[howethomas]

Adds storage.utopia, a storage module that sends each vCon to a Utopia knowledge base. Utopia extracts facts from documents, dates each fact on a world clock and a record clock, and keeps the quoted evidence, so a conserver feeding it can answer "what was said about X as of date Y" with the source vCon behind each answer.

Design

• Render, then upload. Utopia reads JSON as flat text and dates a document only from a leading YYYY-MM-DD line in a Markdown file. The module renders the vCon as dated sentences (On 2026-09-18 at 14:02, A and B had a recorded call lasting 240 seconds.) and uploads <uuid>.md.
• Filename is the key. Utopia assigns its own document ids, so the module finds a vCon's document by filename. No mapping table.
• save uploads first, then tombstones the older <uuid>.md, so a failed upload loses nothing. Unchanged content is skipped by Utopia's sha256 dedup, which also makes retried POSTs safe; the retry session is the one storage.vcon_mcp already uses.
• delete tombstones (reversible in Utopia). get is not offered, since Utopia holds the rendering, not the vCon.
• Auth. Utopia's REST API takes session JWTs only, so the module logs in with a service account and logs in again on a 401.

Lawful basis

Every rendering ends with a lawful basis sentence, or Lawful basis: none recorded. Nothing is filled in. require_analysis_grant (default off) skips vCons whose basis does not grant analysis or has expired.

Tests

common/storage/utopia/test_utopia.py, 7 tests: rendering of a legacy 0.0.1 record and a current one with wtf_transcription and lawful_basis, and save, re-save, supersede, delete, 401 re-login and the grant gate against a loopback fake of Utopia's four endpoints.

uv run pytest common/storage/utopia -q   # 7 passed

Live run

Run once against a local Utopia (ghcr.io/deeplethe/utopia:0.1.0-rc6, qwen2.5:7b for extraction, nomic-embed-text for embeddings), pushing one synthetic record through the real storage.utopia code. All four steps passed:

1. Upload lands in the knowledge base.
2. An identical re-save is skipped by Utopia's sha256 dedup; nothing is tombstoned.
3. Utopia dates the document from its first line and runs fact extraction.
4. A changed re-save uploads a new document and tombstones the old one.

On the 7B model, extraction was thin: five facts from one call, mostly name aliases, none from the party or dialog sentences.

Rerun on a droplet with a hosted chat model over the OpenAI-compatible API, same record, same four steps passing. Extraction then gave 16 facts, including the rendered sentences: each party taking part, their phone and email, who had a conversation with whom, and a callback request from the amended summary. So the rendering works; the thin local result was the model.

That run showed one gap, fixed in cbc84ad: only 1 of 16 facts carried a world time (valid_from), because Utopia extracts chunk by chunk and summary and transcript paragraphs had no date of their own. Each paragraph now opens with On <date>,. Against the same Utopia, dated facts went to 10 of 24. The rest are mostly timeless attributes such as names and contact details.

Not in this PR

• Purging, rather than tombstoning, the original when a redacted vCon arrives (needs KB Admin and Utopia's purge endpoint).
• Seeding party tel / mailto as Utopia entity aliases.

🤖 Generated with Claude Code