A customer portal that can configure, prove, fill and watch a deployment

[bjmeetsfo]

The portal could show which model endpoints a deployment talks to and could not change any of them, so configuring a managed deployment meant shell access to the host. It had no way to list the skills and resources it holds, no way to run the one check that covers the whole pipeline, and /v1/metrics carried only the bulk-import registry — the customer-facing API surface was invisible to the dashboards while the engine below it was fully instrumented.

Seven pages now, all served by the gateway itself.

Page	What it does
/v1/admin	Readiness checklist, each unfinished item carrying the steps to fix it; live import progress; a diagnostics bundle
/v1/admin/setup	79 settings (12 marked required), endpoint probe, live traffic, read-only deployment inventory, change log, export/import
/v1/admin/catalog	List and filter skills and resources; read the stored text; retire a superseded skill
/v1/admin/explore	Run a real retrieve, add a memory, upload documents with byte progress, browse what is stored
/v1/admin/ingestion	Import a directory, watch it in detail, retry what failed
/v1/admin/portal	Keys, with a scope picker that says what each scope allows
/v1/admin/api	All 45 routes, each with the scope it needs and a curl that runs

Six tests that verify rather than assert

Documentation that drifts is worse than none, because it is trusted. Each of these compares a claim against the source and failed on its first run:

• The live/needs restart label is checked by AST-walking every module under tools/ and classifying each environment read as import-time or per-call. It caught 20 mislabelled settings and one phantom knob that nothing outside the display function reads.
• The route list is compared to the gateway's own path literals in both directions, and every documented example is executed and required to answer 2xx. It caught /v1/admin/routes undocumented and /v1/memory/by-key's example missing its required parameter.
• The scope catalogue is pinned to MATRIXARK_TOOL_SCOPES both ways. It caught admin:sso.
• The generated pages are regenerated in a temp copy and required to come out byte-identical.
• The required-settings mark: a required key must exist, must not be a tuning knob, and a secret is required only alongside the variable naming its destination.
• A timed concurrency guard on the readiness report: sequential 188 ms against a 60 ms stub, concurrent 68 ms.

Five defects found by building on top of the code

• post_document gave up immediately on any 4xx — right for a 400, exactly backwards for a 429. The gateway rate-limits ingest and answers 429 with retry-after, so a large import against a busy deployment was abandoning precisely the documents it had been asked to resend. The job reported 1000 documents; the store held fewer, and nothing said so. Mutation-checked: 5 of the 6 new tests fail without the fix.
• X-Scope on /v1/ingest_file is documented as a JSON scope object and was passed through unparsed. It reached the string branch of _apply_identity, which reads a bare string as a namespace label, so an upload carrying {"user_id":"alice"} was filed under acme/{"user_id":"alice"} with the user_id dropped. Silent; the only symptom is a retrieve that later cannot find it.
• An unreachable blob tier on that route escaped as an unhandled 500 with a stack trace. Now a 502 naming the URL it could not reach.
• --faint, the token carrying the help text under every field, measured 3.85:1 in dark and 3.04:1 in light — below AA, on exactly the prose explaining what each setting does.
• .check meant both a checkbox label and a readiness row; the collision was already papered over with inline styles.

Safety posture

• The writable settings are a closed registry; an unregistered key is a 400. Storage locations, cluster addresses and the auth mode are a read-only inventory — repointing a storage directory from a browser form does not reconfigure a running deployment, it strands its data.
• Secrets are write-only: stored owner-only (0600), pushed into the variable named by the matching *_API_KEY_ENV, never returned by any read, and omitted rather than blanked from an export, because a blank is a write that would clear the target's key.
• GET /v1/metrics stays credential-free: aggregate counters only, and the route label is a template, so a client cannot create unbounded series in an operator's Prometheus.

Also

GET /v1/skills, GET /v1/resources, GET /v1/users and POST /v1/skills/update — the backend has had the listing tools since the skill lane landed, reachable only through /v1/mcp. Grafana dashboard and alert rules for the edge. Accessibility: role="status" + aria-live on every region that changes, tab roles on Explore, both colour schemes checked at 375 px with no horizontal overflow.

Five of the seven pages are generated by tools/portal/build_portal_pages.py; the other two are hand-maintained and only have their nav refreshed.

270 tests across the portal, ingestion and gateway suites.

---

Choosing models, driving the memory API, and batching a paste

Both model names were free text on the wire, and a name that is merely misspelt does not fail
loudly: extraction falls back to the local rules and embedding to hash vectors, and both answer
200. The deployment looks healthy while storing far less than it was asked to.

GET /v1/admin/models answers three questions per role — what is worth suggesting, what the
configured endpoint says it actually serves (an OpenAI-compatible server answers `GET

/models`, so asking it turns a guess into a choice), and for embeddings what the stored vectors were actually made with. The probe is opt-in; it is a request against somebody else's service.

Changing the encoder is the one setting on the page that can silently invalidate data already
held. The dangerous case is not a width change — that at least has a width to notice — but two
different encoders of the same width, where nothing in the stack sees a mismatch:
all-MiniLM-L6-v2 and paraphrase-multilingual-MiniLM-L12-v2 are both 384, BAAI/bge-m3 and
voyage-3 are both 1024. Each catalogue entry states its width and names the others it collides
with, derived from the widths rather than written into prose — the hand-written note called out the
384 pair and said nothing about the 1024 one, which a test caught. The warning appears while the
choice is being made, and only when it would actually strand something.

Three engine guards make that change survivable rather than silent. All three were broken and none
had a test:

1. similarity scored the shared prefix of two vectors of different widths — a plausible cosine
   across two unrelated spaces, worse than no number because it ranks. It now refuses;
2. ingest hashed provider.model, the chat model, while the drainer hashed
   provider.embedding_model, so the same encoder produced two hashes depending on which path
   embedded the node;
3. retrieve scored a node embedded by one encoder against a query embedded by another. A vector
   whose recorded encoder is not the active one now falls through to the hybrid lexical pass. A
   hash of 0 means unknown, not foreign, and is still scored.

Six Rust tests, each verified by breaking its guard again on purpose — including the
over-correction that would treat unknown as foreign. Every mutation turns exactly one test red.

A console over the memory API on Explore, listing every method by the name it is called by
rather than by the URL that serves it, generated from one table so an operation that gains an
argument gains a field. Tests hold the table to the routes in both directions, and check each
declared scope against what the gateway gates on. Destructive operations are marked from the
route's own scope and will not run until their name is typed.

Batch ingest takes a paste or a file, one memory per line, and runs it as a server-side job
rather than a loop in the tab — a browser loop over ten thousand records is one closed laptop away
from a half-finished import nobody can resume or count. Job items are now either a path or a
record, so a batch inherits the progress, failure classification, retry-only-the-retryable, cancel
and metrics the directory import already had. Blank lines are dropped and counted rather than
queued to fail forever.

Also fixes the settings audit, which matched a startup function by the outermost segment of a
qualname; the request handler is nested inside the app factory, so every per-request read inside it
read as captured-at-boot. Nothing had noticed because no tracked setting was read there until now.

361 tests.