Add Trino cell observability to the admin console

[fuziontech]

Trino is becoming the primary query execution engine for duckgres, and the admin console had no Trino surface at all — everything Trino lived in provisioner/ and provisioner/opa/. A failed Trino provision was silent unless someone read duckgres_managed_warehouse_trino by hand, and a runaway tenant query could only be killed by someone with cluster access.

This adds the console's Trino views, the identity they need, and fixes an OPA gap found on the way.

What you get

Trino queries — live queries across the cell, longest-running first, each stamped with the duckgres org resolved from the Trino principal. Flags blocked queries (every driver waiting on the metadata store or S3) separately from merely slow ones. Admin-only, audited kill.

Trino cell — coordinator version / environment / uptime / starting, the failure detector's view of the fleet, and every tenant's provisioning state with the reconcile loop's own status message.

Org detail card — that org's state, status_message, ready_at, failed_at, principal, catalog, tier, cell, and live query counts.

The security decision to review

The console reads the coordinator as a new __duckgres_observer principal, deliberately not as __admin_provisioner. Trino gates all of this through the access-control SPI (/v1/query filters through FilterViewQueryOwnedBy, /v1/query/{id} on ViewQueryOwnedBy, kill on KillQueryOwnedBy, /v1/node + /v1/resourceGroupState are MANAGEMENT_READ), so a console credential with no grant sees an empty cluster.

The split is the bargain:

	catalog authority	query visibility
__admin_provisioner	CREATE/DROP CATALOG	own queries only
__duckgres_observer	none	cluster-wide (+ kill)

The observer holds no entry in data.group_catalogs, and the observer group is excluded from tenant_owns_catalog and from the same-org query match, so even a mistaken bundle entry grants no data access. is_observer is the same username-AND-group conjunction as is_admin. One leaked credential yields one half of that authority, never both — which is why this is a second principal rather than a widening of the first.

policy.rego is flagged as security review in CLAUDE.md, so that's the part to read closely.

Bug found along the way

The policy had no batch rule for FilterViewQueryOwnedBy. opa.policy.batched-uri is enabled (without it, filtering a catalog with >1024 tables overruns the OPA client's queue), and OpaBatchAccessControl sends query-owner candidates through it exactly like the catalog filters. With no batch rule the batched answer was the empty set, so the same-org query visibility the policy documents has been inert in production. It didn't look like a failure: Trino short-circuits self-ownership before OPA, so every tenant still saw its own query and the gap read as "org-mates' queries are missing".

allow and batch now share one predicate parameterized on the owner, and TestBatchedQueryFilteringMatchesNonBatched pins them candidate by candidate.

Implementation notes worth a look

• Redaction at decode. SQL is redacted in toTrinoQuery, not per-caller, so no handler can leak it by forgetting. TrinoEnabledOrg.RootPasswordHash never crosses the projection boundary (there's a test that greps every payload for $2a$10$).
• Airlift units are strings. Duration serializes as "%.2f<unit>", DataSize as an exact byte count with a B suffix. Decoding either as a JSON number gives a plausible page of zeroes. Both parsers return 0 on anything unreadable, so a unit-spelling change on a coordinator upgrade costs one column rather than the whole view.
• Cached, and stale-over-blocking. The console polls from every open tab and /v1/query walks every query the coordinator holds. Readers during an in-flight refresh get the previous value, so a slow coordinator can't turn N tabs into N stuck requests during the incident the console exists for.
• Degrades, doesn't error. An unreachable coordinator gives available:false + a reason, with provisioning state still served from the config store — "the cell is down" and "these tenants never provisioned" are different incidents. A 403 is reported distinctly, because that means the bundle hasn't rolled out rather than the cell being down.
• Both sides tag X-Trino-Source (duckgres-provisioner, duckgres-admin), so control-plane traffic is separable from tenant SQL in system.runtime.queries.
• No cell configured (DUCKGRES_TRINO_COORDINATOR_URL unset) leaves every route unregistered.

Known limits

• /v1/node reports heartbeat health keyed by URI and carries no node id or version, so worker version skew is not visible here — the Nodes page's pod-image projection is where that lives. Called out in the UI.
• No tests/mw-dev/e2e/harness.sh coverage, for the same reason the rest of the Trino code has none: mw-dev runs no Trino cell. The harness assertion lands with the chart that deploys one.

Deliberately not in this PR

Durable query history via the trino-http-event-listener plugin (unlocks a Trino errors page and Trino CPU/bytes attribution in billing, which is currently invisible), /metrics scraping into the metrics-proxy allow-list, and DuckLake data-layer health (small-file ratio, delete debt). All three need chart changes in PostHog/charts, so they're follow-ups.

Testing

• golangci-lint run — 0 issues (also 0 in the new files under --build-tags kubernetes; the 62 that tag surfaces are all pre-existing).
• Go: controlplane/... green. The 21 *Postgres failures on this machine are pre-existing — identical count on an untouched checkout, a stale local test DB missing max_hot_idle_workers.
• UI: 160/160 vitest, tsc -b --noEmit clean, eslint clean on new files.
• New coverage: OPA observer + batched-filtering matrix, credential mint/adopt/regenerate/fail-loud, auth-file projection, bundle-grants-no-catalog, coordinator client (airlift parsing, redaction, 403/410 handling), handler behaviour (org annotation, degradation, kill gating + audit, caching), and the UI derivations.

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	2	3	0
E2E/journey files	0	0	0
Workflow files	0	0	0

Signals

• Test cases: +45 / -0
• Assertions: +162 / -6
• Skips or known failures added: 0
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

[fuziontech]

Review

Read the whole diff again with fresh eyes. Three real findings, all now fixed in 4f94932.

1. active=1 hid five in-flight states (the significant one)

The server's active filter allowlisted {RUNNING, QUEUED}. Trino has nine query states and only FINISHED/FAILED are terminal (QueryState.isDone()), so the live view — which requests active=1 by default — silently dropped WAITING_FOR_RESOURCES, DISPATCHING, PLANNING, STARTING and FINISHING.

PLANNING is the one that stings. On a DuckLake-backed cell, planning talks to the tenant's metadata Postgres, so "stuck in PLANNING" is among the most likely things an operator opens this page to find — and it was invisible. Worse, the UI's isActiveTrinoQuery used a different allowlist that included PLANNING/STARTING, so the two sides disagreed about what "active" meant.

Both now define active as the complement of the terminal pair, mirroring QueryState.isDone(), so they can't drift. TestActiveFilterKeepsEveryNonTerminalState walks all nine.

2. ReadSystemInformation is wider than the comment claimed

It's a single operation gating every MANAGEMENT_READ resource, not just the two the console reads. The real set is /v1/node, /v1/resourceGroupState, /v1/thread, /v1/announce (GET), /v1/maxActiveSplits, /v1/integrations/gateway.

All six are GETs of cluster-operational state; none reads a catalog, a table, or SQL text. /v1/thread is the widest (thread names embed query/task ids). The node-registering POST on /v1/announce is INTERNAL_ONLY and needs the node-to-node shared secret, so this grant can't join a fake worker to the cell.

Not a hole, but understating the blast radius of a grant in a file we treat as security review is the wrong default. The set is now enumerated where the rule is written.

3. Duplicate config-store read per request

handleStatus, handleOrgs and handleOrgDetail each built the principal index and then had liveQueries build it again. /trino/status is polled every few seconds by every open tab, so that join was the most-repeated query on the surface. Handlers now pass the index they already hold.

Checked and found fine

• ResourceGroupId serializes via @JsonValue getSegments() as a JSON array, so the []string decode is right (I'd initially suspected a dotted string).
• /v1/query/{id} returns QueryInfo, which does carry resourceGroupId and every queryStats field the client decodes — the detail path is not a degraded shape.
• liveQueries copies before stamping Org, so the shared cache slice is never mutated; the filter/sort path allocates its own backing array.
• The cache's stale-over-blocking path is race-free (populated is read under the lock).
• handleStatus reads /v1/info first because it's PUBLIC in Trino — that's what makes "the cell is down" distinguishable from "the console can't see it", and the 403 path does reach trinoUnavailableReason.
• failureInfo on QueryInfo is deliberately not decoded — it can carry SQL fragments.

Verification after the fixes: golangci-lint run 0 issues, Go tests green, UI 160/160, typecheck and eslint clean.