Skip to content

Wait, don't fail, when a tenant's password file hasn't been mounted yet - #1135

Merged
fuziontech merged 1 commit into
mainfrom
trino-pending-on-secret-mount
Aug 28, 2026
Merged

Wait, don't fail, when a tenant's password file hasn't been mounted yet#1135
fuziontech merged 1 commit into
mainfrom
trino-pending-on-secret-mount

Conversation

@fuziontech

Copy link
Copy Markdown
Member

What

Enabling Trino for an org projects its password onto the tenant Secret and creates the catalog in the same reconcile tick. The Trino pods read that Secret through a mounted volume, so for up to a kubelet sync period the file the catalog points at genuinely does not exist yet, and the coordinator rejects CREATE CATALOG:

Invalid configuration property ducklake.metadata.connection-password-file:
file does not exist: /etc/trino/tenant-secrets/<org>

That window was reported as a failure: the org was stamped Failed with failed_at set and a Trino configuration error parked in status_message, and the tick logged Trino provisioner reconcile failed — for an org that goes Ready on its own about a minute later, unaided.

This classifies that one case as Pending with a reason, which is what the reconcile loop already does for every other "not ready yet" input (org has no managed warehouse row yet, waiting for the duckling to publish a metadata-store credential).

Why now

Observed in production while enabling an org by hand. It matters more from here on: PostHog/posthog#91098 opts every newly onboarded org into a cell, so this stops being a rare manual path and becomes what onboarding normally does. Without this, routine onboarding writes a spurious Failed state and a WARN every time.

Why the match is narrow

Getting this wrong in the other direction would silently downgrade a real failure to "still waiting", so:

  • Trino's statement error is now a typed *TrinoStatementError carrying errorName/errorType/message, instead of being flattened into a string at the point of creation. The classifier uses errors.As, so the type is authoritative — the same principle isInstanceFatalError follows for DuckDB.
  • Within that, it matches the single exact sentence Trino emits, built from our own property name and our own computed path for this org — not a set of loose substrings.

That last point is deliberate. Catalog properties carry tenant-influenced values (bucket names, endpoints) that Trino echoes back in configuration errors, so a loose match could be tripped by an org's own data — the trap CLAUDE.md already documents for DuckDB echoing query text back into INTERNAL Error messages. There's a test for exactly that shape.

If Trino ever rewords the message, the match stops firing and the org fails loudly again — today's behavior, not a new silent state.

TrinoStatementError.Error() reproduces the previous flattened text verbatim, so logs and status messages are otherwise unchanged.

Tests

go test -tags kubernetes ./controlplane/provisioner/... — green.

  • TestReconcile_TenantSecretMountLagIsPendingNotFailed — reproduces the production error verbatim. Asserts the tick returns no error at all (so the loop doesn't log a failure), the org is Provisioning with the mount-lag reason, failed_at is not stamped, and it converges to Ready on the next tick once the file appears.
  • TestTenantSecretNotMountedYetOnlyMatchesThisOrgsMissingPasswordFile — the guard rail. Another org's password file, a different property missing a different file, tenant data echoed back inside an unrelated config error, the same text as an untyped error, an unrelated 503, and nil all stay failures.

No tests/e2e-mw-dev/ change: this touches Trino catalog reconcile classification, none of the activation-pipeline paths CLAUDE.md lists as obligating harness updates, and adds no columns.

Pre-existing on origin/main and unrelated to this change: three controlplane/admin Postgres tests fail locally against a stale local schema (column "max_hot_idle_workers" ... does not exist). Verified by stashing this branch and re-running on a clean tree.

Enabling Trino for an org projects its password onto the tenant Secret and
creates the catalog in the same reconcile tick. The Trino pods read that
Secret through a mounted volume, so for up to a kubelet sync period the file
the catalog points at genuinely does not exist yet, and the coordinator
rejects CREATE CATALOG with a configuration error.

That window was reported as a failure. The org was stamped Failed with
failed_at set and a Trino configuration error in status_message, and the tick
logged "Trino provisioner reconcile failed" — for an org that goes Ready on
its own a minute later, unaided. Observed in production while enabling an org
by hand; it now matters more, because PostHog is about to opt every newly
onboarded org into a cell, so this becomes the normal onboarding path rather
than a rare manual one.

Classify that one case as Pending with a reason, which is what the reconcile
loop already does for every other "not ready yet" input.

Getting the classification wrong in the other direction would silently
downgrade a real failure to "still waiting", so the match is narrow. Trino's
statement error is now a typed *TrinoStatementError carrying errorName and
message rather than a flattened string, and the classifier matches the single
exact sentence Trino emits, built from our own property name and our own
computed path for THIS org. Catalog properties carry tenant-influenced values
that Trino echoes back in configuration errors, so a looser match could be
tripped by an org's own data — the trap isInstanceFatalError already
documents for DuckDB's echoed query text. If Trino rewords the message the
match stops firing and the org fails loudly again, which is today's behavior.

TrinoStatementError.Error() reproduces the previous text verbatim, so logs
and status messages are otherwise unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012qyribmtBS2uxYrrM3pACb
@fuziontech
fuziontech requested a review from a team August 28, 2026 19:30
@github-actions

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 1 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +2 / -0
  • Assertions: +8 / -1
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@fuziontech
fuziontech merged commit 388bebd into main Aug 28, 2026
31 checks passed
@fuziontech
fuziontech deleted the trino-pending-on-secret-mount branch August 28, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant