feat(model,api,taskworker): bootstrap the egress prober's credential without a human - #441
Merged
Ryanmello07 merged 1 commit intoAug 22, 2026
Conversation
…without a human A deployment that had never had an operator hand-mint a prober credential had no egress probing at all, and said nothing about it. The steps were: create a network, POST /network/auth-client, paste the by_client_jwt into the prober's environment. Miss them and every provider goes unprobed, which is indistinguishable from a fleet that is simply healthy -- the failure is silent in exactly the way that matters. This makes the server do it. A task creates the prober's network and balance, mints its client jwt, and re-mints before the jwt expires; a route hands that credential to the prober. No human step remains between a fresh deployment and a probing prober. prober_identity exists to make a job that re-runs forever safe. It is the only record that the account was already created, and it has to be, because looking the network up by name cannot work: the seedphrase branch of NetworkCreate ignores the requested name and generates a random one, so there is nothing to search for on the next run. The singleton primary key makes "at most one prober identity" a schema fact rather than a convention, so the create is an upsert that collides with itself; create_attempts bounds it if the create keeps failing. Re-mints re-auth the SAME client rather than accumulating one client per refresh. The credential route is deliberately on the same auth as the egress routes it sits beside -- the operator secret, fail-closed when the vault resource is missing. A route that hands out a credential is the strongest possible reason not to invent a second, less-examined way in. Read the note on ProberCredentialResult before touching it: the narrow response body is not the protection. The jwt authenticates as the account, and holding it is enough to regenerate the account's seedphrase -- the operator secret is the whole gate. Both rejection branches log loudly, unlike the endpoints next door. A credential endpoint that rejects everything produces a prober that probes nothing, and that misreading has already cost this system eight hours; the log answers "which side is misconfigured" before it is asked. The migration sits before the competition control-plane block, so it does not run on a database already past that index -- see the note above it, and run `bringyourctl db audit --fix` after deploying to an existing database.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standing up an egress prober currently needs a person to create an account, authorise a balance code, and hand-carry a jwt into the prober's environment. This does that server-side on a schedule instead.
A taskworker task creates the prober's network, funds it, mints a client jwt and stores it;
GET /network/prober-credentialserves that jwt to the prober, authenticated with the same operator secret as the existing/network/provider-egress-*routes — one secret, one mechanism.Idempotence is the load-bearing property, since the task re-arms every 6h forever. Four independent layers: a DB-level claim committed before any account is created (so a crash costs one bounded attempt, not an orphan account nothing can find); a hard cap of 5 creation attempts; a single-assignment write-back so a live identity can never be repointed; and per-step conditionals so a steady-state pass performs no writes at all. Tests cover each, and one of them caught a real orphan-network defect during development — a status flag reported success while an unreachable network sat in the table, and only
count(*) FROM networksaw it.The seedphrase is deliberately discarded. This server holds the jwt signing keys, so the credential can be re-minted indefinitely from the stored ids; a seedphrase is a human login credential and no human logs into a machine-managed identity. Persisting it would put a root credential in postgres, recoverable from any dump, to enable a login nobody performs. The trade — the account is unrecoverable by a human by design — is documented at the discard site so nobody "fixes" it later.
ProberCredentialResultis narrow, but its doc comment says plainly that the narrowness is not the protection: the jwt itself carriesnetwork_id/user_id/network_nameas readable claims, and holding it is enough to call/auth/regenerate-seedphrase. The operator secret is the actual gate. An earlier version of that comment claimed otherwise and was wrong.Rebased onto
dbbdaebcafter #436; thedb_migrations.gocollision with it is resolved (blackhole first, thenprober_identity, both before the competition block).Migration placement — same caveat as #436.
ApplyDbMigrationsUpToiterates from the deployed version, so on an already-deployed database this entry is never reached. Runbringyourctl db audit --fixafter deploy; the statement isIF NOT EXISTSand safe to apply out of band.Build, vet and gofmt clean. Leakage-checked: no beta-only config.