feat(prober): hourly blackhole sweep, and fetch the credential instead of hand-carrying it - #5
Merged
Ryanmello07 merged 6 commits intoAug 22, 2026
Conversation
The full pass spends minutes per provider -- ~131 destinations, geolocation
consensus, optionally bandwidth -- so it sweeps a fleet over hours to days. In
that window a provider that silently stops forwarding keeps its last passing
measurement, and every consumer of that measurement keeps believing it. On beta
that meant 98.6% of advertised providers were judged on evidence over six hours
old, and twelve sampled from the stalest cohort answered ok=0/131 while still
being handed to clients.
Nothing about being dark looks different from outside: the provider stays
connected and goes on accepting clients. Only asking it to carry something
finds out.
So this is the cheapest question that still means something -- did ANY traffic
get through -- asked of every provider hourly, on its own loop.
- egresshealth.Blackhole draws up to 3 connectivity destinations and passes on
the FIRST success. Three, not one, so a single destination having a bad
minute cannot condemn a provider; ANY, not all, because a provider reaching
some destinations is degraded rather than dark, and degradation is Check's
department. It reuses fetch, so the table's headers, body caps and Verify
contracts all apply -- a captive portal answering 200 with its own body
fails here exactly as it fails a full run, which is what makes "something
got through" mean anything.
- The sample is drawn fresh per run for the same anti-gaming reason the full
check samples: a provider that knew the three addresses could carry those
and blackhole everything else.
- A tunnel that cannot be opened counts as dark, under its own failure class.
From a client's point of view a provider it cannot build a circuit through
is exactly as useless as one that carries nothing, and not advertising those
is the whole point.
- BlackholeHosts feeds the startup confinement self-check. If the prober could
reach these directly, a provider carrying nothing would still be recorded ok
-- the check would confirm the prober's own connectivity, remove nothing
forever, and look healthy doing it.
- The sweeper stops on ErrBlackholeUnsupported and ErrUnauthorized, and only
those. An old server will never grow the endpoint mid-run, and a rejected
secret is a broken deployment; everything else is retried next tick, so a
momentary server outage cannot silently end blackhole detection for the life
of the process.
Results are submitted as one batch per pass: a sweep produces hundreds of
one-bit answers and a request each would spend more on http than on the checks.
Defaults: -blackhole-interval 1h, -blackhole-limit 500, -blackhole-concurrency
32. Concurrency is higher than the full pass because a check is one round trip
through the tunnel rather than a 131-destination sweep. 0 disables the sweep.
The sweeper reused -probe-timeout (2m20s on beta) as its per-request deadline, which defeats the point of the check. A dark provider burns the full timeout on every destination, so a 500-provider batch at concurrency 32 takes ~37 minutes rather than ~4 -- on a fleet that is mostly dark, which is exactly the fleet this exists to detect, the cheap loop runs as slowly as the full pass it was meant to complement. 15s: a provider that carries traffic answers a 204 probe in well under that, and one that does not must fail fast for the sweep to stay hourly.
A pass took one batch and slept for the interval. -blackhole-limit is the server's per-request ceiling, not the size of the fleet: at 500 per request against ~2,700 eligible providers that covers under a fifth of them per hour, so the oldest evidence ages out faster than the sweep reaches it -- and the requirement is that the whole fleet is checked every interval. Now a pass keeps requesting until the queue returns nothing, bounded at 40 rounds so a server that keeps handing back work cannot hold a pass open forever and starve the interval.
The prober required UR_PROBER_BY_JWT to be provisioned and placed by hand.
The server now mints the prober's network client identity in a bootstrap
task and serves it at GET /network/prober-credential, authenticated with
the same X-UR-Operator-Secret the other operator endpoints use -- so the
last hand-placed secret can go.
ingest.Client.ProberCredential fetches it, with three disjoint outcomes so
a caller cannot take the wrong branch: 404 is ErrCredentialNotReady ("not
yet", keep asking), 401 is a bare ErrUnauthorized (a wrong secret must be
loud and must never be retried forever), everything else is
ErrCredentialUnavailable (transient, retry). A 200 carrying no usable
by_client_jwt is refused rather than returned empty -- decodable is not
usable, the same lesson GeolocationPins learned about a nil pin map.
cmd/egress-prober fills an empty -by-jwt from it, with envFallback's
precedence: an explicitly supplied jwt wins and the endpoint is never
contacted, so the running deployment is untouched and gains no dependency
on it. The fetch sits above parseByJwtClientId, so a fetched credential
takes the identical path a supplied one does and a token this process
cannot use fails at startup instead of becoming a silent outage. The
bootstrap task runs every 6h, so a 404 is a wait on a backoff capped at
5m, ended only by ctx -- an interrupt there exits zero, since a prober
interrupted while waiting is a shutdown, not a broken deployment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtgqtCmKJRXdsQ5ktiqwkg
encoding/json populates the fields it could before returning an UnmarshalTypeError, so a body whose client_id is not a string yields a usable-looking jwt alongside the error. Every other unusable body is backstopped by the emptiness check; this is the only case that fails if the decode error is ignored.
# Conflicts: # cmd/egress-prober/credential_test.go # cmd/egress-prober/main.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes that go together, because the second's wiring sits in code the first adds.
Blackhole sweep
The full pass spends minutes per provider, so it sweeps a fleet over hours to days. In that window a provider that stops carrying traffic keeps its last passing measurement — it stays connected and keeps accepting clients, so nothing else reveals it. On beta, 12 of 12 providers sampled from the stalest cohort answered
ok=0/131while still being advertised.So this asks the cheapest question that still means something — did ANY traffic get through — of every provider hourly, on its own loop. It draws 3 connectivity destinations and passes on the first success, reusing
fetchso the table's headers, body caps and Verify contracts all apply; a captive portal answering 200 with its own body fails here exactly as it fails a full run. The sample is redrawn per run, so a provider cannot learn which three to keep alive.BlackholeHostsfeeds the startup confinement self-check: if the prober could reach these directly, a provider carrying nothing would still be recorded ok.Two defects found in live testing, both fixed here: the sweep reused
-probe-timeout(2m20s), which on a mostly-dark fleet made the cheap loop as slow as the full pass — it now has its own 15s deadline; and a pass took one batch and slept, covering under a fifth of the fleet per hour — it now drains the queue.Credential fetch
Pairs with
GET /network/prober-credential(urnetwork/server#441). WhenUR_PROBER_BY_JWTis empty the prober fetches its jwt with the operator secret it already holds, instead of an operator hand-carrying one. An explicitly supplied jwt still wins, so existing deployments are unaffected — that is pinned by a test.404 means "not ready, keep polling" with backoff rather than a fatal exit, since the server-side bootstrap task runs on its own schedule. A 401 stops the prober loudly: this system already lost 8 hours to a rejected credential that surfaced as every provider reporting
no_consensus, and a silent retry loop would reproduce exactly that. Fetched jwts go through the same startupcheckCredentialthat was added for that incident.Merged with upstream/main; the two conflicts were combined, not chosen between — upstream's
-intervalhelp-text correction is kept alongside our blackhole flags, and both sets of credential tests are retained.go build ./...andgo test -race ./...clean across all 8 packages.