feat(model): diagnose a probe fleet that is failing wholesale - #439
Merged
Ryanmello07 merged 1 commit intoAug 22, 2026
Merged
Conversation
…identically A prober whose platform jwt had been rejected reported EVERY provider as no_consensus for eight hours. Every individual record was well-formed, every endpoint returned 200, the due queue kept handing out work and the prober kept taking it. Per provider the data was indistinguishable from a genuinely unreachable fleet, and nothing anywhere said "credential". The tell was only ever visible in aggregate. When essentially every provider fails, and they all fail the SAME way, the one thing they have in common is not the providers -- it is the prober. That is a statement about the distribution of failures, so it cannot be made per submission; it has to be made over the population, which is what this does. Two properties are load-bearing and should not be optimised away: The success class is excluded from the argmax. provider_egress_probe_attempt stores probe_failure = '' for a successful attempt, so a plain argmax over the tally would diagnose a perfectly healthy fleet as "100% failing with class ''". That is the worst possible output of this function and it is one deleted `continue` away, which is why the guard is written twice. The share is taken over ALL attempts, not over the failures alone. "90% of failures are no_consensus" is unremarkable -- a fleet has a characteristic failure mode. "90% of the entire fleet just failed, all identically" is the incident. The floor of 20 keeps it from crying wolf: three providers failing is evidence about three providers, not about a fleet, and an operator who learns to ignore this message will ignore the real one too. The rule is pure and separately testable from the query, because a warning that never fires is indistinguishable from one that was never written. Nothing calls this yet. It is the diagnosis, deliberately landed before the caller so the rule can be reviewed on its own terms rather than inside whatever sweep ends up emitting it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers the question that cost one deployment eight hours: when every provider comes back failing, is the fleet bad or is the prober's own credential dead? A dominant failure class across the whole fleet is a property of the prober, not of thousands of independent providers.
Reviewer note, stated plainly: this has no callers yet. It is the diagnosis primitive; wiring it into the due-queue handler is a follow-up. Hold this one if you would rather it arrive wired.