fix(loop): max-mode hands the drive the failure that killed every candidate (#425)

[mabry1985]

Fixes #425.

What was wrong

_dispatch_max_mode swallows each candidate's exception. When every candidate died, the drive only ever saw NoChangesError("max-mode: all N candidates produced no diff"). That is a capability verdict, so the card climbed a rung, no matter what had actually killed the candidates.

Reproduced on origin/main through _drive, with rung [codex, sonnet], max_mode_n: 2 and codex refusing its model:

dispatched: codex, codex, opus, opus   escalated ×2   blocked terminal   codex marked: False

That's the trace from the #421 review. The same swallow:

• turned a quota on every candidate into a climb
• blocked a one-provider board as an unexplained terminal no-diff
• hid a timeout from A large card can still time out forever: the design gate is not a breadth cap, and a repeated timeout should decompose rather than park #378's timeout counter and decompose ask
• hid a pre-model seam failure from loop: an infra dispatch failure climbs the model ladder and leaves the card pinned at the top tier #339's block
• climbed on a network blip that a single dispatch would have backed off and retried

The fix

When every candidate raised, no candidate returned anything for the selector to judge. So max-mode re-raises the one error that speaks for them, and from there the drive's own handling applies exactly as for a single dispatch. representative_failure() in loop/_common.py picks it, most specific edge first:

rank	failure	the drive then…
0	a refused model (also beside a quota: the provider is at least partly broken)	marks it, rotates within the rung, or blocks dispatch-infra (#420)
1	a spent quota	rotates, or backs off (#362)
2	a CoderTimeout	counts it for #378, and climbs with #146's note (fanning out again), or blocks pre-first-token as infra
3	a dispatch failure the drive does not retry	#339 blocks it as pre-model unless a candidate reached the model
4	any other dispatch failure (retryable)	backs off and re-runs the fan-out
5	any other WorktreeError	as raised

Ties go to the earliest candidate. Only a WorktreeError can speak. A raw error (say a BrokenPipeError from an untapped host) keeps the old no-diff verdict, which the drive's shutdown and cancel checks see first. Only a fan-out where at least one candidate RETURNED (ran and came back with nothing) is guaranteed to be a capability climb. A candidate that left a diff still wins exactly as before. provider_failure_category() is the one shared definition of a provider failure, used by the drive's rotation and by the ranking.

Found in review, also fixed

• One all-timeout fan-out switched max-mode off for the card for good. The timeout climb put its "prior attempt timed out" note (Retry paths don't propagate failure context — the search repeats the same failure N times, and a timeout escalation upgrades the model while telling it nothing #146) in _ci_feedback. That slot marks a carried-forward fix, which disables fan-out, and it outlives the drive. So the climbed rung ran ONE dispatch, and every later drive of the card also ran single, opening with "a PREVIOUS attempt TIMED OUT". The note is now drive-local: the climb is a fresh build and the new rung fans out again (codex×2 → opus×2). It's spent once the dispatch that carried it has run, and dropped on any other climb.
• The note described the wrong candidate. It was mined from the last gen, which in a fan-out may be a sibling that failed fast ("ran ~0.0s", no tool). The tap now stamps the stop reason of the gen its watchdog killed, on every dispatch path, and the note is mined from that gen.
• A raw error during shutdown blocked the card as unexpected. It's no longer re-raised (see above).

Merge order with #435. #429 on its own blocks a card whose fan-outs keep timing out as transient, and the sweep then rebuilds it twice more. #435 closes that by parking it for a split. The drive-level test of the merged behaviour lives here (test_a_fan_out_that_keeps_timing_out_is_parked_for_its_split). It skips, naming #435, until the park is on the branch, and passes on the three-way merge.

Tests (tests/test_max_mode_provider_425.py)

Covers each precedence boundary, the fallback, and the review items:

• refused on every candidate → same-rung switch, codex marked. One-provider board → dispatch-infra with the evidence. Refusal + quota → the refusal edge.
• quota on every candidate → switch with no backoff, not marked. Quota + timeout → the quota edge; the timeout is not counted.
• timeout + seam failure → A large card can still time out forever: the design gate is not a breadth cap, and a repeated timeout should decompose rather than park #378's counter moves and the climb carries "TIMED OUT".
• seam failure + network blip → blocked as pre-model, one fan-out, no backoff. Network blip on every candidate → re-run after backoff max_attempts times, then transient, never climbs.
• fallback: one candidate refused and one returned empty climbs. Tested both with a stand-in tap and through the real dispatch_coder_tapped.
• review:
  • a climb after an all-timeout fan-out fans out again, asserting the full sequence codex×2 → opus×2;
  • a later drive of the card fans out again with no stale note;
  • the note is spent once the climbed dispatch has run;
  • through the real tap, the note describes the candidate that timed out;
  • every candidate is reaped before the drive handles the failure;
  • a CancelledError child is neither returned nor raised;
  • a raw error on every candidate keeps the shutdown edge.
• unit: the full precedence (every rank beats every later one; ties go to the first; a raw error never speaks), and provider_failure_category matches the drive's definition.

One existing test changes: test_drive_carries_timeout_context_into_the_escalated_prompt pinned the note inside _ci_feedback, which was the bug. It now asserts the note did not land there.

Red-checks: against origin/main, the precedence tests fail with the reproduction above. Against the previous head (6d7a24f), the six review tests that change behaviour fail; the three guards pass there by design. All 10 of the reviewer's reproducers pass on this head.

Mutation checks (8 review mutations plus the earlier precedence ones), each caught:

• the note back in _ci_feedback;
• the last gen again;
• no stamp on the tap;
• raw errors speaking;
• reaping only on the no-diff path;
• a cancelled child counted as raised;
• any raised candidate speaking;
• a note that is never spent;
• each adjacent pair of ranks swapped.

Gates: ruff format --check, ruff check, full suite 1922 passed / 14 skipped (the +1 skip is the merge-only test).

Note: this conflicts textually with #434 in _dispatch_max_mode. Keep #434's discard-by-path and this PR's re-raise.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2V6GRejF7mNukAoYjj2Av

[protoreview]

QA panel review — PASS

code-review-structural · head 80b96469439f · formal

⚠️ PR advanced 1 commit(s) during this round (80b96469439f → 21e1b85f1896); 0 finding(s) in the delta were demoted to possibly addressed.

Overall risk is HIGH: 4 of 5 finders failed (timeouts, truncation, gateway auth), so the production-code change — extracting provider_failure_category into _common.py and wiring it into _drive and _dispatch_max_mode — went effectively unreviewed by the panel. The single surviving finder produced no findings, and the verifier had an empty array to verify. Fix-first: re-run the failed finders (and the skipped structural pass) before merging; do not treat this round as a clean pass. The panel did not disagree on anything because it produced nothing. Coverage gap: the heavily-changed production files (_common.py, the _drive call site, the _dispatch_max_mode re-raise path) received zero substantive review; the test file and changelog likewise went unexamined.

No findings — the review came back clean.

findings JSON (machine-readable)

[]

---

2 panel step(s) hit their time budget and were skipped this round: find_correctness, find_removed_behavior. The verdict stands on the remaining angles; a finding only that step would have caught could be missed — the next push re-runs the full panel.

[protoreview]

⚠️ QA panel exhausted — this PR has not been reviewed.
The review panel failed after 2 attempt(s) on head 6d7a24f48a96. No verdict was posted.
A new push will re-trigger the review.

[protoreview]

⚠️ QA panel exhausted — this PR has not been reviewed.
The review panel failed after 2 attempt(s) on head ea19792cd52c. No verdict was posted.
A new push will re-trigger the review.