fix(board): the working-state snapshot is current or says it is stale (#401)

[mabry1985]

What was wrong

The PM's injected working state showed bd-ezs7 and bd-p8ft in states the store had already moved them out of, and the two were stale in opposite directions. From the audit and agent logs (UTC):

time	event
07:15:08	the health sweep publishes the snapshot, at the start of the sweep: ezs7 in_progress, p8ft blocked
07:16:40	the PM itself calls board_unblock_feature(bd-p8ft), which returns ready
07:17:02	the PM itself calls board_block_feature(bd-ezs7), which returns blocked
07:17:20	the PM writes "bd-p8ft is confirmed board_state=blocked / blocked_class=terminal" and plans around it

The host doesn't cache the snapshot: protoAgent graph/work_providers.collect_work_sections calls the provider every turn. The staleness came from the plugin. The snapshot was republished only by the 300s sweep, before the sweep's own transitions, and it carried nothing that said how old it was.

Fix: how stale the snapshot can be, stated exactly

• Every write marks it stale. Every br write that can move a card (create, update, close, reopen, delete) bumps a process-wide revision, and so does a live config reload of a knob the hints read. The bump happens in a finally around the call, so a write that timed out but may still have landed counts. With fix(loop): a stalled br call costs one read, not the whole tick (#404) #431, a stall reaches that finally only after _run_br_process has stopped the br tree (SIGTERM, then SIGKILL) and reaped it, so nothing the write does can land after the mark.
• A stale snapshot says so. A snapshot records the revision it was read at (the revision is read before the board). When it is behind, the first item is a STALE line naming when it was taken, placed first so a host item cap can't trim it.
• A refresher of its own. It runs separately from the claim tick, so a loop paused at its setup gate still refreshes. It re-reads within 5s of a change (coalesced, so a burst of writes costs one read), and at least every 60s in any case, which bounds writers this process can't see. It backs off exponentially while reads fail. A failed read, fix(loop): a stalled br call costs one read, not the whole tick (#404) #431's BoardTimeout included, is logged as one line with no traceback. Only one read runs at a time: a second one while a read is in flight is skipped. A snapshot older than 120s is marked STALE.
  • The result: the snapshot is either marked STALE or at most about a minute old.
• The sweep publishes last. It publishes after its own transitions. A stall in that read is the sweep's stall, and it goes up to _tick_phase, which ends the tick (fix(loop): a stalled br call costs one read, not the whole tick (#404) #431's rule that the first stall ends the tick). If it were swallowed there, the preflight and the claim scan would each stall on the same wedged store.
• A light read. store.live_cards() is one br list of the open statuses. It adds a br show, in _SHOW_BATCH batches, only for the rows whose hint needs more: blocked cards (their reason and edges) and backlog cards that report dependencies. When a card could be stranded, it also fetches that card's closed dependencies. It replaces the full-board read, which showed every id and scanned br ready (the ~350-id call fix(loop): prevent a stalled br query from aborting the entire reconcile tick #404 caught stalling).
• feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406 compatibility. feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406 landed while this PR was open, and the light read has to keep its "dependencies closed" hints. br list omits dependency edges and closed cards, so, before this change, the light read dropped a stranded backlog card from the view. It also told a cancelled dependency apart from a merged one only by accident. Now both survive the light read, and _report_stranded logs from this read, once per card.
• Blocked hints. A blocked card, ranked first, reads needs a human (<class>): <reason> or retries on its own (<class>): <reason>. (feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406 also annotates the rest with the board's next action.)
• The block-class leftover (the issue's secondary finding).
  • clear_blocked drops the block's class, whatever it is (it used to drop only dispatch-infra), and records the class in its audit comment. It still resets a too-wide park's timeout count (A large card can still time out forever: the design gate is not a breadth cap, and a repeated timeout should decompose rather than park #378), and only for that class.
  • The projection reports blocked_class only while the card is blocked.
  • block_from_review now stamps its own class (terminal) over any stale one, and records a blocked: reason. An old transient can no longer make the sweep auto-heal an exhausted escalation, and boot's preflight-hold release no longer mistakes such a card for an orphaned hold.
  • The loop: an infra dispatch failure climbs the model ladder and leaves the card pinned at the top tier #339 posture is unchanged: tier: labels are never touched.
• The snapshot state lives in a process-stable sys.modules slot, so a plugin reload can't split the counter between the store and the provider. Two stale comments were corrected: failures.py, and a comment in drive.py.

Tests

• tests/test_work_snapshot_staleness_401.py:
  • the incident through real br: the PM's own transitions read back as STALE, and a refresh brings the snapshot current;
  • a write during a read leaves the snapshot stale;
  • the sweep publishes after its own transitions;
  • fix(loop): a stalled br call costs one read, not the whole tick (#404) #431: a br write that commits and then stalls, through a real stalling br wrapper, raises BoardTimeout and still marks the snapshot STALE, while a stalled read marks nothing;
  • fix(loop): a stalled br call costs one read, not the whole tick (#404) #431: a stall in the sweep's snapshot read ends the tick (preflight and claim scan don't run), is logged by the tick with no traceback, and counts toward the refresher's backoff;
  • fix(loop): a stalled br call costs one read, not the whole tick (#404) #431: with the refresher running alongside the ticks and eating a real one-shot stall, the refresher logs one line and keeps running, the next tick still claims the card, and the snapshot comes back current.
• tests/test_stale_blocked_class_401.py (real br):
  • an unblock drops the class and keeps the tiers, for the terminal, dispatch-infra and transient classes;
  • a class left on an unblocked card isn't reported.
• tests/test_work_snapshot_review_401.py, one test per review finding:
  • a failure is logged as one line and backs off, both for a failed call and for a fix(loop): a stalled br call costs one read, not the whole tick (#404) #431 stall;
  • the light read is only list + show of the cards that need detail (real br), and the refresh never takes the full-board read;
  • feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406, real br: the refresher's snapshot names a stranded backlog card with its verb and names a CANCELLED dependency as one, and a backlog card with no dependencies costs no br show;
  • one snapshot read at a time;
  • the refresher picks up a write while the loop is paused;
  • a quiet board is re-read on the age bound;
  • an old snapshot says it's stale;
  • a config reload marks the snapshot stale;
  • blocked hints;
  • block_from_review stamps its class over a stale one and pages instead of self-healing (real br);
  • the counter survives a module reload;
  • boot doesn't release an escalation block as a preflight hold (real br).
  • The write-set tests now stub fix(loop): a stalled br call costs one read, not the whole tick (#404) #431's _run_br_process, the one place a br process runs, where they used to stub subprocess.run.
• Updated existing tests: the test_loop sweep fake serves live_cards, and test_store's unblock test now expects the class to be dropped.
• Dropped: the card-hint test. feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406 fixed the same blank hint on main and has its own test, so this one passed on main and could no longer red-check.

Red-check:

• On main's source (700885d), 28 of the 29 tests in these three files fail. The 29th is the hint test, removed as above. The single-flight test was added after this run; its mutation check is below.
• Mutation checks. Each mutation below makes the named tests fail:
  • the stale mark only on success, with no finally;
  • live_cards showing only blocked rows;
  • no closed-dependency fetch;
  • ignoring dependency_count;
  • the sweep swallowing its stall;
  • the refresher re-raising a stall;
  • a BoardError logged with a traceback;
  • _PROJECTION_WRITES = {"update"};
  • no single-flight;
  • no stranded log from the snapshot read (feat(loop): surface and promote stranded cards whose recorded blockers have cleared #406's own tests catch this one);
  • clear_blocked not resetting a too-wide park's count (A large card can still time out forever: the design gate is not a breadth cap, and a repeated timeout should decompose rather than park #378's test catches this one).
• The earlier round's mutations still fail their tests: the refresher removed from start(), and the revision read after the board.

#431's test_a_stalled_reconcile_read_does_not_cost_the_claim_scan: it is not on main. #431's review round removed it, in 23c014c, when the first stall came to end the tick. Its successor is test_after_a_stall_the_next_tick_claims. In a scratch copy that starts this PR's refresher for every loop #431's tick tests build, all 19 of tests/test_br_timeout_404.py pass, three runs each for the two tick tests. In the next-tick test, the tick hits the stall and the refresher's read succeeds. In the wedged-store test, the tick still makes exactly one stalled call. The refresher then stalls on its own and backs off.

Seam ratchet, recomputed on main with the file's own _external_seams() scanner:

• MAX_UNCOVERED_STORE falls from 18 to 16, because clear_blocked and block_from_review are now REAL.
• live_cards and the new _show_by_id are REAL.
• Store seams are now 33 REAL and 16 UNCOVERED. The worktree seams are unchanged: 24 REAL, 3 EXEMPT, 0 UNCOVERED.

Gates (rebased onto 700885d, which includes #431, #438 and #436): ruff format --check ., ruff check . and pytest -q all pass, with 2139 passed and 16 skipped.

Fixes #401

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2V6GRejF7mNukAoYjj2Av

[protoreview]

QA panel review — PASS

code-review-structural · head e1ace21e994f · formal

This review produced no findings because all five lanes failed on infrastructure (four exceeded their 1200 s budget; the structural lane hit a gateway auth/config error). No code was actually reviewed. The verifier confirmed zero claims to verify and recommends re-running the panel before merging. Do not treat this as a clean pass — it is a complete coverage gap. Re-run the panel (or at minimum the structural and correctness lanes) and gate the merge on a successful pass.

No findings — the review came back clean.

findings JSON (machine-readable)

[]

---

4 panel step(s) hit their time budget and were skipped this round: find_correctness, find_removed_behavior, find_crossfile, find_conventions. The verdict stands on the remaining angles; a finding only that step would have caught could be missed — the next push re-runs the full panel.