fix(board): the working-state snapshot is current or says it is stale (#401)

changelog.d/433.md

@@ -0,0 +1,17 @@
+The agent's working state no longer shows board cards in states they have already left. The
+board's snapshot of open cards was refreshed only by the 300s health sweep, and at the start of
+it. A PM that unblocked one card and blocked another was then shown both in their old states,
+with nothing saying the view was old, and it planned against that. Every board write now marks
+the snapshot stale, including a write that timed out and may still have landed. So does a live
+change to a setting the hints read. A stale snapshot says so in its first line and names when it
+was taken. A refresher of its own, not part of the claim tick, re-reads the board within 5s of a
+change, and at least every 60s regardless. So the view is either marked stale or at most about a
+minute old, even while the loop is paused or another process writes the board. It reads only the
+open cards, not the whole board, and only the detail its hints need, so the cards named as
+stranded behind closed dependencies stay in it. A failed or stalled read is one log line, and
+the refresher backs off; a stall in the sweep's own read ends that tick like any other stall. A
+blocked card's hint names its block reason and whether it retries on its own or needs a human.
+Unblocking a card now clears its block class (the class is noted on the unblock's comment), and a
+card that is not blocked no longer reports one. An exhausted CI or review escalation now records
+a class and reason of its own, so boot no longer mistakes it for an old preflight hold and
+releases it.

docs/lifecycle.md

@@ -285,8 +285,9 @@ again once they close. It is not a claim candidate and it shows up in no skip di
 So the board now names it, wherever a card's next action is shown: the listing, the
 console chip, the agent's working state (which names a backlog card only when it owes a step,
 and ranks it after every in-flight card so a pile of stranded cards can't push a PR awaiting
-merge out of the capped list), and one sweep log line when the card first becomes stranded
-(held in memory, so a restart logs each stranded card once more):
+merge out of the capped list), and one log line when the loop first sees the card stranded,
+on the working-state snapshot's read (held in memory, so a restart logs each stranded card
+once more):
 
 - **backlog, every dependency closed** → `dependencies closed — promote`. The step is
   `board_mark_ready`, and the Ready gate still decides. A `deferred` or `designing` card is

failures.py

@@ -97,11 +97,11 @@ class it would otherwise have (a `502 … timeout` is still transient)."""
 # ── pre-model dispatch / infrastructure failures (#339) ──────────────────────────
 # The `blocked-class:` a pre-model dispatch/infra failure carries. It is deliberately
 # NOT one of `classify()`'s categories: it can't be decided from the message alone
-# (it needs the loop's dispatch-lifecycle evidence too), and it drives two behaviours
-# the message-classes don't — the operator is NOTIFIED rather than auto-healed (it is
-# absent from the loop's self-healing set), and an operator unblock RESETS the card's
-# escalation tier so a host/adapter incident never leaves a `tier:` label the next
-# genuine build inherits.
+# (it needs the loop's dispatch-lifecycle evidence too), and the operator is NOTIFIED
+# rather than auto-healed (it is absent from the loop's self-healing set). The loop never
+# climbs a tier on such a failure, so an unblock (store.clear_blocked) drops the class like
+# any other and leaves the card's `tier:` labels alone: they were earned before the
+# incident.
 PRE_MODEL_DISPATCH_CLASS = "dispatch-infra"
 
 # The seam SHAPES — how a failure raised BELOW the model call reaches the loop. `coder_seam`

loop/core.py

@@ -244,6 +244,14 @@ def __init__(self, cfg: dict, *, gap_reporter: setup_check.GapReporter | None =
         # once per DISTINCT reason, and identical repeats collapse to a one-line
         # "still held (Ns)" WARNING (see _record_preflight_failure).
         self._preflight_failed_at: dict[str, float] = {}
+        # The working-state snapshot's refresher (#401): its task, when it last read, how many
+        # reads in a row failed (its backoff), and whether a read is in flight (one at a time).
+        # -inf, not 0.0: time.monotonic() starts near zero in a fresh container, and the first
+        # read must not wait out MIN_INTERVAL_S.
+        self._snapshot_task: asyncio.Task | None = None
+        self._snapshot_attempted_at = float("-inf")
+        self._snapshot_failures = 0
+        self._snapshot_reading = False
         # ── coder.solve() board seam (ADR 0064 P2, opt-in) ─────────────────────────
         # Route a FRESH build (not a keep-worktree/CI-bounce re-dispatch) through the
         # `coder` plugin's execution-grounded solve() ladder (greedy → best-of-k →

loop/drive.py

@@ -204,6 +204,12 @@ def start(self):
             log.info("[project_board] loop disabled (project_board.loop_enabled=false) — board API still serves")
             return None
         self._task = asyncio.create_task(self._run(), name="project-board-loop")
+        # The working-state snapshot has a refresher of its own, not a step of the tick: a
+        # loop paused at its setup gate runs no ticks, and the snapshot must not go stale
+        # for as long as the pause lasts (#401).
+        self._snapshot_task = asyncio.create_task(
+            self._keep_work_snapshot_current(), name="project-board-work-snapshot"
+        )
         # The RUNNING loop's config, in a process-stable slot: after a reload the
         # routers see the new config while this loop keeps its construction-time
         # `coders`/`repo`/…; /status compares the two and says "restart to apply".
@@ -339,6 +345,9 @@ def reload(self, new_config) -> dict:
                 changed[key] = (cur, new)
         if changed:
             setup_check.publish_loop_snapshot(self.cfg)
+            # The working-state hints read these knobs (auto_merge, review_gate, …), so what
+            # the snapshot renders changed without a board write: mark it stale (#401).
+            work_snapshot.mark_stale()
             log.info(
                 "[project_board] reload applied live: %s (in-flight drives: %d)",
                 ", ".join(f"{k} {o}→{n}" for k, (o, n) in changed.items()),
@@ -352,12 +361,13 @@ async def stop(self):
         _unregister_loop(self)  # drop the process-stable handle (ADR 0326)
         if self._task:
             setup_check.publish_loop_snapshot(None)  # no running loop → nothing to be stale against
-        if self._task:
-            self._task.cancel()
-            try:
-                await self._task
-            except (asyncio.CancelledError, Exception):  # noqa: BLE001
-                pass
+        for task in (self._task, self._snapshot_task):
+            if task:
+                task.cancel()
+                try:
+                    await task
+                except (asyncio.CancelledError, Exception):  # noqa: BLE001
+                    pass
         # Cancel any in-flight drives and await them out. A drive cancelled mid-flight
         # can't run its own cleanup, so its worktree stays in self._inflight — reaped
         # below. (A completed/blocked drive already popped itself.)
@@ -2161,9 +2171,10 @@ async def _drive(self, feature: dict):
                     #     called, so no `tier:`/`attempt:` label is added and no ladder
                     #     budget is spent. It blocks under the `dispatch-infra` class the
                     #     blocked sweep never auto-heals, so the operator is notified with the
-                    #     original infra evidence — and an operator unblock resets the tier
-                    #     posture (store.clear_blocked) so the next genuine build starts at
-                    #     its difficulty-selected tier (#339).
+                    #     original infra evidence. An operator unblock (store.clear_blocked)
+                    #     drops the class and leaves the card's `tier:` labels alone: the
+                    #     incident added none, so the next build resumes the tier the card
+                    #     had earned, or its difficulty-selected one (#339).
                     if pre_model:
                         reason = f"pre-model dispatch failure — infra triage, no tier climb: {exc}"
                         log.warning(

loop/reconcile.py

@@ -16,6 +16,10 @@
 
 _loop = sys.modules[__package__]  # the loop package, for monkeypatch-visible seams
 
+# How often the working-state refresher checks whether a read is due (an in-memory compare;
+# the reads themselves are bounded by work_snapshot.MIN_INTERVAL_S / MAX_AGE_S).
+_SNAPSHOT_POLL_S = 1.0
+
 
 class ReconcileMixin:
     # ── merged-verify exhaustion sentinel ↔ operator reset (ADR 0326, #326) ───────
@@ -317,27 +321,12 @@ async def _sweep(self):
         ``feat-<id>`` worktrees whose feature is gone or already terminal —
         ``done``/``cancelled`` (a missed reap); (c) label terminal features past the
         archive window ``archived``
-        (#115) — the board's growth valve; archival only, nothing is ever deleted. First,
-        it publishes the agent's working-state snapshot, and names any card stranded
+        (#115) — the board's growth valve; archival only, nothing is ever deleted. Last,
+        it publishes the agent's working-state snapshot, which names any card stranded
         outside the ready lane with every dependency closed (#406) — surfaced, never moved.
-        Best-effort; a per-item failure never stops the sweep or the loop."""
+        Best-effort; a per-item failure never stops the sweep or the loop. A stalled store
+        ends the sweep, and the tick with it (#404)."""
         store = self._store()
-        # Publish the board's live projection for the host's <working_state> block (ADR
-        # 0079 Observe). Done on the SWEEP cadence, not per tick: it is one extra `br`
-        # call per sweep, and the provider that reads it must never touch the store
-        # itself — it runs inline on every agent turn. Best-effort, like the rest of the
-        # sweep.
-        #
-        # Annotated first: the snapshot's per-card hint IS the board's `next_action_hint`,
-        # which only `annotate_next_action` writes — a bare listing published every hint
-        # empty. It is also what names a card stranded outside the ready lane (#406), and
-        # the agent this snapshot feeds is the one who can promote it.
-        try:
-            feats = store_mod.annotate_next_action(await asyncio.to_thread(store.list_features), self.cfg)
-            work_snapshot.publish(feats)
-            self._report_stranded(feats)
-        except Exception:  # noqa: BLE001 — never let a snapshot refresh stop the sweep
-            log.warning("[project_board] work snapshot refresh failed (ignored)", exc_info=True)
         for f in await self._list_for_pass(store, "in_progress", "health sweep"):
             fid = f["id"]
             if fid in self._inflight_files:
@@ -372,14 +361,115 @@ async def _sweep(self):
             raise
         except Exception:  # noqa: BLE001
             log.warning("[project_board] sweep archive pass failed", exc_info=True)
+        # (d) the host's <working_state> snapshot, LAST, so it carries this sweep's own
+        # reconciles and unblocks. Published first, as it used to be, it showed a card the
+        # sweep had just moved in the state it had left, for a whole interval (#401). The
+        # refresher would pick those writes up within seconds anyway; publishing here makes
+        # the sweep's result current the moment it ends. Its stall is the sweep's stall.
+        await self._publish_work_snapshot(stall_ends_tick=True)
+
+    # ── the host's <working_state> snapshot (ADR 0079 Observe, #401) ──────────
+    def _take_work_snapshot(self, store) -> None:
+        """Read the open cards and publish them for the host's working-state block.
+
+        The revision is read BEFORE the board, so a write that lands mid-read leaves the
+        snapshot marked stale (and re-read soon) rather than claiming a state its rows may
+        predate. The read is ``live_cards``, the light one: open statuses only, no
+        whole-board ``br show``. Rows are annotated as board_list's are, so a card's hint is
+        the board's own next action. A blocked card, ranked first, gets its block reason and
+        what moves it, which the posture hints leave blank. A card newly stranded outside
+        the ready lane is logged (``_report_stranded``)."""
+        revision = work_snapshot.board_revision()
+        features = store_mod.annotate_next_action(store.live_cards(), self.cfg)
+        for f in features:
+            if f.get("blocked") and not f.get("next_action_hint"):
+                cls = str(f.get("blocked_class") or "").strip()
+                reason = str(f.get("blocked_reason") or "").strip() or "no reason recorded"
+                who = (
+                    f"retries on its own ({cls})"
+                    if cls in _SELF_HEALING_BLOCKS
+                    else f"needs a human ({cls or 'unclassified'})"
+                )
+                f["next_action_hint"] = f"{who}: {reason}"
+        work_snapshot.publish(features, revision=revision)
+        self._report_stranded(features)
+
+    async def _publish_work_snapshot(self, *, stall_ends_tick: bool = False) -> bool:
+        """``_take_work_snapshot``, off the event loop and best-effort. Returns whether it
+        published. A failed read backs the refresher off (see ``_work_snapshot_due``), and
+        the provider keeps marking the old snapshot stale meanwhile.
+
+        A ``BoardError`` (a `br` call that failed, or stalled and was stopped) is an
+        understood, transient outcome, logged as ONE line like ``_tick_phase`` logs it;
+        anything else is a bug and keeps its traceback. The exception is a stall inside a
+        tick (``stall_ends_tick``, the sweep's publish): it goes up to ``_tick_phase``, which
+        ends the tick on it (#404). Swallowed there, the tick went on to the preflight and the
+        claim scan, and each stalled again on the same wedged store.
+
+        One read at a time: the refresher and the sweep can both get here, and a second read
+        while one is in flight is skipped. The one in flight publishes, and a write since it
+        began leaves that snapshot STALE for the refresher to re-read."""
+        if self._snapshot_reading:
+            return False
+        self._snapshot_reading = True
+        self._snapshot_attempted_at = time.monotonic()
+        try:
+            await asyncio.to_thread(self._take_work_snapshot, self._store())
+        except BoardError as exc:
+            self._snapshot_failures += 1
+            if stall_ends_tick and isinstance(exc, store_mod.BoardTimeout):
+                raise
+            log.warning("[project_board] work snapshot refresh failed (retrying with backoff): %s", exc)
+            return False
+        except Exception:  # noqa: BLE001 — never let a snapshot refresh stop the loop
+            self._snapshot_failures += 1
+            log.warning("[project_board] work snapshot refresh failed (retrying with backoff)", exc_info=True)
+            return False
+        finally:
+            self._snapshot_reading = False
+        self._snapshot_failures = 0
+        return True
+
+    def _work_snapshot_due(self) -> bool:
+        """Whether the refresher should read now. In memory, no I/O.
+
+        - Never inside ``MIN_INTERVAL_S`` of the last attempt. A burst of writes (a claim,
+          its labels, a drive's budget stamps) costs ONE read, not one each.
+        - After failures, wait out an exponential backoff (``MIN_INTERVAL_S`` doubling, up to
+          ``MAX_AGE_S``). A store that is failing is not hammered.
+        - Otherwise read when the board changed since the snapshot, when there is none yet,
+          or when it is older than ``MAX_AGE_S``. The last case is what bounds a writer this
+          process can't see."""
+        since = time.monotonic() - self._snapshot_attempted_at
+        wait = work_snapshot.MIN_INTERVAL_S
+        if self._snapshot_failures:
+            wait = min(work_snapshot.MIN_INTERVAL_S * 2**self._snapshot_failures, work_snapshot.MAX_AGE_S)
+        if since < wait:
+            return False
+        taken = work_snapshot.taken_at()
+        return work_snapshot.needs_refresh() or taken is None or time.time() - taken >= work_snapshot.MAX_AGE_S
+
+    async def _keep_work_snapshot_current(self) -> None:
+        """The snapshot's refresher, a task of its own (started with the loop). It is NOT part
+        of the claim tick. A loop paused at its setup gate (a missing coder, say) runs no
+        ticks, and a snapshot refreshed only by ticks stayed STALE for as long as the pause
+        lasted. It polls an in-memory check every ``_SNAPSHOT_POLL_S`` and reads only when
+        ``_work_snapshot_due`` says so."""
+        while not self._stop.is_set() and not self._shutting_down:
+            if self._work_snapshot_due():
+                await self._publish_work_snapshot()
+            try:
+                await asyncio.wait_for(self._stop.wait(), timeout=_SNAPSHOT_POLL_S)
+            except asyncio.TimeoutError:
+                pass
 
     def _report_stranded(self, feats: list[dict]) -> None:
         """Log each card that has just become stranded outside the ready lane, with every
         dependency closed (#406). ``store.stranded_posture`` found them, and they read so
         on every listing and in the agent's working state. This line is the loop's own
-        record of WHEN it first saw each one: once per card, not once per sweep, and again
-        only if the card leaves that state and comes back. Nothing is changed on the card.
-        A backlog card is promoted by the PM, and a block is lifted by whoever set it."""
+        record of WHEN it first saw each one: once per card, not once per snapshot read, and
+        again only if the card leaves that state and comes back. Nothing is changed on the
+        card. A backlog card is promoted by the PM, and a block is lifted by whoever set it."""
         stranded = {
             f["id"]: f
             for f in feats
@@ -389,7 +479,7 @@ def _report_stranded(self, feats: list[dict]) -> None:
         seen = getattr(self, "_stranded_seen", set())
         for fid in sorted(set(stranded) - seen):
             f = stranded[fid]
-            log.info("[project_board] sweep: %s stranded (%s): %s", fid, f["board_state"], f.get("next_action_hint"))
+            log.info("[project_board] %s stranded (%s): %s", fid, f["board_state"], f.get("next_action_hint"))
         self._stranded_seen = set(stranded)
 
     async def _recover_blocked(self, store) -> None: