Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion __init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -949,9 +949,15 @@ def board_block_feature(feature_id: str, reason: str) -> str:
def board_unblock_feature(feature_id: str) -> str:
"""Clear the `blocked` flag so the feature can be re-dispatched — the inverse of
board_block_feature. Removes the blocked label; the puller can claim it again once
it's otherwise `ready`."""
it's otherwise `ready`. A card parked `too-wide` after repeated timeouts also gets
its timeout count reset, so the retry is a real attempt."""
try:
f = get_store(**store_kw).clear_blocked(feature_id)
# The running loop's cached timeout count wins over the label the store just
# reset (#259) — drop it too (#378). Lazy import: the loop imports from here.
from .loop import forget_timeout_count

forget_timeout_count(feature_id)
return json.dumps({"id": f["id"], "state": f["board_state"]})
except BoardError as exc:
return f"Error: {exc}"
Expand Down
8 changes: 7 additions & 1 deletion api.py
Original file line number Diff line number Diff line change
Expand Up @@ -677,7 +677,13 @@ async def _block(fid: str, body: dict = Body(...)):

@router.post("/features/{fid}/unblock")
async def _unblock(fid: str):
return await _guard(lambda: store().clear_blocked(fid))
f = await _guard(lambda: store().clear_blocked(fid))
# The running loop's cached timeout count wins over the label the store just reset
# (#259) — drop it too, so a card parked `too-wide` gets a real retry (#378).
from .loop import forget_timeout_count

forget_timeout_count(fid)
return f

@router.post("/features/{fid}/cancel")
async def _cancel(fid: str, body: dict = Body(default={})):
Expand Down
11 changes: 11 additions & 0 deletions changelog.d/435.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
A card whose fresh builds keep timing out is now parked for its split instead of being rebuilt.
The decompose task a repeated timeout files used to land in backlog, where the loop never
dispatches, so the board's own agent was never actually asked. It is now released once the card
is parked under its own `too-wide` block class, with a reason that names the task. The ask also
waited for the model ladder to run out, so a card on a three-rung ladder timed out three times
first. And it was paired with a self-healing block, so the sweep requeued the oversized card and
rebuilt it twice more, racing its own split. Only a timeout on a fresh build that reached the
model now counts toward `decompose_after_timeouts`: never an infra timeout, never a fix round on
a card that already built. The count clears when a build reaches review, and unblocking a parked
card resets it. The split's steps now run in an order that passes the board's gates: create the
slices, re-point the card's dependents, cancel the card, then mark the slices ready.
33 changes: 27 additions & 6 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,12 +139,33 @@ The gates between a green build and main.
| `kg_lessons_domain` | `"loop-lessons"` | reload **· YAML only** |

`decompose_after_timeouts` is the one that changes behaviour rather than tuning it. A coder
timeout is a **size** signal, not a capability one: it produces no diff and no CI output, so a
retry re-sends a near-identical prompt and climbing the model ladder spends a stronger model on
a card that was never model-limited. After this many timeouts on one card, the loop files a task
asking this agent to split it into buildable slices — once per card, and never for a
pre-first-token timeout (that is an infra fault, and splitting would be the wrong remedy). Set
`0` to switch the ask off and have the card simply block, as it did before.
timeout on a fresh build is a **size** signal, not a capability one. It produces no diff and no
CI output, so a retry re-sends a near-identical prompt, and climbing the model ladder spends a
stronger model on a card that was never model-limited.

Only those timeouts count:

- A pre-first-token timeout is an infra fault that splitting would not fix. It never counts,
and never asks.
- A timeout on a fix round doesn't count either: the card already has a PR, or the coder was
fixing a kept worktree. A card that built in one dispatch is not too wide.
- The count clears when a build reaches review.

On the timeout that reaches this count, the loop parks the card (blocked class `too-wide`) and
files a `ready` task asking this agent to split it. The card does not climb another rung, and
the blocked sweep does not rebuild it, since that would only time out again, racing the split.
The operator is told once, and the block reason names the task, or says that none was filed.

The task gives the agent a fixed order, each step passing the gates the next one relies on:

1. Create the slices, left in backlog.
2. Re-point the card's dependents onto the slices they need.
3. Cancel the card.
4. Mark the slices ready.

The ask is made once per card. Unblocking a parked card resets its count, so a retry after
raising `coder_timeout_s` is a real attempt. Set `0` to switch the ask off and have the card
simply block, as it did before.

## Concurrency

Expand Down
9 changes: 9 additions & 0 deletions failures.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,15 @@ class it would otherwise have (a `502 … timeout` is still transient)."""
return TERMINAL


# The `blocked-class:` of a card the loop PARKED because its fresh builds keep timing out
# (#378): too wide to build in one dispatch, with a split handed to the board's own agent.
# Like `dispatch-infra` below it is the loop's own class, not one of `classify()`'s: the
# blocked sweep never re-runs it (it would only time out again, racing its own split), and
# an operator unblock resets the card's timeout count, so a deliberate retry — after
# raising `coder_timeout_s`, say — gets a real attempt instead of re-parking at once.
TOO_WIDE_CLASS = "too-wide"


# ── pre-model dispatch / infrastructure failures (#339) ──────────────────────────
# The `blocked-class:` a pre-model dispatch/infra failure carries. It is deliberately
# NOT one of `classify()`'s categories: it can't be decided from the message alone
Expand Down
19 changes: 18 additions & 1 deletion loop/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
import types

from .. import br_fetch, coder_seam, config, health, setup_check, work_snapshot, worktree
from ..failures import PRE_MODEL_DISPATCH_CLASS, classify, is_pre_model_dispatch_failure
from ..failures import PRE_MODEL_DISPATCH_CLASS, TOO_WIDE_CLASS, classify, is_pre_model_dispatch_failure
from ..projects import default_project as resolve_default_project
from ..projects import resolve_projects
from .. import store as store_mod
Expand Down Expand Up @@ -1129,6 +1129,21 @@ def reset_merged_verify_budget(fid: str, store) -> bool:
return True


def forget_timeout_count(fid: str) -> bool:
"""Drop the live loop's cached timeout count for ``fid`` after an operator unblock
(#378), so the next ``_budget_get`` re-reads it from the bead. ``clear_blocked`` resets
the persisted count when it releases a card parked as `too-wide`, but the loop's cache
wins over the labels (#259): without this, a retry after raising `coder_timeout_s`
would re-park on its very first timeout. Forgetting — not pinning 0 — is right for any
unblock: the cache simply re-syncs with whatever the bead now says. No lock needed: a
blocked card has no drive in flight to race. Returns False when no loop is running."""
loop = live_loop()
if loop is None:
return False
loop._budget_cache("timeout").pop(fid, None)
return True


def cancel_pr_comment(fid: str) -> str:
return f"cancelled by operator — see card {fid}"

Expand Down Expand Up @@ -1289,6 +1304,7 @@ def _inbox_db_path():
"work_snapshot",
"worktree",
"PRE_MODEL_DISPATCH_CLASS",
"TOO_WIDE_CLASS",
"classify",
"is_pre_model_dispatch_failure",
"resolve_default_project",
Expand Down Expand Up @@ -1370,6 +1386,7 @@ def _inbox_db_path():
"_unregister_loop",
"live_loop",
"reset_merged_verify_budget",
"forget_timeout_count",
"cancel_pr_comment",
"cancel_side_effects",
"_MAX_MODE_JUDGE_SYS",
Expand Down
114 changes: 89 additions & 25 deletions loop/drive.py
Original file line number Diff line number Diff line change
Expand Up @@ -1666,12 +1666,6 @@ async def _drive(self, feature: dict):
# it burned the whole tier ladder in ten seconds (three attempts,
# three tiers, a block; 2026-08-28, bd-cwpv.12/.16) and left `tier:`
# labels that misrouted the card when it was requeued after the reset.
# #378: count timeouts durably, BEFORE the retry/escalate/block fork —
# bd-sxxf timed out, escalated a tier, timed out again and only then
# blocked, so a counter bumped at the block alone would have read 1.
if isinstance(exc, worktree.CoderTimeout):
timeouts = await self._budget_get(store, fid, "timeout") + 1
await self._budget_set(store, fid, "timeout", timeouts)
dispatch_failed = str(exc).startswith("coder dispatch failed") and not policy.retryable
capability = (
isinstance(exc, (worktree.NoChangesError, worktree.CoderTimeout, coder_seam.SolveExhausted))
Expand Down Expand Up @@ -1820,6 +1814,34 @@ async def _drive(self, feature: dict):
await worktree.remove_worktree(repo, wt, branch or "")
self._inflight.pop(fid, None)
return
# 1.7 A card whose FRESH builds keep timing out → PARK it, and hand its split to
# the board's own agent (#378). A repeated timeout is a SIZE signal: no diff,
# no CI output, so neither a stronger rung nor a rebuild has anything new to
# work with (on a ladder, the first timeout's climb already carried #146's
# timeout context). Only a timeout that IS that signal counts:
# • one that reached the model — a pre-first-token timeout is infra, and 1.5
# above has already blocked it, uncounted;
# • on a fresh build — a fix round (a kept worktree, or a card whose PR is
# open) is fixing a card that already BUILT in one dispatch, so it is not
# too wide. It takes the ordinary path below. Parked, it would ask to split
# a card with an open PR, and the split's cancel would close that PR.
# The count is durable (a `budget:` label, so a restart can't reset it) and
# is cleared when a build reaches review or an operator unblocks the park.
# Counting it here, before the climb/block fork, is what lets bd-sxxf's shape
# — timed out, climbed, timed out again — reach the threshold at all. The ask
# used to wait for the ladder to run out, beside a self-healing `transient`
# block: a three-rung ladder timed out three times before it asked, and the
# sweep then rebuilt the card twice more, racing its own split.
if isinstance(exc, worktree.CoderTimeout) and not reusing and not feature.get("pr_url"):
timeouts = await self._budget_get(store, fid, "timeout") + 1
await self._budget_set(store, fid, "timeout", timeouts)
threshold = self.decompose_after_timeouts # 0 = never park (block as before)
if threshold and timeouts >= threshold:
await self._park_for_split(store, fid, timeouts, exc)
if wt:
await worktree.remove_worktree(repo, wt, branch or "")
self._inflight.pop(fid, None)
return
# 2. Capability failure + a ladder → climb a model tier (fresh budget).
if self.escalation_on and capability:
nxt = await asyncio.to_thread(store.escalate, fid, str(exc)[:200])
Expand Down Expand Up @@ -1869,23 +1891,6 @@ async def _drive(self, feature: dict):
await asyncio.to_thread(
store.flag_blocked, fid, f"{policy.category}: {exc}", category=policy.category
)
# #378: a card that has now timed out repeatedly is not model-limited, it
# is too wide — and nothing in the block path says so, which is how one
# sat parked while an operator guessed. Ask the board's own agent to split
# it (once; `request_decomposition` no-ops on a repeat or on a task).
if isinstance(exc, worktree.CoderTimeout) and self.decompose_after_timeouts:
spent = await self._budget_get(store, fid, "timeout")
ask = getattr(store, "request_decomposition", None) # older/stub store: skip
if spent >= self.decompose_after_timeouts and callable(ask):
asked = await asyncio.to_thread(ask, fid, timeouts=spent)
if asked:
log.warning(
"[project_board] %s timed out %dx — filed %s to decompose it "
"(a timeout is a SIZE signal, not a capability one)",
fid,
spent,
asked.get("id", "?"),
)
if wt:
await worktree.remove_worktree(repo, wt, branch or "")
self._inflight.pop(fid, None)
Expand All @@ -1895,8 +1900,11 @@ async def _drive(self, feature: dict):
log.info("[project_board] %s coder done (%d chars) → %s", fid, len(result or ""), pr_url)
await asyncio.to_thread(store.open_review, fid, pr_url=pr_url)
# Gate passed — reset the pre-PR budgets (goal-fix, local-gate, the
# requirement ledger #113, and the empty-result count #198).
await self._budget_reset(store, fid, "goal-fix", "gate-fix", "req-fix", "empty-result", "ledger-only")
# requirement ledger #113, and the empty-result count #198), and the timeout
# count (#378): a card that built in one dispatch is demonstrably not too wide.
await self._budget_reset(
store, fid, "goal-fix", "gate-fix", "req-fix", "empty-result", "ledger-only", "timeout"
)
if self.review_gate:
# Blocking adversarial review (M5). May requeue the feature with
# findings injected — the next drive carries them in the prompt.
Expand Down Expand Up @@ -1934,6 +1942,62 @@ async def _drive(self, feature: dict):
await worktree.remove_worktree(repo, wt, branch or "")
self._inflight.pop(fid, None)

async def _park_for_split(self, store, fid: str, timeouts: int, exc: Exception) -> None:
"""Park a card whose fresh builds keep timing out, and hand its split to the board's
own agent (#378) — in the one order that is safe:

1. FILE the decompose task. ``request_decomposition`` leaves it in backlog, where the
puller cannot see it, and returns the task still open for this card (new, or filed
earlier), or None.
2. PARK the card under ``too-wide`` — never auto-healed; an operator unblock resets the
count — with a reason built from what step 1 returned: it names the task, or says
plainly that none was filed and what to do instead. (Building the reason before the
ask told a card re-parked after an operator requeue that a split was on its way
when none had been filed.)
3. RELEASE the task, only now: the agent that picks it up ends by cancelling this card,
which must never be found still in flight."""
ask = getattr(store, "request_decomposition", None) # older/stub store: skip
task = await asyncio.to_thread(ask, fid, timeouts=timeouts) if callable(ask) else None
task_id = str((task or {}).get("id") or "")
retry = "raise coder_timeout_s and unblock it (unblocking resets its timeout count)"
if task_id:
reason = (
f"too wide to build in one dispatch — timed out {timeouts}x; parked while this agent splits "
f"it into slices ({task_id}). No tier climb and no auto-retry; to retry it whole instead, "
f"{retry}: {exc}"
)
else:
reason = (
f"too wide to build in one dispatch — timed out {timeouts}x, and NO split task was filed (one "
f"was requested for this card before, or the store refused): split it by hand, or {retry}: {exc}"
)
log.warning(
"[project_board] %s timed out %dx on fresh builds — parked as too wide (%s)",
fid,
timeouts,
f"split task {task_id}" if task_id else "no split task filed",
)
await asyncio.to_thread(store.flag_blocked, fid, reason, category=TOO_WIDE_CLASS)
release = getattr(store, "mark_ready", None)
if task_id and str(task.get("board_state") or "backlog") == "backlog" and callable(release):
try:
await asyncio.to_thread(release, task_id)
except Exception: # noqa: BLE001 — the park stands; a human can promote the task
log.warning(
"[project_board] %s split task %s is filed but the Ready gate refused it — it waits in backlog",
fid,
task_id,
exc_info=True,
)
try:
await asyncio.to_thread(
store.comment,
fid,
f"split task {task_id} is filed but NOT ready — the Ready gate refused it; promote it by hand",
)
except Exception: # noqa: BLE001 — the trail is best-effort
log.debug("[project_board] %s split-task comment failed", fid, exc_info=True)

# ── operator cancel during a drive (#211) ────────────────────────────────
@staticmethod
def _cancelled(store, fid: str) -> bool:
Expand Down
Loading
Loading