Integrate #1965 #1992 #2075

docs/p241-native-measurements.md

@@ -0,0 +1,59 @@
+# P241: shared commander load and growth
+
+Owner: pi-cn-phase2. PR #1965. Rebased onto requested main `15ba6edce`.
+
+## Ownership
+
+- `commander/capacity.py:Load` owns native and CLI load, the weighting formula,
+  and selection of receiving child commanders with retained orders. The CLI
+  uses its supplied commander context; it does not construct a competing one.
+- `turn_digests.py:OrderRead.for_commander` owns the shared order/holder selection
+  and incarnation check. Both full edit reads and count-only reads consume it.
+  `OrderEditCountRead.count` serves load and the unjudged standing rule. The
+  competing `CommsUnjudgedTool.readers/count` methods are deleted.
+- The heartbeat reader/writer, its CLI write and its config-directory creation
+  are deleted. Registry parentage/status and held orders supply that fact.
+- `commander/standing.py:MergeGrowth` reads one captured first-parent Git history
+  with numstat, not a fetch followed by two subprocesses per revision.
+  `repo.py:Repo.git` owns bounded checked execution and remote discovery.
+- Main's `CommanderContext`, argument parser, order lifecycle and recorded-check
+  progress implementations are preserved. `ToolCaller.acting_name` selects the
+  explicitly bound caller or the context's configured commander. The CLI uses
+  the existing concrete caller states and FieldCodec normalization, not a nullable
+  name or a second context. Checked order reads obtain their cutoff from the order.
+
+Before-rebase AST evidence uses the existing audit Package/Repository parser
+across Core, tools and tests: `p241-2200-ast-before.txt` in this unit's scratch.
+Receiver bindings are syntactic; dynamic resolution and state freshness are not
+proved by the census. The previous retained source evidence is also preserved.
+
+## Verification and limitations
+
+The 20:52 checkpoint was broken: a CLI FieldCodec reference lacked its import.
+Its retained tests reported 11 passed, one failure and 36 errors. Both ratchets
+exited 2 because the command omitted the required `--head`; those were not
+successful checks. The current rebase removes that obsolete CLI identity change.
+
+Current checks use the shared interpreter, background `timeout 300` commands,
+and the unchanged checker against the requested main. Results and application
+receipts are retained under `/home/ts/.cache/agent-scratch/pi-cn-phase2/` with
+prefix `p241-2200-`. No installation or provider input is performed here.
+
+Final source checkpoint `d00ab51d7`: **49 tests passed** under background
+`timeout 300`; source ratchet **1 solely SourceLines +8**, with no introduced
+findings; tools ratchet **0**. Both ran separately against `15ba6edce`, with
+`--head` supplied. Before/after AST acquisition reported no parse omissions.
+
+Actual configured saved-state native handler read **32 open / 30 overdue /
+1,860 unjudged / total 278.0**. The real CLI then read **32 / 30 / 1,861**;
+the count increased by one between reads, so this is not a frozen parity sample. The
+focused test checks exact native/CLI equality on one retained fixture. This is
+not an installed native-transport or provider-process verification.
+
+The 22:10 readiness deadline was missed: the final fork argument representation
+needed correction. No failed or running check was represented as successful.
+
+This delivers the shared load/growth checkpoint. Broader timing/alert parity and
+P244's comparisons are not claimed complete. Earlier native/CLI samples are
+historical evidence only, not proof of the rebased artifact. No blocked goal is
+reset, and no uncertain operation is replayed.

plans/commander-native-plan.md

@@ -28,42 +28,43 @@
 
 ## Parity matrix
 
-| # | Capability | Script source today | Native owner | Phase |
+<!-- Generated by python -m agent_comms.commander_parity; edit row declarations. -->
+| # | Capability | Current commander | Native owner | Native phases |
 |---|---|---|---|---|
-| 1 | Create a problem with owner, order and deadline | `order.py problem`; `problems.json` | `CommsTodoCreateTool` (exists) | 1 |
-| 2 | Assign, reassign, drop an order | `order.py send`, `--reassign`, `drop`; `deadlines.json` | `CommsOrderTool`, `CommsTodoTool` assignment and release (exist) | 1 |
-| 3 | Complete only on the delivered commit | `order.py done` | `CommsOrderDoneTool` (exists), returning a delivery state: merged to `main`, merged to another branch, not merged | 1, 3 |
-| 4 | Close with installed evidence | `order.py close`; `closed.jsonl` | `CommsProblemCloseTool` (exists) | 1 |
-| 5 | Board, overdue, silent, spinning | `order.py board`, `overdue`; `fleet-board.md` | `CommsTodoListTool` and the fleet tool, derived (exist) | 1 |
-| 6 | Branch hygiene | `branches-*.txt` by hand | `CommsWorkBranchesTool` (exists) | done |
-| 7 | Strikes and the strike limit | `order.py strike`; `strikes.jsonl` | `Strike` rows on the order; `StrikeLimit` declared on the commanded author; one `comms_fire` | 3 |
-| 8 | Verdicts on a unit's tool calls | `judge.py ok`, `back`; `verdicts.jsonl` | `Verdict` on the order; a send-back revises the order Goal and records the strike in one write | 3 |
-| 9 | Plain words in verdicts and strikes | judge and order checks | `PlainText` value type | 3 |
-| 10 | Refuse soft or vague orders | `order.py` | `OrderText` value type on every order action | 3 |
-| 11 | What a unit did during its order | `tool_watch.py`; `edits.jsonl`; `judge.py show` | `comms_order_work`, read from both transcript families by `tool_call_id` | 4 |
-| 12 | What the commander has not judged yet | `judge.py unjudged`; `vibe_watch.py` | `comms_unjudged`: tool calls and unit messages after the last verdict, derived | 4 |
-| 13 | Public story and roasts | `fleet-narrative.md`; `judge.py roast` | `comms_order_story`, `comms_fleet_story`; each send-back verdict posts its plain reason to the order's channel | 5 |
-| 14 | Heartbeat: next unit every minute | `rotate.py` (60 s loop) | a scheduled commander wake owned by Core | 7 |
-| 15 | Standing rules broken right now | `rotate.py broken_rules` | a `StandingRule` family, each member deriving its violations on read | 7 |
-| 16 | Rule review | `rules.py`; `rules.jsonl`; a maximum age | the rules as instruction segments in the commander's turn context, with provenance | 7 |
-| 17 | Load and forking | `capacity.py`: 6 units per commander, 10-minute beat age; `forks/` | `CommanderLoad` derived from open orders and the unjudged queue; forks are registered threads | 7 |
-| 18 | Growth alarm | `growth_watch.py`: merges growing `src` by more than 40 net lines | a growth check derived on read from git per configured repository | 7 |
-| 19 | Storage hygiene | `storage.py`: worktree limit per unit, cleanup due under 30 GB free, nothing allocates under 5 GB | `UnitStorage` derived on read from worktree paths and free disk | 7 |
-| 20 | Commander reaches units only through orders | `send_guard.py` (PreToolUse hook) | stays a hook, at the harness boundary, checking against Core's order tools | 7 |
-| 21 | Paths, repositories, commander identity | `CommanderConfig` from `commander.json` | Core's project and registry configuration | 8 |
-| 33 | Several commanders on one machine and bus, one per project | none (one `commander.json`; a second install overwrites the first) | each commander is a registered thread that owns one project. Orders, strikes, load, board, alarms and forks are keyed by that commander, and the commander for a call comes from its `--from` thread. The send guard refuses a raw send from any registered commander. Two commanders for one project are refused when the second registers | 1, 8 |
-| 22 | Seeing the fleet: board, strikes, queue, story, alarms | `fleet-board.md`, `order.py board`, the terminal output of `rotate.py` and `judge.py` | a commander page in Toad, rendering Core's answers through `ui_model` | 6 |
-| 23 | An order states the user-visible run that closes it | scratch `rts/order.py send` (`Works when:`) | `OrderText` requires it; `CommsProblemCloseTool.works` is checked against it | 3 |
-| 24 | Reassigning past two owners needs the commander's trace | scratch `rts/order.py send --reassign --traced` | the transfer operation reads owners from the Todo's assignment history and refuses without a traced source location | 3 |
-| 25 | Unit wording gets a correction, not a strike | scratch `vibe_watch.py` | `PlainText` applies to the commander's own verdicts and strikes; unit wording never becomes a strike | 3 |
-| 26 | Alerts carry only decisions | none (today every line interrupts) | the commander's wake receives decision events only, each once | 7 |
-| 27 | Momentum prediction per unit, compared with the outcome | none | a prediction record on the order, shown on the commander page beside what happened | 6 |
-| 28 | Measured load and self-report | `capacity.py` unit count | `CommanderLoad`: look debt, judge debt, inflow, calibration, context, plus a one-word self-report calibrated against them | 7 |
-| 29 | No handoff of finished, checked work | none (scratch `order.py send` moved P507 after its fix was pushed and checked) | the transfer operation refuses an order whose current owner has a delivered commit with its check passing; that owner merges it, and the check's recorded run is reused, never rerun by another unit | 3 |
-| 30 | A proof is bound to the build it ran on | none (screenshots and logs are bare file paths) | the run record written by the build check (`tools/build_runtime.py`) carries runtime ID and Core/Toad revisions; `CommsProblemCloseTool` refuses when that runtime does not contain the order's delivered commit | 3 |
-| 31 | An order states its intent as a deletion target | none (orders name a line, so units fix the line: P520 swapped `is_set()` for `except ValueError` around every caller) | the order declares the decision family being deleted, its owner, and a family-wide count from the existing NRA/refactor-audit scanners; completion runs that count on the delivered commit and refuses unless it is zero with no new site of the same kind | 3 |
-| 32 | Where units' time goes, classified | none (`activity.log` has call times but no durations or turn boundaries) | Core's turn records carry each call's start and end and each turn's start reason; `UnitTime` derives per unit and fleet the minutes in model, tool runs (build, check, test, shell), waiting on another owner, waiting on the commander, and idle; the commander marks each large wait necessary or self-inflicted | 6, 7 |
-| 33 | Order progress beside its deadline; the unit never sees a due time | `order.py board`, `overdue`, `due`, `send`; `rotate.py` (before: a due time and "Delivered" only, so merged work drew the strike threat) | `OrderProgress` in Core (`todos.py`), derived on read from the unit's checkout and the repository's PRs, never stored: no commit, committed not pushed, pushed with no PR, open PR, merged; `CommsTodoListTool` returns it per held order. Merged work draws no strike; no first commit within 10 minutes, or no new commit, push or PR change for 15 minutes, is a STALL with the progress line. `CommsOrderTool` sends the unit no due time: the deadline stays the commander's fire line | 1, 7 |
+| 1 | Create a problem with owner, order and deadline | `order.py problem`; `problems.json` | `CommsTodoCreateTool` | `one_order_owner` |
+| 2 | Assign, reassign, drop an order | `order.py send`, `--reassign`, `drop`; `deadlines.json` | `CommsOrderTool`, `CommsTodoTool` assignment and release | `one_order_owner` |
+| 3 | Complete only on the delivered commit | `order.py done` | `CommsOrderDoneTool`: merged to main, another branch, or not merged | `one_order_owner`, `record_plane` |
+| 4 | Close with installed evidence | `order.py close`; `closed.jsonl` | `CommsProblemCloseTool` | `one_order_owner` |
+| 5 | Board, overdue, silent, spinning | `order.py board`, `overdue`; `fleet-board.md` | `CommsTodoListTool` and the fleet tool, derived | `one_order_owner` |
+| 6 | Branch hygiene | `branches-*.txt` by hand | `CommsWorkBranchesTool` | `one_order_owner` |
+| 7 | Strikes and the strike limit | `order.py strike`; `strikes.jsonl` | `Strike` rows, author-declared `StrikeLimit`, one `comms_fire` | `record_plane` |
+| 8 | Verdicts on a unit's tool calls | `judge.py ok`, `back`; `verdicts.jsonl` | Order `Verdict`; send-back revises the Goal and records its strike atomically | `record_plane` |
+| 9 | Plain words in verdicts and strikes | judge and order checks | `PlainText` | `record_plane` |
+| 10 | Refuse soft or vague orders | `order.py` | `OrderText` on every order action | `record_plane` |
+| 11 | What a unit did during its order | `tool_watch.py`; `edits.jsonl`; `judge.py show` | `comms_order_work`: both transcript families by `tool_call_id` | `review_queue` |
+| 12 | What the commander has not judged yet | `judge.py unjudged`; `vibe_watch.py` | `comms_unjudged`: calls and unit messages after the last verdict | `review_queue` |
+| 13 | Public story and roasts | `fleet-narrative.md`; `judge.py roast` | `comms_order_story`, `comms_fleet_story`; send-back reason posted once | `story` |
+| 14 | Heartbeat: next unit every minute | `rotate.py` (60 s loop) | Core-owned scheduled commander wake | `control_plane` |
+| 15 | Standing rules broken right now | `rotate.py broken_rules` | `StandingRule` members derive violations on read | `control_plane` |
+| 16 | Rule review | `rules.py`; `rules.jsonl`; maximum age | Rule instruction segments in the commander's turn context, with provenance | `control_plane` |
+| 17 | Load and forking | `agent-comms-commander capacity`: load and live fork reports; ten-minute beat age | Reporting-only `commander.capacity.Load.of`; no capacity refusal or automatic fork policy | `control_plane` |
+| 18 | Growth alarm | `growth_watch.py`: merges growing src by more than 40 net lines | `commander.config.Repo.main_changes` → `commander_measurements.MergeGrowth.read` | `control_plane` |
+| 19 | Storage hygiene | `storage.py`: checkout limit, cleanup under 30 GB, allocation floor 5 GB | `UnitStorage`, derived from worktree paths and free disk | `control_plane` |
+| 20 | Commander reaches units only through orders | `send_guard.py` (PreToolUse hook) | Harness hook checking Core order tools | `control_plane` |
+| 21 | Paths, repositories, commander identity | `CommanderConfig` from `commander.json` | Core project and registry configuration | `cutover` |
+| 22 | Seeing the fleet: board, strikes, queue, story, alarms | `fleet-board.md`, `order.py board`, `rotate.py`, `judge.py` | Toad commander page rendering Core through `ui_model` | `commander_page` |
+| 23 | An order states the user-visible run that closes it | scratch `rts/order.py send` (`Works when:`) | `OrderText` requires the run; `CommsProblemCloseTool.works` checks it | `record_plane` |
+| 24 | Reassigning past two owners needs the commander's trace | scratch `order.py send --reassign --traced` | Transfer reads Todo owner history and requires traced source location | `record_plane` |
+| 25 | Unit wording gets a correction, not a strike | scratch `vibe_watch.py` | `PlainText` for commander verdicts/strikes; unit wording gets correction | `record_plane` |
+| 26 | Alerts carry only decisions | none; every line interrupts | `messages.AlertKind.require_publishable`, `tools.CommsDecisionTool` | `control_plane` |
+| 27 | Momentum prediction per unit, compared with the outcome | none | Prediction record and later outcome on commander page; `CommitMomentum` is a partial candidate | `commander_page` |
+| 28 | Measured load and self-report | `capacity.py` unit count | `commander.capacity.Load.of` reports orders, overdue, edits and forks; self-report comparison unverified | `control_plane` |
+| 29 | No handoff of finished, checked work | none; scratch transfer moved P507 after its fix was pushed and checked | Transfer refuses delivered checked work; that owner merges; check is reused | `record_plane` |
+| 30 | A proof is bound to the build it ran on | none; screenshots and logs are bare paths | Build run records runtime and Core/Toad revisions; close requires delivered commit | `record_plane` |
+| 31 | An order states its intent as a deletion target | none; P520 renamed a decision rather than deleting it | Order owns deletion family/site/count; delivery refuses nonzero or renamed decisions | `record_plane` |
+| 32 | Where units' time goes, classified | none; activity.log lacks durations and turn boundaries | `UnitTime`: model, build/check/test/shell, dependency/commander waits and idle; classify large waits | `commander_page`, `control_plane` |
+| 33 | Order progress beside its deadline; the unit never sees a due time | `order.py board`, `overdue`, `due`, `send`; `rotate.py` | Derived `OrderProgress`; merged work draws no strike; first commit 10 min, later progress 15 min STALL; units see no due time | `one_order_owner`, `control_plane` |
+| 34 | Several commanders on one machine and bus, one per project | none; a second commander.json install overwrites the first | Registry commander/project ownership; caller from --from; scoped orders, strikes, load, board, alarms and forks; raw-send guard; duplicate project commander refused | `one_order_owner`, `cutover` |
 
 **Dropped deliberately:** old verdicts and `edits.jsonl`. They judged copies of tool calls, and any old call can be judged again from the transcripts.
 

src/agent_comms/commander/capacity.py

@@ -1,43 +1,37 @@
-"""Report commander load and live forks; no command refuses on capacity."""
+"""One commander load report for native tools and the commander's CLI."""
 from __future__ import annotations
 
-import os, time
+import time
 from dataclasses import dataclass
-from pathlib import Path
 
 from ..comms import Comms
-
-BEAT_AGE = 10 * 60
-
-
-def live_forks(forks: Path) -> list[str]:
-    os.makedirs(forks, exist_ok=True)
-    return [n[:-5] for n in os.listdir(forks) if n.endswith('.beat') and time.time() - os.path.getmtime(f'{forks}/{n}') < BEAT_AGE]
-
-
-def beat(forks: Path, name: str) -> None:
-    os.makedirs(forks, exist_ok=True)
-    with open(f'{forks}/{name}.beat', 'a'):
-        os.utime(f'{forks}/{name}.beat')
+from ..field_codec import projected
+from ..tool_result import ToolResult
+from ..turn_digests import OrderEditCountRead
 
 
 @dataclass(frozen=True)
-class Load:
-    """The commander load on one Comms: its open and overdue orders, its unjudged edits, live forks."""
+class Load(ToolResult):
+    """Order and edit debt, with receiving children that themselves command orders."""
     owed: int
     overdue: int
     unjudged: int
     forks: tuple[str, ...]
 
     @classmethod
-    def of(cls, comms: Comms, commander: str, forks: Path) -> Load:
-        from ..tools import CommsUnjudgedTool
-        actor, now = comms.registry.require(commander).incarnation, time.time()
-        owed = [t for t in comms.goals.todos.held_for_commander(actor) if t.pending]
+    def of(cls, comms: Comms, commander: str) -> Load:
+        snapshot, todos, now = comms.registry.snapshot(), comms.goals.todos, time.time()
+        owner = snapshot.require(commander)
+        owed = tuple(t for t in todos.held_for_commander(owner.incarnation) if t.pending)
+        forks = tuple(sorted(
+            thread.name for thread in snapshot.threads.values()
+            if thread.parent == owner.name and snapshot.statuses[thread.name].receiving
+            and todos.held_for_commander(thread.incarnation)
+        ))
         return cls(len(owed), sum(t.overdue(now) for t in owed),
-                   CommsUnjudgedTool.count(comms, commander), tuple(live_forks(forks)))
+                   OrderEditCountRead.count(comms, owner.name), forks)
 
-    @property
+    @projected(view="wire")
     def total(self) -> float:
         return self.owed + 2 * self.overdue + self.unjudged / 10
 
@@ -47,4 +41,4 @@ def __str__(self) -> str:
 
 
 def main(a, ctx):
-    print(Load.of(ctx.comms, ctx.commander, ctx.state / 'forks'))
+    print(Load.of(ctx.comms, ctx.commander))