From 6c431ad6288477549daa317cc8917715f82eb9be Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sun, 13 Sep 2026 23:54:15 -0700 Subject: [PATCH 1/2] Docs: replace the nonexistent hosted-Devin work-order command Hosted Devin work orders were documented as `code-mower devin work-order`, which the packaged CLI never exposed. Point the qualification guide at the maintained DevinWorkOrders library seam and its embedding example, and state what generic session dispatch does and does not verify. Add tests/test_documented_commands.py: every top-level command advertised in current guidance must parse and dispatch under the packaged CLI, and a fabricated command must fail to parse. The check is offline; handlers are replaced with recording stubs. Closes #965 Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 5 +- docs/current-state-and-roadmap.md | 24 ++++ docs/devin-peer-support-qualification.md | 44 +++++++- docs/devin-work-orders.md | 18 +++ docs/public-release-checklist.md | 3 +- tests/test_documented_commands.py | 133 +++++++++++++++++++++++ 6 files changed, 221 insertions(+), 6 deletions(-) create mode 100644 tests/test_documented_commands.py diff --git a/README.md b/README.md index a0a788ef..3a6d685d 100644 --- a/README.md +++ b/README.md @@ -221,7 +221,7 @@ and the [Cloud Data Contract](docs/cloud-data-contract.md). | --- | --- | | Default builders and reviewers | Claude Code + Codex | | Session hosts | Codex, Claude Code, and Cursor qualified; other identities recognized but require explicit handoff/provider transport | -| Devin | Maintained local builder and hosted release-qualification transport; local review remains informational and Devin is not yet a qualified peer orchestrator | +| Devin | Maintained local builder lane and an exact PR-bound hosted work-order library seam (`code_mower.devin_work_orders`, no packaged CLI command); local review remains informational and Devin is not yet a qualified peer orchestrator | | Organizational context | Optional Coworker delivery to approved Claude/Codex/Devin roles | | Work trackers | GitHub Issues by default; Jira Cloud is optional and guarded | | Forge and merge gate | GitHub | @@ -230,7 +230,8 @@ and the [Cloud Data Contract](docs/cloud-data-contract.md). | Slack | Command and authenticated bounded ingress foundation only; no Slack worker delivery, results, or orchestration authority | GitLab, Bitbucket, broad unattended rollout, uncalibrated merge gates, Devin -peer-orchestrator/reviewer parity, a required Graphify dependency, and Slack +peer-orchestrator/reviewer parity, a hosted work-order CLI, runtime role-policy +enforcement, a required Graphify dependency, and Slack worker delivery are not shipped in v1.4.0. The current priorities and boundaries are recorded in [Current State And Roadmap](docs/current-state-and-roadmap.md). diff --git a/docs/current-state-and-roadmap.md b/docs/current-state-and-roadmap.md index 3d5428a4..966b228e 100644 --- a/docs/current-state-and-roadmap.md +++ b/docs/current-state-and-roadmap.md @@ -130,6 +130,30 @@ future hosted-service work. ## Near-Term Roadmap +`v1.4.0` is released and its published artifacts are immutable. The agreed +sequence from the stabilization epic is: + +1. Close the seven `v1.4.0` stabilization implementation children plus the + [#974](https://github.com/codemower-ai/code-mower/issues/974) evidence + verification. These are main-only fixes. +2. Ship those fixes together with Graphify as `v1.4.1` + ([#915](https://github.com/codemower-ai/code-mower/issues/915)). +3. Ship Board as `v1.4.2`. Board work is underway; + [#935](https://github.com/codemower-ai/code-mower/issues/935) is complete. +4. Supervised Slack remains planned for `v1.5.0`; its runtime work is deferred + until the sequence above is complete. + +Devin support is a bounded builder qualification only: a maintained local CLI +builder lane and the exact PR-bound hosted work-order library seam. Hosted +dispatch has no packaged CLI command. Devin review stays informational and +Devin is not a qualified peer orchestrator. + +Role-policy enforcement +([#975](https://github.com/codemower-ai/code-mower/issues/975)) and +effective-authority rendering +([#955](https://github.com/codemower-ai/code-mower/issues/955)) are planned +runtime work, not shipped behavior. + The next three capabilities should ship as independently gated epics rather than one cross-cutting implementation PR. diff --git a/docs/devin-peer-support-qualification.md b/docs/devin-peer-support-qualification.md index 52f888e9..177c82a0 100644 --- a/docs/devin-peer-support-qualification.md +++ b/docs/devin-peer-support-qualification.md @@ -71,9 +71,23 @@ model-quality estimate. ## Hosted API builder deliveries and recovery -Hosted Devin v3 work orders through `code-mower devin work-order`. Time is -public wall time from PR creation to merge; active provider time is -unavailable. Monetary cost is unavailable for both rows. +Hosted Devin v3 work orders were dispatched by the trusted orchestrator through +the packaged `code_mower.devin_work_orders.DevinWorkOrders` library seam; see the +[embedding example](devin-work-orders.md#embedding-example). +No packaged CLI command dispatches a hosted work order, and none is planned +here: a work-order CLI would be a separate product feature with its own issue, +not documentation written ahead of the code. Time is public wall time from PR +creation to merge; active provider time is unavailable. Monetary cost is +unavailable for both rows. + +Generic remote-session dispatch (`code-mower session`, the `RemoteSessions` +seam, and the provider transports beneath it) gives provider-neutral lifecycle +handling: create, status, message, cancel, and result collection. It does not +independently verify issue, branch, pull request, author, base, and exact head. +Only the exact PR-bound `DevinWorkOrders` path makes those observations +independently against GitHub and fails closed when any of them disagree. The +hosted rows below are evidence for that exact-bound path, not for lifecycle +dispatch alone. | Work order | ACU cap | Observed ACU | Public wall time | Active time | Work-order round | Exact-head audit cycles | Intervention | Verified PR / head | Recovery result | | --- | ---: | ---: | ---: | --- | ---: | --- | --- | --- | --- | @@ -278,6 +292,30 @@ branch, author, PR, head-SHA, and base-branch verification are unchanged, and a later valid exact-round collection clears the rejection and returns verified PR evidence. +## Post-release hosted-builder campaign conclusion + +The bounded hosted-builder campaign run after the v1.4.0 release is closed. It +is recorded here as supervised transport evidence only; it does not change the +recommendation posture below. + +- Bounded hosted-builder deliveries: 3 of 5. +- [#865](https://github.com/codemower-ai/code-mower/issues/865) and + [#935](https://github.com/codemower-ai/code-mower/issues/935) were completed + separately by Code Mower Codex in + [PR #971](https://github.com/codemower-ai/code-mower/pull/971) and + [PR #973](https://github.com/codemower-ai/code-mower/pull/973). +- The final #935 recovery attempt was cancelled by the user before delivery. It + is not a completed implementation failure and must not be counted as one. +- Correction to earlier `invalid_request` explanations: the observed failures + were caused by malformed or missing request keys, not by provider-side + rejection of the work order itself. Valid fix and cancel calls in the same + session subsequently succeeded. +- A structured logical completion does not prove that the provider session + exited. Code Mower treats the verified structured result as complete while + the remote session may remain resumable. +- Settled billing for the campaign remains unknown. Observed ACU is a + point-in-time consumption snapshot, never a final invoice. + ## Limitations - Local versus hosted: the local rows report active seconds and the hosted diff --git a/docs/devin-work-orders.md b/docs/devin-work-orders.md index 2a56b853..226eb1ea 100644 --- a/docs/devin-work-orders.md +++ b/docs/devin-work-orders.md @@ -7,12 +7,30 @@ existing `RemoteSessions`, `DevinProvider`, organization-scoped v3 client, priva It does not change campaign dispatch, release policy, review authority, or lane CLI capability declarations. No new lifecycle or automatic polling daemon is introduced. +This is a library seam, not a command. The packaged CLI exposes no hosted +work-order command; an embedding dispatcher calls the library directly, as in +the example below. A work-order CLI would be a separate product feature and +must be tracked as one before it is documented. + +Generic remote-session dispatch and this exact PR-bound path are not +interchangeable. Generic session dispatch — `code-mower session`, the +`RemoteSessions` seam, and the provider transports beneath it — provides +provider-neutral lifecycle handling: create, status, message, cancel, and +result collection. It does not independently verify issue, branch, pull +request, author, base, and exact head; a provider's own claims about those are +not observations. `DevinWorkOrders` adds exactly that independent verification: +every one of those bindings is observed against GitHub by the caller's own +credentials and fails closed on any disagreement. Use the exact-bound path +whenever hosted output becomes builder evidence. + The caller must first apply its trusted issue-author/work-order-comment policy and acquire the repository's existing builder lease. A work-order manifest is not proof of authorization. Pass the approved Markdown body explicitly; the library does not open manifest paths, load context files, or fetch an untrusted issue body. Repository, issue, branch, base, and expected GitHub author ID/login are dispatcher policy inputs. +## Embedding example + ```python from code_mower.devin_sessions import DevinClient from code_mower.devin_work_orders import DevinWorkOrders, WorkOrder diff --git a/docs/public-release-checklist.md b/docs/public-release-checklist.md index ac176c75..0adaaadb 100644 --- a/docs/public-release-checklist.md +++ b/docs/public-release-checklist.md @@ -183,7 +183,8 @@ interpreter. release-hygiene integration tests. - Add static-analysis gates in stages: broaden Ruff for stable subpackages first, then add scoped type checking as module contracts stabilize. -- Add a zero-config first-value experiment, such as `code-mower try OWNER/REPO`, +- Add a zero-config first-value experiment — a future "try this repository" + entry point, not a command that exists today — that can auto-discover recent PR history, generate a draft corpus, and produce a value report without asking a new user to understand the full calibration model first. diff --git a/tests/test_documented_commands.py b/tests/test_documented_commands.py new file mode 100644 index 00000000..bb28cad1 --- /dev/null +++ b/tests/test_documented_commands.py @@ -0,0 +1,133 @@ +"""Documentation contract: advertised top-level commands must exist. + +Current guidance is only useful if a cold operator can run what it names. +These checks are fully offline: command handlers are replaced with recording +stubs, so nothing launches a provider, touches the network, or writes state. + +Historical records are excluded on purpose. The changelog and per-version +release notes describe commands as they were at that release; correcting them +would falsify the record. +""" + +from __future__ import annotations + +import re +import sys +import unittest +from pathlib import Path +from unittest import mock + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from code_mower import cli + + +ROOT = Path(__file__).resolve().parents[1] + +# `code-mower ` where the command is command-shaped, is not a slash +# command (`/code-mower start`), and is not part of a longer token such as +# `code-mower code-mower==1.4.0`. +INVOCATION_RE = re.compile(r"(? list[Path]: + paths = [ROOT / "README.md"] + paths.extend( + path + for path in sorted((ROOT / "docs").rglob("*.md")) + if not HISTORICAL_RE.search(path.as_posix()) + ) + return [path for path in paths if path.is_file()] + + +def _code_segments(text: str) -> list[str]: + """Return fenced-block lines and inline code spans, ignoring prose.""" + segments: list[str] = [] + fence = "" + for line in text.splitlines(): + match = FENCE_RE.match(line) + if match: + marker = match.group(1) + if not fence: + fence = marker + elif marker == fence: + fence = "" + continue + if fence: + segments.append(line) + else: + segments.extend(INLINE_CODE_RE.findall(line)) + return segments + + +def _advertised_commands() -> dict[str, set[str]]: + """Map each advertised command to the files advertising it.""" + advertised: dict[str, set[str]] = {} + for path in _current_guidance_files(): + relative = path.relative_to(ROOT).as_posix() + for segment in _code_segments(path.read_text(encoding="utf-8")): + for command in INVOCATION_RE.findall(segment): + advertised.setdefault(command, set()).add(relative) + return advertised + + +class DocumentedCommandTests(unittest.TestCase): + def test_extractor_finds_the_commands_readme_actually_shows(self) -> None: + # Guards the extractor itself: a silently empty scan would make every + # other check in this file vacuous. + advertised = set(_advertised_commands()) + expected = {"init", "doctor", "session", "lanes", "board", "productivity"} + self.assertTrue(expected.issubset(advertised), msg=sorted(advertised)) + + def test_every_advertised_command_parses_under_the_packaged_cli(self) -> None: + advertised = _advertised_commands() + self.assertTrue(advertised) + calls: list[list[str]] = [] + + def handler(argv: list[str]) -> int: + calls.append(argv) + return 0 + + for command, sources in sorted(advertised.items()): + with self.subTest(command=command): + calls.clear() + self.assertIn( + command, + cli.COMMAND_HANDLERS, + msg=f"{command} is documented in {sorted(sources)} but is not a command", + ) + with mock.patch.dict( + cli.COMMAND_HANDLERS, {command: handler}, clear=False + ): + self.assertEqual(cli.main([command, "--offline-contract-probe"]), 0) + self.assertEqual(calls, [["--offline-contract-probe"]]) + + def test_every_advertised_command_has_a_help_description(self) -> None: + for command in sorted(_advertised_commands()): + with self.subTest(command=command): + self.assertTrue(cli.COMMAND_DESCRIPTIONS.get(command)) + + def test_fabricated_command_is_rejected(self) -> None: + # `code-mower devin work-order` was documented for hosted work orders + # and never existed; hosted dispatch is the DevinWorkOrders library + # seam. A fabricated command must fail to parse rather than dispatch. + for argv in (["devin", "work-order"], ["work-orders"], ["devin-work-order"]): + with self.subTest(argv=argv): + self.assertNotIn(argv[0], cli.COMMAND_HANDLERS) + with self.assertRaises(SystemExit) as raised: + cli.main(argv) + self.assertEqual(raised.exception.code, 2) + + def test_no_document_advertises_a_hosted_work_order_command(self) -> None: + pattern = re.compile(r"code-mower[ \t]+devin[ \t]+work-order") + for path in _current_guidance_files(): + with self.subTest(path=path.relative_to(ROOT).as_posix()): + self.assertIsNone(pattern.search(path.read_text(encoding="utf-8"))) + + +if __name__ == "__main__": + unittest.main() From 7cec22b0ffc54500d0165d4bcea2e86e3abd259d Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Mon, 14 Sep 2026 00:02:11 -0700 Subject: [PATCH 2/2] Docs: read invocations, not prose, and align the roadmap sequence Fix round for the two blocking corrections at 6c431ad. The documentation contract extractor treated every fenced line as a shell invocation, so prose inside the text-fenced prompt block at docs/orchestrator-prompt-pack.md:68 advertised "and" as a command. Extraction is now positional rather than an allowlist of English words: only fences with no info string or a shell info string hold executable examples, and an invocation must sit at a shell command position. A new regression proves the prose case is ignored while fabricated inline and shell commands are still detected, plus a corpus regression pinning the prompt-pack line. docs/current-state-and-roadmap.md no longer contradicts its own checkpoint. The Near-Term Roadmap sections and Delivery Order now carry the #900/#979 sequence: complete v1.4.0 stabilization on main, ship those fixes with Graphify as v1.4.1, then Board as v1.4.2, with supervised Slack deferred to v1.5.0. Board's real implementation state and the existing PR mapping are recorded. The release-baseline sentence and qualification tables are unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- docs/current-state-and-roadmap.md | 160 ++++++++++++++++++------------ tests/test_documented_commands.py | 93 ++++++++++++++--- 2 files changed, 177 insertions(+), 76 deletions(-) diff --git a/docs/current-state-and-roadmap.md b/docs/current-state-and-roadmap.md index 966b228e..b34b6574 100644 --- a/docs/current-state-and-roadmap.md +++ b/docs/current-state-and-roadmap.md @@ -154,72 +154,106 @@ effective-authority rendering ([#955](https://github.com/codemower-ai/code-mower/issues/955)) are planned runtime work, not shipped behavior. -The next three capabilities should ship as independently gated epics rather -than one cross-cutting implementation PR. - -### 1. Devin Peer Support - -Make Devin implement the same user-facing participant lifecycle as Claude and -Codex while keeping local CLI and hosted API mechanics inside separate -transports. The target includes: - -- session dispatch, progress, messaging, cancellation, result collection, and - recovery; -- qualification as a host/orchestrator; -- Coworker and later Graphify context delivery; -- structured current-head reviewer output; and -- a clean/blocked calibration campaign before any reviewer promotion. - -The maintained local builder and hosted Sessions API work are the starting -point. Reviewer authority remains evidence-based. - -### 2. Graphify Repository Context - -Treat Graphify as a repository-context provider beside Coworker, not as a -participant. Start local and code-only: - -- build the optional local provider against the conditions in the - [evaluation record](graphify-evaluation.md), which closes - [issue #876](https://github.com/codemower-ai/code-mower/issues/876) with an - adopt decision; -- add a provider registry and multiple context attachments per session; -- build and refresh graphs with commit/freshness validation — delivered by - `code-mower context-graph`, described in the - [lifecycle record](context-graph-lifecycle.md), which closes - [issue #913](https://github.com/codemower-ai/code-mower/issues/913); -- consume a pinned structured JSON contract; -- generate bounded impact, dependency, symbol, and related-test packets; and -- deliver the same packet shape to Claude, Codex, and Devin. - -Code Mower should own refresh policy and should not rely permanently on parsing -human-oriented MCP prose. - -### 3. Slack Task And Status Interaction - -Treat Slack as an interaction channel, not an orchestrator. A Slack-started -session uses the project or channel's configured default orchestrator unless the -request supplies an explicit one. The first local integration should use Socket -Mode and provide: - -- allowlisted workspace, channel, user, and repository mappings; -- idempotent task creation and Slack-thread-to-session binding; -- redacted progress and completion updates; -- clarification questions with reply-to-resume behavior; -- cancellation and restart reconciliation; and -- no raw private context or private reviewer findings in Slack. - -Slack should consume the durable session lifecycle and event surface introduced -for Devin rather than scrape terminal or Board output. +Each step below is an independently gated epic rather than one cross-cutting +implementation PR. + +### 1. Complete `v1.4.0` Stabilization ([#979](https://github.com/codemower-ai/code-mower/issues/979)) + +Seven main-only implementation PRs plus one evidence verification: + +- accurate advertised commands and live roadmap docs + ([#965](https://github.com/codemower-ai/code-mower/issues/965)); +- optional review defaults + ([#967](https://github.com/codemower-ai/code-mower/issues/967)); +- role-specific qualification and admission + ([#975](https://github.com/codemower-ai/code-mower/issues/975)); +- effective-authority and migration reporting + ([#955](https://github.com/codemower-ai/code-mower/issues/955)), after #975; +- quiescent, capable, consistent takeover + ([#962](https://github.com/codemower-ai/code-mower/issues/962)); +- contributor lineage and reviewer exclusion + ([#963](https://github.com/codemower-ai/code-mower/issues/963)), after #962 + and #975; and +- independent operational acceptance evidence + ([#976](https://github.com/codemower-ai/code-mower/issues/976)). + +#962 runs before #963 where handoff and provenance files overlap. #974 is +evidence-only verification of existing hosted aggregate freshness; a confirmed +hosted defect becomes a separately recorded implementation child and its own +hosted PR rather than an assumed fix. + +### 2. Graphify Repository Context — `v1.4.1` ([#902](https://github.com/codemower-ai/code-mower/issues/902) / release [#915](https://github.com/codemower-ai/code-mower/issues/915)) + +Graphify is a repository-context provider beside Coworker, not a participant. +It is adopted and partly delivered: the +[evaluation record](graphify-evaluation.md) closed +[#876](https://github.com/codemower-ai/code-mower/issues/876) with an adopt +decision, and `code-mower context-graph`, described in the +[lifecycle record](context-graph-lifecycle.md), closed +[#913](https://github.com/codemower-ai/code-mower/issues/913). Remaining work +is the query/packet PR +([#914](https://github.com/codemower-ai/code-mower/issues/914)), which consumes +a pinned structured JSON contract and generates bounded impact, dependency, +symbol, and related-test packets in one shape for Claude, Codex, and Devin. +Release #915 also carries the stabilization docs, review-default, and +operational-evidence work plus explicit freshness evidence. + +Installation stays opt-in, no command requires an index to exist, and Code +Mower owns refresh policy rather than parsing human-oriented MCP prose. + +### 3. Board Clarity And Session Visibility — `v1.4.2` ([#945](https://github.com/codemower-ai/code-mower/issues/945) / release [#952](https://github.com/codemower-ai/code-mower/issues/952)) + +Board implementation is underway rather than unstarted. +[#935](https://github.com/codemower-ai/code-mower/issues/935) is complete and +merged with [#973](https://github.com/codemower-ai/code-mower/issues/973); +[#956](https://github.com/codemower-ai/code-mower/issues/956) and +[#957](https://github.com/codemower-ai/code-mower/issues/957) are drafts behind +main that need refreshing before review. Remaining work is presentation and +producers, persistent Board services +([#961](https://github.com/codemower-ai/code-mower/issues/961)), and integrated +qualification ([#951](https://github.com/codemower-ai/code-mower/issues/951)), +then the release PR #952. #961 is required before #951 and #952, and #951 +consumes #975, #955, #962, #963, and #976 through its integration dependencies. + +Board is a read model over one closed local observation model. Missing or stale +evidence stays explicitly unknown or last-observed; Board never infers runtime +activity from a label, provider name, PID, PR author, lease, or command-line +prose. + +### 4. Supervised Slack Task And Status Interaction — `v1.5.0` ([#903](https://github.com/codemower-ai/code-mower/issues/903) / release [#923](https://github.com/codemower-ai/code-mower/issues/923)) + +Slack is an interaction channel, not an orchestrator. A real qualified Codex or +Claude supervisor controls bounded hosted work: missing supervisor readiness +blocks dispatch, and selecting a provider never promotes its role. Ingress +foundations [#916](https://github.com/codemower-ai/code-mower/issues/916) and +[#917](https://github.com/codemower-ai/code-mower/issues/917) are merged. +Remaining work is OAuth, the qualified-supervisor adapter +([#977](https://github.com/codemower-ai/code-mower/issues/977)), durable +interactions, the bridge, paired telemetry, setup +([#922](https://github.com/codemower-ai/code-mower/issues/922)), and release +acceptance #923. Slack consumes the durable session lifecycle and event surface +rather than scraping terminal or Board output, and carries no raw private +context or private reviewer findings. + +This runtime work is deferred until the sequence above is complete. Board +readiness gates only Slack's end-to-end canary and final acceptance in #923; it +does not block independent Slack OAuth, inbox, interaction, bridge, setup, or +documentation work. ## Delivery Order -1. Ship Devin lifecycle, host, context-recipient, and reviewer parity first. -2. The bounded Graphify evaluation is complete and adopted; ship the optional - local context provider after the shared context registry is stable, against - the conditions in the evaluation record. Installation stays opt-in and no - command requires an index to exist. -3. Define Slack's command and identity contract in parallel, but merge its - worker only after session lifecycle and recovery are stable. +1. Complete `v1.4.0` stabilization on main: the seven #979 implementation PRs + plus the #974 evidence verification. +2. Ship those main-only fixes together with Graphify as `v1.4.1` through #915, + after #914. +3. Ship Board as `v1.4.2` through #952, after #961 and #951. +4. Merge the supervised Slack runtime last, accepted in #923 for `v1.5.0`. + +Elapsed time, implementation difficulty, or an open draft PR never changes this +release order. An explicit evidence-backed Graphify deferral recorded in +#915/#902 may satisfy that one dependency. Merged post-`v1.4.0` fixes, including +#935/#973, count as on main until a later published package is verified to +contain them. Each child issue should produce one reviewable PR with one branch writer, independent current-head review, the normal gate, and package-level validation. diff --git a/tests/test_documented_commands.py b/tests/test_documented_commands.py index bb28cad1..54f34df0 100644 --- a/tests/test_documented_commands.py +++ b/tests/test_documented_commands.py @@ -13,6 +13,7 @@ import re import sys +import textwrap import unittest from pathlib import Path from unittest import mock @@ -24,11 +25,23 @@ ROOT = Path(__file__).resolve().parents[1] -# `code-mower ` where the command is command-shaped, is not a slash -# command (`/code-mower start`), and is not part of a longer token such as -# `code-mower code-mower==1.4.0`. -INVOCATION_RE = re.compile(r"(?` at a shell command position: the start of a line or +# code span, after a prompt marker, or after an operator that begins a new +# command. Prose that merely names the product mid-sentence -- "report +# command -v code-mower and code-mower --version" -- is not an invocation, so +# the following word is not a command. The command itself must be +# command-shaped and not part of a longer token such as `code-mower==1.4.0`. +COMMAND_POSITION = r"(?:^|\$\(|[;&|(])[ \t]*(?:[$>#][ \t]+)?" +INVOCATION_RE = re.compile( + COMMAND_POSITION + r"code-mower[ \t]+([a-z][a-z0-9-]*)(?![\w.=-])" +) +# Only a fence with no info string or a shell info string holds executable +# examples. `text` carries agent prompts, and `json`/`yaml`/`python` carry +# data, so a product name inside them is prose or payload, not a command. +FENCE_RE = re.compile(r"^ {0,3}(```|~~~)[ \t]*([^\s`]*)") +SHELL_FENCE_INFO = frozenset( + {"", "sh", "bash", "shell", "zsh", "console", "shell-session", "terminal"} +) INLINE_CODE_RE = re.compile(r"`([^`\n]+)`") HISTORICAL_RE = re.compile(r"(^|/)(CHANGELOG\.md|release-history\.md|v[\d]+-release-notes\.md)$") @@ -45,33 +58,51 @@ def _current_guidance_files() -> list[Path]: def _code_segments(text: str) -> list[str]: - """Return fenced-block lines and inline code spans, ignoring prose.""" + """Return shell-block lines and inline code spans, ignoring prose. + + Two kinds of text name the product without running it: prose outside code + spans, and prompt or data blocks fenced as something other than a shell. + Neither contributes segments. Inside a fence, backticks are literal text + rather than markdown inline code, so no spans are read from one. + """ segments: list[str] = [] fence = "" + shell_block = False for line in text.splitlines(): match = FENCE_RE.match(line) if match: - marker = match.group(1) + marker, info = match.group(1), match.group(2).lower() if not fence: fence = marker + shell_block = info in SHELL_FENCE_INFO elif marker == fence: fence = "" + shell_block = False continue if fence: - segments.append(line) - else: - segments.extend(INLINE_CODE_RE.findall(line)) + if shell_block: + segments.append(line) + continue + segments.extend(INLINE_CODE_RE.findall(line)) return segments +def _commands_in(text: str) -> set[str]: + """Return the top-level commands one document advertises as runnable.""" + return { + command + for segment in _code_segments(text) + for command in INVOCATION_RE.findall(segment) + } + + def _advertised_commands() -> dict[str, set[str]]: """Map each advertised command to the files advertising it.""" advertised: dict[str, set[str]] = {} for path in _current_guidance_files(): relative = path.relative_to(ROOT).as_posix() - for segment in _code_segments(path.read_text(encoding="utf-8")): - for command in INVOCATION_RE.findall(segment): - advertised.setdefault(command, set()).add(relative) + for command in _commands_in(path.read_text(encoding="utf-8")): + advertised.setdefault(command, set()).add(relative) return advertised @@ -83,6 +114,42 @@ def test_extractor_finds_the_commands_readme_actually_shows(self) -> None: expected = {"init", "doctor", "session", "lanes", "board", "productivity"} self.assertTrue(expected.issubset(advertised), msg=sorted(advertised)) + def test_extractor_reads_invocations_and_not_prose(self) -> None: + # Prose that names the product mid-sentence, and prompt blocks fenced + # as `text`, are not invocations: the word after the product name is + # English, not a command. The rule is positional rather than a list of + # excluded words, so an unseen prose word cannot slip through and an + # unseen command is still caught. + document = textwrap.dedent( + """\ + Run `code-mower inline-fabricated --json` first, and before and + after the install report command -v code-mower and code-mower + --version. + + ```text + Before and after the install, report command -v code-mower and + code-mower --version. Then run code-mower prompt-fabricated. + ``` + + ```bash + code-mower shell-fabricated --repo OWNER/REPO + cd repo && code-mower chained-fabricated + ``` + """ + ) + self.assertEqual( + _commands_in(document), + {"inline-fabricated", "shell-fabricated", "chained-fabricated"}, + ) + + def test_prose_in_the_prompt_pack_is_not_read_as_a_command(self) -> None: + # Regression for the shipped corpus: docs/orchestrator-prompt-pack.md + # tells an agent to "report command -v code-mower and code-mower + # --version" inside a `text` prompt block. + advertised = _advertised_commands() + self.assertNotIn("and", advertised) + self.assertIn("doctor", advertised) + def test_every_advertised_command_parses_under_the_packaged_cli(self) -> None: advertised = _advertised_commands() self.assertTrue(advertised)