Make plan and code reviews faster and need fewer rounds #98

Goal

Make the plan-lifecycle review loops (plan review and code review) and the yolo flow faster and better: fewer rounds, more real defects found per round, less reviewer time per review loop, and no weaker reviews. Token cost is not a limit: the user accepts more tokens when the reviews find more defects or finish sooner. Improve the reviewer prompts, and give each reviewer role a skill with reference checklists, reusing docks skills where they fit. Build a reusable benchmark that replays past review rounds and blind seeded cases, so that every reviewer change is measured, kept only when it wins, and the reviewers can keep improving later. Every change carries its evidence and gets reviewed. The plugin already works well, so a change lands only when the evidence shows that it is better. The change stays portable across Claude Code, Codex, OpenCode, and omp, with omp first; the user runs the reviewers on GPT 6.1 sol through omp's per-agent model setting.

Yolo: the user delegated every decision except the merge to the agent.

Mode: plan-and-implement

Research

Where review time goes today

Times run from the reviewer's first prompt to its final answer, read from the omp transcripts of earlier runs; every reviewer ran on the user's model through the code-reviewer and plan-reviewer agents.

• A round is 11 to 19 model turns with about 6 tool calls each, at 14 to 34 s per turn. Model turns set the time; the reviewers already batch their reads.
• Every round reviews the whole candidate again. Fix #90 and let agents report skill problems to this repository #96 code round 2 re-read the full 2,499-line diff to confirm a 94-line fix (180 s, about 1.5 M tokens).
• 7 of the 12 rounds above checked a small repair or a one-sentence edit. The yolo rule "fix every finding of every severity, then review again" turns each MEDIUM or LOW finding into one more round.
• One reviewer finds few defects per round, and the result varies widely between runs (benchmark 1).

Benchmark 1: lenses and delta rounds

Benchmark 1 replayed #96 on frozen snapshots, with 15 reviewers running at the same time on the user's model. The raw blocks, times, and token counts are in $(git rev-parse --git-path docks-review)/yolo-98/benchmark.md.

The defects are D1, the missing owner recheck at the pre-write read (found by #96 round 1); D2, a defect in shipped skill text (both text lenses rated it HIGH; details went to the repository owner outside this issue); D3, repo-plan-cli does not require bun-1 because runner discovery sees only its Node runner (scripts/tests/test-contracts.mjs); D4, no test exercises the "changed during the settle wait" refusal; D5, the fake gh returns merged and mergedAt even when a query does not select them; D6, a test asserts request text (wiring); and D7, the report-section verify command accepts empty output. All 18 findings of these four runs were real: precision was 100%. D2 to D7 shipped in #96; this plan does not fix them (outside the Goal). No benchmark reviewer found the defect that #96's self-review found (the endless "run reopen again" message).

I checked each defect on main (e09d273f). D3: in a temporary worktree with "toolchains": ["node-24"] for repo-plan-cli, bun scripts/tests/test-contracts.mjs still prints PASS test_contracts contracts=17 discovered=17 selected=17. D4: no case in scripts/tests/plan-cli/reopen.test.mjs names that refusal. D5: the fake gh builds the pull request node with merged and mergedAt before it reads the selection. D6: reopen.test.mjs checks that the request text contains addAssigneesToAssignable. D2 and D7 hold in the current text as well.

Plan review of #96 round 1: one reviewer (307 s, 1.20 M tokens) and two lenses (220 s for the slower lens, 2.50 M tokens) each found 2 of the 6 findings of the historical round 1. A delta plan round on the round-2 body took 246 s and 0.81 M tokens, against 200 s and about 0.95 M tokens for the historical full round 2.

External practice, October 2026

• Find broadly, verify, then post. Anthropic's Code Review runs narrow finders in parallel, then verifies each finding before posting; its prompting guides say a filter phrase such as "only report high-severity" makes the model investigate fully but report less (https://code.claude.com/docs/en/code-review; https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8). Cursor's Bugbot "was too cautious" until it "shifted to aggressive prompts" (https://cursor.com/blog/building-bugbot, 2026-01-15).
• List every qualifying finding in one pass. Codex's /review rubric: "Do not stop at the first qualifying finding. Continue until you've listed every qualifying finding", and its finding bar requires that a finding is discrete, introduced by the change, not an unstated assumption, and that "one must identify the other parts of the code that are provably affected" (https://github.com/openai/codex/blob/main/codex-rs/prompts/templates/review/rubric.md, 2026-07-21). An empty review is a valid answer; a "3-5 findings" cap made reviewers pad, and removing it raised recall from 47 to 75 of 96 (docs(pstack): cut 19 more instructions Opus 5.5 does not need cursor/plugins#419, 2026-09-23).
• Rounds converge with delta review, no repeats, and defect classes. The Claude Code team found that "about 59%" of repeat flags were code from the first pass; diff-only re-review, "no repeats of findings already posted", and "flag the class of bug, so the author fixes every instance" cut mean reviews to pass from 2.08 to 1.32 (https://claude.dev/blog/solving-code-review-at-anthropic/, 2026-10-15). Bugbot reviews "only the changes since the previous Bugbot review" and reads earlier comments "to avoid duplicate suggestions" (https://cursor.com/docs/bugbot); Cloudflare's re-review re-emits unfixed findings (https://blog.cloudflare.com/ai-code-review/, 2026-04-20).
• Evidence for each finding. Codex asks for "the triggering input, expected behavior, and actual behavior" (https://learn.chatgpt.com/docs/code-review); Anthropic's proof requirement raised substantive comments from 16 to 54% (https://claude.com/blog/how-anthropic-secures-its-ai-native-software-development-lifecycle, 2026-07-21).
• Lean prompts for GPT-6. OpenAI's GPT-6 guidance: "overly specific guidance can now hinder results", the model is "more sensitive to instructions contained in skills", state precedence, and make the root document "a minimal router that points to supporting docs" (https://developers.openai.com/blog/rethinking-skills-and-prompts-for-gpt-6-astra, 2026-09-11; https://developers.openai.com/api/docs/guides/latest-model?model=gpt-6.1-sol). Cursor saw Sol models react strongly to emphasized wording (https://cursor.com/blog/agent-swarm-model-economics, 2026-07-20). OpenAI's own multi-agent example reviews a pull request "with three agents: one for correctness, one for security, and one for missing tests" (https://developers.openai.com/api/docs/guides/responses-multi-agent).
• Scale the panel to the change. noodle runs one reviewer under 50 changed lines (https://github.com/poteto/noodle/tree/main/.agents/skills/adversarial-review); Anthropic: "Large or complex changes get more agents… trivial ones get a lightweight pass" (https://claude.com/resources/articles/code-review, 2026-03-09).
• Plan review: check facts, not personas. poteto: "Don't review an abstract plan adversarially" (https://github.com/cursor/plugins/tree/main/pstack, 2026-10); Anthropic warns that "a reviewer prompted to find gaps will usually report some, even when the work is sound" (https://code.claude.com/docs/en/best-practices).
• Benchmark method. Mix historical and seeded defects, keep seeded defects to half the set or less, freeze a held-out split, match findings with a judge from another model family, run each arm more than once, and read the transcripts (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents, 2026-01-09; https://github.com/withmartian/code-review-benchmark/blob/main/methodology/full.md; https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/, 2026-10-05; https://arxiv.org/abs/2606.15689).

Runtime facts

• All four runtimes run several subagents at once: Claude Code (https://code.claude.com/docs/en/sub-agents), Codex (https://developers.openai.com/codex/subagents), OpenCode (https://opencode.ai/docs/agents/), and omp (omp://tools/task.md).
• omp resolves a subagent's model by agent name ("1. task.agentModelOverrides[agentName]…"), and "Claude-dialect plugin agents discard their frontmatter model" (omp://task-agent-discovery.md). Every dispatch of code-reviewer therefore runs on the user's reviewer model; a new agent name would not.
• omp's orchestrate is a prompt keyword that "applies only to the turn containing the keyword" (omp://magic-keywords.md), not an API that a skill can call.
• parseReviewComment in plan.mjs reads only the first verdict line of a comment (.find(Boolean)); a comment with several reviewer blocks would lose a later repair or blocked.
• Agent files cannot carry references: claude plugin validate lints every agents/*.md (plugins/plan-lifecycle/skills/AGENTS.md, Roles). References live in a skill folder, and plan-reviewer.md already loads its skill through ${CLAUDE_PLUGIN_ROOT}.

Decisions

1. Code review round 1 is a three-lens panel for a diff of 50 or more changed lines. Round 1 is a panel round when the full diff changes 50 or more lines. Below that, and in a runtime that cannot run subagents at the same time, round 1 is a full round with one reviewer. A user who wants to save tokens can ask for a full round. A panel dispatches the code-reviewer agent three times in one message, with the lenses correctness, spec-and-tests, and text-and-safety; every dispatch shares the same opening text and puts its lens last, so the runtimes can reuse the prompt prefix. Each lens gets the full diff and a focus diff of the paths in its scope. The manager posts the returned blocks unchanged in one issue comment, in lens order. Evidence: benchmark 1, External practice, and step 6: with the new skill, the panel found more than one reviewer on both dev cases (code-96 0.21 against 0.14, code-89 0.83 against 0.67), it found more on code-96 in every comparison, and it tied on the held-out case at the ceiling; its round time stayed close to one reviewer's (136 to 151 s against 116 to 158 s); it costs about twice the tokens per loop, which the user accepts (Request corrections). Rejected: one reviewer by default with a panel only for high-risk changes (the step 6 correction, reversed after the user removed the token limit); new agent files per lens (omp would not apply the user's code-reviewer model); a reviewer that dispatches lenses itself (the wrappers forbid dispatch); omp's orchestrate keyword (one turn only).
2. A later code-review round is a delta round when it only checks repairs of a completed round. The manager commits the candidate before each round; the commit is the round's reviewed head. Round k after round 1 is a delta round only when round k-1 completed (every reviewer returned a block with a verdict for that head, and no block returned blocked), every change since that head repairs a finding of round k-1 or of the self-review, and the delta diff is at most half as long as the full diff. Otherwise, and after a merge of the default branch, the round is a panel or a full round by the size rule of round 1. A reviewer that fails or returns no block with a verdict is dispatched again before the round is posted. One delta reviewer gets the delta diff, the previous round's comment, and the manager's rejection comment for that round, if any: it marks each previous finding fixed or open and repeats an open one, reviews every delta hunk while tracing changed symbols to callers and callees, and repeats no finding that the manager rejected with evidence. Evidence: benchmark 1 and External practice.
3. The helper reads every verdict line. parseReviewComment returns the most severe verdict among the comment's verdict lines of the comment's kind: blocked, then repair or fixes-required, then pass. A verdict line with an unknown word makes the comment unreadable as a review; a verdict line of the other kind does not count; the severity upgrade for a code pass stays. Rejected: one comment per lens (the latest comment wins and could hide an earlier failure); round grouping (headings carry no required round number).
4. Each reviewer gets a lean prompt with a router skill and reference checklists. A new internal skill code-reviewer holds the finding bar, the evidence rules, the round modes, and the output contract, and routes each lens and the delta mode to one reference file (references/correctness.md, references/spec-and-tests.md, references/text-and-safety.md, references/delta.md); the code-reviewer agent loads it the way plan-reviewer loads its skill, and a lens loads the matching docks code-review reference when docks is installed. The plan-reviewer skill gets the same finding bar and a checklist reference of checkable plan facts (references/checklist.md). The new text asks for every evidenced defect in one pass, also a LOW one, because the manager reproduces each finding; accepts an empty review; sets no count limit; requires the defect and the input or scenario that shows it, with any assumption stated; names the defect class and every instance in the diff; treats the plan's recorded decisions as settled; states precedence; and drops emphasis and repetition. The current prompts stay the baseline; step 6 keeps each new prompt only when the benchmark shows it is better (decision 8).
5. The manager fixes a defect class at once, and overlaps waits. When a finding names a defect class, the fix covers every instance in the candidate in the same commit. In yolo, the manager dispatches code review round 1 first and runs the self-review while it runs; the self-review's fixes go into the delta round with round 1's fixes, also when round 1 passed. While any round runs, the manager does work that does not depend on the verdict, such as acceptance records and the run summary draft. Evidence: External practice (defect classes; a lead that keeps working while subagents run).
6. A project-local benchmark measures every reviewer change. .agents/skills/review-benchmark/ holds the cases, a prepare.mjs script that builds each case workspace (a snapshot without git history, the full, focus, and delta diffs, the plan file, and one dispatch prompt per reviewer and arm), a score.mjs script that parses the returned blocks, prefilters candidate pairs by file or section, and reports recall, valid extra findings, wrong findings, unresolved judgments, verdicts, time, and tokens, and a SKILL.md procedure. The procedure dispatches the reviewers through the runtime and then a blind judge agent from another model family in two separate passes: the match pass pairs each finding with a known defect (match, no match, or unknown), and the adjudication pass checks each finding left unmatched against the snapshot and rates it valid, wrong, or unknown. Each arm runs at least twice with the arms interleaved; the report gives the median and range; held-out answer keys stay closed until the decision; and each escaped defect becomes a new case. A unit test covers the parser and the scorer.
7. Benchmark cases and splits. Dev: code-96 (the repository copy lists 7 known defects; D2's details stay with the owner), code-89 (1 historical and 5 seeded defects), plan-96 (6 known findings), and plan-89 (3 known findings). The seeded code-89 defects and the plan-89 findings entered the prompt author's context during case preparation, so both are dev cases. Held-out: code-H1 (pull request fix(plan-lifecycle): fix #49, #67, #68 and tighten the plugin instructions #70 for plan plan-lifecycle: fix #49, #67, #68 and tighten the plugin instructions #69, 1,204 diff lines, 6 seeded defects) and plan-H1 (plan plan-lifecycle: tear down what a plan created #83, 5 seeded defects), built by a separate agent that returned only paths and counts; their answer keys stay closed until the final run.
8. Decision rule, fixed before any candidate runs. The current workflow is the baseline. Code-review arms: B0, one reviewer with the current prompt (today); B1, the three-lens panel with the current prompt; C1, one reviewer with the new skill; C3, the panel with the new skill. Plan-review arms: the current prompt against the new skill, and the full re-review round against the delta mode.
   • A prompt change (C1 against B0, or the new plan skill against the current one) is kept when its mean recall is higher on the dev cases and at least equal on the held-out cases, its wrong findings do not increase, and its median round time and tokens stay within 25%.
   • The panel policy (C3 against C1, or B1 against B0 when the prompt change loses) is kept when its recall is higher on the dev cases and at least equal on the held-out cases, its wrong findings do not increase, and its modeled loop cost is within 25% of B0's tokens and below B0's time. The modeled loop is round 1 plus the follow-up rounds of the historical loops (Optimize the gate and the plan helper after the Bun switch #89 code review: 1 follow-up; Fix #90 and let agents report skill problems to this repository #96: 2), priced at a measured delta round for the new policy and at a measured full re-review round for B0.
   • A delta mode is kept when it catches the planted repair defects at least as often as a full re-review and costs less.
   • A losing candidate gets at most two more dev iterations, each based on read transcripts, and is dropped if it still loses; the workflow then keeps today's prompt, single reviewer, or full re-review for that part.
   • After step 6, the user removed the token limit (Request corrections). In later decisions, recall and round time decide; tokens are reported but do not block a change.
9. Rejected after research. A separate verification stage: benchmark 1 showed 100% precision, and the manager already reproduces each finding before it fixes it. Repeated shuffled passes with voting and cross-family panels: higher cost, and per-subagent models are not portable. Reasoning-effort tuning: omp needs task.enableEffort, and the other runtimes differ. Blocking only P0 and P1: the yolo landing rule needs a pass with no open finding, and delta rounds make the extra round cheap. A deterministic plan linter: the historical plan findings were semantic. Plan lenses: benchmark 1. Running the shipped helper with Bun instead of Node (the user asked why the skills call node): Bun starts the helper in 13 ms against 26 ms for Node, but one show takes about 600 ms because of its GitHub call, consumers may not have Bun installed, and the gate tests the shipped helper on Node (root AGENTS.md, Commands).

Request corrections

Asked: "uses the orchestrate from omp, in the plan-lifecycle". Did: the manager dispatches the lens reviewers in parallel in every runtime that the plugin supports. Reason: orchestrate is a one-turn omp prompt keyword, not an API (omp://magic-keywords.md), and all four runtimes run subagents at once (Runtime facts).

Asked: "not only optimize the code reviewer but the plan reviewer too". Did: plan review gets a better prompt and a checklist reference, and a delta mode under the benchmark rule; it does not get lenses. Reason: benchmark 1 showed no gain from plan lenses, and External practice advises checking plans against facts instead of adversarial personas.

Asked, after the benchmark: "dont care about spending more tokens to be faster". Did: the Goal drops the token limit, and decision 1 makes the panel the default round 1 for a diff of 50 or more changed lines again. Reason: the panel found more defects than one reviewer at about the same round time (Verification Results).

Engineering standards

• Steps 1 and 2: tdd-workflow (spec tests before the parser change; red baseline in the main context) and code-clarity.
• Step 3: write-skill rules for skill and reference text, including its trigger and near-miss check (a12) and the plugin load check (a11); the agent rules in the root AGENTS.md (Authoring agents); and code-clarity for wording.
• Step 4: code-clarity for the contract, skill, wrapper, and README text.
• Step 5: test-coverage for the scorer tests, code-clarity, and the repository's unit-test registration rules.
• Step 6: the benchmark procedure of step 5.
• Commits: commit-discipline. Self-review during code review round 1: code-review with code-clarity.

Steps

Acceptance

Do not touch

• Review trust (assignee and author), the verdict words, the severity upgrade, the yolo round ceiling, and the landing rules.
• The defects D2 to D7 that benchmark 1 found in Fix #90 and let agents report skill problems to this repository #96's code; this plan reports them and does not fix them.
• The docks plugin.
• The held-out answer keys before the final benchmark run.

Open questions

Verification Results

Red baselines (steps 1 and 2)

• node --test scripts/tests/plan-cli/reviews.test.mjs against the 0.16.0 helper: 2 failed, 6 passed (the several-block cases).
• The prose-verdict cases against 5512b45e: a verdict in a prose line read as undefined (expected fixes-required); a quoted earlier verdict read fixes-required (expected pass); a list-item verdict read pass.
• After the fix: the plan CLI suite passed 66 of 66, and plan.mjs line coverage was 99.68%.

Benchmark (step 6)

Every arm ran as an omp project agent on openai-codex/gpt-6.1-sol at the default effort, two runs per arm, against the 0.16.0 reviewers from e09d273f. A blind judge from another model family matched and rated every finding. No arm had a wrong finding. The full tables are in .agents/skills/review-benchmark/results.md.

Two dev iterations changed the code prompt: it1 stops a full round from loading all lens references (code-89 time 217 s to 167 s), and it2 adds the coverage stance and a shorter finding line (time within 9% of B0 on dev).

Correction to the panel row: by the letter of decision 8 the panel passed (dev recall 0.52 against 0.40; held-out equal; modeled loop tokens +19% for #89 and -10% for #96 against B0 with full re-reviews). That loop model credited the delta rounds' savings to the panel. Like for like, with delta follow-ups in both policies, the panel loop costs 2.93 M against 1.56 M tokens for #89 and 3.26 M against 1.73 M for #96 (+88%). With the new skill, the panel (C3 it2) found more than one reviewer (C1 it2) on both dev cases: code-96 0.21 against 0.14, and code-89 0.83 against 0.67, where one of its two runs found two more defects. With the old prompt, the panel (B1) found more on code-96 (0.57 against 0.00) and tied on code-89 (0.67). Every arm tied on held-out code-H1 at the ceiling. B1 beat C3 on dev, so the new prompt did not help a panel. The panel buys a modest recall gain for about 88% more loop tokens, so decision 1 first made it an option for high-risk changes.

Decision change after the correction: the user removed the token limit ("dont care about spending more tokens to be faster"). The panel found more defects than one reviewer at about the same round time (C3 it2 136 to 151 s against C1 it2 116 to 158 s), so decision 1 makes it the default round 1 for a diff of 50 or more changed lines again.

Limits: code-H1 sat at the ceiling; the dev cases favor the lens texts, which name defect kinds from benchmark 1; two runs per arm detect only large differences; reasoning effort changes recall and time. The code and plan prompt gains are one defect or one finding, which this benchmark cannot prove, so both prompts count as not worse until follow-up plan #100 judges them on a larger benchmark of real defects.

Checks so far

• a1: bun scripts/ci.mjs passed for the repo-wide checks and both plugins after d0fa1f99, on the round 2 repair before 7f7fdbe6, on the panel-default change before db2f2a39, and on the round 4 repair before 4db65007; bun scripts/ci.mjs --plugin plan-lifecycle passed after 845872a1.
• a2: node scripts/tests/plan-cli.mjs: 66 tests, 66 pass, 0 fail.
• a3: bun run test:coverage: plan.mjs line coverage 99.68%.
• a4: the agent guard passed; code-reviewer 15 and plan-reviewer 15 (floor 14).
• a5: productivity/code-reviewer 15 and productivity/plan-reviewer 15 (floor 8).
• a6: 31 shipped skills, one report-section digest (a27c6a6c…), count 31.
• a7: bun test ./scripts/tests/unit/review-benchmark.test.mjs: 17 pass before code review, 23 after the round 1 and round 2 repairs (6 new tests for unresolved and unknown pair judgments, cost medians, and name collisions; they fail against the scorer at 845872a1), and 24 after the round 4 repair (missing pair answers). A smoke case over e09d273f..db2f2a39, whose range changes 30 paths in the benchmark folder, gave 0 benchmark-folder paths in the full diff and in each focus diff. At 4db65007, prepare.mjs case built all 10 case workspaces with their arms, and no generated diff named a benchmark-folder path.
• a8: the tables above and the results log.
• a11: omp -p --plugin-dir ./plugins/plan-lifecycle resolved skill://code-reviewer to the working-tree plugins/plan-lifecycle/skills/productivity/code-reviewer/SKILL.md (first heading # Code Reviewer) and listed code-reviewer and plan-reviewer among the dispatchable agents. The Claude Code check did not run: the local Claude Code CLI (2.1.234) prints "Not logged in · Please run /login", so the omp check stands alone.
• a12: one fresh subagent per query with the 31-skill listing. Should-trigger: 9 of 9 chose code-reviewer, held-back 3 of 3. Near-miss: 9 of 9 routed away (code-review, security, plan-reviewer, test-coverage, fix-workflow, code-clarity, react-component-patterns, skill-agent-pipeline, plan-manager), held-back 3 of 3.
• Plan review: round 6 (full, new plan-reviewer skill) took 165 s and 1.12 M tokens and returned repair with one research gap; round 7 (delta) took 50 s and 0.19 M tokens and passed; round 8 (full, after the token-limit change) took 108 s and 1.13 M tokens and passed.

Code review rounds (a9)

Reviewer time runs from the first prompt to the final answer; tokens are input plus cached input plus cache writes plus output.

show 98 with the branch helper printed code=pass for the three-block round 1 comment. The self-review took about twice as long as the slowest lens, so it, not the panel, set when the round 1 repairs could start.