Build a larger review benchmark from real defects and measure reviewer prompts, efforts, and service tiers #100

Goal

Build a review benchmark that can judge reviewer setups, as a follow-up to #98 (pull request #99). The #98 benchmark had 13 known dev code defects and 6 seeded held-out code defects. Every arm found all held-out defects, and one defect moved recall by 0.14 to 0.17, so the benchmark showed that delta rounds are faster but could not show whether the new prompts find more defects. Collect 50 or more defects from four sources: escaped defects (fixed after a reviewed merge), yolo self-review findings, defects that reviewers missed (found by a later round, another lens, or after the review passed), and seeded defects. Run each configuration 3 times, keep a held-out split closed until the decision, and then judge again the #98 code and plan prompts, the three-lens panel, and the lead from #98 that lens text in the dispatch message found more than lens text in a reference file. Token cost is not a limit; defects found, round time, review-loop rounds, and loop time decide.

Measure the reviewer model at every reasoning effort, with and without OpenAI's priority service tier (omp's fast mode): for openai-codex/gpt-6.1-sol, the efforts low, medium, high, and xhigh, each with the priority tier and with the default tier. The report shows, for each of the eight configurations, the time and the defects found per round, and the rounds and the time that a whole review loop takes. The benchmark documents how to run the same grid for a new model, so that each new reviewer model gets the same table.

Also answer these leads from #98:

• The yolo self-review set the time of code review round 1 (504 s, against 261 s for the slowest lens). Measure whether a split self-review, or no self-review when a panel runs, keeps its unique findings and finishes sooner.
• A finding without a match and without an adjudication shows only in the per-run Unjudged column. Decide whether the report must stop on it, as it stops on a missing pair answer.
• The code reviewer reports every evidenced defect, also a LOW one, and yolo fixes every reproduced finding before the next round. Count the extra findings and the rounds they cause, so the benchmark can show whether this stance adds rounds.
• The judges rated no finding wrong in either Make plan and code reviews faster and need fewer rounds #98 wave. Spot-check a sample of match and adjudication answers against the snapshots, and record the agreement.

Yolo: the user delegated every decision except the merge to the agent.

Mode: plan-and-implement

Research

Defect sources

Two inventories ran on main at 936ad562 with read-only git and gh commands. They recorded identifiers only, so that the plan author does not see the defects and a blind held-out split stays possible. The lists are in the review scratch directory, plan-100-research/.

• Escaped defects: 98 fix commits screened; 32 defects qualify (introduced by a reviewed plan pull request, with a fix that isolates to the defect in 20 cases and partly in 12). Categories: 15 shipped skill text, 12 code, 4 test, 1 docs. They come from 16 plan pull requests.
• Review record: 148 issue comments read; 45 unique defects qualify with a fix: 10 self-review findings, 14 findings that only one lens of a panel reported, 21 defects that a later round found in code that an earlier round had reviewed, and 7 defects found after a plan's review passed (D3 to D7 of Fix #90 and let agents report skill problems to this repository #96 and the parser regression that pull request Fix: quoted review headings start no block #101 fixed). The fix isolates in 16 cases and partly in 36 rows.
• The two lists share 8 fixing commits, so the real candidates are about 69 defects before the case builder removes duplicates and defects without a usable snapshot. That covers the 50-defect target without seeded defects; seeded defects fill categories with few real defects (for example docs).
• D2 of Fix #90 and let agents report skill problems to this repository #96 is out of scope for public case files: the user chose a neutral public fix for it in report-skill-issue: check repository visibility before a report names a repository #103.

Reviewer model settings

• The user's omp config routes code-reviewer and plan-reviewer to the reviewer role, openai-codex/gpt-6.1-sol:xhigh, and sets tier.openai: priority for every OpenAI request. defaultThinkingLevel is high. Production reviews therefore run as xhigh-fast, which is the baseline column of the grid. The config also sets retry.usageAwareFallback: true with anthropic/claude-opus-5-5:xhigh as the reviewer fallback, so a long sweep that reaches a usage limit can move a run to another model without an error.
• The Make plan and code reviews faster and need fewer rounds #98 benchmark agents (review-bench-*) had their own model line, so they did not get the role's effort: the transcripts of those runs record thinkingLevel: high and the service tier priority. The Make plan and code reviews faster and need fewer rounds #98 numbers therefore describe high, not the xhigh that production reviews use. A scan of the Make plan and code reviews faster and need fewer rounds #98 reviewer transcripts found no assistant message from another model; one plan run holds a model_usage side call of gpt-6-luna (362 tokens), which parseTranscript does not count.
• omp sets the effort of an agent-file subagent with the frontmatter key thinking-level (or thinking), and a model selector can carry a :<level> suffix (omp://task-agent-discovery.md, omp://models.md). The tier of a spawned agent comes from tier.subagent (default inherit) or task.agentServiceTierOverrides.<agent>; one omp -p run takes --service-tier <tier> (omp://settings.md, the tier.* and task.agentServiceTierOverrides rows; omp --help).
• Experiment (omp 18.12.6): omp -p --model openai-codex/gpt-6.1-sol:low --service-tier default and ...:xhigh --service-tier priority, each with a one-word prompt. Each transcript records the requested effort (thinking_level_change, thinkingLevel), the requested tier (service_tier_change), and, on the assistant message, the tier that the response reports (serviceTier: "default" with premiumRequests: 0, and serviceTier: "priority" with premiumRequests: 1). A run can therefore prove which effort it asked for and which tier served it.
• Benchmark 1 of Make plan and code reviews faster and need fewer rounds #98 dispatched lens reviewers at the task tool's highest effort and took 250 to 577 s per lens, against 136 to 159 s at high, and found more on code-96. Effort changes both recall and time, so the grid needs both.
• The user is not sure that the priority tier makes reviews faster; the grid measures it at each effort.

Harness facts

• prepare.mjs agents writes one baseline and one candidate agent per role with one shared --model; it writes no effort and no tier (prepareAgents, renderAgent).
• score.mjs transcripts records only seconds and tokens per reviewer (parseTranscript, transcriptsCommand); the report has no column for model, effort, or tier.
• Case files have no field for a defect's fix patch (KIND_FIELDS, FILE_FIELDS, validateCase), so a review loop cannot be replayed.
• Make plan and code reviews faster and need fewer rounds #98 decision 9 rejected effort tuning because omp needs task.enableEffort and other runtimes differ. The user now asks for the sweep; the sweep sets effort per agent file or per run, not through task.enableEffort, and the recommended setting stays a user setting (the plugin's agents carry no model key, root AGENTS.md).
• Reviewers run as subagents whose working directory is the live checkout on main, which holds the fixes of every corpus defect, dev and held-out. A reviewer that reads the live tree (a path under it, or a Grep or Glob without a path) would see fixed code, and recall would measure leakage. A scan of the 96 Make plan and code reviews faster and need fewer rounds #98 reviewer transcripts found no such read: every read named the workspace, the reviewer's own skill and agent files, skill:// URIs, artifact:// output, or public web pages. omp gives a task subagent no working directory of its own, so the harness detects the reads instead of preventing them.

Statistics

• With 50 or more defects and 3 runs, a recall difference of about 15 points between two configurations is detectable; one defect moves recall by 2 points or less.
• Configurations run on the same cases, so comparisons are paired: a bootstrap over cases (10,000 resamples) gives a 90% interval for each difference in recall and in loop time.

Decisions

1. Corpus. A separate case-builder agent builds the cases from the two inventories and from seeded defects, and returns only paths and counts. Each code case is the reviewed candidate of the introducing change (its base and the reviewed head, or the round head), the plan export of that time, and a key with each defect's id, locator, category, source, and fix patch. The builder removes duplicates, keeps at least 50 defects, and splits by case into dev (about 40%) and held-out (about 60%), balanced by category and source. Plan cases come from historical plan-review findings in the same way, without fix patches. During the runs, the held-out keys live outside the repository ($XDG_STATE_HOME/docks/review-bench/keys/, mode 0700), because reviewers can search the checkout; score.mjs (readKnown, the pairs, report, and loop commands) and prepare.mjs (the loop command) take a --keys <dir> option that reads them there, references/judge.md names that input for the judge, and the keys are added to the repository after the decision. Seeded defects come from a different agent than the one that writes reviewer prompts.
2. Configuration grid. A configuration is a model, an effort, and a tier. The first grid is openai-codex/gpt-6.1-sol with low, medium, high, and xhigh, each with the tiers priority (named <effort>-fast) and default (named <effort>): eight configurations. Each uses the production reviewer setup of plan-lifecycle 0.17.1: the current prompts, a panel round 1 for 50 or more changed lines and a full round below that, and delta rounds for repairs. Step 1 picks the dispatch mechanism that applies the effort and the tier to each reviewer (agent files with thinking-level plus a per-agent tier override that the harness writes and removes, or one omp -p process per reviewer with --model <model>:<effort> --service-tier <tier>), and every run must prove its configuration from its transcript. The harness takes any list of models, efforts, and tiers, so a later model runs the same grid.
3. Loop replay. Each run of a code case is a whole review loop that follows the production loop; the key scores the run and does not end it. Round 1 reviews the candidate. The judge matches the findings to the key and adjudicates the rest. Then the fix step repairs every finding that stands: a matched known defect gets its fix patch, and the valid extra findings of the round get one fix that one fixed fixer agent writes for all of them together, as one manager repairs a round in production (the same model, effort, and tier for every configuration, so that only the reviewer setup varies). When a fix patch no longer applies because an earlier repair changed the same lines, the harness skips it if the tree already holds it, and otherwise gives that defect to the same fixer. A finding rated wrong goes into a rejection file with the judge's evidence, as the contract's rejection comment does. The harness commits the repaired tree as the next reviewed head and picks the next round's mode by the production rules of plan-lifecycle 0.17.1 (plan-contract.md, Code-review rounds): a delta round only when the previous round completed, every change since its head repairs one of its findings, and the delta diff is at most half the full diff; otherwise a panel or full round by the size rule. The loop ends when a round returns no open finding (every previous finding fixed, no new finding that stands), or at 5 rounds. The record of each run keeps every round: its mode, its time (the slowest reviewer), its tokens, its findings, and the known defects it found; the loop record adds the rounds, the total time (review rounds only; the fixer's time is reported apart), the total tokens, the known defects found by the end, the known defects that escaped the loop, the valid extra findings, and the rounds that only extra findings caused. Plan cases run one round per run.
4. Report. For each case group (dev, held-out) the report gives one column per configuration, named as in decision 2 (low, low-fast, medium, medium-fast, high, high-fast, xhigh, xhigh-fast; xhigh-fast is the production baseline), in three tables. The decision table covers every case of the group: known defects found and escaped, and the 90% bootstrap interval of each configuration's difference from the best one in round-1 recall, in escaped defects per 50 known, and in loop time. The code-loop table covers only the runs of code cases, which run as loops: for each round index (round 1, round 2, and so on) its time (median, minimum, maximum), its mode counts, and the known defects that it found; then the loop rows: rounds (median and maximum), loop time (median), known defects found by the end, escaped defects, valid extra findings, rounds caused only by extra findings, wrong findings, unjudged and unknown answers, tokens, the loops with a fixer and their fixer time, and the known defects found per loop minute. The single-review table covers plan, plan-delta, and code-delta cases, which have one round: time, known defects found and escaped, valid extra and wrong findings, unjudged and unknown answers, and tokens. Loop time and its interval come from code loops only. Every table shows the model, effort, and served tier that the transcripts show. Time and tokens come only from the assistant messages of the configured model; side calls such as model_usage records of a helper model are reported apart. A run is invalid and leaves the tables when any of its assistant messages comes from another model than its configuration (for example after a usage fallback), when its transcript shows another effort or served tier, or when a reviewer read outside its inputs: every tool-call target must be the run's workspace files or tree copies, the reviewer's own skill folder and agent file, the plan contract of the reviewer's own plugin (the plan-reviewer checklist tells the reviewer to read it there), a skill:// URI of the reviewer's skills or the installed plugin-cache copy of such a skill, an artifact:// URI, or a web page that is not part of the benchmark repository; a Grep, Glob, or find call without a path counts as outside. The reviewer setup is newer than every case, so the optional instruction reads (the plan contract, a plugin-cache copy, and a skill:// URI) count as outside when the case changes the same file: a file in the round's diff, or a file in the Files cells of a plan case. The reviewer's own agent file and skill folder stay allowed also then, because the reviewer cannot work without them; results.md states this limit. Every reviewer prompt says that the working directory is not the snapshot, and that every file other than the reviewer's own agent file and skill folder, including the plan contract and other skills, is read under the snapshot root.
5. Decision rule, fixed before any run. Among the eight configurations, the best is the one with the fewest escaped defects at loop end on the held-out cases. A configuration is not worse than the best when the 90% interval of its escaped-defect difference stays below 2 defects per 50 and its round-1 recall interval stays above -0.05. Among the configurations that are not worse, the recommended one has the lowest median loop time; tokens are reported but do not decide. The same rule then compares, at the recommended configuration: the 0.16.0 and the 0.17.1 code prompts, the panel and one full reviewer, the lens text in the dispatch and in a reference, and the 0.16.0 and 0.17.1 plan prompts on plan cases (round-1 recall for plan cases). Dev runs may change the harness, not the decision rule.
6. Leads from Make plan and code reviews faster and need fewer rounds #98. The self-review lead runs the yolo self-review (the reviewer agent with the docks code-review and code-clarity skills) on the same round-1 candidates at the recommended configuration, as one agent and split by lens, and reports its time and the defects that only it found. The report stops on a finding with neither a match nor an adjudication, as it stops on a missing pair answer. The loop replay counts valid extra findings per round, which measures the extra rounds of the report-every-LOW stance. A second judge from another model family re-judges a random 10% of the match and adjudication answers, and the report gives the agreement rate.
7. Reusable procedure. The benchmark skill gets a section that tells how to run the grid for a new model: the commands with the model, efforts, and tiers as parameters, the proof that each run used its configuration, the report, the decision rule, and where to append the results in results.md. The recommended reviewer setting (effort and tier for the reviewer role) goes into results.md and the run summary as a user setting, with the omp config lines that set it.

Request corrections

Asked: "test all efforts for gpt 6.1 sol ... low, medium, high, xhigh", "compare with and without the --fast too". Did: the grid has the four efforts, each with the priority tier (the user's fast mode) and the default tier. Reason: the user named fast as OpenAI's priority tier; omp's --service-tier and tier.openai set it, and the transcripts record the served tier.

Asked (in #98 decision 9, rejected then): no reasoning-effort tuning. Did: the effort sweep runs, with the effort set per agent file or per run. Reason: the user asked for it, and the #98 numbers came from high while production reviews use xhigh.

Engineering standards

• Steps 1 to 3: tdd-workflow for new scorer behavior (bootstrap intervals, the configuration table, loop aggregation, the unjudged stop), test-coverage for the existing scorer paths that the change touches, and code-clarity.
• Step 4: the blind-split rule of decision 1 (the plan author does not read held-out keys).
• Step 8: write-skill rules for the skill text and code-clarity.
• Commits: commit-discipline. Self-review during code review round 1: code-review with code-clarity.

Steps

Acceptance

Do not touch

• The plugins: this plan changes the project-local benchmark and its tests; the recommended reviewer setting is a user setting.
• D2 of Fix #90 and let agents report skill problems to this repository #96: no case file describes it.
• The held-out keys before the decision.

Open questions

Verification Results

• Step 5, dev grid: 18 dev cases, 8 configurations, 3 runs each: 816 round-1 reviewer dispatches, 192 code loops run to their end (up to 5 rounds), 432 runs. The user limited the machine load during the run, so every reviewer, fixer, and judge ran with at most 16 agents at a time. A first dispatch with 32 at a time was stopped after a few minutes, and its results were discarded, so that every timed run had the same load.
• Step 5 harness repairs that the first loops showed, each committed with a unit test: one fixer per loop round for all valid extra findings (12 separate fixer patches for one function conflicted); a block returned as an object with extra fields reads as its one review field (6 of 816 blocks were unreadable before); a known defect whose fix patch no longer applies after an earlier fixer repair goes to the round's fixer (one loop stopped before). Red baselines: unwrap test 1 failed, 48 passed; fix-patch fallback test, run on the previous commit, 1 failed, 49 passed.
• Dev read scope: 8 of 432 runs were invalid for reads outside their inputs: 2 plan reviewers read the plan contract of their own plugin, 1 reviewer read the installed plugin-cache copy of the code-review skill, and 5 reviewers read the docks code-review or code-clarity skill in the repository that holds their own agent files. Decision 4 now allows the first two kinds; the prompts now point the reviewers to the snapshot copy of repository files, before the held-out run.
• Dev report: the configuration table mixed code loops with single reviews (30 of 54 runs per configuration), so the median rounds were 1 for every configuration and the median fixer time 0. Decision 4 now splits the report into the decision, code-loop, and single-review tables, before the held-out run.
• Dev read scope after the case-change rule (plan review round 8), rescored: 5 of 432 runs invalid, all reads of docks skills in the repository that holds the reviewer's own agent files.
• Step 5, held-out grid: 24 held-out cases (10 code loops, 14 plan cases), 8 configurations, 3 runs each: 1,056 round-1 dispatches, 240 code loops to their end, 576 runs, none invalid. Decision under decision 5: xhigh-fast has the fewest escaped known defects (31.92 per 50 known); no other configuration is not worse (xhigh: escaped interval [-0.41, 4.35], recall interval [-0.03, 0.02]); so xhigh-fast, the production setting, stays. Median code-loop time 262.9 s. Second judge on a random 10%: dev 402 of 418 answers agree (0.96), held-out 606 of 611 (0.99).
• Step 6, first code run discarded: 19 of 30 B0 and 67 of 90 B1 reviewers (the released 0.16.0 code-reviewer, which has no skill of its own) read the installed 0.17.1 code-reviewer skill through skill://. The harness now marks such reads outside for every arm, every prompt forbids them, and no grid reviewer (2,651 transcripts) had read them. All six code arms ran again; 1 of 180 runs is invalid.
• Step 6 harness repair: transcripts picked a dispatch's transcript by name only, and a later workspace in the same session used the same names; it now keeps only transcripts whose prompt names the workspace (unit test added).
• Step 6 results (held-out, xhigh-fast, 3 runs per case): the 0.17.1 code prompt beats the 0.16.0 one (B0 against C1: escaped interval [0.51, 4.76], recall interval [-0.10, -0.01]); the panel beats one full reviewer (C1 against C3: recall interval [-0.10, 0.02]); lens text in the dispatch is not shown not worse than lens text in a reference (B1 against C3: escaped [-5.00, 2.78], recall [-0.06, 0.10]); the 0.17.1 plan prompt is not shown not worse than the 0.16.0 one (P1 against P0: escaped [-1.19, 8.89], recall [-0.18, 0.02]), recorded as a lead because this plan does not change the plugins. The self-review found known defects that no panel run found in 2 cases (one agent) and 1 case (split by lens); the split self-review finds more than one self-reviewer (escaped [0.98, 5.26], recall [-0.11, -0.02] for one agent) at about the same round time.
• Step 7: 24 held-out key files, 35 fix patches, and the held-out inputs moved into cases/heldout/; every held-out case builds without --keys (prepare.mjs case for each, exit 0). Corpus: dev 18 cases with 50 known defects, held-out 24 cases with 59.
• Red baseline, step scorer_report: bun test ./scripts/tests/unit/review-benchmark.test.mjs: 9 failed, 24 passed.
• Red baseline, step loop_replay: bun test ./scripts/tests/unit/review-benchmark.test.mjs: 9 failed, 33 passed.
• Red baseline, step read_scope: bun test ./scripts/tests/unit/review-benchmark.test.mjs: 6 failed, 42 passed.
• Step 5 to 8 harness repairs, each test run on the commit before its fix: unwrap 1 failed, 48 passed; fix-patch fallback 1 failed, 49 passed; report split and pinned reads 6 failed, 47 passed; case-change rule 2 failed, 53 passed; self-review arms and arm report 6 failed, 55 passed; installed reviewer skills 4 failed, 58 passed; transcript lookup 1 failed, 62 passed; probe workspace 1 failed, 63 passed; fixer prompt in names 1 failed, 64 passed.
• a1 bun scripts/ci.mjs: exit 0 at fbae523d ("All ci.mjs checks passed - 2 plugin(s) + repo-wide").
• a2 bun test ./scripts/tests/unit/review-benchmark.test.mjs: 65 pass, 0 fail.
• a3 step 1 probe: eight meta.json files whose effort and served tier equal their configuration; the held-out and dev grids then proved every run from its transcript (no run with another model, effort, or tier).
• a4 every case builds: 18 dev cases and 24 held-out cases (prepare.mjs case, exit 0).
• a5 results.md holds the configuration tables of dev and held-out, the decision, the step 6 comparisons, the leads, the corpus, and the recommended reviewer setting with its omp config lines.
• a6 first fresh-agent run (max-fast): the agent produced a valid report row, but it needed two helps (the effort minimal is not supported by the model, and the probe had no full.diff) and found 8 gaps in the skill text. The gaps are fixed (efforts check, probe folder, probe full.diff, nested transcripts, fixer prompts in names, judge model, worktree folder, one benchmark per checkout).