[Investigation] Agent-led workflow gaps: closure, decision visibility, review provenance and focus #273

Inspected main 436c950c on 2026-10-30 through the GitHub API: 88 Issues, 182 PRs and 942 conversation comments. Record and source inspection only; no code, model or device was run.

Revised 2026-10-31 after the proposal review and the owner's decisions: F1 covers all eight closures and their common cause, F4 updates #248, F5 adds byte sizes measured at main 68108700, and the remedies carry their decisions. Other measurements are unchanged.

Current state: decisions are recorded for R1–R5; R1 software is delivered in PR #296 after current-head review and passing CI. R6 remains open for discussion. Decision needed (Dong1017): R6 byte budgets and the transition for raw evidence in #113 / #51. Next action: record the remaining R6 disposition; R1 does not complete the other remedies.

Goal and acceptance

• Problem / desired outcome: InferRSI is developed agent-first. Agents implement, review and report; humans take part through Issues to scope work, authorize resources and accept results. In this mode, human attention is the scarcest resource and the Issue is the human's control surface. This investigation found five places where agent throughput outruns that control surface. Each has a small remedy (R1–R6 below).

  	Gap	Key evidence	Remedy
  F1	Closure bypasses the acceptance owner	8 closures of 7 human-owned Issues on PR merge, all caused by negated sentences such as "this PR does not close Acceptance: MiniCPM-o 4.5 single-L20X vLLM-Omni throughput optimization #49"	R1, R2
  F2	Pending human decisions are hard to find	5.2% of Issue comments come from an account other than the Issue author; the documented decision labels do not exist	R3
  F3	Review provenance is self-reported	30% of merged PRs have a review from another account	R4
  F4	Throughput flows to locally closable work	No acceptance Issue records a full-gate PASS	R5
  F5	Raw evidence enters Git without a size guard	evidence/ grew from 0.69 MB to 24.2 MB in four days; Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 would add 49.1 MB	R6

• Completion criteria: the acceptance owner records accept, defer or reject for each of R1–R6. Each accepted remedy is delivered by its own scoped task or PR, linked here.

Scope and constraints

• In scope: the workflow between agents and the Issue-level human: closure, decision visibility, review records, work selection and evidence storage.
• Out of scope: engine performance claims, skill content quality, changes to the mission, and rewriting the history of closed Issues.
• Relevant dependencies, shared interfaces or limits: docs/agent-collaboration.md, .github/workflows/ci-draft.yml, .github/workflows/ci-ready.yml, task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248, [Tracking] Minimal engineering benchmarks shared by Workbench, SkillEval and SearchSpace experiments #252, task: agent-maintained issue system: one label scheme, auto-labeling and a pending-decision view #284.

Ownership and authorization

• Proposed by / actual requester: Dong1017. Drafted with Claude Code (claude-opus-5-5) at the requester's direction.
• Status: Accepted for R1 and R5; R2 and R4 deferred; R3 moved to task: agent-maintained issue system: one label scheme, auto-labeling and a pending-decision view #284; R6 under discussion
• Accountable parent (Issue-level owner, if assigned): TBD
• Slice implementer(s) / acting contributor (if assigned): TBD
• Goal-acceptance owner: Dong1017
• Accepted decision or scoped delegation: owner's decisions, 2026-10-31

Inputs and deliverables

• Relevant sources, revisions or linked issues: main 436c950c; GitHub API snapshots of 2026-10-30 and 2026-10-31; the Issues and PRs linked below.
• Expected artifacts: one decision per remedy, recorded here, with links to any follow-up tasks.

Findings

F1. Closure bypasses the acceptance owner

Working rule 6 says: "Use non-closing Refs #N until closure is authorized." To follow it, agents wrote negated sentences in PR bodies. GitHub matches closing keywords regardless of negation, so each of these merges closed a human-owned Issue:

No PR body has used a standalone Closes #N. The two intended closures, #183 and #185, used the Development sidebar link. Rule 6 relies on writers knowing GitHub's matching rules, and no check enforces it.

Separately, #10 was closed as completed while its body records "Initial verdict: INCONCLUSIVE" and it has no closure comment.

F2. Pending human decisions are hard to find

• Issue threads mostly serve as agent work logs. Of 593 Issue comments, 31 (5.2%) come from an account other than the Issue author; with shared accounts, this does not measure human participation. acceptance: Qwen3.8 FP8 vs vLLM 0.32.0 — ≥25% TTFT or TPOT gain, ≤5% regression #67 has 330 comments from one account; acceptance: HunyuanImage-3.0 DiT 1024×1024 single-image latency optimization / local L20X #125 has 48.
• A human cannot list what awaits a decision. The label convention names proposed, deferred and blocked, but none of these labels exists. accepted exists and is used once.
• Decisions waiting on 2026-10-30 included task: pre-registered campaign on experience and context compaction for agents on the #75 task (Qwen3-TTS 0.6B, RTX 4090), with exploratory results #220 (asks maintainers to decide) and RFCs [RFC] Iteration Campaign Plan: one table for model, hardware, goal, search plan and test plan #30, [RFC] Hardware-aware optimization goals throughout the RSI workflow #143, [RFC] Agent plugins: a consistent InferRSI workflow in Codex, DSH and openJiuwen #152 and [RFC] Online quantization knowledge, skills and Skill Indexer selection #254 (all Status: Proposed).

F3. Review provenance is self-reported

• Of 166 merged PRs, 50 (30%) carry a review from an account other than the PR author. 75 have reviews only from the author's own account, which in this workflow are agent sessions; 41 have no review.
• The repository already states the limit. The Agent-assisted-by trailer is "self-reported provenance, not cryptographic signing, independent review or approval" (line 180), and a #61 comment notes that a shared account cannot prove which agent instance acted.
• Fast merges suit agent work. The gap is that a later reader cannot check which reviewer saw which head with which inputs.

F4. Agent throughput flows to locally closable work

• Median time from PR creation to merge: 0.3 h and 1.3 h for the two most active maintainer accounts (148 PRs); 13.6 h for other contributors (18 PRs).
• Five campaign or knowledge PRs opened between 10-21 and 10-23 remain open: Record MiniCPM-o throughput search, quality gates and peer baselines #51, Add Qwen3.8 FP8 correctness and service qualification workflows #55, docs(casebook): three cases from a Qwen3.8 dedicated-engine campaign #90, Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 and knowledge: Drama-3 decode graphs on Ascend 910B1 #132. Each received maintainer review findings between 10-23 and 10-25 and has no author commit since; Record MiniCPM-o throughput search, quality gates and peer baselines #51, Add Qwen3.8 FP8 correctness and service qualification workflows #55 and Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 also have merge conflicts. The wait sits with resolving those findings, not with review attention.
• No acceptance Issue records a full-gate PASS: Acceptance: Qwen-Image-2.1-Turbo / B300×4 campaign #3 (no verdict recorded), Acceptance: Drama 3 / B300×1 / 1.5× throughput at every concurrency #10 and Acceptance: Qwen3-Omni full streaming speech on two L20X GPUs #112 (INCONCLUSIVE), acceptance: Qwen3.8 FP8 vs vLLM 0.32.0 — ≥25% TTFT or TPOT gain, ≤5% regression #67 and acceptance: Qwen3-TTS 0.6B C1 RTF optimization on RTX 4090 #75 (FAIL), Acceptance: MiniCPM-o 4.5 single-L20X vLLM-Omni throughput optimization #49 (target unmet per the README), acceptance: HunyuanImage-3.0 DiT 1024×1024 single-image latency optimization / local L20X #125 (formal gate not run), acceptance: Qwen3.8-27B single-H20 BF16 text correctness — phase 1 #54 and acceptance: Qwen3.8-27B-FP8 single-H20 text engine and HTTP service — phase 1 #57 (superseded). The open acceptance Issues were last updated between 10-21 and 10-27.
• task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248 states that "InferRSI has not measured its own value on the work it exists for". It has no assignee, but since 10-30 the owner side runs a bounded pilot on a CPU KV-reclamation task that compares a workbench-only arm (A2) with workbench plus fixed knowledge (A3); feat: qualify a bounded synthetic CPU KV reclamation task #274, feat: add bounded KV editing and scripted actor qualification #275, feat: add inactive bounded KV pilot launcher #276 and Activate one owner-frozen CPU KV editing pair #280 are merged. Skill Eval treats any workspace change as inputs_changed (per task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248), and RunSpec has no field recording which knowledge an experiment used, so campaign gains cannot yet be attributed to knowledge.
• Inference, not investigated per PR: CPU-verifiable infrastructure work can be completed and closed inside an existing delegation, while campaigns also need accelerator time, paid calls and human authorization.

F5. Raw evidence enters Git without a size guard

• At main 68108700 (2026-10-31), tracked files total 38.5 MB, of which evidence/ is 24.2 MB. Before 10-28, evidence/ was 0.69 MB. evidence: freeze approved 8K search and retain FP8 optimization results #97 added 5.0 MB, and the six [Tracking] Skill Indexer: retrieval efficiency and accuracy #255 PRs added 16.6 MB on 10-30.
• Deleting after merge does not help. perf: reduce knowledge retrieval manifest work with measured evidence #193 added 4.55 MB and chore: remove retrieval audit and evidence artifacts #195 deleted it 16 minutes later, but the bytes remain in history and every clone downloads them.
• Two open draft PRs would add more: Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 adds 49.1 MB in 579 files, more than all tracked files on main, and Record MiniCPM-o throughput search, quality gates and peer baselines #51 adds 13.3 MB in 300 files. The largest file in Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 is 0.72 MB, so a per-file limit alone would not stop it.

Proposed remedies and decisions

The owner's decision record gives the reasons and follow-ups.

Completion and closure

• Who accepts completion and may close: the goal-acceptance owner, after recording a decision for each remedy.
• Remaining uncertainty / next decision: R6 byte budgets, and how Record Qwen3-Omni two-GPU campaign and source-grounded tuning knowledge #113 and Record MiniCPM-o throughput search, quality gates and peer baselines #51 land their raw evidence. The causal reading in F4 is an inference from timing data; individual PRs were not investigated beyond their review state.