Skip to content

feat(agent): adversarial evaluator gate (generator-evaluator separation) - #2447

Merged
topcheer merged 1 commit into
mainfrom
adversarial-evaluator
Sep 16, 2026
Merged

topcheer merged 1 commit into
mainfrom
adversarial-evaluator

Conversation

@topcheer

Copy link
Copy Markdown
Owner

Research-backed capability (cron-runner research round sa-9, concept: Anthropic "Harness design for long-running apps" 2026-03, GAN-inspired generator-evaluator separation; no issue filed).

Gap

The pre-completion gate chain (build/test/lint oracles, spec-gaming, companion, complexity, cross-file, reconcile) is entirely zero-LLM-cost heuristics or command checks - no semantic-level independent evaluation exists. Self-assessment is still done by the generator itself.

Implementation (5 files, +363)

  • internal/agent/evaluator.go: checkAdversarialReviewGate - run diff to a FRESH LLM call with no main-loop history (context isolation). Criteria: spec adherence, correctness, integration, edge cases, honesty. Malformed verdict tolerated as PASS - evaluator failure never blocks completion.
  • Loop closure: FAIL findings injected as user message; generator must fix or argue - evaluator->generator repair loop.
  • Guardrails: max 2 rounds/task, 24KB diff / 6KB findings truncation, 120s timeout, skip in plan mode / no changes / empty provider; diff marked untrusted data; usage attributed to evaluator source.
  • Wiring: agent.go gate chain (after reconcile, before diff-summary); config verify.adversarial_review (default OFF) via agentruntime/verify_config.go.

Tests

evaluator_test.go: 5 verdict-parsing cases + default-off + no-changes skip. Full agent/agentruntime/config package regressions green. gofmt clean, go build -tags goolm clean.

Co-Authored-By: ggcode noreply@ggcode.dev

…ration)

Implements the GAN-inspired pattern from Anthropic's harness design
guidance (2026-03): a generator that reviews its own work is reliably
lenient, so completion is gated on an independent fresh-context LLM
evaluator that reviews the run's diff against the original task.

- new internal/agent/evaluator.go: skeptical system-prompt evaluator
  with explicit grading rubric (spec adherence, correctness,
  integration wiring, edge cases, honesty), VERDICT parsing, diff
  collection (git diff HEAD + untracked names), per-task round budget
  (max 2), diff/findings size caps, prompt-injection hardening
- wired as a pre-completion gate in the run loop after the change
  reconciliation gate; FAIL findings are injected as a user message so
  the generator repairs or explicitly justifies (bounded loop)
- opt-in via config verify.adversarial_review (default off); usage
  accounted under source "evaluator"; evaluator failures never block
  completion
- unit tests for verdict parsing and default-off/skip behavior

Co-Authored-By: ggcode <noreply@ggcode.dev>

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 ✅(默认 off 的 opt-in LLM 门+fail-safe 方向全对)

  • 分离实现正确:fresh-context(evaluator 只收 task+diff,无主循环历史——不被 generator 叙事锚定,GAN 分离的主杠杆);同 provider 复用是可接受简化
  • fail-safe 全对:evaluator 失败/超时/畸形 verdict 一律按 PASS 放行("unavailability must never block completion");仅明确 FAIL+非空 findings 回注——LLM 门挂了不能卡死完成路径,方向正确
  • 注入防御在位:system prompt 明示 diff/task 内容一律 UNTRUSTED——评审对象是任意代码,这是必要防线
  • 资源界完备:2 轮/任务+task prompt 变更重置预算、120s 超时、24KB diff/6KB findings 截断、usage 归集
  • 挂接点:deterministic gates 之后、diff-summary gate 之前,FAIL→user message 回注修(与既有 gate 同模式)
  • 3 测试(verdict 解析表/默认关/无变更跳过)+ config opt-in 链完整

非阻塞观察collectAdversarialDiff 注释称 "committed + staged + unstaged vs HEAD" 与 git diff HEAD 实际语义矛盾(恰不含已 commit 内容——run 中 commit 过则 diff 为空)。对主 checkout 不 commit 的工作流盲区可接受,但注释应改为 "working tree vs HEAD"(免误导后来者)。untracked 文件名面已补偿。

CI 9/9 绿。可合并(排程 5/6,剩 #2448)。

@topcheer
topcheer merged commit 213c90f into main Sep 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant