feat(agent): adversarial evaluator gate (generator-evaluator separation) - #2447
Merged
Merged
Conversation
…ration) Implements the GAN-inspired pattern from Anthropic's harness design guidance (2026-03): a generator that reviews its own work is reliably lenient, so completion is gated on an independent fresh-context LLM evaluator that reviews the run's diff against the original task. - new internal/agent/evaluator.go: skeptical system-prompt evaluator with explicit grading rubric (spec adherence, correctness, integration wiring, edge cases, honesty), VERDICT parsing, diff collection (git diff HEAD + untracked names), per-task round budget (max 2), diff/findings size caps, prompt-injection hardening - wired as a pre-completion gate in the run loop after the change reconciliation gate; FAIL findings are injected as a user message so the generator repairs or explicitly justifies (bounded loop) - opt-in via config verify.adversarial_review (default off); usage accounted under source "evaluator"; evaluator failures never block completion - unit tests for verdict parsing and default-off/skip behavior Co-Authored-By: ggcode <noreply@ggcode.dev>
topcheer
commented
Sep 16, 2026
topcheer
left a comment
Owner
Author
There was a problem hiding this comment.
复审通过 ✅(默认 off 的 opt-in LLM 门+fail-safe 方向全对)
- 分离实现正确:fresh-context(evaluator 只收 task+diff,无主循环历史——不被 generator 叙事锚定,GAN 分离的主杠杆);同 provider 复用是可接受简化
- fail-safe 全对:evaluator 失败/超时/畸形 verdict 一律按 PASS 放行("unavailability must never block completion");仅明确 FAIL+非空 findings 回注——LLM 门挂了不能卡死完成路径,方向正确
- 注入防御在位:system prompt 明示 diff/task 内容一律 UNTRUSTED——评审对象是任意代码,这是必要防线
- 资源界完备:2 轮/任务+task prompt 变更重置预算、120s 超时、24KB diff/6KB findings 截断、usage 归集
- 挂接点:deterministic gates 之后、diff-summary gate 之前,FAIL→user message 回注修(与既有 gate 同模式)
- 3 测试(verdict 解析表/默认关/无变更跳过)+ config opt-in 链完整
非阻塞观察:collectAdversarialDiff 注释称 "committed + staged + unstaged vs HEAD" 与 git diff HEAD 实际语义矛盾(恰不含已 commit 内容——run 中 commit 过则 diff 为空)。对主 checkout 不 commit 的工作流盲区可接受,但注释应改为 "working tree vs HEAD"(免误导后来者)。untracked 文件名面已补偿。
CI 9/9 绿。可合并(排程 5/6,剩 #2448)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Research-backed capability (cron-runner research round sa-9, concept: Anthropic "Harness design for long-running apps" 2026-03, GAN-inspired generator-evaluator separation; no issue filed).
Gap
The pre-completion gate chain (build/test/lint oracles, spec-gaming, companion, complexity, cross-file, reconcile) is entirely zero-LLM-cost heuristics or command checks - no semantic-level independent evaluation exists. Self-assessment is still done by the generator itself.
Implementation (5 files, +363)
Tests
evaluator_test.go: 5 verdict-parsing cases + default-off + no-changes skip. Full agent/agentruntime/config package regressions green. gofmt clean, go build -tags goolm clean.
Co-Authored-By: ggcode noreply@ggcode.dev