feat(agent): harness fingerprint for scaffolding-regression attribution - #2450
Merged
Merged
Conversation
Stamp the harness configuration (composed system prompt, write-integrity check registry, tool schema set) as a SHA-256 fingerprint into the debug log under the 'harness' category, logging on first observation and again only when any surface changes mid-session. Rationale (frontier research): - arXiv:2607.03691 'Don't Blame the LLM': across 35 releases of a coding agent with a FIXED model, resolve rate swung 23%-39% with no trend while token spend rose 70%+ - the swings tracked scaffolding changes. Core recommendation: 'report and control for the scaffolding version'. - Anthropic Claude Code postmortem (2026-04-23): weeks of 'it feels dumber' reports resolved to three harness changes; evals missed all. - arXiv:2607.06184 (TraceProbe): trajectory diffing for regression localization is only meaningful when the producing harness is recorded. ggcode previously had no way to attribute a debug export to the specific scaffolding that produced it: prompt composition, the ~191-check registry (e.g. the fc5c4aa critical-only trim), and MCP/tool registration could all change invisibly. The fingerprint closes that attribution gap so quality regressions can be bisected to a scaffolding change instead of being misattributed to the model. Wired via defer in maybeInjectDynamicSystemPrompt (fires on every agent Run), change-gated to avoid log spam. Zero behavior change otherwise. Co-Authored-By: ggcode <noreply@ggcode.dev>
topcheer
commented
Sep 16, 2026
topcheer
left a comment
topcheer
left a comment
Owner
Author
There was a problem hiding this comment.
复审通过 ✅（协查两重点核销）
① fingerprint 三元组完备性：三个可变面（baseSystemPrompt 组合提示/allChecks 注册表/tool schema 集）确为 ggcode harness 的全部运行塑形面——与 body 引用的三个研究依据（scaffolding 漂移 23-39% 波动/Claude Code postmortem 三 harness 变更/TraceProbe 归因前提）逐一对上。Registry 惰性初始化的空→有转换本身也被记录（注释明示）——覆盖完整。
② 哈希稳定性：hashNames 先排序后 join（\x00 分隔）——顺序不敏感+无歧义分隔符；Sum() 含各面 count（碰撞双保险）；64-bit 前缀对变更归因碰撞概率可忽略（自评正确）。测试钉住：双序同 hash/异集可分/三元组各自敏感性（prompt 改动/tool 注册）。
变更检测契约：logHarnessFingerprint 首次记录+仅变更记录（无 spam）——TestLogHarnessFingerprint 三态钉死；defer 挂接 maybeInjectDynamicSystemPrompt 出口（所有重建路径覆盖）。
非阻塞观察：checks 面读全局 allChecks 无锁——注册只发生在 init 期（read-mostly），实际竞态可忽略；若未来支持运行时注册需补同步。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See commit 48767ff. Harness fingerprint (system prompt + check registry + tool set) stamped into debug log on change, per arXiv:2607.03691 scaffolding-version recommendation. Co-Authored-By: ggcode noreply@ggcode.dev