Skip to content

agent: non-atomic outcome semantics for mutating tool calls - #2458

Merged
topcheer merged 2 commits into
mainfrom
nonatomic-outcome
Sep 16, 2026
Merged

topcheer merged 2 commits into
mainfrom
nonatomic-outcome

Conversation

@topcheer

Copy link
Copy Markdown
Owner

问题

当一个变更型工具调用(run_commandgit_commitwrite_filegit_stash 等)歧义性失败——超时、执行中被取消、panic——副作用可能已部分或全部生效,但 harness 只返回一个裸错误(tool error: context deadline exceeded)。模型无从知晓这一点,会假设调用未发生并原样重发,导致重复提交、重复编辑、半途生效的批量操作

研究依据

  • "Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures"(arXiv:2608.02645):对生产级 agent 框架的调查发现 tool-call 副作用问题是普遍缺陷;论文建议由框架而非模型承载执行语义——对结果分类并引导“先验证再重试”。
  • 与 SAGE(参数域澄清)互补:SAGE 消除调用前歧义,本 PR 消除调用后结果歧义。

实现(工具边界的执行语义,与 transient_retry 同层;非 detector)

  1. 结果分类 classifyMutatingFailure
    • pre-exec(校验/权限/用户拒绝 → 肯定未生效):追加 [no-side-effect],提示安全修参重试,省去一轮验证;
    • ambiguous(超时/取消/panic → 未知):追加 [non-atomic],指示先 git log/status、读文件验证实际状态,再按缺失部分补偿,不要盲目重发
    • shell 工具永不判为 pre-exec(cmd1 && cmd2 在 cmd2 失败时 cmd1 已生效);只读工具完全排除。
  2. 歧义尝试台账 mutatingLedger(每 run 重置、FIFO 有界 128 条,键为 (tool, fnv32a(args))):模型重发完全相同的变更调用时,无论成功/再失败都追加双重生效风险警告。
  3. 接线executeToolCallexecuteWithTransientRetry 前后各一次查找/标注(变更型工具本就不参与自动重试,单次执行语义成立);ledger 随 run 重置;[non-atomic] 注册为 critical guidance tag,不受每轮预算抑制。

测试

  • 新增 mutate_outcome_test.go:分类矩阵(超时/取消/panic/pre-exec/shell 退出码不误判/只读工具不标注)、ledger 记录/重置/驱逐、四类标注路径、critical tag 注册断言;
  • -race 通过;go test -tags goolm ./internal/agent/ 全量通过;go vet/go build(pre-commit)通过。

Junjun Zhang and others added 2 commits September 17, 2026 07:07
When a mutating tool call (run_command, git_commit, write_file, ...) fails
ambiguously -- timeout, cancellation mid-execution, or panic -- the side
effect may be partially or fully applied, but the harness returned a bare
error with no semantics. The model assumes nothing happened and re-issues
the identical call, risking duplicate commits and doubled edits.

Research basis: "Verified Tool Calls Improve LLM Agent Reliability Under
Non-Atomic Failures" (arXiv:2608.02645) recommends the framework carry
execution semantics: classify outcomes and guide verification-before-retry.

Implementation (execution semantics at the tool boundary, same layer as
transient_retry; not a detector):
- classifyMutatingFailure: pre-exec rejection (validation/permission/
  user-declined -> [no-side-effect] safe to retry) vs ambiguous
  mid-execution failure ([non-atomic] verify state, do not blind re-issue).
  Shell tools are never classified pre-exec: cmd1 && cmd2 failing at cmd2
  still applied cmd1. Read-only tools are excluded entirely.
- mutatingLedger: per-run FIFO-bounded map of ambiguous attempts keyed by
  (tool, fnv32a(args)). A re-issue of the identical call is annotated with
  double-apply risk whether it succeeds or fails again.
- Wired into executeToolCall around executeWithTransientRetry; ledger
  resets per run; [non-atomic] registered as a critical guidance tag so
  the note cannot be suppressed by the per-turn budget.

Co-Authored-By: ggcode <noreply@ggcode.dev>
Agent literals in tests (&Agent{}) leave mutateLedger nil; the
executeToolCall wiring probes it on every mutating call. Make
lookupAmbiguous/recordAmbiguous/reset nil-safe (lookup returns 0,
record/reset no-op) so future tests constructing bare Agents cannot
panic, and add TestMutatingLedgerNilSafe covering the full nil-Agent
annotate path.

Co-Authored-By: ggcode <noreply@ggcode.dev>

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 ✅(协查两重点核销)

① 非原子判定白名单准确性:hasNonAtomicSemantics 组合三个既有维护的集合(shellSideEffectTools+gitStateTools+mutatesSourceTree)而非新造平行清单——漂移风险最小化设计;漏判面严格等于 mutatesSourceTree 的既有覆盖面(与 #2452 守卫同源,#1475 家族历史已多轮加固)。表测覆盖分类边界。

② 与 #2452 部分并行交互:mutating 工具本就 batch-wide skip 回落串行(#2452 守卫),注解发生在串行路径的 tool result 上——preExec 路径无关性有专测(TestAnnotateMutatingOutcomePreExec);双重应用检测(retry 成功后回访 ledger)有 DoubleApplyOnRetrySuccess 钉住。

语义设计认可:框架承载执行语义(arXiv:2608.02645 方向)——歧义失败(timeout/cancel/panic)注解"副作用可能已生效,先验证再重试";ledger 有界淘汰(Eviction 测试);成功清账(SuccessClean)。与 effect ledger(#2449)互补:那个管"重试时提示先前尝试",这个管"失败结果本身标注非原子性"——正交域。

8 测试全语义面。CI 绿。可合并。

@topcheer
topcheer merged commit 66ad889 into main Sep 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant