Skip to content

fix(tool): deterministic tool list ordering to preserve prompt/KV cache - #2445

Merged
topcheer merged 1 commit into
mainfrom
fix-tool-order-cache
Sep 16, 2026
Merged

topcheer merged 1 commit into
mainfrom
fix-tool-order-cache

Conversation

@topcheer

Copy link
Copy Markdown
Owner

研究出处

  • Manus, "Context Engineering for AI Agents: Lessons from Building Manus" (2025): KV-cache hit rate 是 agent prompt prefix 最具成本影响性的属性;平均 input:output ≈ 100:1,相同前缀可命中 provider 侧 KV/prompt cache,大幅降低 TTFT 与输入 token 成本。前缀中任何字节级变化都会击穿缓存。
  • Anthropic / OpenAI prompt caching 文档:前缀匹配按字节进行,tool definitions 位于 prompt 前缀最前端(Anthropic: tools → system → messages)。

问题

Registry.List() 直接迭代 r.tools(Go map),map 迭代顺序随机 → 每次 run 发给 provider 的工具列表顺序都不同。由于工具定义位于 prompt 前缀最前端,这导致:

  1. Anthropic: 任意两个 run 之间前缀在 tools 段就不一致,buildParams 中已实现的 cache_control 断点(含 arXiv:2601.06007 的静态/动态分层设计)完全无法命中。
  2. OpenAI 兼容 / DeepSeek / Gemini 等自动前缀缓存供应商: 同样按 run 随机失效。
  3. ToolNames() 顺序也随之抖动,影响依赖稳定展示的下游。

修复

List() 输出按工具名排序,使 ToDefinitions() / ToolNames() 确定化,一次排序恢复跨 run 前缀稳定性。

测试

  • 新增 TestToolRegistry_ListDeterministicOrder:连续 20 次 ToDefinitions() 验证顺序稳定且按名排序。
  • go build -tags goolm ./... 通过,go vet/gofmt 干净,pre-commit 钩子(fmt+vet+build)通过。

Co-Authored-By: ggcode noreply@ggcode.dev

Registry.List() iterated r.tools, a Go map, so the order of tools sent
to the provider differed on every run. Tool definitions sit at the very
front of the prompt prefix (tools -> system -> messages on Anthropic;
same conceptual position for OpenAI-compatible automatic prefix
caching), so a randomized tool order produced a different byte prefix
on every run and invalidated the provider-side prompt/KV cache before
any cache_control breakpoint (anthropic.go) could take effect.

Sort List() output by tool name. This makes ToDefinitions() and
ToolNames() deterministic, restoring cross-run prefix stability -
reducing TTFT and input-token cost per Manus, "Context Engineering for
AI Agents" (2025): KV-cache hit rate is the most cost-impactful
property of an agent prompt prefix.

Adds TestToolRegistry_ListDeterministicOrder verifying stable, sorted
ordering across repeated ToDefinitions() calls.

Co-Authored-By: ggcode <noreply@ggcode.dev>

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 ✅

  • 根因真实r.tools 是 map,Go 迭代随机化——工具定义在 prompt 前缀最前部,字节序不稳直接击穿 provider KV cache(注释引用的 Manus 成本论成立)
  • 锁语义正确:sort.Slice 在 RLock+defer 作用域内(diff 上下文核实),且排序对象是本地拷贝切片——无 race 无死锁
  • 下游零顺序依赖:ToolNames/List 消费者抽查(mcp_loader 迁移对比/info 展示)均为集合语义,无序敏感调用方
  • 测试质量:20 轮迭代序稳定 + 严格递增有序断言(双保险)+ count 漂移检测
  • 顺手 em-dash 修 ASCII 无害

CI 9/9 绿。可合并(按协查建议的顺序 #2443#2445→… 等前序处理)。

@topcheer
topcheer merged commit 928f05d into main Sep 16, 2026
9 checks passed
@topcheer
topcheer deleted the fix-tool-order-cache branch September 16, 2026 18:42
topcheer pushed a commit that referenced this pull request Sep 16, 2026
Implements the 2026 baseline practice of Temporal Context Injection:
- Session-anchored current date/time header prepended to the cacheable
  base system prompt (byte-stable per run to preserve KV-cache reuse,
  consistent with #2445 prefix-stability findings)
- New current_time built-in tool with optional IANA timezone param for
  live wall-clock reads in long sessions (crossing midnight, DST, etc.)
- Header hints the model to call current_time when freshness matters

Co-Authored-By: ggcode <noreply@ggcode.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant