feat(mcp): per-server circuit breaker for MCP tool calls (sa-21) - #2457
Conversation
An MCP server that crashed or became unreachable fails EVERY call to EVERY tool it exposes. Each such call burned the full transport pipeline (120s stdio timeout) and the model typically retried the same or sibling tools several times, costing minutes of wall-clock time and tokens per outage. Trajectory-level guards (recurring_error.go, error_compound.go) only inject guidance text; nothing at the execution layer short-circuits the wasted retries. Add a per-SERVER circuit breaker shared by every mcpTool of one server (shared across Registry.Clone copies via pointer, so swarm teammates share outage state): - CLOSED -> (3 consecutive infrastructure failures) -> OPEN -> (60s cooldown) -> HALF-OPEN single probe; success closes, infra failure reopens with a fresh cooldown. - Only transport-level failures trip it (connection refused/reset, EOF, broken pipe, DNS, i/o/deadline timeouts, TLS, 5xx). Semantic failures (server answered) and plain user Esc never do. - When OPEN, tool calls fast-fail instantly with an actionable message (what happened, why retrying wastes a full LLM round-trip, and the re-plan path), following the permissionDeniedMessage convention. - Zero LLM cost: deterministic O(1) counters; breaker disabled (nil) in hand-built fixtures, so existing behavior is unchanged. Co-Authored-By: ggcode <noreply@ggcode.dev>
topcheer
left a comment
There was a problem hiding this comment.
复审通过 ✅(协查三重点核销,664 行全量走读)
① 参数与误熔断匹配:infra-only 分类是关键防线——语义错误(server answered:JSON-RPC error/isError=true)永远 recordSuccess 不熔断(慢但活的服务器零误伤);清单覆盖 transport 全谱(refused/reset/closed/EOF/DNS/i-o timeout/deadline/TLS/5xx/timed out),方向保守(不匹配即语义失败不触发);context canceled 不匹配任何模式(Esc 非停电)。deadline exceeded 含 server hang 是有意的(hang=不可用)。3 连败阈值对瞬时抖动(单帧丢失)足够缓冲。
② 半开探测并发安全:probing 标志在 gate 的锁内检查+设置——并发调用恰一个成为探测,其余 fast-fail 且消息明示"a probe call is already in flight; await its result";probe 失败 reopens with fresh cooldown;recordFailure 的 OPEN 分支防 race 保持最早 openedAt。全 mu 守卫。
③ tool_search 交互——非问题且语义正确:gate 挡在 Execute(运行时),schema 披露在 registry 初始化层——被熔断 server 的 schema 仍正常披露/占激活位(静态目录),调用时 fast-fail 带恢复指引(换工具/报告用户)。两层职责分离正确:熔断是动态可用性状态,不该回滚静态目录。
fail-fast 消息含 why+actionable path(勿重试/换工具/报告用户)——符合 graceful degrade 文献惯例。never-connected(caller nil)喂入正确(真不可达)。10 测试。
非阻塞观察:文本 Contains 分类与 transport 错误格式耦合(同 #2449 marker 观察)——清单保守方向使漏判安全(不熔断),误判面已被清单的 transport 专属性限定。
CI 9/9 绿。可合并。
Problem
A crashed/unreachable MCP server fails every call to every tool it exposes. Each call burned the full transport pipeline (120s stdio timeout), and the model typically retried the same or sibling tools several times - minutes of wall-clock time and tokens per outage. Trajectory-level guards (recurring_error.go, error_compound.go) only inject guidance text; nothing at the execution layer short-circuits the wasted retries.
Solution
Per-server circuit breaker (sa-21) shared by every
mcpToolof one server:isError=trueresults) and plain user Esc never do.permissionDeniedMessageconvention.Registry.Clone()copies via pointer, so swarm teammates share outage state (correct: a dead server is dead for every agent).nilin hand-built fixtures, so existing behavior is unchanged.Verification
go test -tags goolm -race ./internal/mcp/- all tests pass, including 10 new breaker tests (state machine, probe reopen with fresh cooldown, sibling-tool tripwire, semantic-error non-tripping, 50-goroutine concurrent gate under-race).go build -tags goolm ./...- OK.