Skip to content

feat(mcp): per-server circuit breaker for MCP tool calls (sa-21) - #2457

Merged
topcheer merged 1 commit into
mainfrom
sa-21-tool-circuit-breaker
Sep 16, 2026
Merged

topcheer merged 1 commit into
mainfrom
sa-21-tool-circuit-breaker

Conversation

@topcheer

Copy link
Copy Markdown
Owner

Problem

A crashed/unreachable MCP server fails every call to every tool it exposes. Each call burned the full transport pipeline (120s stdio timeout), and the model typically retried the same or sibling tools several times - minutes of wall-clock time and tokens per outage. Trajectory-level guards (recurring_error.go, error_compound.go) only inject guidance text; nothing at the execution layer short-circuits the wasted retries.

Solution

Per-server circuit breaker (sa-21) shared by every mcpTool of one server:

  • State machine: CLOSED → (3 consecutive infra failures) → OPEN → (60s cooldown) → HALF-OPEN single probe; success closes, infra failure reopens with a fresh cooldown.
  • Conservative classification: only transport-level failures trip it (connection refused/reset, EOF, broken pipe, DNS, i/o/deadline timeouts, TLS, 5xx). Semantic failures (server answered, incl. isError=true results) and plain user Esc never do.
  • OPEN gate fast-fails instantly with an actionable message (what happened, why retrying wastes a full LLM round-trip, re-plan path), following the permissionDeniedMessage convention.
  • Shared across Registry.Clone() copies via pointer, so swarm teammates share outage state (correct: a dead server is dead for every agent).
  • Zero LLM cost: deterministic O(1) counters; breaker is nil in hand-built fixtures, so existing behavior is unchanged.

Verification

  • go test -tags goolm -race ./internal/mcp/ - all tests pass, including 10 new breaker tests (state machine, probe reopen with fresh cooldown, sibling-tool tripwire, semantic-error non-tripping, 50-goroutine concurrent gate under -race).
  • go build -tags goolm ./... - OK.

An MCP server that crashed or became unreachable fails EVERY call to
EVERY tool it exposes. Each such call burned the full transport pipeline
(120s stdio timeout) and the model typically retried the same or sibling
tools several times, costing minutes of wall-clock time and tokens per
outage. Trajectory-level guards (recurring_error.go, error_compound.go)
only inject guidance text; nothing at the execution layer short-circuits
the wasted retries.

Add a per-SERVER circuit breaker shared by every mcpTool of one server
(shared across Registry.Clone copies via pointer, so swarm teammates
share outage state):

- CLOSED -> (3 consecutive infrastructure failures) -> OPEN ->
  (60s cooldown) -> HALF-OPEN single probe; success closes, infra
  failure reopens with a fresh cooldown.
- Only transport-level failures trip it (connection refused/reset,
  EOF, broken pipe, DNS, i/o/deadline timeouts, TLS, 5xx). Semantic
  failures (server answered) and plain user Esc never do.
- When OPEN, tool calls fast-fail instantly with an actionable message
  (what happened, why retrying wastes a full LLM round-trip, and the
  re-plan path), following the permissionDeniedMessage convention.
- Zero LLM cost: deterministic O(1) counters; breaker disabled (nil)
  in hand-built fixtures, so existing behavior is unchanged.

Co-Authored-By: ggcode <noreply@ggcode.dev>

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 ✅(协查三重点核销,664 行全量走读)

① 参数与误熔断匹配:infra-only 分类是关键防线——语义错误(server answered:JSON-RPC error/isError=true)永远 recordSuccess 不熔断(慢但活的服务器零误伤);清单覆盖 transport 全谱(refused/reset/closed/EOF/DNS/i-o timeout/deadline/TLS/5xx/timed out),方向保守(不匹配即语义失败不触发);context canceled 不匹配任何模式(Esc 非停电)。deadline exceeded 含 server hang 是有意的(hang=不可用)。3 连败阈值对瞬时抖动(单帧丢失)足够缓冲。

② 半开探测并发安全:probing 标志在 gate 的锁内检查+设置——并发调用恰一个成为探测,其余 fast-fail 且消息明示"a probe call is already in flight; await its result";probe 失败 reopens with fresh cooldown;recordFailure 的 OPEN 分支防 race 保持最早 openedAt。全 mu 守卫。

③ tool_search 交互——非问题且语义正确:gate 挡在 Execute(运行时),schema 披露在 registry 初始化层——被熔断 server 的 schema 仍正常披露/占激活位(静态目录),调用时 fast-fail 带恢复指引(换工具/报告用户)。两层职责分离正确:熔断是动态可用性状态,不该回滚静态目录。

fail-fast 消息含 why+actionable path(勿重试/换工具/报告用户)——符合 graceful degrade 文献惯例。never-connected(caller nil)喂入正确(真不可达)。10 测试。

非阻塞观察:文本 Contains 分类与 transport 错误格式耦合(同 #2449 marker 观察)——清单保守方向使漏判安全(不熔断),误判面已被清单的 transport 专属性限定。

CI 9/9 绿。可合并。

@topcheer
topcheer merged commit bcf092a into main Sep 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant