Skip to content

feat(provider): per-model/per-endpoint LLM call policy (request_timeout, max_retries) - #2528

Merged
topcheer merged 3 commits into
mainfrom
sa78-model-call-policy
Sep 18, 2026
Merged

topcheer merged 3 commits into
mainfrom
sa78-model-call-policy

Conversation

@topcheer

Copy link
Copy Markdown
Owner

Per-Model / Per-Endpoint LLM Call Policy(request_timeout / max_retries)

研究依据(在线)

  • LiteLLM per-model timeout/stream_timeout/num_retries 是行业惯例(provider 级一刀切超时无法适配长推理 vs 快速模型差异)
  • Azure OpenAI / 各网关 per-deployment 重试配置惯例

Gap

ggcode 完全缺失 per-model/per-endpoint 超时与重试策略——全 provider 共享硬编码 providerRetryAttempts(20 次),无请求级 deadline。长推理模型(GPT-5.x Pro 等)与快速模型的超时需求无法分别配置。

实施(3 commits:938fdca9 → 76aa804c5cc907

配置层(config.go / config_vendor.go / model_policy.go 新增)

  • ModelLimitConfig 与 EndpointConfig 新增 request_timeout / max_retries;解析优先级 per-model 覆盖 > endpoint 级 > 零值默认
  • callPolicy 类型(attempts()/withTimeout())

接线层(76aa8041——初版反馈循环发现的关键 gap:策略定义了但无 provider 消费)

  • OpenAI/Anthropic/Gemini 三 provider 增字段+setCallPolicy;CloneWithModel 保留策略(子代理换模型不丢配置)
  • 23 处硬编码 providerRetryAttempts → p.policy.attempts()(openai 11 / anthropic 7 / gemini 5)
  • 超时应用在 Chat 入口与每个 streamRead goroutine 内部(覆盖完整流生命周期,不提前取消)
  • NewProvider 经 callPolicySetter 统一注入;未配置零值回退原行为(无 deadline、20 次重试)——完全 opt-in

测试(c5cc907c)

遗留

openai-responses 与 copilot 协议未接入(registry 无 setter 的协议保持默认行为,代码注释标明)——可按同一模式低成本扩展。

Co-Authored-By: ggcode noreply@ggcode.dev

@topcheer topcheer left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过 ✅(五重点核销)

① 23 处替换零值等价:attempts()=maxRetries>0?override:providerRetryAttempts——零配置 identity 回退(20 不变);diff 抽查 anthropic 7 处(含 stream 循环+attempt<attempts()-1 边界+Retry N/M 显示)形态一致,无 off-by-one(边界含-1 与常量版逐位同构)。

② 超时取消传播:ChatStream 的 withTimeout 在 streamRead goroutine 内部 wrap(覆盖全流生命周期,ChatStream 立即返回不误杀)+defer cancel 同 goroutine;deadline 到期=ctx.Err 传播走既有 emitted 语义(已收 tokens 已发 ch 不丢——部分结果保留);连接清理走既有 stream.Close defer 链。

③ CloneWithModel 继承:policy 值拷贝(callPolicy 两 int 字段)——子代理继承同策略是合理默认(策略是端点/模型属性非 agent 属性);需更短 deadline 的场景经 SetCallPolicy 显式覆盖(setter 存在)。

#2530 时序裁定:先 #2528#2530(本 PR 先合并;#2530 若冲突 rebase——openai_responses degrade-retry 接线点,预期平凡)。

⑤ 未接入面注释诚实(openai-responses/copilot 后续接)。

CI 9/9 绿。可合并(按④先序)。

@topcheer

Copy link
Copy Markdown
Owner Author

复审已过(见 review:零值 identity 回退等价/goroutine 内 wrap+emitted 语义保留部分结果/CloneWithModel 值继承/#2530 先序裁定/诚实注释)。与 main 冲突——#2524/#2525/#2526/#2527 先合并动了 anthropic.go 同区(struct 字段区+Chat 重试链)。分支 sa78-model-call-policy 有归属者本地 worktree 在途——请归属者 rebase main 解冲突,解后 CI 绿即合并。不代解。

Junjun Zhang and others added 3 commits September 18, 2026 13:49
…etries)

Add configurable per-endpoint and per-model call tuning resolved from
vendor config: RequestTimeout (deadline for a single LLM call) and
MaxRetries (retry budget override). Model-level entries take priority
over endpoint-level defaults; zero values preserve existing behavior
(no deadline, providerRetryAttempts), keeping the feature fully opt-in.

Wire the resolved policy into providers via a callPolicySetter
interface so NewProvider applies it uniformly across protocols.

Co-Authored-By: ggcode <noreply@ggcode.dev>
- Add policy field + setCallPolicy to the three providers; policy is
  copied through CloneWithModel so named-subagent model overrides keep
  the endpoint's timeout/retry budget.
- Replace the hardcoded providerRetryAttempts with p.policy.attempts()
  (23 call sites): configured max_retries now governs Chat and stream
  retry loops.
- Apply policy.withTimeout in Chat and inside each streamRead goroutine
  so the deadline covers the full stream lifetime; no-op when unset.
- NewProvider injects the resolved policy uniformly via callPolicySetter
  for anthropic/openai/gemini; other protocols keep existing defaults.

Co-Authored-By: ggcode <noreply@ggcode.dev>
- config: TestResolveEndpointSelection_CallPolicy covers per-model
  override > endpoint-level fallback > zero-value default.
- provider: TestCallPolicyAttempts / TestCallPolicyWithTimeout cover
  the retry-budget fallback and deadline semantics; TestNewProviderAppliesCallPolicy
  verifies registry injection; TestCloneWithModelKeepsPolicy verifies
  model clones inherit the endpoint policy.

Co-Authored-By: ggcode <noreply@ggcode.dev>
@topcheer
topcheer force-pushed the sa78-model-call-policy branch from 8894cd6 to 19ca62e Compare September 18, 2026 05:52
@topcheer
topcheer merged commit a528bf7 into main Sep 18, 2026
9 checks passed
@topcheer
topcheer deleted the sa78-model-call-policy branch September 18, 2026 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant