Skip to content

fix: 全部账户额度冷却时逐个尝试冷却中的账户 #140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mag1cFall merged 1 commit into from
Nov 11, 2026
+4 −6
Merged

fix: 全部账户额度冷却时逐个尝试冷却中的账户 #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ Worker 容量由热池目标、活动上限和单账户并发共同约束。活

故障重置先等待同账户的活动请求释放租约；等待期间该账户暂停接收新请求。客户端取消只结束自身请求。启动预热在官网 Run 按钮启用后提交，请求在发送前再次检查账户冷却状态。

额度冷却按 `quota_unit`、`quota_limit`、`quota_metric`、错误文案依次判定周期。明确的每日限额在美国太平洋时间次日零点恢复；分钟限额按有效的 `window_start_time` 恢复；通用 429 冷却一分钟。写明恢复时间的套餐限额（`You've reached your usage limit for this model (resets in 10 hours, 2 minutes)`）冷却到该时间；没有重试时间的套餐限额（`You've reached your usage limit` 或 Build 写代码助手步骤带每用户额度标记）按同一账户同一 scope 的连续次数冷却 1、5、15、30、60 分钟，冷却未结束时的并发失败沿用当前期限，该 scope 成功一次后从 1 分钟重新开始；订阅模型遇到额度错误时，后台立即刷新该账户的权益与目录，同一账户每分钟最多一次。`RetryInfo` 与 `Retry-After` 提供的重试时间优先。错误列出的额度项上限全部为 0（`limit: 0`）时冷却类型为无配额，期限同样按上述规则。全局范围由额度元数据确认，其余错误只冷却失败模型或能力 scope；正文后的额度错误与 Live 会话错误使用同一规则。全部候选账户都在冷却且最早恢复晚于 1 分钟时，套餐限额与每日限额冷却中的账户仍接收请求：上游按每个请求判定是否放行，用尽后较短的请求仍可能完成；同一请求再收到这两种限额拒绝时按上游原因冷却并返回该 429，不再换账户。
额度冷却按 `quota_unit`、`quota_limit`、`quota_metric`、错误文案依次判定周期。明确的每日限额在美国太平洋时间次日零点恢复；分钟限额按有效的 `window_start_time` 恢复；通用 429 冷却一分钟。写明恢复时间的套餐限额（`You've reached your usage limit for this model (resets in 10 hours, 2 minutes)`）冷却到该时间；没有重试时间的套餐限额（`You've reached your usage limit` 或 Build 写代码助手步骤带每用户额度标记）按同一账户同一 scope 的连续次数冷却 1、5、15、30、60 分钟，冷却未结束时的并发失败沿用当前期限，该 scope 成功一次后从 1 分钟重新开始；订阅模型遇到额度错误时，后台立即刷新该账户的权益与目录，同一账户每分钟最多一次。`RetryInfo` 与 `Retry-After` 提供的重试时间优先。错误列出的额度项上限全部为 0（`limit: 0`）时冷却类型为无配额，期限同样按上述规则。全局范围由额度元数据确认，其余错误只冷却失败模型或能力 scope；正文后的额度错误与 Live 会话错误使用同一规则。全部候选账户都在冷却且最早恢复晚于 1 分钟时，套餐限额与每日限额冷却中的账户仍接收请求：上游按每个请求判定是否放行，用尽后较短的请求仍可能完成；同一请求依次发往这些账户、每个账户一次，上游放行即完成，被拒的账户按上游原因更新冷却，全部拒绝时返回最后一次的 429。

账户状态：

Expand Down
2 changes: 1 addition & 1 deletion docs/protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2743,7 +2743,7 @@ Gemini 列中注明状态码的行以该状态码返回；Interactions 路由与
| 视频仍在生成 | 409 | `video_not_ready` | `api_error` | `INTERNAL` |
| 文件超过 512 MiB | 413 | `file_too_large` | `request_too_large` | `INTERNAL` |
| 上游配额或限流（上游给出重试时长时带 `Retry-After`，Gemini 错误体带 `RetryInfo`） | 429 | `rate_limit_exceeded` | `rate_limit_error` | `RESOURCE_EXHAUSTED` |
| 候选账户均在冷却且 1 分钟内不恢复（套餐限额与每日限额冷却中的账户先发送一次，上游仍拒绝时返回该拒绝；带 `Retry-After`，Gemini 错误体带 `RetryInfo`） | 429 | `rate_limit_exceeded` | `rate_limit_error` | `RESOURCE_EXHAUSTED` |
| 候选账户均在冷却且 1 分钟内不恢复（套餐限额与每日限额冷却中的账户逐个发送一次，全部拒绝时返回最后一次拒绝；带 `Retry-After`，Gemini 错误体带 `RetryInfo`） | 429 | `rate_limit_exceeded` | `rate_limit_error` | `RESOURCE_EXHAUSTED` |
| 上游过载 | 529 | `upstream_error` | `overloaded_error` | `INTERNAL` |
| 当前客户端请求被管理端取消 | 503 | `request_canceled` | `api_error` | `UNAVAILABLE` |
| 生成服务已停止 | 503 | `service_stopped` | `api_error` | `UNAVAILABLE` |
Expand Down
6 changes: 2 additions & 4 deletions internal/app/runtime.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1755,8 +1755,6 @@ type generateRetry struct {
attemptFailure error
// missingInlineMedia 表示已因复用的内联附件文件不存在而在同一账户重试过
missingInlineMedia bool
// quotaRejected 表示已有尝试收到套餐限额或每日限额拒绝，之后全部账户冷却时不再改用这两种冷却中的账户
quotaRejected bool
// quotaUntil 是本次请求中因额度失败而冷却的账户最早的恢复时刻
quotaUntil time.Time
}
Expand Down Expand Up @@ -1850,6 +1848,7 @@ func (service *trackedService) generateWithRetry(
}
if (unbound || fileBound) && len(retry.attempted) > 0 {
enabled, _ := service.pool.EnabledAccounts()
selection.AllowedAccountIDs = []string{}
for _, accountID := range enabled {
if _, exists := retry.attempted[accountID]; !exists {
selection.AllowedAccountIDs = append(selection.AllowedAccountIDs, accountID)
Expand All @@ -1862,7 +1861,7 @@ func (service *trackedService) generateWithRetry(
"排队等待 | 已等待=%s | 模型=%s", waited, modelID,
))
})
quotaFallback := !retry.quotaRejected && selection.ResourceID == ""
quotaFallback := selection.ResourceID == ""
lease, acquireErr := service.acquireWarmLease(requestCtx, selection, quotaFallback)
var buildCooling *aistudio.AllCoolingError
if acquireErr != nil && selection.Channel == aistudio.ChannelBuild && requestCtx.Err() == nil && !aistudio.RequestBuildOnly(request) &&
Expand Down Expand Up @@ -2231,7 +2230,6 @@ func (service *trackedService) settleAttempt(
}
cooldown, quotaFailure := aistudio.QuotaCooldownForError(err, time.Now())
if quotaFailure {
retry.quotaRejected = retry.quotaRejected || aistudio.RequestQuotaCooldown(cooldown.Reason)
modelAccessScope := lease.CooldownScope(modelID)
scopeLabel := modelAccessScope
if cooldown.Global {
Expand Down