diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 2b8545629c..c9c3094097 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -154,6 +154,8 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | `unsafeAllowNativeLocalExec?` | `boolean` | Ancien booléen de Cursor, équivalent à `nativeLocalExec: "on"` uniquement lorsque le champ plus récent n'est pas défini. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Politique d'exécution locale de Cursor. `off` est la valeur par défaut ; actuellement, `codex-sandbox` échoue de manière sûre comme `off`. | +Pour les requêtes Responses traduites utilisant un pool de plusieurs clés, chaque invocation du fournisseur routé partage une limite de rotations de clés API entre la récupération initiale, les tours des ponts de recherche web, d’image et de vidéo, et les requêtes de continuation jusqu’à la fin de la réponse : au maximum `N - 1` rotations si la clé initiale appartient au pool, ou `N` si elle est extérieure au pool, où `N` est la taille du pool avant le premier envoi. L’expiration d’un délai de refroidissement ou l’agrandissement ultérieur du pool ne renouvelle pas cette limite. Les budgets d’envoi existants applicables peuvent arrêter les tentatives plus tôt ; les ponts sidecar conservent leurs propres limites de reprise après réinitialisation de connexion et de nouvelle tentative vers la même cible, sans que cette limite de rotation leur impose le budget d’envois physiques du core. Si la rotation est refusée, le dernier 429 enregistre toujours le délai de refroidissement de la clé en échec, mais aucune clé de remplacement n’est sélectionnée et la réponse suit le traitement d’erreur existant. + Les fournisseurs à clé API peuvent détenir une clé littérale ou une référence à une variable d'environnement. Les fournisseurs OAuth utilisent le magasin d'identifiants alimenté par `ocx login` ; le comportement de lancement de Claude Code avec abonnement est configuré sous [`claudeCode.authMode`](/fr/reference/configuration/server/#claude-code-claudecode). diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 9b5acfa274..ee9d15459a 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -146,6 +146,8 @@ account を削除しても mapping は保持され、同じ id を再追加す | `unsafeAllowNativeLocalExec?` | `boolean` |カーソルのレガシー ブール値。新しいフィールドが設定されていない場合のみ、`nativeLocalExec: "on"` と同等です。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` |カーソルのローカル実行ポリシー。 `off` がデフォルトです。 `codex-sandbox` は現在、`off` と同様にフェールクローズされます。 | +複数キーのプールを使用する変換済み Responses リクエストでは、ルーティング先プロバイダーの呼び出しごとに、初期リカバリー、ウェブ検索・画像・動画ブリッジの各ラウンド、応答完了のための後続リクエストで API キーの切り替え上限を共有し、最初の送信前のプールサイズを `N` として、最初のキーがそのプール内にある場合は最大 `N - 1` 回、プール外にある場合は最大 `N` 回に固定されます。クールダウンの満了や後からのプール拡張で、この上限が補充されることはありません。適用対象となる既存の送信予算によって、再試行がさらに早く停止する場合もあります。サイドカーブリッジは接続リセット時と同一対象への再試行にそれぞれ既存の上限を維持し、この切り替え上限によって core の物理送信予算がブリッジに新たに適用されることはありません。切り替えが拒否された場合も、最後の 429 に対する失敗したキーのクールダウンは記録されますが、代わりのキーは選択されず、応答には既存のエラー処理が適用されます。 + API キープロバイダーは、リテラルキーまたは環境参照を保持する場合があります。 OAuth プロバイダーは、`ocx login` によって設定された資格情報ストアを使用します。サブスクリプションに基づくクロード コードの起動動作は、[`claudeCode.authMode`](/reference/configuration/server/#claude-code) で構成されます。 ## プロバイダーによるアウトバウンドの安全性診断 diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index b6657ba58d..7cdd7d7280 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -146,6 +146,8 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 레거시 불리언입니다. 더 새로운 필드가 설정되지 않았을 때만 `nativeLocalExec: "on"`과 같습니다. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 로컬 실행 정책입니다. 기본값은 `off`입니다. `codex-sandbox`는 현재 `off`처럼 실패를 닫습니다. | +여러 키가 있는 풀을 사용하는 변환된 Responses 요청에서는 라우팅된 프로바이더 호출마다 최초 복구, 웹 검색·이미지·비디오 브리지의 각 라운드, 응답 완료를 위한 후속 요청이 API 키 회전 한도를 공유하며, 첫 전송 전의 풀 크기를 `N`으로 고정하고 최초 키가 해당 풀에 있으면 최대 `N - 1`회, 풀 밖에 있으면 최대 `N`회 회전할 수 있습니다. 쿨다운이 만료되거나 이후 풀이 커져도 이 한도는 충전되지 않습니다. 적용 대상인 기존 전송 예산에 따라 재시도가 더 일찍 멈출 수 있습니다. 사이드카 브리지는 자체 연결 재설정 및 동일 대상 재시도 한도를 유지하며, 이 회전 한도가 브리지에 core의 물리적 전송 예산을 새로 적용하지는 않습니다. 회전이 거부되면 마지막 429에 따른 실패 키의 쿨다운은 기록하지만 대체 키를 선택하지 않으며, 응답은 기존 오류 처리 방식으로 마무리합니다. + API 키 공급자는 리터럴 키나 환경 참조를 둘 수 있습니다. OAuth 공급자는 `ocx login`으로 채워지는 자격 증명 저장소를 사용합니다. 구독 기반 Claude Code 실행 동작은 [`claudeCode.authMode`](/reference/configuration/server/#claude-code)에서 설정합니다. ## 공급자 진단용 외부 요청 안전성 diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 9a0b822f52..8fd590d0e9 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -225,6 +225,8 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor legacy boolean, equivalent to `nativeLocalExec: "on"` only when the newer field is unset. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor local-exec policy. `off` is default; `codex-sandbox` currently fails closed like `off`. | +For translated Responses requests using a multi-key pool with `authMode: "key"` (excluding OAuth and forward-auth account pools), each routed provider invocation shares its API-key rotation allowance between initial recovery, web-search/image/video bridge rounds, and terminal continuations: at most `N - 1` rotations if the initial key belongs to the pool, or `N` if it is outside the pool, where `N` is the pool size before the first send. Cooldown expiry or later pool growth does not replenish this allowance. Existing applicable send budgets may stop retries sooner; sidecar bridges retain their own connection-reset and same-target retry limits, and this rotation cap does not impose the core's physical-send budget on those bridges. Once rotation is refused, the last 429 still records the failed key's cooldown, but no replacement key is selected and the response follows the existing error handling. Same-key `retryOn429` replays use their separate request-wide retry counter; this allowance counts key rotations only. + With `webSearchBridge` enabled, a search continuation stays bound to the API-key selection that served the first request. Changing the selected key, its reference or resolved value, authentication mode, or base URL during search or provider pacing ends the turn with a bridge error before another diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index ca389d9ffd..a0f5936360 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -159,6 +159,8 @@ cross-route credential fallback не существует. Строки API GPT- | `unsafeAllowNativeLocalExec?` | `boolean` | Legacy boolean Cursor, эквивалентен `nativeLocalExec: "on"` только если новое поле не задано. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Политика local-exec для Cursor. `off` — дефолт; `codex-sandbox` сейчас ведёт себя fail-closed как `off`. | +Для преобразованных запросов Responses с пулом из нескольких API-ключей каждый вызов выбранного маршрутизацией провайдера использует общий предел переключений для первоначального восстановления, раундов мостов веб-поиска, изображений и видео, а также последующих запросов, завершающих ответ: не более `N - 1`, если начальный ключ входит в пул, или `N`, если он находится вне пула; `N` фиксируется как размер пула до первой отправки. Истечение периода ожидания или последующее расширение пула не восстанавливает этот лимит. Применимые существующие бюджеты отправки могут остановить повторы раньше; мосты sidecar сохраняют собственные пределы повторов после сброса соединения и повторных запросов к той же цели, а этот предел переключений не распространяет на них бюджет физических отправок core. Если переключение запрещено, для последнего 429 всё равно записывается период ожидания отказавшего ключа, но другой ключ не выбирается, а ответ обрабатывается существующим механизмом обработки ошибок. + Провайдеры с API-key могут хранить literal key или environment-reference. OAuth-провайдеры используют credential store, заполняемый через `ocx login`; поведение subscription-backed launcher'а Claude Code настраивается через diff --git a/docs-site/src/content/docs/tr/reference/configuration/providers.md b/docs-site/src/content/docs/tr/reference/configuration/providers.md index 3126b9046c..778175d2fd 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/tr/reference/configuration/providers.md @@ -160,6 +160,8 @@ alanlı seçilmiş kimlikleri yalın kimliklere yeniden yazar. | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor eski boolean değeri, yalnızca daha yeni alan ayarlanmadığında `nativeLocalExec: "on"` değerine eşdeğerdir. | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor yerel yürütme politikası. `off` varsayılandır; `codex-sandbox` şu anda `off` gibi kapalı olarak başarısız olur. | +Birden çok anahtar içeren havuz kullanan dönüştürülmüş Responses isteklerinde, yönlendirilen sağlayıcının her çağrısı ilk kurtarma, web araması/görüntü/video köprüsü turları ve yanıtı tamamlayan devam istekleri arasında bir API anahtarı değiştirme sınırını paylaşır: başlangıç anahtarı havuzdaysa en fazla `N - 1`, havuz dışındaysa en fazla `N` değişim yapılabilir; `N`, ilk gönderimden önceki havuz boyutuna sabitlenir. Bekleme süresinin dolması veya havuzun sonradan büyümesi bu hakkı yenilemez. Geçerli mevcut gönderim bütçeleri yeniden denemeleri daha erken durdurabilir; sidecar köprüleri bağlantı sıfırlaması ve aynı hedefe yeniden deneme için kendi sınırlarını korur ve bu anahtar değiştirme sınırı, core'un fiziksel gönderim bütçesini bu köprülere uygulamaz. Anahtar değişimi reddedildiğinde son 429 için başarısız anahtarın bekleme süresi yine kaydedilir, ancak yerine başka bir anahtar seçilmez ve yanıt mevcut hata işleme yolunu izler. + API anahtarı sağlayıcıları değişmez bir anahtar veya bir ortam referansı tutabilir. OAuth sağlayıcıları `ocx login` tarafından doldurulan kimlik bilgisi deposunu kullanır; abonelik destekli Claude Code başlatma davranışı diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index b28a5a2af4..96f58746ea 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -146,6 +146,8 @@ selector，而不是分配一个新名称。 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 旧布尔值；仅当更新字段未设置时，等同于 `nativeLocalExec: "on"`。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本地执行策略。`off` 是默认值；`codex-sandbox` 目前会像 `off` 一样失败关闭。 | +对于使用多密钥池的转换后 Responses 请求，每次路由到提供商的调用，其初始恢复、网页搜索/图像/视频桥接的各轮调用以及用于完成响应的后续请求共同使用 API 密钥轮换额度：初始密钥在池中时最多轮换 `N - 1` 次，不在池中时最多轮换 `N` 次，其中 `N` 固定为首次发送前的密钥池大小。冷却期结束或随后扩大密钥池都不会补充此额度。现有且适用的发送预算可能让重试更早停止；sidecar 桥接仍保留自身的连接重置重试和同目标重试限制，此轮换额度不会将 core 的实际发送预算新增应用于这些桥接。轮换被拒绝时，仍会根据最后一个 429 记录失败密钥的冷却期，但不会选择替代密钥，响应继续按现有错误处理方式处理。 + API key 提供者可以持有字面量 key，或环境引用。OAuth 提供者使用由 `ocx login` 填充的凭据存储；基于订阅的 Claude Code 启动行为在 [`claudeCode.authMode`](/reference/configuration/server/#claude-code) 下配置。 ## 提供者诊断出站安全性 diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 0680fa35aa..5626ef27c5 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -116,6 +116,8 @@ ocx models provider openrouter on | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 舊版布林值，僅在較新欄位未設定時等同於 `nativeLocalExec: "on"`。 | | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本機執行政策。`off` 為預設；`codex-sandbox` 目前像 `off` 般 fail closed。 | +對於使用多金鑰集區的轉換後 Responses 要求，每次路由至提供者的呼叫，其初始復原、網頁搜尋/影像/影片橋接的各輪呼叫以及用於完成回應的後續要求共同使用 API 金鑰輪替額度：初始金鑰在集區中時最多輪替 `N - 1` 次，不在集區中時最多輪替 `N` 次，其中 `N` 固定為首次傳送前的金鑰池大小。冷卻期結束或之後擴大金鑰池都不會補充此額度。現有且適用的傳送預算可能讓重試更早停止；sidecar 橋接仍保留自身的連線重設重試和同目標重試限制，此輪替額度不會將 core 的實際傳送預算新增套用至這些橋接。輪替遭拒時，仍會根據最後一個 429 記錄失敗金鑰的冷卻期，但不會選取替代金鑰，回應繼續依現有錯誤處理方式處理。 + API-key 供應商可持有字面值金鑰或環境參考。OAuth 供應商使用由 `ocx login` 填入的憑證存放；訂閱支援的 Claude Code 啟動行為在 [`claudeCode.authMode`](/zh-tw/reference/configuration/server/#claude-code) 下設定。 ## 供應商診斷對外安全 diff --git a/src/lib/request-execution-budget.ts b/src/lib/request-execution-budget.ts index 80654b0a94..a0b868ddd8 100644 --- a/src/lib/request-execution-budget.ts +++ b/src/lib/request-execution-budget.ts @@ -109,6 +109,8 @@ export interface RequestExecutionBudget extends TransientSendBudget { readonly logicalRequestId: string; readonly policyVersion: string; readonly policy: RequestExecutionBudgetPolicy; + /** Share charged/pending sends; an exact prepaid hop may fund the child's initial send. */ + deriveScope(policy: RequestExecutionBudgetPolicy, prepaid?: SingleUseDispatchPermit): RequestExecutionBudget; reserveDispatch(intent: DispatchIntent): DispatchDecision; /** * Sends still available from the base allowance, capped by a layer's own maximum. @@ -133,49 +135,81 @@ const RESERVE_FUNDED_CLASSES: ReadonlySet = new Set([ ]); let logicalRequestSeq = 0; +interface SharedSendLedger { spent: number; pendingExternalSends: Set; } +const permitBookings = new WeakMap(); export function createRequestExecutionBudget( policy: RequestExecutionBudgetPolicy = CODEX_TEXT_GUARDED_BUDGET_POLICY, logicalRequestId?: string, ): RequestExecutionBudget { - let spent = 0; + return createBudgetScope(policy, + logicalRequestId ?? `lr-${Date.now().toString(36)}-${(logicalRequestSeq += 1).toString(36)}`, + { spent: 0, pendingExternalSends: new Set() }); +} + +function createBudgetScope( + policy: RequestExecutionBudgetPolicy, + logicalRequestId: string, + ledger: SharedSendLedger, + prepaidBooking?: object, +): RequestExecutionBudget { // Reservations whose physical send is reported by a retry helper rather than by the permit. // They are already charged; the reporter's first send settles one instead of charging again. - let pendingExternalSends = 0; let reserveSpent = false; let alternateTargetSends = 0; let targetTransitions = 0; let lastTargetKey: string | undefined; const budget: RequestExecutionBudget = { - get used(): number { return spent; }, + get used(): number { return ledger.spent; }, set used(next: number) { // The retry helpers report their real send count by assigning through this field. A // reservation taken with `countedExternally` has already booked one of those sends, so // the report settles the pending booking first and only the surplus is charged. - const delta = next - spent; + const delta = next - ledger.spent; if (delta <= 0) { - spent = Math.max(0, next); + ledger.spent = Math.max(0, next); return; } - const settled = Math.min(delta, pendingExternalSends); - pendingExternalSends -= settled; - spent += delta - settled; + const settled = Math.min(delta, ledger.pendingExternalSends.size); + for (let i = 0; i < settled; i++) { + const ticket = ledger.pendingExternalSends.values().next().value; + if (ticket) ledger.pendingExternalSends.delete(ticket); + } + ledger.spent += delta - settled; }, - logicalRequestId: logicalRequestId ?? `lr-${Date.now().toString(36)}-${(logicalRequestSeq += 1).toString(36)}`, + logicalRequestId, policyVersion: REQUEST_BUDGET_POLICY_VERSION, policy, + deriveScope(scopePolicy, prepaid) { + const booking = prepaid && permitBookings.get(prepaid); + // Intermediate scopes retain the exact prepaid booking. A dispatched child consumes + // it from the shared ledger, so sibling scopes cannot turn it into a second send. + const inherited = prepaid === undefined ? prepaidBooking : booking?.ledger === ledger ? booking.ticket : undefined; + const inheritedPolicy = prepaid === undefined && inherited && ledger.pendingExternalSends.has(inherited) + ? { ...scopePolicy, maxTotalModelSends: Math.min(scopePolicy.maxTotalModelSends, policy.maxTotalModelSends), + baseSendAllowance: Math.min(scopePolicy.baseSendAllowance, policy.baseSendAllowance), + finalRecoveryAllowance: Math.min(scopePolicy.finalRecoveryAllowance, policy.finalRecoveryAllowance) } + : scopePolicy; + return createBudgetScope(inheritedPolicy, logicalRequestId, ledger, inherited); + }, get reserveSpent() { return reserveSpent; }, get alternateTargetSends() { return alternateTargetSends; }, get targetTransitions() { return targetTransitions; }, get lastTargetKey() { return lastTargetKey; }, remainingBaseSends(cap: number): number { const capped = Number.isFinite(cap) ? Math.trunc(cap) : 0; - return Math.max(0, Math.min(capped, policy.baseSendAllowance - spent)); + const prepaid = prepaidBooking && ledger.pendingExternalSends.has(prepaidBooking) ? 1 : 0; + return Math.max(0, Math.min(capped, Math.max(prepaid, policy.baseSendAllowance - ledger.spent + prepaid))); }, reserveDispatch(intent: DispatchIntent): DispatchDecision { if (intent.replaySafe === false) return { allowed: false, reason: "not-replay-safe" }; - if (spent >= policy.maxTotalModelSends) return { allowed: false, reason: "total-exhausted" }; + // Inner adapters call even their first physical attempt "transient". The explicit hop + // permit identifies the prepaid send; the recovery class does not identify its owner. + const inherited = prepaidBooking + && ledger.pendingExternalSends.has(prepaidBooking) ? prepaidBooking : undefined; + const spentBeforeSend = ledger.spent - (inherited ? 1 : 0); + if (spentBeforeSend >= policy.maxTotalModelSends) return { allowed: false, reason: "total-exhausted" }; const changesTarget = lastTargetKey !== undefined && lastTargetKey !== intent.targetKey; const isAlternateTarget = changesTarget || intent.sendClass === "account-failover" @@ -190,7 +224,7 @@ export function createRequestExecutionBudget( // The base allowance is spent first. Only once it is gone does a recovery class reach // for the single shared reserve -- an account move and a validated rebuild cannot each // take one. - const drawsReserve = policy.baseSendAllowance - spent <= 0; + const drawsReserve = !inherited && policy.baseSendAllowance - spentBeforeSend <= 0; if (drawsReserve) { if (!RESERVE_FUNDED_CLASSES.has(intent.sendClass)) { return { allowed: false, reason: "base-allowance-exhausted" }; @@ -205,17 +239,19 @@ export function createRequestExecutionBudget( // one remaining send admitted two physical sends, which is the per-request multiplication // this budget exists to stop. Everything is booked now; `release()` is the way back. const previousTargetKey = lastTargetKey; - spent += 1; - if (intent.countedExternally === true) pendingExternalSends += 1; + const ticket = {}; + if (inherited) { + prepaidBooking = undefined; + ledger.pendingExternalSends.delete(inherited); + } else ledger.spent += 1; + if (intent.countedExternally === true) ledger.pendingExternalSends.add(ticket); if (drawsReserve) reserveSpent = true; if (isAlternateTarget) alternateTargetSends += 1; if (changesTarget) targetTransitions += 1; lastTargetKey = intent.targetKey; let settled: "open" | "used" | "released" = "open"; - return { - allowed: true, - permit: { + const permit: SingleUseDispatchPermit = { sendClass: intent.sendClass, use(): boolean { if (settled !== "open") return false; @@ -228,17 +264,21 @@ export function createRequestExecutionBudget( // An externally counted reservation the reporter already settled paid for a send // that physically happened. Refunding it would hand the request a free send back. if (intent.countedExternally === true) { - if (pendingExternalSends === 0) return; - pendingExternalSends -= 1; + if (!ledger.pendingExternalSends.has(ticket)) return; + ledger.pendingExternalSends.delete(ticket); } - spent -= 1; + if (inherited) { + ledger.pendingExternalSends.add(inherited); + prepaidBooking = inherited; + } else ledger.spent -= 1; if (drawsReserve) reserveSpent = false; if (isAlternateTarget) alternateTargetSends -= 1; if (changesTarget) targetTransitions -= 1; lastTargetKey = previousTargetKey; }, - }, }; + if (intent.countedExternally === true) permitBookings.set(permit, { ledger, ticket }); + return { allowed: true, permit }; }, }; return budget; diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index d5ccd759e3..ef35b2c505 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -363,6 +363,7 @@ function rotateKeyAfterFailure( now = Date.now(), attemptedKey?: string, attemptedSelection?: ProviderApiKeySelection, + allowRotation = true, ): OcxProviderConfig | null { const provider = config.providers[providerName]; if (!provider) return null; @@ -382,6 +383,10 @@ function rotateKeyAfterFailure( ? pool.find(entry => entry.id === attemptedSelection.entryId && entry.key === failedKey) : pool.find(entry => entry.key === failedKey); + // A spent request still records the failed key, but must not select or persist an + // unattempted replacement. Keep the fresh identity check and the changed:false path. + if (!allowRotation) return { changed: false, value: { failedId: failedEntry?.id } }; + if (freshProvider.apiKey !== failedKey) { const activeEntry = pool.find(entry => entry.key === freshProvider.apiKey); if (activeEntry && !isKeyInCooldown(providerName, activeEntry.id, now)) { @@ -410,6 +415,7 @@ function rotateKeyAfterFailure( }, attemptedSelection); if (outcome.status === "unavailable") return null; if (outcome.status === "superseded") { + if (!allowRotation) return null; // A newer manual selection (including A→B→A) owns subsequent dispatch. Reusing the // same failed key here would loop forever; preserve its original failure instead. return outcome.provider.apiKey !== failedKey ? structuredClone(outcome.provider) : null; @@ -425,6 +431,7 @@ function rotateKeyAfterFailure( keyCooldowns.set(cooldownKey(providerName, outcome.value.failedId), { cooldownUntil: now + cooldownMs }); sweepExpiredOnWrite(now); } + if (!allowRotation) return null; if ("exhaustedCount" in outcome.value) { console.warn(`[key-failover] ${providerName}: all ${outcome.value.exhaustedCount} keys in cooldown after ${failureStatus}; returning the upstream status to the client`); return null; @@ -448,8 +455,9 @@ export function rotateKeyOn429( now = Date.now(), attemptedKey?: string, attemptedSelection?: ProviderApiKeySelection, + allowRotation = true, ): OcxProviderConfig | null { - return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection); + return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection, allowRotation); } /** @@ -486,6 +494,8 @@ interface RotateProviderTransportOptions { attemptedKey?: string; attemptedSelection?: ProviderApiKeySelection; promptCacheKey?: string; + /** False records a proven 429 cooldown without changing the selected key or returning a retry. */ + allowRotation?: boolean; } /** @@ -507,6 +517,7 @@ export function rotateProviderTransportOn429( options.now, options.attemptedKey, options.attemptedSelection ?? routedProvider._apiKeyAttempt, + options.allowRotation, ); if (!rotated) return null; return applyRotatedTransport(providerName, routedProvider, rotated, options.promptCacheKey); @@ -517,7 +528,7 @@ export function rotateProviderTransportOn401( config: OcxConfig, providerName: string, routedProvider: OcxProviderTransport, - options: Omit = {}, + options: Omit = {}, ): OcxProviderTransport | null { const rotated = rotateKeyOn401(config, providerName, options.now, options.attemptedKey, options.attemptedSelection ?? routedProvider._apiKeyAttempt); diff --git a/src/server/responses/adapter-continuation.ts b/src/server/responses/adapter-continuation.ts index a1db9398d5..f87514d901 100644 --- a/src/server/responses/adapter-continuation.ts +++ b/src/server/responses/adapter-continuation.ts @@ -24,6 +24,8 @@ import { } from "../../providers/key-failover"; import { fetchWithTransientRetry, + TRANSIENT_RETRY_MAX_ATTEMPTS, + SendBudgetExhaustedError, fetchWithResetRetry, applyUpstreamRecoveryInit, prepareSameTarget429Wait, @@ -87,6 +89,11 @@ export function createAdapterContinuations( | "remainingTransientSendBudget" | "noteTransientSends" | "reserveCredentialHop" + | "pendingHopPermit" + | "keyPoolFailovers" + | "recoverySendAllowance" + | "recoveryClassFor" + | "sendBudgetExhausted" >, adapterExchange: Pick< AdapterExchange, @@ -95,6 +102,7 @@ export function createAdapterContinuations( | "rateLimitPolicy" | "rateLimitRetries" | "stallTimeoutMs" + | "keyPool429RetryAllowed" >, ) { const { options, logCtx, config } = requestContext; @@ -115,6 +123,9 @@ export function createAdapterContinuations( remainingTransientSendBudget, noteTransientSends, reserveCredentialHop, + recoverySendAllowance, + recoveryClassFor, + sendBudgetExhausted, } = sendBudgetState; @@ -203,8 +214,17 @@ export function createAdapterContinuations( const fetchContinuationWithRetryPolicy = (route.provider.adapter === "google" || continuationTransientPolicy) ? fetchWithTransientRetry : fetchWithResetRetry; - return await fetchContinuationWithRetryPolicy( + const continuationCap = continuationTransientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS; + const allowance = replayKind + ? recoverySendAllowance(continuationCap, recoveryClassFor(replayKind), + `${route.providerName}|${route.modelId}|${replayKind}`) + : { attempts: remainingTransientSendBudget(continuationCap), permit: undefined }; + try { + return await fetchContinuationWithRetryPolicy( recovery => { + if (allowance.permit && !allowance.permit.use()) { + throw new SendBudgetExhaustedError(safeHostLabel(builtContinuationRequest.url)); + } noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? replayKind); return fetchWithHeaderTimeout( builtContinuationRequest.url, @@ -229,19 +249,19 @@ export function createAdapterContinuations( // Same request-scoped budget as the initial send and the 429/rotation refetches: // a terminal-guard continuation is another leg of ONE request, so handing it a // fresh `attempts` would let one request exceed the configured total-send ceiling. - ...(continuationTransientPolicy - ? { - attempts: remainingTransientSendBudget(continuationTransientPolicy.attempts), - onSendsConsumed: noteTransientSends, - } - : {}), + attempts: allowance.attempts, + onSendsConsumed: noteTransientSends, }, ); + } finally { + allowance.permit?.release(); + } } finally { builtContinuationRequest.releaseBodyObservation?.(); } }; while (true) { + const continuationHop = sendBudgetState.pendingHopPermit; try { const recoveryKind = nextContinuationRecoveryKind; nextContinuationRecoveryKind = undefined; @@ -253,6 +273,9 @@ export function createAdapterContinuations( yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` }; } return; + } finally { + sendBudgetState.pendingHopPermit = undefined; + continuationHop?.release(); } // Same-target 429 wait-and-retry (opt-in `retryOn429`) before key/account failover: @@ -262,6 +285,7 @@ export function createAdapterContinuations( response.status === 429 && rateLimitPolicy !== null && adapterExchange.rateLimitRetries < rateLimitPolicy.attempts + && !sendBudgetExhausted() ) { adapterExchange.rateLimitRetries += 1; // Release unread body + heartbeat-fed wait via the shared same-target helper. @@ -308,8 +332,10 @@ export function createAdapterContinuations( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: nextParsed.options.promptCacheKey, + allowRotation: adapterExchange.keyPool429RetryAllowed(), }); if (rotated) { + sendBudgetState.keyPoolFailovers += 1; try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } route.provider = rotated; invalidateSameTargetRequest(); @@ -388,6 +414,7 @@ export function createAdapterContinuations( const hop = reserveCredentialHop( "auth-recovery", `${route.providerName}|${route.modelId}|continuation-oauth-429`, + !transportState.activeAdapter.fetchResponse, ); const nextAccountId = hop.allowed ? rotateGenericOAuthAccountOn429( @@ -418,6 +445,7 @@ export function createAdapterContinuations( sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.activeAdapter.name, logCtx.accountLogLabel); recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, transportState.activeAdapter.name); nextContinuationRecoveryKind = "oauth-account-429"; + sendBudgetState.pendingHopPermit = transportState.activeAdapter.fetchResponse ? undefined : hop.permit; continue; } } catch { diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 57eadf6a28..b5fd739c0c 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -29,6 +29,7 @@ import { } from "../../providers/key-failover"; import { fetchWithTransientRetry, + TRANSIENT_RETRY_MAX_ATTEMPTS, fetchWithResetRetry, applyUpstreamRecoveryInit, SendBudgetExhaustedError, @@ -127,6 +128,9 @@ export async function prepareAdapterExchange( | "recoveryClassFor" | "sendBudgetExhausted" | "reserveCredentialHop" + | "pendingHopPermit" + | "keyPoolFailovers" + | "keyPool429RetryAllowed" >, ) { const { options, config, logCtx, req } = requestContext; @@ -171,6 +175,7 @@ export async function prepareAdapterExchange( ? Math.floor(config.stallTimeoutSec * 1000) : 300_000; transportState.activeAdapter = transportState.adapter; + const keyPool429RetryAllowed = () => sendBudgetState.keyPool429RetryAllowed(!!transportState.activeAdapter.fetchResponse); // One immutable, body-safe outbound request per same-target sequence (URL, serialized body, // auth headers, generated compat headers). Same-target 429 replays reuse it verbatim; the @@ -315,15 +320,9 @@ export async function prepareAdapterExchange( { abortSignal: upstream.signal, label: safeHostLabel(builtInitialRequest.url), - ...(transientPolicy - // Draws the remainder, not the raw policy. A combo child inherits the parent's - // holder but used to take a fresh full allowance on its own first send, so the - // shared counter was inherited without ever being read as a limit. - ? { - attempts: remainingTransientSendBudget(transientPolicy.attempts), - onSendsConsumed: noteTransientSends, - } - : {}), + // Count both retry policies; reset-only still hops immediately on HTTP 5xx. + attempts: remainingTransientSendBudget(transientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS), + onSendsConsumed: noteTransientSends, }, ); } @@ -448,13 +447,11 @@ export async function prepareAdapterExchange( // Same rule as the passthrough rebuild: spend the base allowance first, then the one // shared final-recovery reserve, so a recovery that follows a spent streak still gets // its single send instead of dying at three. - const refetchAllowance = refetchTransientPolicy - ? recoverySendAllowance( - refetchTransientPolicy.attempts, - recoveryClassFor(recovery), - `${route.providerName}|${route.modelId}|${recovery}`, - ) - : undefined; + const refetchAllowance = recoverySendAllowance( + refetchTransientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS, + recoveryClassFor(recovery), + `${route.providerName}|${route.modelId}|${recovery}`, + ); try { return await refetchWithPolicy( recoveryKind => { @@ -644,8 +641,10 @@ export async function prepareAdapterExchange( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: parsed.options.promptCacheKey, + allowRotation: keyPool429RetryAllowed(), }); if (!rotated) break; + sendBudgetState.keyPoolFailovers += 1; // Release the failed response's socket before retrying; unread bodies otherwise linger // until runtime cleanup (one per rotated key under a rate-limit storm). try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } @@ -725,6 +724,7 @@ export async function prepareAdapterExchange( const hop = reserveCredentialHop( "auth-recovery", `${route.providerName}|${route.modelId}|adapter-recovery-oauth-429`, + !transportState.activeAdapter.fetchResponse, ); if (!hop.allowed) break; const nextAccountId = rotateGenericOAuthAccountOn429( @@ -755,22 +755,22 @@ export async function prepareAdapterExchange( ); sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.activeAdapter.name, logCtx.accountLogLabel); recordAttemptCredentialSource(logCtx.activeAttempt, route.providerName, route.provider, transportState.activeAdapter.name); - // Confirm at the dispatch boundary, not here: a rebuild can fail while shaping the - // request and return `{ failed }` without reaching the wire, and a permit confirmed - // before that would hold the charge for a send that never happened. - const result = await rebuildAndRefetch("oauth-account-429", () => { hop.permit?.use(); }); - if ("failed" in result) { - // A no-op if the boundary was reached; a refund if the rebuild died before it. + // Retry helpers settle the externally counted hop themselves. Adapter-owned + // sends confirm only after pacing, preserving the release-on-rebuild-failure fix. + sendBudgetState.pendingHopPermit = transportState.activeAdapter.fetchResponse ? undefined : hop.permit; + try { + const result = await rebuildAndRefetch("oauth-account-429", + transportState.activeAdapter.fetchResponse ? () => { hop.permit?.use(); } : undefined); + if ("failed" in result) return result.failed; + upstreamResponse = result; + } finally { + sendBudgetState.pendingHopPermit = undefined; + // Idempotent: refund only an unused reservation, never a dispatched send. hop.permit?.release(); - return result.failed; } - upstreamResponse = result; } catch { - // A throw before the send — snapshot fetch, credential application, adapter - // resolution — must hand the reservation back. Without this the ladder charges the - // request for a send it never made, and a later recovery in the same request is - // refused on an allowance nothing spent. release() is idempotent and a no-op once - // used, so a throw from the rebuild keeps its charge. + // Snapshot, credential and adapter-resolution failures may precede dispatch. + // Both adapter and retry-helper paths must refund an unused reservation. hop.permit?.release(); break; } @@ -954,6 +954,7 @@ export async function prepareAdapterExchange( return { upstream, + keyPool429RetryAllowed, cleanupUpstreamAbort, connectMs, stallTimeoutMs, diff --git a/src/server/responses/combo-send-budget.ts b/src/server/responses/combo-send-budget.ts new file mode 100644 index 0000000000..7ec4b94539 --- /dev/null +++ b/src/server/responses/combo-send-budget.ts @@ -0,0 +1,81 @@ +import { CODEX_TEXT_GUARDED_BUDGET_POLICY, type RequestExecutionBudget, type RequestExecutionBudgetPolicy, type SingleUseDispatchPermit } from "../../lib/request-execution-budget"; + +/** + * Sends one combo target may run on its own before the ladder moves on. A target is a whole + * request as far as its own provider is concerned, so this is the guarded profile's base + * allowance rather than a separate number to keep in sync. + */ +export const COMBO_TARGET_BASE_SENDS = CODEX_TEXT_GUARDED_BUDGET_POLICY.baseSendAllowance; + +/** + * A combo's execution policy is DECLARED by the combo, not inherited from the single-target + * profile. + * + * `maxTargetTransitions: 1` and `maxAlternateTargetSends: 1` describe an account move, and + * applying them to a combo would refuse the second hop of a three-target combo -- which is why + * combo was left off `reserveDispatch` when the per-request split landed. The transitions a + * combo may make are exactly the targets it declares minus the one it starts on. What stays + * capped is the TOTAL: the first target's full ladder, one send for every further declared + * target, and the one shared final-recovery reserve. A one-target combo reduces to the guarded + * profile exactly, and a three-target combo whose every target fails hard reaches upstream six + * times instead of the twelve #4546 measured. + */ +export function comboExecutionBudgetPolicy(declaredTargets: number): RequestExecutionBudgetPolicy { + const targets = Math.max(1, Math.trunc(declaredTargets)); + const hops = targets - 1; + const reserve = CODEX_TEXT_GUARDED_BUDGET_POLICY.finalRecoveryAllowance; + const total = COMBO_TARGET_BASE_SENDS + hops + reserve; + return { + maxTotalModelSends: total, + baseSendAllowance: total - reserve, + finalRecoveryAllowance: reserve, + maxAlternateTargetSends: Math.max(1, hops), + maxTargetTransitions: Math.max(1, hops), + }; +} + +/** + * A budget scope that keeps its own recovery ledgers but spends the SAME request-wide counter. + * + * The factory shares both charged sends and pending external reports. Forwarding `used` alone + * cannot share reservation checks held in the factory closure, and would let each target + * refill its allowance. The reserve, alternate-target and transition ledgers stay + * per-scope on purpose: a combo target's account failover is its own recovery decision, while + * the request total still bounds every target together. + */ +export function deriveSendBudgetScope( + parent: RequestExecutionBudget, + policy: RequestExecutionBudgetPolicy, + prepaid?: SingleUseDispatchPermit, +): RequestExecutionBudget { + return parent.deriveScope(policy, prepaid); +} + +/** + * The ladder one combo target may run, expressed as an allowance on the request-wide counter. + * + * `used + COMBO_TARGET_BASE_SENDS` gives this target its own ladder from wherever the request + * already stands, and the clamp holds back one send for each target still declared after it: a + * first target that 5xx-streaks must not eat the send the last declared target is entitled to. + * That guarantee is the difference between a per-target policy and a shared pool the first + * target drains. + */ +export function comboTargetSendBudget( + comboScope: RequestExecutionBudget, + targetsDeclaredAfterThisOne: number, + prepaid?: SingleUseDispatchPermit, +): RequestExecutionBudget { + const policy = comboScope.policy; + const heldForLaterTargets = Math.max(0, targetsDeclaredAfterThisOne); + const ceiling = Math.min(policy.maxTotalModelSends, + Math.max(comboScope.used, 1, policy.maxTotalModelSends - heldForLaterTargets)); + return deriveSendBudgetScope(comboScope, { + maxTotalModelSends: ceiling, + baseSendAllowance: Math.min(ceiling, comboScope.used - (prepaid ? 1 : 0) + COMBO_TARGET_BASE_SENDS), + finalRecoveryAllowance: policy.finalRecoveryAllowance, + // Within one target the account-move shape is unchanged: three same-account sends plus one + // alternate is the recovery live traffic depends on, and a combo does not widen it. + maxAlternateTargetSends: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxAlternateTargetSends, + maxTargetTransitions: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxTargetTransitions, + }, prepaid); +} diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 6b5b979d37..2b9967a33a 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1218,6 +1218,20 @@ export async function handleResponsesCompact( } else if (quotaFailure && !storedPool401ReplayAttempted) { const fallbackModel = compactHandoffRoute(req, raw.model); if (fallbackModel && !req.signal.aborted) { + const hop = sendBudget.reserveDispatch({ + sendClass: "account-failover", + targetKey: `compact-handoff:${fallbackModel}`, + countedExternally: true, + }); + if (!hop.allowed) return buffered; + // The routed child's initial send spends this exact recovery booking. Its base + // exposes the prepaid send even after native compact used the three-send ladder; + // it shares the total ledger and receives no additional recovery reserve. + const handoffBudget = sendBudget.deriveScope({ + ...sendBudget.policy, + baseSendAllowance: Math.max(sendBudget.policy.baseSendAllowance, sendBudget.used), + finalRecoveryAllowance: 0, + }, hop.permit); const fallbackReq = new Request(req.url, { method: "POST", headers: req.headers, @@ -1233,13 +1247,15 @@ export async function handleResponsesCompact( admission, // The handoff child is the same logical compact on a second model, so it inherits // the holder. Forwarding `options` alone was not enough: the child minted its own. - { ...options, sendBudget }, + { ...options, sendBudget: handoffBudget }, ); if (fallback.ok || fallback.status === 499) return fallback; await fallback.body?.cancel().catch(() => undefined); } catch { // The previous-model rejection is the authoritative failure when the // remembered handoff route can no longer compact this thread. + } finally { + hop.permit.release(); } } } diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 43863fc294..3f4325ce43 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1,12 +1,4 @@ -import { - CODEX_TEXT_GUARDED_BUDGET_POLICY, - createRequestExecutionBudget, - isRequestExecutionBudget, -} from "../../lib/request-execution-budget"; -import type { - RequestExecutionBudgetPolicy, - RequestExecutionBudget, -} from "../../lib/request-execution-budget"; +import { isRequestExecutionBudget } from "../../lib/request-execution-budget"; import type { OcxConfig } from "../../types"; import type { RequestLogContext } from "../request-log"; import type { HandleResponsesOptions, ResponsesDispatchers, ConsumedComboFailure } from "./core-options"; @@ -66,93 +58,8 @@ import { import { preflightComboStreamResponse } from "./combo-stream-preflight"; import { streamingContextOverflowResponse, jsonContextOverflowResponse } from "./context-overflow"; -/** - * Sends one combo target may run on its own before the ladder moves on. A target is a whole - * request as far as its own provider is concerned, so this is the guarded profile's base - * allowance rather than a separate number to keep in sync. - */ -export const COMBO_TARGET_BASE_SENDS = CODEX_TEXT_GUARDED_BUDGET_POLICY.baseSendAllowance; - - -/** - * A combo's execution policy is DECLARED by the combo, not inherited from the single-target - * profile. - * - * `maxTargetTransitions: 1` and `maxAlternateTargetSends: 1` describe an account move, and - * applying them to a combo would refuse the second hop of a three-target combo -- which is why - * combo was left off `reserveDispatch` when the per-request split landed. The transitions a - * combo may make are exactly the targets it declares minus the one it starts on. What stays - * capped is the TOTAL: the first target's full ladder, one send for every further declared - * target, and the one shared final-recovery reserve. A one-target combo reduces to the guarded - * profile exactly, and a three-target combo whose every target fails hard reaches upstream six - * times instead of the twelve #4546 measured. - */ -export function comboExecutionBudgetPolicy(declaredTargets: number): RequestExecutionBudgetPolicy { - const targets = Math.max(1, Math.trunc(declaredTargets)); - const hops = targets - 1; - const reserve = CODEX_TEXT_GUARDED_BUDGET_POLICY.finalRecoveryAllowance; - const total = COMBO_TARGET_BASE_SENDS + hops + reserve; - return { - maxTotalModelSends: total, - baseSendAllowance: total - reserve, - finalRecoveryAllowance: reserve, - maxAlternateTargetSends: Math.max(1, hops), - maxTargetTransitions: Math.max(1, hops), - }; -} - - -/** - * A budget scope that keeps its own recovery ledgers but spends the SAME request-wide counter. - * - * `used` is redefined as an accessor onto the parent because the factory reads it back off this - * object -- `remainingBaseSends` and the total check both do -- so a copied number would let a - * combo target run its ladder against a stale total, which is precisely the per-layer counting - * this work exists to remove. The reserve, alternate-target and transition ledgers stay - * per-scope on purpose: a combo target's account failover is its own recovery decision, while - * the request total still bounds every target together. - */ -export function deriveSendBudgetScope( - parent: RequestExecutionBudget, - policy: RequestExecutionBudgetPolicy, -): RequestExecutionBudget { - const scope = createRequestExecutionBudget(policy, parent.logicalRequestId); - Object.defineProperty(scope, "used", { - get: () => parent.used, - set: (value: number) => { parent.used = value; }, - enumerable: true, - configurable: true, - }); - return scope; -} - - -/** - * The ladder one combo target may run, expressed as an allowance on the request-wide counter. - * - * `used + COMBO_TARGET_BASE_SENDS` gives this target its own ladder from wherever the request - * already stands, and the clamp holds back one send for each target still declared after it: a - * first target that 5xx-streaks must not eat the send the last declared target is entitled to. - * That guarantee is the difference between a per-target policy and a shared pool the first - * target drains. - */ -export function comboTargetSendBudget( - comboScope: RequestExecutionBudget, - targetsDeclaredAfterThisOne: number, -): RequestExecutionBudget { - const policy = comboScope.policy; - const heldForLaterTargets = Math.max(0, targetsDeclaredAfterThisOne); - const ceiling = Math.max(1, policy.maxTotalModelSends - heldForLaterTargets); - return deriveSendBudgetScope(comboScope, { - maxTotalModelSends: policy.maxTotalModelSends, - baseSendAllowance: Math.min(ceiling, comboScope.used + COMBO_TARGET_BASE_SENDS), - finalRecoveryAllowance: policy.finalRecoveryAllowance, - // Within one target the account-move shape is unchanged: three same-account sends plus one - // alternate is the recovery live traffic depends on, and a combo does not widen it. - maxAlternateTargetSends: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxAlternateTargetSends, - maxTargetTransitions: CODEX_TEXT_GUARDED_BUDGET_POLICY.maxTargetTransitions, - }); -} +import { comboExecutionBudgetPolicy, deriveSendBudgetScope, comboTargetSendBudget } from "./combo-send-budget"; +export { COMBO_TARGET_BASE_SENDS, comboExecutionBudgetPolicy, deriveSendBudgetScope, comboTargetSendBudget } from "./combo-send-budget"; export async function executeComboResponses( @@ -391,7 +298,7 @@ export async function executeComboResponses( break; } const targetSendBudget = comboSendScope - ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched) + ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched, hopDecision?.allowed ? hopDecision.permit : undefined) : options.sendBudget; comboTargetsDispatched += 1; const childLog: RequestLogContext = { diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 1bef92779a..342eec4e63 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -111,7 +111,7 @@ async function handleResponsesInner( requestState, sidecarState, ); - const sendBudgetState = createResponsesSendBudget(requestContext); + const sendBudgetState = createResponsesSendBudget(requestContext, requestState); if (sendBudgetState instanceof Response) return sendBudgetState; if ("passthrough" in transportState.adapter && transportState.adapter.passthrough && !sidecarState.routedCompaction) { return await executePassthroughResponse( diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index c5879e106f..202e2992c6 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -1,4 +1,6 @@ import type { ResponsesRequestContext } from "./core-options"; +import type { PreparedResponsesRequest } from "./request-prepare"; +import { transientRetryPolicyFor } from "../../providers/key-failover"; import { createRequestExecutionBudget, isRequestExecutionBudget } from "../../lib/request-execution-budget"; import { chargeWorkflowSends, workflowSendCeilingReached } from "../../lib/workflow-budget"; import { workflowRefusalResponse } from "../workflow-refusal"; @@ -10,8 +12,16 @@ import type { SingleUseDispatchPermit, SendClass } from "../../lib/request-execu /** Owns the shared request send counter and recovery permits. */ export function createResponsesSendBudget( requestContext: Pick, + requestState: Pick, ) { const { options, req, logCtx } = requestContext; + const { route } = requestState; + // Capture before sidecar dispatch; every phase shares this rotation count. + const initialKeyPool = route.provider.apiKeyPool ?? []; + const initialKeyReference = route.provider._apiKeyAttempt?.reference ?? route.provider.apiKey; + const initialKeyIsPooled = initialKeyPool.some(entry => entry.key === initialKeyReference); + const maxKeyPoolFailovers = Math.max(0, initialKeyPool.length - (initialKeyIsPooled ? 1 : 0)); + let keyPoolFailovers = 0; // One transient-retry budget for the whole LOGICAL request, read ABOVE the passthrough branch @@ -142,7 +152,29 @@ export function createResponsesSendBudget( const recoveryClassFor = (recovery: AttemptRecoveryKind): SendClass => /401|429|oauth|rate-limit|key/.test(recovery) ? "auth-recovery" : "repair"; + const keyPool429RetryAllowed = (adapterOwnsSends: boolean): boolean => { + if (keyPoolFailovers >= maxKeyPoolFailovers) return false; + if (adapterOwnsSends) return !adapterSendBudget + || adapterSendBudget.remainingBaseSends(adapterSendBudget.policy.baseSendAllowance) > 0; + const policy = transientRetryPolicyFor(route.provider); + const attempts = policy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS; + if (!Number.isInteger(attempts) || attempts <= 0) return false; + if (remainingTransientSendBudget(attempts) > 0) return true; + if (!isRequestExecutionBudget(sendBudget)) return false; + const decision = sendBudget.reserveDispatch({ + sendClass: "auth-recovery", targetKey: `${route.providerName}|${route.modelId}|key-429`, + countedExternally: true, + }); + if (!decision.allowed) return false; + decision.permit.release(); + return true; + }; + return { + maxKeyPoolFailovers, + get keyPoolFailovers() { return keyPoolFailovers; }, + set keyPoolFailovers(value: number) { keyPoolFailovers = value; }, + keyPool429RetryAllowed, workflowRootId, noteTransientSends, remainingTransientSendBudget, diff --git a/src/server/responses/sidecar-execution.ts b/src/server/responses/sidecar-execution.ts index 7987d5ca93..36af7605e6 100644 --- a/src/server/responses/sidecar-execution.ts +++ b/src/server/responses/sidecar-execution.ts @@ -74,7 +74,7 @@ export async function executeResponsesSidecars( | "notifyResponseComplete" | "cancelResponseCompletion" >, - sendBudgetState: Pick, + sendBudgetState: Pick, ) { const { config, options, logCtx } = requestContext; const { @@ -159,8 +159,10 @@ export async function executeResponsesSidecars( now: Date.now(), attemptedKey: route.provider.apiKey, promptCacheKey: parsed.options.promptCacheKey, + allowRotation: sendBudgetState.keyPoolFailovers < sendBudgetState.maxKeyPoolFailovers, }); if (rotated) { + sendBudgetState.keyPoolFailovers += 1; route.provider = rotated; } else if ( // A POSITIVE gate, not an early return. An early `return null` here made every later arm diff --git a/structure/adapters/registry.md b/structure/adapters/registry.md index 83e8f4f466..dc39725b44 100644 --- a/structure/adapters/registry.md +++ b/structure/adapters/registry.md @@ -4,7 +4,7 @@ Request-local adapter bindings are separate from registry authority in the Respo [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. Response-attached WebSocket telemetry follows the [stage record identity contract](../transports/responses.md#passthrough-sse-stream-shapes-314). diff --git a/structure/catalog.md b/structure/catalog.md index 2fd03722df..0b5b4e872b 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -4,7 +4,7 @@ Catalog discovery remains separate from the Responses final-route [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Shared parsing and streaming follow the [request-copy](transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](transports/byte-accounting.md#stream-buffer-accounting) contracts. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). diff --git a/structure/clients/claude-desktop.md b/structure/clients/claude-desktop.md index 01a583c182..28cb0f6ed1 100644 --- a/structure/clients/claude-desktop.md +++ b/structure/clients/claude-desktop.md @@ -4,7 +4,7 @@ Desktop callers retain their existing ingress through the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Codex-native model discovery follows the [shared retirement policy](../catalog.md#shared-catalog). That projection does not migrate existing user-selected Desktop configuration or usage history. diff --git a/structure/clients/integrations.md b/structure/clients/integrations.md index eb4ace5975..fc84009944 100644 --- a/structure/clients/integrations.md +++ b/structure/clients/integrations.md @@ -1,6 +1,6 @@ # Client Integrations -Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. +Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](../transports/responses.md#bounded-api-key-429-rotation). The client-integration subsystem writes one generated OpenCodex provider contribution into a third-party client's existing config without taking ownership of the rest of that file. Its core diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index a7420072bf..fc15593bc2 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -4,7 +4,7 @@ Vision preprocessing and image/video/search execution use the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Hosted Responses image-tool eligibility uses the shared compatibility policy without a Codex Spark exception; standalone Images retain the separate relay contract below. See diff --git a/structure/data-planes/inbound-compat.md b/structure/data-planes/inbound-compat.md index 33ca8c76b0..811eb2971a 100644 --- a/structure/data-planes/inbound-compat.md +++ b/structure/data-planes/inbound-compat.md @@ -4,7 +4,7 @@ Compatibility callers retain the public Responses ingress described by the [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. ## Standalone file transcription diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 4628ca5e50..7709788aa7 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -4,7 +4,7 @@ The shared server request path follows the Responses [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Response-attached WebSocket telemetry follows the [stage record identity contract](transports/responses.md#passthrough-sse-stream-shapes-314). Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. ## Dashboard serving diff --git a/structure/ops/docs-and-release.md b/structure/ops/docs-and-release.md index 0c0ffe38d0..aec7c5323d 100644 --- a/structure/ops/docs-and-release.md +++ b/structure/ops/docs-and-release.md @@ -1,7 +1,7 @@ # Docs And Release The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](../transports/responses.md#bounded-api-key-429-rotation). Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts. diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index e28575b432..96f2e968d6 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -4,7 +4,7 @@ Service endpoints are unchanged by the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Service startup and restore use the [catalog retirement policy](../catalog.md#shared-catalog); retirement does not itself change service registration or user-selected model configuration. diff --git a/structure/overview.md b/structure/overview.md index 0151115adc..d80872ccc8 100644 --- a/structure/overview.md +++ b/structure/overview.md @@ -1,7 +1,7 @@ # Overview The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local. Nested compact/combo recovery follows the [prepaid send-budget contract](transports/responses.md#bounded-api-key-429-rotation). Shared parsing and streaming follow the [request-copy](transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](transports/byte-accounting.md#stream-buffer-accounting) contracts. diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index dc982c5639..64bec93299 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -4,7 +4,7 @@ xAI uses the same shared credential and delivery policies through the Responses [core module ownership](../transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](../transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Codex-native retirement is scoped to OpenAI catalog/quota evidence. Shared Responses handling retains xAI provider behavior; see diff --git a/structure/runtime.md b/structure/runtime.md index bd9ebbd561..f8d0044efb 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -4,7 +4,7 @@ Responses admission and finalization are composed through the [core module ownership](transports/responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Chat request serialization owns the destination-scoped [OpenCode Go instruction ordering](providers/chat-compat.md#opencode-go-chronological-instructions); diff --git a/structure/subagents.md b/structure/subagents.md index c0c92891f7..5e32fdf423 100644 --- a/structure/subagents.md +++ b/structure/subagents.md @@ -10,7 +10,7 @@ Encrypted-task and fallback request handling follow the Responses only explicit true on Responses ingress to the final canonical ChatGPT forward route activates it. A default top-level collaboration catalog is required. The compiler preserves caller objects, aliases the namespace and three message functions, and removes only their true encryption marker. -Declaration/reference collisions refuse the whole rewrite without changing the request. +Declaration/reference collisions refuse the whole rewrite without changing the request. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](transports/responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. `src/adapters/openai-responses.ts` returns request-local alias capabilities. The Responses core refreshes them after every request rebuild and restores JSON, SSE and WebSocket identities after diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 7f01dee197..218cc14ebd 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -5,7 +5,7 @@ Responses body-reader limits and lifetime handling follow the How opencodex measures request and stream bytes without allocating copies solely to count them. These contracts are shared by request parsing, SSE rewriting, the provider adapters and -the translator budget, which is why so many documents link here rather than restating them. Response-attached WebSocket telemetry follows the [stage record identity contract](responses.md#passthrough-sse-stream-shapes-314). +the translator budget, which is why so many documents link here rather than restating them. Response-attached WebSocket telemetry follows the [stage record identity contract](responses.md#passthrough-sse-stream-shapes-314). Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. ## Request-copy accounting diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index ca80373a8e..67540ee3b1 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -4,7 +4,7 @@ The existing Responses transport is divided by responsibility in the [core module ownership](responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. The Chat adapter's [OpenCode Go instruction ordering](../providers/chat-compat.md#opencode-go-chronological-instructions) changes translated message placement only; endpoint selection and transport stay with their existing owners. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 4a6a664cad..e02c1d5e6d 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -113,13 +113,7 @@ within their route; neither route falls through to the other. See ### Pre-dispatch API-key pool pick -Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries pick a -warm key before the first send (`selectProactiveApiKeyTransport` in -`src/providers/key-failover.ts`). The pick is inert unless that strategy is set and the -committed key is already cooling or missing from the pool: a healthy committed key, including -a manual selection, is left alone and the common path returns null without a config write. -`forgetApiKeyRotationCursor` drops the process-local round-robin cursor when the operator -edits the pool, so a later pick cannot second-guess that choice. +Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries pick a warm key before the first send (`selectProactiveApiKeyTransport` in `src/providers/key-failover.ts`). The pick is inert unless that strategy is set and the committed key is already cooling or missing from the pool: a healthy committed key, including a manual selection, is left alone and the common path returns null without a config write. `forgetApiKeyRotationCursor` drops the process-local round-robin cursor when the operator edits the pool, so a later pick cannot second-guess that choice. On the shared Responses path the assignment lands in `src/server/responses/core.ts` immediately before `resolveProviderTransport`. `route.provider` is copied into @@ -142,6 +136,11 @@ are not at risk on a stored row, because the config schema requires both. Reactive 429 rotation (`rotateProviderTransportOn429`) remains the recovery path after a send has already earned a throttle. +### Bounded API-key 429 rotation + +The request-owned budget in `src/server/responses/request-send-budget.ts` captures a failover allowance before sidecar or adapter dispatch: the initial pool size minus one when the initially selected key reference belongs to that pool, or the full initial pool size when it is outside the pool and shares that failover count across ordinary recovery, web-search/image/video bridge rounds and terminal continuations. Initialization precedes sidecar dispatch. Sidecars enforce this count only at their API-key rotation callback, retaining their own reset and same-target retry policies; the ordinary core physical-send budget is not newly applied to them, and OAuth/Anthropic account fallback remains available independently. Cooldown expiry and later pool growth do not refill it. This is a failover count, not a distinct-key guarantee or a new combo-wide budget; existing shared physical-send admission remains authoritative. Before cancelling a superseded response or choosing another key, the ordinary response path checks both limits. Generic reset-only initial fetches, key-rotation rebuilds and terminal continuations consume the same physical-send budget; opting out of transient-5xx retries does not bypass the child ceiling or enable 5xx retries. Initial and continuation recovery admission probes release their unused dispatch reservations immediately; the actual refetch owns the charged send permit. Generic OAuth hops replayed through the retry helpers mark their reservation as externally counted and pass it into the refetch, so its first physical send settles the booking once. An unused retry-helper hop is refunded on failure or abort. Terminal-continuation key recovery in `src/server/responses/adapter-continuation.ts` can draw the same single final reserve after the base allowance is spent. The quota-confirmed compact handoff in `src/server/responses/compact.ts` also transfers an exact prepaid reservation into its routed child; native three-send exhaustion still leaves the one allowed recovery send, without minting a second budget. An intermediate combo scope retains that exact booking and cannot enlarge its inherited total or recovery allowance; an already prepaid first send is preserved even when the remaining budget cannot fund later declared targets. Combo policy and child-scope construction live in `src/server/responses/combo-send-budget.ts`; their scopes derive through `src/lib/request-execution-budget.ts`, sharing both charged reservations and pending external-send settlement. Remaining-allowance checks and unused-permit refunds use that same ledger, while recovery and target-transition ledgers remain scope-local. Each child has a hard total ceiling that preserves one physical send for every later declared target, including when an earlier target uses recovery. An exact parent hop permit supplies the child's prepaid initial send; adapter-owned reservations and external send reports settle that booking once, rather than charging the hop and the physical send twice. Three plain-5xx targets use six physical sends, with the final combo hop eligible for recovery allowance. `tests/lib/execution-budget-permits.test.ts` checks interleaved scopes and refunds; `tests/responses/responses-send-budget-counts.test.ts` counts physical sends and verifies that every target is reached. On refusal, `src/providers/key-failover.ts` accepts `allowRotation: false` to record the proven failed key's cooldown through the existing selection-identity fence, without selecting, persisting or announcing an unattempted replacement. The original failure stays available to normal response handling, including an already-started streaming continuation. `tests/server/server-key-failover-e2e.test.ts` covers expiry, an initially unpooled key, actual web-search/image bridge dispatch, shared continuation limits and earlier send-budget exhaustion; `tests/adapters/key-failover.test.ts` verifies health-only recording, unchanged disk bytes and manual-selection ownership. + + ### Routed service-tier capability OpenAI-compatible service-tier support is resolved only after the final provider/model wire is diff --git a/structure/transports/streaming-health.md b/structure/transports/streaming-health.md index 682fbb2ca2..b60bffcba0 100644 --- a/structure/transports/streaming-health.md +++ b/structure/transports/streaming-health.md @@ -4,7 +4,7 @@ Native and translated delivery now have separate owners in the [core module ownership](responses.md#core-module-ownership). This surface retains its existing behavior. The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages) -is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. +is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](responses.md#bounded-api-key-429-rotation). Responses dispatch scopes share [reservation and settlement accounting](responses.md#bounded-api-key-429-rotation) while keeping recovery ledgers local; generic reset-only fetches and OAuth replays settle physical sends once; nested compact/combo scopes retain the prepaid recovery without enlarging its ceiling or charging twice. Codex WebSocket quota-family normalization remains generic; retired-model evidence is filtered by the [OpenAI quota owner](../providers/openai-tiers.md#public-provider-contract), not by diff --git a/tests/adapters/adapter-inner-send-budget.test.ts b/tests/adapters/adapter-inner-send-budget.test.ts index 09406cb506..2ba454295d 100644 --- a/tests/adapters/adapter-inner-send-budget.test.ts +++ b/tests/adapters/adapter-inner-send-budget.test.ts @@ -6,6 +6,7 @@ import type { CursorRunRequest, CursorServerMessage } from "../../src/adapters/c import type { CursorTransport } from "../../src/adapters/cursor/transport"; import { createRequestExecutionBudget, type RequestExecutionBudgetPolicy } from "../../src/lib/request-execution-budget"; import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; +import { comboExecutionBudgetPolicy, comboTargetSendBudget } from "../../src/server/responses/combo-send-budget"; /** * Adapters that retry INSIDE one adapter call are the layer a per-request cap cannot see from @@ -52,6 +53,23 @@ function alwaysResets(): { calls: number } { } describe("Kiro inner retries and the request send budget", () => { + test("combo hop bookings fund actual Kiro sends once and leave every target reachable", async () => { + const upstream = alwaysResets(); + const combo = createRequestExecutionBudget(comboExecutionBudgetPolicy(3)); + const counts: number[] = []; + for (let target = 0; target < 3; target++) { + const hop = combo.reserveDispatch({ sendClass: target === 0 ? "initial" : "combo-failover", targetKey: `target-${target}`, countedExternally: true }); + if (!hop.allowed) throw new Error(`target ${target} was starved`); + hop.permit.use(); + const child = comboTargetSendBudget(combo, 2 - target, hop.permit); + const before = upstream.calls; + await expect(fetchKiroWithRetry(kiroRequest, { timeoutMs: 5_000, sendBudget: child })).rejects.toBeDefined(); + counts.push(upstream.calls - before); + } + expect(counts).toEqual([3, 2, 1]); + expect(combo.used).toBe(upstream.calls); + expect(upstream.calls).toBe(6); + }); test("a context without a budget keeps the adapter's own reset ladder", async () => { const upstream = alwaysResets(); const observed: Array<{ ordinal: number; recovery?: string }> = []; diff --git a/tests/adapters/key-failover.test.ts b/tests/adapters/key-failover.test.ts index 561c1cefa9..757b1df82f 100644 --- a/tests/adapters/key-failover.test.ts +++ b/tests/adapters/key-failover.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtempSync, unlinkSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { createOpenAIChatAdapter } from "../../src/adapters/openai-chat"; @@ -95,6 +95,29 @@ describe("hasKeyPoolFailover", () => { }); describe("rotateKeyOn429", () => { + test.each([false, true])("health-only 429 preserves selection ownership and disk bytes (superseded=%s)", superseded => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); + const routed = routedProviderConfig("p", config.providers.p); + if (superseded) { + expect(setActiveProviderApiKey(config, "p", "k2")).toBe(true); + expect(setActiveProviderApiKey(config, "p", "k1")).toBe(true); + } + const before = readFileSync(getConfigPath(), "utf8"); + const events: unknown[] = []; + const unsubscribe = subscribeAccountSelections(event => { events.push(event); }); + const now = 10_000; + try { + expect(rotateProviderTransportOn429(config, "p", routed, { + allowRotation: false, attemptedKey: routed.apiKey, retryAfter: "2", now, + })).toBeNull(); + expect(readFileSync(getConfigPath(), "utf8")).toBe(before); + expect(loadConfig().providers.p.apiKey).toBe("key-alpha-000111222333"); + expect(events).toEqual([]); + expect(getKeyCooldownUntil("p", "k1", now)).toBe(superseded ? null : now + 2_000); + expect(getKeyCooldownUntil("p", "k2", now)).toBeNull(); + } finally { unsubscribe(); } + }); + test("an old attempt cannot overwrite a newer manual key selection or its ABA revision", () => { const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); const routed = routedProviderConfig("p", config.providers.p); diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 35fea3be44..52e31367f0 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -17,6 +17,7 @@ export const RESPONSES_CORE_MODULES = [ "core-auth.ts", "core-normalize.ts", "core-combo.ts", + "combo-send-budget.ts", "request-prepare.ts", "request-transport.ts", "request-sidecar-auth.ts", diff --git a/tests/lib/execution-budget-permits.test.ts b/tests/lib/execution-budget-permits.test.ts index 6aff76df69..e3cc28f558 100644 --- a/tests/lib/execution-budget-permits.test.ts +++ b/tests/lib/execution-budget-permits.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "bun:test"; import { readFileSync } from "node:fs"; +import { comboExecutionBudgetPolicy, comboTargetSendBudget, deriveSendBudgetScope } from "../../src/server/responses/combo-send-budget"; import { CODEX_TEXT_GUARDED_BUDGET_POLICY, createRequestExecutionBudget, @@ -126,6 +127,136 @@ describe("atomic dispatch permits", () => { }); }); +describe("combo scopes share reservation accounting", () => { + test("a prepaid compact handoff survives an intermediate scope without a second charge", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "handoff", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + const handoff = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + const combo = deriveSendBudgetScope(handoff, ONE_SEND_LEFT); + expect(combo.remainingBaseSends(3)).toBe(1); + const first = combo.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + expect(first.allowed).toBe(true); + if (!first.allowed) throw new Error("prepaid send lost"); + first.permit.use(); parent.used += 1; + expect(parent.used).toBe(1); + expect(handoff.reserveDispatch({ sendClass: "initial", targetKey: "duplicate" }).allowed).toBe(false); + }); + + test("a combo handoff keeps its prepaid recovery and cannot enlarge the compact ceiling", () => { + const parent = createRequestExecutionBudget(); parent.used = 3; + const hop = parent.reserveDispatch({ sendClass: "account-failover", targetKey: "handoff", countedExternally: true }); + if (!hop.allowed) throw new Error("expected recovery"); + const handoff = deriveSendBudgetScope(parent, { ...parent.policy, baseSendAllowance: 4, finalRecoveryAllowance: 0 }, hop.permit); + const combo = deriveSendBudgetScope(handoff, comboExecutionBudgetPolicy(3)); + expect(combo.policy.maxTotalModelSends).toBe(4); + const booked = combo.reserveDispatch({ sendClass: "initial", targetKey: "first", countedExternally: true }); + if (!booked.allowed) throw new Error("lost prepaid recovery"); + const target = comboTargetSendBudget(combo, 2, booked.permit); + expect(target.remainingBaseSends(3)).toBe(1); + const physical = target.reserveDispatch({ sendClass: "transient", targetKey: "first" }); + expect(physical.allowed).toBe(true); + expect(parent.used).toBe(4); + expect(combo.reserveDispatch({ sendClass: "combo-failover", targetKey: "second" }).allowed).toBe(false); + }); + + test("a child sees the last send reserved by its parent before dispatch", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const child = deriveSendBudgetScope(parent, ONE_SEND_LEFT); + const reserved = parent.reserveDispatch({ sendClass: "initial", targetKey: "parent" }); + expect(reserved.allowed).toBe(true); + expect(child.remainingBaseSends(5)).toBe(0); + expect(child.reserveDispatch({ sendClass: "initial", targetKey: "child" }).allowed).toBe(false); + }); + + test("child release refunds the shared booking and external reports settle it once", () => { + const parent = createRequestExecutionBudget(); + const child = deriveSendBudgetScope(parent, parent.policy); + const first = child.reserveDispatch({ sendClass: "initial", targetKey: "child", countedExternally: true }); + if (!first.allowed) throw new Error("expected first permit"); + expect(parent.used).toBe(1); + first.permit.release(); + expect(parent.used).toBe(0); + const sent = child.reserveDispatch({ sendClass: "initial", targetKey: "child", countedExternally: true }); + if (!sent.allowed) throw new Error("expected second permit"); + parent.used += 1; + expect(child.used).toBe(1); + sent.permit.release(); + expect(parent.used).toBe(1); + child.used += 1; + expect(parent.used).toBe(2); + }); + + test("three failed targets cannot each refill the request-wide ladder", () => { + const parent = createRequestExecutionBudget(); + const combo = deriveSendBudgetScope(parent, comboExecutionBudgetPolicy(3)); + let sends = 0; + const byTarget: number[] = []; + for (let target = 0; target < 3; target++) { + const scope = comboTargetSendBudget(combo, 2 - target); + const before = sends; + for (let attempt = 0; attempt < 8; attempt++) { + const decision = scope.reserveDispatch({ sendClass: attempt === 0 ? "initial" : "auth-recovery", targetKey: `target-${target}` }); + if (!decision.allowed) break; + expect(decision.permit.use()).toBe(true); + sends++; + } + byTarget.push(sends - before); + } + expect(byTarget).toEqual([4, 1, 1]); + expect(sends).toBe(combo.policy.maxTotalModelSends); + expect(parent.used).toBe(sends); + expect(combo.remainingBaseSends(100)).toBe(0); + }); + + test("target transition ledgers remain local to each scope", () => { + const parent = createRequestExecutionBudget(comboExecutionBudgetPolicy(4)); + for (const name of ["a", "b"]) { + const child = deriveSendBudgetScope(parent, parent.policy); + for (const key of [name, `${name}-alternate`]) { + const decision = child.reserveDispatch({ sendClass: "auth-recovery", targetKey: key }); + expect(decision.allowed).toBe(true); + if (decision.allowed) decision.permit.use(); + } + expect(child.targetTransitions).toBe(1); + } + expect(parent.targetTransitions).toBe(0); + expect(parent.used).toBe(4); + }); + + test("two external reporters cannot adopt the same prepaid hop", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + hop.permit.use(); + const first = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + const second = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + expect(first.reserveDispatch({ sendClass: "transient", targetKey: "a", countedExternally: true }).allowed).toBe(true); + expect(second.reserveDispatch({ sendClass: "transient", targetKey: "b", countedExternally: true }).allowed).toBe(false); + parent.used += 1; + expect(parent.used).toBe(1); + }); + + test("an adapter adopts its exact prepaid hop once and can return it before dispatch", () => { + const parent = createRequestExecutionBudget(ONE_SEND_LEFT); + const hop = parent.reserveDispatch({ sendClass: "initial", targetKey: "combo", countedExternally: true }); + if (!hop.allowed) throw new Error("expected hop"); + hop.permit.use(); + const child = deriveSendBudgetScope(parent, ONE_SEND_LEFT, hop.permit); + expect(child.remainingBaseSends(3)).toBe(1); + const first = child.reserveDispatch({ sendClass: "initial", targetKey: "adapter" }); + if (!first.allowed) throw new Error("expected prepaid initial send"); + expect(parent.used).toBe(1); + first.permit.release(); + expect(child.remainingBaseSends(3)).toBe(1); + const retry = child.reserveDispatch({ sendClass: "initial", targetKey: "adapter" }); + if (!retry.allowed) throw new Error("expected returned booking"); + expect(retry.permit.use()).toBe(true); + expect(parent.used).toBe(1); + expect(child.reserveDispatch({ sendClass: "transient", targetKey: "adapter" }).allowed).toBe(false); + }); +}); + describe("layer caps intersect the shared budget", () => { test("a roster credential hop walks within the shared total; a cross-pool move does not", () => { // The two classes answer different questions and must not be conflated. A credential @@ -249,8 +380,18 @@ describe("generic-OAuth hop reservations are handed back when no send happens", "adapter-recovery-oauth-429", "attemptOpaqueBlobRecovery", ); - expect(block).toContain('rebuildAndRefetch("oauth-account-429", () => { hop.permit?.use(); })'); - expect(block).toMatch(/if \("failed" in result\) \{[^}]*hop\.permit\?\.release\(\)/); + // The confirm callback is adapter-owned: when the adapter carries its own fetchResponse the + // dispatch boundary lives inside it, so the callback is handed down; otherwise the reservation + // is parked in pendingHopPermit for the retry helper to settle. Match the shape rather than an + // exact source substring, because an exact multi-line substring breaks on any reformatting in + // this block without the behaviour having changed. + expect(block).toMatch(/rebuildAndRefetch\(\s*"oauth-account-429",[\s\S]{0,240}?hop\.permit\?\.use\(\)/); + expect(block).toMatch(/pendingHopPermit = [^;]*fetchResponse \? undefined : hop\.permit/); + // The refund moved out of the failed arm and into a finally, so one idempotent release now + // covers the failed return, the success path and a throw. Assert that shape rather than a + // per-branch release that no longer exists. + expect(block).toMatch(/if \("failed" in result\) return result\.failed;/); + expect(block).toMatch(/finally \{[^}]*hop\.permit\?\.release\(\)/); expect(block).toMatch(refundsOnThrow); }); diff --git a/tests/lib/transient-budget-scope-source.test.ts b/tests/lib/transient-budget-scope-source.test.ts index 8bc9c8d0ea..26cb2f79ec 100644 --- a/tests/lib/transient-budget-scope-source.test.ts +++ b/tests/lib/transient-budget-scope-source.test.ts @@ -66,15 +66,15 @@ describe("transient send budget stays request-scoped", () => { // EVERY leg asks for the remainder now, including the adapter initial send. That one used // to pass the raw policy on the argument that nothing had been spent yet -- true for a first // turn, false for a combo child, which inherits the parent's holder and then took a fresh - // full allowance on its own first send. Five sites spell it directly; the two rebuild legs + // full allowance on its own first send. Five sites spell it directly; rebuild and recovery legs // go through recoverySendAllowance, which spends the base allowance first and only then // draws the single shared final-recovery reserve. expect(core.match(/attempts: remainingTransientSendBudget\(/g)).toHaveLength(5); - expect(core).toContain("attempts: remainingTransientSendBudget(transientPolicy.attempts)"); - expect(core).toContain("attempts: remainingTransientSendBudget(continuationTransientPolicy.attempts)"); + expect(core).toContain("attempts: remainingTransientSendBudget(transientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS)"); + expect(core).toContain("attempts: remainingTransientSendBudget(continuationCap)"); // The reserve path: an account move and a validated rebuild share ONE final send, so a // request cannot take both and reach five. - expect(core.match(/recoverySendAllowance\(/g)).toHaveLength(2); + expect(core.match(/recoverySendAllowance\(/g)).toHaveLength(3); expect(core).toContain("countedExternally: true"); // The passthrough legs have no adapter policy to draw from, so they name the helper's own // ceiling rather than re-spelling the number. @@ -141,7 +141,9 @@ describe("every dispatch path reports into the shared budget", () => { // fresh four. expect(compact).toContain("turnAdmissionLease, sendBudget,"); // The handoff child already inherited; both paths must keep doing so. - expect(compact).toContain("{ ...options, sendBudget }"); + expect(compact).toContain("{ ...options, sendBudget: handoffBudget }"); + expect(compact).toContain("sendBudget.deriveScope({"); + expect(compact).toContain("}, hop.permit)"); }); test("credential hops keep their roster cap AND reserve from the shared budget", () => { diff --git a/tests/responses/responses-compaction-routing.test.ts b/tests/responses/responses-compaction-routing.test.ts index c612aed502..78852e0665 100644 --- a/tests/responses/responses-compaction-routing.test.ts +++ b/tests/responses/responses-compaction-routing.test.ts @@ -1464,10 +1464,10 @@ describe("compact alternate-account attempt (#913)", () => { }); }); - test("a quota-blocked previous-model compact retries the same thread's successful routed handoff target (#2723)", async () => { + test.each([false, true])("a quota-blocked previous-model compact retries its routed handoff (combo: %s)", async combo => { await withPoolEnv("ocx-compact-routed-handoff-", async config => { config.providers.deepseek = { - adapter: "openai-chat", + adapter: combo ? "openai-responses" : "openai-chat", baseUrl: "https://api.deepseek.com", authMode: "key", apiKey: "deepseek-test-key", @@ -1481,6 +1481,7 @@ describe("compact alternate-account attempt (#913)", () => { models: ["gpt-5.6-sol"], }; const headers = { "x-codex-parent-thread-id": "compact-routed-handoff-thread" }; + config.combos = { summarizer: { strategy: "failover", targets: [{ provider: "deepseek", model: "deepseek-v4-flash" }] } }; const calls: Array<{ model: string; nativeCompact: boolean }> = []; globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { const url = typeof input === "string" @@ -1496,12 +1497,12 @@ describe("compact alternate-account attempt (#913)", () => { status: 502, }); } - return jsonResponse(completedPayload("DeepSeek handoff summary")); + return combo ? sseResponse([{ type: "response.completed", response: completedPayload("DeepSeek handoff summary") }]) + : jsonResponse(completedPayload("DeepSeek handoff summary")); }) as typeof fetch; - const manual = await handleResponsesCompact( compactionRequest( - baseCompactionBody({ model: "deepseek/deepseek-v4-flash" }), + baseCompactionBody({ model: combo ? "combo/summarizer" : "deepseek/deepseek-v4-flash" }), undefined, headers, ), @@ -1525,7 +1526,6 @@ describe("compact alternate-account attempt (#913)", () => { expect(calls.length).toBeGreaterThan(0); expect(calls.every(call => call.model === "gpt-5.6-sol" && call.nativeCompact)).toBe(true); calls.length = 0; - const logCtx: RequestLogContext = { model: "", provider: "" }; const automatic = await handleResponsesCompact( compactionRequest( @@ -1540,9 +1540,9 @@ describe("compact alternate-account attempt (#913)", () => { expect(automatic.status).toBe(200); const output = await automatic.json() as { output?: unknown[] }; expect(output.output?.length).toBeGreaterThan(0); - expect(logCtx.provider).toBe("deepseek"); + expect(logCtx.provider).toBe(combo ? "combo" : "deepseek"); expect(calls.at(-1)).toEqual({ model: "deepseek-v4-flash", nativeCompact: false }); - expect(calls.slice(0, -1).length).toBeGreaterThan(0); + expect(calls.slice(0, -1)).toHaveLength(3); // Native ladder + one handoff = four sends. expect(calls.slice(0, -1).every(call => ( call.model === "gpt-5.6-sol" && call.nativeCompact ))).toBe(true); diff --git a/tests/responses/responses-core-modules.test.ts b/tests/responses/responses-core-modules.test.ts index 2d94f02f98..b13733ed1b 100644 --- a/tests/responses/responses-core-modules.test.ts +++ b/tests/responses/responses-core-modules.test.ts @@ -118,7 +118,9 @@ function budgetOwner(sendBudget: TransientSendBudget) { req: new Request("http://localhost/v1/responses"), logCtx: { model: "test", provider: "test" }, options: { translatorBudget, sendBudget }, - }); + }, { route: { providerName: "test", modelId: "test", provider: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://budget-fixture.invalid/v1", + } } } as Parameters[1]); if (result instanceof Response) { translatorBudget.dispose(); throw new Error("Unexpected workflow refusal without a workflow root"); diff --git a/tests/responses/responses-send-budget-counts.test.ts b/tests/responses/responses-send-budget-counts.test.ts index 78f5a42856..b1eb4ff7b8 100644 --- a/tests/responses/responses-send-budget-counts.test.ts +++ b/tests/responses/responses-send-budget-counts.test.ts @@ -11,11 +11,10 @@ import type { OcxConfig } from "../../src/types"; * only assertion that catches a regression here is the exact number of times the proxy reached * upstream for one client turn. * - * These rows use a key-auth `openai-chat` provider with `transientRetryOn5xx` because that is the - * counted path: the generic adapter branch draws `attempts` from the request budget and reports - * every physical send back through `onSendsConsumed`, and `noteAttemptSend` records the same send - * on the attempt. An adapter without an opted-in transient policy keeps reset-only semantics and - * hops on the first 5xx, so it would pin a 1 for every shape and prove nothing. + * These 5xx rows opt into `transientRetryOn5xx` to exercise same-target retries. Both generic + * retry policies draw from the request budget and report physical sends through + * `onSendsConsumed`. Reset-only still hops on the first HTTP 5xx; its key-rotation and socket + * reset counts are covered by the server-key-failover end-to-end fixture instead. */ const originalFetch = globalThis.fetch; @@ -120,7 +119,7 @@ describe("upstream sends per logical request", () => { expect(sendCounts(logCtx)).toEqual([3]); }); - test("a three-target combo fan-out gives every declared target a send and totals six", async () => { + test("a three-target 5xx combo reaches every target within six physical sends", async () => { const upstream = alwaysFailing(502, "upstream busy"); const logCtx: RequestLogContext = { model: "", provider: "" }; @@ -133,10 +132,8 @@ describe("upstream sends per logical request", () => { // the later targets to zero. The first target runs its own ladder, each later target draws // what is left, and the clamp holds back one send for every target still declared, so the // last target is still reached. - // Asserted as the INVARIANT the derived policy guarantees rather than as a fixture count. - // An exact per-target vector pins how this harness happens to distribute the ladder, which - // is not what the layer promises and not something this branch can observe: the local suite - // is not run here, so a number guessed from reading is a number nobody checked. + // Count physical sends through the adapter; unit-only shared-counter assertions are not + // enough to show that retries and later targets consume the same reservation ledger. const bearers = upstream.authorizations; // Every declared target is still reached. Starving the last target is the failure mode that // sharing one counter WITHOUT a per-target policy produces. @@ -147,15 +144,10 @@ describe("upstream sends per logical request", () => { // Bounded by the derived total: the first target's ladder, one send per further declared // target, and the single shared final-recovery reserve. The measured regression in #4546 was // twelve, four per target, because each child drew a fresh full allowance. - // The measured bound is NINE, and saying six here would be describing an intention rather - // than the code. #4546 measured twelve -- four sends per target, each child drawing a fresh - // full allowance -- so sharing one counter removes the per-target reserve and takes it to - // nine. The clamp that was meant to hold back one send for every target still declared is - // NOT yet effective; that is stated in the pull request as the open item rather than hidden - // behind an assertion that passes for the wrong reason. - expect(bearers.length).toBeLessThanOrEqual(9); - expect(bearers.length).toBeLessThan(12); - expect(bearers.length).toBeGreaterThanOrEqual(3); + // The final combo hop can use the shared recovery allowance. A prepaid hop must settle + // against its first physical send rather than shrinking the next target's ladder twice. + expect(bearers).toHaveLength(6); + expect(totalSends(logCtx)).toBe(6); }); // REMOVED: "a 401 before the 5xx streak spends one of the same three sends". diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index 418ef993ca..8ac9f72877 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -1,9 +1,9 @@ -import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; import { mkdtempSync} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { loadConfig, saveConfig } from "../../src/config"; -import { clearKeyCooldowns, rotateKeyOn429 } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, getKeyCooldownUntil, rotateKeyOn429 } from "../../src/providers/key-failover"; import { deriveXaiConvId } from "../../src/providers/xai-transport"; import { clearReasoningReplayCacheForTests } from "../../src/responses/reasoning-replay-cache"; import { startServer } from "../../src/server"; @@ -15,6 +15,8 @@ import { resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTe import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; import { routedProviderConfig } from "../../src/router"; import type { OcxProviderTransport } from "../../src/providers/xai-transport"; +import { getAccountSet, saveCredential, setActiveAccount } from "../../src/oauth/store"; +import { clearGenericFailoverHealth } from "../../src/oauth/generic-account-failover"; let testDir = ""; let previousHome: string | undefined; @@ -43,6 +45,257 @@ afterEach(() => { }); describe("server 429 key failover (end-to-end)", () => { + test.each([false, true])("OAuth hops charge each reset-only refetch once (third succeeds: %s)", async succeeds => { + const originalFetch = globalThis.fetch; + const authorizations: string[] = []; + let server: ReturnType | undefined; + clearGenericFailoverHealth(); + try { + for (let i = 0; i < 3; i++) { + await saveCredential("nous", { access: `synthetic-oauth-${i}`, refresh: `synthetic-refresh-${i}`, + expires: Date.now() + 3_600_000, accountId: `fixture-account-${i}` }, { addAccount: true }); + } + await setActiveAccount("nous", getAccountSet("nous")!.accounts[0]!.id); + saveConfig({ port: 0, hostname: "127.0.0.1", defaultProvider: "nous", providers: { nous: { + adapter: "openai-chat", authMode: "oauth", baseUrl: "https://oauth-refetch.invalid/v1", models: ["test"], + } } } as OcxConfig); + server = startServer(0); + globalThis.fetch = (async (_input, init) => { + authorizations.push(new Headers(init?.headers).get("authorization") ?? ""); + if (succeeds && authorizations.length === 3) return Response.json({ id: "third-account", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "third account works" }, finish_reason: "stop" }] }); + return Response.json({ error: { message: `oauth-quota-${authorizations.length}`, type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": "30" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "nous/test", stream: false, input: "hello" }), + }); + const text = await response.text(); + expect(authorizations).toEqual(["Bearer synthetic-oauth-0", "Bearer synthetic-oauth-1", "Bearer synthetic-oauth-2"]); + expect(response.status).toBe(succeeds ? 200 : 429); + expect(text).toContain(succeeds ? "third account works" : "oauth-quota-3"); + } finally { + globalThis.fetch = originalFetch; + await server?.stop(true); + clearGenericFailoverHealth(); + } + }); + + test.each(["429", "5xx", "reset"] as const)("reset-only combo key rotations preserve later targets (%s)", async mode => { + const originalFetch = globalThis.fetch; + const counts = [0, 0, 0]; + const targets = counts.map((_, index) => ({ provider: `reset-key-t${index}`, model: "test" })); + const providers = Object.fromEntries(targets.map(({ provider }) => [provider, { + adapter: "openai-chat", authMode: "key", baseUrl: `https://${provider}.invalid/v1`, + apiKey: `synthetic-${provider}-0`, + apiKeyPool: Array.from({ length: 6 }, (_, key) => ({ id: `k${key}`, key: `synthetic-${provider}-${key}` })), + }])); + const config = { port: 0, hostname: "127.0.0.1", defaultProvider: targets[0]!.provider, + providers, combos: { fan: { strategy: "failover", targets } }, + } as OcxConfig; + let server: ReturnType | undefined; + try { + saveConfig(config); + server = startServer(0); + globalThis.fetch = (async (input) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + const index = targets.findIndex(target => url.hostname === `${target.provider}.invalid`); + if (index < 0) throw new Error("unexpected reset-only combo fixture request"); + counts[index]!++; + if (counts.reduce((a, b) => a + b, 0) > 18) throw new Error("fixture send ceiling exceeded"); + if (mode === "reset" && (index !== 0 || counts[index]! > 1)) { + throw Object.assign(new Error("socket reset fixture"), { code: "ECONNRESET" }); + } + return Response.json({ error: { message: "key quota exhausted", type: "rate_limit_error" } }, + { status: mode === "5xx" ? 502 : 429, headers: { "retry-after": "0" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.timeout(10_000), + body: JSON.stringify({ model: "combo/fan", input: "hello", stream: false }), + }); + await response.text(); + expect(response.ok).toBe(false); + expect(counts).toEqual(mode === "5xx" ? [1, 1, 1] : mode === "reset" ? [3, 2, 1] : [4, 1, 1]); + expect(counts.reduce((a, b) => a + b, 0)).toBe(mode === "5xx" ? 3 : 6); + } finally { globalThis.fetch = originalFetch; await server?.stop(true); } + }, 15_000); + + test.each(["web-search", "image"] as const)( + "%s bridge bounds rotations after short cooldowns expire", + async bridge => { + const originalFetch = globalThis.fetch; + const endpoint = "https://sidecar-key429-fixture.invalid/v1/chat/completions"; + const seen: Array<{ authorization: string | null; tools: string[] }> = []; + let now = Date.now(); + let restoreClock: (() => void) | undefined; + let server: ReturnType | undefined; + const config = { + port: 0, hostname: "127.0.0.1", defaultProvider: "sidecar429", + providers: { + sidecar429: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://sidecar-key429-fixture.invalid/v1", + apiKey: "synthetic-sidecar-a", apiKeyPool: [ + { id: "a", key: "synthetic-sidecar-a" }, { id: "b", key: "synthetic-sidecar-b" }, + ], + }, + // Arms image planning without OAuth or an actual image-service request. + ...(bridge === "image" ? { xai: { + adapter: "openai-chat", authMode: "key", baseUrl: "https://image-plan-fixture.invalid/v1", + apiKey: "synthetic-image-plan-token", + } } : {}), + }, + ...(bridge === "web-search" + ? { webSearchSidecar: { enabled: true, backend: "exa", exaApiKey: "synthetic-exa-plan-token" } } + : { images: { bridgeEnabled: true } }), + } as OcxConfig; + try { + saveConfig(config); + server = startServer(0); + const clock = spyOn(Date, "now").mockImplementation(() => now); + restoreClock = () => clock.mockRestore(); + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== endpoint) throw new Error("unexpected outbound request in sidecar 429 fixture"); + const body = JSON.parse(String(init?.body)) as { tools?: Array<{ function?: { name?: string } }> }; + seen.push({ authorization: new Headers(init?.headers).get("authorization"), + tools: (body.tools ?? []).map(tool => tool.function?.name ?? ""), + }); + // A broken rotation loop is bounded by six mocked sends, never by an infinite wait. + if (seen.length >= 6) throw new Error("sidecar 429 fixture send ceiling exceeded"); + now += 1_000; + return Response.json({ error: { message: `sidecar429-final-${seen.length}`, type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": "0" } }); + }) as typeof fetch; + const response = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.timeout(10_000), + body: JSON.stringify({ model: "sidecar429/test", input: "Use the provided tool.", stream: true, + tools: [{ type: bridge === "web-search" ? "web_search" : "image_generation" }], + }), + }); + const text = await response.text(); + expect(seen.map(call => call.authorization)).toEqual(["Bearer synthetic-sidecar-a", "Bearer synthetic-sidecar-b"]); + // Verifies bridge activation rather than accidentally exercising the generic 429 loop. + for (const call of seen) expect(call.tools).toContain(bridge === "web-search" ? "web_search" : "image_gen"); + expect(response.status).toBe(429); + expect(text).toContain("Provider error 429"); + expect(text).toContain("sidecar429-final-2"); + expect(getKeyCooldownUntil("sidecar429", "b", now)).toBe(now + 1); + } finally { + globalThis.fetch = originalFetch; + restoreClock?.(); + await server?.stop(true); + } + }, 15_000, + ); + + test.each(["exhausted", "continuation", "transient", "budget-exhausted", "recovery-success", "unpooled"] as const)( + "429 rotation stays request-bounded after every earlier cooldown expires (%s)", + async mode => { + const originalFetch = globalThis.fetch; + const endpoint = "https://key429-fixture.invalid/v1/chat/completions"; + const usesFinalReserve = mode === "budget-exhausted" || mode === "recovery-success"; + const expectedSends = mode === "exhausted" ? 2 : usesFinalReserve ? 4 : 3; + const seen: string[] = []; + const cancelled: number[] = []; + let now = Date.now(); + let restoreClock: (() => void) | undefined; + let server: ReturnType | undefined; + const config = { + port: 0, hostname: "127.0.0.1", defaultProvider: "key429fixture", + providers: { key429fixture: { + adapter: "openai-chat", baseUrl: "https://key429-fixture.invalid/v1", authMode: "key", + apiKey: mode === "unpooled" ? "synthetic-key-outside" : "synthetic-key-a", apiKeyPool: [ + { id: "a", key: "synthetic-key-a" }, { id: "b", key: "synthetic-key-b" }, + ...(usesFinalReserve ? [{ id: "c", key: "synthetic-key-c" }] : []), + ], + ...(mode === "continuation" ? { terminalContinuationGuard: true } : {}), + ...(mode === "transient" || usesFinalReserve ? { transientRetryOn5xx: { enabled: true, attempts: 3 } } : {}), + } }, + } as OcxConfig; + try { + saveConfig(config); + server = startServer(0); + const clock = spyOn(Date, "now").mockImplementation(() => now); + restoreClock = () => clock.mockRestore(); + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url !== endpoint) throw new Error("unexpected outbound request in key-rotation fixture"); + seen.push(new Headers(init?.headers).get("authorization") ?? ""); + // The old loop must terminate deterministically instead of waiting for a test timeout. + if (seen.length > 6) throw new Error("key-rotation fixture send ceiling exceeded"); + const send = seen.length; + // Retry-After: 0 means a 1ms cooldown. Every subsequent response arrives after it. + now += 1_000; + if (mode === "recovery-success" && send === 4) { + return Response.json({ id: "chatcmpl-final-recovery", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "recovered-with-final-send" }, finish_reason: "stop" }], + usage: { prompt_tokens: 2, completion_tokens: 2, total_tokens: 4 }, + }); + } + if (mode === "continuation" && send === 2) { + return Response.json({ id: "chatcmpl-plan", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "I will edit the file now." }, finish_reason: "stop" }], + usage: { prompt_tokens: 2, completion_tokens: 2, total_tokens: 4 }, + }); + } + const status = (mode === "transient" && send === 1) || (usesFinalReserve && send < 3) ? 503 : 429; + const text = JSON.stringify({ error: { message: `key429-final-${send}`, type: "rate_limit_error" } }); + const bytes = new TextEncoder().encode(text); + const body = new ReadableStream({ + start(controller) { controller.enqueue(bytes); controller.close(); }, + cancel() { cancelled.push(send); }, + }); + return new Response(body, { status, headers: { + "content-type": "application/json", "retry-after": "0", + } }); + }) as typeof fetch; + const result = await originalFetch(new URL("/v1/responses", server.url), { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "key429fixture/test", stream: false, + input: mode === "continuation" ? "Edit the file using the tool." : "hello", + ...(mode === "continuation" ? { tools: [{ type: "function", name: "edit_file", + description: "Synthetic edit tool; never executed", parameters: { type: "object", properties: {} }, + }] } : {}), + }), + }); + const text = await result.text(); + expect(seen).toHaveLength(expectedSends); + expect(seen).toEqual(mode === "unpooled" + ? ["Bearer synthetic-key-outside", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : usesFinalReserve + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : mode === "transient" + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-a", "Bearer synthetic-key-b"] + : mode === "continuation" + ? ["Bearer synthetic-key-a", "Bearer synthetic-key-b", "Bearer synthetic-key-b"] + : ["Bearer synthetic-key-a", "Bearer synthetic-key-b"]); + // Exhausted initial rotation must not cancel the final error it returns to the caller. + if (mode === "recovery-success") { + // The admission probe must not spend the reserve before the real B dispatch. + expect(result.status).toBe(200); + expect(text).toContain("recovered-with-final-send"); + } else if (mode !== "continuation") { + expect(result.status).toBe(429); + expect(result.headers.get("retry-after")).toBe("0"); + expect(text).toContain(`key429-final-${expectedSends}`); + expect(cancelled).not.toContain(expectedSends); + } else { + // A continuation error is represented inside the already-started Responses result. + expect(text).toContain("key429-final-3"); + } + expect(cancelled).toContain(usesFinalReserve ? 3 : mode === "transient" ? 2 : 1); + expect(getKeyCooldownUntil("key429fixture", "b", now)).toBe(mode === "recovery-success" ? null : now + 1); + } finally { + globalThis.fetch = originalFetch; + restoreClock?.(); + await server?.stop(true); + } + }, 15_000, + ); + test("physical key selection rejects disabled, removed, and changed-auth providers", () => { const provider = { adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first" } as const; const config = { providers: { current: { ...provider } } } as unknown as OcxConfig;