feat(core,runtime,desktop): add Fast mode for OpenAI Responses relays - #3309
Conversation
Astro-Han
left a comment
There was a problem hiding this comment.
COMMENT.
Splitting Fast out of #3300 is the right call, and the concern I raised there is genuinely addressed: service_tier: "fast" is now asserted against a captured Responses request body rather than an intermediate providerOptions object, and gpt-5-relay does satisfy the pinned SDK's priority-processing rule, so that capture proves what it claims. I re-derived it at 2a66ac7a6.
Three findings inline; the two P1s I think need to land before merge.
Two things above the line level:
-
This PR is not only Fast. It re-implements #3300's relay taxonomy wholesale —
isRelayProviderType,PROVIDER_DEFAULTS.relayModelProfiles, and the codec/UI admission ofopenai-responses-compatible— frommainrather than from #3300. Two open PRs each export the sameisRelayProviderTypefrom the same file: whichever lands first makes the other conflict, and neither is a single revertable intent any more. Stacking #3309 on #3300 leaves the taxonomy with exactly one owner. -
mergeable_stateisdirtyat this head, so the branch needs a rebase before any CI result on it means much.
AI disclosure: this review was assisted by Claude (Opus) for code search and cross-checking. Every finding below I re-derived myself against the source at 2a66ac7a6 and the pinned @ai-sdk/openai@4.0.42; nothing unverified is included.
| } | ||
| /> | ||
| </CapabilityRow> | ||
| <CapabilityRow label={copy.fastMode} description={copy.fastModeHelp}> |
There was a problem hiding this comment.
[P1] Gate the Fast row on the same condition the runtime uses, or name the constraint in fastModeHelp. This CapabilityRow renders for every id in capabilityModelIds whenever showsCapabilities is true — that is, for both relay provider types and for any model id — while buildFamilyWire only emits serviceTier when wire === 'openai-responses' and reasoningReplay.contract.adapter === 'openai' and supportsFastServiceTier(modelId) (packages/runtime/src/model-factory.ts:525-531). On an openai-compatible (Chat Completions) relay the selector appears, the choice is normalized, persisted into relayModelProfiles, and then silently dropped at request build; your own test asserts exactly that, buildProviderOptions(chat, 'fast-model') deep-equals {}. The PR body says Chat-compatible relays "do not receive a misleading option that the SDK would strip" — the runtime does not, but the settings UI does, and the help copy states no constraint, so the user sees a saved setting that never reaches the wire. Confirmed by reading the source at this head. Regression test needed: a renderer test asserting the Fast row is absent for an openai-compatible connection and for a Responses relay model id outside the supported families.
| rawEntry, | ||
| `relay model profile for ${modelId}`, | ||
| ['thinkingLevels', 'vision', 'contextWindow'], | ||
| ['thinkingLevels', 'vision', 'contextWindow', 'serviceTier'], |
There was a problem hiding this comment.
[P1] Bump RUNTIME_HOST_COMPATIBILITY_EPOCH for this new Runtime Policy field. RelayModelProfile crosses the Host protocol as ConnectionCatalogPageItem.relayProfile (packages/runtime-host/src/protocol/runtime-policy.ts), and this key list is the admission check on the Host's write path through normalizeConnectionCatalogEntryUpdateForProvider -> packages/storage/src/runtime-policy/connection-catalog-document.ts:194. exactRecord rejects unknown keys outright (packages/core/src/runtime-policy/domain-codec.ts:27-29), so a new Client paired with an epoch-27 Host — a pair the handshake still admits, because this PR leaves the epoch at 27 — fails the entire connection update with relay model profile for <id> contains unknown field 'serviceTier' the first time a user saves Fast, and the same decode makes an already-written document unreadable after a downgrade. This is the case epoch 27's own comment describes, "Older peers cannot safely preserve that field", and the precedent is direct: #2463 introduced this exact key list and bumped the epoch, #2565 bumped it for connection fields. Note that 28 is currently contested by #3236, #3199 and #3133, all taking 28 from a base at 27 — take 28 and re-check it at merge time. Confirmed by reading the source at this head. No new regression test is needed beyond the constant itself; runtime-policy-codec.test.ts already covers the key.
| } | ||
|
|
||
| /** The pinned OpenAI SDK strips fast for model ids outside these families. */ | ||
| function supportsFastServiceTier(modelId: string): boolean { |
There was a problem hiding this comment.
[P2] Mirror @ai-sdk/openai@4.0.42's supportsPriorityProcessing here instead of approximating it — this predicate diverges from it in both directions. The SDK's rule (node_modules/@ai-sdk/openai/dist/index.js:52) is modelId.startsWith('gpt-4') || gptVersion.major >= 5 && !isGptNanoModel && !isGptChatModel || oSeriesVersion >= 3, with gptVersion parsed by /^gpt-(\d+)(?:\.(\d+))?(?:-(.+))?$/. False positives: gpt-5-nano and gpt-5-chat-latest match this regex but the SDK excludes them, so service_tier is deleted at dist/index.js:6353 and the toggle becomes the silent no-op this gate exists to prevent. False negatives: gpt-4o, gpt-4o-mini, gpt-4.1, gpt-5.1 and gpt-5.1-codex all satisfy the SDK but fail gpt-4(?:-|$) / gpt-5(?:-|$), so Fast is refused for models that would have accepted it — gpt-5.1 most notably. getOpenAILanguageModelCapabilities is not exported from the package, so the predicate does have to be restated by hand, which is what makes pinning it worthwhile. Confirmed by reading the pinned SDK in node_modules. Regression test needed: a table over gpt-4o, gpt-4.1, gpt-5, gpt-5.1, gpt-5-nano, gpt-5-chat-latest, o3-mini, o4-mini and a plain relay id, asserting the emitted providerOptions for each.
…es relays Generated-by: Codex
Generated-by: Codex
2a66ac7 to
0854f7c
Compare
|
Addressed the review findings in 0854f7c:
Verified core/runtime/runtime-host builds, focused Fast tests (5 passed), Biome on changed files, and |
Astro-Han
left a comment
There was a problem hiding this comment.
APPROVE. Both P1s from my last review are fixed, and fixed at the right level.
- The UI/runtime split.
supportsRelayFastServiceTieris now one exported predicate that both sides call —provider-connection-detail.tsxgates the wholeCapabilityRowonshowsFastMode, andmodel-factory.ts:530gates the wire on the same function with the same arguments. That is stronger than what I asked for: the previous shape had two independent conditions that could drift, and now there is one, so a saved Fast declaration cannot exist without a path to the wire. - The epoch.
RUNTIME_HOST_COMPATIBILITY_EPOCHis 28 with a comment that names the actual reason ("Relay model profiles carry the Fast service-tier declaration"). That closes the mixed-version hole where a new Client and an epoch-27 Host would shake hands and then fail the first save with an unknown-field error. - The predicate itself. It now mirrors
@ai-sdk/openai@4.0.42'ssupportsPriorityProcessingrather than approximating it —gpt-5-nanoandgpt-5-chat-latestare correctly excluded, andgpt-4o,gpt-4.1andgpt-5.1are correctly included. I checked each against the pinned SDK. One narrow residual is inline as a P3.
Two things to flag that are not findings against this PR:
Epoch 28 is now contested by five open PRs — #2521, #2641, #3236, #3322 and this one — against a main at 27. The = 28 line merges cleanly because every branch writes identical text and each PR's own lower-bound assertion still passes, so whichever lands second silently shares an epoch with the first and the mixed-version guarantee is lost without any conflict to warn you. This needs a human check of the constant at merge time, every time, until #3313 removes the class.
The taxonomy overlap with #3300 is resolved from my side. This head no longer re-implements it in a way that would conflict — isRelayProviderType and the relayModelProfiles registry flag are still here, so whichever of the two lands first, the other will need a rebase, but that is ordinary and no longer a competing definition.
AI disclosure: this review was assisted by Claude (Opus) for code search and cross-checking. Everything above I re-derived myself against the source at 0854f7c48 and the pinned @ai-sdk/openai@4.0.42 in node_modules.
| const gptMajor = gptMatch?.[1] === undefined ? undefined : Number(gptMatch[1]); | ||
| const gptVariant = gptMatch?.[3]; | ||
| const isGptNanoModel = gptVariant?.startsWith('nano') ?? false; | ||
| const isGptChatModel = gptVariant?.startsWith('chat') ?? false; |
There was a problem hiding this comment.
[P3] Add the minor == null condition the SDK's isGptChatModel carries, or this gate is narrower than the rule it mirrors. @ai-sdk/openai@4.0.42 computes it as gptVersion.minor == null && variant.startsWith('chat') (dist/index.js:49) — a chat variant only disqualifies a model that has no minor version. This line drops the minor check, so gpt-5.1-chat is refused Fast here while the SDK would accept it and send service_tier. That is a false negative on one hypothetical id rather than the silent-no-op class the earlier finding was about, so it costs a capability rather than misleading anyone, and I would not hold the merge for it. Everything else in this function matches the SDK verbatim, including startsWith('gpt-4'), the /^gpt-(\d+)(?:\.(\d+))?(?:-(.+))?$/ parse and the /^o(\d+)(?:-|$)/ o-series rule. Confirmed by reading the pinned SDK and the source at this head. Regression test needed: add gpt-5.1-chat to whatever table pins this predicate.
Summary
Adds an optional Fast mode to custom OpenAI relay profiles. It maps to OpenAI
service_tier: "fast"on the Responses wire.Fast is deliberately limited to the OpenAI Responses adapter and OpenAI-named model families that the pinned SDK recognizes for priority processing. Arbitrary relay model IDs and OpenAI Chat-compatible relays do not receive a misleading option that the SDK would strip.
Refs #3300
Verification
service_tier: "fast"packages/uitype driftAI use
Tool(s) and scope: Codex performed codebase research, implementation, test authoring, and verification.
Checklist
Does this PR entail a change in behavior?