feat(openai): surface finish_reason on chat-completions streaming final responses - #2249
Open
nazq wants to merge 1 commit into
Open
feat(openai): surface finish_reason on chat-completions streaming final responses#2249nazq wants to merge 1 commit into
nazq wants to merge 1 commit into
Conversation
…al responses The compatible-stream state machine parsed each choice's finish_reason, used it to flush pending tool calls, and dropped it: the streaming final response carried usage and nothing else, so a max_tokens truncation was indistinguishable from a natural stop. The non-streaming response keeps the value. Track the last streamed finish reason in the shared loop and surface it as StreamingCompletionResponse.finish_reason for OpenAI's chat completions dialect, every provider riding GenericCompletionModel, and Copilot chat -- which previously collapsed everything except tool_calls. Nonstandard values survive as Other; the deprecated function_call value is normalized to ToolCalls. The field skips serialization when None, so persisted payloads round-trip unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2248.
Refs #2090, #2235.
The OpenAI Chat Completions streaming path parses each choice's
finish_reason,uses it once to decide when to flush pending tool calls, and then drops it. The
streaming
StreamingCompletionResponsecarriedusageand nothing else, so amax_tokenstruncation was indistinguishable from a natural stop — the same gap#2236 closes for Anthropic, and the one its notes explicitly left open for the
OpenAI-compatible side. The non-streaming response keeps the value
(
choices[].finish_reasonon the raw response); only streaming consumers lose it.Reproduce with any streaming completion capped by
max_tokens: the finalresponse reports normal-looking usage and no signal that the output was cut off.
A consumer that must refuse to act on a truncated response has nothing to branch
on without re-parsing provider SSE itself.
What this does
StreamingCompletionResponsegains the field:FinishReasonis the typed enum this module already deserializes from the wire(
ToolCalls,Stop,ContentFilter,Length,Other(String)); it now derivesClone,Serialize, andEqso it can live on the response.Internally, the shared compatible-stream state machine
(
providers/internal/openai_chat_completions_compatible.rs) previouslynormalized finish reasons into
CompatibleFinishReason { ToolCalls, Other },which conflated "stopped for length", "stopped naturally", and "no finish reason
on this chunk". The enum now carries the full set, the choice field is
Option<CompatibleFinishReason>so absence is distinct from nonstandard, theloop tracks the last streamed reason, and
build_final_responsereceives it.Because the state machine is shared, the fix lands in one place for:
.completions_api()), and every providerdriving completions through
GenericCompletionModel(Azure, Groq, DeepSeek,Mistral, OpenRouter, Together, Moonshot, MiniMax, Z.ai, Hugging Face,
Perplexity, and the other compatible providers);
tool_calls— it now surfaces the same typed value.Two mapping details, both pinned by tests: the deprecated
function_callwirevalue keeps its existing tool-calls semantics and surfaces as
FinishReason::ToolCalls; nonstandard values some gateways emit (e.g.DeepSeek's
insufficient_system_resource) survive verbatim asOther(String).Design choice
This mirrors #1776, which added
finish_reasonto Gemini's streaming responsetype, and #2236, which does the same for Anthropic: the provider-native typed
value on the provider's streaming final response. It deliberately does not
touch the generic
CompletionResponse/streaming::StreamingCompletionResponsetypes — #2090 tracks a normalized cross-provider
FinishReason, and the variantset there is still open. Once that lands, this field is exactly the input the
OpenAI-family mapping needs.
Alternatives considered:
discussion; doing it per-provider first matches how
Usagegrew.Option<String>— rejected; the typed enum already exists on the wirepath and
Other(String)preserves fidelity for nonstandard values.design, so the value has to flow through the shared loop.
Compatibility
Additive. The field is
Option, skipped whenNoneon serialization, andOptionfields default on deserialization, so payloads persisted before thischange round-trip unchanged, and payloads written by this change deserialize
under older readers that ignore unknown fields. The reshaped
CompatibleFinishReasonand thebuild_final_responsesignature arepub(crate). No public API is removed or changed in shape.Tests
Unit tests in
providers/openai/completion/streaming.rs:stop/length/content_filtereach survive to the final response, withusage intact;
tool_callssurfaces asToolCalls; deprecatedfunction_callnormalizes toToolCalls;insufficient_system_resourceis preserved asOther;None;Nonekeepsthe serialized form unchanged, standard and nonstandard reasons round-trip.
Copilot:
chat_stream_surfaces_finish_reason_on_final_responsepins the typedvalue through
CopilotStreamingResponse::Chat.Cassette:
completions_api_stream_surfaces_length_finish_reasonreplays amax_tokens-truncated chat-completions stream and assertsLengthon the finalresponse. The fixture is authored in scrubbed cassette form (it passes the
cassette-safety scan and the registered-scenario check).
The tests are load-bearing, not incidentally green: reverting only the surface
(
finish_reason: Noneat the final-response construction site) turns four ofthem red.
Verified locally on the pinned toolchain:
cargo fmt --checkcargo clippy -p rig-core --all-targets --all-features— cleancargo test -p rig-core --lib— 952 passedcargo test -p rig --all-features --test openai— 130 passedcargo test -p rig --all-features --test copilot— 67 passeddoubleword, llamafile, chatgpt — all green