Skip to content

feat(openai): surface finish_reason on chat-completions streaming final responses - #2249

Open
nazq wants to merge 1 commit into
0xPlaygrounds:mainfrom
nazq:fix/streaming-stop-reason
Open

feat(openai): surface finish_reason on chat-completions streaming final responses#2249
nazq wants to merge 1 commit into
0xPlaygrounds:mainfrom
nazq:fix/streaming-stop-reason

Conversation

@nazq

@nazq nazq commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #2248.

Refs #2090, #2235.

The OpenAI Chat Completions streaming path parses each choice's finish_reason,
uses it once to decide when to flush pending tool calls, and then drops it. The
streaming StreamingCompletionResponse carried usage and nothing else, so a
max_tokens truncation was indistinguishable from a natural stop — the same gap
#2236 closes for Anthropic, and the one its notes explicitly left open for the
OpenAI-compatible side. The non-streaming response keeps the value
(choices[].finish_reason on the raw response); only streaming consumers lose it.

Reproduce with any streaming completion capped by max_tokens: the final
response reports normal-looking usage and no signal that the output was cut off.
A consumer that must refuse to act on a truncated response has nothing to branch
on without re-parsing provider SSE itself.

What this does

StreamingCompletionResponse gains the field:

pub struct StreamingCompletionResponse<U = Usage> {
    pub usage: U,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub finish_reason: Option<FinishReason>,   // added
}

FinishReason is the typed enum this module already deserializes from the wire
(ToolCalls, Stop, ContentFilter, Length, Other(String)); it now derives
Clone, Serialize, and Eq so it can live on the response.

Internally, the shared compatible-stream state machine
(providers/internal/openai_chat_completions_compatible.rs) previously
normalized finish reasons into CompatibleFinishReason { ToolCalls, Other },
which conflated "stopped for length", "stopped naturally", and "no finish reason
on this chunk". The enum now carries the full set, the choice field is
Option<CompatibleFinishReason> so absence is distinct from nonstandard, the
loop tracks the last streamed reason, and build_final_response receives it.

Because the state machine is shared, the fix lands in one place for:

  • OpenAI's chat-completions dialect (.completions_api()), and every provider
    driving completions through GenericCompletionModel (Azure, Groq, DeepSeek,
    Mistral, OpenRouter, Together, Moonshot, MiniMax, Z.ai, Hugging Face,
    Perplexity, and the other compatible providers);
  • Copilot chat, whose profile previously collapsed everything except
    tool_calls — it now surfaces the same typed value.

Two mapping details, both pinned by tests: the deprecated function_call wire
value keeps its existing tool-calls semantics and surfaces as
FinishReason::ToolCalls; nonstandard values some gateways emit (e.g.
DeepSeek's insufficient_system_resource) survive verbatim as
Other(String).

Design choice

This mirrors #1776, which added finish_reason to Gemini's streaming response
type, and #2236, which does the same for Anthropic: the provider-native typed
value on the provider's streaming final response. It deliberately does not
touch the generic CompletionResponse / streaming::StreamingCompletionResponse
types — #2090 tracks a normalized cross-provider FinishReason, and the variant
set there is still open. Once that lands, this field is exactly the input the
OpenAI-family mapping needs.

Alternatives considered:

  • Normalized enum on the generic response — rejected here as feat(core): expose a normalized finish reason / stop status on CompletionResponse #2090's design
    discussion; doing it per-provider first matches how Usage grew.
  • Raw Option<String> — rejected; the typed enum already exists on the wire
    path and Other(String) preserves fidelity for nonstandard values.
  • Capturing inside each profile — not possible; profiles are stateless by
    design, so the value has to flow through the shared loop.

Compatibility

Additive. The field is Option, skipped when None on serialization, and
Option fields default on deserialization, so payloads persisted before this
change round-trip unchanged, and payloads written by this change deserialize
under older readers that ignore unknown fields. The reshaped
CompatibleFinishReason and the build_final_response signature are
pub(crate). No public API is removed or changed in shape.

Tests

Unit tests in providers/openai/completion/streaming.rs:

  • stop / length / content_filter each survive to the final response, with
    usage intact;
  • tool_calls surfaces as ToolCalls; deprecated function_call normalizes to
    ToolCalls;
  • nonstandard insufficient_system_resource is preserved as Other;
  • a stream with no finish reason yields None;
  • serde round-trip: legacy payloads without the field deserialize, None keeps
    the serialized form unchanged, standard and nonstandard reasons round-trip.

Copilot: chat_stream_surfaces_finish_reason_on_final_response pins the typed
value through CopilotStreamingResponse::Chat.

Cassette: completions_api_stream_surfaces_length_finish_reason replays a
max_tokens-truncated chat-completions stream and asserts Length on the final
response. The fixture is authored in scrubbed cassette form (it passes the
cassette-safety scan and the registered-scenario check).

The tests are load-bearing, not incidentally green: reverting only the surface
(finish_reason: None at the final-response construction site) turns four of
them red.

Verified locally on the pinned toolchain:

  • cargo fmt --check
  • cargo clippy -p rig-core --all-targets --all-features — clean
  • cargo test -p rig-core --lib — 952 passed
  • cargo test -p rig --all-features --test openai — 130 passed
  • cargo test -p rig --all-features --test copilot — 67 passed
  • full cassette suites for groq, deepseek, mistral, openrouter, perplexity,
    doubleword, llamafile, chatgpt — all green

…al responses

The compatible-stream state machine parsed each choice's finish_reason,
used it to flush pending tool calls, and dropped it: the streaming final
response carried usage and nothing else, so a max_tokens truncation was
indistinguishable from a natural stop. The non-streaming response keeps
the value.

Track the last streamed finish reason in the shared loop and surface it
as StreamingCompletionResponse.finish_reason for OpenAI's chat
completions dialect, every provider riding GenericCompletionModel, and
Copilot chat -- which previously collapsed everything except tool_calls.
Nonstandard values survive as Other; the deprecated function_call value
is normalized to ToolCalls. The field skips serialization when None, so
persisted payloads round-trip unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(openai): chat-completions-compatible streaming drops finish_reason

1 participant