Skip to content

fix(providers): fail an OpenAI-compatible stream on an in-band error payload - #1056

Open
tomastola wants to merge 2 commits into
mainfrom
ROCMAI-700-fix-inband-stream-error
Open

tomastola wants to merge 2 commits into
mainfrom
ROCMAI-700-fix-inband-stream-error

Conversation

@tomastola

Copy link
Copy Markdown
Collaborator

Why

vLLM and other OpenAI-compatible servers report a failure that happens mid-generation in-band: data: {"error":{...}} (older releases send a bare {"object":"error",...}), followed by data: [DONE]. parse_openai_sse_line skipped the error line, so the stream ended as a clean finish that held only the text generated before the failure.

In this tree rocm chat and the dash chat use non-streaming requests. The only streaming consumer of this parser is the serve post-launch smoke test, which treats any stream error as empty metrics. Before this change it computed throughput and TTFT from the truncated text; now it reports empty metrics like any other stream failure. No user-facing CLI surface streams, so no e2e scenario is added (AGENTS.md section 3 allows that with this explanation).

What changes

The parser now returns an error, OpenAI-compatible provider stream failed: <server message> (with a placeholder when the server gave no message), mirroring the Anthropic parser. Callers already propagate it with ?, so text delivered before the failure still reaches the callback and no clean done event is emitted. A null error field is not an error.

docs/testing.md: its known-gap example pointed at this gap, so it now names the still-open #522 one instead.

Verification

  • The existing ignored reproducer in_band_stream_errors_are_not_reported_as_a_clean_finish is un-ignored and strengthened: it asserts the exact error message, that the earlier text was delivered, and that no done event was emitted.
  • Seven new fixtures in the split-invariance sweep (nested error, bare error, no [DONE], null error, string error, empty-error placeholder, non-object payload), plus a test that lines after an error are not read.
  • With the guard disabled, 3 of 7 sse_emitter tests fail; with it, they pass.
  • Run: fmt, clippy -D warnings, module-size, cargo xtask smoke, and the proptests at PROPTEST_CASES=10000 (43 passed).

Not verified:

  • The error shapes are written from vLLM's handler, not captured from a live server.
  • The full workspace test run had 2 failures in files this PR does not touch, and was not compared against a clean main. runtimes_cmd::tests::concurrent_active_runtime_marker_writes_keep_a_complete_marker_published found the marker file unreadable during a concurrent rewrite. storage::tests::prune_removes_the_install_it_named_and_not_a_case_twin_of_it looks like it assumes a case-sensitive filesystem and was run on virtiofs.
  • lychee and hawkeye are not installed in the dev container, so the docs-link and licence-header hooks ran only in CI.

Scope

Not done here, as a separate follow-up: the engine's error message is printed unsanitised (the Anthropic path is the same), which relates to #983.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (None found: the search matched nothing.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (N/A, no new subcommand or subsystem.)
  • If this PR adds, renames or removes a production source file, docs/architecture.md has the matching entry. (N/A, no new files.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, not only the wording, per AGENTS.md §3. (The reproducer asserts the exact error text and that no done event was emitted.)

Fixes #516

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

…payload

vLLM and other OpenAI-compatible servers report a mid-generation failure
in-band as `data: {"error":{...}}` (older releases as a bare
`{"object":"error",...}`), followed by `data: [DONE]`.
`parse_openai_sse_line` read only `choices[..].content`, skipped the error
line, and so the stream ended cleanly with the truncated text.

It now returns an error carrying the server's message, as the Anthropic
parser already does. Text delivered before the failure still reaches the
callback, and a null `error` is not an error.

The reproducer test for this gap is un-ignored and fixtures cover the
error shapes; docs/testing.md's known-gap example now points at a gap
that is still open. The error shapes were written from vLLM's handler,
not captured from a live server.

Fixes #516

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the ROCMAI-700-fix-inband-stream-error branch from 18cee21 to 088e75f Compare November 17, 2026 10:33
@tomastola
tomastola marked this pull request as ready for review November 17, 2026 13:07
@tomastola
tomastola requested a review from a team as a code owner November 17, 2026 13:07
@tomastola

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (head 088e75f). All checks except E2E tests (MI300X, current) have passed or are still running; that lane is still queued behind the runner backlog. Marking ready for review in the meantime.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The in-band error payload now fails the stream before any further line is read, and the new tests pin both the exact message and the absence of a done event. CI is green on all required checks; only the GPU lane is pending.

Non-blocking notes:

  • {"error":{},"message":"x"} prints the placeholder rather than "x" (inline comment).
  • A non-string error.message silently becomes the placeholder.
  • The proptests module doc is rewrapped awkwardly (cosmetic).
  • The server error text is printed unsanitised; deferring that to #983 is fine.

let error = payload.get("error").filter(|error| !error.is_null());
let message = error
.and_then(|error| error.get("message"))
.or(error)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: when error is an object with no string message, .or(error) wins over the top-level message fallback on the next line, so {"error":{},"message":"x"} prints the placeholder. Only matters for odd servers; consider .or(error.filter(|e| e.is_string())).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed in e454b39, which only carries the doc fixes below; this nit is left open for you to resolve or say you want it fixed here. For {"error":{},"message":"x"} the placeholder is what prints today; the stream still fails either way.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I traced the fix itself end-to-end (both the local-provider path through local_http.rs's read loop and the remote path through stream_json_with_headers): the ? on push_bytes in both readers means the new bail! in parse_openai_sse_line genuinely stops the stream before any later line is read, in every case the new fixtures exercise (nested/bare error object, string error, null-error passthrough, non-object payload, no-[DONE] tail). The split-invariance sweep exercising all seven new OPENAI_FIXTURES entries at every one/two-byte split is solid coverage, and I agree with the existing approval's read of the message-precedence and sanitization nits — not repeating those.

Blocking

This fix flips a documented "not detected" behavior to "detected", and the change doesn't update the three places that document the old behavior, so the tree now contains explicit, incorrect claims about current behavior (a requirement this repo's own AGENTS.md §5 states directly: "when a statement such as 'X is not supported' or 'there is no Y' becomes false, grep the same surfaces for the old claim before pushing").

The PR body correctly notes the only real streaming consumer of this parser is run_smoke_test's post-launch probe (serve_summary.rs, provider "local" → local_http.rs → OpenAiSseEmitter → parse_openai_sse_line). Before this change, an in-band error line was skipped, the stream still finished cleanly, and provider_stream_chat_with_callback returned Ok, so the smoke test was SmokeOutcome::Measured. After this change, the same line makes parse_openai_sse_line return Err, which propagates unmodified through stream_chat_with_callback and provider_stream_chat_with_callback (no catch in between), so run_smoke_test now takes the Err branch and returns SmokeOutcome::Failed instead. That's an improvement in what gets detected, but three existing comments/docs still assert the pre-fix behavior:

  • apps/rocm/src/serve_summary.rs:90 (doc on SmokeOutcome::Measured): "Not detected here, and so Measured: an error reported inside a 200 stream as a data: payload, ..."
  • apps/rocm/src/serve_summary.rs:382 (doc on run_smoke_test): "Also Measured, and so not detected: an error data: payload inside a 200 stream, ..."
  • docs/testing.md:4401-4408: lists an in-band data: error as "Not detected (no failure is reported and the status reads ready)" and separately as a case whose tokens "are measured."

All three now describe the opposite of what the code does post-fix. None of them are touched by this diff (the only docs/testing.md edit is the unrelated known-gap example swap at line 668), so this isn't something the existing approval would have seen by reading the diff alone. Fix: update all three to say the in-band case is now detected/Failed, or fold them into one blocking finding's worth of edits — whichever reads better in context.

Confidence 85 — traced the full propagation path (parse_openai_sse_line → push_bytes → stream_json_from_local_endpoint/stream_json_with_headers → stream_chat_with_callback → provider_stream_chat_with_callback → run_smoke_test) against the current code, and confirmed no test pins the stale claim either way.

CI

Re-checked at head 088e75f3: all required checks are green; only E2E tests (MI300X, current) is still in_progress (queued behind the runner backlog per the author's own comment), nothing failing.

🤖 Automated by an AI agent

run_smoke_test reports an error sent as a `data:` payload inside a 200
stream as SmokeOutcome::Failed since the previous commit. Update the
SmokeOutcome and run_smoke_test doc comments, the Serve Smoke Test
Failure section of docs/testing.md and the matching bullet in
docs/manual-testing.md, which still listed that case as not detected,
and pin the outcome with a smoke-test test against a mock engine.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola requested a review from juhovainio November 17, 2026 14:47

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The in-band error fix looks right to me: the stream now fails on an error payload, the reproducer is un-ignored and asserts the exact message, the delivered text and no done event, and the new fixtures plus the lines-after-error test cover the edges. Two small, non-blocking notes inline.

Beyond those:

  • The server's error text goes into the summary unsanitised. The PR already flags this and ties it to #983, so I'm fine leaving it there.
  • Failing the stream rather than keeping partial text is the product call #516 leaves open. It mirrors the Anthropic parser, and text already delivered stays delivered, which seems reasonable.
  • No Gherkin scenario, justified because no user-facing CLI surface streams. The serve smoke-test summary change is arguably user-observable, but the socket-mock test asserts Failed plus the message, so I'm OK with that.

if has_error {
let error = payload.get("error").filter(|error| !error.is_null());
let message = error
.and_then(|error| error.get("message"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge case: .or(error) short-circuits when error is an object without a string message, so a sibling top-level "message" is never used and the placeholder is shown. Consider as_str-filtering each candidate before chaining. Non-blocking.

Comment thread docs/testing.md
that follows content tokens. The reason shown is the server's own message.
- Not detected (no failure is reported and the status reads `ready`): a `200`
answer with no `data:` line at all (HTML, plain JSON, an empty body); a
close-delimited stream cut after a token; and a complete stream with no `content` token, which is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is much longer than its neighbours after the edit; consider rewrapping for diff-friendliness. Non-blocking.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change. Fixes #516 (open issue; the only PR linked to it): the issue explicitly leaves "discard vs. warn vs. keep silently" as an open product call and asks only that the defect become visible — this PR resolves that narrower ask (fail the stream, keep text already delivered, mirror the Anthropic parser's behavior) without resolving the wider product question, which is consistent with the issue's own framing. No other ticket named.

Blocking

None

Non-blocking

  • Top-level message fallback is dead for most of the shapes it should cover — apps/rocm/src/providers/wire.rs:471-477
    .and_then(|error| error.get("message")).or(error).or_else(|| payload.get("message")) resolves at the .or(error) step to the original error value (object/array, not a string) whenever error is present but has no message key or isn't itself a string — which is Some, so the top-level-message fallback never runs. .and_then(as_str) then discovers it isn't a string and falls to the placeholder. {"error":{"code":500},"message":"CUDA OOM"} bails with the generic placeholder instead of "CUDA OOM". No fixture exercises this combination.
    Confidence 85 · logic · Fix: resolve each tier to &str before chaining (see inline comment).
  • has_error treats any non-null error value as a failure, including falsy scalars — apps/rocm/src/providers/wire.rs:467-468
    payload.get("error").is_some_and(|error| !error.is_null()) is true for false, 0, "", [], {} — not just the object/string shapes vLLM is documented as sending. Plausible but unconfirmed against a real server; the PR itself notes these shapes weren't captured live.
    Confidence 70 · logic · Fix: narrow to the documented shapes, or add a fixture pinning the broader behavior as intentional.
  • Unwrapped line breaks the file's prose-wrap convention — docs/testing.md:4290
    99 characters vs. ~75-82 elsewhere in the same paragraph; leftover from moving a clause up to the "Detected" bullet.
    Confidence 100 · mechanical · Fix: rewrap the bullet.

Decisions for the author

  • Failing the whole stream vs. keeping partial text with a warning — tradeoff
    The PR fails the stream entirely on an in-band error (text already delivered via the callback is kept; no further line, including [DONE], is read). Issue #516 explicitly leaves this choice open as a product decision; this PR's answer mirrors the existing Anthropic-parser behavior. Worth the author's explicit confirmation this is the intended default, not merely the simplest option.
  • Engine error text reaches the smoke-test summary unsanitised — tradeoff
    Already tracked under #983 and explicitly deferred there in the PR body; the Anthropic path has the same property. Reasonable to leave as is.

Positive signals

  • The fix is exercised at three independent levels: a hand-computed fixture table, an exhaustive one/two-byte split-invariance sweep (so chunk boundaries can't hide the new branch), and a full integration test (serve_smoke_test_fails_on_an_in_band_error_payload_in_a_200_stream) driving the real socket/chunked-transfer/run_smoke_test path end to end.
  • The un-ignored in_band_stream_errors_are_not_reported_as_a_clean_finish pins resulting state, not just wording: exact error string, preserved partial content, and !events.iter().any(|e| e.done) — it would fail if the fix were reverted.
  • The three doc/comment locations that a prior review round found describing the pre-fix "not detected" behavior (serve_summary.rs's two doc comments, docs/testing.md's detected/not-detected list) are now all updated and consistent with each other and with the actual code, and docs/manual-testing.md was updated too even though no one had flagged it.

Deployment notes

None. This only changes the internal post-launch smoke-test outcome (Measured → Failed) for an in-band stream error; no public CLI surface streams today, so there's no user-facing behavior change and no migration.

What this covered

Read the full diff (origin/main 2583a97f...pr-1056 e454b394168619c890eea6db68f290835e181e3c) across all 6 changed files, via two independent fan-out passes grouped by coupling — (1) apps/rocm/src/providers/wire.rs, apps/rocm/src/providers/wire/sse_emitter_proptests.rs, and the new test in apps/rocm/src/providers.rs, each read completely with surrounding context (full file / callers / the Anthropic parser it's compared to); (2) apps/rocm/src/serve_summary.rs's doc comments, docs/testing.md, and docs/manual-testing.md, cross-checked against each other and against the actual runtime behavior. I independently verified both logic findings by hand-tracing serde_json::Value/Option semantics, and confirmed by repo-wide grep that no other surface still states the old "not detected" claim. Checked this repo's AGENTS.md (§3 test-coverage rules, §5's "update the old claim" rule, §6 guardrails) — no violations. CI at head is green except E2E tests (MI300X, current), still queued behind the runner backlog, nothing failing.

Did not run: a "prior changes" pass (no earlier review comments on these specific lines to re-check, distinct from this PR's own review thread, which this pass does not read). Not reconciled against this PR's existing review threads or prior rounds — this is a blind pass.


🤖 Automated by an AI agent

Comment on lines +471 to +476
let message = error
.and_then(|error| error.get("message"))
.or(error)
.or_else(|| payload.get("message"))
.and_then(serde_json::Value::as_str)
.unwrap_or("unknown OpenAI-compatible stream error");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (logic, confidence 85): the top-level message fallback is reachable only when the error key is entirely absent (the bare {"object":"error","message":...} shape), not whenever error.message is missing or non-string.

.and_then(|error| error.get("message")) returns None for {"error":{"code":500},...} (no message key) or {"error":[1,2],...} (not an object). .or(error) then substitutes the original error value (the object/array itself, not a string) — which is Some, so .or_else(|| payload.get("message")) never runs. Only then does .and_then(as_str) discover the candidate isn't a string and fall through to the placeholder.

Concretely, {"error":{"code":500},"message":"CUDA OOM"} bails with "unknown OpenAI-compatible stream error" instead of the informative top-level "CUDA OOM", even though the chain's shape (and the existing non-blocking note on this PR about non-string error.message) suggests top-level message was meant as a genuine fallback tier. No fixture exercises this combination.

Fix: resolve each tier to &str before chaining, e.g. error.and_then(|e| e.get("message")).and_then(Value::as_str).or_else(|| error.and_then(Value::as_str)).or_else(|| payload.get("message").and_then(Value::as_str)).unwrap_or(placeholder).

Comment on lines +467 to +468
let has_error = payload.get("error").is_some_and(|error| !error.is_null())
|| payload.get("object").and_then(serde_json::Value::as_str) == Some("error");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (logic/edge case, confidence 70): payload.get("error").is_some_and(|error| !error.is_null()) treats any non-null error value as a failure — including false, 0, "", [], or {} — not just the object/string shapes vLLM is documented as sending. If some OpenAI-compatible server ever includes a benign falsy "error" field alongside real content, this would hard-fail a stream the old code silently tolerated. Unconfirmed against any real server (the PR itself notes these shapes weren't captured live), but worth either narrowing the check to the documented shapes or adding a fixture that pins the broader behavior as intentional.

Comment thread docs/testing.md
that follows content tokens. The reason shown is the server's own message.
- Not detected (no failure is reported and the status reads `ready`): a `200`
answer with no `data:` line at all (HTML, plain JSON, an empty body); a
close-delimited stream cut after a token; and a complete stream with no `content` token, which is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (mechanical, confidence 100): this line is 99 characters, breaking the paragraph's hand-wrap convention (~75-82 chars elsewhere, including the lines immediately above and below it). Looks like a leftover from moving the "an engine error sent as a data: payload…" clause up to the "Detected" bullet — the remaining "Not detected" bullet wasn't re-wrapped. Minor rewrap would fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chat: an error sent inside the stream is dropped and the answer is shown as complete

2 participants