Repository navigation
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
…payload
vLLM and other OpenAI-compatible servers report a mid-generation failure
in-band as `data: {"error":{...}}` (older releases as a bare
`{"object":"error",...}`), followed by `data: [DONE]`.
`parse_openai_sse_line` read only `choices[..].content`, skipped the error
line, and so the stream ended cleanly with the truncated text.
It now returns an error carrying the server's message, as the Anthropic
parser already does. Text delivered before the failure still reaches the
callback, and a null `error` is not an error.
The reproducer test for this gap is un-ignored and fixtures cover the
error shapes; docs/testing.md's known-gap example now points at a gap
that is still open. The error shapes were written from vLLM's handler,
not captured from a live server.
Fixes #516
Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
18cee21 to
088e75f
Compare
|
Rebased onto current main (head 088e75f). All checks except |
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. The in-band error payload now fails the stream before any further line is read, and the new tests pin both the exact message and the absence of a done event. CI is green on all required checks; only the GPU lane is pending.
Non-blocking notes:
{"error":{},"message":"x"}prints the placeholder rather than "x" (inline comment).- A non-string
error.messagesilently becomes the placeholder. - The proptests module doc is rewrapped awkwardly (cosmetic).
- The server error text is printed unsanitised; deferring that to #983 is fine.
| let error = payload.get("error").filter(|error| !error.is_null()); | ||
| let message = error | ||
| .and_then(|error| error.get("message")) | ||
| .or(error) |
There was a problem hiding this comment.
Nit: when error is an object with no string message, .or(error) wins over the top-level message fallback on the next line, so {"error":{},"message":"x"} prints the placeholder. Only matters for odd servers; consider .or(error.filter(|e| e.is_string())).
There was a problem hiding this comment.
Not changed in e454b39, which only carries the doc fixes below; this nit is left open for you to resolve or say you want it fixed here. For {"error":{},"message":"x"} the placeholder is what prints today; the stream still fails either way.
juhovainio
left a comment
There was a problem hiding this comment.
I traced the fix itself end-to-end (both the local-provider path through local_http.rs's read loop and the remote path through stream_json_with_headers): the ? on push_bytes in both readers means the new bail! in parse_openai_sse_line genuinely stops the stream before any later line is read, in every case the new fixtures exercise (nested/bare error object, string error, null-error passthrough, non-object payload, no-[DONE] tail). The split-invariance sweep exercising all seven new OPENAI_FIXTURES entries at every one/two-byte split is solid coverage, and I agree with the existing approval's read of the message-precedence and sanitization nits — not repeating those.
Blocking
This fix flips a documented "not detected" behavior to "detected", and the change doesn't update the three places that document the old behavior, so the tree now contains explicit, incorrect claims about current behavior (a requirement this repo's own AGENTS.md §5 states directly: "when a statement such as 'X is not supported' or 'there is no Y' becomes false, grep the same surfaces for the old claim before pushing").
The PR body correctly notes the only real streaming consumer of this parser is run_smoke_test's post-launch probe (serve_summary.rs, provider "local" → local_http.rs → OpenAiSseEmitter → parse_openai_sse_line). Before this change, an in-band error line was skipped, the stream still finished cleanly, and provider_stream_chat_with_callback returned Ok, so the smoke test was SmokeOutcome::Measured. After this change, the same line makes parse_openai_sse_line return Err, which propagates unmodified through stream_chat_with_callback and provider_stream_chat_with_callback (no catch in between), so run_smoke_test now takes the Err branch and returns SmokeOutcome::Failed instead. That's an improvement in what gets detected, but three existing comments/docs still assert the pre-fix behavior:
apps/rocm/src/serve_summary.rs:90(doc onSmokeOutcome::Measured): "Not detected here, and soMeasured: an error reported inside a 200 stream as adata:payload, ..."apps/rocm/src/serve_summary.rs:382(doc onrun_smoke_test): "AlsoMeasured, and so not detected: an errordata:payload inside a 200 stream, ..."docs/testing.md:4401-4408: lists an in-banddata:error as "Not detected (no failure is reported and the status readsready)" and separately as a case whose tokens "are measured."
All three now describe the opposite of what the code does post-fix. None of them are touched by this diff (the only docs/testing.md edit is the unrelated known-gap example swap at line 668), so this isn't something the existing approval would have seen by reading the diff alone. Fix: update all three to say the in-band case is now detected/Failed, or fold them into one blocking finding's worth of edits — whichever reads better in context.
Confidence 85 — traced the full propagation path (parse_openai_sse_line → push_bytes → stream_json_from_local_endpoint/stream_json_with_headers → stream_chat_with_callback → provider_stream_chat_with_callback → run_smoke_test) against the current code, and confirmed no test pins the stale claim either way.
CI
Re-checked at head 088e75f3: all required checks are green; only E2E tests (MI300X, current) is still in_progress (queued behind the runner backlog per the author's own comment), nothing failing.
🤖 Automated by an AI agent
run_smoke_test reports an error sent as a `data:` payload inside a 200 stream as SmokeOutcome::Failed since the previous commit. Update the SmokeOutcome and run_smoke_test doc comments, the Serve Smoke Test Failure section of docs/testing.md and the matching bullet in docs/manual-testing.md, which still listed that case as not detected, and pin the outcome with a smoke-test test against a mock engine. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
The in-band error fix looks right to me: the stream now fails on an error payload, the reproducer is un-ignored and asserts the exact message, the delivered text and no done event, and the new fixtures plus the lines-after-error test cover the edges. Two small, non-blocking notes inline.
Beyond those:
- The server's error text goes into the summary unsanitised. The PR already flags this and ties it to #983, so I'm fine leaving it there.
- Failing the stream rather than keeping partial text is the product call #516 leaves open. It mirrors the Anthropic parser, and text already delivered stays delivered, which seems reasonable.
- No Gherkin scenario, justified because no user-facing CLI surface streams. The serve smoke-test summary change is arguably user-observable, but the socket-mock test asserts
Failedplus the message, so I'm OK with that.
| if has_error { | ||
| let error = payload.get("error").filter(|error| !error.is_null()); | ||
| let message = error | ||
| .and_then(|error| error.get("message")) |
There was a problem hiding this comment.
Edge case: .or(error) short-circuits when error is an object without a string message, so a sibling top-level "message" is never used and the placeholder is shown. Consider as_str-filtering each candidate before chaining. Non-blocking.
| that follows content tokens. The reason shown is the server's own message. | ||
| - Not detected (no failure is reported and the status reads `ready`): a `200` | ||
| answer with no `data:` line at all (HTML, plain JSON, an empty body); a | ||
| close-delimited stream cut after a token; and a complete stream with no `content` token, which is |
There was a problem hiding this comment.
This line is much longer than its neighbours after the edit; consider rewrapping for diff-friendliness. Non-blocking.
juhovainio
left a comment
There was a problem hiding this comment.
Review — no blocking findings
Full review of the whole change. Fixes #516 (open issue; the only PR linked to it): the issue explicitly leaves "discard vs. warn vs. keep silently" as an open product call and asks only that the defect become visible — this PR resolves that narrower ask (fail the stream, keep text already delivered, mirror the Anthropic parser's behavior) without resolving the wider product question, which is consistent with the issue's own framing. No other ticket named.
Blocking
None
Non-blocking
- Top-level
messagefallback is dead for most of the shapes it should cover —apps/rocm/src/providers/wire.rs:471-477
.and_then(|error| error.get("message")).or(error).or_else(|| payload.get("message"))resolves at the.or(error)step to the originalerrorvalue (object/array, not a string) whenevererroris present but has nomessagekey or isn't itself a string — which isSome, so the top-level-messagefallback never runs..and_then(as_str)then discovers it isn't a string and falls to the placeholder.{"error":{"code":500},"message":"CUDA OOM"}bails with the generic placeholder instead of "CUDA OOM". No fixture exercises this combination.
Confidence 85 · logic · Fix: resolve each tier to&strbefore chaining (see inline comment). has_errortreats any non-nullerrorvalue as a failure, including falsy scalars —apps/rocm/src/providers/wire.rs:467-468
payload.get("error").is_some_and(|error| !error.is_null())is true forfalse,0,"",[],{}— not just the object/string shapes vLLM is documented as sending. Plausible but unconfirmed against a real server; the PR itself notes these shapes weren't captured live.
Confidence 70 · logic · Fix: narrow to the documented shapes, or add a fixture pinning the broader behavior as intentional.- Unwrapped line breaks the file's prose-wrap convention —
docs/testing.md:4290
99 characters vs. ~75-82 elsewhere in the same paragraph; leftover from moving a clause up to the "Detected" bullet.
Confidence 100 · mechanical · Fix: rewrap the bullet.
Decisions for the author
- Failing the whole stream vs. keeping partial text with a warning — tradeoff
The PR fails the stream entirely on an in-band error (text already delivered via the callback is kept; no further line, including[DONE], is read). Issue #516 explicitly leaves this choice open as a product decision; this PR's answer mirrors the existing Anthropic-parser behavior. Worth the author's explicit confirmation this is the intended default, not merely the simplest option. - Engine error text reaches the smoke-test summary unsanitised — tradeoff
Already tracked under #983 and explicitly deferred there in the PR body; the Anthropic path has the same property. Reasonable to leave as is.
Positive signals
- The fix is exercised at three independent levels: a hand-computed fixture table, an exhaustive one/two-byte split-invariance sweep (so chunk boundaries can't hide the new branch), and a full integration test (
serve_smoke_test_fails_on_an_in_band_error_payload_in_a_200_stream) driving the real socket/chunked-transfer/run_smoke_testpath end to end. - The un-ignored
in_band_stream_errors_are_not_reported_as_a_clean_finishpins resulting state, not just wording: exact error string, preserved partial content, and!events.iter().any(|e| e.done)— it would fail if the fix were reverted. - The three doc/comment locations that a prior review round found describing the pre-fix "not detected" behavior (
serve_summary.rs's two doc comments,docs/testing.md's detected/not-detected list) are now all updated and consistent with each other and with the actual code, anddocs/manual-testing.mdwas updated too even though no one had flagged it.
Deployment notes
None. This only changes the internal post-launch smoke-test outcome (Measured → Failed) for an in-band stream error; no public CLI surface streams today, so there's no user-facing behavior change and no migration.
What this covered
Read the full diff (origin/main 2583a97f...pr-1056 e454b394168619c890eea6db68f290835e181e3c) across all 6 changed files, via two independent fan-out passes grouped by coupling — (1) apps/rocm/src/providers/wire.rs, apps/rocm/src/providers/wire/sse_emitter_proptests.rs, and the new test in apps/rocm/src/providers.rs, each read completely with surrounding context (full file / callers / the Anthropic parser it's compared to); (2) apps/rocm/src/serve_summary.rs's doc comments, docs/testing.md, and docs/manual-testing.md, cross-checked against each other and against the actual runtime behavior. I independently verified both logic findings by hand-tracing serde_json::Value/Option semantics, and confirmed by repo-wide grep that no other surface still states the old "not detected" claim. Checked this repo's AGENTS.md (§3 test-coverage rules, §5's "update the old claim" rule, §6 guardrails) — no violations. CI at head is green except E2E tests (MI300X, current), still queued behind the runner backlog, nothing failing.
Did not run: a "prior changes" pass (no earlier review comments on these specific lines to re-check, distinct from this PR's own review thread, which this pass does not read). Not reconciled against this PR's existing review threads or prior rounds — this is a blind pass.
🤖 Automated by an AI agent
| let message = error | ||
| .and_then(|error| error.get("message")) | ||
| .or(error) | ||
| .or_else(|| payload.get("message")) | ||
| .and_then(serde_json::Value::as_str) | ||
| .unwrap_or("unknown OpenAI-compatible stream error"); |
There was a problem hiding this comment.
Non-blocking (logic, confidence 85): the top-level message fallback is reachable only when the error key is entirely absent (the bare {"object":"error","message":...} shape), not whenever error.message is missing or non-string.
.and_then(|error| error.get("message")) returns None for {"error":{"code":500},...} (no message key) or {"error":[1,2],...} (not an object). .or(error) then substitutes the original error value (the object/array itself, not a string) — which is Some, so .or_else(|| payload.get("message")) never runs. Only then does .and_then(as_str) discover the candidate isn't a string and fall through to the placeholder.
Concretely, {"error":{"code":500},"message":"CUDA OOM"} bails with "unknown OpenAI-compatible stream error" instead of the informative top-level "CUDA OOM", even though the chain's shape (and the existing non-blocking note on this PR about non-string error.message) suggests top-level message was meant as a genuine fallback tier. No fixture exercises this combination.
Fix: resolve each tier to &str before chaining, e.g. error.and_then(|e| e.get("message")).and_then(Value::as_str).or_else(|| error.and_then(Value::as_str)).or_else(|| payload.get("message").and_then(Value::as_str)).unwrap_or(placeholder).
| let has_error = payload.get("error").is_some_and(|error| !error.is_null()) | ||
| || payload.get("object").and_then(serde_json::Value::as_str) == Some("error"); |
There was a problem hiding this comment.
Non-blocking (logic/edge case, confidence 70): payload.get("error").is_some_and(|error| !error.is_null()) treats any non-null error value as a failure — including false, 0, "", [], or {} — not just the object/string shapes vLLM is documented as sending. If some OpenAI-compatible server ever includes a benign falsy "error" field alongside real content, this would hard-fail a stream the old code silently tolerated. Unconfirmed against any real server (the PR itself notes these shapes weren't captured live), but worth either narrowing the check to the documented shapes or adding a fixture that pins the broader behavior as intentional.
| that follows content tokens. The reason shown is the server's own message. | ||
| - Not detected (no failure is reported and the status reads `ready`): a `200` | ||
| answer with no `data:` line at all (HTML, plain JSON, an empty body); a | ||
| close-delimited stream cut after a token; and a complete stream with no `content` token, which is |
There was a problem hiding this comment.
Non-blocking (mechanical, confidence 100): this line is 99 characters, breaking the paragraph's hand-wrap convention (~75-82 chars elsewhere, including the lines immediately above and below it). Looks like a leftover from moving the "an engine error sent as a data: payload…" clause up to the "Detected" bullet — the remaining "Not detected" bullet wasn't re-wrapped. Minor rewrap would fix it.
Why
vLLM and other OpenAI-compatible servers report a failure that happens mid-generation in-band:
data: {"error":{...}}(older releases send a bare{"object":"error",...}), followed bydata: [DONE].parse_openai_sse_lineskipped the error line, so the stream ended as a clean finish that held only the text generated before the failure.In this tree
rocm chatand the dash chat use non-streaming requests. The only streaming consumer of this parser is the serve post-launch smoke test, which treats any stream error as empty metrics. Before this change it computed throughput and TTFT from the truncated text; now it reports empty metrics like any other stream failure. No user-facing CLI surface streams, so no e2e scenario is added (AGENTS.md section 3 allows that with this explanation).What changes
The parser now returns an error,
OpenAI-compatible provider stream failed: <server message>(with a placeholder when the server gave no message), mirroring the Anthropic parser. Callers already propagate it with?, so text delivered before the failure still reaches the callback and no clean done event is emitted. A nullerrorfield is not an error.docs/testing.md: its known-gap example pointed at this gap, so it now names the still-open #522 one instead.Verification
in_band_stream_errors_are_not_reported_as_a_clean_finishis un-ignored and strengthened: it asserts the exact error message, that the earlier text was delivered, and that no done event was emitted.[DONE], null error, string error, empty-error placeholder, non-object payload), plus a test that lines after an error are not read.sse_emittertests fail; with it, they pass.-D warnings, module-size,cargo xtask smoke, and the proptests atPROPTEST_CASES=10000(43 passed).Not verified:
runtimes_cmd::tests::concurrent_active_runtime_marker_writes_keep_a_complete_marker_publishedfound the marker file unreadable during a concurrent rewrite.storage::tests::prune_removes_the_install_it_named_and_not_a_case_twin_of_itlooks like it assumes a case-sensitive filesystem and was run on virtiofs.Scope
Not done here, as a separate follow-up: the engine's error message is printed unsanitised (the Anthropic path is the same), which relates to #983.
Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (None found: the search matched nothing.)docs/architecture.md. (N/A, no new subcommand or subsystem.)docs/architecture.mdhas the matching entry. (N/A, no new files.)Fixes #516