Skip to content

fix(serve): say so in the summary when the post-launch smoke test fails (ROCMAI-703) - #1050

Merged
tomastola merged 4 commits into
mainfrom
ROCMAI-703-fix-serve-smoke-failure-summary
Nov 17, 2026
Merged

tomastola merged 4 commits into
mainfrom
ROCMAI-703-fix-serve-smoke-failure-summary

Conversation

@tomastola

@tomastola tomastola commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #523

Why

After rocm serve launched a service, the summary printed status: ready with usage instructions even when the first post-launch test request had failed, for example when the engine died mid-response. The only hint was that the two metrics read n/a, with no reason given.

The cause is that run_smoke_test returned empty metrics on any error and discarded the error itself, while the summary decided "ready" from the launch-time status alone.

What changes

  • The smoke test now returns an outcome (SmokeOutcome) instead of bare metrics.
  • On failure the summary heading reads Deployment summary (smoke test failed), the status reads started, smoke test failed, and a note says the smoke test could not complete, gives a one-line reason, and points to rocm logs --service <id>. The reason is cleared of terminal control and format characters and length-capped; text after a cut-off CSI escape sequence is kept (an unterminated OSC, DCS or APC string sequence still swallows the rest of the reason, see the follow-ups).
  • Launch stays best-effort: the exit code is 0, the service record is unchanged and the server is not stopped. --no-smoke-test and already-running summaries are unchanged. Only the interactive summary runs the smoke test.
  • docs/testing.md lists what is and is not detected, and the README and docs/manual-testing.md point to that list.

What is not detected

  • A 200 response whose stream carries an in-band data: error payload. That is handled separately in fix(providers): fail an OpenAI-compatible stream on an in-band error payload #1056.
  • A 200 response with no data: line.
  • A close-delimited stream that is cut after a token.
  • A clean stream that ends without a content token. This is deliberately not a failure, because reasoning models can legitimately produce no content token.
  • Known limitation: an audit-log or service-resolution error also surfaces as a failed smoke test.

Verification

  • Unit tests cover the failure summary, a stream with no content, control and format characters and newlines in the reason, a cut-off escape sequence, and the already-running summary. The existing mid-stream engine-death test pinned the old behaviour and now asserts the summary is not presented as ready and carries the reason. The assert messages are constant strings, so the summary (which carries the endpoint api key) is never echoed on failure.
  • Mutation check: forcing the render branch back to the old behaviour turns the key tests red.
  • End to end: the serve-59 scenario launches a service that reaches ready and then fails its first chat request, using a second ready mode of the existing e2e fault-injection hook. It asserts, over a pty, the heading, the status row, the engine reason, exactly one chat request and the logs command, with exit 0. The scenario is tagged @requires-oom-fault-injection, @requires-no-gpu and @requires-os:linux, so it runs on the GitHub-hosted mock E2E lane (where xtask builds the binary with the hook) and resolves to skip on every prebuilt-binary lane (the self-hosted GPU lanes and the nightly lanes). The hook is compiled out of normal release builds, checked with strings on a release binary.
  • Passing: cargo clippy -D warnings, fmt, the 45 smoke/summary tests, the ASCII test, check-module-size, check-architecture-doc, and the serve-41 and serve-59 e2e scenarios. The workspace tests pass on a Linux filesystem; on a case-insensitive host mount two unrelated tests (the storage case-twin test and the runtimes_cmd marker test) fail, identically on main.
  • Not verified: the prek license-headers hook (needs hawkeye) and lychee did not run locally. CodeQL cannot run locally, so the assert changes were checked by inspection against the rust/cleartext-logging rule.

Follow-ups (not in this PR)

  • Send the smoke request to the launched service instead of routing by model name: the reason and the logs pointer can belong to another service, and a dead launch can read as ready when an older ready service for the same model answers.
  • An audit-log write error and a service-resolution error surface as a failed smoke test.
  • --no-smoke-test suppressing the new failure verdict has no test of its own: the gate in serve_cmd.rs is older code, no e2e scenario drives the flag, and the unit test for the skipped case renders default metrics by hand. A sibling scenario using the ready hook with --no-smoke-test would cover it.
  • An unterminated OSC, DCS or APC sequence in an engine's reason swallows the remainder in one_line_reason.
  • The local provider treats a 200 status line without a reason phrase, and EOF after [DONE] without the terminating chunk, as failures. This strictness is pre-existing and is now visible.
  • The e2e hook keeps its OOM names and parses its mode from the environment in two places, and its fabricated record carries the CLI's own pid without start ticks, a pid-reuse hazard at teardown.
  • In-band data: error payloads (fix(providers): fail an OpenAI-compatible stream on an in-band error payload #1056, chat: an error sent inside the stream is dropped and the answer is shown as complete #516).
  • A few test-label and wording nits in docs: the "exactly one chat request" step cannot tell the readiness probe's request from the smoke request, the ready-latch comment names the wrong probe, a serve-41 feature comment about the daemon step, and the README sentence order.
  • Carry Option<SmokeOutcome> in DeploymentSummary instead of two fields.

Scope

Other open PRs edit the same summary code (#817, #678, #723, #725), so overlap with them is possible.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation).
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.

Comment thread apps/rocm/src/providers.rs Fixed
Comment thread apps/rocm/src/providers.rs Fixed
Comment thread apps/rocm/src/providers.rs Fixed
Comment thread apps/rocm/src/providers.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola
tomastola force-pushed the ROCMAI-703-fix-serve-smoke-failure-summary branch 2 times, most recently from 29402d1 to 3c5faa7 Compare November 16, 2026 19:43
@tomastola
tomastola marked this pull request as ready for review November 16, 2026 21:54
@tomastola
tomastola requested a review from a team as a code owner November 16, 2026 21:54
@tomastola
tomastola requested a review from volen-silo November 16, 2026 21:54
@tomastola

Copy link
Copy Markdown
Collaborator Author

All required checks are green on 3c5faa7; the self-hosted E2E lane (MI300X) is still queued for lack of a runner and is not required.

@tomastola
tomastola requested a review from juhovainio November 16, 2026 21:54
…ls (ROCMAI-703)

After `rocm serve`, the deployment summary read `status: ready` with n/a
metrics even when the first test request had failed: run_smoke_test
discarded the error and returned empty metrics, so a server that could not
answer looked the same as one that was not measured.

run_smoke_test now returns the failure reason. When the test fails, the
summary is headed "(smoke test failed)", the status reads "started, smoke
test failed", and a note gives a one-line reason and points at `rocm logs`.
The launch stays best-effort: the server keeps running and the exit code is
unchanged.

Fixes #523

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…choing the api key (ROCMAI-703)

The first commit's tests echoed the rendered summary, which carries the
endpoint api key, into assert messages, and CodeQL rust/cleartext-logging
flagged them. The asserts now carry constant messages.

Also in this change:
- the failure reason is cleared of terminal control and format characters,
  and text after a cut-off escape sequence is kept;
- a clean stream with no content token (a reasoning model) is no longer
  treated as a failure;
- serve-55, a second `ready` mode of the existing e2e fault-injection hook
  (compiled out of release builds), asserts the heading, status row, engine
  reason, one chat request and the logs command over a pty with exit 0;
- the README and test docs state what is and is not detected.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… (ROCMAI-703)

The testing guide claimed a close-delimited stream cut after a token shows
`n/a` metrics; it shows the metrics measured up to the cut. The README and
the manual test steps now point at the guide's list of failure classes that
are not detected, so they no longer imply the note catches everything.

The failure reason no longer lets an engine add a backtick-quoted command
next to the note's own logs hint. A test pins the note's "server may not be
able to serve requests" warning. Comments no longer list scenario ids that
drift. The guide records that two ready services on one model can make the
verdict describe the other service.

The earlier commit's message refers to the e2e scenario as serve-55; it is
serve-56 after merging main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…tes claim (ROCMAI-703)

A review found claims the code cannot back. The daemon-left-behind step
cannot discriminate the autostart guard because the isolated config
enables no watcher. The testing guide overstated `n/a` metrics for
undetected classes and described a two-ready-services case that cannot
happen, while omitting the real false pass: the smoke request is routed
by model name, not to the launched service. A length-cap boundary had no
test.

The text now says what the code does, the cap is tested at 200/201
characters, a load-bearing line in the ready hook is commented, and prose
cites @id tags instead of scenario numbers that had already drifted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the ROCMAI-703-fix-serve-smoke-failure-summary branch from 3c5faa7 to bdd4252 Compare November 17, 2026 03:33

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (17 files, +955/-125) against merge-base 9101286a…head bdd4252a, split across the production summary/smoke-test code and the e2e harness/scenario changes. Implements ROCMAI-703: a failed post-launch smoke test now shows up in the deployment summary (heading, status row, one-line engine reason, logs pointer) instead of silently reading ready with n/a metrics. The PR description is unusually precise and candid — every item below was already named as a known gap in its own "Follow-ups" section, which this review independently reached the same way before cross-checking against that section.

Blocking

None.

Non-blocking

  • A doc comment's cross-reference is now dangling — tests/e2e-cucumber/features/model_serving.feature:20 (inline comment). The serve-02 scenario's header comment used to point at serve-55 for "what holds the per-engine refs in the meantime"; this PR repoints it at serve-60, which is this PR's own new, unrelated smoke-failure-summary scenario. Reads as a mechanical renumbering slip rather than intent.
  • The new {then} steps interpolate the full PTY screen into their panic messages (assert_summary_reports_failed_smoke_test, assert_summary_gives_engine_reason, etc. in serving_steps.rs), unlike the production unit test added by this same PR, which deliberately keeps its assert messages as constant strings specifically so the rendered summary (which can carry the endpoint's API key) is never echoed on failure/CodeQL's rust/cleartext-logging check. No live leak today — this scenario never passes --host, so the endpoint binds loopback and api_key is None — but the pattern is worth not copying verbatim into a future scenario that does bind publicly.

Decisions for the author

  • serve-60's "exactly one chat request" assertion is less vacuous than the PR's own follow-up suggests — the follow-up list says this step "cannot tell the readiness probe's request from the smoke request." Tracing it: /v1/models never touches the mock's completion_requests counter, and the ready-mode fabricated launch short-circuits start_managed_service entirely, so the real readiness-probe wait/probe path never runs for this scenario at all — there is no readiness-probe chat request to confuse with the smoke test's for this specific path. Worth confirming whether the follow-up was describing a different/future scenario shape, since as written today the assertion does discriminate.
  • The fabricated ready-mode launch record compounds the already-disclosed pid-reuse hazard — the PR's own follow-ups already flag that the e2e hook's fabricated record carries the CLI's own pid with no start-tick, a pid-reuse hazard at teardown. The new ready mode marks that record status: "ready" with inference_verified_at_unix_ms set, i.e. a record that reads as genuinely live and verified rather than mid-launch. Any out-of-scope code that trusts a "ready" record's pid more readily than a "starting" one would be more exposed under this mode than the pre-existing OOM mode. Already acknowledged as a known gap, just sharpening it since this PR adds the specific mode that makes it worse.

Positive signals

  • Reason propagation is traced end to end with no normalization/loss on the way: run_smoke_test → SmokeOutcome::Failed(reason) → serve_cmd.rs's (metrics, smoke_failure) → DeploymentSummary.smoke_failure → render_summary's heading/status/note all carry the same value.
  • one_line_reason's terminal-control sanitizer was hand-traced against crates/rocm-core/src/terminal.rs's escape-sequence scanner for both disclosed edge cases (truncated CSI kept, unterminated OSC/DCS/APC swallowed) and matches the PR's own description; the backtick→apostrophe substitution is directly asserted by a test using a real chunk-size error that can embed one.
  • --no-smoke-test and the already-running summary path are both gated before the smoke test ever runs, and a dedicated test (an_already_running_summary_ignores_a_smoke_failure) pins that an already-running summary can't show a smoke failure even if one were somehow present.
  • The API key is sourced from an independent field never concatenated into the failure reason — no leak path found in the production code despite the summary printing right next to it.
  • The scenario's "exactly one chat request", summary-text, and service-record-unchanged assertions are each honest about what they do and don't prove (two are explicitly commented as "passes with the fix reverted" rather than silently relying on an untested assumption).
  • All production file line counts (excluding #[cfg(test)] modules, per this repo's own module-size convention) stay under the limit even after this PR's growth.

Deployment notes

None required — exit code stays 0, the service record is left unchanged, and --no-smoke-test / already-running behavior is unchanged; this only adds visibility into an existing failure mode. CI is fully green at this head (47 checks, including both self-hosted GPU E2E lanes and serve-59/the new scenario).

What this covered

Every one of the 17 changed files read in full against the PR head (bdd4252a7a3decb39322880e9ebf8e3acdc8cf8b, merge-base 9101286acfe1d071b2d1ae43739f937593589ad7), split into a production-code pass (cli.rs, providers.rs, serve_cmd.rs, serve_summary.rs, service_launch.rs, service_launch/readiness.rs, plus the unmodified providers/local_http.rs and crates/rocm-core/src/terminal.rs these changes depend on) and an e2e-harness pass (README.md, model_serving.feature, capability.rs, expectation.rs, serving_steps.rs, xtask/src/e2e.rs, tests/e2e-cucumber/src/mock_server.rs). Each pass verified its claims directly against the actual code rather than taking the PR description at face value.

Did not run: no local cargo test/clippy — relied on this PR's own CI, which is green across every required and informational check at this head, including both self-hosted GPU E2E lanes. Not reconciled against any other review thread — the only pre-existing activity on this PR was an automated security-scan comment reporting no findings, not human review feedback.


🤖 Automated by an AI agent

# host while its xfail row stands, and the vLLM run stops before printing a
# plan, so the step panics there whatever the lemonade half did. It becomes
# regression coverage when that row is narrowed. serve-55 and the unit tests in
# regression coverage when that row is narrowed. serve-60 and the unit tests in

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this comment's cross-reference looks like a renumbering slip rather than a deliberate edit. Before this PR, serve-55 genuinely was about per-engine short-name refs (the scenario this comment sits above). serve-60 is this PR's own new smoke-failure-summary scenario, which is unrelated to short names or per-engine refs. The pointer is now dangling/wrong — worth restoring the original serve-55 reference (or whatever scenario actually covers that narrowing today) rather than pointing at this PR's own addition.

# host while its xfail row stands, and the vLLM run stops before printing a
# plan, so the step panics there whatever the lemonade half did. It becomes
# regression coverage when that row is narrowed. serve-55 and the unit tests in
# regression coverage when that row is narrowed. serve-60 and the unit tests in

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this comment's cross-reference was changed from serve-55 to serve-60, but serve-60 is this same PR's new smoke-failure-summary scenario, which has nothing to do with per-engine short-name refs. serve-55 was (and still is) the scenario that actually covers that (confirmed via git log -S). This reads like an incidental renumbering slip rather than a deliberate edit — the comment now points a future reader at the wrong regression coverage once the xfail row here is narrowed.

Confidence 80 · mechanical · Fix: restore the serve-55 reference (or point at whichever scenario currently covers per-engine short-name refs).

Comment thread docs/testing.md
promise in the README and `rocm serve --help`.
- "no background automation daemon was left behind" is hygiene only: it records
that no daemon was started but does not discriminate the
`if !e2e_simulate_oom_launch()` guard in `serve_cmd.rs`, because no watcher is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this new citation of serve_cmd.rs is a bare backtick path rather than a relative markdown link ([serve_cmd.rs](../apps/rocm/src/serve_cmd.rs)), per this repo's own docs-link convention (AGENTS.md §5). docs/testing.md isn't in that rule's exemption list, and this reads as a directed citation (pointing the reader at a specific guard in that file) rather than a generic/historical mention. lychee/docs-links won't catch this since it's not a broken link, just an unlinked one — which is exactly the gap that convention exists to close.

Confidence 65 · mechanical · Fix: link it, e.g. [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs).

.expect("no interactive serve summary")
.screen_text();
let flattened = screen_without_whitespace(&screen);
assert!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this then step (and its siblings added for serve-60) interpolates the full PTY {screen} into the panic message on assertion failure. Elsewhere in this same PR (the providers.rs unit tests, and serve_summary.rs's own test comments) the author is deliberate about not doing this, specifically to avoid echoing a summary that can carry the endpoint API key and tripping CodeQL's rust/cleartext-logging rule. No real leak happens here today — this scenario never passes --host, so the endpoint binds loopback and api_key is None — but the pattern now exists in two different shapes in the same codebase, and copying this then step as a template for a future public-bind scenario would carry the key into CI logs on a failing assertion.

Confidence 70 · architectural · Fix: assert on a redacted/derived value here too (or add a comment noting why it's safe in this scenario specifically, the way the unit tests do), so the safe pattern doesn't get overwritten by copy-paste.

@tomastola
tomastola added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit cab5db1 Nov 17, 2026
48 checks passed
@tomastola
tomastola deleted the ROCMAI-703-fix-serve-smoke-failure-summary branch November 17, 2026 09:46
r0x0r added a commit that referenced this pull request Nov 17, 2026
Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary
prose and adds its own serve scenario, so all three conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for
its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario
becomes `serve-61` and moves after it, so each number still equals its
position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two
`-n "serve-60"` repro filters in docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. They are unnumbered `##` headings, so nothing renumbers.

README.md: both sides rewrite the one paragraph describing the deployment
summary, so this is a real merge rather than a choice between them. The
summary sentence keeps this branch's `device:` row and main's endpoint/model/
smoke-test list; main's smoke-test-failure paragraph and this branch's
"device line is omitted when the engine cannot say yet" paragraph both
survive; and the shared closing sentence is kept once. Taking either side
whole would have silently dropped the other's user-facing contract.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Main landed ROCMAI-703 (#1050), which adds its own serve scenario and its own
testing.md section, so both conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60`.
Main's keeps 60; this branch's two become `serve-61` and `serve-62` and move
after it, so each number still equals its position. Four prose references
follow: the two scenario entries and the repro filter in `docs/testing.md`
(now `serve-(61|62)`), the `(serve-62)` note in
`service_launch/readiness.rs`, and the feature file's own
"exactly like serve-60's crash" line. The other `-n "serve-60"` filters in
docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. Unnumbered headings, so nothing renumbers.

This is the third time these two scenarios have moved (57/58 -> 59/60 ->
60/61 -> 61/62); the `@id:` tags are the binding key and have never changed,
so no step binding moved with them.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Tracks #678 past its merge of ROCMAI-703 (#1050), which renumbered that
branch's APU scenario to serve-61 and reworked the README's
deployment-summary paragraph. Merged clean -- no conflicts here.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary
prose and adds its own serve scenario, so all three conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for
its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario
becomes `serve-61` and moves after it, so each number still equals its
position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two
`-n "serve-60"` repro filters in docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. They are unnumbered `##` headings, so nothing renumbers.

README.md: both sides rewrite the one paragraph describing the deployment
summary, so this is a real merge rather than a choice between them. The
summary sentence keeps this branch's `device:` row and main's endpoint/model/
smoke-test list; main's smoke-test-failure paragraph and this branch's
"device line is omitted when the engine cannot say yet" paragraph both
survive; and the shared closing sentence is kept once. Taking either side
whole would have silently dropped the other's user-facing contract.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve: the launch summary says the service is ready even when the post-launch smoke test failed

3 participants