Repository navigation
fix(serve): say so in the summary when the post-launch smoke test fails (ROCMAI-703) - #1050
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
29402d1 to
3c5faa7
Compare
|
All required checks are green on 3c5faa7; the self-hosted E2E lane (MI300X) is still queued for lack of a runner and is not required. |
…ls (ROCMAI-703) After `rocm serve`, the deployment summary read `status: ready` with n/a metrics even when the first test request had failed: run_smoke_test discarded the error and returned empty metrics, so a server that could not answer looked the same as one that was not measured. run_smoke_test now returns the failure reason. When the test fails, the summary is headed "(smoke test failed)", the status reads "started, smoke test failed", and a note gives a one-line reason and points at `rocm logs`. The launch stays best-effort: the server keeps running and the exit code is unchanged. Fixes #523 Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…choing the api key (ROCMAI-703) The first commit's tests echoed the rendered summary, which carries the endpoint api key, into assert messages, and CodeQL rust/cleartext-logging flagged them. The asserts now carry constant messages. Also in this change: - the failure reason is cleared of terminal control and format characters, and text after a cut-off escape sequence is kept; - a clean stream with no content token (a reasoning model) is no longer treated as a failure; - serve-55, a second `ready` mode of the existing e2e fault-injection hook (compiled out of release builds), asserts the heading, status row, engine reason, one chat request and the logs command over a pty with exit 0; - the README and test docs state what is and is not detected. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
… (ROCMAI-703) The testing guide claimed a close-delimited stream cut after a token shows `n/a` metrics; it shows the metrics measured up to the cut. The README and the manual test steps now point at the guide's list of failure classes that are not detected, so they no longer imply the note catches everything. The failure reason no longer lets an engine add a backtick-quoted command next to the note's own logs hint. A test pins the note's "server may not be able to serve requests" warning. Comments no longer list scenario ids that drift. The guide records that two ready services on one model can make the verdict describe the other service. The earlier commit's message refers to the e2e scenario as serve-55; it is serve-56 after merging main. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
…tes claim (ROCMAI-703) A review found claims the code cannot back. The daemon-left-behind step cannot discriminate the autostart guard because the isolated config enables no watcher. The testing guide overstated `n/a` metrics for undetected classes and described a two-ready-services case that cannot happen, while omitting the real false pass: the smoke request is routed by model name, not to the launched service. A length-cap boundary had no test. The text now says what the code does, the cap is tested at 200/201 characters, a load-bearing line in the ready hook is commented, and prose cites @id tags instead of scenario numbers that had already drifted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
3c5faa7 to
bdd4252
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Review — no blocking findings
Full review of the whole change (17 files, +955/-125) against merge-base 9101286a…head bdd4252a, split across the production summary/smoke-test code and the e2e harness/scenario changes. Implements ROCMAI-703: a failed post-launch smoke test now shows up in the deployment summary (heading, status row, one-line engine reason, logs pointer) instead of silently reading ready with n/a metrics. The PR description is unusually precise and candid — every item below was already named as a known gap in its own "Follow-ups" section, which this review independently reached the same way before cross-checking against that section.
Blocking
None.
Non-blocking
- A doc comment's cross-reference is now dangling —
tests/e2e-cucumber/features/model_serving.feature:20(inline comment). Theserve-02scenario's header comment used to point atserve-55for "what holds the per-engine refs in the meantime"; this PR repoints it atserve-60, which is this PR's own new, unrelated smoke-failure-summary scenario. Reads as a mechanical renumbering slip rather than intent. - The new
{then}steps interpolate the full PTY screen into their panic messages (assert_summary_reports_failed_smoke_test,assert_summary_gives_engine_reason, etc. inserving_steps.rs), unlike the production unit test added by this same PR, which deliberately keeps its assert messages as constant strings specifically so the rendered summary (which can carry the endpoint's API key) is never echoed on failure/CodeQL'srust/cleartext-loggingcheck. No live leak today — this scenario never passes--host, so the endpoint binds loopback andapi_keyisNone— but the pattern is worth not copying verbatim into a future scenario that does bind publicly.
Decisions for the author
serve-60's "exactly one chat request" assertion is less vacuous than the PR's own follow-up suggests — the follow-up list says this step "cannot tell the readiness probe's request from the smoke request." Tracing it:/v1/modelsnever touches the mock'scompletion_requestscounter, and theready-mode fabricated launch short-circuitsstart_managed_serviceentirely, so the real readiness-probe wait/probe path never runs for this scenario at all — there is no readiness-probe chat request to confuse with the smoke test's for this specific path. Worth confirming whether the follow-up was describing a different/future scenario shape, since as written today the assertion does discriminate.- The fabricated
ready-mode launch record compounds the already-disclosed pid-reuse hazard — the PR's own follow-ups already flag that the e2e hook's fabricated record carries the CLI's own pid with no start-tick, a pid-reuse hazard at teardown. The newreadymode marks that recordstatus: "ready"withinference_verified_at_unix_msset, i.e. a record that reads as genuinely live and verified rather than mid-launch. Any out-of-scope code that trusts a "ready" record's pid more readily than a "starting" one would be more exposed under this mode than the pre-existing OOM mode. Already acknowledged as a known gap, just sharpening it since this PR adds the specific mode that makes it worse.
Positive signals
- Reason propagation is traced end to end with no normalization/loss on the way:
run_smoke_test→SmokeOutcome::Failed(reason)→serve_cmd.rs's(metrics, smoke_failure)→DeploymentSummary.smoke_failure→render_summary's heading/status/note all carry the same value. one_line_reason's terminal-control sanitizer was hand-traced againstcrates/rocm-core/src/terminal.rs's escape-sequence scanner for both disclosed edge cases (truncated CSI kept, unterminated OSC/DCS/APC swallowed) and matches the PR's own description; the backtick→apostrophe substitution is directly asserted by a test using a real chunk-size error that can embed one.--no-smoke-testand the already-running summary path are both gated before the smoke test ever runs, and a dedicated test (an_already_running_summary_ignores_a_smoke_failure) pins that an already-running summary can't show a smoke failure even if one were somehow present.- The API key is sourced from an independent field never concatenated into the failure reason — no leak path found in the production code despite the summary printing right next to it.
- The scenario's "exactly one chat request", summary-text, and service-record-unchanged assertions are each honest about what they do and don't prove (two are explicitly commented as "passes with the fix reverted" rather than silently relying on an untested assumption).
- All production file line counts (excluding
#[cfg(test)]modules, per this repo's own module-size convention) stay under the limit even after this PR's growth.
Deployment notes
None required — exit code stays 0, the service record is left unchanged, and --no-smoke-test / already-running behavior is unchanged; this only adds visibility into an existing failure mode. CI is fully green at this head (47 checks, including both self-hosted GPU E2E lanes and serve-59/the new scenario).
What this covered
Every one of the 17 changed files read in full against the PR head (bdd4252a7a3decb39322880e9ebf8e3acdc8cf8b, merge-base 9101286acfe1d071b2d1ae43739f937593589ad7), split into a production-code pass (cli.rs, providers.rs, serve_cmd.rs, serve_summary.rs, service_launch.rs, service_launch/readiness.rs, plus the unmodified providers/local_http.rs and crates/rocm-core/src/terminal.rs these changes depend on) and an e2e-harness pass (README.md, model_serving.feature, capability.rs, expectation.rs, serving_steps.rs, xtask/src/e2e.rs, tests/e2e-cucumber/src/mock_server.rs). Each pass verified its claims directly against the actual code rather than taking the PR description at face value.
Did not run: no local cargo test/clippy — relied on this PR's own CI, which is green across every required and informational check at this head, including both self-hosted GPU E2E lanes. Not reconciled against any other review thread — the only pre-existing activity on this PR was an automated security-scan comment reporting no findings, not human review feedback.
🤖 Automated by an AI agent
| # host while its xfail row stands, and the vLLM run stops before printing a | ||
| # plan, so the step panics there whatever the lemonade half did. It becomes | ||
| # regression coverage when that row is narrowed. serve-55 and the unit tests in | ||
| # regression coverage when that row is narrowed. serve-60 and the unit tests in |
There was a problem hiding this comment.
Non-blocking: this comment's cross-reference looks like a renumbering slip rather than a deliberate edit. Before this PR, serve-55 genuinely was about per-engine short-name refs (the scenario this comment sits above). serve-60 is this PR's own new smoke-failure-summary scenario, which is unrelated to short names or per-engine refs. The pointer is now dangling/wrong — worth restoring the original serve-55 reference (or whatever scenario actually covers that narrowing today) rather than pointing at this PR's own addition.
| # host while its xfail row stands, and the vLLM run stops before printing a | ||
| # plan, so the step panics there whatever the lemonade half did. It becomes | ||
| # regression coverage when that row is narrowed. serve-55 and the unit tests in | ||
| # regression coverage when that row is narrowed. serve-60 and the unit tests in |
There was a problem hiding this comment.
Non-blocking: this comment's cross-reference was changed from serve-55 to serve-60, but serve-60 is this same PR's new smoke-failure-summary scenario, which has nothing to do with per-engine short-name refs. serve-55 was (and still is) the scenario that actually covers that (confirmed via git log -S). This reads like an incidental renumbering slip rather than a deliberate edit — the comment now points a future reader at the wrong regression coverage once the xfail row here is narrowed.
Confidence 80 · mechanical · Fix: restore the serve-55 reference (or point at whichever scenario currently covers per-engine short-name refs).
| promise in the README and `rocm serve --help`. | ||
| - "no background automation daemon was left behind" is hygiene only: it records | ||
| that no daemon was started but does not discriminate the | ||
| `if !e2e_simulate_oom_launch()` guard in `serve_cmd.rs`, because no watcher is |
There was a problem hiding this comment.
Non-blocking: this new citation of serve_cmd.rs is a bare backtick path rather than a relative markdown link ([serve_cmd.rs](../apps/rocm/src/serve_cmd.rs)), per this repo's own docs-link convention (AGENTS.md §5). docs/testing.md isn't in that rule's exemption list, and this reads as a directed citation (pointing the reader at a specific guard in that file) rather than a generic/historical mention. lychee/docs-links won't catch this since it's not a broken link, just an unlinked one — which is exactly the gap that convention exists to close.
Confidence 65 · mechanical · Fix: link it, e.g. [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs).
| .expect("no interactive serve summary") | ||
| .screen_text(); | ||
| let flattened = screen_without_whitespace(&screen); | ||
| assert!( |
There was a problem hiding this comment.
Non-blocking: this then step (and its siblings added for serve-60) interpolates the full PTY {screen} into the panic message on assertion failure. Elsewhere in this same PR (the providers.rs unit tests, and serve_summary.rs's own test comments) the author is deliberate about not doing this, specifically to avoid echoing a summary that can carry the endpoint API key and tripping CodeQL's rust/cleartext-logging rule. No real leak happens here today — this scenario never passes --host, so the endpoint binds loopback and api_key is None — but the pattern now exists in two different shapes in the same codebase, and copying this then step as a template for a future public-bind scenario would carry the key into CI logs on a failing assertion.
Confidence 70 · architectural · Fix: assert on a redacted/derived value here too (or add a comment noting why it's safe in this scenario specifically, the way the unit tests do), so the safe pattern doesn't get overwritten by copy-paste.
Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary prose and adds its own serve scenario, so all three conflicts are with it. tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario becomes `serve-61` and moves after it, so each number still equals its position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two `-n "serve-60"` repro filters in docs are main's own and are left alone. docs/testing.md: two independent sections appended at the same place, kept whole. They are unnumbered `##` headings, so nothing renumbers. README.md: both sides rewrite the one paragraph describing the deployment summary, so this is a real merge rather than a choice between them. The summary sentence keeps this branch's `device:` row and main's endpoint/model/ smoke-test list; main's smoke-test-failure paragraph and this branch's "device line is omitted when the engine cannot say yet" paragraph both survive; and the shared closing sentence is kept once. Taking either side whole would have silently dropped the other's user-facing contract. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed ROCMAI-703 (#1050), which adds its own serve scenario and its own testing.md section, so both conflicts are with it. tests/e2e-cucumber/features/model_serving.feature: main took `serve-60`. Main's keeps 60; this branch's two become `serve-61` and `serve-62` and move after it, so each number still equals its position. Four prose references follow: the two scenario entries and the repro filter in `docs/testing.md` (now `serve-(61|62)`), the `(serve-62)` note in `service_launch/readiness.rs`, and the feature file's own "exactly like serve-60's crash" line. The other `-n "serve-60"` filters in docs are main's own and are left alone. docs/testing.md: two independent sections appended at the same place, kept whole. Unnumbered headings, so nothing renumbers. This is the third time these two scenarios have moved (57/58 -> 59/60 -> 60/61 -> 61/62); the `@id:` tags are the binding key and have never changed, so no step binding moved with them. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Main landed ROCMAI-703 (#1050), which rewrites the same deployment-summary prose and adds its own serve scenario, so all three conflicts are with it. tests/e2e-cucumber/features/model_serving.feature: main took `serve-60` for its smoke-test-failure scenario. Main's keeps 60; this branch's APU scenario becomes `serve-61` and moves after it, so each number still equals its position. `apps/rocm/src/therock/family.rs`'s prose reference follows. The two `-n "serve-60"` repro filters in docs are main's own and are left alone. docs/testing.md: two independent sections appended at the same place, kept whole. They are unnumbered `##` headings, so nothing renumbers. README.md: both sides rewrite the one paragraph describing the deployment summary, so this is a real merge rather than a choice between them. The summary sentence keeps this branch's `device:` row and main's endpoint/model/ smoke-test list; main's smoke-test-failure paragraph and this branch's "device line is omitted when the engine cannot say yet" paragraph both survive; and the shared closing sentence is kept once. Taking either side whole would have silently dropped the other's user-facing contract. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Fixes #523
Why
After
rocm servelaunched a service, the summary printedstatus: readywith usage instructions even when the first post-launch test request had failed, for example when the engine died mid-response. The only hint was that the two metrics readn/a, with no reason given.The cause is that
run_smoke_testreturned empty metrics on any error and discarded the error itself, while the summary decided "ready" from the launch-time status alone.What changes
SmokeOutcome) instead of bare metrics.Deployment summary (smoke test failed), the status readsstarted, smoke test failed, and a note says the smoke test could not complete, gives a one-line reason, and points torocm logs --service <id>. The reason is cleared of terminal control and format characters and length-capped; text after a cut-off CSI escape sequence is kept (an unterminated OSC, DCS or APC string sequence still swallows the rest of the reason, see the follow-ups).--no-smoke-testand already-running summaries are unchanged. Only the interactive summary runs the smoke test.docs/testing.mdlists what is and is not detected, and the README anddocs/manual-testing.mdpoint to that list.What is not detected
data:error payload. That is handled separately in fix(providers): fail an OpenAI-compatible stream on an in-band error payload #1056.data:line.Verification
serve-59scenario launches a service that reaches ready and then fails its first chat request, using a secondreadymode of the existing e2e fault-injection hook. It asserts, over a pty, the heading, the status row, the engine reason, exactly one chat request and the logs command, with exit 0. The scenario is tagged@requires-oom-fault-injection,@requires-no-gpuand@requires-os:linux, so it runs on the GitHub-hosted mock E2E lane (where xtask builds the binary with the hook) and resolves to skip on every prebuilt-binary lane (the self-hosted GPU lanes and the nightly lanes). The hook is compiled out of normal release builds, checked withstringson a release binary.cargo clippy -D warnings, fmt, the 45 smoke/summary tests, the ASCII test,check-module-size,check-architecture-doc, and theserve-41andserve-59e2e scenarios. The workspace tests pass on a Linux filesystem; on a case-insensitive host mount two unrelated tests (the storage case-twin test and the runtimes_cmd marker test) fail, identically onmain.hawkeye) and lychee did not run locally. CodeQL cannot run locally, so the assert changes were checked by inspection against therust/cleartext-loggingrule.Follow-ups (not in this PR)
--no-smoke-testsuppressing the new failure verdict has no test of its own: the gate inserve_cmd.rsis older code, no e2e scenario drives the flag, and the unit test for the skipped case renders default metrics by hand. A sibling scenario using the ready hook with--no-smoke-testwould cover it.one_line_reason.[DONE]without the terminating chunk, as failures. This strictness is pre-existing and is now visible.data:error payloads (fix(providers): fail an OpenAI-compatible stream on an in-band error payload #1056, chat: an error sent inside the stream is dropped and the answer is shown as complete #516).serve-41feature comment about the daemon step, and the README sentence order.Option<SmokeOutcome>inDeploymentSummaryinstead of two fields.Scope
Other open PRs edit the same summary code (#817, #678, #723, #725), so overlap with them is possible.
Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows.docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation)..rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.