Repository navigation
Conversation
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 0557c5d
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change. No ticket named in the PR description.
Blocking
-
Commit is not GitHub-"Verified"; the
commit-signatures/verify-commitsgate fails — head commit0557c5db. This repo's ownAGENTS.md§2 states signing/sign-off are enforced (not advisory): "a blocking CI gate (commit-signatures) re-checks every PR with GitHub 'Verified' status." The job output names the fix directly: sign off (git commit -s --amend/git rebase --signoff <base>) or enable commit signing and re-sign, then re-push.
Confidence 100 · mechanical · Fix: sign and/or sign off the commit per the job's own remediation text, then force-push. -
clippyfails the build:clippy::missing_const_for_fnonrecorded_status_after_stop,apps/rocm/src/serve_progress.rs:479, under this repo's-D warningsgate — the job output shows this is not advisory, it aborts compilation ("could not compilerocm(bin "rocm") due to 1 previous error").
Confidence 100 · mechanical · Fix: see the next finding — fixing that dead branch naturally resolves this too (const fnbecomes valid once the unreachable arm is simplified). -
New CodeQL alert
rust/cleartext-logging(security_severity: high),apps/rocm/src/serve_progress.rs:1068— flagsassert!(rendered.contains(&hint), "{rendered}")in the new testfailure_message_names_the_same_logs_command_as_the_summaryas writingapi_keyto a log sink. In this specific callapi_key: None, so no real secret is exposed today, but the scanner is reacting to the general shape: formatting a wholeDeploymentSummary(which legitimately carries a real API key whenSome) into a test-failure message is exactly the pattern that would leak one in a future test built the same way. This is a new, currently-open, failing required check on this commit.
Confidence 90 · architectural · Fix: assert a narrower, non-secret-bearing condition (e.g. compare just the logs-hint substring location without echoing the whole rendered block on failure), or justify/dismiss the alert explicitly if the team considers test-onlyNonedata out of scope. -
recorded_status_after_stop's_ => "starting"arm is dead, untested code —apps/rocm/src/serve_progress.rs:479-485. Tracingdecide_serve_startup(same file, ~line 403): the!process_alivebranch always returnsProcessExitedbefore any ofDownloadStalled/Silent/SilentLoad/Deadlinecan be reached, so everyServeStartupStopthis module ever constructs with one of those four reasons necessarily hasprocess_alive == true. The function's only producer isstop_decision, and no test ever builds a non-terminal-reason stop withprocess_alive: falseto reach the final arm.
Confidence 85 · logic · Fix: collapse to_ => "running"with adebug_assert!(stop.process_alive)(or a comment) documenting the invariant — this also makes the function triviallyconst fn, resolving the clippy failure above. -
apps/rocm/src/serve_summary.rs:180hand-rollsrocm logs --service {}instead of calling the sibling helperservice_logs_hintthat this same PR introduces and that the very same function already calls correctly two lines earlier (line 143).service_hints.rs's doc comment states the helper's entire purpose: "Shared so a startup failure and the deployment summary cannot name two different log commands." The two strings are byte-identical today, but nothing ties them together going forward — exactly the drift class the helper exists to prevent, violated within the PR that introduces it.
Confidence 85 · mechanical · Fix: replace the inlineformat!at line 180 withcrate::service_hints::service_logs_hint(&summary.service_id). -
apps/rocm/src/serve_cmd.rs:977:return Err(error.context(helper))makes the secondary daemon-helper failure the headline, demoting the real startup failure —anyhow::Error::context(self, context)keepsself(the startup failure, built fromServeStartupStop::user_message, which namesrocm logs/rocm services stop) as the cause and makescontext(the helper failure) the new top-levelDisplaymessage.main.rsprints errors via{e:?}(Debug), so the original message survives in the chain, but it's no longer the first line the user reads — backwards from the two errors' relative importance, and only reachable in the rare caseensure_background_helper_running_quietitself fails right after a startup stop.
Confidence 73 · logic · Fix: reverse the composition (e.g.helper.context(error)) so the serve-startup failure stays the headline.
Non-blocking
-
State-matrix doc table omits the deadline as an exit for "downloading"/"loading" rows —
apps/rocm/src/serve_progress.rs:19-20.decide_serve_startupchecks the overall deadline unconditionally before any phase-specific stall check (confirmed by the existing testwait_keeps_a_changing_download_alive_past_the_stall_limit, whose own comment says "a moving download must hit the deadline, not the stall limit"), so these two rows do exit on the deadline too — theunreachablerow's column correctly lists both ("silent limit, or the deadline"), these two don't. Code and tests are correct; only the table is incomplete.
Confidence 78 · mechanical · Fix: add "(or the deadline)" to both rows, matching theunreachablerow's phrasing. -
wait_for_serve_startupcallsobserve()+step()twice per 250 ms poll tick (once before the HTTP probe, once after), duplicating ~15 lines of near-identical logic. Correctness is unaffected (a terminal decision from the first call returns immediately, so the second never double-fires a stop/ready), but it can callon_viewtwice with the same label in one tick and is a simplification opportunity.
Confidence 70 · mechanical · Fix: factor the shared "observe, compute token/timing, step" sequence into one helper called twice, or restructure to a single call per iteration.
Decisions for the author
rocm_core::SERVE_STARTUP_DEADLINE(30 minutes) is shared verbatim between the interactive CLI wait's give-up deadline androcmd's background-recovery watcher's "don't restart a live, still-starting process" grace window (live_startup_still_in_progress) — tradeoff. This couples two independently-motivated concerns (interactive UX patience vs. background recovery safety) to one constant: tuning one later (e.g. shortening the interactive wait for snappier UX) would silently also shrink the watcher's grace window for every managed service, not just ones started interactively. Confirm this coupling is intentional rather than incidental.- The stall/silent timers are fixed global constants (
DOWNLOAD_STALL= 3 min,SILENT_UNREACHABLE= 2 min,SILENT_LOAD= 5 min), not configurable per engine — tradeoff. If any of the three supported engines (lemonade/vllm/aim) can legitimately go quiet longer than 5 minutes during a real (non-hung) load step, that engine would now hit a false "stalled" error under the new interactive path, where previously it was never an error (the old 45-second behavior only ever printed a time-boxed summary). Worth confirming this was checked against all three engines' real log cadence, not just the synthetic timings the unit tests use.
Positive signals
decide_serve_startupis a pure function fully decoupled from I/O/timing, letting every lifecycle row be driven directly by a unit test with exact phase/label/reason assertions rather than weak substring checks — and the tests do this well.- The "readiness never downgrades" invariant (the
bestvariable inwait_for_serve_startup, documented at lines 374-376) is an explicit, well-reasoned guard against flicker: one refused connection after the model has listed does not regress the status line. probe_service_http_readinessis shared between the interactive wait and the historical polling path specifically so the status line and the real readiness check "cannot disagree about ready" — a clean elimination of a split-brain bug class.service_launch.rs'sstart_managed_servicealways persists the updatedManagedServiceRecord(status,inference_verified_at, audit log) before ever branching tobail!on a startup stop — there is no path that mutates state and fails to persist it.- Interactive vs. piped mode is correctly gated on
IsTerminal::is_terminal(&stdout())(serve_cmd.rs:845), so none of the new stall-detection/hard-error behavior reaches a script or CI pipe — the lenient 45-second status-budget path is untouched for non-interactive callers. - The PR's own test-plan is unusually candid about what wasn't validated locally ("Full workspace
cargo testand clippy were not run"), and that disclosure correctly predicted the CI failures found above.
Deployment notes
The shared SERVE_STARTUP_DEADLINE constant changes rocmd's recovery-watcher behavior for any managed service (not only ones started under the new interactive path) that is less than 30 minutes old, still "running" with a live pid, but endpoint-unreachable: it will no longer be auto-recovered during that window, where it may have been before this change. This looks intentional (it's exactly what the new recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one test pins), but it's worth calling out explicitly for anyone operating rocmd in production, since it widens the window before automatic recovery kicks in for any stuck or slow-starting service, not just ones launched interactively.
What this covered
Read completely: apps/rocm/src/serve_progress.rs (new, 1447 lines, including all tests), serve_cmd.rs, serve_summary.rs, service_launch.rs, service_hints.rs, engine_runtime_select.rs, main.rs, apps/rocmd/src/watchers.rs, crates/rocm-core/src/lib.rs and net.rs, xtask/src/ascii_output_contract.rs, README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, and this repo's AGENTS.md in full. Diff resolved against cc56179c…0557c5db by fetching the PR head read-only (git fetch https://github.com/ROCm/rocm-cli.git pull/817/head); the PR was never checked out. Work was split across three independent, blind, file-scoped fan-out workers (core state-machine module; wiring/integration layer; supporting infra + docs), then consolidated and every surviving finding re-verified against the source here, plus a design-review pass and a check of this PR's live CI status (get_check_runs) and code-scanning alerts.
Did not run: cargo build/test/clippy locally (relied on CI's own run, which failed — see Blocking); the e2e-cucumber/GPU-gated suite (no GPU available here, and the PR itself states none was run); a direct check of the three serving engines' (lemonade/vllm/aim) real log cadence against the new silent-load/download-stall timers (raised as an open question under Decisions for the author instead, since it can't be settled from the diff alone). The history/prior-changes passes were done lightly (confirmed these files' recent history is module-extraction refactors with no directly relevant prior review threads surfaced), not exhaustively. Not reconciled against any PR discussion or comments — the existing github-advanced-security[bot] comment was read only for its CodeQL alert detail, not treated as review discussion.
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for this. Keeping startup progress visible until the server is ready or stalled is the right fix, and pulling the decision into a pure decide_serve_startup with a test per state is a nice structure. A few things need fixing before this can merge.
Blocking
- CI is red on this head for three checks that come from this change:
clippy(missing_const_for_fnonrecorded_status_after_stop),Commit signatures + sign-off, and CodeQL (rust/cleartext-loggingon the test assert atserve_progress.rs:1068). Because clippy fails, the unit-test jobs were skipped, so none of the new tests have run in CI yet. recorded_status_after_stophas an unreachable_ => "starting"arm (details inline). It is dead code, and it is what clippy trips on.- The PR body doesn't show that the new tests fail when the code they cover is broken (revert a guard, swap a branch, confirm red), and it says full workspace test and clippy weren't run. Please state which mutations you tried and rerun the full gate locally.
Worth addressing (inline comments)
- Any
start_managed_serviceerror now starts the background helper and masks the real startup failure message. - The rocmd recovery guard may not cover a record that is still
starting, or one left behind by Ctrl-C. - Stall limits (2 / 5 min) can now fail a server that would have become ready, and they aren't documented.
- Engine log lines are echoed to the spinner and error text with only ASCII filtering. Launch args can contain tokens, so some redaction seems prudent.
- The doubled observe/step block, and a stale "half-minute" in
docs/manual-testing.md.
Also: the rocmd behavior change, the new rocm-core deadline constant and the service_logs_hint refactor aren't mentioned in the PR body. They touch another binary, so I'd list them. The two Strix Halo Ubuntu E2E lanes also failed after ~45 min; I didn't dig into those logs, so please check they aren't related.
| /// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one` | ||
| /// leaves alone until [`rocm_core::SERVE_STARTUP_DEADLINE`]. Engine failure | ||
| /// and a dead process are `failed`, which recovery is supposed to restart. | ||
| pub(crate) fn recorded_status_after_stop(stop: &ServeStartupStop) -> &'static str { |
There was a problem hiding this comment.
The _ => "starting" arm can't be reached: decide_serve_startup returns ProcessExited before any other reason when the process is dead, so every other reason has process_alive == true. Collapse it to _ => "running" (with a comment or debug_assert!) and make the fn const fn. That also clears the missing_const_for_fn clippy failure.
There was a problem hiding this comment.
Fixed in 1a9dfb7: the unreachable arm is gone, every stall/deadline reason maps to running with a debug_assert that the process is alive, and the fn is const.
| notes: Vec::new(), | ||
| }; | ||
| let rendered = crate::serve_summary::render_summary(&summary); | ||
| assert!(rendered.contains(&hint), "{rendered}"); |
There was a problem hiding this comment.
CodeQL flags rust/cleartext-logging on "{rendered}" here, since the summary carries the api key. Assert without echoing the whole rendered summary (e.g. print only hint).
There was a problem hiding this comment.
Fixed in 1a9dfb7: the assert message names only the hint, not the rendered summary.
| let mut token = String::new(); | ||
| let mut best = EndpointReadiness::Unreachable; | ||
|
|
||
| loop { |
There was a problem hiding this comment.
The observe / token / step block is repeated again around line 589. I'd pull it into one helper. watch_engine is also applied in both adjusted_signals and decide_serve_startup, and step fires on_view twice per loop, so the spinner ticks at double rate.
There was a problem hiding this comment.
Fixed in 1a9dfb7: each poll now does one observe, one probe (skipped when the engine failed or the process is gone), one decision and at most one status-line update. effective_signals is the single watch_engine adjustment. wait_observes_and_updates_the_status_line_once_per_poll fails if the engine is sampled or the line drawn twice per poll.
| // The process is already recorded. Start supervision before | ||
| // returning the startup failure so a server that is still | ||
| // running is not left with no daemon. | ||
| if let Err(helper) = ensure_background_helper_running_quiet(summary_mode) { |
There was a problem hiding this comment.
Any error from start_managed_service reaches this arm, including a spawn failure where nothing was recorded, so the comment "the process is already recorded" isn't always true and the helper starts needlessly. error.context(helper) also makes the helper error the headline and pushes the startup-failure text (what the user needs to read) down to "Caused by". A typed error for the stop case would avoid both.
There was a problem hiding this comment.
Fixed in 1a9dfb7: start_managed_service returns a typed ServeStartupFailed for a startup stop, and only that error starts the helper (supervise_after_launch_error, unit-tested on all three rows). A helper that fails to start is printed as a warning; the startup failure stays the error.
| // A live process that has not opened its port is still the | ||
| // interactive serve wait's job. Restarting it here kills a | ||
| // download the CLI is reporting. A dead pid is not spared. | ||
| if record.status == "running" && live_startup_still_in_progress(&record, now) { |
There was a problem hiding this comment.
This guard only spares status == "running". If the record is still starting during the wait (status looks to be written only after it), or is left over after a Ctrl-C of a wait that can now last up to 30 minutes, stale-start recovery could still restart a live download. Is starting covered? The new test only builds a hand-made running record.
There was a problem hiding this comment.
Good catch. The CLI writes running before the wait, but rocm services persists the engine's own starting state into the record, and that was restarted after 5 minutes. Fixed in 1a9dfb7: a live starting record is spared until SERVE_STARTUP_DEADLINE, the same as running; a Ctrl-C leaves one of those two, so it is covered. New test: recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline (inside the deadline, dead pid, at the deadline).
| Some(service_id), | ||
| ); | ||
| if let Some(stop) = startup_stop { | ||
| bail!("{}", stop.user_message(requested_model, engine, service_id)); |
There was a problem hiding this comment.
This bail! turns a stall into a non-zero exit for a server that may still become ready (Silent at 2 min, SilentLoad at 5 min, e.g. a container pull that logs elsewhere, or a long weight load / graph capture with no output). Previously interactive serve printed a "not ready" summary and exited 0. Could you confirm the limits against real vllm/lemonade/aim log cadence and document them in README / manual-testing?
There was a problem hiding this comment.
Documented in README, manual-testing and testing.md in 1a9dfb7, including the non-zero exit and that the server keeps running. I have not measured real vllm/lemonade/aim log cadence against the 2/5-minute limits; that needs a GPU lane.
| The default command, without `--foreground` or `--verbose`, leaves the server | ||
| in the background. While it starts, the status line follows the engine: a | ||
| download percent, a load line, or a wait for the model to serve. A model that | ||
| is still loading after the first half-minute keeps that line; it is not |
There was a problem hiding this comment.
"after the first half-minute" disagrees with the 45-second piped budget in docs/testing.md; please reword.
0557c5d to
1a9dfb7
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 1a9dfb7
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (base cc56179c…head 1a9dfb75, the review-fix commit on top of the previously-reviewed 0557c5db). Implements "keep startup progress until the server is ready or stalled" for interactive rocm serve. This commit is itself the author's response to two earlier review rounds (one from this same automation, one human); all three of their blocking CI findings (clippy missing_const_for_fn, the CodeQL rust/cleartext-logging alert, commit-signatures) and the dead _ => "starting" arm are fixed — confirmed by re-reading the current code, not assumed.
Blocking
-
No CI check has executed for this exact head commit.
get_status/get_check_runsfor1a9dfb75333f2c52c1f0210986587da2e4ca68dcreturn only a single Read-the-Docs status;actions_listfinds no workflow run anywhere referencing this sha (the three runs on this branch are all still pinned to the superseded0557c5db). None ofbuild-and-test,clippy,Test (affected crates),windows-build-and-test,Commit signatures + sign-off,CodeQL, or the E2E suite have run on the commit actually being merged. Per this method's own rule, an unverified class is a finding, not an assumption.
Confidence 100 · Fix: push a no-op commit or otherwise re-trigger CI for this head and confirm green before merge. -
service_launch.rs's attached-serve path (run_attached_service) still hand-rollsrocm logs <id>/rocm logs {service_id}at three call sites (lines ~949, 994, 1035), bypassing theservice_logs_hinthelper this very commit introduces to prevent exactly this drift.rocm logs(main.rs) parses a bare positional as a free-text search query, not a service id — only--service <id>selects one service's log (confirmed against the clap definition). So these three hints are not equivalent toservice_logs_hint(id)("rocm logs --service {id}") and are likely to find nothing when run as printed.service_hints.rs's own doc comment states the helper exists "so a startup failure and the deployment summary cannot name two different log commands" — this commit correctly routesserve_summary.rsthrough it but missed the sibling site in the same file family.
Confidence 80 · logic · Fix: replace all three call sites withservice_hints::service_logs_hint(&service_id). -
The interactive wait can now issue an unthrottled real inference probe (
openai_chat_completion_probe, 8s timeout) on every ~250ms poll for up to the full 30-minuteSERVE_STARTUP_DEADLINE, once the model reachesListing.probe_service_http_readiness(engine_runtime_select.rs) calls the raw probe directly rather than through the throttled/latchedINFERENCE_PROBE_RETRY_INTERVAL(15s) pathcrates/rocm-core/src/net.rsdocuments and that commitfdfa620(#150) added specifically for this scenario ("One observed lemonade serve listed the model in 4s and could not answer for the next 6 minutes… the unthrottled cost lands exactly where it is most visible"). Previously this same unthrottled probing was bounded to 45 seconds; it is now reachable for up to 40x longer with no comment indicating the tradeoff was deliberately re-accepted.
Confidence 75 · architectural · Fix: route through the throttled probe helper (engine_state_inference_verified/managed_service_endpoint_model_ready), or explicitly document why the interactive path accepts the extra probe volume.
Non-blocking
-
engine_runtime_select.rs's module doc (lines 76-77) still claimswait_for_service_http_ready_with_progressis "read fromservice_launch.rs", but this PR's diff removes that import/call entirely (confirmed via repo-wide grep: the only remaining caller is the internalwait_for_service_http_readywrapper). This exact module recently had an identical class of stale-caller-claim fixed days earlier (commit9301dfc, #804) — the same drift has recurred one paragraph away.
Confidence 85 · mechanical · Fix: update the claim to name the real caller, or drop it in favor of the repo'sdocs/architecture.mdcross-reference convention. -
No unit test in
service_launch.rsexercises the new persistence/typed-error wiring instart_managed_servicedirectly (the "persistlaunch_statusbefore returningErr(ServeStartupFailed(...))" ordering, and thereadiness == Serving && startup_stop.is_none()guard). The decision table itself (decide_serve_startup) is well unit-tested inserve_progress.rs, but the call-site wiring added here has zero test coverage of its own.
Confidence 70 · architectural
Decisions for the author
live_startup_still_in_progress(rocmd's recovery watcher) trusts raw PID liveness with no start-tick identity check, over a 30-minute window — tradeoff. Elsewhere in this crate, PID-trust decisions are paired with a start-tick token specifically because of PID reuse (service_record.rs'snames_same_processesdoc comment); this new guard checks onlyprocess_is_running(pid). The bad outcome (PID reuse during the 30-minute window causing a dead service to be wrongly treated as still starting) is low-probability but non-zero, and this is a real, if narrow, widening from the previous purely time-based 5-minute check. Confirm this is an accepted tradeoff for the initial release of this guard.SERVE_STARTUP_DEADLINE(30 min) is shared verbatim between the interactive CLI wait's give-up deadline and rocmd's recovery-watcher grace window — tradeoff, already raised in the prior round and still true: tuning one later silently reshapes the other.
Positive signals
decide_serve_startupremains a pure, fully unit-tested state machine with one test per lifecycle row; the prior round's dead-arm/clippy finding is cleanly resolved (collapsed to_ => "running"), andconst fnnow compiles.- Credential redaction is now applied to engine log lines before they reach the spinner/status line or failure text (
redact_log_line, maskingapi_key, bearer/basic auth,hf_tokens, compact JSON key fields) — directly answers the prior round's "no redaction" concern, verified by reading the implementation. supervise_after_launch_error's error composition was specifically re-verified this round and is correct:start_managed_servicereturnsErr(ServeStartupFailed(...))with no further.context()wrapping, sodowncast_ref::<ServeStartupFailed>()matches at the top level and the startup-stop message stays the user-visible headline — this fixes the prior round's confidence-73 finding about a.context()headline swap; no further chain-walking assumption was needed.- The "readiness never downgrades" (
best) invariant and the sharedprobe_service_http_readinesshelper (eliminating a split-brain between the status line and the real readiness check) are both sound, well-documented designs. - Interactive vs. piped dispatch is correctly gated on
IsTerminal::is_terminal(&stdout()), so none of the new stall-detection/hard-error behavior reaches a script or CI pipe.
Deployment notes
- Unchanged from the prior round:
SERVE_STARTUP_DEADLINE(30 min) widens rocmd's auto-recovery grace window for any managed service that isrunning-but-endpoint-unreachable and under 30 minutes old, not just ones launched interactively. - Engine log lines now undergo redaction before reaching any user-visible surface — a behavior change worth noting for anyone who greps raw
rocm serveoutput for a credential during debugging.
What this covered
Read completely at head 1a9dfb75333f2c52c1f0210986587da2e4ca68dc (base cc56179c): serve_progress.rs, serve_cmd.rs, serve_summary.rs, service_launch.rs, service_hints.rs, engine_runtime_select.rs, apps/rocmd/src/watchers.rs, crates/rocm-core/src/lib.rs/net.rs, main.rs's touched regions, README/docs/testing/manual-testing diffs, and the full AGENTS.md. Six independent blind fan-out workers covered: the core state machine (context for this round); serve_cmd/serve_summary/service_hints wiring; service_launch/engine_runtime_select (two independent passes); the agent-instruction/history/code-comments source-reading pass (two independent passes); rocmd watchers + the shared deadline constant. Every candidate surfaced by those workers was re-verified against the source before inclusion above; several (the three prior round's CI failures, the headline-swap concern) were independently re-confirmed as now fixed rather than carried forward on faith.
Did not run: cargo build/test/clippy locally (no Rust toolchain in this environment) — relied on direct code tracing plus the CI-is-absent finding above, since there is no green run to lean on for this commit. The three supported engines' (lemonade/vllm/aim) real log cadence against the new silent-load/download-stall timers could not be checked from the diff alone (same open question as the prior round). Not reconciled against this PR's own review discussion, which was not read.
🤖 Generated with Claude Code
1a9dfb7 to
cba35c6
Compare
|
Response to the automated review of Blocking:
Non-blocking:
Decisions:
Also from the rebase: |
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head cba35c6
Review — needs work
Full review of the whole change (base 1530f6b1…head cba35c61). No ticket named in the PR description. This is the third review round on this PR; the prior two rounds' blocking findings (clippy missing_const_for_fn, a CodeQL cleartext-logging alert, a missing commit-signature, a dead _ => "starting" arm, an unthrottled inference probe, and a hand-rolled rocm logs <id> hint) are all confirmed fixed by direct re-reading of the current code, not assumed.
Blocking
-
CI is currently red on this exact head.
Windows build and test (debug)(and the gatingwindows-build-and-testcheck) fail with real test failures inside this PR's own newserve_progress.rstests:wait_stops_a_hung_download_before_the_deadline(assertion left == right failed: left: Deadline, right: DownloadStalledatapps/rocm/src/serve_progress.rs:1524),wait_keeps_a_changing_download_alive_past_the_stall_limit, andpiped_wait_ignores_engine_failure_and_a_dead_process. These look like real-clock/thread::sleep-based timing tests that are not robust to Windows' coarser scheduling/timer resolution — the PR's own "Not verified" note only flags thecfg(not(windows))integration tests inservice_launch::start, not these platform-neutralserve_progress.rsunit tests, which it implies already run clean on Windows.
Confidence 100 · logic · Fix: make the timing-sensitive assertions robust to coarser clocks (fake/injectable clock rather than real sleeps, or wider margins), and confirm green on the Windows runner before merge. -
service_logs_hint(new this PR) skips the module's own mandatoryshell_quotestep —apps/rocm/src/service_hints.rs:51-53. The module's doc comment states the invariant directly: "The service id goes throughshell_quotehere, once, so every hint is a line whose words are exactly the argv it names" — and the siblingservice_action_hint/remote_service_stop_hintin the very same file both call it.service_logs_hintdoes not.ServiceIdvalidation makes this unlikely to bite today, but it is a confirmed violation of an explicitly adopted, documented pattern in the one module whose stated purpose is preventing exactly this class of drift (it even cites a prior incident, #417).
Confidence 88 · mechanical/architectural · Fix: wrapservice_idinshell_quote(service_id)the same way the sibling functions do; add a hostile-id test alongside the existingservice_hints.rscoverage. -
docs/architecture.md'sservice_hints.rsentry is incomplete — it listsserve_progress.rsas a new caller but omitsserve_cmd.rs, which newly callsservice_hints::service_logs_hintdirectly atapps/rocm/src/serve_cmd.rs:1182(confirmed: noservice_hintsreference existed inserve_cmd.rsbefore this PR). The adjacentshell_quote.rsentry's blanket claim of covering "theservice_hints.rshints" is now also inaccurate given the finding above.cargo xtask check-architecture-doconly checks file presence, not prose accuracy — perAGENTS.md§6, "the check reads no prose, so keeping an entry's description true is still on you."
Confidence 85 · mechanical · Fix: addserve_cmd.rsto theservice_hints.rsentry's caller list. -
No Gherkin scenario was added for new, CLI-observable interactive-serve behavior, despite this repo's own established convention for exactly this situation.
AGENTS.md§3 requires user-observable behavior (status-line output, exit codes) to be covered by a scenario, and explicitly anticipates a GPU-only scenario: "if the scenario can only run on a gated lane (@requires-gpu,@nightly), say so in the PR text and name the lane" — i.e. write it and tag it, not omit it.tests/e2e-cucumber/features/model_serving.featurealready carries 30 existing@requires-gpu-tagged scenarios (confirmed by direct grep), so this is a well-established, actively-used pattern in this exact file. This PR's diff to that file adds only an explanatory comment ("There is no Gherkin scenario for this yet... a scenario would be@requires-gpu"), not a tagged scenario.
Confidence 80 · architectural (agent-instruction adherence) · Fix: add a@requires-gpuscenario for the core happy-path (status line follows a download/load, then prints the deployment summary) alongside the existing unit coverage, per the established pattern in this file. -
Branch-mutation test gap: the Unreachable and Loading silent-stall checks in
decide_serve_startupcan't be told apart from a field-swap regression. The Unreachable check readstiming.elapsed(serve_progress.rs:536-538) and the Loading check readstiming.since_progress(serve_progress.rs:548-550), but every test covering either branch (unreachable_keeps_the_starting_label_until_the_silent_limit,unreachable_with_no_log_stops_when_nothing_arrives,loading_past_the_old_budget_keeps_reporting_the_log_line,loading_with_no_new_log_stops) passeselapsed/since_progressas either equal or moving in the same direction — directly confirmed by reading the fixtures. Swapping the two fields in either branch would pass the whole suite. The analogous Downloading-phase check has exactly this protection (wait_keeps_a_changing_download_alive_past_the_stall_limitkeepssince_progressresetting whileelapsedgrows past the stall limit); Unreachable/Loading have no equivalent.
Confidence 82 · logic/test-quality · Fix: add one integration-level test per phase (mirroringwait_keeps_a_changing_download_alive_past_the_stall_limit) that keeps the relevant progress signal moving while overall elapsed time grows, proving the deadline alone — not the silent-stall check — is what fires. -
Potential panic in
mask_valueon a multi-byte UTF-8 trailing character —apps/rocm/src/serve_progress.rs:821-835.word.rfind(|c| ...)returns the byte offset where the matched character starts; the code then doesindex + 1to getbody_end, which is only a valid slice boundary for a 1-byte (ASCII) character. For a value ending in e.g.é(2 bytes),body_endlands mid-character and&word[body_end..]panics ("byte index N is not a char boundary"). Traced and confirmed by byte-level analysis. This is not reachable in the currently-wired production path only because every real caller (sanitize_log_line→latest_meaningful_log_line, used byobserve_launch) ASCII-filters the line beforeredact_log_line/mask_valueever see it — an unenforced calling convention on aStringdocumented only as "already one sanitized line," not a type-level guarantee, and no test in the file exercisesredact_log_line/mask_valuewith non-ASCII input.
Confidence 85 · logic · Fix: computebody_endfrom a char boundary (e.g.word.char_indices().rev().find(...), orword.len()minus the matched char's byte length) rather thanindex + 1; add a non-ASCII regression test.
Non-blocking
-
is_secret_name's suffix allowlist misses common variants likesecret_key/private_key—apps/rocm/src/serve_progress.rs:792-806. It matches vianame.ends_with(suffix)against["api_key", "apikey", "token", "password", "secret"]; a name such assecret_keyhas "secret" as a prefix, not a suffix, soends_with("secret")is false and the value is never masked. Verified directly against the suffix list. Narrower than what the redaction feature is billed as doing, though within the PR's own stated scope ("api-key/token/secret values") it's a partial, not total, miss.
Confidence 70 · logic · Fix: also match on these names as whole words/components (split on_/-and check membership), not only as a trailing suffix. -
cli.rs's new serve help text overclaims for the "reuse an already-starting service" path. The new text says "while it starts, the status line follows what the engine is doing... until the server is ready or that wait fails" (apps/rocm/src/cli.rs, theservedoc comment). But for a plainrocm serve <model>(no--wait) that matches an equivalent service another invocation is still starting,wait_on_existing_service(apps/rocm/src/service_launch/readiness.rs:393) returns immediately with the untouched"starting"status —if !wait_until_ready || report.status == "ready" { return Ok(report); }— before ever invoking the wait loop or the "follows the engine" behavior at all. This is pre-existing behavior for the reuse path, not newly introduced, but the new, broader help text makes no exception for it.
Confidence 75 · architectural/doc-accuracy · Fix: scope the help text's claim to a fresh launch, or note the reuse-of-starting-service case explicitly. -
A manifest-write I/O failure in the "server still running" branch of
start_managed_servicecan bury the real startup-stop diagnosis —apps/rocm/src/service_launch/start.rs:185-189followed by the typed-failure return at line ~203. When a stall/deadline stop leaves the server running, the code callsrecord_wait_outcome(paths, &mut record, launch_status, ...)?before converting the stop into the typedServeStartupFailedthe user is meant to see; if that write fails, the raw I/O error propagates instead of the actionable stop reason. The sibling "ended" branch just above is deliberately robust to this (settle_ended_launch/settle_exited_servicepersist best-effort vialet _ = ...write()), so this is a real asymmetry introduced by this refactor.
Confidence 72 · logic · Fix: make the write best-effort here too (log/ignore a write failure) so the typed stop is still returned.
Decisions for the author
- List-valued secrets leak all but their first item — tradeoff, already self-documented in
redact_log_line's own doc comment ("A list value ('api_key': ['a', 'b']) has only its first item masked"). vLLM accepts repeated--api-keyflags, so a printed launch-args dict with multiple keys would leak every key after the first. Confirm this is an accepted scope boundary for now rather than something to close in this PR. rocm_core::SERVE_STARTUP_DEADLINE(30 min) is still shared verbatim between the interactive CLI wait's give-up deadline androcmd's recovery-watcher grace window — tradeoff, raised in both prior rounds and unchanged. Tuning one later (e.g. a snappier interactive UX deadline) silently reshapes the daemon's auto-recovery window for every managed service, not just interactively-started ones.live_startup_still_in_progress(rocmd's recovery watcher) still trusts raw PID liveness with no start-tick identity check, over the full 30-minute window — tradeoff, raised in the prior round and unchanged. PID reuse during that window could cause a dead service to be wrongly treated as still starting; low-probability but non-zero, and this crate has an existing start-tick pattern elsewhere for exactly this reason.- Two parallel inference-probe-throttling mechanisms now exist — the disk-persisted one in
crates/rocm-core/src/endpoint_readiness.rs/service_record.rs(for the background health-check path) and the new in-memoryInferenceProbeThrottleinserve_progress.rs(for the interactive wait). They share only the numeric constant (INFERENCE_PROBE_RETRY_INTERVAL), not the throttling logic itself. Reasonable given the different contexts (cross-invocation persistence vs. in-loop state), but worth confirming the duplication is intentional rather than something to unify later.
Positive signals
- The prior round's blocking CI findings (clippy, CodeQL cleartext-logging alert, missing commit signature) and the dead
_ => "starting"match arm are all cleanly resolved, confirmed by direct re-reading rather than assumed. - Inference-probe backoff is now correctly implemented and well tested (
InferenceProbeThrottle, exact gap-sequence assertions from 1s to 15s) — this resolves the prior round's blocking finding about an unthrottled probe reachable for the full 30-minute wait. apps/rocm/src/service_launch/attach.rs's three previously hand-rolledrocm logs <id>strings are now fully replaced with the sharedservice_hints::service_logs_hinthelper, backed by a dedicated test (attached_logs_rows_select_the_service_log) that round-trips the printed command through the real CLI parser.wait_for_serve_startupnow performs exactly one observe/probe/decide pass per poll tick, resolving the prior round's "double observe per tick" note.- The "readiness never downgrades" invariant in
wait_for_serve_startup'sbestvariable is correctly enforced on every path (independently confirmed by two fan-out workers and the consolidating pass). rocmd's new recovery-watcher guard is tested at precise boundary conditions (age exactly at the deadline; dead vs. live pid; bothrunningandstartingstatuses).- The new state machine's extraction into its own
serve_progress.rsfile, rather than growing an existing file, matches this repo's documented "new subsystems get their own file from day one" guardrail, and stays comfortably under the 1,500-production-line cap (~859 lines).
Deployment notes
SERVE_STARTUP_DEADLINE(30 min) widensrocmd's auto-recovery grace window for any managed service that isrunning-but-endpoint-unreachable and under 30 minutes old, not only ones launched interactively (unchanged from prior rounds).- Engine log lines now undergo redaction before reaching any user-visible surface (status line, failure text) — worth noting for anyone debugging by grepping raw
rocm serveoutput for a credential, though see the suffix-allowlist gap above for a residual limitation. - Two GPU-hardware E2E lanes (
E2E tests (MI300X, current),E2E tests (W7900, current)) were still queued and had not completed as of this review's CI check; their result is unverified here.
What this covered
Read completely: all 19 changed files (apps/rocm/src/serve_progress.rs — new, ~1961 lines including tests; serve_cmd.rs, serve_summary.rs, service_hints.rs, engine_runtime_select.rs, cli.rs, main.rs's touched region; service_launch/{attach,readiness,start}.rs; apps/rocmd/src/watchers/server_recover.rs; crates/rocm-core/src/{lib.rs,endpoint_readiness.rs,net.rs,service_record.rs}; README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, xtask/src/ascii_output_contract.rs), plus this repo's root AGENTS.md in full. Diff resolved against 1530f6b1…cba35c61 by fetching the PR head read-only (git fetch origin +pull/817/head:refs/remotes/pr/817); the PR was never checked out.
Three independent, blind, file-scoped fan-out workers ran and reported: the core state-machine module (serve_progress.rs); the wiring/integration layer (serve_cmd.rs/serve_summary.rs/service_hints.rs/engine_runtime_select.rs/cli.rs/main.rs); and the service_launch module (attach.rs/readiness.rs/start.rs). Every candidate they surfaced was independently re-verified against the source here before inclusion or rejection.
Degraded for three planned passes: this environment hit a shared concurrency limit (repeated "concurrent subagent limit reached" errors from unrelated sessions sharing the host) that prevented spawning independent fan-out workers for (a) the rocmd/rocm-core watcher pass, (b) the docs/testing/xtask pass, and (c) a dedicated agent-instruction/history/prior-changes pass. I performed all three myself directly — reading every file in those groups completely, cross-checking git history and blame, and checking AGENTS.md compliance by hand — so that portion of the review ran single-context rather than as an independent blind pass; findings from it (e.g. the architecture.md staleness, the Gherkin-scenario gap, the help-text and manifest-write findings) should be weighed with that in mind, though each was independently verified against the source.
Live CI status was checked directly (get_check_runs/get_status) against this exact head commit, which surfaced the Windows test failures above; this was not something any fan-out worker was tasked with. Not reconciled against this PR's own review discussion or prior review comments, which were read only for the duplicate-review claim-check (commit IDs), not as review content.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 442ae1e9
Review — needs work
Full review of the whole change (base 1530f6b1…head 442ae1e9). No ticket named in the PR description.
Blocking
-
No Gherkin scenario added for the new, CLI-observable interactive-serve behavior, despite this repo's own established convention for exactly this case.
AGENTS.md§3 requires user-observable behavior (status-line output, exit codes) to be covered by a scenario, and explicitly anticipates a GPU-only one: "if the scenario can only run on a gated lane (@requires-gpu,@nightly), say so in the PR text and name the lane" — write it and tag it, not omit it.tests/e2e-cucumber/features/model_serving.featurealready carries 30 existing@requires-gpu/@nightly-tagged scenarios, an established, actively-used pattern in this exact file. This PR's diff to that file (tests/e2e-cucumber/features/model_serving.feature:3-10) adds only an explanatory comment ("There is no Gherkin scenario for this yet... a scenario would be@requires-gpu"), not a tagged scenario, anddocs/testing.md's matching paragraph relies on unit tests instead — which AGENTS.md §3 explicitly rules out for this purpose ("a unit test asserting the internal helper does NOT discharge this; it proves the function, not the behavior"). The repo also already has a proven no-GPU technique for a structurally similar case (serve-41,@requires-no-gpu, using thee2e-oom-fault-injectionseam to fabricate a failed-launch state) that the PR's comment doesn't address as an alternative.
Confidence 85 · architectural (agent-instruction adherence) · Fix: add an@requires-gpu(or a fabricated-state@requires-no-gpu, following theserve-41pattern) scenario for at least the ready-path and one failure path, per this file's established convention. -
service_hints::service_logs_hintskips the module's own mandatedshell_quotestep, anddocs/architecture.mdrepeats the now-false claim that it doesn't.apps/rocm/src/service_hints.rs:51-53:pub(crate) fn service_logs_hint(service_id: &str) -> String { format!("rocm logs --service {service_id}") }— noshell_quotecall. The module's own doc comment states the invariant directly: "The service id goes throughshell_quotehere, once, so every hint is a line whose words are exactly the argv it names" — and the siblingservice_action_hint/remote_service_stop_hintin the very same file both call it;service_logs_hintdoes not.rocm_core::ServiceId::newonly rejects empty/whitespace-only ids, path separators,.., and control characters — it does not reject spaces,;,$(), or quotes — so a legally-accepted id breaks the "hint names exactly the argv it runs" guarantee this file exists to provide (and cites a prior incident for, #417).docs/architecture.md's entry for this file repeats the same now-inaccurate claim verbatim ("It quotes the service id withshell_quote, so a hint names exactly the argv it runs."). The file's own proptest (a_hint_denotes_the_argv_it_names) covers the two sibling builders but notservice_logs_hint, so nothing in the suite catches this.
Confidence 90 · mechanical/architectural · Fix: wrapservice_idinshell_quote(service_id)the same way the sibling functions do; add it to the existing proptest's hint list; the architecture.md entry becomes accurate once the code is fixed. -
The
Unreachable/silent-port stall check indecide_serve_startupusestiming.elapsedinstead oftiming.since_progress, unlike both of its sibling checks, and unlike the behavior this PR's own README documents.apps/rocm/src/serve_progress.rs:536-538:if view.phase == ServeStartupPhase::Unreachable && timing.elapsed >= policy.bounds.silent_unreachable. TheDownloadingcheck (:524-525) and theLoadingcheck (:548-549) both correctly key offtiming.since_progress(time since the state last changed), andREADME.md's own new text describes this exit as "when the port stays closed with no engine log for 2 minutes" — a duration of silence, not total time since launch. Becauseclassify()(:383-436) falls intoUnreachablewhenever the latest tailed log window has nois_meaningfulline at all (readiness not yetListing), a server that produced real download/load output for most of a long startup and then has a brief gap (a scrolled-past tail window, a quiet transition between phases) would be judged by totalelapsed, not by how long it has actually been silent — so a wait already past 2 minutes of total elapsed age can fail immediately withSilentthe moment it first goes quiet, even though the silence itself just started. No test distinguishes the two fields for this branch:unreachable_with_no_log_stops_when_nothing_arrivespassesat(SILENT_UNREACHABLE, SILENT_UNREACHABLE)(both fields equal), so a field swap here would not be caught — unlike the analogousDownloadingcase, which has a dedicated integration test (wait_keeps_a_changing_download_alive_past_the_stall_limit) provingsince_progress, notelapsed, governs.
Confidence 85 · logic · Fix: usetiming.since_progressfor theUnreachablecheck, matching its siblings and the README's own description; add an integration-level test mirroringwait_keeps_a_changing_download_alive_past_the_stall_limitfor this phase. -
rocmd's new "don't restart a live, still-starting service" guard trusts a bare PID-liveness check, with no process-identity verification, even though the crate already ships the exact mechanism for this and the needed data is already on the record.apps/rocmd/src/watchers/server_recover.rs:403-413(live_startup_still_in_progress) checks onlyprocess_is_running(pid)forrecord.supervisor_pid/record.engine_pid.crates/rocm-core/src/service_record.rsalready persistssupervisor_start_ticks/engine_start_ticksspecifically so a PID can't be trusted after recycling, andcrates/rocm-core/src/proc_lifecycle.rsalready ships a ready-made classifier for precisely this question —ProcessIdentity/identity_state/IdentityState::{Matches,Recycled,Indeterminate,Gone}— whose own doc comment states the problem this guard reintroduces: "Stopping a managed service by a persisted PID is unsafe on its own: PIDs are recycled, so a stale PID may have been reassigned to an unrelated process." The new guard does not call it. Window:SERVE_STARTUP_DEADLINE= 30 minutes. If the supervisor or engine process actually died and its PID was reused within that window, the guard reports "still starting" and suppresses recovery on a service that is actually dead for up to 30 minutes — a real availability regression against pre-PR behavior, using data and a helper that are already present and unused.
Confidence 85 · logic/architectural · Fix: buildProcessIdentity::new(pid, record.supervisor_start_ticks)(and the engine equivalent) and callidentity_state, treating onlyMatchesas "still alive";Recycled/Gonemust not suppress recovery. -
The live-startup guard above is applied in the scan path but not in the event re-verification path that actually authorizes a restart, contradicting the guard's own stated single-point-of-truth design.
apps/rocmd/src/watchers/server_recover.rs:92-105(service_record_matches_recovery_event) callsmanifest_service_recovery_reason(record, ...).is_some()andendpoint_service_recovery_reason(record).is_some()directly — the unguarded helpers — whilefind_recoverable_service's scan (:328-336) andunreachable_endpoint_recovery_reason(:366-379) go through the newlive_startup_still_in_progressguard.unreachable_endpoint_recovery_reason's own doc comment states the intent directly: "One function for both places that fall back to the endpoint, so neither can drop the guard" — butservice_record_matches_recovery_event, the function that actually authorizeshandle_server_recover_event's restart (dispatched fromapps/rocmd/src/watchers.rs:184), is a third call site that was not routed through it. No test exercises this recheck path against a live-startup record, so the gap is untested as well as unguarded.
Confidence 75 · logic · Fix: route both ofservice_record_matches_recovery_event's branches throughunreachable_endpoint_recovery_reason/ the same"starting"-status checkfind_recoverable_serviceuses, or factor that check into one function both call.
Non-blocking
-
mask_valuecan panic on a multi-byte UTF-8 trailing character.apps/rocm/src/serve_progress.rs:821-835:body_endis computed asindex + 1fromword.rfind(...), whereindexis the byte offset where the matched character starts — valid only for a 1-byte (ASCII) character. A value ending in a multi-byte character (e.g.é) makesbody_endland mid-character, and&word[body_end..]panics ("byte index N is not a char boundary"). Not reachable today: the only production path (sanitize_log_line→latest_meaningful_log_line, used byobserve_launch) ASCII-filters the line beforeredact_log_line/mask_valueever see it. Butwait_for_serve_startup's own loop (:673-675) callsredact_log_linedirectly onsignals.log_linewith no such guarantee at the type level — it is an unenforced calling convention, not a contract — and no test exercises non-ASCII input tomask_value/redact_log_linedirectly.
Confidence 80 · logic · Fix: computebody_endfrom the end of the matched character (e.g.index + c.len_utf8()), notindex + 1; add a non-ASCII regression test. -
A manifest-write I/O failure in the "stall left the server running" branch of
start_managed_servicecan bury the typed, actionable startup-stop message.apps/rocm/src/service_launch/start.rs:185-190callsrecord_wait_outcome(paths, &mut record, launch_status, ...)?— whose last line islatest.write()?(a real, propagating error) — before constructing and returning theServeStartupFailed{server_running: true, message: stop.user_message(...)}the user is meant to see. If that write fails, a bare I/O/anyhow error propagates instead. The sibling "ended" branch immediately above (:164-177) is deliberately robust to this:settle_ended_launch→settle_exited_service's write is best-effort (let _ = latest.write();). This is a real asymmetry this PR introduces between its two new typed-failure branches.
Confidence 78 · logic · Fix: make the write best-effort here too (log/ignore a write failure) so the typed stop is still returned, matching the sibling branch. -
Two
docs/architecture.mdentries this PR touches understate their actual callers. Theservice_hints.rsentry listsserve_progress.rsas a caller but omitsserve_cmd.rs, which newly callsservice_hints::service_logs_hintdirectly atapps/rocm/src/serve_cmd.rs:1182(confirmed: noservice_hintsreference existed inserve_cmd.rsbefore this PR). Separately, theserve_progress.rsentry names onlyservice_launch/readiness.rs'swait_for_launch_startupas a caller, butserve_cmd.rs(starting_label,ServeStartupPolicy, matches onServeStartupFailed) andservice_launch/start.rs(starting_label,ServeStartupFailed,ServeStartupPolicy/ServeStartupView/PIPED_SERVE_STATUS_BUDGET) call into it directly too.cargo xtask check-architecture-doconly checks file presence, not prose accuracy — perAGENTS.md§6, "the check reads no prose, so keeping an entry's description true is still on you."
Confidence 75 · mechanical · Fix: addserve_cmd.rsto theservice_hints.rsentry's caller list, and addserve_cmd.rs/service_launch/start.rsto theserve_progress.rsentry's. -
cli.rs's new top-levelservehelp text overclaims for the "reuse an already-starting service, no--wait" path. The new text says "while it starts, the status line follows what the engine is doing... until the server is ready or that wait fails" (apps/rocm/src/cli.rs, theServedoc comment, diff around line 426-432). For a plainrocm serve <model>that matches an equivalent service another invocation is still starting,wait_on_existing_service(apps/rocm/src/service_launch/readiness.rs:393-395) returns immediately —if !wait_until_ready || report.status == "ready" { return Ok(report); }— before any status-line following happens at all. This is pre-existing behavior for the reuse path, not newly introduced, but the new, broader help text makes no exception for it.
Confidence 72 · architectural/doc-accuracy · Fix: scope the claim to a fresh launch (or one additionally waited on), or note the reuse-of-starting-service case explicitly.
Decisions for the author
rocm_core::SERVE_STARTUP_DEADLINE(30 min) is shared verbatim between the interactive CLI wait's give-up deadline androcmd's recovery-watcher grace window — tradeoff. This is documented at its definition (crates/rocm-core/src/endpoint_readiness.rs:213-217: "rocmd's server-recover watcher uses the same bound so it does not restart a live process that has not opened its port yet"), so the coupling is visible rather than silent. It still means tuning one later (e.g. a snappier interactive UX deadline) reshapes the daemon's auto-recovery window for every managed service, not just interactively-started ones. Confirm this is an accepted, intentional coupling.- List-valued secrets leak all but their first item, and a single-token compact-JSON object with two secret fields leaks the second one unmasked (
mask_compact_value,apps/rocm/src/serve_progress.rs:839-852, only splits on the first:). Partial, not total, redaction within the PR's own stated scope ("api-key/token/secret values"); worth confirming this residual gap is an accepted scope boundary for now.
Positive signals
decide_serve_startupis a pure function fully decoupled from I/O/timing, letting every lifecycle row be driven directly by a unit test with exact phase/label/reason assertions;wait_keeps_a_changing_download_alive_past_the_stall_limitis a genuinely mutation-resistant integration test proving theDownloadingstall check is keyed onsince_progress, notelapsed(the siblingUnreachablecheck lacks this protection — see Blocking).- Inference-probe backoff (
InferenceProbeThrottle) is well tested with an exact gap-sequence assertion ([1, 2, 4, 8, 15, 15, ...]), and redaction is proven to run before truncation (a_key_cut_at_the_length_limit_is_masked_before_truncation), closing an easy leak-via-partial-prefix bug. apps/rocm/src/service_launch/attach.rs's three previously hand-rolledrocm logs <id>strings are now fully replaced withservice_hints::service_logs_hint, backed by a test that round-trips the printed command through the real CLI parser rather than just pinning its text.apps/rocmd's crate-layering invariant holds: it depends only onrocm-core/rocm-engine-protocoland generic crates, never onrocm.- The new recovery-watcher boundary tests are precise (age exactly at the deadline; dead vs. live pid;
>=comparison matched by the fixtures' intent), andu32::MAX-as-dead-pid is handled safely (libc::pid_t::try_fromfails first, so no accidental broadcast-kill). start_managed_service's two new typed-failure branches both persist state and emit an audit event before returning their error (modulo the Non-blocking write-ordering asymmetry above);--wait's death path gained an audit event it did not previously have.- All new timing numbers added across
README.md,docs/manual-testing.md, anddocs/testing.md(3 min / 2 min / 5 min / 30 min / 45 s / 1 s→15 s) were checked against the actual constants and match exactly;xtask/src/ascii_output_contract.rs's new exemption entries match the two non-ASCII literals actually present inserve_progress.rs. serve_progress.rs's production code is ~859 lines, comfortably under the 1,500-line module-size cap, and has its owndocs/architecture.mdentry from day one, matching this repo's "new subsystems get their own file" guardrail.
Deployment notes
SERVE_STARTUP_DEADLINE(30 min) widensrocmd's auto-recovery grace window for any managed service that isrunning-but-endpoint-unreachable and under 30 minutes old, not only ones launched interactively — documented at its definition, but worth calling out for anyone operatingrocmdin production.- Engine log lines now undergo redaction before reaching any user-visible surface (status line, failure text) — worth noting for anyone debugging by grepping raw
rocm serveoutput for a credential, though see the compact-JSON and list-valued gaps above for residual limits. - CI for this exact head (
442ae1e9) is green across the fullCI/CodeQL/Security scanworkflows (clippy, commit-signatures, build-and-test, affected-crate tests, Windows build/test, coverage ratchet, docs/link checks all passed). The self-hosted GPU hardware E2E lane (E2E self-hosted, covering live vLLM/Lemonade/AIM) was still queued/pending with zero jobs started as of this review and had not run; the PR's own text already flags the GPU-engine log-cadence question as unverified for the same reason.
What this covered
Read completely: all 19 changed files (apps/rocm/src/serve_progress.rs — new, ~1966 lines including tests; serve_cmd.rs, serve_summary.rs, service_hints.rs, engine_runtime_select.rs, cli.rs's touched regions, main.rs's one-line diff; service_launch/{attach,readiness,start}.rs; apps/rocmd/src/watchers/server_recover.rs; crates/rocm-core/src/{lib.rs,endpoint_readiness.rs}; README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, xtask/src/ascii_output_contract.rs), plus this repo's root AGENTS.md in full, and crates/rocm-core/src/{service_record.rs,proc_lifecycle.rs} as supporting context for the rocmd findings. Diff resolved against 1530f6b1…442ae1e9 by fetching the PR head read-only (git fetch https://github.com/ROCm/rocm-cli.git pull/817/head into a scratch clone); the PR was never checked out.
Five independent, blind, file-scoped fan-out workers ran in parallel and reported: the core state-machine module (serve_progress.rs); the serve_cmd/serve_summary/service_hints/engine_runtime_select/cli.rs wiring layer; the service_launch module (attach.rs/readiness.rs/start.rs); the rocmd watcher plus shared rocm-core constants; and the docs/testing/xtask surface. Every candidate they surfaced was independently re-verified against the source here (direct byte-offset tracing for the UTF-8 panic, direct confirmation of the shell_quote/doc-claim gap, direct confirmation that ManagedServiceRecord already carries the start-tick fields identity_state needs) before inclusion or rejection; several weaker candidates (suffix-only secret-name matching, a dormant unclamped-backoff path, a hardcoded-phrase collision, a second typed-error-shape inconsistency) were read and dropped as below the reporting bar. The history and prior-changes passes were done directly by the consolidating reviewer (git log/blame on every touched pre-existing file, and a direct check of one specific prior-review-linked claim about engine_runtime_select.rs's module doc, which turned out to still be accurate at this head and is not reported as a finding).
Live CI was checked directly (get_check_runs/list_workflow_runs/list_workflow_jobs) against this exact head commit, re-polled until the CI workflow reached a final success conclusion; this was not something any fan-out worker was tasked with. Not reconciled against this PR's own review discussion or prior review comments, which were not read as review content (only consulted for the duplicate-review commit-SHA check, per this task's own instructions).
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
Response to the automated review of Blocking:
Non-blocking:
Decisions:
|
Interactive rocm serve stopped after 45 seconds with a static not-ready summary while a download or load was still moving. The status line now follows the engine log until the endpoint can serve or that progress stalls, and a failure names the logs and stop commands. Piped output keeps the 45-second status budget, and `--wait` keeps its own wait, with no stall limit or deadline. Like `--wait`, the interactive wait first prints the service id, how to follow its log and a public bind's key, since it can fail without a summary. Liveness is read from the manifest with the zombie-aware pid check, so a dead supervisor ends the wait. A startup stop is a typed ServeStartupFailed: only it starts the background helper, and a helper failure is a warning beside it. Engine log lines are redacted (endpoint key, --api-key/token/secret values, compact JSON keys, Authorization headers, hf_ tokens) before they reach the status line or failure text. rocmd's server-recover watcher no longer restarts a live `running` or `starting` record younger than SERVE_STARTUP_DEADLINE, so it does not kill a download the CLI is still reporting. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Once the endpoint lists the model, the interactive wait sent an 8-second inference probe on every 250 ms poll for up to 30 minutes. The probe now runs at once, then backs off from 1 to 15 seconds (INFERENCE_PROBE_RETRY_INTERVAL), timed from when the last one returned. Piped output keeps its per-poll probing within the 45-second budget. An engine failure or process exit during the interactive wait was recorded `failed` and handed to the background helper, which a launch the CLI has already reported as failed must not be. It is now settled the way a `--wait` that saw the server exit is (`stopped`, or the engine's own `failed`), with the same memory guidance and an audit event, and only a stall or the deadline, which leave the server running, start the helper. The attached path printed `rocm logs <id>`, which `rocm logs` reads as a free-text search. Every logs hint now goes through service_logs_hint. rocmd's two endpoint-unreachable checks share one guarded helper, and the dead --wait parameter of wait_for_record_ready is gone. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…once Three serve_progress tests pointed the wait at a closed loopback port. On Windows a refused loopback connect takes about two seconds, so each poll outlasted the tests' stall and deadline limits and the Windows job failed. They now use a local endpoint that answers 503 at once, which is the same "model not listed" reading without the platform's connect delay. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ndary service_logs_hint printed the service id bare, unlike every other hint in service_hints.rs, whose module promises a line that pastes back as the argv it names. It now goes through shell_quote too. mask_value cut a value one byte past the start of its last character, which panics when that character is wider than a byte. It now cuts at the character's end. A secret word anywhere in an underscore name (`secret_key`, `--private-key`) is masked as well as one at the end. After a stall or deadline stop the record rewrite is best-effort, so the stop stays the error the user reads. A test pins which clock each silent limit reads, and the help and README scope the wait announcement to a fresh launch. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…identity The unreachable silent limit counted the whole wait, so a long startup that went quiet for a moment could stop at once. It now counts from the last change of what the wait observes, as the load limit does and as the README describes. rocmd's live-startup guard trusted a bare pid for up to 30 minutes. It now checks the recorded start time with identity_state, so a recycled pid does not keep a dead service from being recovered. The event re-check that authorizes a restart now uses the same guarded reasons as the scan, so an event raised before a startup was spared cannot restart it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
eb66e15 to
ad1b07c
Compare
|
Response to the automated review of Already fixed on the next head ( New in
Not changed:
|
|
@juhovainio, on the points in your review body that were not inline threads (inline ones are answered in their threads):
Since your review: the rebase picked up |
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for the work on this. The interactive wait, the state matrix, and the paired message-plus-behaviour tests in start.rs are solid. One thing needs to change before merge, and a few smaller points are worth a look.
Needs to change
- No Gherkin scenario covers the user-observable
rocm servechange. Interactive serve now exits non-zero in new situations (stall, deadline, engine failure) and prints a new announce block and changed log hints. AGENTS.md section 3 asks for a scenario intests/e2e-cucumber/features/and, when it can only run on a gated lane, for the PR text to name that lane. A comment explaining the absence doesn't cover it. Please add the@requires-gpuscenario (or get an explicit maintainer exemption and say so in the PR text).
Worth a look (non-blocking)
- The 5-minute silent-load limit can make
rocm serveexit non-zero on a healthy model that is just quiet while loading. The server keeps running, but scripts will see a failure. Consider a longer default or an override. - A stalled or deadline-stopped server is recorded as
runningwith no inference verification. Please confirm nothing reads that as ready (services list, reuse of a matching live service, chat). - The inline inference probe can block for up to 8s, which freezes the spinner and delays noticing an engine failure.
- The rocmd guard defers recovery for up to 30 minutes after create or restart. Looks intended and tested; just confirming it is the tradeoff you want for a hung engine.
- "Ctrl-C stops waiting, the server keeps running" is now printed for the interactive wait but no test asserts it for that path.
- Redaction is heuristic (list values mask only the first item,
user:pass@hostis not handled), so "credentials masked" in the README is broader than the guarantee. "Known credential forms" would be more accurate. - Minor:
start_managed_serviceis at 20 arguments,decide_serve_startuprepeats near-identicalstop_decisioncalls, andquoted_logmatches on label strings thatclassifyalso produces.
| pub(crate) const SILENT_UNREACHABLE: Duration = Duration::from_mins(2); | ||
|
|
||
| /// A load line that stops changing while the port is still down. | ||
| pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5); |
There was a problem hiding this comment.
A 5-minute silent limit will fail a healthy engine that is quiet while loading weights or capturing graphs, with a non-zero exit. The PR notes the real engine cadence is unverified; consider a longer default or an env override, or confirm the tradeoff is intended.
There was a problem hiding this comment.
Intended tradeoff, not verified against every engine: five minutes with no change in the log line and the port still down is treated as a hang. The stall only ends the wait. It exits non-zero and the message says the server is still running and how to follow it (rocm logs --service <id> --follow) or stop it, so a healthy engine that was just quiet is not killed. I did not add an env override here to keep the change scoped; if you want one, say so and I will add it as a follow-up.
| if !signals.engine_failed && signals.process_alive { | ||
| let listed = | ||
| service_http_lists_model(engine_name, host, port, &probe_model, endpoint_api_key); | ||
| match throttle.readiness(listed, &mut Instant::now, || { |
There was a problem hiding this comment.
The inference probe runs inline here and can block up to the probe timeout, so the spinner stops ticking and an engine failure isn't seen until it returns. Backoff bounds it; maybe note it in the doc comment if intended.
There was a problem hiding this comment.
Intended, and now stated in the doc comment on wait_for_serve_startup (eb3ce2a): the probe runs inline, so a poll that sends one can take up to the probe timeout, during which the spinner does not tick and an engine failure is not read. The backoff (1 s growing to 15 s) bounds how often that happens.
| } | ||
| // A stall or the deadline left the server up: `running`, which `rocmd` | ||
| // leaves alone until the startup deadline. | ||
| let launch_status = if startup_stop.is_some() { |
There was a problem hiding this comment.
A stalled or deadline-stopped live server is recorded running with no inference verification. Please confirm no reader (services list, reuse of a live matching service, chat) treats running as serving.
There was a problem hiding this comment.
Checked the readers of running: managed_service_status (services list/status) and providers (chat) both re-run managed_service_endpoint_readiness for a running record and only promote it to ready when the inference probe answers, otherwise it stays running. running was already what a model-listed-but-unverified launch recorded (status_for_readiness(Listing)), so a stalled launch is no different. I did not trace the reuse-of-a-live-matching-service path separately: it keys on liveness, not on the status string, as far as I read it. Say if you want that path pinned with a test.
| @@ -1,5 +1,14 @@ | |||
| Feature: Model serving | |||
|
|
|||
| # Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows | |||
There was a problem hiding this comment.
AGENTS.md section 3 asks for a scenario for user-observable CLI behaviour, using a @requires-gpu lane when it can't run elsewhere. A comment explaining the absence doesn't cover it; please add the gated scenario or get an explicit exemption.
There was a problem hiding this comment.
Added in eb3ce2a: scenario serve-55 (serve-interactive-startup-progress, @requires-gpu) runs rocm serve under a real PTY, checks the wait line says Ctrl-C stops waiting (not the server), runs the rocm logs --service <id> command it names, and checks the serve exits 0 with the endpoint listing the model. It cannot run in default CI or on the no-GPU lane, only on the GPU lanes; I could not run it locally (no GPU), so it is compile- and clippy-checked only and not yet seen to fail or pass. The stall, deadline and engine-failed branches stay covered by the unit state matrix, since they need a controllable engine.
Resolve docs/architecture.md against the per-crate table layout from #1016: serve_progress.rs gets a row, and the service_hints.rs, service_launch/readiness.rs, service_launch/start.rs and watchers/server_recover.rs rows carry this branch's changes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (base b109db12c7a59f5f5c6f38d1e4a2191ff8851b53 against head ad1b07cd37499935a2192133e2cec86f05619d8c). No ticket referenced in the PR description.
Blocking
-
Engine-log secret redaction can be bypassed for exactly the formats it's meant to catch (a quoted "Authorization"/"Bearer" header, or a compact
"key":"value"dump whose value contains a space) —apps/rocm/src/serve_progress.rs:748-756,apps/rocm/src/serve_progress.rs:815-821,apps/rocm/src/serve_progress.rs:846-859redact_log_lineprocesses the sanitized log line token-by-token on spaces, and only re-arms its "mask the next word" state when the previous token is the literal, unquoted wordbearer/basic. A dict-style header rendering such as{'Authorization': 'Bearer <token>'}(common in debug logs from Python request libraries) splits into'Authorization':,'Bearer, and<token>'; because the second token carries its opening quote attached,word.eq_ignore_ascii_case("bearer")is false, so the function masks the harmless word "Bearer" itself and never re-arms for the real secret that follows — which is then printed unchanged. The compact-JSON path (mask_compact_value) has the same gap from a different angle: it only looks for a closing quote inside the same whitespace-delimited token, so"api_key":"sk live 123"(a value containing a space) comes back completely untouched, not even partially masked, because no closing quote is found in the first fragment and the continuation token is never revisited.Confirmed by tracing the exact branches against
"'Authorization': 'Bearer abc123secret'"and"api_key":"sk live 123", then reproducing the same control flow in a short standalone script and running it: both leak the full secret verbatim in the output. Confidence 90 · logic · Fix: track "currently inside a value for a flagged key" across tokens, so a continuation word is still masked after a quotedBearer/Basicmarker, andmask_compact_valuekeeps masking into the next token when no closing quote turns up in the first one, rather than only special-casing the exact unquoted wordsbearer/basic.
Non-blocking
- Legitimate field names containing "token" get redacted as if they were secrets —
apps/rocm/src/serve_progress.rs:807-809
is_secret_nametreats any underscore-separated name containing the bare wordtoken(among others) as a secret, so an ordinary line liketoken_type_ids=[0, 0, 1](a common ML tensor field, not a credential) has its value replaced with[redacted], hiding debugging information the feature isn't meant to hide. Confidence 70 · logic · Fix: narrow the match (e.g. names ending in_token/token) rather than "contains the word token anywhere in an underscore-joined name".
Decisions for the author
- A newly-added audit event on the
--waitdeath path — tradeoff
Settling an ended launch through the shared helper now records an "error"-level audit event on the--waitpath too, which previously didn't get one. This looks intentional (it gives the interactive and--waitfailure paths the same audit trail), but since it's a side effect of a refactor rather than something the description calls out explicitly, it's worth confirming that was deliberate.
Positive signals
- The "time since progress stalled" timers in
wait_for_serve_startupare anchored to when the tracked status last changed, not to when the wait started — the specific off-by-reference-point mistake this kind of stall detector commonly makes — and that's covered by a test using plainDurationvalues rather than real sleeps. mask_value/mask_compact_valueoperate on character boundaries (viachar_indices) rather than raw byte offsets, with a test using a literal multi-byte character sitting exactly at the truncation limit — real, deliberate protection against a UTF-8-slicing panic.- The previously hand-rolled
rocm logs <id>hint text (whichrocm logsitself would have parsed as a free-text search rather than selecting that service) is now fully routed through one shared, shell-quoted helper everywhere it's printed; one of the new tests proves this by feeding the printed hint back through the real command-line parser instead of just string-matching it.
Deployment notes
- The CLI's interactive startup deadline and the separate daemon's recovery watcher now share one constant, so the two can't drift out of sync with each other. Piped/non-interactive output is unchanged and keeps its original fixed-budget behavior.
What this covered
Read the full diff between the PR's merge base and its current head (19 files, roughly 3,000 added lines, dominated by one new ~2,000-line module). The new startup-decision module, the launch/wait integration it plugs into, and the readiness/attach/hint/summary files were each read in full, independently, with particular attention to the credential-redaction logic, where the finding above was traced against the exact source and then confirmed by reproducing the same control flow in a standalone script. The CLI flag/help-text changes, the recovery-watcher change in the companion daemon (including its pid-recycling and race-condition handling, which held up well), the shared constant it was given, and the README/docs/feature-file changes were also read in full; the repository's own contributor guide was checked against this change for its module-size limit, its "every source file gets an entry in the architecture doc" rule, and its doc/behavior-sync rules, all of which this change satisfies. These last few areas got one thorough read rather than a second independent pass.
Every required CI check passes (Linux and Windows build/test, lint, commit sign-off, static analysis, the security-scan suite, and the project's own module-size/architecture-doc/doc-link gates). One self-hosted GPU lane's consolidated report shows red, but only because that lane's run was itself cancelled; another workflow was running at the same time on a branch whose name indicates that exact lane is being retired, so this reads as known infrastructure churn rather than anything this change broke. Separately, GitHub currently reports this PR as not mergeable against main (a conflict to resolve by rebasing), which is a note for the author rather than a comment on the code.
Not reconciled against any review discussion on the PR.
|
Merged main (2ffa4f9) to resolve conflicts with #1016; the only conflict was
The code merged cleanly into main's |
…tartup Scenario serve-55 runs `rocm serve` under a real terminal, checks the wait line names a log command that actually runs, and that the serve exits 0 with the endpoint listing the model. Also documents that the inference probe runs inline in the startup wait. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 2ffa4f9
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (19 files, +3081/-116: apps/rocm/src/serve_progress.rs new; serve_cmd.rs, service_launch/{start,readiness,attach}.rs, apps/rocmd/src/watchers/server_recover.rs, crates/rocm-core, CLI wiring/hints, docs, and the e2e feature/xtask fixture all touched). No ticket named in the PR title or body; the description is the sole stated intent.
Blocking
None.
Non-blocking
-
redact_log_line's documented "list value" limitation has no test —apps/rocm/src/serve_progress.rs:738(doc comment),redact_log_line/mask_value(740-842), testredact_log_line_masks_keys_tokens_and_secret_flags(~1902-1963).
The doc comment explicitly documents a security-relevant limitation: "A list value ('api_key': ['a', 'b']) has only its first item masked." Tracing the code confirms the current behavior matches that claim exactly (word-splitting +mask_nextconsumes only the first list item), but no test in the file exercises a list-shaped value at all, so a future change tomask_value's bracket/mask_next-reset handling that silently makes this worse (e.g. drops masking of even the first item) would go uncaught.
Confidence 72 · logic/test-coverage · Fix: add a case asserting both that the first list item is masked and the second is not, e.g. on"non-default args: {'api_key': ['sk-abc', 'sk-def']}". -
The new redaction only covers the new status-line/error surface; the same secrets remain fully exposed through
--verbose/--foregroundstreaming androcm logs, which this PR's own new failure text actively points users to —apps/rocm/src/serve_progress.rs:591-640(ServeStartupStop::user_message, which on every failure path tells the user to runservice_logs_hint/rocm logs --service <id>) vs.apps/rocm/src/service_launch/attach.rs:286-353(stream_attached_logs/stream_attached_logs_no_tty, which forward the on-disk log file verbatim with no call toredact_log_line/sanitize_log_line— confirmed unchanged by this PR's diff) andapps/rocm/src/services_text.rs:519-570(render_service_logs_text, backingrocm logs --service <id>, also prints raw lines with no redaction call, also unchanged). The on-disk log file itself (apps/rocm/src/service_launch/spawn.rs'sattach_background_stdio, byte-identical pre/post-PR) has always held the engine's raw stdout/stderr, credentials included. So a secret masked for an instant on the interactive status line is fully visible an instant later via the exact command this PR's own messages recommend. This is a pre-existing gap (the wider unredacted surfaces predate this PR and the PR's own wording — "before they reach the status line or an error" — is accurately scoped to what it actually touches, not a broader claim), and this PR does not enlarge it; raising it because it sits squarely in the pattern of the change (credential handling in engine logs) and the new remediation text is what actively routes users to the unredacted view.
Confidence 85 (verified by tracing all four files and confirming empty diffs on the unredacting paths) · architectural · Fix: non-blocking for this PR, but worth a follow-up — either redact at log-write time (so every reader benefits) or caveat thatrocm logs/--verbosestill show raw content.
Decisions for the author
None.
Positive signals
decide_serve_startup's branch order mirrors the module doc's state matrix closely, and nearly every row/edge case called out in the doc comment (e.g. "a log line that still says 'download' does not pull a listed model back into the download stall") has a directly corresponding, boundary-precise unit test rather than a loose substring check.- Credential redaction is adversarially solid where it does apply: known-endpoint-key substring replacement happens before word-level heuristics, double-masking is guarded against, Unicode/multi-byte values are handled via
char_indices/len_utf8rather than byte slicing, and redaction is confirmed to run before truncation (so a key straddling the 160-char cutoff can't leak its prefix). - The
rocmdserver-recover guard (live_startup_still_in_progress) and the CLI-side interactive wait are backed by the same sharedrocm_core::SERVE_STARTUP_DEADLINEconstant rather than a duplicated literal, crate layering stays clean (rocmdimports nothing fromrocm), and both restart-triggering code paths (periodic scan and the event re-check) route through the identical guard function. service_hints::service_logs_hintgenuinely centralizes previously-duplicated (and in one case bugged — a barerocm logs <id>thatrocm logsparsed as free-text search) hint text, and the "hostile id" property test verifies the shell-quoting round-trips through realshlexparsing rather than a cosmetic string compare.- Documentation discipline is unusually good: every numeric constant (3 min download stall, 2 min silent-unreachable, 5 min silent-load, 30 min deadline, 45 s piped budget, 15 s max inference-probe backoff) matches exactly across README.md, docs/testing.md, docs/manual-testing.md and the code, and
docs/architecture.mdwas updated in the same change for the new file and every touched file's changed role. - The "no Gherkin scenario for the interactive flow" gap is handled the way this repo's own rules ask for: named explicitly in
tests/e2e-cucumber/features/model_serving.featureanddocs/testing.mdwith the@requires-gpurationale, and backed by state-matrix unit tests instead.
Deployment notes
rocm serve run interactively (in a real terminal, not piped) now blocks for up to 30 minutes before giving up on a stalled/failing startup, versus the previous fixed ~45-second wait. Piped/captured output is explicitly unchanged. Anyone wrapping rocm serve from a pseudo-terminal in semi-automated tooling (not a plain pipe) should expect the longer worst-case wait; this is already called out in the PR description's "Unchanged" section but is worth flagging explicitly here.
What this covered
Read the full diff between 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe and 2ffa4f9e09cf8b87553d61bab192d331ff47df8d (19 files). Every changed file was read completely (not just diff hunks) by one of six independent fan-out workers, grouped by coupling: (1) serve_progress.rs alone (new 2031-line file, production + tests); (2) service_launch/{readiness,start}.rs; (3) serve_cmd.rs + service_launch/attach.rs; (4) apps/rocmd/src/watchers/server_recover.rs + the two rocm-core constant files; (5) cli.rs, engine_runtime_select.rs, serve_summary.rs, service_hints.rs, main.rs; (6) README/docs/e2e-feature/xtask-fixture consistency. A seventh worker ran the agent-instruction-adherence pass (against this repo's root AGENTS.md) and a git-history pass across the whole diff. Two additional targeted investigations (Step 7 design review) checked (a) whether the new redaction is undermined by other log-reading paths — yielded the non-blocking finding above — and (b) whether a coincidental "5 minutes" value shared between rocmd's pre-existing stale-start check and this PR's new SILENT_LOAD constant creates an implicit, undocumented coupling — traced directly and found not to be an issue: rocmd's new live_startup_still_in_progress guard (keyed on the 30-minute deadline plus pid-identity matching) short-circuits before the old 5-minute staleness check ever runs for a live, identity-matched process, so the two "5 minute" constants govern genuinely independent conditions, not a hidden shared assumption.
CI: all checks green at the reviewed commit except one GPU-lane check (E2E tests (MI300X, current)) still QUEUED at review time (not failed) and a couple of platform-matrix jobs correctly SKIPPED.
Did not run: cannot execute cargo test/cargo clippy/cargo xtask check-module-size/check-architecture-doc/check-crate-edges/coverage in this environment (no Rust toolchain or working-tree checkout available) — CI's green status for all of these was taken at face value rather than re-derived. The "prior changes" pass (review comments on earlier, unrelated PRs touching these same files) did not run; no practical way to retrieve that history here beyond the git commit log, which the history pass did check. Not reconciled against any prior review round or PR discussion thread on this PR — this review was produced blind, from the diff and the PR's own description only, per method.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Requesting changes for two process reasons; I found no blocking defect in the code itself.
- The PR is currently CONFLICTING with main. Because of that no CI has run on this head (only readthedocs), so everything under "Verification" is a local claim. Please rebase and get the checks green.
- The new failure behaviour (non-zero exit, message, settled record) is covered only by unit tests. The only scenario is the happy path and it is
@requires-gpu. Per the contribution rules, a scenario for "engine exits during startup" on a lane that has runners, or a note in the PR text naming the gap, is needed (inline comment below).
The rest are smaller points, mostly inline:
- A stall or deadline stop records the server as
runningeven when it never listed the model. - The new
rocmdguard is broader than interactive serve and delays recovery of any live service with a closed port for up to 30 minutes. - An engine-failed record with a live process stays
failed, sorocmdcan restart it behind the error the user just read. - The download stall check keys on the last
NN%, which can false-positive on a slow multi-GB shard. - Docs say "the port stays closed" but the code arms on the endpoint not listing the model.
Also worth a look, no anchor:
- The 30-minute wait policy keys on stdout being a TTY while the spinner draws on stderr, so
rocm serve 2>login a terminal waits 30 minutes with nothing drawn. serve_cmd.rsis at about 1462 of the 1500-line limit,wait_for_serve_startuptakes 8 arguments, andstop_decisionis called with 6 arguments at 6 sites.- Please confirm the fixed 3/2/5/30-minute limits are deliberate and not per-engine.
- Piped output exits 0 after 45s while a terminal now exits non-zero for the same slow server; worth stating as intended.
On the plus side, the injected-clock backoff tests, measuring silent limits from the last change, and the process-start-time check in the rocmd guard are well done.
|
|
||
| /// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent | ||
| /// does not is still stalled. | ||
| fn download_fingerprint(line: &str) -> String { |
There was a problem hiding this comment.
download_fingerprint keys on the last NN%. A Fetching N files: 20%| 1/5 bar only moves when a whole file finishes, so a slow multi-GB shard holds the same percent for longer than DOWNLOAD_STALL (3 min) and a healthy download is reported as stalled (non-zero exit). Please consider a byte/size-growth signal (log or cache-dir size/mtime) as an extra progress token, or say in the PR which real engine log cadence this was checked against (it is listed as not verified).
There was a problem hiding this comment.
Not changed in this PR, and stated in its body as a known limit. You are right about the mechanism: with only the Fetching N files bar visible, a single slow shard holds the percent past 3 minutes. The per-file byte bars that huggingface_hub prints change the line (and so the token) while a file downloads, but I have not checked that every engine writes them to the service log, which is why the cadence is listed as unverified. A stall only ends the wait: the server keeps running and the message says how to follow it. A cache-size growth signal is the right fix; I would like to do it as a follow-up rather than widen this PR. Say if you consider it blocking.
| } | ||
| // A stall or the deadline left the server up: `running`, which `rocmd` | ||
| // leaves alone until the startup deadline. | ||
| let launch_status = if startup_stop.is_some() { |
There was a problem hiding this comment.
A stall/deadline stop writes running even when readiness is Unreachable. Elsewhere running means "endpoint lists the model" (status_for_readiness(Listing)), so rocm services, dash and chat will show a server that has not opened its port as up. The new starting guard in rocmd already spares a live start until the deadline, so starting (or status_for_readiness(readiness)) would do. The test pins running.
There was a problem hiding this comment.
Fixed in e4933a9: a stall or deadline stop now records status_for_readiness(readiness), so a launch whose endpoint never listed the model stays starting; running is only written when the model is listed. rocmd already spares a live starting record until the startup deadline. a_startup_stop_records_the_live_server_and_returns_the_typed_failure now pins starting, and goes red with the old running branch restored.
| { | ||
| // The engine failed or the process ended: settled the way a `--wait` | ||
| // that saw the server exit is, with the same memory guidance. | ||
| let message = stop.user_message(requested_model, engine, service_id); |
There was a problem hiding this comment.
For the engine-failed-but-process-alive case the record is left failed (settle_exited_service keeps the engine state). mark_launch_abandoned documents that rocmd restarts a failed record with no stop marker, which is why the --wait death path uses stopped. An already-running helper can therefore restart this server behind the error printed here, and the message ("stop stops it") does not say so. Same for a Deadline stop at age >= 30 min (see found_at_deadline). Please either settle it stopped/terminate it, or say in the message that it may be restarted.
There was a problem hiding this comment.
Addressed in the message, in e4933a9: for an engine that reported failed while its process is still up, the text now adds "A running rocm daemon with server-recover enabled may restart it first." (pinned in only_a_stall_or_the_deadline_leaves_the_server_running, red without it). I did not settle that record stopped or kill it: the engine itself wrote failed and the process is still the engine's, so overriding it would hide the engine's own verdict from rocm services. A process that has exited is settled stopped (now checked end to end by serve-60). For a Deadline stop past 30 min the process is alive and the message already says it is still running and names the stop command.
There was a problem hiding this comment.
Correction to my reply above about the Deadline stop: the message did not say the server may be restarted. Fixed in ced7f76: every stop that leaves the server running (a stall or the deadline) now adds "If it is still not answering 30 minutes after it started, a running rocm daemon with server-recover enabled may restart it.", pinned in only_a_stall_or_the_deadline_leaves_the_server_running.
| /// That is the window interactive `rocm serve` spends following a download | ||
| /// or load before the port accepts connections. Endpoint-unreachable | ||
| /// recovery during it restarts the process out from under that wait. | ||
| fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u128) -> bool { |
There was a problem hiding this comment.
This guard spares any live-pid running/starting record with a closed port for 30 min after creation/last restart, for every managed service and not only one the interactive serve is waiting on. A service that was serving and whose listener then died is no longer recovered for up to 30 min. Consider scoping it to records the CLI wait left (a marker), or stating the trade-off as a decision in the PR.
There was a problem hiding this comment.
Kept, and stated in the PR body as a trade-off. It cannot be narrowed to starting: the engines write their own state running before their port opens (router_serve.rs and direct_llama.rs in Lemonade, state.rs in AIM), and rocm services copies that into the record, so a running record with a closed port is the normal shape of a Lemonade/AIM startup. The cost is the one you describe: a service that served and then lost its listener within 30 minutes of its start or last restart, with its process still alive, waits until the deadline to be recovered. A dead process, or a recycled pid, is still recovered at once.
| second). A model that is still loading is not reported as a finished "not | ||
| ready" summary. The wait fails, and `rocm serve` exits non-zero, when the | ||
| engine reports failure or its process exits, when a download percent stops | ||
| advancing for 3 minutes, when the port stays closed with no engine log for |
There was a problem hiding this comment.
The code arms this exit on EndpointReadiness::Unreachable (the endpoint does not list the model), not on a closed port: an open port answering 503 with no log also counts, and the message says "never answered". Please reword here and in docs/testing.md and docs/manual-testing.md ("does not list the model" / "has not listed the model").
There was a problem hiding this comment.
Fixed in e4933a9: README.md, docs/testing.md and docs/manual-testing.md now say "the endpoint has not listed the model" instead of "the port stays closed", and so do the doc comments on SILENT_UNREACHABLE and SILENT_LOAD.
| # lists the model. The named command is run, not just read: a hint that | ||
| # points at a command that does not work would still pass a string check. | ||
| @id:serve-interactive-startup-progress @requires-gpu | ||
| Scenario: serve-55 - Interactive serve shows how to follow startup and exits when the model is served |
There was a problem hiding this comment.
Only the happy path is a scenario and it is @requires-gpu, so nothing on a lane with runners exercises the new exit code/message. The repo already has a mock-lane seam (e2e-oom-fault-injection, stand-in processes in comfyui_steps.rs) for engines that cannot run in CI. A scenario for "engine exits during startup -> non-zero exit, names rocm logs --service/rocm services stop, record settled stopped" would cover the user-visible failure. If not added, name the gap and the lane in the PR text.
There was a problem hiding this comment.
Added in e4933a9: serve-60 (serve-startup-exit-names-log, @requires-no-gpu @requires-os:linux @requires-oom-fault-injection, so it runs on the mock lane). With ROCM_E2E_SIMULATE_STARTUP_EXIT=1 an e2e-oom-fault-injection build swaps only the spawned child for a stand-in that logs a line and exits during startup; the record, log, wait and settling are production code. It checks the non-zero exit and "the server process exited" message, runs the rocm logs --service <id> the failure names and finds the engine's last line, checks the record is stopped, and that no daemon was left. It passes locally, and fails at the stopped step when settle_exited_service writes failed.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head eb3ce2a0
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (base 8f086ad9...head eb3ce2a0).
No ticket named. (One commit message references #1016 as the prior-art docs table layout it builds on, not an issue this PR closes.)
Blocking
None
Non-blocking
None
Decisions for the author
redact_log_line's list-value masking covers only the first element — answered-by-intent
redact_log_lineinapps/rocm/src/serve_progress.rs:745masks a secret-named value, but when that value is a bracketed/comma list (e.g.'api_key': ['a', 'b']), only the single word immediately after the key is passed tomask_value(mask_nextis a one-shot flag,serve_progress.rs:751-758); any further list items are emitted verbatim. I traced this by hand against the current head and it reproduces exactly as described. The PR's own description states this plainly: "A list value has only its first item masked." — matching the in-code doc comment atserve_progress.rs:743-744word for word, so this is disclosed, deliberate scope rather than an oversight. Absent that disclosure this would have been a Blocking finding (confidence 85): it's security-sensitive redaction code (the whole point is keeping secrets out of the terminal/status line), and the 1,164-line test module for this file (redact_log_line_masks_keys_tokens_and_secret_flags,serve_progress.rs:~1907-1968) has no case with a bracketed list value, so the documented gap is also completely unpinned by any test. Confirm this is still the intended scope, and consider at least a test that pins today's "first item only" behavior so a future change can't silently narrow it further (e.g. to "no items masked").- Fixed, global stall/deadline constants rather than per-engine tuning — tradeoff
DOWNLOAD_STALL(3 min),SILENT_UNREACHABLE(2 min),SILENT_LOAD(5 min) andSERVE_STARTUP_DEADLINE(30 min, inrocm-core) apply uniformly to every engine (vLLM, Lemonade, AIM), even though their download/load cadence can differ substantially. The PR's own "Scope" section says this is deliberate: "The stall limits are fixed constants, not per-engine settings." Both sides are real: fixed constants are simple and auditable in one place; per-engine tuning would better fit an engine whose load phase legitimately goes quiet for longer than 5 minutes on constrained hardware, at the cost of more configuration surface. Worth revisiting if a real engine's startup profile turns out to routinely trip a stall limit that isn't actually a hang. - No audit event records "the interactive wait gave up while the server kept running" — tradeoff
Inapps/rocm/src/service_launch/start.rs, an exited/failed launch (settle_ended_launch) always writes anerror-levelmanaged_service_launchaudit event, but a stall/deadline stop that leaves the process alive only ever has the earlierinfo-level "launched … readiness=running" event on record — there's no audit trail entry marking that the CLI's wait itself gave up. This looks intentional (the launch did succeed; only the terminal wait timed out, and the module's own state-matrix doc treats "still running, stalled" as distinct from "failed"), but it does mean an operator scanning the audit log for failures would see a clean "info: launched" line for what the interactive user experienced as a non-zero exit. Confirm that's the intended audit semantics.
Positive signals
engine_runtime_select.rs's readiness probe is explicitly shared between the HTTP-polling wait andserve_progress::wait_for_serve_startup, with a doc comment calling out why: "so the interactive status line and this wait cannot disagree about 'ready'" — avoids a classic two-sources-of-truth readiness bug.service_launch/attach.rs's hint-helper refactor is an actual bug fix, not just deduplication: the old inline hints printedrocm logs {id}(parsed byrocm logsas a free-text search, not--service <id>); the fix is backed by a test that round-trips the printed line through the realclapparser and asserts it parses asLogs { service: Some(id), .. }.- The secret-masking helpers (
mask_value,mask_compact_value) are genuinely char-boundary-safe — they locate the mask boundary viachar_indices()/findon chars rather than raw byte offsets, verified directly against the source and exercised by dedicated multi-byte-character test cases. - All three stall/silence timers in
serve_progress.rsconsistently key off time-since-last-observed-change rather than total elapsed wait time, verified directly indecide_serve_startupand the token/progress_atbookkeeping inwait_for_serve_startup. rocmd's server-recover guard (apps/rocmd/src/watchers/server_recover.rs) checks a pid's recorded start time viaidentity_state, not just "is some process alive at this pid" — a recycled pid is correctly still recovered — and both the periodic scan and the event-triggered re-check route through the same guarded helper, so neither can diverge from the other.- Every timing number and behavioral claim added to README.md/docs/testing.md/docs/manual-testing.md (stall windows, the 30-minute deadline, inference-probe backoff, which hint text is shown when) was checked directly against the corresponding constant or function in source and matched exactly.
Deployment notes
- At review time, only the docs/readthedocs build has completed against the exact head commit (
eb3ce2a0): CI (build/test/clippy), the PR security scan, CodeQL, and the GPU-gated E2E self-hosted lane last completed successfully against the previous commit (2ffa4f9e), not this one — confirm all of those are green oneb3ce2a0itself before merging, rather than assuming the prior commit's results still apply. SERVE_STARTUP_DEADLINEis now a single constant inrocm-coreread by both therocmCLI's interactive wait and the separaterocmddaemon's recovery guard — correct today since both ship from the same build, but worth knowing that deployingrocmandrocmdfrom different versions could make them disagree about how long a fresh startup is protected from forced recovery.
What this covered
Every file changed in this PR, read completely at the head commit, by five independent workers grouped by coupling (the new apps/rocm/src/serve_progress.rs startup-wait module; the serve_cmd.rs/serve_summary.rs/cli.rs/main.rs/engine_runtime_select.rs command-wiring group; the service_hints.rs/service_launch/{attach,readiness,start}.rs service-launch group; the rocmd recovery-watcher plus rocm-core's shared readiness/constants; and the README/docs/xtask/Cucumber-scenario group), plus my own direct re-verification of the one candidate finding and of CI/check status against the live commit. Base 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe…head eb3ce2a0d8adc0517862a67f04114df535b94bf9.
Did not run as separate dedicated passes: a repo-wide history/git blame sweep and a "prior review comments on earlier commits of this PR" pass — this PR has earlier review rounds on previous commits that were deliberately not read here, per the blind-review method, so whether any earlier feedback was addressed is outside what this report can say. Agent-instruction adherence (this repo's AGENTS.md) and in-file code-comment compliance were folded into each file-group worker's brief rather than run as fully separate passes, which is a reasonable scaling choice for a PR this size but is disclosed rather than assumed equivalent to a dedicated pass.
One process note for transparency: a single combined gh pr view call made early on (intended only to check CI status) also returned the PR's description and commit messages before the code-side findings were settled. No part of that content was passed to the five code-review workers, who worked blind from the diff and file contents only; the one finding that survived to this report was independently verified by direct re-reading of the current source, and the PR's stated intent was consulted only afterward, as the method requires, to check it against the already-settled findings.
Not reconciled against any prior review discussion on this PR.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…tup exit on the mock lane A stall or deadline stop now records the status that matches how far the endpoint got (`starting` when it never listed the model), instead of `running`, which elsewhere means the model is listed. When the engine reports failed but its process is still up, the message now says a running `rocm daemon` with `server-recover` may restart it. Scenario serve-60 drives an engine that exits during interactive startup on the no-GPU mock lane: an `e2e-oom-fault-injection` build swaps the managed child for a stand-in when ROCM_E2E_SIMULATE_STARTUP_EXIT=1. It checks the non-zero exit and message, runs the `rocm logs --service` command the failure names, and checks the record is settled `stopped`. The docs say the silent limits arm while the endpoint has not listed the model, which is what the code checks, rather than "the port stays closed". Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…artup deadline A stall or deadline stop leaves the server running, and rocmd spares a live startup only until SERVE_STARTUP_DEADLINE. The message now says a running daemon with server-recover may restart it after that. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The interactive wait was chosen by stdout alone, while its status line and failures go to stderr, so `rocm serve m 2>log` in a terminal could wait up to 30 minutes with nothing drawn. A redirected stderr now keeps the piped 45-second status budget. Also applies rustfmt to the previous commit. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
@juhovainio thanks for the review. The inline threads each have a reply; these are the points from the review body that had no anchor:
|
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
I went through the whole diff. I found nothing that blocks on its own, but there are a few things I'd like addressed before this merges.
Please fix
- A download line that ends at 100% followed by a quiet engine is still classified as Downloading, so a healthy server that logs nothing for 3 minutes after the download finishes gets reported as a download stall (and now exits non-zero in a terminal). See the inline comment on
line_is_download. - The 2-minute silent-unreachable limit can fire on a slow start that has written no log line yet (for example a container-engine pre-spawn step). The PR body says the stall limits are unverified against real engine cadence; please confirm on the GPU lane or narrow the condition.
classifybuilds the "the engine reported failed" / "the server process exited" labels andquoted_logrecognises them by string equality, and the e2e step greps the same words. AGENTS.md section 3 asks for one shared constant or deriving the text from the decision. Carrying an explicit optional quoted line would remove the cross-function matching (and a real log line equal to a label would no longer be silently dropped).- The message for a live engine-
failedrecord says a running daemon with server-recover "may restart it first". I could not find a test in the diff that shows rocmd doing that; only therunning/startingguard is covered. Please point at the test that proves it, or soften the sentence.
Scope: this bundles the startup wait, redaction, probe backoff, the rocmd recovery guard, the rocm logs --service hint fix and a new audit event on --wait. The rocmd guard and the hint/audit fixes could land separately and would be easier to review apart.
For the author to confirm
- The rocmd guard delays recovery of a live service that lost its listener for up to 30 minutes.
- Piped serve exits 0 at 45s while a terminal now exits non-zero for the same slow server.
- Stall constants are fixed rather than per-engine.
CI: prek (lint / hygiene) is failing on the current head; I couldn't see which hook, so please look. Windows jobs were still pending when I checked.
This is a read of the diff only; I did not build or run it.
| line.chars().filter(char::is_ascii_alphabetic).count() >= 4 && line.len() >= 6 | ||
| } | ||
|
|
||
| fn line_is_download(line: &str) -> bool { |
There was a problem hiding this comment.
A download line that ends at 100% and is followed by a quiet engine keeps the phase at Downloading, so a healthy server that logs nothing for 3 minutes after the download is reported as a download stall. Consider treating 100% (or a completed download) as Loading.
There was a problem hiding this comment.
Fixed in 7fa1193: a download line whose last percent is 100 is now a finished download (line_is_active_download), so a quiet engine after it is classified Loading and judged by the 5-minute load limit, not the 3-minute download stall. A line below 100 is still a download. a_finished_download_line_is_judged_as_loading covers both and fails when the 100% check is removed. README, docs/testing.md and docs/manual-testing.md say so.
| readiness, | ||
| ); | ||
| } | ||
| if view.phase == ServeStartupPhase::Unreachable |
There was a problem hiding this comment.
This 2-minute silent limit also fires on a start that has written no meaningful log yet (e.g. a slow pre-spawn step on a container engine). The PR body says the stall limits are unverified against real engine cadence; please confirm on the GPU lanes or note the risk.
There was a problem hiding this comment.
Narrowed in 7fa1193 rather than confirmed. I cannot confirm it on a GPU lane: no lane drives a stalled interactive serve, and I have no GPU here. A start with no meaningful log line now gets as long as a quiet load: SILENT_UNREACHABLE is defined as SILENT_LOAD (5 minutes, was 2). a_start_with_no_log_gets_as_long_as_a_quiet_load pins that, and fails with the old 2-minute value. The comment on the constant (reworded in 4c6cf19) no longer gives an image pull as the example, since docker pull writes its own status lines to the log. A stop on this limit still leaves the server running. The PR body keeps "not verified against real engine cadence" as a known gap.
| } | ||
| } | ||
|
|
||
| fn quoted_log(model: &str, engine: &str, view: &ServeStartupView) -> Option<String> { |
There was a problem hiding this comment.
quoted_log recognises the two fixed failure labels by comparing strings produced in classify. AGENTS.md section 3 asks for a shared constant or deriving this from the decision; carry an explicit optional quoted line instead of matching label text.
There was a problem hiding this comment.
Fixed in 7fa1193: quoted_log is gone. ServeStartupView now carries log_line: Option<String>, set in classify where the label is chosen (the engine line for Failed/Listing/Downloading/Loading, None for the fallback labels), and stop_decision copies it, so it no longer needs model/engine. An engine line that reads exactly like a fallback label is now quoted. a_stop_quotes_the_engine_line_not_a_fallback_label covers both cases and fails if the line is filtered by text again. The e2e step that greps "the server process exited" asserts the user-visible headline (user_message builds it from the stop reason), not the label match.
…ne by decision A download line at 100% followed by a quiet engine was held to the 3-minute download stall; it is now judged by the 5-minute load limit. A stop quoted the log by comparing the label with this module's own fallback strings. The view now carries the engine line it shows, set where the label is chosen, so an engine line that reads like a fallback label is still quoted, and stop_decision no longer needs model and engine. A start with no log line yet now gets as long as a quiet load (5 minutes, was 2), so a slow pre-spawn step is not reported as silent sooner. A rocmd test now shows that a `failed` record whose process is still up is recovered, which the engine-failed message warns about. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Conflicts: README.md (serve paragraph keeps both the startup wait and the smoke-test-failure text), serve_cmd.rs and rocm-core lib.rs (both sides' additions), docs/architecture.md (both fault-injection seams), and model_serving.feature, where main took serve-59/60, so this branch's scenarios become serve-61/62. A DeploymentSummary test fixture gains main's new smoke_failure field. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The SILENT_UNREACHABLE comment no longer gives a container image pull as its example, which writes its own lines to the log, and names the test that pins the limit; the start.rs note about rocmd's guard names the function. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
@juhovainio thanks. Each of your three inline comments has a reply. The points from the review body:
|
Why
Interactive
rocm servestopped after 45 seconds and printed a finished "Deployment summary (not ready yet)" while the model was still downloading or loading. The user saw a static spinner, then a summary that looked final, and had to guess whether the server would ever come up.What changes
SERVE_STARTUP_DEADLINEinrocm-core). A stall or the deadline leaves the server running, records it with the status that matches how far the endpoint got (startingif it never listed the model,runningif it did), namesrocm logs --service <id>androcm services stop <id> --yes, and says a runningrocm daemonwithserver-recovermay restart it once 30 minutes have passed since it started. An engine failure or exit is settled the same way a--waitsettles a server that exited (stopped, or the engine's ownfailed, with the same memory guidance and an audit event, which a--waitthat sees the server exit now records too), and the CLI does not start the background helper for it. If the engine reported failure but its process is still up, the message names the stop command and says the daemon may restart it first (afailedrecord is a recovery candidate whether or not its process is alive:a_failed_record_with_a_live_process_is_still_recoveredinapps/rocmd/src/watchers/server_recover.rs); the record keeps the engine's ownfailed.--wait, the interactive wait for a fresh launch first prints the service id,rocm logs --service <id> --followand, for a public bind, the API key, because it can fail without printing a summary.INFERENCE_PROBE_RETRY_INTERVAL), instead of an 8-second request every 250 ms poll.--api-key/token/secret/password/private-key values, compact JSON keys,Authorizationheaders,hf_tokens) before they reach the status line or an error. A list value has only its first item masked.rocmd(another binary) does not restart a live startup. The server-recover watcher spares a liverunningservice whose port is not open yet, and a livestartingone past the 5-minute stale-start window, until the 30-minute deadline. A dead pid, a pid whose recorded start time does not match (recycled), or a recoverable engine healthcheck is still recovered. The event re-check that authorizes a restart applies the same guard as the scan. Trade-off: the guard coversrunningtoo because Lemonade and AIM write their own staterunningbefore their port opens, so it also delays, until 30 minutes after its start or last restart, the recovery of a live service that served and then lost its listener.service_logs_hintis now the only place that spellsrocm logs --service <id>, with the id shell-quoted like the other hints: the deployment summary, the startup failure, the--waitmessages and the attached (--verbose) path's threelogs:rows use it. Those rows used to printrocm logs <id>, whichrocm logsparses as a free-text search, not that service's log.Unchanged: piped output, or a redirected stderr (
rocm serve m 2>log, where nothing would be drawn), keeps the 45-second status budget and returns a readiness status;--waitkeeps its own wait, with no CLI-side stall limit or deadline;--verbose/--foregroundstill stream the log.Verification
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo clippy -p e2e-cucumber --test e2e -D warnings,cargo xtask check-architecture-docandcheck-module-sizepass locally.cargo test --workspace --all-targetspasses locally except three tests that fail on this host onmaintoo (service_control::…stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_failsand twobuild_metadatagit_watch_paths_*tests), none of which this PR touches. Clippy is clean with and withoutrocm/e2e-oom-fault-injection,rocm/e2e-test-hooks.cargo xtask smokewas not run: on this host its isolated build cannot run (a rustc wrapper and no network namespace for bubblewrap).inference_probe_backs_off_to_the_retry_interval_after_listing(fake clock) andwait_spaces_inference_probes_while_the_model_is_listed(real wait against a fake endpoint) both fail;a_server_that_exits_during_the_wait_is_settled_stopped_and_not_supervisedfails;running:a_startup_stop_records_the_live_server_and_returns_the_typed_failurefails;a_background_serve_follows_the_startup_only_with_both_streams_on_a_terminalfails;a_finished_download_line_is_judged_as_loadingfails;a_stop_quotes_the_engine_line_not_a_fallback_labelfails;a_start_with_no_log_gets_as_long_as_a_quiet_loadfails;only_a_stall_or_the_deadline_leaves_the_server_runningfails;observe_launchreading the held copy's status, or reporting every process alive:observe_launch_reads_failure_and_liveness_from_the_manifestfails;effective_signalsmade a no-op:piped_wait_ignores_engine_failure_and_a_dead_processfails;wait_follows_a_download_until_the_server_servesfails;logs:row back torocm logs <id>:attached_logs_rows_select_the_service_log(parses the printed command with the real CLI) fails;--wait:a_wait_that_can_fail_without_a_summary_is_announcedfails;a_hint_denotes_the_argv_it_names(property test over hostile ids) fails;index + 1:redact_log_line_masks_keys_tokens_and_secret_flagspanics on a non-ASCII value;silent_limits_count_from_the_last_changefails;rocmdguard trusting a bare pid, or the event re-check skipping the guard:recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_oneandrecovery_leaves_a_live_starting_record_alone_until_the_startup_deadlinefail.serve-61(@requires-gpu).model_serving.featuredrives interactiverocm serveunder a real PTY: the wait line namesrocm logs --service <id>, that command runs, and the serve exits 0 with the endpoint listing the model. It needs a live engine, so it runs on the GPU lanes only; it was not run locally (no GPU), only compiled and clippy-checked.serve-62(mock lane,@requires-no-gpu @requires-os:linux @requires-oom-fault-injection). An engine that exits during interactive startup: withROCM_E2E_SIMULATE_STARTUP_EXIT=1, ane2e-oom-fault-injectionbuild (never shipped) swaps only the spawned child for a stand-in that logs a line and exits; record, log, wait and settling are production code. It checks the non-zero exit and the "server process exited" message, runs therocm logs --service <id>the failure names, checks the record isstoppedand no daemon was left. Passes locally, with serve-41 and serve-60 that share the fault-injection feature (cargo xtask e2e -- -n "serve-(41|60|62) "), and fails at thestoppedstep when settling writesfailed.decide_serve_startup/wait_for_serve_startupunit tests inapps/rocm/src/serve_progress.rs, the start path by theservice_launch::starttests, and therocmdguard byrecovery_leaves_a_live_*inapps/rocmd/src/watchers/server_recover.rs.Fetching N filesbar, one shard that takes longer than 3 minutes reads as a stall (the wait ends and exits non-zero, the server keeps running); a cache-size growth signal is the follow-up; the start-path tests arecfg(not(windows))(they spawnsleep), so on Windowsobserve_launchand the interactive wait are covered only by the platform-neutral unit tests.Scope
The stall limits are fixed constants, not per-engine settings, by choice: one auditable set until real engine cadence says otherwise. Piped output exiting 0 after 45 seconds while a terminal now exits non-zero for the same slow server is intended: the piped path is for scripts and CI that want a readiness status, the terminal path tells a person the startup failed.
--waitis not changed to follow the log.The pieces stay in one PR because each is part of the wait working: the
rocmdguard keeps the helper from restarting a startup the wait is reporting on, the logs hint is the command every failure message names, and the--waitaudit event comes from the same settling code the interactive failure uses.Commit history note
The scenarios this PR adds are
serve-61(@id:serve-interactive-startup-progress) andserve-62(@id:serve-startup-exit-names-log). They were renumbered whenmaintookserve-59andserve-60. Earlier commit messages on this branch use the numbers they had at the time (serve-55,serve-59,serve-60). The@idtags did not change.