Skip to content

fix(serve): keep startup progress until the server is ready or stalled - #817

Open
rominf wants to merge 15 commits into
mainfrom
fix/serve-startup-progress
Open

rominf wants to merge 15 commits into
mainfrom
fix/serve-startup-progress

Conversation

@rominf

@rominf rominf commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Interactive rocm serve stopped after 45 seconds and printed a finished "Deployment summary (not ready yet)" while the model was still downloading or loading. The user saw a static spinner, then a summary that looked final, and had to guess whether the server would ever come up.

What changes

  • Interactive serve follows the startup. When stdout and stderr are both terminals, the status line follows the engine log (a download percent, a load line, or a wait for the model to serve) until the endpoint answers an inference request. A ready server prints the deployment summary as before.
  • The wait has bounded exits. It fails, with a non-zero exit, when the engine reports failure or its process exits, when a download percent below 100 stops for 3 minutes (a line at 100% is judged as loading), when the endpoint has not listed the model and there is no engine log for 5 minutes (as long as the load limit) (counted from the last change, not from the launch), when a load line stops changing for 5 minutes while the model is not listed, or after 30 minutes (SERVE_STARTUP_DEADLINE in rocm-core). A stall or the deadline leaves the server running, records it with the status that matches how far the endpoint got (starting if it never listed the model, running if it did), names rocm logs --service <id> and rocm services stop <id> --yes, and says a running rocm daemon with server-recover may restart it once 30 minutes have passed since it started. An engine failure or exit is settled the same way a --wait settles a server that exited (stopped, or the engine's own failed, with the same memory guidance and an audit event, which a --wait that sees the server exit now records too), and the CLI does not start the background helper for it. If the engine reported failure but its process is still up, the message names the stop command and says the daemon may restart it first (a failed record is a recovery candidate whether or not its process is alive: a_failed_record_with_a_live_process_is_still_recovered in apps/rocmd/src/watchers/server_recover.rs); the record keeps the engine's own failed.
  • The wait announces itself. Like --wait, the interactive wait for a fresh launch first prints the service id, rocm logs --service <id> --follow and, for a public bind, the API key, because it can fail without printing a summary.
  • Inference probes are spaced. Once the endpoint lists the model, the one-token inference probe runs at once, then backs off from 1 second to 15 seconds (INFERENCE_PROBE_RETRY_INTERVAL), instead of an 8-second request every 250 ms poll.
  • Engine log lines are redacted (endpoint key, --api-key/token/secret/password/private-key values, compact JSON keys, Authorization headers, hf_ tokens) before they reach the status line or an error. A list value has only its first item masked.
  • rocmd (another binary) does not restart a live startup. The server-recover watcher spares a live running service whose port is not open yet, and a live starting one past the 5-minute stale-start window, until the 30-minute deadline. A dead pid, a pid whose recorded start time does not match (recycled), or a recoverable engine healthcheck is still recovered. The event re-check that authorizes a restart applies the same guard as the scan. Trade-off: the guard covers running too because Lemonade and AIM write their own state running before their port opens, so it also delays, until 30 minutes after its start or last restart, the recovery of a live service that served and then lost its listener.
  • One logs hint. service_logs_hint is now the only place that spells rocm logs --service <id>, with the id shell-quoted like the other hints: the deployment summary, the startup failure, the --wait messages and the attached (--verbose) path's three logs: rows use it. Those rows used to print rocm logs <id>, which rocm logs parses as a free-text search, not that service's log.

Unchanged: piped output, or a redirected stderr (rocm serve m 2>log, where nothing would be drawn), keeps the 45-second status budget and returns a readiness status; --wait keeps its own wait, with no CLI-side stall limit or deadline; --verbose/--foreground still stream the log.

Verification

  • cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, cargo clippy -p e2e-cucumber --test e2e -D warnings, cargo xtask check-architecture-doc and check-module-size pass locally.
  • cargo test --workspace --all-targets passes locally except three tests that fail on this host on main too (service_control::…stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails and two build_metadata git_watch_paths_* tests), none of which this PR touches. Clippy is clean with and without rocm/e2e-oom-fault-injection,rocm/e2e-test-hooks.
  • cargo xtask smoke was not run: on this host its isolated build cannot run (a rustc wrapper and no network namespace for bubblewrap).
  • New tests were broken on purpose and went red:
    • inference backoff disabled: inference_probe_backs_off_to_the_retry_interval_after_listing (fake clock) and wait_spaces_inference_probes_while_the_model_is_listed (real wait against a fake endpoint) both fail;
    • an ended launch sent down the live-stop path: a_server_that_exits_during_the_wait_is_settled_stopped_and_not_supervised fails;
    • the typed failure not returned, or a live stop that never listed the model recorded running: a_startup_stop_records_the_live_server_and_returns_the_typed_failure fails;
    • the policy chosen from stdout alone again: a_background_serve_follows_the_startup_only_with_both_streams_on_a_terminal fails;
    • a 100% download line treated as a download again: a_finished_download_line_is_judged_as_loading fails;
    • the quoted engine line filtered by label text again: a_stop_quotes_the_engine_line_not_a_fallback_label fails;
    • the no-log limit back at 2 minutes: a_start_with_no_log_gets_as_long_as_a_quiet_load fails;
    • either new sentence about a daemon restart removed from the stop message: only_a_stall_or_the_deadline_leaves_the_server_running fails;
    • observe_launch reading the held copy's status, or reporting every process alive: observe_launch_reads_failure_and_liveness_from_the_manifest fails;
    • effective_signals made a no-op: piped_wait_ignores_engine_failure_and_a_dead_process fails;
    • download lines not classified as downloads: wait_follows_a_download_until_the_server_serves fails;
    • the attached logs: row back to rocm logs <id>: attached_logs_rows_select_the_service_log (parses the printed command with the real CLI) fails;
    • the wait announced only for --wait: a_wait_that_can_fail_without_a_summary_is_announced fails;
    • the logs hint printing the id unquoted: a_hint_denotes_the_argv_it_names (property test over hostile ids) fails;
    • a redacted value cut at index + 1: redact_log_line_masks_keys_tokens_and_secret_flags panics on a non-ASCII value;
    • either silent check reading total elapsed time instead of time since the last change: silent_limits_count_from_the_last_change fails;
    • the rocmd guard trusting a bare pid, or the event re-check skipping the guard: recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one and recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline fail.
  • Scenario serve-61 (@requires-gpu). model_serving.feature drives interactive rocm serve under a real PTY: the wait line names rocm logs --service <id>, that command runs, and the serve exits 0 with the endpoint listing the model. It needs a live engine, so it runs on the GPU lanes only; it was not run locally (no GPU), only compiled and clippy-checked.
  • Scenario serve-62 (mock lane, @requires-no-gpu @requires-os:linux @requires-oom-fault-injection). An engine that exits during interactive startup: with ROCM_E2E_SIMULATE_STARTUP_EXIT=1, an e2e-oom-fault-injection build (never shipped) swaps only the spawned child for a stand-in that logs a line and exits; record, log, wait and settling are production code. It checks the non-zero exit and the "server process exited" message, runs the rocm logs --service <id> the failure names, checks the record is stopped and no daemon was left. Passes locally, with serve-41 and serve-60 that share the fault-injection feature (cargo xtask e2e -- -n "serve-(41|60|62) "), and fails at the stopped step when settling writes failed.
  • Gap: the stall, deadline and engine-failed-while-alive branches have no scenario: they need an engine whose log and readiness a test controls over minutes. They are covered by the decide_serve_startup/wait_for_serve_startup unit tests in apps/rocm/src/serve_progress.rs, the start path by the service_launch::start tests, and the rocmd guard by recovery_leaves_a_live_* in apps/rocmd/src/watchers/server_recover.rs.
  • Three timing tests pointed the wait at a closed port, which on Windows takes about two seconds to refuse; they now use a local endpoint that answers 503 at once, and the Windows job passes.
  • Not verified: the stall limits against real vLLM, Lemonade and AIM log cadence (needs a GPU lane). Known limit: the download stall keys on the last percent in the log line, so if an engine prints only a Fetching N files bar, one shard that takes longer than 3 minutes reads as a stall (the wait ends and exits non-zero, the server keeps running); a cache-size growth signal is the follow-up; the start-path tests are cfg(not(windows)) (they spawn sleep), so on Windows observe_launch and the interactive wait are covered only by the platform-neutral unit tests.

Scope

The stall limits are fixed constants, not per-engine settings, by choice: one auditable set until real engine cadence says otherwise. Piped output exiting 0 after 45 seconds while a terminal now exits non-zero for the same slow server is intended: the piped path is for scripts and CI that want a readiness status, the terminal path tells a person the startup failed. --wait is not changed to follow the log.

The pieces stay in one PR because each is part of the wait working: the rocmd guard keeps the helper from restarting a startup the wait is reporting on, the logs hint is the command every failure message names, and the --wait audit event comes from the same settling code the interactive failure uses.

Commit history note

The scenarios this PR adds are serve-61 (@id:serve-interactive-startup-progress) and serve-62 (@id:serve-startup-exit-names-log). They were renumbered when main took serve-59 and serve-60. Earlier commit messages on this branch use the numbers they had at the time (serve-55, serve-59, serve-60). The @id tags did not change.

@rominf
rominf requested a review from a team as a code owner October 30, 2026 16:29
@rominf
rominf requested a review from volen-silo October 30, 2026 16:29
Comment thread apps/rocm/src/serve_progress.rs Fixed

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 0557c5d

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change. No ticket named in the PR description.

Blocking

  • Commit is not GitHub-"Verified"; the commit-signatures/verify-commits gate fails — head commit 0557c5db. This repo's own AGENTS.md §2 states signing/sign-off are enforced (not advisory): "a blocking CI gate (commit-signatures) re-checks every PR with GitHub 'Verified' status." The job output names the fix directly: sign off (git commit -s --amend / git rebase --signoff <base>) or enable commit signing and re-sign, then re-push.
    Confidence 100 · mechanical · Fix: sign and/or sign off the commit per the job's own remediation text, then force-push.

  • clippy fails the build: clippy::missing_const_for_fn on recorded_status_after_stop, apps/rocm/src/serve_progress.rs:479, under this repo's -D warnings gate — the job output shows this is not advisory, it aborts compilation ("could not compile rocm (bin "rocm") due to 1 previous error").
    Confidence 100 · mechanical · Fix: see the next finding — fixing that dead branch naturally resolves this too (const fn becomes valid once the unreachable arm is simplified).

  • New CodeQL alert rust/cleartext-logging (security_severity: high), apps/rocm/src/serve_progress.rs:1068 — flags assert!(rendered.contains(&hint), "{rendered}") in the new test failure_message_names_the_same_logs_command_as_the_summary as writing api_key to a log sink. In this specific call api_key: None, so no real secret is exposed today, but the scanner is reacting to the general shape: formatting a whole DeploymentSummary (which legitimately carries a real API key when Some) into a test-failure message is exactly the pattern that would leak one in a future test built the same way. This is a new, currently-open, failing required check on this commit.
    Confidence 90 · architectural · Fix: assert a narrower, non-secret-bearing condition (e.g. compare just the logs-hint substring location without echoing the whole rendered block on failure), or justify/dismiss the alert explicitly if the team considers test-only None data out of scope.

  • recorded_status_after_stop's _ => "starting" arm is dead, untested code — apps/rocm/src/serve_progress.rs:479-485. Tracing decide_serve_startup (same file, ~line 403): the !process_alive branch always returns ProcessExited before any of DownloadStalled/Silent/SilentLoad/Deadline can be reached, so every ServeStartupStop this module ever constructs with one of those four reasons necessarily has process_alive == true. The function's only producer is stop_decision, and no test ever builds a non-terminal-reason stop with process_alive: false to reach the final arm.
    Confidence 85 · logic · Fix: collapse to _ => "running" with a debug_assert!(stop.process_alive) (or a comment) documenting the invariant — this also makes the function trivially const fn, resolving the clippy failure above.

  • apps/rocm/src/serve_summary.rs:180 hand-rolls rocm logs --service {} instead of calling the sibling helper service_logs_hint that this same PR introduces and that the very same function already calls correctly two lines earlier (line 143). service_hints.rs's doc comment states the helper's entire purpose: "Shared so a startup failure and the deployment summary cannot name two different log commands." The two strings are byte-identical today, but nothing ties them together going forward — exactly the drift class the helper exists to prevent, violated within the PR that introduces it.
    Confidence 85 · mechanical · Fix: replace the inline format! at line 180 with crate::service_hints::service_logs_hint(&summary.service_id).

  • apps/rocm/src/serve_cmd.rs:977: return Err(error.context(helper)) makes the secondary daemon-helper failure the headline, demoting the real startup failure — anyhow::Error::context(self, context) keeps self (the startup failure, built from ServeStartupStop::user_message, which names rocm logs/rocm services stop) as the cause and makes context (the helper failure) the new top-level Display message. main.rs prints errors via {e:?} (Debug), so the original message survives in the chain, but it's no longer the first line the user reads — backwards from the two errors' relative importance, and only reachable in the rare case ensure_background_helper_running_quiet itself fails right after a startup stop.
    Confidence 73 · logic · Fix: reverse the composition (e.g. helper.context(error)) so the serve-startup failure stays the headline.

Non-blocking

  • State-matrix doc table omits the deadline as an exit for "downloading"/"loading" rows — apps/rocm/src/serve_progress.rs:19-20. decide_serve_startup checks the overall deadline unconditionally before any phase-specific stall check (confirmed by the existing test wait_keeps_a_changing_download_alive_past_the_stall_limit, whose own comment says "a moving download must hit the deadline, not the stall limit"), so these two rows do exit on the deadline too — the unreachable row's column correctly lists both ("silent limit, or the deadline"), these two don't. Code and tests are correct; only the table is incomplete.
    Confidence 78 · mechanical · Fix: add "(or the deadline)" to both rows, matching the unreachable row's phrasing.

  • wait_for_serve_startup calls observe()+step() twice per 250 ms poll tick (once before the HTTP probe, once after), duplicating ~15 lines of near-identical logic. Correctness is unaffected (a terminal decision from the first call returns immediately, so the second never double-fires a stop/ready), but it can call on_view twice with the same label in one tick and is a simplification opportunity.
    Confidence 70 · mechanical · Fix: factor the shared "observe, compute token/timing, step" sequence into one helper called twice, or restructure to a single call per iteration.

Decisions for the author

  • rocm_core::SERVE_STARTUP_DEADLINE (30 minutes) is shared verbatim between the interactive CLI wait's give-up deadline and rocmd's background-recovery watcher's "don't restart a live, still-starting process" grace window (live_startup_still_in_progress) — tradeoff. This couples two independently-motivated concerns (interactive UX patience vs. background recovery safety) to one constant: tuning one later (e.g. shortening the interactive wait for snappier UX) would silently also shrink the watcher's grace window for every managed service, not just ones started interactively. Confirm this coupling is intentional rather than incidental.
  • The stall/silent timers are fixed global constants (DOWNLOAD_STALL = 3 min, SILENT_UNREACHABLE = 2 min, SILENT_LOAD = 5 min), not configurable per engine — tradeoff. If any of the three supported engines (lemonade/vllm/aim) can legitimately go quiet longer than 5 minutes during a real (non-hung) load step, that engine would now hit a false "stalled" error under the new interactive path, where previously it was never an error (the old 45-second behavior only ever printed a time-boxed summary). Worth confirming this was checked against all three engines' real log cadence, not just the synthetic timings the unit tests use.

Positive signals

  • decide_serve_startup is a pure function fully decoupled from I/O/timing, letting every lifecycle row be driven directly by a unit test with exact phase/label/reason assertions rather than weak substring checks — and the tests do this well.
  • The "readiness never downgrades" invariant (the best variable in wait_for_serve_startup, documented at lines 374-376) is an explicit, well-reasoned guard against flicker: one refused connection after the model has listed does not regress the status line.
  • probe_service_http_readiness is shared between the interactive wait and the historical polling path specifically so the status line and the real readiness check "cannot disagree about ready" — a clean elimination of a split-brain bug class.
  • service_launch.rs's start_managed_service always persists the updated ManagedServiceRecord (status, inference_verified_at, audit log) before ever branching to bail! on a startup stop — there is no path that mutates state and fails to persist it.
  • Interactive vs. piped mode is correctly gated on IsTerminal::is_terminal(&stdout()) (serve_cmd.rs:845), so none of the new stall-detection/hard-error behavior reaches a script or CI pipe — the lenient 45-second status-budget path is untouched for non-interactive callers.
  • The PR's own test-plan is unusually candid about what wasn't validated locally ("Full workspace cargo test and clippy were not run"), and that disclosure correctly predicted the CI failures found above.

Deployment notes

The shared SERVE_STARTUP_DEADLINE constant changes rocmd's recovery-watcher behavior for any managed service (not only ones started under the new interactive path) that is less than 30 minutes old, still "running" with a live pid, but endpoint-unreachable: it will no longer be auto-recovered during that window, where it may have been before this change. This looks intentional (it's exactly what the new recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one test pins), but it's worth calling out explicitly for anyone operating rocmd in production, since it widens the window before automatic recovery kicks in for any stuck or slow-starting service, not just ones launched interactively.

What this covered

Read completely: apps/rocm/src/serve_progress.rs (new, 1447 lines, including all tests), serve_cmd.rs, serve_summary.rs, service_launch.rs, service_hints.rs, engine_runtime_select.rs, main.rs, apps/rocmd/src/watchers.rs, crates/rocm-core/src/lib.rs and net.rs, xtask/src/ascii_output_contract.rs, README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, and this repo's AGENTS.md in full. Diff resolved against cc56179c…0557c5db by fetching the PR head read-only (git fetch https://github.com/ROCm/rocm-cli.git pull/817/head); the PR was never checked out. Work was split across three independent, blind, file-scoped fan-out workers (core state-machine module; wiring/integration layer; supporting infra + docs), then consolidated and every surviving finding re-verified against the source here, plus a design-review pass and a check of this PR's live CI status (get_check_runs) and code-scanning alerts.

Did not run: cargo build/test/clippy locally (relied on CI's own run, which failed — see Blocking); the e2e-cucumber/GPU-gated suite (no GPU available here, and the PR itself states none was run); a direct check of the three serving engines' (lemonade/vllm/aim) real log cadence against the new silent-load/download-stall timers (raised as an open question under Decisions for the author instead, since it can't be settled from the diff alone). The history/prior-changes passes were done lightly (confirmed these files' recent history is module-extraction refactors with no directly relevant prior review threads surfaced), not exhaustively. Not reconciled against any PR discussion or comments — the existing github-advanced-security[bot] comment was read only for its CodeQL alert detail, not treated as review discussion.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Keeping startup progress visible until the server is ready or stalled is the right fix, and pulling the decision into a pure decide_serve_startup with a test per state is a nice structure. A few things need fixing before this can merge.

Blocking

  • CI is red on this head for three checks that come from this change: clippy (missing_const_for_fn on recorded_status_after_stop), Commit signatures + sign-off, and CodeQL (rust/cleartext-logging on the test assert at serve_progress.rs:1068). Because clippy fails, the unit-test jobs were skipped, so none of the new tests have run in CI yet.
  • recorded_status_after_stop has an unreachable _ => "starting" arm (details inline). It is dead code, and it is what clippy trips on.
  • The PR body doesn't show that the new tests fail when the code they cover is broken (revert a guard, swap a branch, confirm red), and it says full workspace test and clippy weren't run. Please state which mutations you tried and rerun the full gate locally.

Worth addressing (inline comments)

  • Any start_managed_service error now starts the background helper and masks the real startup failure message.
  • The rocmd recovery guard may not cover a record that is still starting, or one left behind by Ctrl-C.
  • Stall limits (2 / 5 min) can now fail a server that would have become ready, and they aren't documented.
  • Engine log lines are echoed to the spinner and error text with only ASCII filtering. Launch args can contain tokens, so some redaction seems prudent.
  • The doubled observe/step block, and a stale "half-minute" in docs/manual-testing.md.

Also: the rocmd behavior change, the new rocm-core deadline constant and the service_logs_hint refactor aren't mentioned in the PR body. They touch another binary, so I'd list them. The two Strix Halo Ubuntu E2E lanes also failed after ~45 min; I didn't dig into those logs, so please check they aren't related.

Comment thread apps/rocm/src/serve_progress.rs Outdated
/// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one`
/// leaves alone until [`rocm_core::SERVE_STARTUP_DEADLINE`]. Engine failure
/// and a dead process are `failed`, which recovery is supposed to restart.
pub(crate) fn recorded_status_after_stop(stop: &ServeStartupStop) -> &'static str {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _ => "starting" arm can't be reached: decide_serve_startup returns ProcessExited before any other reason when the process is dead, so every other reason has process_alive == true. Collapse it to _ => "running" (with a comment or debug_assert!) and make the fn const fn. That also clears the missing_const_for_fn clippy failure.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1a9dfb7: the unreachable arm is gone, every stall/deadline reason maps to running with a debug_assert that the process is alive, and the fn is const.

Comment thread apps/rocm/src/serve_progress.rs Outdated
notes: Vec::new(),
};
let rendered = crate::serve_summary::render_summary(&summary);
assert!(rendered.contains(&hint), "{rendered}");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL flags rust/cleartext-logging on "{rendered}" here, since the summary carries the api key. Assert without echoing the whole rendered summary (e.g. print only hint).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1a9dfb7: the assert message names only the hint, not the rendered summary.

let mut token = String::new();
let mut best = EndpointReadiness::Unreachable;

loop {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The observe / token / step block is repeated again around line 589. I'd pull it into one helper. watch_engine is also applied in both adjusted_signals and decide_serve_startup, and step fires on_view twice per loop, so the spinner ticks at double rate.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1a9dfb7: each poll now does one observe, one probe (skipped when the engine failed or the process is gone), one decision and at most one status-line update. effective_signals is the single watch_engine adjustment. wait_observes_and_updates_the_status_line_once_per_poll fails if the engine is sampled or the line drawn twice per poll.

Comment thread apps/rocm/src/serve_cmd.rs Outdated
// The process is already recorded. Start supervision before
// returning the startup failure so a server that is still
// running is not left with no daemon.
if let Err(helper) = ensure_background_helper_running_quiet(summary_mode) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any error from start_managed_service reaches this arm, including a spawn failure where nothing was recorded, so the comment "the process is already recorded" isn't always true and the helper starts needlessly. error.context(helper) also makes the helper error the headline and pushes the startup-failure text (what the user needs to read) down to "Caused by". A typed error for the stop case would avoid both.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1a9dfb7: start_managed_service returns a typed ServeStartupFailed for a startup stop, and only that error starts the helper (supervise_after_launch_error, unit-tested on all three rows). A helper that fails to start is printed as a warning; the startup failure stays the error.

Comment thread apps/rocmd/src/watchers.rs Outdated
// A live process that has not opened its port is still the
// interactive serve wait's job. Restarting it here kills a
// download the CLI is reporting. A dead pid is not spared.
if record.status == "running" && live_startup_still_in_progress(&record, now) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard only spares status == "running". If the record is still starting during the wait (status looks to be written only after it), or is left over after a Ctrl-C of a wait that can now last up to 30 minutes, stale-start recovery could still restart a live download. Is starting covered? The new test only builds a hand-made running record.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The CLI writes running before the wait, but rocm services persists the engine's own starting state into the record, and that was restarted after 5 minutes. Fixed in 1a9dfb7: a live starting record is spared until SERVE_STARTUP_DEADLINE, the same as running; a Ctrl-C leaves one of those two, so it is covered. New test: recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline (inside the deadline, dead pid, at the deadline).

Comment thread apps/rocm/src/service_launch.rs Outdated
Some(service_id),
);
if let Some(stop) = startup_stop {
bail!("{}", stop.user_message(requested_model, engine, service_id));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bail! turns a stall into a non-zero exit for a server that may still become ready (Silent at 2 min, SilentLoad at 5 min, e.g. a container pull that logs elsewhere, or a long weight load / graph capture with no output). Previously interactive serve printed a "not ready" summary and exited 0. Could you confirm the limits against real vllm/lemonade/aim log cadence and document them in README / manual-testing?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in README, manual-testing and testing.md in 1a9dfb7, including the non-zero exit and that the server keeps running. I have not measured real vllm/lemonade/aim log cadence against the 2/5-minute limits; that needs a GPU lane.

Comment thread docs/manual-testing.md Outdated
The default command, without `--foreground` or `--verbose`, leaves the server
in the background. While it starts, the status line follows the engine: a
download percent, a load line, or a wait for the model to serve. A model that
is still loading after the first half-minute keeps that line; it is not

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"after the first half-minute" disagrees with the 45-second piped budget in docs/testing.md; please reword.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in 1a9dfb7.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 1a9dfb7

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (base cc56179c…head 1a9dfb75, the review-fix commit on top of the previously-reviewed 0557c5db). Implements "keep startup progress until the server is ready or stalled" for interactive rocm serve. This commit is itself the author's response to two earlier review rounds (one from this same automation, one human); all three of their blocking CI findings (clippy missing_const_for_fn, the CodeQL rust/cleartext-logging alert, commit-signatures) and the dead _ => "starting" arm are fixed — confirmed by re-reading the current code, not assumed.

Blocking

  • No CI check has executed for this exact head commit. get_status/get_check_runs for 1a9dfb75333f2c52c1f0210986587da2e4ca68dc return only a single Read-the-Docs status; actions_list finds no workflow run anywhere referencing this sha (the three runs on this branch are all still pinned to the superseded 0557c5db). None of build-and-test, clippy, Test (affected crates), windows-build-and-test, Commit signatures + sign-off, CodeQL, or the E2E suite have run on the commit actually being merged. Per this method's own rule, an unverified class is a finding, not an assumption.
    Confidence 100 · Fix: push a no-op commit or otherwise re-trigger CI for this head and confirm green before merge.

  • service_launch.rs's attached-serve path (run_attached_service) still hand-rolls rocm logs <id>/rocm logs {service_id} at three call sites (lines ~949, 994, 1035), bypassing the service_logs_hint helper this very commit introduces to prevent exactly this drift. rocm logs (main.rs) parses a bare positional as a free-text search query, not a service id — only --service <id> selects one service's log (confirmed against the clap definition). So these three hints are not equivalent to service_logs_hint(id) ("rocm logs --service {id}") and are likely to find nothing when run as printed. service_hints.rs's own doc comment states the helper exists "so a startup failure and the deployment summary cannot name two different log commands" — this commit correctly routes serve_summary.rs through it but missed the sibling site in the same file family.
    Confidence 80 · logic · Fix: replace all three call sites with service_hints::service_logs_hint(&service_id).

  • The interactive wait can now issue an unthrottled real inference probe (openai_chat_completion_probe, 8s timeout) on every ~250ms poll for up to the full 30-minute SERVE_STARTUP_DEADLINE, once the model reaches Listing. probe_service_http_readiness (engine_runtime_select.rs) calls the raw probe directly rather than through the throttled/latched INFERENCE_PROBE_RETRY_INTERVAL (15s) path crates/rocm-core/src/net.rs documents and that commit fdfa620 (#150) added specifically for this scenario ("One observed lemonade serve listed the model in 4s and could not answer for the next 6 minutes… the unthrottled cost lands exactly where it is most visible"). Previously this same unthrottled probing was bounded to 45 seconds; it is now reachable for up to 40x longer with no comment indicating the tradeoff was deliberately re-accepted.
    Confidence 75 · architectural · Fix: route through the throttled probe helper (engine_state_inference_verified/managed_service_endpoint_model_ready), or explicitly document why the interactive path accepts the extra probe volume.

Non-blocking

  • engine_runtime_select.rs's module doc (lines 76-77) still claims wait_for_service_http_ready_with_progress is "read from service_launch.rs", but this PR's diff removes that import/call entirely (confirmed via repo-wide grep: the only remaining caller is the internal wait_for_service_http_ready wrapper). This exact module recently had an identical class of stale-caller-claim fixed days earlier (commit 9301dfc, #804) — the same drift has recurred one paragraph away.
    Confidence 85 · mechanical · Fix: update the claim to name the real caller, or drop it in favor of the repo's docs/architecture.md cross-reference convention.

  • No unit test in service_launch.rs exercises the new persistence/typed-error wiring in start_managed_service directly (the "persist launch_status before returning Err(ServeStartupFailed(...))" ordering, and the readiness == Serving && startup_stop.is_none() guard). The decision table itself (decide_serve_startup) is well unit-tested in serve_progress.rs, but the call-site wiring added here has zero test coverage of its own.
    Confidence 70 · architectural

Decisions for the author

  • live_startup_still_in_progress (rocmd's recovery watcher) trusts raw PID liveness with no start-tick identity check, over a 30-minute window — tradeoff. Elsewhere in this crate, PID-trust decisions are paired with a start-tick token specifically because of PID reuse (service_record.rs's names_same_processes doc comment); this new guard checks only process_is_running(pid). The bad outcome (PID reuse during the 30-minute window causing a dead service to be wrongly treated as still starting) is low-probability but non-zero, and this is a real, if narrow, widening from the previous purely time-based 5-minute check. Confirm this is an accepted tradeoff for the initial release of this guard.
  • SERVE_STARTUP_DEADLINE (30 min) is shared verbatim between the interactive CLI wait's give-up deadline and rocmd's recovery-watcher grace window — tradeoff, already raised in the prior round and still true: tuning one later silently reshapes the other.

Positive signals

  • decide_serve_startup remains a pure, fully unit-tested state machine with one test per lifecycle row; the prior round's dead-arm/clippy finding is cleanly resolved (collapsed to _ => "running"), and const fn now compiles.
  • Credential redaction is now applied to engine log lines before they reach the spinner/status line or failure text (redact_log_line, masking api_key, bearer/basic auth, hf_ tokens, compact JSON key fields) — directly answers the prior round's "no redaction" concern, verified by reading the implementation.
  • supervise_after_launch_error's error composition was specifically re-verified this round and is correct: start_managed_service returns Err(ServeStartupFailed(...)) with no further .context() wrapping, so downcast_ref::<ServeStartupFailed>() matches at the top level and the startup-stop message stays the user-visible headline — this fixes the prior round's confidence-73 finding about a .context() headline swap; no further chain-walking assumption was needed.
  • The "readiness never downgrades" (best) invariant and the shared probe_service_http_readiness helper (eliminating a split-brain between the status line and the real readiness check) are both sound, well-documented designs.
  • Interactive vs. piped dispatch is correctly gated on IsTerminal::is_terminal(&stdout()), so none of the new stall-detection/hard-error behavior reaches a script or CI pipe.

Deployment notes

  • Unchanged from the prior round: SERVE_STARTUP_DEADLINE (30 min) widens rocmd's auto-recovery grace window for any managed service that is running-but-endpoint-unreachable and under 30 minutes old, not just ones launched interactively.
  • Engine log lines now undergo redaction before reaching any user-visible surface — a behavior change worth noting for anyone who greps raw rocm serve output for a credential during debugging.

What this covered

Read completely at head 1a9dfb75333f2c52c1f0210986587da2e4ca68dc (base cc56179c): serve_progress.rs, serve_cmd.rs, serve_summary.rs, service_launch.rs, service_hints.rs, engine_runtime_select.rs, apps/rocmd/src/watchers.rs, crates/rocm-core/src/lib.rs/net.rs, main.rs's touched regions, README/docs/testing/manual-testing diffs, and the full AGENTS.md. Six independent blind fan-out workers covered: the core state machine (context for this round); serve_cmd/serve_summary/service_hints wiring; service_launch/engine_runtime_select (two independent passes); the agent-instruction/history/code-comments source-reading pass (two independent passes); rocmd watchers + the shared deadline constant. Every candidate surfaced by those workers was re-verified against the source before inclusion above; several (the three prior round's CI failures, the headline-swap concern) were independently re-confirmed as now fixed rather than carried forward on faith.

Did not run: cargo build/test/clippy locally (no Rust toolchain in this environment) — relied on direct code tracing plus the CI-is-absent finding above, since there is no green run to lean on for this commit. The three supported engines' (lemonade/vllm/aim) real log cadence against the new silent-load/download-stall timers could not be checked from the diff alone (same open question as the prior round). Not reconciled against this PR's own review discussion, which was not read.


🤖 Generated with Claude Code

@rominf

rominf commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Response to the automated review of 1a9dfb75. The branch is rebased onto current main; the two PR commits are squashed into one and the fixes below are a second commit.

Blocking:

  • CI had not run on that head. It has run on the new head; status is in the checks.
  • run_attached_service printed rocm logs <id>. Fixed. Confirmed first: rocm logs takes a bare argument as a free-text search, so the three rows did not select the service's log. All three now go through service_logs_hint (logs_row in service_launch/attach.rs), and so do the --wait messages main added meanwhile. attached_logs_rows_select_the_service_log parses each printed row with the real CLI and asserts it selects that service and searches nothing; it fails with the row reverted to rocm logs <id>.
  • Unthrottled inference probe after Listing. Fixed. Once the endpoint lists the model, the inference probe runs at once and then backs off 1, 2, 4, 8 and then every 15 seconds (INFERENCE_PROBE_RETRY_INTERVAL, the same interval rocm-core uses), timed from when the previous probe returned (InferenceProbeThrottle in serve_progress.rs). Piped output keeps probing each poll within its 45-second budget, as before. inference_probe_backs_off_to_the_retry_interval_after_listing drives it under a fake clock (ten minutes of polls, each probe taking the full 8 s timeout); wait_spaces_inference_probes_while_the_model_is_listed runs the real wait against a fake endpoint that lists and never answers. Both fail with the backoff disabled.

Non-blocking:

  • Stale caller claim in engine_runtime_select.rs. Updated: the module doc now names service_http_lists_model and service_http_inference_answers and their reader; probe_service_http_readiness is private again.
  • No test of start_managed_service's stop wiring. Added, through the spawn hook: a_startup_stop_records_the_live_server_and_returns_the_typed_failure (a live child, silent limit: record running, no inference verification, typed failure naming the logs and stop commands, child still alive) and a_server_that_exits_during_the_wait_is_settled_stopped_and_not_supervised. Both fail when the wiring is broken.

Decisions:

  • PID reuse in the rocmd guard. Accepted for now: the guard spares a service only while a recorded pid is alive and the record is younger than 30 minutes, and a reused pid inside that window is the narrow case. Pairing it with the start-tick identity is a reasonable follow-up.
  • Shared SERVE_STARTUP_DEADLINE. Intentional: the rocmd grace window is defined as "while the interactive wait may still be following this startup", so the two should move together.

Also from the rebase: main added rocm serve --wait. --wait keeps its own wait (no CLI-side stall limit or deadline); the stage-following wait applies to a fresh launch without it. An engine failure or process exit during the interactive wait is now settled like a --wait that saw the server exit (stopped, or the engine's own failed, plus the memory guidance) instead of being recorded failed and handed to the helper.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head cba35c6

Review — needs work

Full review of the whole change (base 1530f6b1…head cba35c61). No ticket named in the PR description. This is the third review round on this PR; the prior two rounds' blocking findings (clippy missing_const_for_fn, a CodeQL cleartext-logging alert, a missing commit-signature, a dead _ => "starting" arm, an unthrottled inference probe, and a hand-rolled rocm logs <id> hint) are all confirmed fixed by direct re-reading of the current code, not assumed.

Blocking

  • CI is currently red on this exact head. Windows build and test (debug) (and the gating windows-build-and-test check) fail with real test failures inside this PR's own new serve_progress.rs tests: wait_stops_a_hung_download_before_the_deadline (assertion left == right failed: left: Deadline, right: DownloadStalled at apps/rocm/src/serve_progress.rs:1524), wait_keeps_a_changing_download_alive_past_the_stall_limit, and piped_wait_ignores_engine_failure_and_a_dead_process. These look like real-clock/thread::sleep-based timing tests that are not robust to Windows' coarser scheduling/timer resolution — the PR's own "Not verified" note only flags the cfg(not(windows)) integration tests in service_launch::start, not these platform-neutral serve_progress.rs unit tests, which it implies already run clean on Windows.
    Confidence 100 · logic · Fix: make the timing-sensitive assertions robust to coarser clocks (fake/injectable clock rather than real sleeps, or wider margins), and confirm green on the Windows runner before merge.

  • service_logs_hint (new this PR) skips the module's own mandatory shell_quote step — apps/rocm/src/service_hints.rs:51-53. The module's doc comment states the invariant directly: "The service id goes through shell_quote here, once, so every hint is a line whose words are exactly the argv it names" — and the sibling service_action_hint/remote_service_stop_hint in the very same file both call it. service_logs_hint does not. ServiceId validation makes this unlikely to bite today, but it is a confirmed violation of an explicitly adopted, documented pattern in the one module whose stated purpose is preventing exactly this class of drift (it even cites a prior incident, #417).
    Confidence 88 · mechanical/architectural · Fix: wrap service_id in shell_quote(service_id) the same way the sibling functions do; add a hostile-id test alongside the existing service_hints.rs coverage.

  • docs/architecture.md's service_hints.rs entry is incomplete — it lists serve_progress.rs as a new caller but omits serve_cmd.rs, which newly calls service_hints::service_logs_hint directly at apps/rocm/src/serve_cmd.rs:1182 (confirmed: no service_hints reference existed in serve_cmd.rs before this PR). The adjacent shell_quote.rs entry's blanket claim of covering "the service_hints.rs hints" is now also inaccurate given the finding above. cargo xtask check-architecture-doc only checks file presence, not prose accuracy — per AGENTS.md §6, "the check reads no prose, so keeping an entry's description true is still on you."
    Confidence 85 · mechanical · Fix: add serve_cmd.rs to the service_hints.rs entry's caller list.

  • No Gherkin scenario was added for new, CLI-observable interactive-serve behavior, despite this repo's own established convention for exactly this situation. AGENTS.md §3 requires user-observable behavior (status-line output, exit codes) to be covered by a scenario, and explicitly anticipates a GPU-only scenario: "if the scenario can only run on a gated lane (@requires-gpu, @nightly), say so in the PR text and name the lane" — i.e. write it and tag it, not omit it. tests/e2e-cucumber/features/model_serving.feature already carries 30 existing @requires-gpu-tagged scenarios (confirmed by direct grep), so this is a well-established, actively-used pattern in this exact file. This PR's diff to that file adds only an explanatory comment ("There is no Gherkin scenario for this yet... a scenario would be @requires-gpu"), not a tagged scenario.
    Confidence 80 · architectural (agent-instruction adherence) · Fix: add a @requires-gpu scenario for the core happy-path (status line follows a download/load, then prints the deployment summary) alongside the existing unit coverage, per the established pattern in this file.

  • Branch-mutation test gap: the Unreachable and Loading silent-stall checks in decide_serve_startup can't be told apart from a field-swap regression. The Unreachable check reads timing.elapsed (serve_progress.rs:536-538) and the Loading check reads timing.since_progress (serve_progress.rs:548-550), but every test covering either branch (unreachable_keeps_the_starting_label_until_the_silent_limit, unreachable_with_no_log_stops_when_nothing_arrives, loading_past_the_old_budget_keeps_reporting_the_log_line, loading_with_no_new_log_stops) passes elapsed/since_progress as either equal or moving in the same direction — directly confirmed by reading the fixtures. Swapping the two fields in either branch would pass the whole suite. The analogous Downloading-phase check has exactly this protection (wait_keeps_a_changing_download_alive_past_the_stall_limit keeps since_progress resetting while elapsed grows past the stall limit); Unreachable/Loading have no equivalent.
    Confidence 82 · logic/test-quality · Fix: add one integration-level test per phase (mirroring wait_keeps_a_changing_download_alive_past_the_stall_limit) that keeps the relevant progress signal moving while overall elapsed time grows, proving the deadline alone — not the silent-stall check — is what fires.

  • Potential panic in mask_value on a multi-byte UTF-8 trailing character — apps/rocm/src/serve_progress.rs:821-835. word.rfind(|c| ...) returns the byte offset where the matched character starts; the code then does index + 1 to get body_end, which is only a valid slice boundary for a 1-byte (ASCII) character. For a value ending in e.g. é (2 bytes), body_end lands mid-character and &word[body_end..] panics ("byte index N is not a char boundary"). Traced and confirmed by byte-level analysis. This is not reachable in the currently-wired production path only because every real caller (sanitize_log_line → latest_meaningful_log_line, used by observe_launch) ASCII-filters the line before redact_log_line/mask_value ever see it — an unenforced calling convention on a String documented only as "already one sanitized line," not a type-level guarantee, and no test in the file exercises redact_log_line/mask_value with non-ASCII input.
    Confidence 85 · logic · Fix: compute body_end from a char boundary (e.g. word.char_indices().rev().find(...), or word.len() minus the matched char's byte length) rather than index + 1; add a non-ASCII regression test.

Non-blocking

  • is_secret_name's suffix allowlist misses common variants like secret_key/private_key — apps/rocm/src/serve_progress.rs:792-806. It matches via name.ends_with(suffix) against ["api_key", "apikey", "token", "password", "secret"]; a name such as secret_key has "secret" as a prefix, not a suffix, so ends_with("secret") is false and the value is never masked. Verified directly against the suffix list. Narrower than what the redaction feature is billed as doing, though within the PR's own stated scope ("api-key/token/secret values") it's a partial, not total, miss.
    Confidence 70 · logic · Fix: also match on these names as whole words/components (split on _/- and check membership), not only as a trailing suffix.

  • cli.rs's new serve help text overclaims for the "reuse an already-starting service" path. The new text says "while it starts, the status line follows what the engine is doing... until the server is ready or that wait fails" (apps/rocm/src/cli.rs, the serve doc comment). But for a plain rocm serve <model> (no --wait) that matches an equivalent service another invocation is still starting, wait_on_existing_service (apps/rocm/src/service_launch/readiness.rs:393) returns immediately with the untouched "starting" status — if !wait_until_ready || report.status == "ready" { return Ok(report); } — before ever invoking the wait loop or the "follows the engine" behavior at all. This is pre-existing behavior for the reuse path, not newly introduced, but the new, broader help text makes no exception for it.
    Confidence 75 · architectural/doc-accuracy · Fix: scope the help text's claim to a fresh launch, or note the reuse-of-starting-service case explicitly.

  • A manifest-write I/O failure in the "server still running" branch of start_managed_service can bury the real startup-stop diagnosis — apps/rocm/src/service_launch/start.rs:185-189 followed by the typed-failure return at line ~203. When a stall/deadline stop leaves the server running, the code calls record_wait_outcome(paths, &mut record, launch_status, ...)? before converting the stop into the typed ServeStartupFailed the user is meant to see; if that write fails, the raw I/O error propagates instead of the actionable stop reason. The sibling "ended" branch just above is deliberately robust to this (settle_ended_launch/settle_exited_service persist best-effort via let _ = ...write()), so this is a real asymmetry introduced by this refactor.
    Confidence 72 · logic · Fix: make the write best-effort here too (log/ignore a write failure) so the typed stop is still returned.

Decisions for the author

  • List-valued secrets leak all but their first item — tradeoff, already self-documented in redact_log_line's own doc comment ("A list value ('api_key': ['a', 'b']) has only its first item masked"). vLLM accepts repeated --api-key flags, so a printed launch-args dict with multiple keys would leak every key after the first. Confirm this is an accepted scope boundary for now rather than something to close in this PR.
  • rocm_core::SERVE_STARTUP_DEADLINE (30 min) is still shared verbatim between the interactive CLI wait's give-up deadline and rocmd's recovery-watcher grace window — tradeoff, raised in both prior rounds and unchanged. Tuning one later (e.g. a snappier interactive UX deadline) silently reshapes the daemon's auto-recovery window for every managed service, not just interactively-started ones.
  • live_startup_still_in_progress (rocmd's recovery watcher) still trusts raw PID liveness with no start-tick identity check, over the full 30-minute window — tradeoff, raised in the prior round and unchanged. PID reuse during that window could cause a dead service to be wrongly treated as still starting; low-probability but non-zero, and this crate has an existing start-tick pattern elsewhere for exactly this reason.
  • Two parallel inference-probe-throttling mechanisms now exist — the disk-persisted one in crates/rocm-core/src/endpoint_readiness.rs/service_record.rs (for the background health-check path) and the new in-memory InferenceProbeThrottle in serve_progress.rs (for the interactive wait). They share only the numeric constant (INFERENCE_PROBE_RETRY_INTERVAL), not the throttling logic itself. Reasonable given the different contexts (cross-invocation persistence vs. in-loop state), but worth confirming the duplication is intentional rather than something to unify later.

Positive signals

  • The prior round's blocking CI findings (clippy, CodeQL cleartext-logging alert, missing commit signature) and the dead _ => "starting" match arm are all cleanly resolved, confirmed by direct re-reading rather than assumed.
  • Inference-probe backoff is now correctly implemented and well tested (InferenceProbeThrottle, exact gap-sequence assertions from 1s to 15s) — this resolves the prior round's blocking finding about an unthrottled probe reachable for the full 30-minute wait.
  • apps/rocm/src/service_launch/attach.rs's three previously hand-rolled rocm logs <id> strings are now fully replaced with the shared service_hints::service_logs_hint helper, backed by a dedicated test (attached_logs_rows_select_the_service_log) that round-trips the printed command through the real CLI parser.
  • wait_for_serve_startup now performs exactly one observe/probe/decide pass per poll tick, resolving the prior round's "double observe per tick" note.
  • The "readiness never downgrades" invariant in wait_for_serve_startup's best variable is correctly enforced on every path (independently confirmed by two fan-out workers and the consolidating pass).
  • rocmd's new recovery-watcher guard is tested at precise boundary conditions (age exactly at the deadline; dead vs. live pid; both running and starting statuses).
  • The new state machine's extraction into its own serve_progress.rs file, rather than growing an existing file, matches this repo's documented "new subsystems get their own file from day one" guardrail, and stays comfortably under the 1,500-production-line cap (~859 lines).

Deployment notes

  • SERVE_STARTUP_DEADLINE (30 min) widens rocmd's auto-recovery grace window for any managed service that is running-but-endpoint-unreachable and under 30 minutes old, not only ones launched interactively (unchanged from prior rounds).
  • Engine log lines now undergo redaction before reaching any user-visible surface (status line, failure text) — worth noting for anyone debugging by grepping raw rocm serve output for a credential, though see the suffix-allowlist gap above for a residual limitation.
  • Two GPU-hardware E2E lanes (E2E tests (MI300X, current), E2E tests (W7900, current)) were still queued and had not completed as of this review's CI check; their result is unverified here.

What this covered

Read completely: all 19 changed files (apps/rocm/src/serve_progress.rs — new, ~1961 lines including tests; serve_cmd.rs, serve_summary.rs, service_hints.rs, engine_runtime_select.rs, cli.rs, main.rs's touched region; service_launch/{attach,readiness,start}.rs; apps/rocmd/src/watchers/server_recover.rs; crates/rocm-core/src/{lib.rs,endpoint_readiness.rs,net.rs,service_record.rs}; README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, xtask/src/ascii_output_contract.rs), plus this repo's root AGENTS.md in full. Diff resolved against 1530f6b1…cba35c61 by fetching the PR head read-only (git fetch origin +pull/817/head:refs/remotes/pr/817); the PR was never checked out.

Three independent, blind, file-scoped fan-out workers ran and reported: the core state-machine module (serve_progress.rs); the wiring/integration layer (serve_cmd.rs/serve_summary.rs/service_hints.rs/engine_runtime_select.rs/cli.rs/main.rs); and the service_launch module (attach.rs/readiness.rs/start.rs). Every candidate they surfaced was independently re-verified against the source here before inclusion or rejection.

Degraded for three planned passes: this environment hit a shared concurrency limit (repeated "concurrent subagent limit reached" errors from unrelated sessions sharing the host) that prevented spawning independent fan-out workers for (a) the rocmd/rocm-core watcher pass, (b) the docs/testing/xtask pass, and (c) a dedicated agent-instruction/history/prior-changes pass. I performed all three myself directly — reading every file in those groups completely, cross-checking git history and blame, and checking AGENTS.md compliance by hand — so that portion of the review ran single-context rather than as an independent blind pass; findings from it (e.g. the architecture.md staleness, the Gherkin-scenario gap, the help-text and manifest-write findings) should be weighed with that in mind, though each was independently verified against the source.

Live CI status was checked directly (get_check_runs/get_status) against this exact head commit, which surfaced the Windows test failures above; this was not something any fan-out worker was tasked with. Not reconciled against this PR's own review discussion or prior review comments, which were read only for the duplicate-review claim-check (commit IDs), not as review content.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 442ae1e9

Review — needs work

Full review of the whole change (base 1530f6b1…head 442ae1e9). No ticket named in the PR description.

Blocking

  • No Gherkin scenario added for the new, CLI-observable interactive-serve behavior, despite this repo's own established convention for exactly this case. AGENTS.md §3 requires user-observable behavior (status-line output, exit codes) to be covered by a scenario, and explicitly anticipates a GPU-only one: "if the scenario can only run on a gated lane (@requires-gpu, @nightly), say so in the PR text and name the lane" — write it and tag it, not omit it. tests/e2e-cucumber/features/model_serving.feature already carries 30 existing @requires-gpu/@nightly-tagged scenarios, an established, actively-used pattern in this exact file. This PR's diff to that file (tests/e2e-cucumber/features/model_serving.feature:3-10) adds only an explanatory comment ("There is no Gherkin scenario for this yet... a scenario would be @requires-gpu"), not a tagged scenario, and docs/testing.md's matching paragraph relies on unit tests instead — which AGENTS.md §3 explicitly rules out for this purpose ("a unit test asserting the internal helper does NOT discharge this; it proves the function, not the behavior"). The repo also already has a proven no-GPU technique for a structurally similar case (serve-41, @requires-no-gpu, using the e2e-oom-fault-injection seam to fabricate a failed-launch state) that the PR's comment doesn't address as an alternative.
    Confidence 85 · architectural (agent-instruction adherence) · Fix: add an @requires-gpu (or a fabricated-state @requires-no-gpu, following the serve-41 pattern) scenario for at least the ready-path and one failure path, per this file's established convention.

  • service_hints::service_logs_hint skips the module's own mandated shell_quote step, and docs/architecture.md repeats the now-false claim that it doesn't. apps/rocm/src/service_hints.rs:51-53: pub(crate) fn service_logs_hint(service_id: &str) -> String { format!("rocm logs --service {service_id}") } — no shell_quote call. The module's own doc comment states the invariant directly: "The service id goes through shell_quote here, once, so every hint is a line whose words are exactly the argv it names" — and the sibling service_action_hint/remote_service_stop_hint in the very same file both call it; service_logs_hint does not. rocm_core::ServiceId::new only rejects empty/whitespace-only ids, path separators, .., and control characters — it does not reject spaces, ;, $(), or quotes — so a legally-accepted id breaks the "hint names exactly the argv it runs" guarantee this file exists to provide (and cites a prior incident for, #417). docs/architecture.md's entry for this file repeats the same now-inaccurate claim verbatim ("It quotes the service id with shell_quote, so a hint names exactly the argv it runs."). The file's own proptest (a_hint_denotes_the_argv_it_names) covers the two sibling builders but not service_logs_hint, so nothing in the suite catches this.
    Confidence 90 · mechanical/architectural · Fix: wrap service_id in shell_quote(service_id) the same way the sibling functions do; add it to the existing proptest's hint list; the architecture.md entry becomes accurate once the code is fixed.

  • The Unreachable/silent-port stall check in decide_serve_startup uses timing.elapsed instead of timing.since_progress, unlike both of its sibling checks, and unlike the behavior this PR's own README documents. apps/rocm/src/serve_progress.rs:536-538: if view.phase == ServeStartupPhase::Unreachable && timing.elapsed >= policy.bounds.silent_unreachable. The Downloading check (:524-525) and the Loading check (:548-549) both correctly key off timing.since_progress (time since the state last changed), and README.md's own new text describes this exit as "when the port stays closed with no engine log for 2 minutes" — a duration of silence, not total time since launch. Because classify() (:383-436) falls into Unreachable whenever the latest tailed log window has no is_meaningful line at all (readiness not yet Listing), a server that produced real download/load output for most of a long startup and then has a brief gap (a scrolled-past tail window, a quiet transition between phases) would be judged by total elapsed, not by how long it has actually been silent — so a wait already past 2 minutes of total elapsed age can fail immediately with Silent the moment it first goes quiet, even though the silence itself just started. No test distinguishes the two fields for this branch: unreachable_with_no_log_stops_when_nothing_arrives passes at(SILENT_UNREACHABLE, SILENT_UNREACHABLE) (both fields equal), so a field swap here would not be caught — unlike the analogous Downloading case, which has a dedicated integration test (wait_keeps_a_changing_download_alive_past_the_stall_limit) proving since_progress, not elapsed, governs.
    Confidence 85 · logic · Fix: use timing.since_progress for the Unreachable check, matching its siblings and the README's own description; add an integration-level test mirroring wait_keeps_a_changing_download_alive_past_the_stall_limit for this phase.

  • rocmd's new "don't restart a live, still-starting service" guard trusts a bare PID-liveness check, with no process-identity verification, even though the crate already ships the exact mechanism for this and the needed data is already on the record. apps/rocmd/src/watchers/server_recover.rs:403-413 (live_startup_still_in_progress) checks only process_is_running(pid) for record.supervisor_pid/record.engine_pid. crates/rocm-core/src/service_record.rs already persists supervisor_start_ticks/engine_start_ticks specifically so a PID can't be trusted after recycling, and crates/rocm-core/src/proc_lifecycle.rs already ships a ready-made classifier for precisely this question — ProcessIdentity/identity_state/IdentityState::{Matches,Recycled,Indeterminate,Gone} — whose own doc comment states the problem this guard reintroduces: "Stopping a managed service by a persisted PID is unsafe on its own: PIDs are recycled, so a stale PID may have been reassigned to an unrelated process." The new guard does not call it. Window: SERVE_STARTUP_DEADLINE = 30 minutes. If the supervisor or engine process actually died and its PID was reused within that window, the guard reports "still starting" and suppresses recovery on a service that is actually dead for up to 30 minutes — a real availability regression against pre-PR behavior, using data and a helper that are already present and unused.
    Confidence 85 · logic/architectural · Fix: build ProcessIdentity::new(pid, record.supervisor_start_ticks) (and the engine equivalent) and call identity_state, treating only Matches as "still alive"; Recycled/Gone must not suppress recovery.

  • The live-startup guard above is applied in the scan path but not in the event re-verification path that actually authorizes a restart, contradicting the guard's own stated single-point-of-truth design. apps/rocmd/src/watchers/server_recover.rs:92-105 (service_record_matches_recovery_event) calls manifest_service_recovery_reason(record, ...).is_some() and endpoint_service_recovery_reason(record).is_some() directly — the unguarded helpers — while find_recoverable_service's scan (:328-336) and unreachable_endpoint_recovery_reason (:366-379) go through the new live_startup_still_in_progress guard. unreachable_endpoint_recovery_reason's own doc comment states the intent directly: "One function for both places that fall back to the endpoint, so neither can drop the guard" — but service_record_matches_recovery_event, the function that actually authorizes handle_server_recover_event's restart (dispatched from apps/rocmd/src/watchers.rs:184), is a third call site that was not routed through it. No test exercises this recheck path against a live-startup record, so the gap is untested as well as unguarded.
    Confidence 75 · logic · Fix: route both of service_record_matches_recovery_event's branches through unreachable_endpoint_recovery_reason / the same "starting"-status check find_recoverable_service uses, or factor that check into one function both call.

Non-blocking

  • mask_value can panic on a multi-byte UTF-8 trailing character. apps/rocm/src/serve_progress.rs:821-835: body_end is computed as index + 1 from word.rfind(...), where index is the byte offset where the matched character starts — valid only for a 1-byte (ASCII) character. A value ending in a multi-byte character (e.g. é) makes body_end land mid-character, and &word[body_end..] panics ("byte index N is not a char boundary"). Not reachable today: the only production path (sanitize_log_line → latest_meaningful_log_line, used by observe_launch) ASCII-filters the line before redact_log_line/mask_value ever see it. But wait_for_serve_startup's own loop (:673-675) calls redact_log_line directly on signals.log_line with no such guarantee at the type level — it is an unenforced calling convention, not a contract — and no test exercises non-ASCII input to mask_value/redact_log_line directly.
    Confidence 80 · logic · Fix: compute body_end from the end of the matched character (e.g. index + c.len_utf8()), not index + 1; add a non-ASCII regression test.

  • A manifest-write I/O failure in the "stall left the server running" branch of start_managed_service can bury the typed, actionable startup-stop message. apps/rocm/src/service_launch/start.rs:185-190 calls record_wait_outcome(paths, &mut record, launch_status, ...)? — whose last line is latest.write()? (a real, propagating error) — before constructing and returning the ServeStartupFailed{server_running: true, message: stop.user_message(...)} the user is meant to see. If that write fails, a bare I/O/anyhow error propagates instead. The sibling "ended" branch immediately above (:164-177) is deliberately robust to this: settle_ended_launch → settle_exited_service's write is best-effort (let _ = latest.write();). This is a real asymmetry this PR introduces between its two new typed-failure branches.
    Confidence 78 · logic · Fix: make the write best-effort here too (log/ignore a write failure) so the typed stop is still returned, matching the sibling branch.

  • Two docs/architecture.md entries this PR touches understate their actual callers. The service_hints.rs entry lists serve_progress.rs as a caller but omits serve_cmd.rs, which newly calls service_hints::service_logs_hint directly at apps/rocm/src/serve_cmd.rs:1182 (confirmed: no service_hints reference existed in serve_cmd.rs before this PR). Separately, the serve_progress.rs entry names only service_launch/readiness.rs's wait_for_launch_startup as a caller, but serve_cmd.rs (starting_label, ServeStartupPolicy, matches on ServeStartupFailed) and service_launch/start.rs (starting_label, ServeStartupFailed, ServeStartupPolicy/ServeStartupView/PIPED_SERVE_STATUS_BUDGET) call into it directly too. cargo xtask check-architecture-doc only checks file presence, not prose accuracy — per AGENTS.md §6, "the check reads no prose, so keeping an entry's description true is still on you."
    Confidence 75 · mechanical · Fix: add serve_cmd.rs to the service_hints.rs entry's caller list, and add serve_cmd.rs/service_launch/start.rs to the serve_progress.rs entry's.

  • cli.rs's new top-level serve help text overclaims for the "reuse an already-starting service, no --wait" path. The new text says "while it starts, the status line follows what the engine is doing... until the server is ready or that wait fails" (apps/rocm/src/cli.rs, the Serve doc comment, diff around line 426-432). For a plain rocm serve <model> that matches an equivalent service another invocation is still starting, wait_on_existing_service (apps/rocm/src/service_launch/readiness.rs:393-395) returns immediately — if !wait_until_ready || report.status == "ready" { return Ok(report); } — before any status-line following happens at all. This is pre-existing behavior for the reuse path, not newly introduced, but the new, broader help text makes no exception for it.
    Confidence 72 · architectural/doc-accuracy · Fix: scope the claim to a fresh launch (or one additionally waited on), or note the reuse-of-starting-service case explicitly.

Decisions for the author

  • rocm_core::SERVE_STARTUP_DEADLINE (30 min) is shared verbatim between the interactive CLI wait's give-up deadline and rocmd's recovery-watcher grace window — tradeoff. This is documented at its definition (crates/rocm-core/src/endpoint_readiness.rs:213-217: "rocmd's server-recover watcher uses the same bound so it does not restart a live process that has not opened its port yet"), so the coupling is visible rather than silent. It still means tuning one later (e.g. a snappier interactive UX deadline) reshapes the daemon's auto-recovery window for every managed service, not just interactively-started ones. Confirm this is an accepted, intentional coupling.
  • List-valued secrets leak all but their first item, and a single-token compact-JSON object with two secret fields leaks the second one unmasked (mask_compact_value, apps/rocm/src/serve_progress.rs:839-852, only splits on the first :). Partial, not total, redaction within the PR's own stated scope ("api-key/token/secret values"); worth confirming this residual gap is an accepted scope boundary for now.

Positive signals

  • decide_serve_startup is a pure function fully decoupled from I/O/timing, letting every lifecycle row be driven directly by a unit test with exact phase/label/reason assertions; wait_keeps_a_changing_download_alive_past_the_stall_limit is a genuinely mutation-resistant integration test proving the Downloading stall check is keyed on since_progress, not elapsed (the sibling Unreachable check lacks this protection — see Blocking).
  • Inference-probe backoff (InferenceProbeThrottle) is well tested with an exact gap-sequence assertion ([1, 2, 4, 8, 15, 15, ...]), and redaction is proven to run before truncation (a_key_cut_at_the_length_limit_is_masked_before_truncation), closing an easy leak-via-partial-prefix bug.
  • apps/rocm/src/service_launch/attach.rs's three previously hand-rolled rocm logs <id> strings are now fully replaced with service_hints::service_logs_hint, backed by a test that round-trips the printed command through the real CLI parser rather than just pinning its text.
  • apps/rocmd's crate-layering invariant holds: it depends only on rocm-core/rocm-engine-protocol and generic crates, never on rocm.
  • The new recovery-watcher boundary tests are precise (age exactly at the deadline; dead vs. live pid; >= comparison matched by the fixtures' intent), and u32::MAX-as-dead-pid is handled safely (libc::pid_t::try_from fails first, so no accidental broadcast-kill).
  • start_managed_service's two new typed-failure branches both persist state and emit an audit event before returning their error (modulo the Non-blocking write-ordering asymmetry above); --wait's death path gained an audit event it did not previously have.
  • All new timing numbers added across README.md, docs/manual-testing.md, and docs/testing.md (3 min / 2 min / 5 min / 30 min / 45 s / 1 s→15 s) were checked against the actual constants and match exactly; xtask/src/ascii_output_contract.rs's new exemption entries match the two non-ASCII literals actually present in serve_progress.rs.
  • serve_progress.rs's production code is ~859 lines, comfortably under the 1,500-line module-size cap, and has its own docs/architecture.md entry from day one, matching this repo's "new subsystems get their own file" guardrail.

Deployment notes

  • SERVE_STARTUP_DEADLINE (30 min) widens rocmd's auto-recovery grace window for any managed service that is running-but-endpoint-unreachable and under 30 minutes old, not only ones launched interactively — documented at its definition, but worth calling out for anyone operating rocmd in production.
  • Engine log lines now undergo redaction before reaching any user-visible surface (status line, failure text) — worth noting for anyone debugging by grepping raw rocm serve output for a credential, though see the compact-JSON and list-valued gaps above for residual limits.
  • CI for this exact head (442ae1e9) is green across the full CI/CodeQL/Security scan workflows (clippy, commit-signatures, build-and-test, affected-crate tests, Windows build/test, coverage ratchet, docs/link checks all passed). The self-hosted GPU hardware E2E lane (E2E self-hosted, covering live vLLM/Lemonade/AIM) was still queued/pending with zero jobs started as of this review and had not run; the PR's own text already flags the GPU-engine log-cadence question as unverified for the same reason.

What this covered

Read completely: all 19 changed files (apps/rocm/src/serve_progress.rs — new, ~1966 lines including tests; serve_cmd.rs, serve_summary.rs, service_hints.rs, engine_runtime_select.rs, cli.rs's touched regions, main.rs's one-line diff; service_launch/{attach,readiness,start}.rs; apps/rocmd/src/watchers/server_recover.rs; crates/rocm-core/src/{lib.rs,endpoint_readiness.rs}; README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, xtask/src/ascii_output_contract.rs), plus this repo's root AGENTS.md in full, and crates/rocm-core/src/{service_record.rs,proc_lifecycle.rs} as supporting context for the rocmd findings. Diff resolved against 1530f6b1…442ae1e9 by fetching the PR head read-only (git fetch https://github.com/ROCm/rocm-cli.git pull/817/head into a scratch clone); the PR was never checked out.

Five independent, blind, file-scoped fan-out workers ran in parallel and reported: the core state-machine module (serve_progress.rs); the serve_cmd/serve_summary/service_hints/engine_runtime_select/cli.rs wiring layer; the service_launch module (attach.rs/readiness.rs/start.rs); the rocmd watcher plus shared rocm-core constants; and the docs/testing/xtask surface. Every candidate they surfaced was independently re-verified against the source here (direct byte-offset tracing for the UTF-8 panic, direct confirmation of the shell_quote/doc-claim gap, direct confirmation that ManagedServiceRecord already carries the start-tick fields identity_state needs) before inclusion or rejection; several weaker candidates (suffix-only secret-name matching, a dormant unclamped-backoff path, a hardcoded-phrase collision, a second typed-error-shape inconsistency) were read and dropped as below the reporting bar. The history and prior-changes passes were done directly by the consolidating reviewer (git log/blame on every touched pre-existing file, and a direct check of one specific prior-review-linked claim about engine_runtime_select.rs's module doc, which turned out to still be accurate at this head and is not reported as a finding).

Live CI was checked directly (get_check_runs/list_workflow_runs/list_workflow_jobs) against this exact head commit, re-polled until the CI workflow reached a final success conclusion; this was not something any fan-out worker was tasked with. Not reconciled against this PR's own review discussion or prior review comments, which were not read as review content (only consulted for the duplicate-review commit-SHA check, per this task's own instructions).


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@rominf

rominf commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Response to the automated review of cba35c61. Fixed in 442ae1e9 and 58f4888b unless noted.

Blocking:

  • Windows CI red. Fixed in 442ae1e9. Three serve_progress tests pointed the wait at a closed loopback port; on Windows a refused connect takes about two seconds, so each poll outlasted the tests' limits. They now use a local endpoint that answers 503 at once. windows-build-and-test passed on 442ae1e9.
  • service_logs_hint skipped shell_quote. Fixed. The property test a_hint_denotes_the_argv_it_names now covers the logs hint over hostile ids and fails with the id printed bare.
  • docs/architecture.md service_hints.rs callers. Added serve_cmd.rs. With the quoting fixed, the shell_quote.rs entry's claim holds again.
  • No Gherkin scenario. Not added in this PR. A scenario needs rocm serve against a live engine in a PTY: the mock lane stubs the HTTP endpoint and never runs rocm serve, so it would be @requires-gpu with new PTY serve steps. I have left that decision to the maintainers rather than add GPU-only steps I cannot run here; the gap is stated in the description.
  • Unreachable/Loading clock swap untested. Fixed: silent_limits_read_their_own_clocks moves one clock past each limit while holding the other below it; swapping the field in either check fails it.
  • mask_value multi-byte panic. Fixed: the value is cut at the end of its last character. redact_log_line_masks_keys_tokens_and_secret_flags has two non-ASCII rows and panics with the old index + 1.

Non-blocking:

  • secret_key / private_key not masked. Fixed: a secret word anywhere in an underscore name is masked, and private_key as a suffix. Covered in the same redaction test.
  • Help overclaims for reusing a starting service. Scoped: the help and README now say the announcement and status line apply to a fresh launch.
  • A failed record write after a live stop hides the stop. Fixed: after a stall or deadline stop the rewrite is best-effort, like the ended path, so the stop is the error returned.

Decisions:

  • List-valued secrets. Accepted for this PR and stated in the description; masking every item of a list is a follow-up.
  • Shared SERVE_STARTUP_DEADLINE and PID liveness without start-tick identity in the rocmd guard: answered in the response to the previous round.
  • Two probe throttles. Intentional: the rocm-core one is persisted across invocations on the record; the interactive one lives for one wait and backs off within it. They share the 15-second interval.

Interactive rocm serve stopped after 45 seconds with a static not-ready
summary while a download or load was still moving. The status line now
follows the engine log until the endpoint can serve or that progress
stalls, and a failure names the logs and stop commands. Piped output keeps
the 45-second status budget, and `--wait` keeps its own wait, with no
stall limit or deadline.

Like `--wait`, the interactive wait first prints the service id, how to
follow its log and a public bind's key, since it can fail without a
summary. Liveness is read from the manifest with the zombie-aware pid
check, so a dead supervisor ends the wait.

A startup stop is a typed ServeStartupFailed: only it starts the
background helper, and a helper failure is a warning beside it. Engine
log lines are redacted (endpoint key, --api-key/token/secret values,
compact JSON keys, Authorization headers, hf_ tokens) before they reach
the status line or failure text.

rocmd's server-recover watcher no longer restarts a live `running` or
`starting` record younger than SERVE_STARTUP_DEADLINE, so it does not
kill a download the CLI is still reporting.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Once the endpoint lists the model, the interactive wait sent an 8-second
inference probe on every 250 ms poll for up to 30 minutes. The probe now
runs at once, then backs off from 1 to 15 seconds
(INFERENCE_PROBE_RETRY_INTERVAL), timed from when the last one returned.
Piped output keeps its per-poll probing within the 45-second budget.

An engine failure or process exit during the interactive wait was
recorded `failed` and handed to the background helper, which a launch the
CLI has already reported as failed must not be. It is now settled the
way a `--wait` that saw the server exit is (`stopped`, or the engine's own
`failed`), with the same memory guidance and an audit event, and only a
stall or the deadline, which leave the server running, start the helper.

The attached path printed `rocm logs <id>`, which `rocm logs` reads as a
free-text search. Every logs hint now goes through service_logs_hint.

rocmd's two endpoint-unreachable checks share one guarded helper, and the
dead --wait parameter of wait_for_record_ready is gone.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…once

Three serve_progress tests pointed the wait at a closed loopback port.
On Windows a refused loopback connect takes about two seconds, so each
poll outlasted the tests' stall and deadline limits and the Windows job
failed. They now use a local endpoint that answers 503 at once, which is
the same "model not listed" reading without the platform's connect delay.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…ndary

service_logs_hint printed the service id bare, unlike every other hint in
service_hints.rs, whose module promises a line that pastes back as the
argv it names. It now goes through shell_quote too.

mask_value cut a value one byte past the start of its last character,
which panics when that character is wider than a byte. It now cuts at the
character's end. A secret word anywhere in an underscore name
(`secret_key`, `--private-key`) is masked as well as one at the end.

After a stall or deadline stop the record rewrite is best-effort, so the
stop stays the error the user reads. A test pins which clock each silent
limit reads, and the help and README scope the wait announcement to a
fresh launch.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…identity

The unreachable silent limit counted the whole wait, so a long startup
that went quiet for a moment could stop at once. It now counts from the
last change of what the wait observes, as the load limit does and as the
README describes.

rocmd's live-startup guard trusted a bare pid for up to 30 minutes. It
now checks the recorded start time with identity_state, so a recycled pid
does not keep a dead service from being recovered. The event re-check
that authorizes a restart now uses the same guarded reasons as the scan,
so an event raised before a startup was spared cannot restart it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/serve-startup-progress branch from eb66e15 to ad1b07c Compare November 11, 2026 10:31
@rominf

rominf commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Response to the automated review of 442ae1e9. The branch has since been rebased on current main; fixes are in the commits after 802eee2a.

Already fixed on the next head (58f4888b, now 9e8419b8), as answered in the previous response: service_logs_hint quoting (with the property test), the mask_value multi-byte cut, the best-effort record write after a live stop, serve_cmd.rs in the service_hints.rs entry, and the help text scoped to a fresh launch.

New in ad1b07cd:

  • Unreachable silent check used total elapsed time. Fixed: it now counts from the last change of what the wait observes, like the load check and as the README says. silent_limits_count_from_the_last_change holds one clock past the limit and the other below it for each check; reading elapsed in either fails it.
  • rocmd guard trusted a bare pid. Fixed: live_startup_still_in_progress checks the recorded start time through rocm_core::identity_state and spares only Matches (a record with no start time keeps the legacy best-effort match). Both rocmd tests have a recycled-pid row (start time that is not the process's own), which is recovered.
  • Event re-check skipped the guard. Fixed: service_record_matches_recovery_event uses the same guarded reasons as the scan (stale_manifest_recovery_reason, unreachable_endpoint_recovery_reason). Both rocmd tests assert an earlier manifest or endpoint event no longer authorizes a restart of a live startup; they fail with the unguarded helpers.
  • serve_progress.rs entry callers. Added service_launch/start.rs and serve_cmd.rs.

Not changed:

  • Gherkin scenario. Same answer as before: it needs rocm serve against a live engine in a PTY. The serve-41 seam fabricates a launch's on-disk state without running the wait, so it cannot show the status line following the engine. I have left the choice of a @requires-gpu scenario to the maintainers; the gap is stated in the description.
  • Compact JSON with two secret fields, and list values. Accepted for this PR: the first secret in a one-word object and the first item of a list are masked; the rest is a follow-up.
  • Shared SERVE_STARTUP_DEADLINE. Intentional, as answered earlier.

@rominf

rominf commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio, on the points in your review body that were not inline threads (inline ones are answered in their threads):

  • CI red on that head. Clippy, CodeQL and signatures were fixed in the earlier round. The branch is now rebased on current main, and every check passed on ad1b07cd, including windows-build-and-test and the E2E consolidated report.
  • Mutations. The PR description now lists, per new test, what was broken and that the test went red: the inference backoff, the ended-launch path, the typed stop error and recorded status, observe_launch's failure and liveness fields, the piped path's effective_signals, download classification, the silent limits' clocks, the rocmd pid identity and event re-check, the logs-hint quoting, the attached logs: rows, and the wait announcement.
  • Full gate. cargo fmt --check, workspace clippy with -D warnings, the e2e clippy, cargo test --workspace --all-targets, check-architecture-doc and check-module-size were run locally; the only failures are three tests that fail on this host on main too, plus two timing tests in untouched modules that failed once under load and pass alone. Details are in the description.
  • PR body coverage. It now lists the rocmd behaviour change, the new rocm-core deadline constant and the service_logs_hint refactor.
  • Strix Halo E2E lanes. All four Strix Halo lanes (Ubuntu release and nightly, Windows, WSL2) passed on cba35c61, the first head after the rebase; the later heads changed only CLI and rocmd code and tests, and the GPU lanes were not re-triggered for them.

Since your review: the rebase picked up main's rocm serve --wait, which keeps its own wait; an engine failure or exit during the interactive wait is settled like a --wait that saw the server exit, so it is no longer recorded failed and handed to the helper; the inference probe backs off after the model lists; and the rocmd guard now checks the pid's recorded start time and also covers the event re-check. Re-requesting your review.

@rominf
rominf requested a review from juhovainio November 11, 2026 10:55

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this. The interactive wait, the state matrix, and the paired message-plus-behaviour tests in start.rs are solid. One thing needs to change before merge, and a few smaller points are worth a look.

Needs to change

  • No Gherkin scenario covers the user-observable rocm serve change. Interactive serve now exits non-zero in new situations (stall, deadline, engine failure) and prints a new announce block and changed log hints. AGENTS.md section 3 asks for a scenario in tests/e2e-cucumber/features/ and, when it can only run on a gated lane, for the PR text to name that lane. A comment explaining the absence doesn't cover it. Please add the @requires-gpu scenario (or get an explicit maintainer exemption and say so in the PR text).

Worth a look (non-blocking)

  • The 5-minute silent-load limit can make rocm serve exit non-zero on a healthy model that is just quiet while loading. The server keeps running, but scripts will see a failure. Consider a longer default or an override.
  • A stalled or deadline-stopped server is recorded as running with no inference verification. Please confirm nothing reads that as ready (services list, reuse of a matching live service, chat).
  • The inline inference probe can block for up to 8s, which freezes the spinner and delays noticing an engine failure.
  • The rocmd guard defers recovery for up to 30 minutes after create or restart. Looks intended and tested; just confirming it is the tradeoff you want for a hung engine.
  • "Ctrl-C stops waiting, the server keeps running" is now printed for the interactive wait but no test asserts it for that path.
  • Redaction is heuristic (list values mask only the first item, user:pass@host is not handled), so "credentials masked" in the README is broader than the guarantee. "Known credential forms" would be more accurate.
  • Minor: start_managed_service is at 20 arguments, decide_serve_startup repeats near-identical stop_decision calls, and quoted_log matches on label strings that classify also produces.

pub(crate) const SILENT_UNREACHABLE: Duration = Duration::from_mins(2);

/// A load line that stops changing while the port is still down.
pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A 5-minute silent limit will fail a healthy engine that is quiet while loading weights or capturing graphs, with a non-zero exit. The PR notes the real engine cadence is unverified; consider a longer default or an env override, or confirm the tradeoff is intended.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended tradeoff, not verified against every engine: five minutes with no change in the log line and the port still down is treated as a hang. The stall only ends the wait. It exits non-zero and the message says the server is still running and how to follow it (rocm logs --service <id> --follow) or stop it, so a healthy engine that was just quiet is not killed. I did not add an env override here to keep the change scoped; if you want one, say so and I will add it as a follow-up.

if !signals.engine_failed && signals.process_alive {
let listed =
service_http_lists_model(engine_name, host, port, &probe_model, endpoint_api_key);
match throttle.readiness(listed, &mut Instant::now, || {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inference probe runs inline here and can block up to the probe timeout, so the spinner stops ticking and an engine failure isn't seen until it returns. Backoff bounds it; maybe note it in the doc comment if intended.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended, and now stated in the doc comment on wait_for_serve_startup (eb3ce2a): the probe runs inline, so a poll that sends one can take up to the probe timeout, during which the spinner does not tick and an engine failure is not read. The backoff (1 s growing to 15 s) bounds how often that happens.

Comment thread apps/rocm/src/service_launch/start.rs Outdated
}
// A stall or the deadline left the server up: `running`, which `rocmd`
// leaves alone until the startup deadline.
let launch_status = if startup_stop.is_some() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A stalled or deadline-stopped live server is recorded running with no inference verification. Please confirm no reader (services list, reuse of a live matching service, chat) treats running as serving.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the readers of running: managed_service_status (services list/status) and providers (chat) both re-run managed_service_endpoint_readiness for a running record and only promote it to ready when the inference probe answers, otherwise it stays running. running was already what a model-listed-but-unverified launch recorded (status_for_readiness(Listing)), so a stalled launch is no different. I did not trace the reuse-of-a-live-matching-service path separately: it keys on liveness, not on the status string, as far as I read it. Say if you want that path pinned with a test.

@@ -1,5 +1,14 @@
Feature: Model serving

# Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGENTS.md section 3 asks for a scenario for user-observable CLI behaviour, using a @requires-gpu lane when it can't run elsewhere. A comment explaining the absence doesn't cover it; please add the gated scenario or get an explicit exemption.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in eb3ce2a: scenario serve-55 (serve-interactive-startup-progress, @requires-gpu) runs rocm serve under a real PTY, checks the wait line says Ctrl-C stops waiting (not the server), runs the rocm logs --service <id> command it names, and checks the serve exits 0 with the endpoint listing the model. It cannot run in default CI or on the no-GPU lane, only on the GPU lanes; I could not run it locally (no GPU), so it is compile- and clippy-checked only and not yet seen to fail or pass. The stall, deadline and engine-failed branches stay covered by the unit state matrix, since they need a controllable engine.

Resolve docs/architecture.md against the per-crate table layout from
#1016: serve_progress.rs gets a row, and the service_hints.rs,
service_launch/readiness.rs, service_launch/start.rs and
watchers/server_recover.rs rows carry this branch's changes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (base b109db12c7a59f5f5c6f38d1e4a2191ff8851b53 against head ad1b07cd37499935a2192133e2cec86f05619d8c). No ticket referenced in the PR description.

Blocking

  • Engine-log secret redaction can be bypassed for exactly the formats it's meant to catch (a quoted "Authorization"/"Bearer" header, or a compact "key":"value" dump whose value contains a space) — apps/rocm/src/serve_progress.rs:748-756, apps/rocm/src/serve_progress.rs:815-821, apps/rocm/src/serve_progress.rs:846-859

    redact_log_line processes the sanitized log line token-by-token on spaces, and only re-arms its "mask the next word" state when the previous token is the literal, unquoted word bearer/basic. A dict-style header rendering such as {'Authorization': 'Bearer <token>'} (common in debug logs from Python request libraries) splits into 'Authorization':, 'Bearer, and <token>'; because the second token carries its opening quote attached, word.eq_ignore_ascii_case("bearer") is false, so the function masks the harmless word "Bearer" itself and never re-arms for the real secret that follows — which is then printed unchanged. The compact-JSON path (mask_compact_value) has the same gap from a different angle: it only looks for a closing quote inside the same whitespace-delimited token, so "api_key":"sk live 123" (a value containing a space) comes back completely untouched, not even partially masked, because no closing quote is found in the first fragment and the continuation token is never revisited.

    Confirmed by tracing the exact branches against "'Authorization': 'Bearer abc123secret'" and "api_key":"sk live 123", then reproducing the same control flow in a short standalone script and running it: both leak the full secret verbatim in the output. Confidence 90 · logic · Fix: track "currently inside a value for a flagged key" across tokens, so a continuation word is still masked after a quoted Bearer/Basic marker, and mask_compact_value keeps masking into the next token when no closing quote turns up in the first one, rather than only special-casing the exact unquoted words bearer/basic.

Non-blocking

  • Legitimate field names containing "token" get redacted as if they were secrets — apps/rocm/src/serve_progress.rs:807-809
    is_secret_name treats any underscore-separated name containing the bare word token (among others) as a secret, so an ordinary line like token_type_ids=[0, 0, 1] (a common ML tensor field, not a credential) has its value replaced with [redacted], hiding debugging information the feature isn't meant to hide. Confidence 70 · logic · Fix: narrow the match (e.g. names ending in _token/token) rather than "contains the word token anywhere in an underscore-joined name".

Decisions for the author

  • A newly-added audit event on the --wait death path — tradeoff
    Settling an ended launch through the shared helper now records an "error"-level audit event on the --wait path too, which previously didn't get one. This looks intentional (it gives the interactive and --wait failure paths the same audit trail), but since it's a side effect of a refactor rather than something the description calls out explicitly, it's worth confirming that was deliberate.

Positive signals

  • The "time since progress stalled" timers in wait_for_serve_startup are anchored to when the tracked status last changed, not to when the wait started — the specific off-by-reference-point mistake this kind of stall detector commonly makes — and that's covered by a test using plain Duration values rather than real sleeps.
  • mask_value/mask_compact_value operate on character boundaries (via char_indices) rather than raw byte offsets, with a test using a literal multi-byte character sitting exactly at the truncation limit — real, deliberate protection against a UTF-8-slicing panic.
  • The previously hand-rolled rocm logs <id> hint text (which rocm logs itself would have parsed as a free-text search rather than selecting that service) is now fully routed through one shared, shell-quoted helper everywhere it's printed; one of the new tests proves this by feeding the printed hint back through the real command-line parser instead of just string-matching it.

Deployment notes

  • The CLI's interactive startup deadline and the separate daemon's recovery watcher now share one constant, so the two can't drift out of sync with each other. Piped/non-interactive output is unchanged and keeps its original fixed-budget behavior.

What this covered

Read the full diff between the PR's merge base and its current head (19 files, roughly 3,000 added lines, dominated by one new ~2,000-line module). The new startup-decision module, the launch/wait integration it plugs into, and the readiness/attach/hint/summary files were each read in full, independently, with particular attention to the credential-redaction logic, where the finding above was traced against the exact source and then confirmed by reproducing the same control flow in a standalone script. The CLI flag/help-text changes, the recovery-watcher change in the companion daemon (including its pid-recycling and race-condition handling, which held up well), the shared constant it was given, and the README/docs/feature-file changes were also read in full; the repository's own contributor guide was checked against this change for its module-size limit, its "every source file gets an entry in the architecture doc" rule, and its doc/behavior-sync rules, all of which this change satisfies. These last few areas got one thorough read rather than a second independent pass.

Every required CI check passes (Linux and Windows build/test, lint, commit sign-off, static analysis, the security-scan suite, and the project's own module-size/architecture-doc/doc-link gates). One self-hosted GPU lane's consolidated report shows red, but only because that lane's run was itself cancelled; another workflow was running at the same time on a branch whose name indicates that exact lane is being retired, so this reads as known infrastructure churn rather than anything this change broke. Separately, GitHub currently reports this PR as not mergeable against main (a conflict to resolve by rebasing), which is a note for the author rather than a comment on the code.

Not reconciled against any review discussion on the PR.

@rominf

rominf commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main (2ffa4f9) to resolve conflicts with #1016; the only conflict was docs/architecture.md:

  • ported this PR's entries into the new per-crate tables: a serve_progress.rs row, the startup-wait and ServeStartupFailed changes on the service_launch/readiness.rs and service_launch/start.rs rows, the startup grace on the watchers/server_recover.rs row, the rocm logs --service hint on service_hints.rs (its caller list now names serve_cmd.rs, serve_progress.rs and service_launch/readiness.rs), and a boundary bullet saying which module owns what in the interactive wait.

The code merged cleanly into main's service_launch/ layout. No behaviour change beyond the port.

…tartup

Scenario serve-55 runs `rocm serve` under a real terminal, checks the wait
line names a log command that actually runs, and that the serve exits 0 with
the endpoint listing the model. Also documents that the inference probe runs
inline in the startup wait.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf requested a review from juhovainio November 13, 2026 10:37

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 2ffa4f9

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (19 files, +3081/-116: apps/rocm/src/serve_progress.rs new; serve_cmd.rs, service_launch/{start,readiness,attach}.rs, apps/rocmd/src/watchers/server_recover.rs, crates/rocm-core, CLI wiring/hints, docs, and the e2e feature/xtask fixture all touched). No ticket named in the PR title or body; the description is the sole stated intent.

Blocking

None.

Non-blocking

  • redact_log_line's documented "list value" limitation has no test — apps/rocm/src/serve_progress.rs:738 (doc comment), redact_log_line/mask_value (740-842), test redact_log_line_masks_keys_tokens_and_secret_flags (~1902-1963).
    The doc comment explicitly documents a security-relevant limitation: "A list value ('api_key': ['a', 'b']) has only its first item masked." Tracing the code confirms the current behavior matches that claim exactly (word-splitting + mask_next consumes only the first list item), but no test in the file exercises a list-shaped value at all, so a future change to mask_value's bracket/mask_next-reset handling that silently makes this worse (e.g. drops masking of even the first item) would go uncaught.
    Confidence 72 · logic/test-coverage · Fix: add a case asserting both that the first list item is masked and the second is not, e.g. on "non-default args: {'api_key': ['sk-abc', 'sk-def']}".

  • The new redaction only covers the new status-line/error surface; the same secrets remain fully exposed through --verbose/--foreground streaming and rocm logs, which this PR's own new failure text actively points users to — apps/rocm/src/serve_progress.rs:591-640 (ServeStartupStop::user_message, which on every failure path tells the user to run service_logs_hint/rocm logs --service <id>) vs. apps/rocm/src/service_launch/attach.rs:286-353 (stream_attached_logs/stream_attached_logs_no_tty, which forward the on-disk log file verbatim with no call to redact_log_line/sanitize_log_line — confirmed unchanged by this PR's diff) and apps/rocm/src/services_text.rs:519-570 (render_service_logs_text, backing rocm logs --service <id>, also prints raw lines with no redaction call, also unchanged). The on-disk log file itself (apps/rocm/src/service_launch/spawn.rs's attach_background_stdio, byte-identical pre/post-PR) has always held the engine's raw stdout/stderr, credentials included. So a secret masked for an instant on the interactive status line is fully visible an instant later via the exact command this PR's own messages recommend. This is a pre-existing gap (the wider unredacted surfaces predate this PR and the PR's own wording — "before they reach the status line or an error" — is accurately scoped to what it actually touches, not a broader claim), and this PR does not enlarge it; raising it because it sits squarely in the pattern of the change (credential handling in engine logs) and the new remediation text is what actively routes users to the unredacted view.
    Confidence 85 (verified by tracing all four files and confirming empty diffs on the unredacting paths) · architectural · Fix: non-blocking for this PR, but worth a follow-up — either redact at log-write time (so every reader benefits) or caveat that rocm logs/--verbose still show raw content.

Decisions for the author

None.

Positive signals

  • decide_serve_startup's branch order mirrors the module doc's state matrix closely, and nearly every row/edge case called out in the doc comment (e.g. "a log line that still says 'download' does not pull a listed model back into the download stall") has a directly corresponding, boundary-precise unit test rather than a loose substring check.
  • Credential redaction is adversarially solid where it does apply: known-endpoint-key substring replacement happens before word-level heuristics, double-masking is guarded against, Unicode/multi-byte values are handled via char_indices/len_utf8 rather than byte slicing, and redaction is confirmed to run before truncation (so a key straddling the 160-char cutoff can't leak its prefix).
  • The rocmd server-recover guard (live_startup_still_in_progress) and the CLI-side interactive wait are backed by the same shared rocm_core::SERVE_STARTUP_DEADLINE constant rather than a duplicated literal, crate layering stays clean (rocmd imports nothing from rocm), and both restart-triggering code paths (periodic scan and the event re-check) route through the identical guard function.
  • service_hints::service_logs_hint genuinely centralizes previously-duplicated (and in one case bugged — a bare rocm logs <id> that rocm logs parsed as free-text search) hint text, and the "hostile id" property test verifies the shell-quoting round-trips through real shlex parsing rather than a cosmetic string compare.
  • Documentation discipline is unusually good: every numeric constant (3 min download stall, 2 min silent-unreachable, 5 min silent-load, 30 min deadline, 45 s piped budget, 15 s max inference-probe backoff) matches exactly across README.md, docs/testing.md, docs/manual-testing.md and the code, and docs/architecture.md was updated in the same change for the new file and every touched file's changed role.
  • The "no Gherkin scenario for the interactive flow" gap is handled the way this repo's own rules ask for: named explicitly in tests/e2e-cucumber/features/model_serving.feature and docs/testing.md with the @requires-gpu rationale, and backed by state-matrix unit tests instead.

Deployment notes

rocm serve run interactively (in a real terminal, not piped) now blocks for up to 30 minutes before giving up on a stalled/failing startup, versus the previous fixed ~45-second wait. Piped/captured output is explicitly unchanged. Anyone wrapping rocm serve from a pseudo-terminal in semi-automated tooling (not a plain pipe) should expect the longer worst-case wait; this is already called out in the PR description's "Unchanged" section but is worth flagging explicitly here.

What this covered

Read the full diff between 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe and 2ffa4f9e09cf8b87553d61bab192d331ff47df8d (19 files). Every changed file was read completely (not just diff hunks) by one of six independent fan-out workers, grouped by coupling: (1) serve_progress.rs alone (new 2031-line file, production + tests); (2) service_launch/{readiness,start}.rs; (3) serve_cmd.rs + service_launch/attach.rs; (4) apps/rocmd/src/watchers/server_recover.rs + the two rocm-core constant files; (5) cli.rs, engine_runtime_select.rs, serve_summary.rs, service_hints.rs, main.rs; (6) README/docs/e2e-feature/xtask-fixture consistency. A seventh worker ran the agent-instruction-adherence pass (against this repo's root AGENTS.md) and a git-history pass across the whole diff. Two additional targeted investigations (Step 7 design review) checked (a) whether the new redaction is undermined by other log-reading paths — yielded the non-blocking finding above — and (b) whether a coincidental "5 minutes" value shared between rocmd's pre-existing stale-start check and this PR's new SILENT_LOAD constant creates an implicit, undocumented coupling — traced directly and found not to be an issue: rocmd's new live_startup_still_in_progress guard (keyed on the 30-minute deadline plus pid-identity matching) short-circuits before the old 5-minute staleness check ever runs for a live, identity-matched process, so the two "5 minute" constants govern genuinely independent conditions, not a hidden shared assumption.

CI: all checks green at the reviewed commit except one GPU-lane check (E2E tests (MI300X, current)) still QUEUED at review time (not failed) and a couple of platform-matrix jobs correctly SKIPPED.

Did not run: cannot execute cargo test/cargo clippy/cargo xtask check-module-size/check-architecture-doc/check-crate-edges/coverage in this environment (no Rust toolchain or working-tree checkout available) — CI's green status for all of these was taken at face value rather than re-derived. The "prior changes" pass (review comments on earlier, unrelated PRs touching these same files) did not run; no practical way to retrieve that history here beyond the git commit log, which the history pass did check. Not reconciled against any prior review round or PR discussion thread on this PR — this review was produced blind, from the diff and the PR's own description only, per method.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for two process reasons; I found no blocking defect in the code itself.

  1. The PR is currently CONFLICTING with main. Because of that no CI has run on this head (only readthedocs), so everything under "Verification" is a local claim. Please rebase and get the checks green.
  2. The new failure behaviour (non-zero exit, message, settled record) is covered only by unit tests. The only scenario is the happy path and it is @requires-gpu. Per the contribution rules, a scenario for "engine exits during startup" on a lane that has runners, or a note in the PR text naming the gap, is needed (inline comment below).

The rest are smaller points, mostly inline:

  • A stall or deadline stop records the server as running even when it never listed the model.
  • The new rocmd guard is broader than interactive serve and delays recovery of any live service with a closed port for up to 30 minutes.
  • An engine-failed record with a live process stays failed, so rocmd can restart it behind the error the user just read.
  • The download stall check keys on the last NN%, which can false-positive on a slow multi-GB shard.
  • Docs say "the port stays closed" but the code arms on the endpoint not listing the model.

Also worth a look, no anchor:

  • The 30-minute wait policy keys on stdout being a TTY while the spinner draws on stderr, so rocm serve 2>log in a terminal waits 30 minutes with nothing drawn.
  • serve_cmd.rs is at about 1462 of the 1500-line limit, wait_for_serve_startup takes 8 arguments, and stop_decision is called with 6 arguments at 6 sites.
  • Please confirm the fixed 3/2/5/30-minute limits are deliberate and not per-engine.
  • Piped output exits 0 after 45s while a terminal now exits non-zero for the same slow server; worth stating as intended.

On the plus side, the injected-clock backoff tests, measuring silent limits from the last change, and the process-start-time check in the rocmd guard are well done.


/// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent
/// does not is still stalled.
fn download_fingerprint(line: &str) -> String {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

download_fingerprint keys on the last NN%. A Fetching N files: 20%| 1/5 bar only moves when a whole file finishes, so a slow multi-GB shard holds the same percent for longer than DOWNLOAD_STALL (3 min) and a healthy download is reported as stalled (non-zero exit). Please consider a byte/size-growth signal (log or cache-dir size/mtime) as an extra progress token, or say in the PR which real engine log cadence this was checked against (it is listed as not verified).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed in this PR, and stated in its body as a known limit. You are right about the mechanism: with only the Fetching N files bar visible, a single slow shard holds the percent past 3 minutes. The per-file byte bars that huggingface_hub prints change the line (and so the token) while a file downloads, but I have not checked that every engine writes them to the service log, which is why the cadence is listed as unverified. A stall only ends the wait: the server keeps running and the message says how to follow it. A cache-size growth signal is the right fix; I would like to do it as a follow-up rather than widen this PR. Say if you consider it blocking.

Comment thread apps/rocm/src/service_launch/start.rs Outdated
}
// A stall or the deadline left the server up: `running`, which `rocmd`
// leaves alone until the startup deadline.
let launch_status = if startup_stop.is_some() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A stall/deadline stop writes running even when readiness is Unreachable. Elsewhere running means "endpoint lists the model" (status_for_readiness(Listing)), so rocm services, dash and chat will show a server that has not opened its port as up. The new starting guard in rocmd already spares a live start until the deadline, so starting (or status_for_readiness(readiness)) would do. The test pins running.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e4933a9: a stall or deadline stop now records status_for_readiness(readiness), so a launch whose endpoint never listed the model stays starting; running is only written when the model is listed. rocmd already spares a live starting record until the startup deadline. a_startup_stop_records_the_live_server_and_returns_the_typed_failure now pins starting, and goes red with the old running branch restored.

{
// The engine failed or the process ended: settled the way a `--wait`
// that saw the server exit is, with the same memory guidance.
let message = stop.user_message(requested_model, engine, service_id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the engine-failed-but-process-alive case the record is left failed (settle_exited_service keeps the engine state). mark_launch_abandoned documents that rocmd restarts a failed record with no stop marker, which is why the --wait death path uses stopped. An already-running helper can therefore restart this server behind the error printed here, and the message ("stop stops it") does not say so. Same for a Deadline stop at age >= 30 min (see found_at_deadline). Please either settle it stopped/terminate it, or say in the message that it may be restarted.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in the message, in e4933a9: for an engine that reported failed while its process is still up, the text now adds "A running rocm daemon with server-recover enabled may restart it first." (pinned in only_a_stall_or_the_deadline_leaves_the_server_running, red without it). I did not settle that record stopped or kill it: the engine itself wrote failed and the process is still the engine's, so overriding it would hide the engine's own verdict from rocm services. A process that has exited is settled stopped (now checked end to end by serve-60). For a Deadline stop past 30 min the process is alive and the message already says it is still running and names the stop command.

@rominf rominf Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my reply above about the Deadline stop: the message did not say the server may be restarted. Fixed in ced7f76: every stop that leaves the server running (a stall or the deadline) now adds "If it is still not answering 30 minutes after it started, a running rocm daemon with server-recover enabled may restart it.", pinned in only_a_stall_or_the_deadline_leaves_the_server_running.

/// That is the window interactive `rocm serve` spends following a download
/// or load before the port accepts connections. Endpoint-unreachable
/// recovery during it restarts the process out from under that wait.
fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u128) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard spares any live-pid running/starting record with a closed port for 30 min after creation/last restart, for every managed service and not only one the interactive serve is waiting on. A service that was serving and whose listener then died is no longer recovered for up to 30 min. Consider scoping it to records the CLI wait left (a marker), or stating the trade-off as a decision in the PR.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept, and stated in the PR body as a trade-off. It cannot be narrowed to starting: the engines write their own state running before their port opens (router_serve.rs and direct_llama.rs in Lemonade, state.rs in AIM), and rocm services copies that into the record, so a running record with a closed port is the normal shape of a Lemonade/AIM startup. The cost is the one you describe: a service that served and then lost its listener within 30 minutes of its start or last restart, with its process still alive, waits until the deadline to be recovered. A dead process, or a recycled pid, is still recovered at once.

Comment thread README.md Outdated
second). A model that is still loading is not reported as a finished "not
ready" summary. The wait fails, and `rocm serve` exits non-zero, when the
engine reports failure or its process exits, when a download percent stops
advancing for 3 minutes, when the port stays closed with no engine log for

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code arms this exit on EndpointReadiness::Unreachable (the endpoint does not list the model), not on a closed port: an open port answering 503 with no log also counts, and the message says "never answered". Please reword here and in docs/testing.md and docs/manual-testing.md ("does not list the model" / "has not listed the model").

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e4933a9: README.md, docs/testing.md and docs/manual-testing.md now say "the endpoint has not listed the model" instead of "the port stays closed", and so do the doc comments on SILENT_UNREACHABLE and SILENT_LOAD.

# lists the model. The named command is run, not just read: a hint that
# points at a command that does not work would still pass a string check.
@id:serve-interactive-startup-progress @requires-gpu
Scenario: serve-55 - Interactive serve shows how to follow startup and exits when the model is served

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the happy path is a scenario and it is @requires-gpu, so nothing on a lane with runners exercises the new exit code/message. The repo already has a mock-lane seam (e2e-oom-fault-injection, stand-in processes in comfyui_steps.rs) for engines that cannot run in CI. A scenario for "engine exits during startup -> non-zero exit, names rocm logs --service/rocm services stop, record settled stopped" would cover the user-visible failure. If not added, name the gap and the lane in the PR text.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in e4933a9: serve-60 (serve-startup-exit-names-log, @requires-no-gpu @requires-os:linux @requires-oom-fault-injection, so it runs on the mock lane). With ROCM_E2E_SIMULATE_STARTUP_EXIT=1 an e2e-oom-fault-injection build swaps only the spawned child for a stand-in that logs a line and exits during startup; the record, log, wait and settling are production code. It checks the non-zero exit and "the server process exited" message, runs the rocm logs --service <id> the failure names and finds the engine's last line, checks the record is stopped, and that no daemon was left. It passes locally, and fails at the stopped step when settle_exited_service writes failed.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head eb3ce2a0

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (base 8f086ad9...head eb3ce2a0).
No ticket named. (One commit message references #1016 as the prior-art docs table layout it builds on, not an issue this PR closes.)

Blocking

None

Non-blocking

None

Decisions for the author

  • redact_log_line's list-value masking covers only the first element — answered-by-intent
    redact_log_line in apps/rocm/src/serve_progress.rs:745 masks a secret-named value, but when that value is a bracketed/comma list (e.g. 'api_key': ['a', 'b']), only the single word immediately after the key is passed to mask_value (mask_next is a one-shot flag, serve_progress.rs:751-758); any further list items are emitted verbatim. I traced this by hand against the current head and it reproduces exactly as described. The PR's own description states this plainly: "A list value has only its first item masked." — matching the in-code doc comment at serve_progress.rs:743-744 word for word, so this is disclosed, deliberate scope rather than an oversight. Absent that disclosure this would have been a Blocking finding (confidence 85): it's security-sensitive redaction code (the whole point is keeping secrets out of the terminal/status line), and the 1,164-line test module for this file (redact_log_line_masks_keys_tokens_and_secret_flags, serve_progress.rs:~1907-1968) has no case with a bracketed list value, so the documented gap is also completely unpinned by any test. Confirm this is still the intended scope, and consider at least a test that pins today's "first item only" behavior so a future change can't silently narrow it further (e.g. to "no items masked").
  • Fixed, global stall/deadline constants rather than per-engine tuning — tradeoff
    DOWNLOAD_STALL (3 min), SILENT_UNREACHABLE (2 min), SILENT_LOAD (5 min) and SERVE_STARTUP_DEADLINE (30 min, in rocm-core) apply uniformly to every engine (vLLM, Lemonade, AIM), even though their download/load cadence can differ substantially. The PR's own "Scope" section says this is deliberate: "The stall limits are fixed constants, not per-engine settings." Both sides are real: fixed constants are simple and auditable in one place; per-engine tuning would better fit an engine whose load phase legitimately goes quiet for longer than 5 minutes on constrained hardware, at the cost of more configuration surface. Worth revisiting if a real engine's startup profile turns out to routinely trip a stall limit that isn't actually a hang.
  • No audit event records "the interactive wait gave up while the server kept running" — tradeoff
    In apps/rocm/src/service_launch/start.rs, an exited/failed launch (settle_ended_launch) always writes an error-level managed_service_launch audit event, but a stall/deadline stop that leaves the process alive only ever has the earlier info-level "launched … readiness=running" event on record — there's no audit trail entry marking that the CLI's wait itself gave up. This looks intentional (the launch did succeed; only the terminal wait timed out, and the module's own state-matrix doc treats "still running, stalled" as distinct from "failed"), but it does mean an operator scanning the audit log for failures would see a clean "info: launched" line for what the interactive user experienced as a non-zero exit. Confirm that's the intended audit semantics.

Positive signals

  • engine_runtime_select.rs's readiness probe is explicitly shared between the HTTP-polling wait and serve_progress::wait_for_serve_startup, with a doc comment calling out why: "so the interactive status line and this wait cannot disagree about 'ready'" — avoids a classic two-sources-of-truth readiness bug.
  • service_launch/attach.rs's hint-helper refactor is an actual bug fix, not just deduplication: the old inline hints printed rocm logs {id} (parsed by rocm logs as a free-text search, not --service <id>); the fix is backed by a test that round-trips the printed line through the real clap parser and asserts it parses as Logs { service: Some(id), .. }.
  • The secret-masking helpers (mask_value, mask_compact_value) are genuinely char-boundary-safe — they locate the mask boundary via char_indices()/find on chars rather than raw byte offsets, verified directly against the source and exercised by dedicated multi-byte-character test cases.
  • All three stall/silence timers in serve_progress.rs consistently key off time-since-last-observed-change rather than total elapsed wait time, verified directly in decide_serve_startup and the token/progress_at bookkeeping in wait_for_serve_startup.
  • rocmd's server-recover guard (apps/rocmd/src/watchers/server_recover.rs) checks a pid's recorded start time via identity_state, not just "is some process alive at this pid" — a recycled pid is correctly still recovered — and both the periodic scan and the event-triggered re-check route through the same guarded helper, so neither can diverge from the other.
  • Every timing number and behavioral claim added to README.md/docs/testing.md/docs/manual-testing.md (stall windows, the 30-minute deadline, inference-probe backoff, which hint text is shown when) was checked directly against the corresponding constant or function in source and matched exactly.

Deployment notes

  • At review time, only the docs/readthedocs build has completed against the exact head commit (eb3ce2a0): CI (build/test/clippy), the PR security scan, CodeQL, and the GPU-gated E2E self-hosted lane last completed successfully against the previous commit (2ffa4f9e), not this one — confirm all of those are green on eb3ce2a0 itself before merging, rather than assuming the prior commit's results still apply.
  • SERVE_STARTUP_DEADLINE is now a single constant in rocm-core read by both the rocm CLI's interactive wait and the separate rocmd daemon's recovery guard — correct today since both ship from the same build, but worth knowing that deploying rocm and rocmd from different versions could make them disagree about how long a fresh startup is protected from forced recovery.

What this covered

Every file changed in this PR, read completely at the head commit, by five independent workers grouped by coupling (the new apps/rocm/src/serve_progress.rs startup-wait module; the serve_cmd.rs/serve_summary.rs/cli.rs/main.rs/engine_runtime_select.rs command-wiring group; the service_hints.rs/service_launch/{attach,readiness,start}.rs service-launch group; the rocmd recovery-watcher plus rocm-core's shared readiness/constants; and the README/docs/xtask/Cucumber-scenario group), plus my own direct re-verification of the one candidate finding and of CI/check status against the live commit. Base 8f086ad9c0fe85e7bab2bd8272dc88638938e4fe…head eb3ce2a0d8adc0517862a67f04114df535b94bf9.

Did not run as separate dedicated passes: a repo-wide history/git blame sweep and a "prior review comments on earlier commits of this PR" pass — this PR has earlier review rounds on previous commits that were deliberately not read here, per the blind-review method, so whether any earlier feedback was addressed is outside what this report can say. Agent-instruction adherence (this repo's AGENTS.md) and in-file code-comment compliance were folded into each file-group worker's brief rather than run as fully separate passes, which is a reasonable scaling choice for a PR this size but is disclosed rather than assumed equivalent to a dedicated pass.

One process note for transparency: a single combined gh pr view call made early on (intended only to check CI status) also returned the PR's description and commit messages before the code-side findings were settled. No part of that content was passed to the five code-review workers, who worked blind from the diff and file contents only; the one finding that survived to this report was independently verified by direct re-reading of the current source, and the PR's stated intent was consulted only afterward, as the method requires, to check it against the already-settled findings.

Not reconciled against any prior review discussion on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…tup exit on the mock lane

A stall or deadline stop now records the status that matches how far the
endpoint got (`starting` when it never listed the model), instead of
`running`, which elsewhere means the model is listed.

When the engine reports failed but its process is still up, the message now
says a running `rocm daemon` with `server-recover` may restart it.

Scenario serve-60 drives an engine that exits during interactive startup on
the no-GPU mock lane: an `e2e-oom-fault-injection` build swaps the managed
child for a stand-in when ROCM_E2E_SIMULATE_STARTUP_EXIT=1. It checks the
non-zero exit and message, runs the `rocm logs --service` command the
failure names, and checks the record is settled `stopped`.

The docs say the silent limits arm while the endpoint has not listed the
model, which is what the code checks, rather than "the port stays closed".

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…artup deadline

A stall or deadline stop leaves the server running, and rocmd spares a
live startup only until SERVE_STARTUP_DEADLINE. The message now says a
running daemon with server-recover may restart it after that.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The interactive wait was chosen by stdout alone, while its status line and
failures go to stderr, so `rocm serve m 2>log` in a terminal could wait up
to 30 minutes with nothing drawn. A redirected stderr now keeps the piped
45-second status budget. Also applies rustfmt to the previous commit.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio thanks for the review. The inline threads each have a reply; these are the points from the review body that had no anchor:

  • Conflicts and CI. main is merged in (0493acf, no force-push, so the review context stays). CI is running on the new head f04ebd1.
  • Failure scenario. serve-60 is on the mock lane and covers an engine that exits during interactive startup (details in that thread). The stall, deadline and engine-failed-while-alive branches still have no scenario. The PR body names that gap and what covers them instead.
  • stdout TTY vs spinner on stderr. Fixed in f04ebd1. The startup is followed only when both streams are terminals, so rocm serve m 2>log keeps the 45-second piped budget. This is pinned by a_background_serve_follows_the_startup_only_with_both_streams_on_a_terminal.
  • Fixed 3/2/5/30-minute limits. Deliberate: one auditable set of limits until real engine cadence says otherwise. The PR body says so under Scope.
  • Piped exits 0 after 45 s while a terminal exits non-zero. Intended. The piped path is for scripts and CI that want a readiness status. The terminal path tells a person that the startup failed. The PR body says so too.
  • Size and arity. serve_cmd.rs is still under the 1,500-line limit (check-module-size passes). I left wait_for_serve_startup's 8 arguments and the stop_decision call sites as they are. Folding them into a context struct is a mechanical refactor that I would rather do separately than grow this PR.

@rominf
rominf requested a review from juhovainio November 16, 2026 13:06
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole diff. I found nothing that blocks on its own, but there are a few things I'd like addressed before this merges.

Please fix

  • A download line that ends at 100% followed by a quiet engine is still classified as Downloading, so a healthy server that logs nothing for 3 minutes after the download finishes gets reported as a download stall (and now exits non-zero in a terminal). See the inline comment on line_is_download.
  • The 2-minute silent-unreachable limit can fire on a slow start that has written no log line yet (for example a container-engine pre-spawn step). The PR body says the stall limits are unverified against real engine cadence; please confirm on the GPU lane or narrow the condition.
  • classify builds the "the engine reported failed" / "the server process exited" labels and quoted_log recognises them by string equality, and the e2e step greps the same words. AGENTS.md section 3 asks for one shared constant or deriving the text from the decision. Carrying an explicit optional quoted line would remove the cross-function matching (and a real log line equal to a label would no longer be silently dropped).
  • The message for a live engine-failed record says a running daemon with server-recover "may restart it first". I could not find a test in the diff that shows rocmd doing that; only the running/starting guard is covered. Please point at the test that proves it, or soften the sentence.

Scope: this bundles the startup wait, redaction, probe backoff, the rocmd recovery guard, the rocm logs --service hint fix and a new audit event on --wait. The rocmd guard and the hint/audit fixes could land separately and would be easier to review apart.

For the author to confirm

  • The rocmd guard delays recovery of a live service that lost its listener for up to 30 minutes.
  • Piped serve exits 0 at 45s while a terminal now exits non-zero for the same slow server.
  • Stall constants are fixed rather than per-engine.

CI: prek (lint / hygiene) is failing on the current head; I couldn't see which hook, so please look. Windows jobs were still pending when I checked.

This is a read of the diff only; I did not build or run it.

line.chars().filter(char::is_ascii_alphabetic).count() >= 4 && line.len() >= 6
}

fn line_is_download(line: &str) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A download line that ends at 100% and is followed by a quiet engine keeps the phase at Downloading, so a healthy server that logs nothing for 3 minutes after the download is reported as a download stall. Consider treating 100% (or a completed download) as Loading.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7fa1193: a download line whose last percent is 100 is now a finished download (line_is_active_download), so a quiet engine after it is classified Loading and judged by the 5-minute load limit, not the 3-minute download stall. A line below 100 is still a download. a_finished_download_line_is_judged_as_loading covers both and fails when the 100% check is removed. README, docs/testing.md and docs/manual-testing.md say so.

readiness,
);
}
if view.phase == ServeStartupPhase::Unreachable

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 2-minute silent limit also fires on a start that has written no meaningful log yet (e.g. a slow pre-spawn step on a container engine). The PR body says the stall limits are unverified against real engine cadence; please confirm on the GPU lanes or note the risk.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Narrowed in 7fa1193 rather than confirmed. I cannot confirm it on a GPU lane: no lane drives a stalled interactive serve, and I have no GPU here. A start with no meaningful log line now gets as long as a quiet load: SILENT_UNREACHABLE is defined as SILENT_LOAD (5 minutes, was 2). a_start_with_no_log_gets_as_long_as_a_quiet_load pins that, and fails with the old 2-minute value. The comment on the constant (reworded in 4c6cf19) no longer gives an image pull as the example, since docker pull writes its own status lines to the log. A stop on this limit still leaves the server running. The PR body keeps "not verified against real engine cadence" as a known gap.

Comment thread apps/rocm/src/serve_progress.rs Outdated
}
}

fn quoted_log(model: &str, engine: &str, view: &ServeStartupView) -> Option<String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quoted_log recognises the two fixed failure labels by comparing strings produced in classify. AGENTS.md section 3 asks for a shared constant or deriving this from the decision; carry an explicit optional quoted line instead of matching label text.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7fa1193: quoted_log is gone. ServeStartupView now carries log_line: Option<String>, set in classify where the label is chosen (the engine line for Failed/Listing/Downloading/Loading, None for the fallback labels), and stop_decision copies it, so it no longer needs model/engine. An engine line that reads exactly like a fallback label is now quoted. a_stop_quotes_the_engine_line_not_a_fallback_label covers both cases and fails if the line is filtered by text again. The e2e step that greps "the server process exited" asserts the user-visible headline (user_message builds it from the stop reason), not the label match.

…ne by decision

A download line at 100% followed by a quiet engine was held to the
3-minute download stall; it is now judged by the 5-minute load limit.

A stop quoted the log by comparing the label with this module's own
fallback strings. The view now carries the engine line it shows, set
where the label is chosen, so an engine line that reads like a fallback
label is still quoted, and stop_decision no longer needs model and engine.

A start with no log line yet now gets as long as a quiet load (5 minutes,
was 2), so a slow pre-spawn step is not reported as silent sooner.

A rocmd test now shows that a `failed` record whose process is still up is
recovered, which the engine-failed message warns about.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Conflicts: README.md (serve paragraph keeps both the startup wait and the
smoke-test-failure text), serve_cmd.rs and rocm-core lib.rs (both sides'
additions), docs/architecture.md (both fault-injection seams), and
model_serving.feature, where main took serve-59/60, so this branch's
scenarios become serve-61/62. A DeploymentSummary test fixture gains main's
new smoke_failure field.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The SILENT_UNREACHABLE comment no longer gives a container image pull as its
example, which writes its own lines to the log, and names the test that
pins the limit; the start.rs note about rocmd's guard names the function.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio thanks. Each of your three inline comments has a reply. The points from the review body:

  • "May restart it first" for a live engine-failed record. The claim is true. A failed record is a recovery candidate whether or not its process is alive (manifest_service_recovery_reason, with no liveness guard for failed). A test now shows it: a_failed_record_with_a_live_process_is_still_recovered in apps/rocmd/src/watchers/server_recover.rs (7fa1193). The scan and the event re-check both restart a failed record whose process is this test's own live pid. The serve-side test that pins the sentence now names that test.
  • Scope. I kept it as one PR. The rocmd guard is what stops the helper from restarting a startup the new wait is reporting on. The logs hint is the command every new failure message names. The --wait audit event comes from settle_ended_launch, the same settling code the interactive failure uses. Shipping the wait without the guard would let the helper kill a download that the CLI says is in progress. I've added this to the PR body.
  • For you to confirm:
    • The recovery delay (up to 30 minutes for a live service that lost its listener) is intended. It is stated as a trade-off in the PR body, with the reason: Lemonade and AIM write running before their port opens.
    • Piped exits 0 at 45 s while a terminal exits non-zero: intended, and stated under Scope.
    • Fixed stall constants: intended, also under Scope.
  • CI. prek was the end-of-file hook: model_serving.feature ended with a blank line. That was fixed in 3d41093, and every check went green on that head. main is merged in again as d6d2e90. main had taken serve-59 and serve-60, so this PR's scenarios are now serve-61 (GPU, happy path) and serve-62 (mock lane, engine exits during startup). serve-62, serve-60 and serve-41, which share the fault-injection feature, pass locally on the merge.

@rominf
rominf requested a review from juhovainio November 17, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants