From c44e0b0bc095c15954a56ba7dd8853299fd6e0ed Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Tue, 10 Nov 2026 13:35:49 +0200 Subject: [PATCH 01/14] fix(serve): keep startup progress until the server is ready or stalled Interactive rocm serve stopped after 45 seconds with a static not-ready summary while a download or load was still moving. The status line now follows the engine log until the endpoint can serve or that progress stalls, and a failure names the logs and stop commands. Piped output keeps the 45-second status budget, and `--wait` keeps its own wait, with no stall limit or deadline. Like `--wait`, the interactive wait first prints the service id, how to follow its log and a public bind's key, since it can fail without a summary. Liveness is read from the manifest with the zombie-aware pid check, so a dead supervisor ends the wait. A startup stop is a typed ServeStartupFailed: only it starts the background helper, and a helper failure is a warning beside it. Engine log lines are redacted (endpoint key, --api-key/token/secret values, compact JSON keys, Authorization headers, hf_ tokens) before they reach the status line or failure text. rocmd's server-recover watcher no longer restarts a live `running` or `starting` record younger than SERVE_STARTUP_DEADLINE, so it does not kill a download the CLI is still reporting. Signed-off-by: Roman Inflianskas --- README.md | 33 +- apps/rocm/src/cli.rs | 13 +- apps/rocm/src/engine_runtime_select.rs | 83 +- apps/rocm/src/main.rs | 1 + apps/rocm/src/serve_cmd.rs | 105 +- apps/rocm/src/serve_progress.rs | 1720 +++++++++++++++++ apps/rocm/src/serve_summary.rs | 18 +- apps/rocm/src/service_hints.rs | 9 + apps/rocm/src/service_launch/readiness.rs | 48 + apps/rocm/src/service_launch/start.rs | 93 +- apps/rocmd/src/watchers/server_recover.rs | 171 +- crates/rocm-core/src/endpoint_readiness.rs | 6 + crates/rocm-core/src/lib.rs | 2 +- docs/architecture.md | 3 +- docs/manual-testing.md | 24 + docs/testing.md | 28 + .../features/model_serving.feature | 8 + xtask/src/ascii_output_contract.rs | 9 +- 18 files changed, 2295 insertions(+), 79 deletions(-) create mode 100644 apps/rocm/src/serve_progress.rs diff --git a/README.md b/README.md index 6cd4841af2..55e1a3e232 100644 --- a/README.md +++ b/README.md @@ -1149,12 +1149,29 @@ Likewise, an explicit `--port` that differs from the port the running service is already on is refused rather than silently ignored; omit `--port` to reuse the running service, or stop it first to serve on the new port. -By default the server runs in the background under ROCm CLI's supervision and -prints a deployment summary: a progress indicator while it starts, then a table -with the status, the full inference endpoint, the API-qualified model name, and a -quick smoke test (time to first token and approximate tokens per second). Control -returns to your shell with the server still running; manage it later with `rocm -services` (below). +By default the server runs in the background under ROCm CLI's supervision. +In a terminal it first prints the service id, how to follow its log +(`rocm logs --service --follow`) and, for a public bind, the +API key. While it starts, the status line follows the engine: a download, a weight +load, or the latest log line, until the server is ready or that wait fails +(the engine exits, a download stops advancing, or the startup deadline +passes). When the server is ready it prints a deployment summary: a table +with the status, the full inference endpoint, the API-qualified model name, +and a quick smoke test (time to first token and approximate tokens per +second). A model that is still loading is not reported as a finished "not +ready" summary. The wait fails, and `rocm serve` exits non-zero, when the +engine reports failure or its process exits, when a download percent stops +advancing for 3 minutes, when the port stays closed with no engine log for +2 minutes, when a load log line stops changing for 5 minutes with the port +still closed, or after 30 minutes in all. A stall leaves the server running: +the message names `rocm logs --service ` and +`rocm services stop --yes`, and the background helper does not +restart a live server for being slow until those 30 minutes pass. Engine log +lines shown on the status line have credentials (the endpoint key, `--api-key` +and token values, `hf_` tokens) masked. Piped output (not a terminal) keeps +the 45-second wait and reports the server as it stood then. Control returns +to your shell with the server still running; manage it later with +`rocm services` (below). `--verbose` (or `--foreground`) instead attaches to the server in the current terminal and streams every engine log line. Use it to debug a startup problem. @@ -1172,8 +1189,8 @@ can: `--managed` is the explicit form of the default background behavior. `--wait` keeps `rocm serve` waiting until the server answers requests or its engine gives up (vLLM and AIM after 5 minutes by default; raise it with -`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), instead -of returning `starting` after 45 seconds; if the server exits first (crashes, is killed, or is stopped with +`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), with +no stall limit or deadline, instead of the default wait above; if the server exits first (crashes, is killed, or is stopped with `rocm services stop`), it fails with an error that points at the server's logs instead of waiting forever. Ctrl-C ends the wait and leaves the server starting in the background; a generated API key (public bind) is printed when the wait diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 02cda2dc2e..8b5945fb87 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -430,8 +430,12 @@ rocm model --verbose" /// Picks an engine and ROCm runtime automatically unless overridden. An /// Anthropic `/v1/messages` endpoint is published beside the OpenAI /// one (see `rocm services list`). By default the - /// server runs as a managed background service and prints a deployment summary - /// (status, endpoint, model, and smoke-test metrics); use --verbose to stream engine + /// server runs as a managed background service. While it starts, the status line + /// follows what the engine is doing (download, load, or the latest log line) until + /// the server is ready or that wait fails. A ready server then prints a deployment + /// summary (status, endpoint, model, and smoke-test metrics). A wait that fails + /// while the server is still running names `rocm logs` and `rocm services stop` + /// instead of a finished "not ready" summary. Use --verbose to stream engine /// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the /// server running, or Ctrl-C to stop it. If that stop answers that it could not confirm /// the server exited, it says so on stderr and exits with an error; if the stop itself @@ -520,8 +524,9 @@ rocm serve qwen --verbose --device gpu_required")] /// Keep waiting until the server answers requests or its engine gives up /// (vLLM and AIM after 5 minutes by default, raised with /// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), - /// instead of returning `starting` after 45 seconds. - /// Also waits on a matching server that is already starting. Fails with a + /// with no stall limit or deadline. Without it, a terminal gives up when + /// startup progress stalls or after 30 minutes, and piped output returns + /// `starting` after 45 seconds. Also waits on a matching server that is already starting. Fails with a /// pointer to the logs if the server exits first. Cannot be /// combined with --foreground or --verbose, which already stay attached. #[arg(long, conflicts_with_all = ["foreground", "verbose"])] diff --git a/apps/rocm/src/engine_runtime_select.rs b/apps/rocm/src/engine_runtime_select.rs index aa65af269d..879e9a2b8d 100644 --- a/apps/rocm/src/engine_runtime_select.rs +++ b/apps/rocm/src/engine_runtime_select.rs @@ -72,7 +72,9 @@ //! `service_launch.rs` and `service_control.rs`'s `restart_internal_managed_service_with`. //! `wait_for_service_http_ready` is `pub(crate)`, read from `service_control.rs`'s //! `restart_internal_managed_service_with`; `wait_for_service_http_ready_with_progress` -//! is `pub(crate)`, read from `service_launch.rs`. `http_post_local_service_json` +//! is `pub(crate)`, read from `service_launch.rs`. `probe_service_http_readiness` +//! is `pub(crate)`, read from `serve_progress.rs`'s `wait_for_serve_startup` and +//! from `wait_for_service_http_ready_with_progress`. `http_post_local_service_json` //! is `pub(crate)`, read from `service_control.rs`'s `unload_lemonade_service_model`. //! `service_model_names_match` is `pub(crate)`, read from `chat/turn.rs`. //! `service_http_readiness_paths`, `http_get_local_service`, @@ -348,43 +350,14 @@ pub(crate) fn wait_for_service_http_ready_with_progress( on_tick: &mut dyn FnMut(Duration), ) -> EndpointReadiness { let start = std::time::Instant::now(); - let endpoint = format_http_base_url(host, port); let mut best = EndpointReadiness::Unreachable; while start.elapsed() < timeout { // Re-evaluated each pass: a container engine reports the name its server // answers to only after it starts, so a name fixed up front can be wrong. - let canonical_model_id = model_name(); - let canonical_model_id = canonical_model_id.as_str(); - let listed = service_http_readiness_paths(engine).iter().any(|path| { - http_get_local_service( - host, - port, - path, - endpoint_api_key, - Duration::from_millis(750), - ) - .is_ok_and(|(status, body)| { - service_http_readiness_response_ready( - engine, - path, - status, - &body, - canonical_model_id, - ) - }) - }); - if listed { - best = EndpointReadiness::Listing; - if rocm_core::openai_chat_completion_probe( - &endpoint, - canonical_model_id, - endpoint_api_key, - rocm_core::INFERENCE_PROBE_TIMEOUT, - ) - .unwrap_or(false) - { - return EndpointReadiness::Serving; - } + match probe_service_http_readiness(engine, host, port, &model_name(), endpoint_api_key) { + EndpointReadiness::Serving => return EndpointReadiness::Serving, + EndpointReadiness::Listing => best = EndpointReadiness::Listing, + EndpointReadiness::Unreachable => {} } on_tick(start.elapsed()); thread::sleep(Duration::from_millis(250)); @@ -392,6 +365,48 @@ pub(crate) fn wait_for_service_http_ready_with_progress( best } +/// One readiness probe: [`EndpointReadiness::Serving`] only after an inference +/// request comes back, [`EndpointReadiness::Listing`] when the model is +/// advertised but that request does not, otherwise unreachable. +/// +/// Shared with [`crate::serve_progress::wait_for_serve_startup`] so the +/// interactive status line and this wait cannot disagree about "ready". +pub(crate) fn probe_service_http_readiness( + engine: &str, + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> EndpointReadiness { + let endpoint = format_http_base_url(host, port); + let listed = service_http_readiness_paths(engine).iter().any(|path| { + http_get_local_service( + host, + port, + path, + endpoint_api_key, + Duration::from_millis(750), + ) + .is_ok_and(|(status, body)| { + service_http_readiness_response_ready(engine, path, status, &body, canonical_model_id) + }) + }); + if !listed { + return EndpointReadiness::Unreachable; + } + if rocm_core::openai_chat_completion_probe( + &endpoint, + canonical_model_id, + endpoint_api_key, + rocm_core::INFERENCE_PROBE_TIMEOUT, + ) + .unwrap_or(false) + { + return EndpointReadiness::Serving; + } + EndpointReadiness::Listing +} + fn service_http_readiness_paths(engine: &str) -> &'static [&'static str] { match engine { "lemonade" => &["/v1/health", "/v1/models"], diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index d713b0c105..1018516f8d 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -61,6 +61,7 @@ mod runtime_uninstall; mod runtimes_cmd; mod serve_cmd; mod serve_port; +mod serve_progress; mod serve_recipe; mod serve_summary; mod service_control; diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 711e169b6c..fb2ea6724e 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -67,6 +67,7 @@ use crate::gpu_policy::{ }; use crate::managed_service_status::{find_live_managed_service, live_managed_services_for_model}; use crate::serve_port; +use crate::serve_progress::{self, ServeStartupView}; use crate::serve_recipe::{ ServeGenerationDefaults, engine_env_ignored_note, engine_recipe_enables_tool_choice, engine_recipe_with_extra_env, engine_recipe_with_generation_defaults, @@ -1132,10 +1133,21 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { } if background { - let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(format!( - "Starting {model} on {selected_engine}…" - ))); + let starting = serve_progress::starting_label(&model, &selected_engine); + let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(starting.clone())); spinner.borrow_mut().tick(); + let mut shown = starting; + // Interactive summary follows download / load / health until ready or + // a bounded failure. Piped output keeps the historical 45-second + // status budget so a serve that never answers does not block CI. + // `--wait` keeps waiting past either, until the server answers or dies. + let policy = if summary_mode { + serve_progress::ServeStartupPolicy::interactive() + } else { + serve_progress::ServeStartupPolicy::status_budget( + serve_progress::PIPED_SERVE_STATUS_BUDGET, + ) + }; // The key is shown when the wait begins, since Ctrl-C or a failed wait // would otherwise leave a running authenticated server whose key was never // printed. It is then not repeated in the summary below. @@ -1158,6 +1170,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { require_api_key, gpu_refusal_waived_for_reuse, wait, + &policy, // Named with the id the launch resolved to: an already-starting // server keeps its own, not the one generated above. The spinner // has no id on it, and a non-TTY stderr gets no spinner at all. @@ -1181,13 +1194,26 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { print!("{config}"); } }, - &mut |_elapsed| spinner.borrow_mut().tick(), + &mut |view: &ServeStartupView| { + let mut spinner = spinner.borrow_mut(); + if view.label == shown { + spinner.tick(); + } else { + shown.clone_from(&view.label); + spinner.set_label(view.label.clone()); + } + }, ); // `Spinner` has no `Drop`: clear its row so `Error:` is not glued to it. let report = match launched { Ok(report) => report, Err(error) => { spinner.borrow_mut().clear(); + if let Some(warning) = supervise_after_launch_error(&error, || { + ensure_background_helper_running_quiet(summary_mode) + }) { + eprintln!("{warning}"); + } return Err(error); } }; @@ -1404,6 +1430,31 @@ fn collect_serve_notes( notes } +/// Start supervision after `start_managed_service` failed, when that failure +/// left a launched server behind. Returns the warning to print when the +/// helper itself would not start. +/// +/// Only a startup stop ([`serve_progress::ServeStartupFailed`]) is known to +/// leave a process that may still be running. Other errors either come before +/// the child exists (the claiming record is written first, as `starting`, and +/// `rocmd` treats it as any stale start), or from a child that already exited +/// or an I/O error writing its record, which did not start the helper before +/// this wait existed either. The startup failure stays the error the user +/// reads; a helper failure is a warning beside it. +fn supervise_after_launch_error( + error: &anyhow::Error, + start_helper: impl FnOnce() -> Result<()>, +) -> Option { + error + .downcast_ref::() + .and_then(|_| start_helper().err()) + .map(|helper| { + format!( + "warning: the background helper did not start, so nothing restarts this server if it fails: {helper:#}" + ) + }) +} + #[cfg(test)] mod tests { use super::*; @@ -2174,4 +2225,50 @@ mod tests { "non-vLLM engines must not be told to pass a vLLM-only flag: {lemonade:?}" ); } + + #[test] + fn a_startup_stop_starts_supervision_and_keeps_its_own_error() { + use std::cell::Cell; + let stop = || -> anyhow::Error { + serve_progress::ServeStartupFailed( + "qwen on vllm stopped making download progress".to_owned(), + ) + .into() + }; + // Row: startup stop, helper starts. Helper called, nothing to warn. + let called = Cell::new(false); + let warning = super::supervise_after_launch_error(&stop(), || { + called.set(true); + Ok(()) + }); + assert!(called.get()); + assert_eq!(warning, None); + + // Row: startup stop, helper fails. The warning names the helper + // error; the startup failure is still what the caller returns. + let error = stop(); + let warning = super::supervise_after_launch_error(&error, || anyhow::bail!("no daemon")) + .expect("a helper that failed is reported"); + assert!( + warning.starts_with("warning: the background helper did not start"), + "{warning}" + ); + assert!(warning.contains("no daemon"), "{warning}"); + assert_eq!( + error.to_string(), + "qwen on vllm stopped making download progress" + ); + + // Row: any other launch error. No helper. + let called = Cell::new(false); + let warning = super::supervise_after_launch_error(&anyhow::anyhow!("spawn failed"), || { + called.set(true); + Ok(()) + }); + assert!( + !called.get(), + "a launch that never ran a server starts no helper" + ); + assert_eq!(warning, None); + } } diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs new file mode 100644 index 0000000000..deeac7f569 --- /dev/null +++ b/apps/rocm/src/serve_progress.rs @@ -0,0 +1,1720 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Interactive `rocm serve` startup progress. +//! +//! Summary mode used to tick one static "Starting …" label and, after 45 +//! seconds, print a finished "not ready yet" summary while the server kept +//! loading. This module decides the status line from the endpoint and the +//! engine log, and when that wait is allowed to stop. +//! +//! State matrix for the wait. Every pending row has an exit when nothing +//! new arrives; a download that stops advancing does not spin until the +//! overall deadline. +//! +//! | State | What we observe | Status line | Exit if data never arrives | +//! | --- | --- | --- | --- | +//! | unreachable | port down, no meaningful log, process alive | `Starting {model} on {engine}…` | silent limit, or the deadline | +//! | downloading | latest log line is a download, the endpoint is not listing the model, and progress is still moving | that log line | stall limit once the line (or its percent) freezes, or the deadline | +//! | loading | latest log line is anything else meaningful, port still down | that log line | silent-load limit once the line freezes, or the deadline | +//! | listing | HTTP lists the model, inference has not answered | the log line, or `Waiting for {model} to serve…` | the overall deadline (the server is answering) | +//! | serving | inference answered | wait returns ready | immediate | +//! | failed | engine state `failed`, or every recorded pid is dead | last log line, or a fixed failure line | immediate | +//! +//! A log line that still says "download" does not pull a listed model back +//! into the download stall. The server has answered; the exit stays the +//! deadline. +//! +//! A piped serve keeps the historical status budget and returns the best +//! readiness instead of this failure. That path has no status line; holding +//! it for a full model load would block CI and the assistant on a server +//! that never answers. + +use std::fmt::Write as _; +use std::path::Path; +use std::thread; +use std::time::{Duration, Instant}; + +use rocm_core::EndpointReadiness; + +use crate::engine_runtime_select::probe_service_http_readiness; +use crate::service_hints::{service_logs_hint, service_stop_hint}; + +/// How long a download percent (or line) may stay unchanged before it is hung. +pub(crate) const DOWNLOAD_STALL: Duration = Duration::from_mins(3); + +/// Piped / captured `rocm serve` returns the best readiness after this long. +/// Interactive summary mode follows the stage instead. +pub(crate) const PIPED_SERVE_STATUS_BUDGET: Duration = Duration::from_secs(45); + +/// Port down and no meaningful log line. +pub(crate) const SILENT_UNREACHABLE: Duration = Duration::from_mins(2); + +/// A load line that stops changing while the port is still down. +pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5); + +const POLL: Duration = Duration::from_millis(250); +const LOG_TAIL_LINES: usize = 40; +const LOG_TAIL_BYTES: u64 = 64 * 1024; +const MAX_LABEL_CHARS: usize = 160; + +/// How the interactive wait treats time, and whether giving up is an error. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupPolicy { + pub bounds: ServeStartupBounds, + pub poll: Duration, + /// Stop with an error at the deadline and the stall limits. + /// + /// `false` for piped output: the deadline returns the best readiness, the + /// way the 45-second wait did. + pub stop_on_give_up: bool, + /// Read engine-failed and process-liveness. + /// + /// Off for the piped path so a serve that exits in CI still returns a + /// readiness status. + pub watch_engine: bool, +} + +/// The clocks [`decide_serve_startup`] compares against. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupBounds { + pub deadline: Duration, + pub download_stall: Duration, + pub silent_unreachable: Duration, + pub silent_load: Duration, +} + +impl ServeStartupPolicy { + /// Terminal summary: follow the stage until ready or a bounded failure. + pub(crate) const fn interactive() -> Self { + Self { + bounds: ServeStartupBounds { + deadline: rocm_core::SERVE_STARTUP_DEADLINE, + download_stall: DOWNLOAD_STALL, + silent_unreachable: SILENT_UNREACHABLE, + silent_load: SILENT_LOAD, + }, + poll: POLL, + stop_on_give_up: true, + watch_engine: true, + } + } + + /// Piped / captured output. `timeout` is the historical readiness budget. + pub(crate) const fn status_budget(timeout: Duration) -> Self { + Self { + bounds: ServeStartupBounds { + deadline: timeout, + download_stall: Duration::MAX, + silent_unreachable: Duration::MAX, + silent_load: Duration::MAX, + }, + poll: POLL, + stop_on_give_up: false, + watch_engine: false, + } + } +} + +/// One sample of engine and process state, separate from the HTTP probe. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupSignals { + pub engine_failed: bool, + pub process_alive: bool, + /// Latest meaningful log line, already one sanitized line. + pub log_line: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupTiming { + pub elapsed: Duration, + pub since_progress: Duration, +} + +/// Lifecycle row the status line is on. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ServeStartupPhase { + Unreachable, + Downloading, + Loading, + Listing, + Serving, + Failed, +} + +/// What the spinner should show for this sample. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupView { + pub phase: ServeStartupPhase, + pub label: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ServeStartupStopReason { + EngineFailed, + ProcessExited, + DownloadStalled, + Silent, + SilentLoad, + Deadline, +} + +/// The wait stopped before the server could serve. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupStop { + pub phase: ServeStartupPhase, + pub reason: ServeStartupStopReason, + /// Spinner label for this sample. + /// + /// May contain the spinner ellipsis. The failure text quotes + /// [`Self::log_line`] instead, which is ASCII. + pub label: String, + pub log_line: Option, + pub process_alive: bool, + pub readiness: EndpointReadiness, +} + +/// The error `start_managed_service` returns when the wait stopped. +/// +/// The process is recorded (and may still be running), unlike a spawn +/// failure, so the caller starts supervision for this error and no other. +/// `Display` is [`ServeStartupStop::user_message`]. +#[derive(Debug)] +pub(crate) struct ServeStartupFailed(pub String); + +impl std::fmt::Display for ServeStartupFailed { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.0) + } +} + +impl std::error::Error for ServeStartupFailed {} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ServeStartupDecision { + Continue(ServeStartupView), + Ready, + /// Piped path: the status budget elapsed. Hand back the best readiness. + BudgetElapsed(EndpointReadiness), + Stop(ServeStartupStop), +} + +pub(crate) fn starting_label(model: &str, engine: &str) -> String { + format!("Starting {model} on {engine}…") +} + +pub(crate) fn waiting_label(model: &str) -> String { + format!("Waiting for {model} to serve…") +} + +/// Latest meaningful engine-log line, sanitized to one ASCII line. +/// +/// Carriage-return progress bars keep only their last segment. ANSI and +/// other control bytes are stripped. Blank lines and lines with no words +/// are skipped so a rule of dashes cannot cover a real stage. Credentials +/// are masked ([`redact_log_line`]) before the line is truncated, so a key +/// cut at the length limit cannot show its prefix. +pub(crate) fn latest_meaningful_log_line( + path: &Path, + endpoint_api_key: Option<&str>, +) -> Option { + let lines = rocm_core::tail_lines(path, LOG_TAIL_LINES, LOG_TAIL_BYTES).ok()?; + lines.into_iter().rev().find_map(|line| { + let sanitized = sanitize_log_line(&line, endpoint_api_key); + is_meaningful(&sanitized).then_some(sanitized) + }) +} + +pub(crate) fn sanitize_log_line(raw: &str, endpoint_api_key: Option<&str>) -> String { + let segment = raw + .rsplit(['\r', '\n']) + .find(|part| !part.trim().is_empty()) + .unwrap_or(raw); + let stripped = rocm_core::terminal::strip_terminal_control_sequences(segment); + let ascii: String = stripped.chars().filter(char::is_ascii).collect(); + let collapsed = ascii.split_whitespace().collect::>().join(" "); + truncate_label(&redact_log_line(&collapsed, endpoint_api_key)) +} + +fn truncate_label(line: &str) -> String { + if line.chars().count() <= MAX_LABEL_CHARS { + return line.to_owned(); + } + let mut out: String = line + .chars() + .take(MAX_LABEL_CHARS.saturating_sub(3)) + .collect(); + out.push_str("..."); + out +} + +fn is_meaningful(line: &str) -> bool { + line.chars().filter(char::is_ascii_alphabetic).count() >= 4 && line.len() >= 6 +} + +fn line_is_download(line: &str) -> bool { + let lower = line.to_ascii_lowercase(); + lower.contains("download") || lower.contains("fetching ") || lower.contains("pulling ") +} + +/// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent +/// does not is still stalled. +fn download_fingerprint(line: &str) -> String { + last_percent(line).map_or_else(|| line.to_owned(), |pct| format!("pct:{pct}")) +} + +fn last_percent(line: &str) -> Option<&str> { + let bytes = line.as_bytes(); + let mut search_end = bytes.len(); + while let Some(rel) = line[..search_end].rfind('%') { + let mut start = rel; + while start > 0 && bytes[start - 1].is_ascii_digit() { + start -= 1; + } + if start < rel { + return Some(&line[start..rel]); + } + search_end = rel; + } + None +} + +fn progress_token(readiness: EndpointReadiness, signals: &ServeStartupSignals) -> String { + if signals.engine_failed { + return "engine-failed".to_owned(); + } + if !signals.process_alive { + return "process-exited".to_owned(); + } + if readiness == EndpointReadiness::Listing { + return format!("listing:{}", signals.log_line.as_deref().unwrap_or("")); + } + if let Some(line) = signals + .log_line + .as_deref() + .filter(|line| line_is_download(line)) + { + return format!("download:{}", download_fingerprint(line)); + } + if let Some(line) = &signals.log_line { + return format!("log:{line}"); + } + "unreachable".to_owned() +} + +fn classify( + model: &str, + engine: &str, + readiness: EndpointReadiness, + signals: &ServeStartupSignals, +) -> ServeStartupView { + if signals.engine_failed || !signals.process_alive { + let label = signals.log_line.clone().unwrap_or_else(|| { + if signals.engine_failed { + "the engine reported failed".to_owned() + } else { + "the server process exited".to_owned() + } + }); + return ServeStartupView { + phase: ServeStartupPhase::Failed, + label, + }; + } + if readiness == EndpointReadiness::Serving { + return ServeStartupView { + phase: ServeStartupPhase::Serving, + label: starting_label(model, engine), + }; + } + // Listing wins over a stale download line. vLLM advertises the model + // before inference answers, and the last log line often still says + // "download". That must not arm the download stall. + if readiness == EndpointReadiness::Listing { + return ServeStartupView { + phase: ServeStartupPhase::Listing, + label: signals + .log_line + .clone() + .unwrap_or_else(|| waiting_label(model)), + }; + } + if let Some(line) = signals + .log_line + .as_ref() + .filter(|line| line_is_download(line)) + { + return ServeStartupView { + phase: ServeStartupPhase::Downloading, + label: line.clone(), + }; + } + if let Some(line) = &signals.log_line { + return ServeStartupView { + phase: ServeStartupPhase::Loading, + label: line.clone(), + }; + } + ServeStartupView { + phase: ServeStartupPhase::Unreachable, + label: starting_label(model, engine), + } +} + +fn quoted_log(model: &str, engine: &str, view: &ServeStartupView) -> Option { + if view.label == starting_label(model, engine) || view.label == waiting_label(model) { + return None; + } + match view.label.as_str() { + "the engine reported failed" | "the server process exited" => None, + _ => Some(view.label.clone()), + } +} + +fn stop_decision( + model: &str, + engine: &str, + view: &ServeStartupView, + reason: ServeStartupStopReason, + process_alive: bool, + readiness: EndpointReadiness, +) -> ServeStartupDecision { + ServeStartupDecision::Stop(ServeStartupStop { + phase: view.phase, + reason, + log_line: quoted_log(model, engine, view), + label: view.label.clone(), + process_alive, + readiness, + }) +} + +/// Decide the next status line and whether the wait stops. +/// +/// `readiness` is the best probe so far, never a downgrade: one refused +/// connection after the model has listed does not put the line back on +/// "Starting" or arm the silent-unreachable exit. +pub(crate) fn decide_serve_startup( + model: &str, + engine: &str, + readiness: EndpointReadiness, + signals: &ServeStartupSignals, + timing: ServeStartupTiming, + policy: &ServeStartupPolicy, +) -> ServeStartupDecision { + let signals = effective_signals(signals.clone(), policy); + let view = classify(model, engine, readiness, &signals); + let process_alive = signals.process_alive; + + if signals.engine_failed { + return stop_decision( + model, + engine, + &view, + ServeStartupStopReason::EngineFailed, + process_alive, + readiness, + ); + } + if !process_alive { + return stop_decision( + model, + engine, + &view, + ServeStartupStopReason::ProcessExited, + process_alive, + readiness, + ); + } + if readiness == EndpointReadiness::Serving || view.phase == ServeStartupPhase::Serving { + return ServeStartupDecision::Ready; + } + if timing.elapsed >= policy.bounds.deadline { + return if policy.stop_on_give_up { + stop_decision( + model, + engine, + &view, + ServeStartupStopReason::Deadline, + process_alive, + readiness, + ) + } else { + ServeStartupDecision::BudgetElapsed(readiness) + }; + } + if view.phase == ServeStartupPhase::Downloading + && timing.since_progress >= policy.bounds.download_stall + { + return stop_decision( + model, + engine, + &view, + ServeStartupStopReason::DownloadStalled, + process_alive, + readiness, + ); + } + if view.phase == ServeStartupPhase::Unreachable + && timing.elapsed >= policy.bounds.silent_unreachable + { + return stop_decision( + model, + engine, + &view, + ServeStartupStopReason::Silent, + process_alive, + readiness, + ); + } + if view.phase == ServeStartupPhase::Loading + && timing.since_progress >= policy.bounds.silent_load + { + return stop_decision( + model, + engine, + &view, + ServeStartupStopReason::SilentLoad, + process_alive, + readiness, + ); + } + ServeStartupDecision::Continue(view) +} + +/// The status written to the service record when the wait stops. +/// +/// Engine failure and a dead process are `failed`, which recovery is supposed +/// to restart. Every other reason leaves the process alive and is `running`: +/// [`decide_serve_startup`] returns `ProcessExited` before it checks any +/// stall or the deadline, so a stall or deadline stop always has +/// `process_alive`. A live `running` record is what +/// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one`, and a +/// live `starting` one what +/// `recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline`, +/// leaves alone until [`rocm_core::SERVE_STARTUP_DEADLINE`], so `rocmd` does +/// not kill the server [`ServeStartupStop::user_message`] just said is still +/// running. +pub(crate) const fn recorded_status_after_stop(stop: &ServeStartupStop) -> &'static str { + match stop.reason { + ServeStartupStopReason::EngineFailed | ServeStartupStopReason::ProcessExited => "failed", + ServeStartupStopReason::DownloadStalled + | ServeStartupStopReason::Silent + | ServeStartupStopReason::SilentLoad + | ServeStartupStopReason::Deadline => { + debug_assert!( + stop.process_alive, + "a stall or deadline stop is only reached with a live process" + ); + "running" + } + } +} + +impl ServeStartupStop { + /// Text the CLI prints when the wait stops. + /// + /// A log line is quoted only when [`Self::label`] is that line. The + /// spinner's own "Starting" and "Waiting" labels are not quoted. + pub(crate) fn user_message(&self, model: &str, engine: &str, service_id: &str) -> String { + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + let headline = match self.reason { + ServeStartupStopReason::EngineFailed => { + format!("{model} on {engine} failed: the engine reported failed") + } + ServeStartupStopReason::ProcessExited => { + format!("{model} on {engine} failed: the server process exited") + } + ServeStartupStopReason::DownloadStalled => { + format!("{model} on {engine} stopped making download progress") + } + ServeStartupStopReason::Silent => { + format!("{model} on {engine} produced no log and never answered") + } + ServeStartupStopReason::SilentLoad => { + format!("{model} on {engine} stopped making progress while loading") + } + ServeStartupStopReason::Deadline => { + format!("{model} on {engine} did not become ready within {minutes} minutes") + } + }; + let mut message = headline; + if let Some(line) = &self.log_line { + message.push_str(": "); + message.push_str(line); + } + message.push('\n'); + let logs = service_logs_hint(service_id); + if self.process_alive { + let stop = service_stop_hint(service_id); + let _ = write!( + message, + "The server is still running (service {service_id}). Follow it with `{logs}`, or stop it with `{stop}`." + ); + } else { + let _ = write!( + message, + "The server process is gone (service {service_id}). The log is `{logs}`." + ); + } + message + } +} + +/// Poll until the endpoint can serve, reporting each stage through `on_view`. +/// +/// `observe` returns the model name the probe should ask for (a container +/// engine may not know it until it starts) and the latest engine/process +/// sample. The same HTTP probe as the historical readiness wait is used, so +/// "serving" still means an inference request came back. +#[allow(clippy::too_many_arguments)] +pub(crate) fn wait_for_serve_startup( + engine_name: &str, + model: &str, + host: &str, + port: u16, + endpoint_api_key: Option<&str>, + policy: &ServeStartupPolicy, + observe: &mut dyn FnMut() -> (String, ServeStartupSignals), + on_view: &mut dyn FnMut(&ServeStartupView), +) -> Result { + let started = Instant::now(); + let mut progress_at = started; + let mut token = String::new(); + let mut best = EndpointReadiness::Unreachable; + + // One observe, one probe, one decision and at most one status-line update + // per poll, so the spinner advances once per tick. + loop { + let (probe_model, signals) = observe(); + let mut signals = effective_signals(signals, policy); + signals.log_line = signals + .log_line + .map(|line| redact_log_line(&line, endpoint_api_key)); + // A failed engine or a dead process is decided without waiting on a + // probe that cannot succeed. + if !signals.engine_failed && signals.process_alive { + match probe_service_http_readiness( + engine_name, + host, + port, + &probe_model, + endpoint_api_key, + ) { + EndpointReadiness::Serving => return Ok(EndpointReadiness::Serving), + EndpointReadiness::Listing => best = EndpointReadiness::Listing, + EndpointReadiness::Unreachable => {} + } + } + let next_token = progress_token(best, &signals); + if next_token != token { + token = next_token; + progress_at = Instant::now(); + } + let timing = ServeStartupTiming { + elapsed: started.elapsed(), + since_progress: progress_at.elapsed(), + }; + match decide_serve_startup(model, engine_name, best, &signals, timing, policy) { + ServeStartupDecision::Continue(view) => on_view(&view), + ServeStartupDecision::Ready => return Ok(EndpointReadiness::Serving), + ServeStartupDecision::BudgetElapsed(readiness) => return Ok(readiness), + ServeStartupDecision::Stop(stop) => return Err(stop), + } + + let remaining = policy.bounds.deadline.saturating_sub(started.elapsed()); + if remaining.is_zero() { + continue; + } + thread::sleep(policy.poll.min(remaining)); + } +} + +/// The signals the policy looks at. +/// +/// The piped path ignores engine failure and process liveness so a serve that +/// exits in CI still returns a readiness status. +const fn effective_signals( + mut signals: ServeStartupSignals, + policy: &ServeStartupPolicy, +) -> ServeStartupSignals { + if !policy.watch_engine { + signals.engine_failed = false; + signals.process_alive = true; + } + signals +} + +const REDACTED: &str = "[redacted]"; + +/// Mask credentials in an engine log line before it reaches the status line +/// or the failure text. +/// +/// Engines echo their launch arguments (vLLM prints its non-default args), +/// and those can carry the endpoint key or a hub token. The line is already +/// one ASCII line with single spaces. Masked: the endpoint key wherever it +/// appears, the value after a secret-named flag or key (`--api-key X`, +/// `api_key=X`, `'api_key': 'X'`, `"api_key":"X"`, `Authorization: Bearer X`), +/// and Hugging Face `hf_` tokens. A list value (`'api_key': ['a', 'b']`) has +/// only its first item masked. +pub(crate) fn redact_log_line(line: &str, endpoint_api_key: Option<&str>) -> String { + let line = match endpoint_api_key.filter(|key| !key.is_empty()) { + Some(key) => line.replace(key, REDACTED), + None => line.to_owned(), + }; + let mut out: Vec = Vec::new(); + let mut mask_next = false; + for word in line.split(' ') { + if mask_next { + mask_next = false; + if word.eq_ignore_ascii_case("bearer") || word.eq_ignore_ascii_case("basic") { + out.push(word.to_owned()); + mask_next = true; + continue; + } + out.push(mask_value(word)); + continue; + } + if let Some((name, value)) = word.split_once('=') + && is_secret_name(name) + { + out.push(if value.is_empty() { + format!("{name}=") + } else { + format!("{name}={}", mask_value(value)) + }); + continue; + } + // Compact JSON or YAML: `"api_key":"X"`. + if let Some((name, value)) = word.split_once(':') + && !value.is_empty() + && is_secret_name(name) + { + out.push(format!("{name}:{}", mask_compact_value(value))); + continue; + } + if names_a_following_secret(word) { + mask_next = true; + out.push(word.to_owned()); + continue; + } + if looks_like_hf_token(word) { + out.push(REDACTED.to_owned()); + continue; + } + out.push(word.to_owned()); + } + out.join(" ") +} + +/// Whether `word` names a secret (`--api-key`, `HF_TOKEN`, `'api_key':`, +/// `Authorization:`, `Bearer`). +fn is_secret_name(word: &str) -> bool { + let name = word + .trim_start_matches('-') + .trim_matches(|c: char| matches!(c, '\'' | '"' | ':' | ',' | '{' | '(')) + .to_ascii_lowercase() + .replace('-', "_"); + if name.is_empty() { + return false; + } + name == "authorization" + || name == "bearer" + || ["api_key", "apikey", "token", "password", "secret"] + .iter() + .any(|suffix| name.ends_with(suffix)) +} + +/// A flag (`--api-key`), a key ending in `:` (`'api_key':`, +/// `Authorization:`), or `Bearer`, whose next word is the secret. A bare word +/// such as "token" in prose does not mask what follows it. +fn names_a_following_secret(word: &str) -> bool { + if word.eq_ignore_ascii_case("bearer") { + return true; + } + let unquoted = word.trim_end_matches(['\'', '"', ',']); + (word.starts_with('-') || unquoted.ends_with(':')) && is_secret_name(word) +} + +/// Keep the quoting and trailing punctuation around a masked value, so a +/// Python-dict log line stays readable. +fn mask_value(word: &str) -> String { + if word.contains(REDACTED) { + return word.to_owned(); + } + let body_start = word + .find(|c: char| !matches!(c, '\'' | '"' | '(' | '[')) + .unwrap_or(word.len()); + let body_end = word + .rfind(|c: char| !matches!(c, '\'' | '"' | ',' | ')' | ']' | '}')) + .map_or(body_start, |index| index + 1); + if body_end <= body_start { + return word.to_owned(); + } + format!("{}{REDACTED}{}", &word[..body_start], &word[body_end..]) +} + +/// Mask one compact value (`"X","port":8000}` or `X,port:8000}`) and keep +/// what follows it, so the rest of a one-word JSON object stays readable. +fn mask_compact_value(value: &str) -> String { + if let Some(quote) = value.chars().next().filter(|c| matches!(c, '"' | '\'')) { + let rest = &value[1..]; + return match rest.find(quote) { + Some(end) if end > 0 => format!("{quote}{REDACTED}{}", &rest[end..]), + _ => value.to_owned(), + }; + } + let end = value.find([',', '}', ']']).unwrap_or(value.len()); + if end == 0 { + return value.to_owned(); + } + format!("{REDACTED}{}", &value[end..]) +} + +fn looks_like_hf_token(word: &str) -> bool { + let body = word.trim_matches(|c: char| !c.is_ascii_alphanumeric() && c != '_'); + body.strip_prefix("hf_") + .is_some_and(|rest| rest.len() >= 20 && rest.chars().all(|c| c.is_ascii_alphanumeric())) +} + +#[cfg(test)] +mod tests { + use std::io::{Read, Write}; + use std::net::TcpListener; + use std::thread; + use std::time::{Duration, Instant}; + + use super::*; + use crate::service_hints::{service_logs_hint, service_stop_hint}; + use crate::test_support::unique_suffix; + + fn alive() -> ServeStartupSignals { + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: None, + } + } + + fn at(elapsed_ms: u64, since_ms: u64) -> ServeStartupTiming { + ServeStartupTiming { + elapsed: Duration::from_millis(elapsed_ms), + since_progress: Duration::from_millis(since_ms), + } + } + + fn interactive() -> ServeStartupPolicy { + ServeStartupPolicy::interactive() + } + + fn assert_continue(decision: ServeStartupDecision) -> ServeStartupView { + match decision { + ServeStartupDecision::Continue(view) => view, + other => panic!("expected the wait to keep going, got {other:?}"), + } + } + + fn assert_stop(decision: ServeStartupDecision) -> ServeStartupStop { + match decision { + ServeStartupDecision::Stop(stop) => stop, + other => panic!("expected the wait to stop, got {other:?}"), + } + } + + #[test] + fn unreachable_keeps_the_starting_label_until_the_silent_limit() { + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Unreachable); + assert_eq!(view.label, starting_label("qwen", "vllm")); + } + + #[test] + fn unreachable_with_no_log_stops_when_nothing_arrives() { + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at( + SILENT_UNREACHABLE.as_millis() as u64, + SILENT_UNREACHABLE.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::Silent); + assert_eq!(stop.phase, ServeStartupPhase::Unreachable); + assert_eq!(stop.label, starting_label("qwen", "vllm")); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!( + message.contains("produced no log and never answered"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + assert!(message.contains(&service_stop_hint("svc-1")), "{message}"); + assert!( + !message.contains("Deployment summary"), + "a silent start must not become a finished summary: {message}" + ); + // The stop command is the one the approval gate runs. That it stops + // a live service is `@id:service-stop-yes-stops-a-running-service` + // in tests/e2e-cucumber/features/service_stop.feature. The logs + // command is `services_text::render_service_logs_text`, proven to + // tail the service log by `render_service_logs_text_tails_manifest_log`. + } + + #[test] + fn downloading_past_the_old_budget_keeps_reporting_the_log_line() { + let line = "Downloading model.safetensors: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + // 50s is past the old 45s budget. Progress moved 1s ago, so this is + // not a hung download. + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(50_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Downloading); + assert_eq!(view.label, line); + } + + #[test] + fn hung_download_stops_and_the_message_quotes_the_stalled_line() { + let line = "Downloading model.safetensors: 12%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at( + DOWNLOAD_STALL.as_millis() as u64 + 1_000, + DOWNLOAD_STALL.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!( + message.contains("stopped making download progress"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + assert!( + !message.contains("not ready yet"), + "a hung download is a failure, not a finished summary: {message}" + ); + assert!( + Duration::from_millis(DOWNLOAD_STALL.as_millis() as u64) + < rocm_core::SERVE_STARTUP_DEADLINE, + "the stall must fire before the overall deadline" + ); + } + + #[test] + fn a_percent_that_does_not_change_is_the_download_fingerprint() { + assert_eq!( + download_fingerprint("Downloading model.safetensors: 12%| ETA 0:40"), + "pct:12" + ); + assert_eq!( + download_fingerprint("Downloading model.safetensors: 12%| ETA 0:01"), + "pct:12", + "a moving ETA at the same percent is not progress" + ); + assert_ne!( + download_fingerprint("Downloading model.safetensors: 13%"), + download_fingerprint("Downloading model.safetensors: 12%") + ); + } + + #[test] + fn loading_past_the_old_budget_keeps_reporting_the_log_line() { + let line = "Loading safetensors checkpoint shards: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(50_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + assert_eq!(view.label, line); + } + + #[test] + fn loading_with_no_new_log_stops() { + let line = "Loading safetensors checkpoint shards: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at( + SILENT_LOAD.as_millis() as u64 + 1_000, + SILENT_LOAD.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!( + message.contains("stopped making progress while loading"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + } + + #[test] + fn listing_past_the_old_budget_keeps_waiting() { + let mut signals = alive(); + signals.log_line = Some("Loading safetensors checkpoint shards: 80%".to_owned()); + // The log has been quiet for longer than the load stall. Listing + // still waits: the server is answering, and a GPU load can sit on + // one line. The exit is the overall deadline, not the stall. + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &signals, + at(50_000, SILENT_LOAD.as_millis() as u64), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, "Loading safetensors checkpoint shards: 80%"); + } + + #[test] + fn listing_keeps_a_stale_download_line_off_the_download_stall() { + // vLLM often still has "download" as the last log line after it has + // started advertising the model. That line must not arm the stall: + // the server has answered, and the exit is the overall deadline. + let line = "Downloading model.safetensors: 100%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &signals, + at(50_000, DOWNLOAD_STALL.as_millis() as u64), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, line); + } + + #[test] + fn listing_without_a_log_says_it_is_waiting_to_serve() { + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &alive(), + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, waiting_label("qwen")); + } + + #[test] + fn listing_at_the_deadline_stops_without_a_not_ready_summary() { + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &alive(), + at(rocm_core::SERVE_STARTUP_DEADLINE.as_millis() as u64, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + assert_eq!(stop.label, waiting_label("qwen")); + let message = stop.user_message("qwen", "vllm", "svc-1"); + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + assert!(message.contains(&format!("{minutes} minutes")), "{message}"); + assert!(message.contains("The server is still running"), "{message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + assert!( + !message.contains("not ready yet") && !message.contains("Deployment summary"), + "{message}" + ); + } + + #[test] + fn serving_is_ready() { + let decision = decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Serving, + &alive(), + at(1_000, 0), + &interactive(), + ); + assert!(matches!(decision, ServeStartupDecision::Ready)); + } + + #[test] + fn engine_failed_stops_immediately_and_quotes_the_log() { + let mut signals = alive(); + signals.engine_failed = true; + signals.log_line = Some("CUDA out of memory".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(0, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::EngineFailed); + assert_eq!(stop.label, "CUDA out of memory"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains("the engine reported failed"), "{message}"); + assert!(message.contains("CUDA out of memory"), "{message}"); + } + + #[test] + fn process_exited_stops_without_claiming_the_server_is_running() { + let mut signals = alive(); + signals.process_alive = false; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(0, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::ProcessExited); + assert!(!stop.process_alive); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains("the server process exited"), "{message}"); + assert!(message.contains("The server process is gone"), "{message}"); + assert!( + !message.contains("still running"), + "a dead process must not be described as still running: {message}" + ); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + } + + #[test] + fn piped_serve_returns_at_the_status_budget_instead_of_following_a_download() { + let mut signals = alive(); + signals.log_line = Some("Downloading model.safetensors: 40%".to_owned()); + let policy = ServeStartupPolicy::status_budget(Duration::from_secs(45)); + let decision = decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(45_000, 0), + &policy, + ); + assert_eq!( + decision, + ServeStartupDecision::BudgetElapsed(EndpointReadiness::Unreachable) + ); + } + + #[test] + fn sanitize_log_line_keeps_the_latest_progress_segment_and_drops_ansi() { + let raw = "\u{1b}[32mold line\rDownloading model.safetensors: 12%\u{1b}[0m"; + assert_eq!( + sanitize_log_line(raw, None), + "Downloading model.safetensors: 12%" + ); + } + + #[test] + fn latest_meaningful_log_line_skips_blank_and_rule_lines() { + let path = std::env::temp_dir().join(format!("serve-progress-{}.log", unique_suffix())); + std::fs::write( + &path, + "Loading safetensors checkpoint shards: 40%\n\n----------\n", + ) + .unwrap(); + let line = latest_meaningful_log_line(&path, None); + let _ = std::fs::remove_file(&path); + assert_eq!( + line.as_deref(), + Some("Loading safetensors checkpoint shards: 40%") + ); + } + + #[test] + fn failure_message_names_the_same_logs_command_as_the_summary() { + let hint = service_logs_hint("svc-1"); + let summary = crate::serve_summary::DeploymentSummary { + engine: "vllm".to_owned(), + requested_model: "qwen".to_owned(), + api_model: "qwen".to_owned(), + chat_endpoint: "http://127.0.0.1:1/v1/chat/completions".to_owned(), + anthropic_endpoint: None, + service_id: "svc-1".to_owned(), + status: "starting".to_owned(), + already_running: false, + metrics: crate::serve_summary::SmokeMetrics::default(), + api_key: None, + notes: Vec::new(), + }; + let rendered = crate::serve_summary::render_summary(&summary); + // The message names only the hint: the rendered summary can carry an + // endpoint key. + assert!( + rendered.contains(&hint), + "the deployment summary must name `{hint}`" + ); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at(SILENT_UNREACHABLE.as_millis() as u64, 0), + &interactive(), + )); + assert!(stop.user_message("qwen", "vllm", "svc-1").contains(&hint)); + } + + fn short_policy(deadline: Duration, download_stall: Duration) -> ServeStartupPolicy { + ServeStartupPolicy { + bounds: ServeStartupBounds { + deadline, + download_stall, + silent_unreachable: Duration::from_secs(30), + silent_load: Duration::from_secs(30), + }, + poll: Duration::from_millis(20), + stop_on_give_up: true, + watch_engine: true, + } + } + + #[test] + fn wait_follows_a_download_until_the_server_serves() { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let started = Instant::now(); + let serve_after = Duration::from_millis(400); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let inference_ready = started.elapsed() >= serve_after; + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + if inference_ready { + ( + "HTTP/1.1 200 OK", + r#"{"choices":[{"message":{"content":"ok"}}]}"#, + ) + } else { + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } + } else { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + + let mut labels = Vec::new(); + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: Some("Downloading model.safetensors: 40%".to_owned()), + }, + ) + }, + &mut |view| labels.push(view.label.clone()), + ); + drop(server); + + assert_eq!(result.unwrap(), EndpointReadiness::Serving); + assert!( + started.elapsed() >= serve_after, + "the wait returned before the server could serve" + ); + assert!( + labels + .iter() + .any(|label| label == "Downloading model.safetensors: 40%"), + "the status line must show the download while the wait continues: {labels:?}" + ); + } + + #[test] + fn wait_stops_a_hung_download_before_the_deadline() { + let line = "Downloading model.safetensors: 12%"; + let stall = Duration::from_millis(200); + let deadline = Duration::from_secs(5); + let started = Instant::now(); + let mut labels = Vec::new(); + // Nothing is listening: the probe fails fast on connection refused. + let port = { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + port + }; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(deadline, stall), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: Some(line.to_owned()), + }, + ) + }, + &mut |view| labels.push(view.label.clone()), + ); + let elapsed = started.elapsed(); + let stop = result.expect_err("a frozen download must fail the wait"); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!(message.contains("still running"), "{message}"); + assert!( + labels.iter().any(|label| label == line), + "the spinner must have shown the stalled line: {labels:?}" + ); + assert!( + elapsed < Duration::from_secs(3), + "stall fired at {elapsed:?}, not before the {deadline:?} deadline" + ); + } + + #[test] + fn wait_reports_serving_once_inference_answers_from_the_first_probe() { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let body = if request.starts_with("POST /v1/chat/completions ") { + r#"{"choices":[{"message":{"content":"ok"}}]}"# + } else { + r#"{"data":[{"id":"qwen"}]}"# + }; + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + let started = Instant::now(); + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || ("qwen".to_owned(), alive()), + &mut |_| {}, + ); + drop(server); + assert_eq!(result.unwrap(), EndpointReadiness::Serving); + assert!( + started.elapsed() < SILENT_UNREACHABLE, + "a server that is already serving must not wait out the silent limit" + ); + } + + fn refused_port() -> u16 { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + port + } + + #[test] + fn stop_records_failed_only_for_a_dead_server() { + let stop = |reason, process_alive, readiness| ServeStartupStop { + phase: ServeStartupPhase::Loading, + reason, + label: String::new(), + log_line: None, + process_alive, + readiness, + }; + assert_eq!( + recorded_status_after_stop(&stop( + ServeStartupStopReason::EngineFailed, + true, + EndpointReadiness::Unreachable + )), + "failed" + ); + assert_eq!( + recorded_status_after_stop(&stop( + ServeStartupStopReason::ProcessExited, + false, + EndpointReadiness::Listing + )), + "failed" + ); + // A live server that stalled is still up: the record must not say + // failed while the message tells the user it is still running, and + // must not say `starting`, which rocmd restarts once it is stale. + assert_eq!( + recorded_status_after_stop(&stop( + ServeStartupStopReason::DownloadStalled, + true, + EndpointReadiness::Unreachable + )), + "running" + ); + assert_eq!( + recorded_status_after_stop(&stop( + ServeStartupStopReason::Silent, + true, + EndpointReadiness::Unreachable + )), + "running" + ); + assert_eq!( + recorded_status_after_stop(&stop( + ServeStartupStopReason::Deadline, + true, + EndpointReadiness::Listing + )), + "running" + ); + } + + #[test] + fn wait_stops_at_once_when_the_process_has_exited() { + let started = Instant::now(); + let mut views = 0; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + refused_port(), + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: false, + log_line: Some("Traceback: boom".to_owned()), + }, + ) + }, + &mut |_| views += 1, + ); + let stop = result.expect_err("a dead process cannot become ready"); + assert_eq!(stop.reason, ServeStartupStopReason::ProcessExited); + assert_eq!(views, 0, "no spinner frame for a server that is gone"); + assert!(started.elapsed() < Duration::from_secs(2)); + } + + #[test] + fn wait_hands_back_listing_at_the_deadline_with_a_stop() { + // Row: listing, inference never answers. The interactive wait ends in + // a deadline stop (not a finished summary) carrying `Listing`; the + // piped budget returns `Listing` as `Ok`. + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } else { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + + let mut labels = Vec::new(); + let interactive_result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_millis(1_500), Duration::from_secs(30)), + &mut || ("qwen".to_owned(), alive()), + &mut |view| labels.push(view.label.clone()), + ); + let mut budget = ServeStartupPolicy::status_budget(Duration::from_millis(1_500)); + budget.poll = Duration::from_millis(20); + let piped_result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &budget, + &mut || ("qwen".to_owned(), alive()), + &mut |_| {}, + ); + drop(server); + + let stop = interactive_result.expect_err("the interactive wait must end in a stop"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + assert!( + labels.iter().any(|label| *label == waiting_label("qwen")), + "once the model is listed the line says it is waiting to serve: {labels:?}" + ); + assert_eq!(piped_result.unwrap(), EndpointReadiness::Listing); + } + + #[test] + fn wait_keeps_a_changing_download_alive_past_the_stall_limit() { + // A percent that keeps advancing re-arms the stall clock, so a long + // healthy download is not mistaken for a hung one. + let stall = Duration::from_millis(300); + let mut step = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + refused_port(), + None, + &short_policy(Duration::from_millis(1_200), stall), + &mut || { + step += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + // A new percent on every sample re-arms the stall clock. + // Wall-clock buckets can sit still long enough to look hung. + log_line: Some(format!("Downloading model.safetensors: {step}%")), + }, + ) + }, + &mut |_| {}, + ); + let stop = result.expect_err("the deadline still ends the wait"); + assert_eq!( + stop.reason, + ServeStartupStopReason::Deadline, + "a moving download must hit the deadline, not the stall limit" + ); + assert!(step > 3); + } + + #[test] + fn wait_observes_and_updates_the_status_line_once_per_poll() { + // Row: unreachable, refused port. One observe and one status-line + // update per poll, so the spinner does not tick twice per tick. + let poll = Duration::from_millis(20); + let deadline = Duration::from_millis(400); + let mut policy = short_policy(deadline, Duration::from_secs(30)); + policy.poll = poll; + let mut observes = 0_u32; + let mut views = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + refused_port(), + None, + &policy, + &mut || { + observes += 1; + ("qwen".to_owned(), alive()) + }, + &mut |_| views += 1, + ); + let stop = result.expect_err("nothing answers, so the deadline ends the wait"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + // The last observe ends in the stop, which draws no frame. + assert_eq!(views + 1, observes, "one frame per observed poll"); + let polls = u32::try_from(deadline.as_millis() / poll.as_millis()).unwrap(); + assert!( + observes <= polls + 2, + "{observes} observes in {polls} polls: the engine is sampled more than once per poll" + ); + } + + #[test] + fn redact_log_line_masks_keys_tokens_and_secret_flags() { + let key = "sk-endpoint-0123456789"; + for (raw, expected) in [ + ( + "non-default args: {'model': 'qwen', 'api_key': 'sk-endpoint-0123456789', 'port': 8000}", + "non-default args: {'model': 'qwen', 'api_key': '[redacted]', 'port': 8000}", + ), + ( + "launch --api-key abc123 --port 8000", + "launch --api-key [redacted] --port 8000", + ), + ( + "env HF_TOKEN=hf_abcdefghijklmnopqrstuvwx set", + "env HF_TOKEN=[redacted] set", + ), + ( + "using hf_abcdefghijklmnopqrstuvwx for the hub", + "using [redacted] for the hub", + ), + ( + "header Authorization: Bearer abc.def.ghi sent", + "header Authorization: Bearer [redacted] sent", + ), + ( + r#"config {"api_key":"abc123","port":8000}"#, + r#"config {"api_key":"[redacted]","port":8000}"#, + ), + ( + "header Authorization: Basic dXNlcjpwYXNz sent", + "header Authorization: Basic [redacted] sent", + ), + ( + "serving with key sk-endpoint-0123456789 on 0.0.0.0", + "serving with key [redacted] on 0.0.0.0", + ), + ] { + assert_eq!(redact_log_line(raw, Some(key)), expected, "{raw}"); + } + // Ordinary progress lines, including prose about tokens, are untouched. + for line in [ + "Loading safetensors checkpoint shards: 40%", + "Downloading tokenizer.json: 100%", + "the token budget is 8192 tokens", + "max_num_batched_tokens=8192", + ] { + assert_eq!(redact_log_line(line, Some(key)), line); + } + } + + #[test] + fn a_secret_in_the_engine_log_never_reaches_the_status_line_or_the_failure() { + let key = "sk-endpoint-0123456789"; + let mut labels = Vec::new(); + let mut samples = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + refused_port(), + Some(key), + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + // A live sample first, so the status line draws the line, + // then the failure that ends the wait. + samples += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: samples > 1, + process_alive: true, + log_line: Some(format!( + "error: bad args --api-key {key} --hf-token hf_abcdefghijklmnopqrstuvwx" + )), + }, + ) + }, + &mut |view| labels.push(view.label.clone()), + ); + let stop = result.expect_err("a failed engine ends the wait"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + for text in [&stop.label, &message] { + assert!( + !text.contains(key), + "endpoint key leaked: {}", + text.replace(key, "") + ); + assert!( + !text.contains("hf_abcdefghijklmnopqrstuvwx"), + "hub token leaked" + ); + } + assert!( + message.contains("--api-key [redacted]"), + "the line is still quoted, masked" + ); + assert!(!labels.is_empty(), "the live sample drew a status line"); + for label in &labels { + assert!(!label.contains(key), "endpoint key on the status line"); + assert!(label.contains("--api-key [redacted]"), "{label}"); + } + } + + #[test] + fn a_key_cut_at_the_length_limit_is_masked_before_truncation() { + let key = "sk-endpoint-0123456789abcdef"; + // Put the key across the MAX_LABEL_CHARS boundary. + let pad = "x".repeat(MAX_LABEL_CHARS - 25); + let raw = format!("args {pad} {key} trailing words"); + let line = sanitize_log_line(&raw, Some(key)); + assert!( + !line.contains("sk-endpoint"), + "key prefix survived truncation: {line}" + ); + assert!(line.chars().count() <= MAX_LABEL_CHARS); + } +} diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index cd0ab34bc6..90100c3053 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -5,7 +5,8 @@ //! Engine-agnostic deployment summary for `rocm serve`. //! //! By default `rocm serve ` no longer streams raw engine logs. Instead it -//! shows an animated status indicator while the server starts, runs a small +//! shows a status line that follows the engine (a download, a load, or the +//! latest log line) while the server starts, runs a small //! inference smoke test, and then prints a compact summary table: deployment //! status, the full inference endpoint (with port), the API-qualified model name, //! and the measured time-to-first-token / throughput. @@ -97,10 +98,11 @@ pub(crate) fn format_tps(tps: Option) -> String { /// Render the deployment summary table as plain text (returned rather than /// printed so it is unit-testable and identical across engines). pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { - // The server answered its health check within the startup window. A launch - // that timed out lands here with `status == "starting"`; the heading and a - // note must make that visibly different from a healthy deployment so the - // summary is never mistaken for success. + // A ready server answered an inference request. Any other status on a fresh + // launch gets a distinct heading so it cannot be read as success. + // Interactive `rocm serve` does not render this heading for a startup that + // is still loading: that wait ends in `ServeStartupStop::user_message` + // (`listing_at_the_deadline_stops_without_a_not_ready_summary`). let ready = summary.status == "ready"; let heading = if summary.already_running { "Deployment summary (already running)" @@ -148,7 +150,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ), ( "logs", - format!("rocm logs --service {}", summary.service_id), + crate::service_hints::service_logs_hint(&summary.service_id), ), ], ] @@ -185,8 +187,8 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { }; let _ = writeln!( out, - " note: {explanation} - follow `rocm logs --service {} --follow` or check `rocm services list`", - summary.service_id + " note: {explanation} - follow `{} --follow` or check `rocm services list`", + crate::service_hints::service_logs_hint(&summary.service_id) ); } for note in &summary.notes { diff --git a/apps/rocm/src/service_hints.rs b/apps/rocm/src/service_hints.rs index 6556c2e70a..9216dd4b92 100644 --- a/apps/rocm/src/service_hints.rs +++ b/apps/rocm/src/service_hints.rs @@ -53,6 +53,15 @@ pub(crate) fn service_stop_hint(service_id: &str) -> String { service_action_hint("stop", service_id) } +/// The directly-runnable `rocm logs --service ` hint. +/// +/// `rocm logs --service` prints `services_text::render_service_logs_text` for +/// that service. Shared so a startup failure and the deployment summary cannot +/// name two different log commands. +pub(crate) fn service_logs_hint(service_id: &str) -> String { + format!("rocm logs --service {service_id}") +} + /// The directly-runnable ` services stop --yes` hint for a /// service on a remote machine, where `remote_cli` is that machine's own /// `rocm` binary name (not necessarily `"rocm"` — it is whatever the remote diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index 1e14c85599..826b830183 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -18,6 +18,7 @@ use super::spawn::ManagedLaunchReport; use crate::device_policy_name; use crate::endpoint_keys; use crate::engine_runtime_select::wait_for_service_http_ready_with_progress; +use crate::serve_progress; use crate::status_for_readiness; #[cfg(feature = "e2e-oom-fault-injection")] use anyhow::Context; @@ -308,6 +309,53 @@ pub(super) fn wait_for_record_ready( } } +/// One sample of a freshly launched service for the startup wait: the model +/// name its endpoint answers to and the engine/process signals. +/// +/// Read from the manifest on disk, like [`service_still_live`], so a +/// `rocm services stop` from another terminal is seen. Liveness is +/// [`service_pid_is_alive`]: the supervisor is an unreaped child of this +/// process, so a dead one still passes `process_is_running`. +fn observe_launch( + paths: &AppPaths, + held: &ManagedServiceRecord, + endpoint_api_key: Option<&str>, + pid_alive: impl Fn(u32) -> bool, +) -> (String, serve_progress::ServeStartupSignals) { + let mut latest = ManagedServiceRecord::read_manifest(paths, &held.service_id) + .unwrap_or_else(|_| held.clone()); + let _ = latest.refresh_from_engine_state(); + let pids = crate::managed_service_status::recorded_service_pids(&latest); + let signals = serve_progress::ServeStartupSignals { + engine_failed: latest.status == "failed", + process_alive: pids.is_empty() || pids.into_iter().any(pid_alive), + log_line: serve_progress::latest_meaningful_log_line(&latest.log_path, endpoint_api_key), + }; + (latest.served_model_name().to_owned(), signals) +} + +/// Follow a fresh launch (without `--wait`) until it serves, the policy's +/// budget runs out, or the wait stops. See [`serve_progress`]. +pub(super) fn wait_for_launch_startup( + paths: &AppPaths, + record: &ManagedServiceRecord, + requested_model: &str, + endpoint_api_key: Option<&str>, + policy: &serve_progress::ServeStartupPolicy, + on_view: &mut dyn FnMut(&serve_progress::ServeStartupView), +) -> Result { + serve_progress::wait_for_serve_startup( + &record.engine, + requested_model, + &record.host, + record.port, + endpoint_api_key, + policy, + &mut || observe_launch(paths, record, endpoint_api_key, service_pid_is_alive), + on_view, + ) +} + /// An equivalent service is already live. Without `--wait`, or once it is /// serving, report it as is; otherwise wait for it like a fresh launch would. pub(super) fn wait_on_existing_service( diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 06cb93697e..fbd83e961b 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -7,13 +7,14 @@ use super::endpoint_auth::drop_orphaned_endpoint_key_on_already_running; use super::readiness::{ - died_during_wait, record_wait_outcome, service_pid_is_alive, wait_for_record_ready, - wait_on_existing_service, + died_during_wait, record_wait_outcome, service_pid_is_alive, wait_for_launch_startup, + wait_for_record_ready, wait_on_existing_service, }; #[cfg(feature = "e2e-oom-fault-injection")] use super::readiness::{e2e_simulate_oom_launch, simulate_oom_managed_launch}; use super::spawn::{ManagedLaunchReport, ManagedSpawn, spawn_managed_engine_child}; use crate::serve_port; +use crate::serve_progress::{self, ServeStartupPhase, ServeStartupPolicy, ServeStartupView}; use crate::{record_cli_audit_event, status_for_readiness}; use anyhow::Result; use rocm_core::format_http_base_url; @@ -22,6 +23,7 @@ use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse} use std::fmt::Write as _; #[cfg(windows)] use std::thread; +#[cfg(windows)] use std::time::Duration; /// Launch a managed service in the background: run the locked allocation and @@ -55,9 +57,15 @@ pub(crate) fn start_managed_service( require_api_key: bool, gpu_refusal_waived_for_reuse: bool, wait_until_ready: bool, + policy: &ServeStartupPolicy, on_wait_begin: &mut dyn FnMut(&str, u16), - on_wait_tick: &mut dyn FnMut(Duration), + on_view: &mut dyn FnMut(&ServeStartupView), ) -> Result { + // `--wait` keeps the label it started with: its wait reports no stage. + let steady = ServeStartupView { + phase: ServeStartupPhase::Unreachable, + label: serve_progress::starting_label(requested_model, engine), + }; #[cfg(feature = "e2e-oom-fault-injection")] if e2e_simulate_oom_launch() { return simulate_oom_managed_launch( @@ -104,7 +112,9 @@ pub(crate) fn start_managed_service( report, wait_until_ready, on_wait_begin, - on_wait_tick, + &mut |_elapsed| { + on_view(&steady); + }, ); } ManagedSpawn::Spawned { record, child_pid } => (*record, child_pid), @@ -121,16 +131,28 @@ pub(crate) fn start_managed_service( #[cfg(windows)] thread::sleep(Duration::from_millis(200)); - if wait_until_ready { + if announces_wait(wait_until_ready, policy) { on_wait_begin(service_id, port); } - let readiness = wait_for_record_ready( - paths, - &record, - wait_until_ready, - endpoint_api_key, - on_wait_tick, - ); + let (readiness, startup_stop) = if wait_until_ready { + let readiness = + wait_for_record_ready(paths, &record, true, endpoint_api_key, &mut |_elapsed| { + on_view(&steady); + }); + (readiness, None) + } else { + match wait_for_launch_startup( + paths, + &record, + requested_model, + endpoint_api_key, + policy, + on_view, + ) { + Ok(readiness) => (readiness, None), + Err(stop) => (stop.readiness, Some(stop)), + } + }; if let Some(error) = died_during_wait( paths, service_id, @@ -140,12 +162,15 @@ pub(crate) fn start_managed_service( ) { return Err(error); } - let launch_status = status_for_readiness(readiness); + let launch_status = match &startup_stop { + Some(stop) => serve_progress::recorded_status_after_stop(stop), + None => status_for_readiness(readiness), + }; record_wait_outcome( paths, &mut record, launch_status, - readiness == EndpointReadiness::Serving, + readiness == EndpointReadiness::Serving && startup_stop.is_none(), )?; let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); record_cli_audit_event( @@ -159,6 +184,14 @@ pub(crate) fn start_managed_service( ), Some(service_id), ); + if let Some(stop) = startup_stop { + return Err(serve_progress::ServeStartupFailed(stop.user_message( + requested_model, + engine, + service_id, + )) + .into()); + } Ok(ManagedLaunchReport { service_id: service_id.to_owned(), endpoint_url, @@ -170,6 +203,17 @@ pub(crate) fn start_managed_service( }) } +/// Whether a fresh launch announces its wait (the server id, how to follow its +/// log, and a public bind's key) before waiting. +/// +/// `--wait` does, and so does the interactive wait, which can run for +/// [`rocm_core::SERVE_STARTUP_DEADLINE`] and can end in an error that prints no +/// summary: without the announcement a public bind's key would never be shown. +/// The piped status budget returns a report either way and prints the key there. +const fn announces_wait(wait_until_ready: bool, policy: &ServeStartupPolicy) -> bool { + wait_until_ready || policy.stop_on_give_up +} + /// Reproduce the original plain, line-by-line managed-launch output. Used for /// non-interactive output (piped, CI, the chat assistant's `serve --managed`), /// where the animated summary is inappropriate. The interactive path renders the @@ -263,6 +307,21 @@ mod tests { use std::fs; #[cfg(not(windows))] use std::io; + #[cfg(not(windows))] + use std::time::Duration; + + /// A wait that can end in an error without a summary announces itself, so + /// a public bind's key is printed before it. The piped status budget + /// always returns a report, and the summary prints the key there. + #[test] + fn a_wait_that_can_fail_without_a_summary_is_announced() { + let interactive = ServeStartupPolicy::interactive(); + let piped = ServeStartupPolicy::status_budget(serve_progress::PIPED_SERVE_STATUS_BUDGET); + assert!(announces_wait(false, &interactive)); + assert!(announces_wait(true, &interactive)); + assert!(announces_wait(true, &piped)); + assert!(!announces_wait(false, &piped)); + } #[test] fn wait_begin_shows_the_key_only_for_the_launch_it_belongs_to() { @@ -369,8 +428,9 @@ mod tests { false, false, false, + &ServeStartupPolicy::status_budget(Duration::ZERO), &mut |_, _| {}, - &mut |_elapsed| {}, + &mut |_view| {}, ); let Err(error) = result else { let _ = fs::remove_dir_all(root); @@ -466,8 +526,9 @@ mod tests { false, false, false, + &ServeStartupPolicy::status_budget(Duration::ZERO), &mut |_, _| {}, - &mut |_elapsed| {}, + &mut |_view| {}, ) .map(|_| ()) }; diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index aac0a9e9d5..ec564bf4d4 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -13,7 +13,7 @@ use crate::supervise::restart_managed_service; use anyhow::{Context, Result}; use rocm_core::{ AppPaths, AutomationRuntimeState, AutomationTriggerEvent, ManagedServiceRecord, WatcherMode, - format_host_port, unix_time_millis, + format_host_port, process_is_running, unix_time_millis, }; #[cfg(test)] use serde_json::json; @@ -327,6 +327,14 @@ pub(crate) fn find_recoverable_service( continue; } if let Some(reason) = manifest_service_recovery_reason(&record, now) { + // `rocm services` persists an engine's own `starting` state into + // the record, so a live server still downloading under the + // interactive serve wait (or left by a Ctrl-C of it) can read + // `starting` past the stale-start window. It is not stale until + // the startup deadline. + if record.status == "starting" && live_startup_still_in_progress(&record, now) { + continue; + } return Ok(Some((record, reason))); } if matches!(record.status.as_str(), "ready" | "running") { @@ -335,6 +343,12 @@ pub(crate) fn find_recoverable_service( &record.engine, &record.service_id, ) else { + // A live process that has not opened its port is still the + // interactive serve wait's job. Restarting it here kills a + // download the CLI is reporting. A dead pid is not spared. + if record.status == "running" && live_startup_still_in_progress(&record, now) { + continue; + } if let Some(reason) = endpoint_service_recovery_reason(&record) { return Ok(Some((record, reason))); } @@ -347,6 +361,9 @@ pub(crate) fn find_recoverable_service( ))); } if let Some(reason) = endpoint_service_recovery_reason(&record) { + if record.status == "running" && live_startup_still_in_progress(&record, now) { + continue; + } return Ok(Some((record, reason))); } } @@ -368,6 +385,26 @@ pub(crate) const fn stop_requested(record: &ManagedServiceRecord) -> bool { record.stop_requested_unix_ms.is_some() } +/// A recorded process is alive and the service is younger than +/// [`rocm_core::SERVE_STARTUP_DEADLINE`]. +/// +/// That is the window interactive `rocm serve` spends following a download +/// or load before the port accepts connections. Endpoint-unreachable +/// recovery during it restarts the process out from under that wait. +fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u128) -> bool { + let started_at = record + .last_restart_unix_ms + .unwrap_or(record.created_at_unix_ms); + let age_ms = now_unix_ms.saturating_sub(started_at); + if age_ms >= rocm_core::SERVE_STARTUP_DEADLINE.as_millis() { + return false; + } + [Some(record.supervisor_pid), record.engine_pid] + .into_iter() + .flatten() + .any(|pid| pid != 0 && process_is_running(pid)) +} + fn endpoint_service_recovery_reason(record: &ManagedServiceRecord) -> Option { (!wait_for_port(&record.host, record.port, ENDPOINT_HEALTH_TIMEOUT)) .then(|| "endpoint_status_unreachable".to_owned()) @@ -1122,4 +1159,136 @@ mod tests { assert_eq!(record.service_id, "svc-backoff-b"); Ok(()) } + + /// A live process that has not opened its port yet is the interactive + /// `rocm serve` wait, not a crash. Endpoint recovery must leave it alone + /// until [`rocm_core::SERVE_STARTUP_DEADLINE`]. A dead pid at the same age + /// is still recovered: the guard is the live process, not the clock alone. + #[test] + fn recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one() -> Result<()> { + let (root, paths) = temp_app_paths("recover-live-startup"); + paths.ensure()?; + + let mut live = ManagedServiceRecord::new( + &paths, + "svc-live-startup", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + 11461, + "managed", + std::process::id(), + None, + None, + None, + ); + live.status = "running".to_owned(); + let now = unix_time_millis(); + let deadline_ms = rocm_core::SERVE_STARTUP_DEADLINE.as_millis(); + // Two seconds inside the bound, so the check below cannot cross it. + // A shorter window than the shared deadline would recover this. + live.created_at_unix_ms = now.saturating_sub(deadline_ms.saturating_sub(2_000)); + live.write()?; + + let found_inside = find_recoverable_service(&paths)?; + + live.supervisor_pid = u32::MAX; + live.engine_pid = None; + live.write()?; + let found_dead = find_recoverable_service(&paths)?; + + live.supervisor_pid = std::process::id(); + // Age equal to the deadline is no longer the interactive wait. + // A longer window than the shared deadline would spare this. + live.created_at_unix_ms = now.saturating_sub(deadline_ms); + live.write()?; + let found_at_deadline = find_recoverable_service(&paths)?; + fs::remove_dir_all(root).ok(); + + assert!( + found_inside.is_none(), + "a live process inside the startup deadline must not be restarted: {found_inside:?}" + ); + assert_eq!( + found_dead.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-startup".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + "a dead pid is not a startup still in progress" + ); + assert_eq!( + found_at_deadline.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-startup".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + "at the startup deadline a live process with a closed port is recoverable" + ); + Ok(()) + } + + /// `rocm services` writes an engine's own `starting` state into the + /// record, so a live startup can read `starting` past the five-minute + /// stale-start window. That record is spared until + /// [`rocm_core::SERVE_STARTUP_DEADLINE`] like a `running` one; a dead pid + /// at the same age, or a live one past the deadline, is still stale. + #[test] + fn recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline() -> Result<()> { + let (root, paths) = temp_app_paths("recover-live-starting"); + paths.ensure()?; + + let mut live = ManagedServiceRecord::new( + &paths, + "svc-live-starting", + "lemonade", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + 11462, + "managed", + std::process::id(), + None, + None, + None, + ); + live.status = "starting".to_owned(); + let now = unix_time_millis(); + let deadline_ms = rocm_core::SERVE_STARTUP_DEADLINE.as_millis(); + // Past the stale-start window, two seconds inside the startup deadline. + live.created_at_unix_ms = now.saturating_sub(deadline_ms.saturating_sub(2_000)); + assert!(now.saturating_sub(live.created_at_unix_ms) >= SERVER_TRANSIENT_STALE_MS); + live.write()?; + let found_inside = find_recoverable_service(&paths)?; + + live.supervisor_pid = u32::MAX; + live.write()?; + let found_dead = find_recoverable_service(&paths)?; + + live.supervisor_pid = std::process::id(); + live.created_at_unix_ms = now.saturating_sub(deadline_ms); + live.write()?; + let found_at_deadline = find_recoverable_service(&paths)?; + fs::remove_dir_all(root).ok(); + + assert!( + found_inside.is_none(), + "a live starting process inside the startup deadline must not be restarted: {found_inside:?}" + ); + for (found, why) in [ + (found_dead, "a dead pid is a stale start"), + (found_at_deadline, "a live start past the deadline is stale"), + ] { + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-starting".to_owned(), + "manifest_status_starting_stale".to_owned() + )), + "{why}" + ); + } + Ok(()) + } } diff --git a/crates/rocm-core/src/endpoint_readiness.rs b/crates/rocm-core/src/endpoint_readiness.rs index fe4eec3a5e..897b850bab 100644 --- a/crates/rocm-core/src/endpoint_readiness.rs +++ b/crates/rocm-core/src/endpoint_readiness.rs @@ -211,6 +211,12 @@ pub fn managed_service_endpoint_model_ready( openai_models_endpoint_has_model(&record.endpoint_url, expected, endpoint_api_key, timeout) } +/// Interactive `rocm serve` startup deadline. +/// +/// `rocmd`'s server-recover watcher uses the same bound so it does not restart +/// a live process that has not opened its port yet. +pub const SERVE_STARTUP_DEADLINE: Duration = Duration::from_mins(30); + /// How far along a managed service's endpoint is. /// /// The middle state is the one that matters: an engine lists a model within diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index a8181620d0..27f4ad5dea 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -117,7 +117,7 @@ pub use download::{ }; pub use endpoint_key::{endpoint_api_key_has_forbidden_chars, generate_endpoint_api_key}; pub use endpoint_readiness::{ - EndpointReadiness, EndpointReadinessOutcome, INFERENCE_PROBE_TIMEOUT, + EndpointReadiness, EndpointReadinessOutcome, INFERENCE_PROBE_TIMEOUT, SERVE_STARTUP_DEADLINE, engine_state_inference_verified, managed_service_endpoint_model_ready, managed_service_endpoint_readiness, merge_json_state_file, openai_chat_completion_probe, openai_chat_completion_status, openai_models_endpoint_has_model, diff --git a/docs/architecture.md b/docs/architecture.md index 75cdbc5dd7..06f959d100 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -89,11 +89,12 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): `rocm runtimes uninstall` and the storage pruning that removes a registered runtime. `plan_runtime_uninstall` builds the plan (`RuntimeUninstallPlan`, with the `InstallRootDecision` for its folder), `print_runtime_uninstall_plan` shows it, `confirm_and_revalidate_runtime_uninstall` asks and then checks that nothing changed while the prompt was open (`revalidate_runtime_uninstall_plan`), and `apply_runtime_uninstall` unregisters the runtime, clears the active and default state that named it and removes its install root when `should_remove_runtime_install_root` says ROCm CLI owns it (`local_runtime_manifest_matches`, `ensure_runtime_install_root_is_safe_to_remove`). `RuntimeUninstallPlan` also records whether Lemonade's install record points inside the runtime's folder (`lemonade_install_inside`, from `rocm_engine_lemonade::recorded_runtime_dir`), which `print_runtime_uninstall_plan` reports, and whether the folder holds a Lemonade tree at all (`holds_lemonade_tree`), which decides whether an unreadable record is worth reporting; `apply_runtime_uninstall` clears that record once the folder is removed (`RuntimeUninstallResult::lemonade_record`, a `RecordCleanup` that also carries a failed deletion or an unreadable record, worded by `lemonade_record_report_line`). `uninstall_runtime` is the non-interactive entry point. It reads the registry, the active-runtime marker and the path comparison from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and does not depend on `runtimes_cmd.rs`, which calls into it from `runtimes()` and re-exports `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line`. The planning and apply entry points, `RuntimeUninstallPlan`, `RuntimeUninstallConfirmation` and `RuntimeUninstallResult` (with its fields) are `pub(crate)` for `runtimes()`; `revalidate_runtime_uninstall_plan` and the folder-safety helpers are private. Its tests cover planning, revalidation, the folder-removal rules and uninstalling a read-only imported runtime; the uninstall tests that first activate a runtime stay with the activation tests in `runtimes_cmd.rs`. - [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) — mechanical relocation, owns types (Phase 6c, ROCMAI-89): `runtimes()` and its activate/rollback handlers (the activation transaction, including `write_active_runtime_marker`), plus the runtime list rendering; the import and adopt handlers it dispatches to live in [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) and the uninstall plan, confirmation and apply in [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs); registry selection and manifest validation moved to [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), which this module re-exports for the callers that import them from here. `RuntimesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) (part of the `Command`/`Cli` clap definition, the same way `EnginesCommand` does). `RuntimeActivationResult` is `pub(crate)`, read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary`; `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line` (from `runtime_uninstall.rs`) and `ActiveRuntimeMarker` and `active_runtime_marker_path` (from `runtime_registry.rs`) are re-exported for [`storage.rs`](../apps/rocm/src/storage.rs) and [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs); `recover_setup_runtime_registration`, `newest_installed_runtime_manifest` and `setup_runtime_paths` (from `runtime_adopt.rs`) and `normalize_path_for_compare` (from `runtime_registry.rs`) are re-exported for the modules that import them from here, such as [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs), [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) and [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs). [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) is deliberately not one of them: it imports from `runtime_adopt.rs` and `runtime_registry.rs` directly, because going through this module would lead back to it via `runtime_services.rs`. The dependency on the sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) above runs the other way: that module calls into this one (`activate_runtime`, `newest_installed_runtime_manifest`, `recover_setup_runtime_registration`), while this module reaches it only from its test module. - [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) — mechanical relocation, owns types (Phase 6a): `serve()`, plus the engine selection that runs ahead of it (`select_serve_engine`, `recipe_preferred_engine`, `serve_model_ref_for_engine`) and the AIM pre-flight that runs inside it: the Hugging Face token check, the GPU-family profile check and pin, the multi-GPU refusal (`aim_gpu_preflight`), and the board note. The engine-recipe overrides `serve()` applies to the launch hint live in [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs); this module imports them by name. It also holds `collect_serve_notes`, the GPU/device warnings folded into the interactive summary, which was relocated here from `main.rs` with its two tests. Its tests cover engine selection, the AIM pre-flight and the reuse pre-gate; the AIM tests build their hint with `protocol_engine_recipe_hint` and `engine_recipe_with_extra_env` from `serve_recipe.rs`, which `serve()` also calls. +- [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) — interactive `rocm serve` startup status: the stage line (download, load, health) and when that wait stops. [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs)'s `wait_for_launch_startup` calls it for a fresh launch without `--wait`; the historical piped readiness budget stays a separate policy on the same waiter. - [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs) — mechanical relocation, owns a type (split out of [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which imports the entry points by name): the pure transforms `serve()` applies to the selected engine's launch recipe hint. `protocol_engine_recipe_hint` maps the catalog's engine record onto the protocol hint, and `render_serve_engine_recipe_lines` prints it for the serve plan. The explicit controls are then layered on top: `--tool-call-parser` (`engine_recipe_with_tool_call_override`), `--gpu-memory-utilization` (`parse_gpu_memory_utilization`, `engine_recipe_with_gpu_memory_utilization_override`), `--temperature`/`--top-p`/`--max-tokens` (`ServeGenerationDefaults`, `engine_recipe_with_generation_defaults`) and `--engine-env` (`engine_recipe_with_extra_env`, `engine_env_ignored_note`, with the env names it refuses to override in `RESERVED_ENGINE_ENV_KEYS`); `engine_recipe_enables_tool_choice` reads the result back. The per-flag rewriters (`set_vllm_tool_call_parser`, `set_vllm_gpu_memory_utilization`, `set_vllm_override_generation_config`, `set_lemonade_generation_defaults`) are private. The entry points above, and `ServeGenerationDefaults`, are `pub(crate)` for `serve()` and for the AIM tests in `serve_cmd.rs`; `RESERVED_ENGINE_ENV_KEYS` is private. Among first-party crates the module depends only on `rocm_core` and `rocm_engine_protocol`, and nothing in it calls back into `serve_cmd.rs`. Its tests cover each override on its own. - [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) — full domain extraction. - [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`stop_recorded_processes`, which hands the recorded PIDs to the routine [`proc_lifecycle.rs`](../crates/rocm-core/src/proc_lifecycle.rs) shares with `rocmd`, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which passes it to [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `run_attached_service` as the stop for a Ctrl-C, the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. This module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which does not import it back. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. - [`serve_port.rs`](../apps/rocm/src/serve_port.rs) — full domain extraction: the central `rocm serve` port policy (automatic vs explicit request, reservation by live managed services, OS preflight lease), used by [`service_launch.rs`](../apps/rocm/src/service_launch.rs); the CLI restart path shares the `rocm-core` lease primitives (`lease_loopback_port_or_bind_error`, `is_canonical_loopback_host`) rather than this module. -- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. +- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` and `rocm logs --service ` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. - [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, since split into a directory of child modules, owns no code of its own beyond its declarations: spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind, divided among the children listed below. The file declares the children, a `test_support` module compiled only for tests (the fixtures more than one child's tests share), and `pub(crate) use` re-exports of the items other modules call, so [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs) and [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) still reach them as `crate::service_launch::name`; [`main.rs`](../apps/rocm/src/main.rs) does not. An item that sibling children share is `pub(super)` and reached through `super::`; everything else is private to the child that owns it. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path. This module does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it: `run_attached_service` takes the stop for a Ctrl-C as a parameter, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) passes `run_internal_sandbox_tool`. - [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: bind-host validation and the endpoint API key policy. `validate_bind_host` refuses a non-loopback host without `--allow-public-bind` (`is_loopback_host` is the inverse of `rocm_engine_protocol::is_public_bind_host`, which `rocmd` shares); `resolve_endpoint_auth` decides the key a launch uses (none for a loopback bind unless a key is required, otherwise the supplied key trimmed or a generated one, refusing an empty key or one with a control character); `drop_orphaned_endpoint_key_on_already_running` clears a key staged for an attempt that spawned nothing; `ensure_public_bind_engine_supported` refuses a public bind with Windows managed Lemonade, whose key cannot be enforced; and `ensure_public_service_has_endpoint_key` is the guard a respawn runs so a public (or `--require-api-key`) service whose key file is gone is not reopened without authentication. All six are `pub(crate)` and re-exported by the parent: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads `ensure_public_service_has_endpoint_key`, [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) reads `is_loopback_host`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads the rest. - [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the part of a managed launch that decides before anything spawns. `resolve_managed_port_in_transaction` refreshes liveness, answers an equivalent live service before leasing any port (`ManagedPortDecision`), and otherwise leases a concrete port against both the ports live records still reserve (`live_managed_services`, `reserved_service_ports`) and the real OS through [`serve_port.rs`](../apps/rocm/src/serve_port.rs); a caller must hold the launch lock across it and the record write that follows. `abandon_failed_managed_launch` undoes a failed attempt before that lock is released: a spawned child is terminated first, through the same `rocm_core::terminate_recorded_pid_within` routine a stop uses (so on Windows its descendants are taken down too), and its record kept, marked stopped, while an attempt that never spawned has every artifact removed. A spawned child whose start token was not captured, on a host that records tokens (Linux and Windows, which records the process creation time), is not signalled (`launch_cleanup_refusal`): it was gone before it could be captured and its pid may belong to another process. `managed_service_reuse_refusal` (with `ManagedReuseRefusal`) is the one answer to whether a live matching service can serve a request (recipe, `--require-api-key`, device policy), and `no_usable_gpu_refusal` words the no-usable-GPU refusal once for `serve()`'s pre-flight and for the spawn. `managed_service_reuse_refusal` and `no_usable_gpu_refusal` are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), and `ManagedReuseRefusal` is `pub(crate)` too because the first returns it (only [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) matches on it); `ManagedPortDecision`, `live_managed_services`, `resolve_managed_port_in_transaction` and `abandon_failed_managed_launch` are `pub(super)` for [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs). diff --git a/docs/manual-testing.md b/docs/manual-testing.md index a2eb7dd454..eb63754ab8 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -352,6 +352,16 @@ Expected result: - The OpenAI-compatible endpoint answers a simple chat request. - `rocm services list` shows an `anthropic endpoint:` URL, and `POST /v1/messages` with `{"model":"x","max_tokens":16,"messages":[{"role":"user","content":"Hello"}]}` returns an Anthropic `message` that names the served model. The same request with `"stream":true` answers with `text/event-stream` (`message_start` through `message_stop`, text in `content_block_delta` events). A request with `tools` and `"tool_choice":{"type":"tool","name":""}` returns a `tool_use` content block naming that tool with its arguments and `"stop_reason":"tool_use"`; sending the call back as a `tool_result` block gets the model's answer. Combining `"stream":true` with `tools` is refused with an `invalid_request_error`. +The same model without `--foreground` or `--verbose` is the interactive +summary. While the server starts, the status line follows the engine (a +download percent, a load line, or a wait for the model to serve) and keeps +updating past the first minute. It does not finish with +`Deployment summary (not ready yet)` while the server is still loading. A +ready server then prints the deployment summary. If that wait fails and the +process is still up, the message names `rocm logs --service ` and +`rocm services stop --yes`, and `rocm services` still lists the +server. + If the llama.cpp server exits at start with an access violation (`0xC0000005` in `MSVCP140.dll`), the Visual C++ runtime on the machine is probably older than 14.40. Run `rocm diagnose` and expect `fix-21-msvc-redist-outdated`. `rocm fix @@ -480,6 +490,20 @@ Expected result: and how to free it (`rocm services stop --yes`), without spawning an engine process or writing a new record. +The default command, without `--foreground` or `--verbose`, leaves the server +in the background. While it starts, the status line follows the engine: a +download percent, a load line, or a wait for the model to serve. A model that +is still loading after 45 seconds (the old wait, which piped output still +uses) keeps that line; it is not replaced by a finished "not ready yet" +summary. The wait fails with a non-zero exit only when the engine fails or +exits, a download percent stops for 3 minutes, the port stays closed with no +engine log for 2 minutes, a load line stops changing for 5 minutes, or 30 +minutes pass. When the endpoint answers, the +command prints the deployment summary and returns to the shell. If the wait +fails and the process is still up, the message names `rocm logs --service +` and `rocm services stop --yes`, and `rocm services` +still lists the server. + ### Windows display driver check On a Windows machine with an AMD GPU, check what the CLI says about the display diff --git a/docs/testing.md b/docs/testing.md index 408eb9fd0d..4cae57726f 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1985,6 +1985,34 @@ cargo test -p rocm --bin rocm serve_engine_selection These tests verify that explicit/configured engines still win and shared model recipes only choose a preferred engine when no user/default engine is set. +Interactive `rocm serve` (a terminal, without `--verbose` or `--foreground`) +keeps a status line through download, load, and health until the server can +serve or that wait fails. A download that is still moving is not cut off at +45 seconds, and a failure names `rocm logs --service ` and +`rocm services stop --yes` instead of printing +`Deployment summary (not ready yet)`. The wait fails when the engine fails or +exits, a download percent stops for 3 minutes (`DOWNLOAD_STALL`), the port +stays closed with no log for 2 minutes (`SILENT_UNREACHABLE`), a load line +stops changing for 5 minutes (`SILENT_LOAD`), or after 30 minutes +(`rocm_core::SERVE_STARTUP_DEADLINE`). Engine log lines are masked of +credentials before they are shown (`redact_log_line`). `rocmd` does not +restart a live `running` or `starting` service younger than that deadline. +Piped output keeps the 45-second status budget. Like `--wait`, the +interactive wait first prints the service id, `rocm logs --service + --follow` and a public bind's key (`announces_wait` in +[start.rs](../apps/rocm/src/service_launch/start.rs)), since it can fail +without printing a summary; `--wait` keeps its own wait, with no stall limit +or deadline. The status line is a TTY spinner, and the harness does not allocate a +PTY, so this is not a default-lane Gherkin scenario (a live engine would also +be `@requires-gpu`). `decide_serve_startup` and `wait_for_serve_startup` in +[serve_progress.rs](../apps/rocm/src/serve_progress.rs) are the state matrix, +covered by: + +```bash +cargo test -p rocm --bin rocm serve_progress +cargo test -p rocmd recovery_leaves_a_live +``` + Public-endpoint authentication focused tests: ```bash diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 1786d007e2..467b12a0e7 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -1,5 +1,13 @@ Feature: Model serving + # Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows + # download, load, and health on the status line until the server is ready or + # the startup wait fails. Seeing that line requires a TTY spinner and a live + # engine log, so it is not a default-lane scenario: the harness does not + # allocate a PTY, and a real serve is `@requires-gpu`. The state matrix is + # covered without a GPU by `decide_serve_startup` and `wait_for_serve_startup` + # in apps/rocm/src/serve_progress.rs. + # Per-platform expectations (pass / xfail / skip) are resolved at runtime from # the host capability probe + expectations.toml, keyed by the @id tag — not by # @expected-failure tags. Engine requirements are declared via @requires-engine diff --git a/xtask/src/ascii_output_contract.rs b/xtask/src/ascii_output_contract.rs index 18827f11d6..28e994f965 100644 --- a/xtask/src/ascii_output_contract.rs +++ b/xtask/src/ascii_output_contract.rs @@ -183,8 +183,13 @@ mod tests { SPINNER_LABEL, ), ( - "apps/rocm/src/serve_cmd.rs", - "Starting {model} on {selected_engine}…", + "apps/rocm/src/serve_progress.rs", + "Starting {model} on {engine}…", + SPINNER_LABEL, + ), + ( + "apps/rocm/src/serve_progress.rs", + "Waiting for {model} to serve…", SPINNER_LABEL, ), ( From 06dfc794e53bc9a4068c05c8d5f725bf2d6a1d94 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Tue, 10 Nov 2026 14:41:08 +0200 Subject: [PATCH 02/14] fix(serve): space inference probes and settle an ended startup Once the endpoint lists the model, the interactive wait sent an 8-second inference probe on every 250 ms poll for up to 30 minutes. The probe now runs at once, then backs off from 1 to 15 seconds (INFERENCE_PROBE_RETRY_INTERVAL), timed from when the last one returned. Piped output keeps its per-poll probing within the 45-second budget. An engine failure or process exit during the interactive wait was recorded `failed` and handed to the background helper, which a launch the CLI has already reported as failed must not be. It is now settled the way a `--wait` that saw the server exit is (`stopped`, or the engine's own `failed`), with the same memory guidance and an audit event, and only a stall or the deadline, which leave the server running, start the helper. The attached path printed `rocm logs `, which `rocm logs` reads as a free-text search. Every logs hint now goes through service_logs_hint. rocmd's two endpoint-unreachable checks share one guarded helper, and the dead --wait parameter of wait_for_record_ready is gone. Signed-off-by: Roman Inflianskas --- README.md | 6 +- apps/rocm/src/cli.rs | 5 +- apps/rocm/src/engine_runtime_select.rs | 58 ++- apps/rocm/src/serve_cmd.rs | 51 +- apps/rocm/src/serve_progress.rs | 439 ++++++++++++++---- apps/rocm/src/service_launch/attach.rs | 36 +- apps/rocm/src/service_launch/readiness.rs | 104 ++++- apps/rocm/src/service_launch/start.rs | 214 ++++++++- apps/rocmd/src/watchers/server_recover.rs | 34 +- docs/architecture.md | 6 +- docs/testing.md | 26 +- .../features/model_serving.feature | 7 +- 12 files changed, 783 insertions(+), 203 deletions(-) diff --git a/README.md b/README.md index 55e1a3e232..2f7fc480b3 100644 --- a/README.md +++ b/README.md @@ -1168,7 +1168,9 @@ the message names `rocm logs --service ` and `rocm services stop --yes`, and the background helper does not restart a live server for being slow until those 30 minutes pass. Engine log lines shown on the status line have credentials (the endpoint key, `--api-key` -and token values, `hf_` tokens) masked. Piped output (not a terminal) keeps +and token values, `hf_` tokens) masked. Once the server lists the model, the wait +checks whether it can answer with a one-token request, first right away and +then at growing intervals up to every 15 seconds. Piped output (not a terminal) keeps the 45-second wait and reports the server as it stood then. Control returns to your shell with the server still running; manage it later with `rocm services` (below). @@ -1190,7 +1192,7 @@ can: `--wait` keeps `rocm serve` waiting until the server answers requests or its engine gives up (vLLM and AIM after 5 minutes by default; raise it with `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), with -no stall limit or deadline, instead of the default wait above; if the server exits first (crashes, is killed, or is stopped with +no CLI-side stall limit or deadline, instead of the default wait above; if the server exits first (crashes, is killed, or is stopped with `rocm services stop`), it fails with an error that points at the server's logs instead of waiting forever. Ctrl-C ends the wait and leaves the server starting in the background; a generated API key (public bind) is printed when the wait diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 8b5945fb87..8db47837f7 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -430,7 +430,8 @@ rocm model --verbose" /// Picks an engine and ROCm runtime automatically unless overridden. An /// Anthropic `/v1/messages` endpoint is published beside the OpenAI /// one (see `rocm services list`). By default the - /// server runs as a managed background service. While it starts, the status line + /// server runs as a managed background service. In a terminal it first prints the + /// service id and how to follow its log; while it starts, the status line /// follows what the engine is doing (download, load, or the latest log line) until /// the server is ready or that wait fails. A ready server then prints a deployment /// summary (status, endpoint, model, and smoke-test metrics). A wait that fails @@ -524,7 +525,7 @@ rocm serve qwen --verbose --device gpu_required")] /// Keep waiting until the server answers requests or its engine gives up /// (vLLM and AIM after 5 minutes by default, raised with /// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), - /// with no stall limit or deadline. Without it, a terminal gives up when + /// with no CLI-side stall limit or deadline. Without it, a terminal gives up when /// startup progress stalls or after 30 minutes, and piped output returns /// `starting` after 45 seconds. Also waits on a matching server that is already starting. Fails with a /// pointer to the logs if the server exits first. Cannot be diff --git a/apps/rocm/src/engine_runtime_select.rs b/apps/rocm/src/engine_runtime_select.rs index 879e9a2b8d..c003d9febe 100644 --- a/apps/rocm/src/engine_runtime_select.rs +++ b/apps/rocm/src/engine_runtime_select.rs @@ -72,9 +72,9 @@ //! `service_launch.rs` and `service_control.rs`'s `restart_internal_managed_service_with`. //! `wait_for_service_http_ready` is `pub(crate)`, read from `service_control.rs`'s //! `restart_internal_managed_service_with`; `wait_for_service_http_ready_with_progress` -//! is `pub(crate)`, read from `service_launch.rs`. `probe_service_http_readiness` -//! is `pub(crate)`, read from `serve_progress.rs`'s `wait_for_serve_startup` and -//! from `wait_for_service_http_ready_with_progress`. `http_post_local_service_json` +//! is `pub(crate)`, read from `service_launch.rs`. `service_http_lists_model` and +//! `service_http_inference_answers`, the two halves of its readiness probe, are +//! `pub(crate)`, read from `serve_progress.rs`'s `wait_for_serve_startup`. `http_post_local_service_json` //! is `pub(crate)`, read from `service_control.rs`'s `unload_lemonade_service_model`. //! `service_model_names_match` is `pub(crate)`, read from `chat/turn.rs`. //! `service_http_readiness_paths`, `http_get_local_service`, @@ -369,17 +369,36 @@ pub(crate) fn wait_for_service_http_ready_with_progress( /// request comes back, [`EndpointReadiness::Listing`] when the model is /// advertised but that request does not, otherwise unreachable. /// -/// Shared with [`crate::serve_progress::wait_for_serve_startup`] so the -/// interactive status line and this wait cannot disagree about "ready". -pub(crate) fn probe_service_http_readiness( +/// Probes inference on every call. [`crate::serve_progress::wait_for_serve_startup`] +/// spaces those probes out instead, through [`service_http_lists_model`] and +/// [`service_http_inference_answers`], the same two checks, so the interactive +/// status line and this wait cannot disagree about "ready". +fn probe_service_http_readiness( engine: &str, host: &str, port: u16, canonical_model_id: &str, endpoint_api_key: Option<&str>, ) -> EndpointReadiness { - let endpoint = format_http_base_url(host, port); - let listed = service_http_readiness_paths(engine).iter().any(|path| { + if !service_http_lists_model(engine, host, port, canonical_model_id, endpoint_api_key) { + return EndpointReadiness::Unreachable; + } + if service_http_inference_answers(host, port, canonical_model_id, endpoint_api_key) { + return EndpointReadiness::Serving; + } + EndpointReadiness::Listing +} + +/// The cheap half of a readiness probe: the endpoint answers and advertises +/// the model (a short GET per readiness path). +pub(crate) fn service_http_lists_model( + engine: &str, + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> bool { + service_http_readiness_paths(engine).iter().any(|path| { http_get_local_service( host, port, @@ -390,21 +409,24 @@ pub(crate) fn probe_service_http_readiness( .is_ok_and(|(status, body)| { service_http_readiness_response_ready(engine, path, status, &body, canonical_model_id) }) - }); - if !listed { - return EndpointReadiness::Unreachable; - } - if rocm_core::openai_chat_completion_probe( - &endpoint, + }) +} + +/// The expensive half: a real one-token chat request, which can wait up to +/// [`rocm_core::INFERENCE_PROBE_TIMEOUT`] on a model that is still loading. +pub(crate) fn service_http_inference_answers( + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> bool { + rocm_core::openai_chat_completion_probe( + &format_http_base_url(host, port), canonical_model_id, endpoint_api_key, rocm_core::INFERENCE_PROBE_TIMEOUT, ) .unwrap_or(false) - { - return EndpointReadiness::Serving; - } - EndpointReadiness::Listing } fn service_http_readiness_paths(engine: &str) -> &'static [&'static str] { diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index fb2ea6724e..96ea5130a5 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -1178,7 +1178,8 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { spinner.borrow_mut().clear(); eprintln!( "Waiting for {waiting_on}; Ctrl-C stops waiting, the server keeps running. \ - Follow: rocm logs --service {waiting_on} --follow" + Follow: {} --follow", + crate::service_hints::service_logs_hint(waiting_on) ); if let Some(config) = wait_begin_client_config( waiting_on, @@ -1434,19 +1435,23 @@ fn collect_serve_notes( /// left a launched server behind. Returns the warning to print when the /// helper itself would not start. /// -/// Only a startup stop ([`serve_progress::ServeStartupFailed`]) is known to -/// leave a process that may still be running. Other errors either come before -/// the child exists (the claiming record is written first, as `starting`, and -/// `rocmd` treats it as any stale start), or from a child that already exited -/// or an I/O error writing its record, which did not start the helper before -/// this wait existed either. The startup failure stays the error the user -/// reads; a helper failure is a warning beside it. +/// Only a startup stop that left the server running +/// ([`serve_progress::ServeStartupFailed`] with `server_running`) leaves a +/// process to supervise. A stop on an engine that failed or a process that +/// exited has already been settled and reported as failed, so the CLI does not +/// start the helper for it. Other errors either come before the child +/// exists (the claiming record is written first, as `starting`, and `rocmd` +/// treats it as any stale start), or from a child that already exited or an +/// I/O error writing its record, which did not start the helper before this +/// wait existed either. The startup failure stays the error the user reads; a +/// helper failure is a warning beside it. fn supervise_after_launch_error( error: &anyhow::Error, start_helper: impl FnOnce() -> Result<()>, ) -> Option { error .downcast_ref::() + .filter(|stop| stop.server_running) .and_then(|_| start_helper().err()) .map(|helper| { format!( @@ -2230,9 +2235,10 @@ mod tests { fn a_startup_stop_starts_supervision_and_keeps_its_own_error() { use std::cell::Cell; let stop = || -> anyhow::Error { - serve_progress::ServeStartupFailed( - "qwen on vllm stopped making download progress".to_owned(), - ) + serve_progress::ServeStartupFailed { + message: "qwen on vllm stopped making download progress".to_owned(), + server_running: true, + } .into() }; // Row: startup stop, helper starts. Helper called, nothing to warn. @@ -2246,18 +2252,29 @@ mod tests { // Row: startup stop, helper fails. The warning names the helper // error; the startup failure is still what the caller returns. - let error = stop(); - let warning = super::supervise_after_launch_error(&error, || anyhow::bail!("no daemon")) + let warning = super::supervise_after_launch_error(&stop(), || anyhow::bail!("no daemon")) .expect("a helper that failed is reported"); assert!( warning.starts_with("warning: the background helper did not start"), "{warning}" ); assert!(warning.contains("no daemon"), "{warning}"); - assert_eq!( - error.to_string(), - "qwen on vllm stopped making download progress" - ); + + // Row: the server ended (engine failed, process exited). It was + // settled and reported as failed, so no helper is started to restart + // it behind that error. + let called = Cell::new(false); + let ended: anyhow::Error = serve_progress::ServeStartupFailed { + message: "qwen on vllm failed: the server process exited".to_owned(), + server_running: false, + } + .into(); + let warning = super::supervise_after_launch_error(&ended, || { + called.set(true); + Ok(()) + }); + assert!(!called.get(), "an ended launch starts no helper"); + assert_eq!(warning, None); // Row: any other launch error. No helper. let called = Cell::new(false); diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index deeac7f569..830611fa54 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -38,7 +38,7 @@ use std::time::{Duration, Instant}; use rocm_core::EndpointReadiness; -use crate::engine_runtime_select::probe_service_http_readiness; +use crate::engine_runtime_select::{service_http_inference_answers, service_http_lists_model}; use crate::service_hints::{service_logs_hint, service_stop_hint}; /// How long a download percent (or line) may stay unchanged before it is hung. @@ -74,6 +74,74 @@ pub(crate) struct ServeStartupPolicy { /// Off for the piped path so a serve that exits in CI still returns a /// readiness status. pub watch_engine: bool, + /// How far apart inference probes are once the endpoint lists the model. + pub inference_backoff: InferenceProbeBackoff, +} + +/// Spacing between inference probes while the endpoint lists the model but +/// cannot answer yet. +/// +/// Listing is a short GET; the inference probe is a real generation request +/// that can hold the engine for up to [`rocm_core::INFERENCE_PROBE_TIMEOUT`]. +/// The first probe runs as soon as the model is listed, then the gap starts at +/// `first` and doubles up to `max`. A zero `first` probes every poll. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct InferenceProbeBackoff { + pub first: Duration, + pub max: Duration, +} + +impl InferenceProbeBackoff { + /// Probe on every poll: the piped status budget, bounded at 45 seconds. + pub(crate) const EVERY_POLL: Self = Self { + first: Duration::ZERO, + max: Duration::ZERO, + }; +} + +/// First gap between inference probes on the interactive wait. +pub(crate) const INFERENCE_PROBE_FIRST_GAP: Duration = Duration::from_secs(1); + +/// When the next inference probe may run. See [`InferenceProbeBackoff`]. +#[derive(Debug)] +pub(crate) struct InferenceProbeThrottle { + backoff: InferenceProbeBackoff, + gap: Duration, + next_at: Option, +} + +impl InferenceProbeThrottle { + pub(crate) const fn new(backoff: InferenceProbeBackoff) -> Self { + Self { + backoff, + gap: backoff.first, + next_at: None, + } + } + + /// Readiness for one poll. `probe` runs only when the model is listed and + /// the gap since the last failed probe has passed; the next gap is counted + /// from when that probe returned, read from `clock`. + pub(crate) fn readiness( + &mut self, + listed: bool, + clock: &mut dyn FnMut() -> Instant, + probe: impl FnOnce() -> bool, + ) -> EndpointReadiness { + if !listed { + return EndpointReadiness::Unreachable; + } + if self.next_at.is_some_and(|at| clock() < at) { + return EndpointReadiness::Listing; + } + if probe() { + return EndpointReadiness::Serving; + } + let returned = clock(); + self.next_at = Some(returned.checked_add(self.gap).unwrap_or(returned)); + self.gap = self.gap.saturating_mul(2).min(self.backoff.max); + EndpointReadiness::Listing + } } /// The clocks [`decide_serve_startup`] compares against. @@ -98,6 +166,10 @@ impl ServeStartupPolicy { poll: POLL, stop_on_give_up: true, watch_engine: true, + inference_backoff: InferenceProbeBackoff { + first: INFERENCE_PROBE_FIRST_GAP, + max: rocm_core::INFERENCE_PROBE_RETRY_INTERVAL, + }, } } @@ -113,6 +185,7 @@ impl ServeStartupPolicy { poll: POLL, stop_on_give_up: false, watch_engine: false, + inference_backoff: InferenceProbeBackoff::EVERY_POLL, } } } @@ -177,15 +250,19 @@ pub(crate) struct ServeStartupStop { /// The error `start_managed_service` returns when the wait stopped. /// -/// The process is recorded (and may still be running), unlike a spawn -/// failure, so the caller starts supervision for this error and no other. -/// `Display` is [`ServeStartupStop::user_message`]. +/// `Display` is [`ServeStartupStop::user_message`], plus memory guidance when +/// the server did not start. Only a stop that left the server running +/// (`server_running`) has a process for the caller to supervise; a spawn +/// failure or a server that ended has none. #[derive(Debug)] -pub(crate) struct ServeStartupFailed(pub String); +pub(crate) struct ServeStartupFailed { + pub message: String, + pub server_running: bool, +} impl std::fmt::Display for ServeStartupFailed { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.write_str(&self.0) + f.write_str(&self.message) } } @@ -483,36 +560,30 @@ pub(crate) fn decide_serve_startup( ServeStartupDecision::Continue(view) } -/// The status written to the service record when the wait stops. -/// -/// Engine failure and a dead process are `failed`, which recovery is supposed -/// to restart. Every other reason leaves the process alive and is `running`: -/// [`decide_serve_startup`] returns `ProcessExited` before it checks any -/// stall or the deadline, so a stall or deadline stop always has -/// `process_alive`. A live `running` record is what -/// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one`, and a -/// live `starting` one what -/// `recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline`, -/// leaves alone until [`rocm_core::SERVE_STARTUP_DEADLINE`], so `rocmd` does -/// not kill the server [`ServeStartupStop::user_message`] just said is still -/// running. -pub(crate) const fn recorded_status_after_stop(stop: &ServeStartupStop) -> &'static str { - match stop.reason { - ServeStartupStopReason::EngineFailed | ServeStartupStopReason::ProcessExited => "failed", - ServeStartupStopReason::DownloadStalled - | ServeStartupStopReason::Silent - | ServeStartupStopReason::SilentLoad - | ServeStartupStopReason::Deadline => { - debug_assert!( - stop.process_alive, - "a stall or deadline stop is only reached with a live process" - ); - "running" - } +impl ServeStartupStop { + /// The wait gave up on a server that is still up: a stall or the deadline. + /// + /// Engine failure and a dead process end the launch instead. Those are + /// settled like a `--wait` that saw the server exit: a dead process is + /// recorded `stopped`; a record the engine already marked `failed` keeps + /// that status, and `rocmd`'s recovery treats it like any `failed` service + /// (its message names the stop command while its process is still up); a + /// stop another terminal recorded is left alone. The CLI does not start the + /// background helper for either. A server left running is recorded + /// `running`, which `rocmd` leaves alone until + /// [`rocm_core::SERVE_STARTUP_DEADLINE`] (see + /// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one`), + /// so it does not kill the server this message says is still running. + /// [`decide_serve_startup`] returns `ProcessExited` before it checks any + /// stall or the deadline, so those stops always have `process_alive`. + pub(crate) const fn leaves_server_running(&self) -> bool { + self.process_alive + && !matches!( + self.reason, + ServeStartupStopReason::EngineFailed | ServeStartupStopReason::ProcessExited + ) } -} -impl ServeStartupStop { /// Text the CLI prints when the wait stops. /// /// A log line is quoted only when [`Self::label`] is that line. The @@ -546,16 +617,23 @@ impl ServeStartupStop { } message.push('\n'); let logs = service_logs_hint(service_id); - if self.process_alive { + if self.leaves_server_running() { let stop = service_stop_hint(service_id); let _ = write!( message, "The server is still running (service {service_id}). Follow it with `{logs}`, or stop it with `{stop}`." ); + } else if self.process_alive { + // The engine said failed but its process has not exited yet. + let stop = service_stop_hint(service_id); + let _ = write!( + message, + "Its process is still up (service {service_id}). The log is `{logs}`; `{stop}` stops it." + ); } else { let _ = write!( message, - "The server process is gone (service {service_id}). The log is `{logs}`." + "The server did not start (service {service_id}). The log is `{logs}`." ); } message @@ -566,8 +644,10 @@ impl ServeStartupStop { /// /// `observe` returns the model name the probe should ask for (a container /// engine may not know it until it starts) and the latest engine/process -/// sample. The same HTTP probe as the historical readiness wait is used, so -/// "serving" still means an inference request came back. +/// sample. The same HTTP checks as the historical readiness wait are used, so +/// "serving" still means an inference request came back. Once the model is +/// listed, the inference request is spaced by the policy's +/// [`InferenceProbeBackoff`] rather than sent every poll. #[allow(clippy::too_many_arguments)] pub(crate) fn wait_for_serve_startup( engine_name: &str, @@ -583,6 +663,7 @@ pub(crate) fn wait_for_serve_startup( let mut progress_at = started; let mut token = String::new(); let mut best = EndpointReadiness::Unreachable; + let mut throttle = InferenceProbeThrottle::new(policy.inference_backoff); // One observe, one probe, one decision and at most one status-line update // per poll, so the spinner advances once per tick. @@ -595,14 +676,12 @@ pub(crate) fn wait_for_serve_startup( // A failed engine or a dead process is decided without waiting on a // probe that cannot succeed. if !signals.engine_failed && signals.process_alive { - match probe_service_http_readiness( - engine_name, - host, - port, - &probe_model, - endpoint_api_key, - ) { - EndpointReadiness::Serving => return Ok(EndpointReadiness::Serving), + let listed = + service_http_lists_model(engine_name, host, port, &probe_model, endpoint_api_key); + match throttle.readiness(listed, &mut Instant::now, || { + service_http_inference_answers(host, port, &probe_model, endpoint_api_key) + }) { + EndpointReadiness::Serving => best = EndpointReadiness::Serving, EndpointReadiness::Listing => best = EndpointReadiness::Listing, EndpointReadiness::Unreachable => {} } @@ -1113,7 +1192,7 @@ mod tests { assert!(!stop.process_alive); let message = stop.user_message("qwen", "vllm", "svc-1"); assert!(message.contains("the server process exited"), "{message}"); - assert!(message.contains("The server process is gone"), "{message}"); + assert!(message.contains("The server did not start"), "{message}"); assert!( !message.contains("still running"), "a dead process must not be described as still running: {message}" @@ -1210,9 +1289,129 @@ mod tests { poll: Duration::from_millis(20), stop_on_give_up: true, watch_engine: true, + inference_backoff: InferenceProbeBackoff::EVERY_POLL, } } + /// The interactive wait's inference probe, under a clock that moves only + /// when the test says: a model listed but never answering is probed at 0, + /// 1, 3, 7 and 15 seconds, then every 15, never once per 250 ms poll. Each + /// probe takes the full 8-second timeout, and the next gap counts from + /// when it returned. + #[test] + fn inference_probe_backs_off_to_the_retry_interval_after_listing() { + use std::cell::Cell; + let start = Instant::now(); + let now = Cell::new(start); + let mut clock = || now.get(); + let mut throttle = + InferenceProbeThrottle::new(ServeStartupPolicy::interactive().inference_backoff); + let mut probed_at = Vec::new(); + // Ten minutes of 250 ms polls. + while now.get() - start < Duration::from_mins(10) { + let readiness = throttle.readiness(true, &mut clock, || { + probed_at.push((now.get() - start).as_secs()); + now.set(now.get() + rocm_core::INFERENCE_PROBE_TIMEOUT); + false + }); + assert_eq!(readiness, EndpointReadiness::Listing); + now.set(now.get() + POLL); + } + let gaps: Vec = probed_at + .windows(2) + .map(|pair| pair[1] - pair[0] - rocm_core::INFERENCE_PROBE_TIMEOUT.as_secs()) + .collect(); + assert_eq!(&gaps[..5], &[1, 2, 4, 8, 15], "probed at {probed_at:?}"); + assert!( + gaps[4..].iter().all(|gap| *gap == 15), + "the gap must stay at the retry interval: {gaps:?}" + ); + // 2400 polls; one probe each would be 2400 eight-second requests. + assert!(probed_at.len() < 30, "probed {} times", probed_at.len()); + } + + /// Not listed, the probe never runs; listed, the first probe is not + /// delayed, and a probe that answers is ready at once. + #[test] + fn inference_probe_runs_at_once_when_listed_and_not_before() { + let now = Instant::now(); + let mut clock = || now; + let mut throttle = + InferenceProbeThrottle::new(ServeStartupPolicy::interactive().inference_backoff); + assert_eq!( + throttle.readiness(false, &mut clock, || panic!("not listed, no probe")), + EndpointReadiness::Unreachable + ); + assert_eq!( + throttle.readiness(true, &mut clock, || true), + EndpointReadiness::Serving + ); + } + + /// The same backoff through the real wait: an endpoint that lists the model + /// and refuses every chat request gets a handful of inference requests over + /// a second of 10 ms polls, not one per poll. + #[test] + fn wait_spaces_inference_probes_while_the_model_is_listed() { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let chat_requests = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&chat_requests); + let _server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + counted.fetch_add(1, Ordering::SeqCst); + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } else { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + let mut policy = short_policy(Duration::from_secs(1), Duration::from_secs(30)); + policy.poll = Duration::from_millis(10); + policy.inference_backoff = InferenceProbeBackoff { + first: Duration::from_millis(100), + max: Duration::from_millis(200), + }; + let mut polls = 0_usize; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &policy, + &mut || { + polls += 1; + ("qwen".to_owned(), alive()) + }, + &mut |_| {}, + ); + let stop = result.expect_err("a model that never answers hits the deadline"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + let probes = chat_requests.load(Ordering::SeqCst); + // 0, 100, 300, 500, 700, 900 ms: at most a few more under a slow host, + // where an unthrottled wait sends one per poll. + assert!( + (1..=8).contains(&probes), + "{probes} inference requests over {polls} polls" + ); + } + #[test] fn wait_follows_a_download_until_the_server_serves() { let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); @@ -1237,8 +1436,11 @@ mod tests { } else { ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) } - } else { + } else if inference_ready { ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + } else { + // Still downloading: the endpoint does not list the model. + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) }; let _ = write!( stream, @@ -1248,7 +1450,7 @@ mod tests { } }); - let mut labels = Vec::new(); + let mut views = Vec::new(); let result = wait_for_serve_startup( "vllm", "qwen", @@ -1266,7 +1468,7 @@ mod tests { }, ) }, - &mut |view| labels.push(view.label.clone()), + &mut |view| views.push(view.clone()), ); drop(server); @@ -1276,10 +1478,11 @@ mod tests { "the wait returned before the server could serve" ); assert!( - labels + views .iter() - .any(|label| label == "Downloading model.safetensors: 40%"), - "the status line must show the download while the wait continues: {labels:?}" + .any(|view| view.phase == ServeStartupPhase::Downloading + && view.label == "Downloading model.safetensors: 40%"), + "the status line must show the download while the wait continues: {views:?}" ); } @@ -1383,59 +1586,97 @@ mod tests { port } + /// Piped output returns a readiness status at the budget even when the + /// engine reports failure or the process is gone: that is the historical + /// 45-second contract scripts and CI read, and it has no stop. #[test] - fn stop_records_failed_only_for_a_dead_server() { - let stop = |reason, process_alive, readiness| ServeStartupStop { + fn piped_wait_ignores_engine_failure_and_a_dead_process() { + let port = { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + drop(listener); + port + }; + let mut policy = ServeStartupPolicy::status_budget(Duration::from_millis(200)); + policy.poll = Duration::from_millis(20); + let mut polls = 0; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &policy, + &mut || { + polls += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: true, + process_alive: false, + log_line: None, + }, + ) + }, + &mut |_| {}, + ); + assert_eq!(result, Ok(EndpointReadiness::Unreachable)); + assert!( + polls > 1, + "the piped wait must run to its budget, not stop at once" + ); + } + + #[test] + fn only_a_stall_or_the_deadline_leaves_the_server_running() { + let stop = |reason, process_alive| ServeStartupStop { phase: ServeStartupPhase::Loading, reason, label: String::new(), log_line: None, process_alive, - readiness, + readiness: EndpointReadiness::Unreachable, }; - assert_eq!( - recorded_status_after_stop(&stop( - ServeStartupStopReason::EngineFailed, - true, - EndpointReadiness::Unreachable - )), - "failed" - ); - assert_eq!( - recorded_status_after_stop(&stop( - ServeStartupStopReason::ProcessExited, - false, - EndpointReadiness::Listing - )), - "failed" - ); - // A live server that stalled is still up: the record must not say - // failed while the message tells the user it is still running, and - // must not say `starting`, which rocmd restarts once it is stale. - assert_eq!( - recorded_status_after_stop(&stop( - ServeStartupStopReason::DownloadStalled, - true, - EndpointReadiness::Unreachable - )), - "running" - ); - assert_eq!( - recorded_status_after_stop(&stop( - ServeStartupStopReason::Silent, - true, - EndpointReadiness::Unreachable - )), - "running" - ); - assert_eq!( - recorded_status_after_stop(&stop( - ServeStartupStopReason::Deadline, - true, - EndpointReadiness::Listing - )), - "running" - ); + // The engine said failed, or the process is gone: the launch is over, + // so the message does not say the server is still running. (That the + // CLI does not supervise it is + // `a_startup_stop_starts_supervision_and_keeps_its_own_error`.) + for (reason, alive) in [ + (ServeStartupStopReason::EngineFailed, true), + (ServeStartupStopReason::EngineFailed, false), + (ServeStartupStopReason::ProcessExited, false), + ] { + let stop = stop(reason, alive); + assert!(!stop.leaves_server_running(), "{reason:?}"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(!message.contains("still running"), "{reason:?}: {message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + // A failed engine whose process is still up gets the command that + // stops it; one that is gone does not. + assert_eq!( + message.contains(&service_stop_hint("svc-1")), + alive, + "{reason:?} alive={alive}: {message}" + ); + assert_eq!( + message.contains("Its process is still up"), + alive, + "{reason:?} alive={alive}: {message}" + ); + assert_eq!( + message.contains("The server did not start"), + !alive, + "{reason:?} alive={alive}: {message}" + ); + } + for reason in [ + ServeStartupStopReason::DownloadStalled, + ServeStartupStopReason::Silent, + ServeStartupStopReason::SilentLoad, + ServeStartupStopReason::Deadline, + ] { + assert!(stop(reason, true).leaves_server_running(), "{reason:?}"); + } } #[test] diff --git a/apps/rocm/src/service_launch/attach.rs b/apps/rocm/src/service_launch/attach.rs index d76a90e8d4..f12da3a5be 100644 --- a/apps/rocm/src/service_launch/attach.rs +++ b/apps/rocm/src/service_launch/attach.rs @@ -105,7 +105,7 @@ pub(crate) fn run_attached_service( println!(" service_id: {}", report.service_id); println!(" endpoint: {}", report.endpoint_url); println!(" status: {}", report.status); - println!(" logs: rocm logs {}", report.service_id); + println!("{}", logs_row("logs", &report.service_id)); println!( " stop: {}", service_hints::service_stop_hint(&report.service_id) @@ -169,7 +169,7 @@ pub(crate) fn run_attached_service( println!(" service_id: {service_id}"); println!(" endpoint: {endpoint}"); println!(" list: rocm services"); - println!(" logs: rocm logs {service_id}"); + println!("{}", logs_row("logs", &service_id)); println!(" stop: {}", service_hints::service_stop_hint(&service_id)); Ok(()) } @@ -189,12 +189,22 @@ pub(crate) fn run_attached_service( AttachOutcome::ServerExited => { println!("server process exited"); println!(" service_id: {service_id}"); - println!(" recent logs: rocm logs {service_id}"); + println!("{}", logs_row("recent logs", &service_id)); Ok(()) } } } +/// The ` {label}: rocm logs --service ` row of the attached path's +/// reports. A bare `rocm logs ` would be a free-text search, not that +/// service's log. +fn logs_row(label: &str, service_id: &str) -> String { + format!( + " {label}: {}", + service_hints::service_logs_hint(service_id) + ) +} + /// Restores cooked terminal mode when dropped, so [`stream_attached_logs`] leaves /// the terminal usable on every exit path (normal return, `?` error, or panic). struct RawModeGuard; @@ -433,6 +443,26 @@ fn finish_attached_stop( #[cfg(test)] mod tests { use super::*; + + /// Every `logs` row the attached path prints names a command that selects + /// that service's log. `rocm logs ` parses as a free-text search. + #[test] + fn attached_logs_rows_select_the_service_log() { + use clap::Parser as _; + for label in ["logs", "recent logs"] { + let row = logs_row(label, "svc-1"); + let command = row + .strip_prefix(&format!(" {label}: ")) + .expect("row starts with its label"); + let parsed = crate::cli::Cli::try_parse_from(command.split_whitespace()) + .expect("the printed command parses"); + let Some(crate::cli::Command::Logs { service, query, .. }) = parsed.command else { + panic!("`{command}` is not `rocm logs`"); + }; + assert_eq!(service.as_deref(), Some("svc-1"), "`{command}`"); + assert!(query.is_empty(), "`{command}` searches for {query:?}"); + } + } use crate::endpoint_keys; use crate::tests::test_paths; use rocm_core::ManagedServiceRecord; diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index 826b830183..c8befa5246 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -31,8 +31,9 @@ use rocm_engine_protocol::{DevicePolicy, ResolveModelResponse}; use std::fmt::Write as _; use std::time::Duration; -/// How long one readiness wait polls before `rocm serve` reports `starting`. -const READINESS_WAIT: Duration = Duration::from_secs(45); +/// How long one `--wait` readiness pass polls: the same budget after which a +/// piped `rocm serve` reports `starting`. +const READINESS_WAIT: Duration = crate::serve_progress::PIPED_SERVE_STATUS_BUDGET; /// `rocm serve --wait`: keep waiting past [`READINESS_WAIT`] (a first start /// downloads the model, which can take far longer) until the server answers or @@ -128,14 +129,43 @@ pub(super) fn died_during_wait( { return None; } + let message = format!( + "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `{}` for its output", + crate::service_hints::service_logs_hint(service_id) + ); + Some(anyhow!(settle_ended_launch( + paths, + service_id, + "exited during --wait", + message + ))) +} + +/// Settle a launch whose server ended before it became ready, record it in +/// the audit log (`why` says how it ended), and return `message` with the +/// memory guidance a plain `serve` would print for it. +/// +/// Shared by a `--wait` that saw the server exit and the interactive wait's +/// engine-failed or process-exited stop: the record is settled by +/// [`settle_exited_service`] (`stopped`, or the engine's own `failed`, never +/// over a stop already recorded), and a vLLM launch that ran out of VRAM gets +/// the same note either way. "starting" is the status the note keys on for +/// "never became ready", whatever the settled record now says. +pub(super) fn settle_ended_launch( + paths: &AppPaths, + service_id: &str, + why: &str, + mut message: String, +) -> String { settle_exited_service(paths, service_id); - let mut message = format!( - "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `rocm logs --service {service_id}` for its output" + crate::record_cli_audit_event( + paths, + "service", + "managed_service_launch", + "error", + format!("managed service ended before it became ready: {why}"), + Some(service_id), ); - // A plain `serve` adds the memory guidance for a vLLM launch that ran out of - // VRAM; the wait's failure is the same launch, so it gets the same note. - // "starting" is the status the note keys on for "never became ready", which is - // what a `--wait` death is, whatever the settled record now says. if let Ok(record) = ManagedServiceRecord::read_manifest(paths, service_id) { for note in crate::serve_summary::append_oom_serve_note( Vec::new(), @@ -147,7 +177,7 @@ pub(super) fn died_during_wait( let _ = write!(message, "\n note: {note}"); } } - Some(anyhow!(message)) + message } /// Record the outcome of a finished wait on the manifest as it is now, changing @@ -275,12 +305,11 @@ pub(super) fn simulate_oom_managed_launch( }) } -/// Probe `record`'s endpoint until it answers: once within the readiness -/// budget, or (with `wait_until_ready`) for as long as the server stays alive. +/// Probe `record`'s endpoint until it answers (`--wait`), for as long as the +/// server stays alive. Each pass is one [`READINESS_WAIT`]-long readiness wait. pub(super) fn wait_for_record_ready( paths: &AppPaths, record: &ManagedServiceRecord, - wait_until_ready: bool, endpoint_api_key: Option<&str>, on_wait_tick: &mut dyn FnMut(Duration), ) -> EndpointReadiness { @@ -299,14 +328,10 @@ pub(super) fn wait_for_record_ready( on_tick, ) }; - if wait_until_ready { - wait_past_readiness_budget( - || probe(&mut *on_wait_tick), - || service_still_live(paths, &record.service_id, service_pid_is_alive), - ) - } else { - probe(on_wait_tick) - } + wait_past_readiness_budget( + || probe(&mut *on_wait_tick), + || service_still_live(paths, &record.service_id, service_pid_is_alive), + ) } /// One sample of a freshly launched service for the startup wait: the model @@ -377,8 +402,7 @@ pub(super) fn wait_on_existing_service( // The key the caller generated belongs to a launch that did not happen; the // existing service answers to its own. let stored_key = endpoint_keys::endpoint_api_key(paths, &report.service_id); - let readiness = - wait_for_record_ready(paths, &record, true, stored_key.as_deref(), on_wait_tick); + let readiness = wait_for_record_ready(paths, &record, stored_key.as_deref(), on_wait_tick); if let Some(error) = died_during_wait( paths, &report.service_id, @@ -405,6 +429,42 @@ mod tests { use crate::tests::test_paths; use std::fs; + /// The startup wait's sample comes from the manifest on disk: the + /// engine's own `failed`, liveness by pid (supervisor or engine), and the + /// name the endpoint answers to. + #[test] + fn observe_launch_reads_failure_and_liveness_from_the_manifest() -> anyhow::Result<()> { + let (root, paths) = test_paths("observe-launch"); + paths.ensure()?; + let mut held = reusable_record(&paths); + held.supervisor_pid = 41; + held.engine_pid = Some(42); + held.write()?; + let sample = |pid_alive: fn(u32) -> bool| observe_launch(&paths, &held, None, pid_alive); + + let (model, signals) = sample(|pid| pid == 42); + assert_eq!(model, held.served_model_name()); + assert!(!signals.engine_failed); + assert!(signals.process_alive, "a live engine pid is alive"); + assert!( + sample(|pid| pid == 41).1.process_alive, + "a live supervisor is alive" + ); + assert!( + !sample(|_| false).1.process_alive, + "every recorded pid dead" + ); + + // Another writer's `failed` on disk is what the wait sees, not the + // copy it holds. + let mut on_disk = ManagedServiceRecord::read_manifest(&paths, &held.service_id)?; + on_disk.status = "failed".to_owned(); + on_disk.write()?; + assert!(sample(|_| true).1.engine_failed); + let _ = fs::remove_dir_all(root); + Ok(()) + } + #[test] fn wait_past_readiness_budget_keeps_probing_a_live_starting_server() { let mut readings = vec![ diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index fbd83e961b..31bab6400a 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -7,8 +7,8 @@ use super::endpoint_auth::drop_orphaned_endpoint_key_on_already_running; use super::readiness::{ - died_during_wait, record_wait_outcome, service_pid_is_alive, wait_for_launch_startup, - wait_for_record_ready, wait_on_existing_service, + died_during_wait, record_wait_outcome, service_pid_is_alive, settle_ended_launch, + wait_for_launch_startup, wait_for_record_ready, wait_on_existing_service, }; #[cfg(feature = "e2e-oom-fault-injection")] use super::readiness::{e2e_simulate_oom_launch, simulate_oom_managed_launch}; @@ -135,10 +135,9 @@ pub(crate) fn start_managed_service( on_wait_begin(service_id, port); } let (readiness, startup_stop) = if wait_until_ready { - let readiness = - wait_for_record_ready(paths, &record, true, endpoint_api_key, &mut |_elapsed| { - on_view(&steady); - }); + let readiness = wait_for_record_ready(paths, &record, endpoint_api_key, &mut |_elapsed| { + on_view(&steady); + }); (readiness, None) } else { match wait_for_launch_startup( @@ -162,15 +161,32 @@ pub(crate) fn start_managed_service( ) { return Err(error); } - let launch_status = match &startup_stop { - Some(stop) => serve_progress::recorded_status_after_stop(stop), - None => status_for_readiness(readiness), + if let Some(stop) = startup_stop + .as_ref() + .filter(|stop| !stop.leaves_server_running()) + { + // The engine failed or the process ended: settled the way a `--wait` + // that saw the server exit is, with the same memory guidance. + let message = stop.user_message(requested_model, engine, service_id); + let why = format!("{:?}", stop.reason); + return Err(serve_progress::ServeStartupFailed { + message: settle_ended_launch(paths, service_id, &why, message), + server_running: false, + } + .into()); + } + // A stall or the deadline left the server up: `running`, which `rocmd` + // leaves alone until the startup deadline. + let launch_status = if startup_stop.is_some() { + "running" + } else { + status_for_readiness(readiness) }; record_wait_outcome( paths, &mut record, launch_status, - readiness == EndpointReadiness::Serving && startup_stop.is_none(), + readiness == EndpointReadiness::Serving, )?; let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); record_cli_audit_event( @@ -185,11 +201,10 @@ pub(crate) fn start_managed_service( Some(service_id), ); if let Some(stop) = startup_stop { - return Err(serve_progress::ServeStartupFailed(stop.user_message( - requested_model, - engine, - service_id, - )) + return Err(serve_progress::ServeStartupFailed { + message: stop.user_message(requested_model, engine, service_id), + server_running: true, + } .into()); } Ok(ManagedLaunchReport { @@ -452,6 +467,175 @@ mod tests { Ok(()) } + /// What a fresh launch through the spawn hook left behind. + #[cfg(not(windows))] + struct HookedLaunch { + result: Result, + record: Result, + announced: Vec, + child_alive_after: bool, + audit: Vec, + } + + /// Launch `service_id` with `argv` as the engine child, under a short + /// interactive policy: nothing listens on the port, so the wait ends on + /// the silent limit or on the child's exit. + #[cfg(not(windows))] + fn launch_hooked_child( + service_id: &str, + argv: &'static [&'static str], + silent_unreachable: Duration, + ) -> Result { + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + let (root, paths) = test_paths(service_id); + paths.ensure()?; + let child = std::rc::Rc::new(std::cell::RefCell::new(None::)); + let spawned = child.clone(); + TEST_SPAWN_HOOK.with(|hook| { + *hook.borrow_mut() = Some(std::rc::Rc::new(move |_command| { + let process = std::process::Command::new(argv[0]) + .args(&argv[1..]) + .spawn()?; + *spawned.borrow_mut() = Some(process.id()); + Ok(process) + })); + }); + let mut policy = ServeStartupPolicy::interactive(); + policy.poll = Duration::from_millis(20); + policy.bounds.silent_unreachable = silent_unreachable; + policy.bounds.deadline = Duration::from_secs(10); + let mut announced = Vec::new(); + let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + let result = start_managed_service( + &paths, + "lemonade", + service_id, + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Auto, + &resolve.device_policy, + &[], + None, + None, + None, + None, + launch_lock, + false, + false, + false, + &policy, + &mut |id, _port| announced.push(id.to_owned()), + &mut |_view| {}, + ); + TEST_SPAWN_HOOK.with(|hook| *hook.borrow_mut() = None); + let record = ManagedServiceRecord::read_manifest(&paths, service_id); + let audit = rocm_core::load_recent_audit_events(&paths, 20).unwrap_or_default(); + let pid = *child.borrow(); + let child_alive_after = pid.is_some_and(super::service_pid_is_alive); + if let Some(pid) = pid { + let _ = rocm_core::terminate_process(pid); + } + let _ = fs::remove_dir_all(root); + Ok(HookedLaunch { + result, + record, + announced, + child_alive_after, + audit, + }) + } + + #[cfg(not(windows))] + fn startup_failure(result: Result) -> serve_progress::ServeStartupFailed { + let Err(error) = result else { + panic!("the startup wait must stop"); + }; + error + .downcast::() + .expect("the stop is the typed startup failure") + } + + /// A fresh launch whose wait stops on a live server: the wait is announced + /// first, the server is left running and recorded `running` without an + /// inference verification, and the typed failure names that service's + /// logs and stop commands and asks `serve` to supervise it. + #[cfg(not(windows))] + #[test] + fn a_startup_stop_records_the_live_server_and_returns_the_typed_failure() -> Result<()> { + let service_id = "lemonade-startup-stop"; + let launch = launch_hooked_child(service_id, &["sleep", "30"], Duration::from_millis(300))?; + assert!(launch.child_alive_after, "the wait must not end the server"); + let failure = startup_failure(launch.result); + assert!(failure.server_running, "{}", failure.message); + for hint in [ + crate::service_hints::service_logs_hint(service_id), + crate::service_hints::service_stop_hint(service_id), + ] { + assert!(failure.message.contains(&hint), "{}", failure.message); + } + assert_eq!(launch.announced, [service_id], "the wait is announced once"); + let record = launch.record?; + assert_eq!(record.status, "running", "the live server stays recorded"); + assert_eq!(record.inference_verified_at_unix_ms, None); + Ok(()) + } + + /// A launch whose server exits during the wait is over: settled `stopped` + /// the way a `--wait` that saw it exit is, not `failed`, which `rocmd` + /// would restart behind the error the user just read; and the typed + /// failure does not ask `serve` to supervise it. + #[cfg(not(windows))] + #[test] + fn a_server_that_exits_during_the_wait_is_settled_stopped_and_not_supervised() -> Result<()> { + let service_id = "lemonade-startup-exit"; + // The silent limit is far past the child's life, so the exit is what + // ends the wait. + let launch = launch_hooked_child(service_id, &["sleep", "0.3"], Duration::from_secs(5))?; + let failure = startup_failure(launch.result); + assert!(!failure.server_running, "{}", failure.message); + assert!( + failure.message.contains("the server process exited"), + "{}", + failure.message + ); + assert!( + !failure + .message + .contains(&crate::service_hints::service_stop_hint(service_id)), + "{}", + failure.message + ); + assert_eq!(launch.record?.status, "stopped"); + assert!( + launch + .audit + .iter() + .any(|event| event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(service_id) + && event.message.contains("ProcessExited")), + "the ended launch is audited: {:?}", + launch.audit + ); + Ok(()) + } + /// The launch lock must still be held when the child is spawned: it is what /// serializes allocation against publication. Observed through the spawn /// hook, since releasing it early changes nothing else a test can see. diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index ec564bf4d4..0c57bba560 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -343,13 +343,7 @@ pub(crate) fn find_recoverable_service( &record.engine, &record.service_id, ) else { - // A live process that has not opened its port is still the - // interactive serve wait's job. Restarting it here kills a - // download the CLI is reporting. A dead pid is not spared. - if record.status == "running" && live_startup_still_in_progress(&record, now) { - continue; - } - if let Some(reason) = endpoint_service_recovery_reason(&record) { + if let Some(reason) = unreachable_endpoint_recovery_reason(&record, now) { return Ok(Some((record, reason))); } continue; @@ -360,10 +354,7 @@ pub(crate) fn find_recoverable_service( format!("healthcheck_status_{}", healthcheck.status), ))); } - if let Some(reason) = endpoint_service_recovery_reason(&record) { - if record.status == "running" && live_startup_still_in_progress(&record, now) { - continue; - } + if let Some(reason) = unreachable_endpoint_recovery_reason(&record, now) { return Ok(Some((record, reason))); } } @@ -371,6 +362,23 @@ pub(crate) fn find_recoverable_service( Ok(None) } +/// Why a `ready`/`running` service whose endpoint does not answer should be +/// recovered, or `None` when it should not. +/// +/// A live `running` process that has not opened its port is still the +/// interactive serve wait's job: restarting it kills a download the CLI is +/// reporting. A dead pid is not spared. One function for both places that +/// fall back to the endpoint, so neither can drop the guard. +fn unreachable_endpoint_recovery_reason( + record: &ManagedServiceRecord, + now_unix_ms: u128, +) -> Option { + if record.status == "running" && live_startup_still_in_progress(record, now_unix_ms) { + return None; + } + endpoint_service_recovery_reason(record) +} + /// Whether an operator asked for this service to stop and that stop is still /// standing. /// @@ -1176,7 +1184,7 @@ mod tests { "qwen", "Qwen/Qwen3.5", "127.0.0.1", - 11461, + free_loopback_port(), "managed", std::process::id(), None, @@ -1246,7 +1254,7 @@ mod tests { "qwen", "Qwen/Qwen3.5", "127.0.0.1", - 11462, + free_loopback_port(), "managed", std::process::id(), None, diff --git a/docs/architecture.md b/docs/architecture.md index 06f959d100..7b54a80d3c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -99,8 +99,8 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: bind-host validation and the endpoint API key policy. `validate_bind_host` refuses a non-loopback host without `--allow-public-bind` (`is_loopback_host` is the inverse of `rocm_engine_protocol::is_public_bind_host`, which `rocmd` shares); `resolve_endpoint_auth` decides the key a launch uses (none for a loopback bind unless a key is required, otherwise the supplied key trimmed or a generated one, refusing an empty key or one with a control character); `drop_orphaned_endpoint_key_on_already_running` clears a key staged for an attempt that spawned nothing; `ensure_public_bind_engine_supported` refuses a public bind with Windows managed Lemonade, whose key cannot be enforced; and `ensure_public_service_has_endpoint_key` is the guard a respawn runs so a public (or `--require-api-key`) service whose key file is gone is not reopened without authentication. All six are `pub(crate)` and re-exported by the parent: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads `ensure_public_service_has_endpoint_key`, [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) reads `is_loopback_host`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads the rest. - [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the part of a managed launch that decides before anything spawns. `resolve_managed_port_in_transaction` refreshes liveness, answers an equivalent live service before leasing any port (`ManagedPortDecision`), and otherwise leases a concrete port against both the ports live records still reserve (`live_managed_services`, `reserved_service_ports`) and the real OS through [`serve_port.rs`](../apps/rocm/src/serve_port.rs); a caller must hold the launch lock across it and the record write that follows. `abandon_failed_managed_launch` undoes a failed attempt before that lock is released: a spawned child is terminated first, through the same `rocm_core::terminate_recorded_pid_within` routine a stop uses (so on Windows its descendants are taken down too), and its record kept, marked stopped, while an attempt that never spawned has every artifact removed. A spawned child whose start token was not captured, on a host that records tokens (Linux and Windows, which records the process creation time), is not signalled (`launch_cleanup_refusal`): it was gone before it could be captured and its pid may belong to another process. `managed_service_reuse_refusal` (with `ManagedReuseRefusal`) is the one answer to whether a live matching service can serve a request (recipe, `--require-api-key`, device policy), and `no_usable_gpu_refusal` words the no-usable-GPU refusal once for `serve()`'s pre-flight and for the spawn. `managed_service_reuse_refusal` and `no_usable_gpu_refusal` are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), and `ManagedReuseRefusal` is `pub(crate)` too because the first returns it (only [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) matches on it); `ManagedPortDecision`, `live_managed_services`, `resolve_managed_port_in_transaction` and `abandon_failed_managed_launch` are `pub(super)` for [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs). - [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the detached engine child that the background and attached serve paths share. The process and stdio plumbing (`detach_background_command`, `attach_background_stdio`, `managed_service_process_command`, `managed_engine_startup_failure_detail`) is `pub(crate)` and, off Windows, re-exported: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads all four for a restart and [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) reads the first three. On Windows `managed_engine_startup_failure_detail` is re-exported for test builds only, because the [`services_text.rs`](../apps/rocm/src/services_text.rs) log-tail tests call it through `crate::service_launch` there too. `ManagedLaunchReport` (`pub(crate)`) is the engine-neutral result of a launch, and `ManagedSpawn` (`pub(super)`) says whether the launch found a live service or spawned a child. `spawn_managed_engine_child` is the whole serialized transaction under the launch lock (duplicate detection, port lease, manifest and log creation, argv preparation, the bounded immediate-exit check and record publication) and returns before any readiness wait; `launch_managed_engine_child_locked` is its side-effecting half, split out so that every failure after the record's first write goes through `abandon_failed_managed_launch`. The tests reach the spawn through a test-only, thread-scoped hook (`TEST_SPAWN_HOOK`, `pub(super)` so the tests in [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) can observe the state at the instant of the spawn). -- [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns private items: waiting on a service after it is launched or found live. `wait_for_record_ready` and `wait_on_existing_service` run the wait (one wait is bounded by `READINESS_WAIT`, and `wait_past_readiness_budget` is what `rocm serve --wait` uses to keep polling past it until the server answers or stops being live), `service_pid_is_alive`, `record_still_live` and `service_still_live` decide whether the tracked process is still there, `settle_exited_service` demotes the manifest of one that is not, and `died_during_wait` and `record_wait_outcome` end a wait, writing its outcome back without discarding what changed on disk meanwhile. `e2e_simulate_oom_launch` and `simulate_oom_managed_launch` are the end-to-end fault-injection seam behind the `e2e-oom-fault-injection` feature: the first is a `const fn` returning false without the feature, is `pub(crate)` and re-exported, and is read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs); the second fabricates the on-disk state of a launch that ran out of memory. `wait_for_record_ready`, `wait_on_existing_service`, `service_pid_is_alive`, `died_during_wait`, `record_wait_outcome` and `simulate_oom_managed_launch` are `pub(super)` for [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs); the rest are private. -- [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: the background launch path. `start_managed_service` runs the spawn transaction, releases the launch lock once the record is published, waits for HTTP readiness and returns a `ManagedLaunchReport`; if the launch fails before a record exists it drops the endpoint key `serve()` staged for the attempt. `print_managed_launch_plain` renders the report line by line for output that is not interactive, and `wait_begin_client_config` returns the one-time client configuration for a generated key when a wait begins, built by `render_endpoint_client_config` (`pub(super)`, since [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) prints it too). The three functions other modules call are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs). +- [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns private items: waiting on a service after it is launched or found live. `wait_for_record_ready` and `wait_on_existing_service` run the wait (one wait is bounded by `READINESS_WAIT`, and `wait_past_readiness_budget` is what `rocm serve --wait` uses to keep polling past it until the server answers or stops being live), `service_pid_is_alive`, `record_still_live` and `service_still_live` decide whether the tracked process is still there, `settle_exited_service` demotes the manifest of one that is not (`settle_ended_launch` does that and adds the memory guidance, for `died_during_wait` and for the interactive wait's engine-failed or process-exited stop), `wait_for_launch_startup` runs a fresh launch's wait without `--wait` through [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) with `observe_launch` sampling the manifest for it, and `died_during_wait` and `record_wait_outcome` end a wait, writing its outcome back without discarding what changed on disk meanwhile. `e2e_simulate_oom_launch` and `simulate_oom_managed_launch` are the end-to-end fault-injection seam behind the `e2e-oom-fault-injection` feature: the first is a `const fn` returning false without the feature, is `pub(crate)` and re-exported, and is read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs); the second fabricates the on-disk state of a launch that ran out of memory. `wait_for_record_ready`, `wait_for_launch_startup`, `wait_on_existing_service`, `service_pid_is_alive`, `settle_ended_launch`, `died_during_wait`, `record_wait_outcome` and `simulate_oom_managed_launch` are `pub(super)` for [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs); the rest are private. +- [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: the background launch path. `start_managed_service` runs the spawn transaction, releases the launch lock once the record is published, announces the wait when it can end without a summary (`announces_wait`: `--wait`, or the interactive policy), waits for HTTP readiness and returns a `ManagedLaunchReport` or the typed `ServeStartupFailed`; if the launch fails before a record exists it drops the endpoint key `serve()` staged for the attempt. `print_managed_launch_plain` renders the report line by line for output that is not interactive, and `wait_begin_client_config` returns the one-time client configuration for a generated key when a wait begins, built by `render_endpoint_client_config` (`pub(super)`, since [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) prints it too). The three functions other modules call are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs). - [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns a private type: starting the background automation helper (`rocm daemon`) when a watcher needs it. `ensure_background_helper_running` and `ensure_background_helper_running_quiet` do nothing unless an automation is enabled and [`background_helper.rs`](../apps/rocm/src/background_helper.rs)'s `background_helper_already_running` says no helper is running; `ensure_background_helper_running_with` takes the spawn as a parameter so the decision is testable, and `spawn_background_helper` does the detached spawn. Concurrent callers queue on an autostart lock, and `AutostartClaim` (with `AUTOSTART_CLAIM_TTL_MS`, `read_autostart_claim`, `write_autostart_claim` and `autostart_spawn_in_flight`) records a spawn in flight so the next holder defers instead of starting a second helper. Both entry points are `pub(crate)` and re-exported: [`automations.rs`](../apps/rocm/src/automations.rs) calls the first, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) and [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) call the quiet one. - [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns private types: the attached serve path (`--verbose`/`--foreground`). `run_attached_service` spawns the same detached child the background path does and follows its log in the terminal until the user detaches with Ctrl-D (the server keeps running) or stops it with Ctrl-C, or the engine exits; `AttachOutcome` is that result and `detach_key_outcome` maps a key press to it. `stream_attached_logs` reads keys in crossterm raw mode (`RawModeGuard` restores the terminal on every exit path) and `stream_attached_logs_no_tty` follows the log until the process exits when stdin is not a terminal. `finish_attached_stop` turns the answer of the stop that Ctrl-C ran into output: a confirmed stop is reported on stdout, an unconfirmed one becomes the error, and a failed stop signals the child this session launched. `run_attached_service` is `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs). - [`service_records.rs`](../apps/rocm/src/service_records.rs) — mechanical relocation, owns types: removing one managed-service record and every file it owns (`remove_managed_service_record`, `service_record_artifacts`) and the `rocm services prune` plan/render/apply steps (`build_service_prune_plan`, `render_service_prune_plan`, `apply_service_prune_plan`, `prune_managed_service_records`) with the age gate behind them (`DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS`, `service_prune_min_age_hours`). It reads record state through [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) and [`services_text.rs`](../apps/rocm/src/services_text.rs) and builds its advice text with [`service_hints.rs`](../apps/rocm/src/service_hints.rs); none of the three depends on it. `remove_managed_service_record_with` takes `--force` and the read of each recorded PID: `--force` removes a record only when none of its recorded processes is confirmed running, one could not be queried and a stop was requested, and reports which. `remove_managed_service_record_with`, `prune_managed_service_records`, `service_prune_min_age_hours` and `DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS` are `pub(crate)` (`services()` in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) calls the first three; the `ServicesCommand::Prune` clap definition in [`cli.rs`](../apps/rocm/src/cli.rs) uses the constant), and `ServicePruneOutcome`, the return type of `prune_managed_service_records`, is `pub(crate)` with `pub(crate)` `text`, `removed_records`, `removed_files` and `failures` fields that `services()` reads. Every other item is private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths`, `plant_service_record` and `plant_service_record_without_its_key` fixtures through `crate::tests::` (the last two widened to `pub(crate)` for that). @@ -170,7 +170,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses (its engine child runs on the `rocm` binary that `DaemonHost::rocm_binary` finds, before the first write, since `rocmd` has no engine commands, and it polls readiness through the same host), and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor; the binary that runs `supervise` comes from [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), and a restart that cannot find it fails before it writes the record). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and `stop_requested` from [`watchers.rs`](../apps/rocmd/src/watchers.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. - [`watchers.rs`](../apps/rocmd/src/watchers.rs) — the module root of the built-in watchers: the registry (`reconcile_watcher_snapshots`) and the event collector and dispatcher (`evaluate_watchers`, `collect_automation_events_with_gpu_snapshot`, which takes the GPU snapshot as a closure so tests can inject one, and `evaluate_watchers_for_events`), which route each event to its watcher's handler. `find_recoverable_service` and `stop_requested` are re-exported from here because [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s and [`service.rs`](../apps/rocmd/src/service.rs)'s tests reach them by the `crate::watchers::` path. Each watcher's handler lives in the child below that carries its name, together with the policy they share, and a child reaches only [`policy.rs`](../apps/rocmd/src/watchers/policy.rs) and, for the thermal-protect watcher, [`gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) among the others. - [`watchers/policy.rs`](../apps/rocmd/src/watchers/policy.rs) — what a watcher does in each mode and the proposals it queues: `WatcherPolicyAction` and `watcher_policy_action` (observe, queue a proposal or run contained, chosen by the watcher mode), `queue_proposal` and `queue_proposal_with_arguments` with the action-to-tool and action-to-arguments tables behind them, and `pending_proposal_exists`, the check the thermal-protect and server-recover handlers use so they do not queue a duplicate of a proposal that is still pending. It uses no other watcher module. - - [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) — the `server-recover` watcher: finding a managed service that needs recovery (`find_recoverable_service`, from a failed or stale manifest, an unreachable endpoint or a failing engine healthcheck, skipping a service whose stop request still stands via `stop_requested`), the recovery event kinds and plain-language reasons, the backoff that spaces recovery attempts (`server_recover_due`), and the handler that restarts the service or queues a restart proposal by watcher mode. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s `restart_managed_service`. + - [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) — the `server-recover` watcher: finding a managed service that needs recovery (`find_recoverable_service`, from a failed or stale manifest, an unreachable endpoint or a failing engine healthcheck, skipping a service whose stop request still stands via `stop_requested`, and sparing a live `running` or `starting` service younger than `rocm_core::SERVE_STARTUP_DEADLINE` whose port is not open yet via `live_startup_still_in_progress`), the recovery event kinds and plain-language reasons, the backoff that spaces recovery attempts (`server_recover_due`), and the handler that restarts the service or queues a restart proposal by watcher mode. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s `restart_managed_service`. - [`watchers/gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) — the `gpu-metrics` watcher and the GPU readings it works from: the telemetry-event handler (`handle_gpu_metrics_event`, which records telemetry and takes no action), the interval check (`gpu_metrics_due`), the one-line snapshot summary, and the reading of amd-smi monitor entries into thermal-pressure and memory-pressure events (`gpu_pressure_events`, `gpu_pressure_event`) with the thresholds they use. It uses [`persistence.rs`](../apps/rocmd/src/persistence.rs) and no other watcher module, and [`gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) reads its interval and thresholds. - [`watchers/gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) — the `gpu-thermal-protect` watcher: the pressure-event handler (`handle_gpu_thermal_protect_event`), which records the event and, outside observe mode, picks the one running managed service as the target and queues a reviewed stop request unless one is already pending, and the interval check (`gpu_thermal_protect_due`). It uses [`gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) for the interval and thresholds, [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`common.rs`](../apps/rocmd/src/common.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). - [`watchers/therock_update.rs`](../apps/rocmd/src/watchers/therock_update.rs) — the `therock-update` watcher: the schedule check (`therock_update_due`, every six hours) and the handler, which in observe mode records a reminder, in propose mode also queues an update-check proposal, and in contained mode runs the read-only `check_updates` sandbox tool and records a notification when an update or repair is available, never applying one. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`sandbox.rs`](../apps/rocmd/src/sandbox.rs), [`sandbox_tool.rs`](../apps/rocmd/src/sandbox_tool.rs), [`common.rs`](../apps/rocmd/src/common.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). diff --git a/docs/testing.md b/docs/testing.md index 4cae57726f..97f6facde5 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1995,16 +1995,30 @@ exits, a download percent stops for 3 minutes (`DOWNLOAD_STALL`), the port stays closed with no log for 2 minutes (`SILENT_UNREACHABLE`), a load line stops changing for 5 minutes (`SILENT_LOAD`), or after 30 minutes (`rocm_core::SERVE_STARTUP_DEADLINE`). Engine log lines are masked of -credentials before they are shown (`redact_log_line`). `rocmd` does not -restart a live `running` or `starting` service younger than that deadline. +credentials before they are shown (`redact_log_line`). Once the endpoint lists the model, +the one-token inference request is not sent every poll: it backs off from 1 +second to `INFERENCE_PROBE_RETRY_INTERVAL` (15 seconds), timed from when the +previous request returned (`InferenceProbeThrottle`, tested under a fake clock +by `inference_probe_backs_off_to_the_retry_interval_after_listing`). `rocmd` does not +restart a live `running` service whose endpoint does not answer, or a live +`starting` one past the stale-start window, while it is younger than that +deadline; a recoverable engine healthcheck is still restarted. Piped output keeps the 45-second status budget. Like `--wait`, the interactive wait first prints the service id, `rocm logs --service --follow` and a public bind's key (`announces_wait` in [start.rs](../apps/rocm/src/service_launch/start.rs)), since it can fail -without printing a summary; `--wait` keeps its own wait, with no stall limit -or deadline. The status line is a TTY spinner, and the harness does not allocate a -PTY, so this is not a default-lane Gherkin scenario (a live engine would also -be `@requires-gpu`). `decide_serve_startup` and `wait_for_serve_startup` in +without printing a summary; `--wait` keeps its own wait, with no CLI-side +stall limit or deadline. A wait that ends because the engine failed or the +process exited settles the record the way a `--wait` does (`stopped`, or the +engine's own `failed`), records an audit event and adds the same memory +guidance (a `--wait` that sees the server exit now records the event too); +when the engine failed but its process is still up, the message names the +stop command. Only a stall or the +deadline leaves the server running for the background helper to supervise. +There is no Gherkin scenario for this yet. The harness can drive a PTY +(`download_progress_pty.feature`), but interactive `rocm serve` needs a live +engine: the no-GPU mock lane stubs the HTTP endpoint and never runs +`rocm serve`, so a scenario would be `@requires-gpu`. `decide_serve_startup` and `wait_for_serve_startup` in [serve_progress.rs](../apps/rocm/src/serve_progress.rs) are the state matrix, covered by: diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 467b12a0e7..c127384159 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -2,9 +2,10 @@ Feature: Model serving # Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows # download, load, and health on the status line until the server is ready or - # the startup wait fails. Seeing that line requires a TTY spinner and a live - # engine log, so it is not a default-lane scenario: the harness does not - # allocate a PTY, and a real serve is `@requires-gpu`. The state matrix is + # the startup wait fails. There is no scenario for it yet: the harness can + # drive a PTY (download_progress_pty.feature), but interactive `rocm serve` + # needs a live engine, and the no-GPU mock lane stubs the HTTP endpoint + # without running `rocm serve`, so one would be `@requires-gpu`. The state matrix is # covered without a GPU by `decide_serve_startup` and `wait_for_serve_startup` # in apps/rocm/src/serve_progress.rs. From 802eee2a230ef4defc44c601e6c0c0c75082f710 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Wed, 11 Nov 2026 10:53:32 +0200 Subject: [PATCH 03/14] test(serve): give the startup-wait tests an endpoint that answers at once Three serve_progress tests pointed the wait at a closed loopback port. On Windows a refused loopback connect takes about two seconds, so each poll outlasted the tests' stall and deadline limits and the Windows job failed. They now use a local endpoint that answers 503 at once, which is the same "model not listed" reading without the platform's connect delay. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 43 ++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index 830611fa54..bd9ea74d59 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -1493,13 +1493,8 @@ mod tests { let deadline = Duration::from_secs(5); let started = Instant::now(); let mut labels = Vec::new(); - // Nothing is listening: the probe fails fast on connection refused. - let port = { - let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); - let port = listener.local_addr().unwrap().port(); - drop(listener); - port - }; + // The endpoint never lists the model, and answers at once. + let port = not_listing_port(); let result = wait_for_serve_startup( "vllm", "qwen", @@ -1579,10 +1574,25 @@ mod tests { ); } - fn refused_port() -> u16 { + /// A local endpoint that answers every request with 503 at once: the + /// model is never listed. Not a closed port: on Windows a refused loopback + /// connect takes about two seconds, which outlasts these tests' limits. + fn not_listing_port() -> u16 { let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); let port = listener.local_addr().unwrap().port(); - drop(listener); + thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let _ = stream.read(&mut buffer); + let body = r#"{"error":"loading"}"#; + let _ = write!( + stream, + "HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); port } @@ -1591,12 +1601,7 @@ mod tests { /// 45-second contract scripts and CI read, and it has no stop. #[test] fn piped_wait_ignores_engine_failure_and_a_dead_process() { - let port = { - let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); - let port = listener.local_addr().unwrap().port(); - drop(listener); - port - }; + let port = not_listing_port(); let mut policy = ServeStartupPolicy::status_budget(Duration::from_millis(200)); policy.poll = Duration::from_millis(20); let mut polls = 0; @@ -1687,7 +1692,7 @@ mod tests { "vllm", "qwen", "127.0.0.1", - refused_port(), + not_listing_port(), None, &short_policy(Duration::from_secs(5), Duration::from_secs(30)), &mut || { @@ -1781,7 +1786,7 @@ mod tests { "vllm", "qwen", "127.0.0.1", - refused_port(), + not_listing_port(), None, &short_policy(Duration::from_millis(1_200), stall), &mut || { @@ -1822,7 +1827,7 @@ mod tests { "vllm", "qwen", "127.0.0.1", - refused_port(), + not_listing_port(), None, &policy, &mut || { @@ -1901,7 +1906,7 @@ mod tests { "vllm", "qwen", "127.0.0.1", - refused_port(), + not_listing_port(), Some(key), &short_policy(Duration::from_secs(5), Duration::from_secs(30)), &mut || { From 9e8419b8d97e42d2fec1ce5ee792846686be76e7 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Wed, 11 Nov 2026 11:35:49 +0200 Subject: [PATCH 04/14] fix(serve): quote the logs hint and cut redacted values on a char boundary service_logs_hint printed the service id bare, unlike every other hint in service_hints.rs, whose module promises a line that pastes back as the argv it names. It now goes through shell_quote too. mask_value cut a value one byte past the start of its last character, which panics when that character is wider than a byte. It now cuts at the character's end. A secret word anywhere in an underscore name (`secret_key`, `--private-key`) is masked as well as one at the end. After a stall or deadline stop the record rewrite is best-effort, so the stop stays the error the user reads. A test pins which clock each silent limit reads, and the help and README scope the wait announcement to a fresh launch. Signed-off-by: Roman Inflianskas --- README.md | 2 +- apps/rocm/src/cli.rs | 4 +- apps/rocm/src/serve_progress.rs | 67 +++++++++++++++++++++++++-- apps/rocm/src/service_hints.rs | 19 +++++++- apps/rocm/src/service_launch/start.rs | 9 +++- docs/architecture.md | 2 +- 6 files changed, 93 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 2f7fc480b3..327d14c996 100644 --- a/README.md +++ b/README.md @@ -1150,7 +1150,7 @@ is already on is refused rather than silently ignored; omit `--port` to reuse the running service, or stop it first to serve on the new port. By default the server runs in the background under ROCm CLI's supervision. -In a terminal it first prints the service id, how to follow its log +For a fresh launch in a terminal it first prints the service id, how to follow its log (`rocm logs --service --follow`) and, for a public bind, the API key. While it starts, the status line follows the engine: a download, a weight load, or the latest log line, until the server is ready or that wait fails diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 8db47837f7..3d9201ecdd 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -430,8 +430,8 @@ rocm model --verbose" /// Picks an engine and ROCm runtime automatically unless overridden. An /// Anthropic `/v1/messages` endpoint is published beside the OpenAI /// one (see `rocm services list`). By default the - /// server runs as a managed background service. In a terminal it first prints the - /// service id and how to follow its log; while it starts, the status line + /// server runs as a managed background service. For a fresh launch in a terminal it + /// first prints the service id and how to follow its log; while it starts, the status line /// follows what the engine is doing (download, load, or the latest log line) until /// the server is ready or that wait fails. A ready server then prints a deployment /// summary (status, endpoint, model, and smoke-test metrics). A wait that fails diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index bd9ea74d59..d2b88a7204 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -800,9 +800,13 @@ fn is_secret_name(word: &str) -> bool { } name == "authorization" || name == "bearer" - || ["api_key", "apikey", "token", "password", "secret"] + || ["api_key", "apikey", "token", "password", "secret", "private_key"] .iter() .any(|suffix| name.ends_with(suffix)) + // `secret_key`, `token_value`: a secret word anywhere in a `_` name. + || name + .split('_') + .any(|part| matches!(part, "secret" | "token" | "password" | "passwd")) } /// A flag (`--api-key`), a key ending in `:` (`'api_key':`, @@ -825,9 +829,12 @@ fn mask_value(word: &str) -> String { let body_start = word .find(|c: char| !matches!(c, '\'' | '"' | '(' | '[')) .unwrap_or(word.len()); + // The end of the last body character, which may be wider than one byte. let body_end = word - .rfind(|c: char| !matches!(c, '\'' | '"' | ',' | ')' | ']' | '}')) - .map_or(body_start, |index| index + 1); + .char_indices() + .rev() + .find(|(_, c)| !matches!(c, '\'' | '"' | ',' | ')' | ']' | '}')) + .map_or(body_start, |(index, c)| index + c.len_utf8()); if body_end <= body_start { return word.to_owned(); } @@ -1632,6 +1639,46 @@ mod tests { ); } + /// The two silent limits read different clocks: the unreachable one + /// counts from the start of the wait, the load one from the last change + /// of the log line. Each row moves one clock past its limit and holds the + /// other below it, so swapping the fields in either check fails a row. + #[test] + fn silent_limits_read_their_own_clocks() { + let decide = |log_line: Option<&str>, elapsed: Duration, since: Duration| { + let mut signals = alive(); + signals.log_line = log_line.map(str::to_owned); + decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + ServeStartupTiming { + elapsed, + since_progress: since, + }, + &interactive(), + ) + }; + let short = Duration::from_secs(1); + // Unreachable, no log: the time since the start decides. + let stop = assert_stop(decide(None, SILENT_UNREACHABLE, short)); + assert_eq!(stop.reason, ServeStartupStopReason::Silent); + assert!(matches!( + decide(None, short, SILENT_UNREACHABLE), + ServeStartupDecision::Continue(_) + )); + // Loading: the time since the line last changed decides, however long + // the wait has run. + let load = Some("Loading safetensors checkpoint shards: 40%"); + let stop = assert_stop(decide(load, SILENT_LOAD, SILENT_LOAD)); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + assert!(matches!( + decide(load, SILENT_LOAD * 2, short), + ServeStartupDecision::Continue(_) + )); + } + #[test] fn only_a_stall_or_the_deadline_leaves_the_server_running() { let stop = |reason, process_alive| ServeStartupStop { @@ -1851,6 +1898,20 @@ mod tests { fn redact_log_line_masks_keys_tokens_and_secret_flags() { let key = "sk-endpoint-0123456789"; for (raw, expected) in [ + ( + "launch --secret-key abc123 --private-key def456 --port 8000", + "launch --secret-key [redacted] --private-key [redacted] --port 8000", + ), + // A value ending in a multi-byte character is cut at a character + // boundary, not one byte past the start of that character. + ( + "launch --api-key clé --port 8000", + "launch --api-key [redacted] --port 8000", + ), + ( + "launch --api-key 'clé', --port 8000", + "launch --api-key '[redacted]', --port 8000", + ), ( "non-default args: {'model': 'qwen', 'api_key': 'sk-endpoint-0123456789', 'port': 8000}", "non-default args: {'model': 'qwen', 'api_key': '[redacted]', 'port': 8000}", diff --git a/apps/rocm/src/service_hints.rs b/apps/rocm/src/service_hints.rs index 9216dd4b92..bc81279a49 100644 --- a/apps/rocm/src/service_hints.rs +++ b/apps/rocm/src/service_hints.rs @@ -59,7 +59,7 @@ pub(crate) fn service_stop_hint(service_id: &str) -> String { /// that service. Shared so a startup failure and the deployment summary cannot /// name two different log commands. pub(crate) fn service_logs_hint(service_id: &str) -> String { - format!("rocm logs --service {service_id}") + format!("rocm logs --service {}", shell_quote(service_id)) } /// The directly-runnable ` services stop --yes` hint for a @@ -128,6 +128,10 @@ mod tests { remote_service_stop_hint("rocm", "vllm-qwen-8000"), "rocm services stop vllm-qwen-8000 --yes" ); + assert_eq!( + service_logs_hint("vllm-qwen-8000"), + "rocm logs --service vllm-qwen-8000" + ); } proptest! { @@ -153,6 +157,19 @@ mod tests { hint ); } + // Mutation: printing `service_id` bare in the logs hint fails this. + let logs = service_logs_hint(&service_id); + prop_assert_eq!( + shlex::split(&logs), + Some(vec![ + "rocm".to_owned(), + "logs".to_owned(), + "--service".to_owned(), + service_id, + ]), + "{}", + logs + ); } } } diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 31bab6400a..0c4cde39ff 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -182,12 +182,17 @@ pub(crate) fn start_managed_service( } else { status_for_readiness(readiness) }; - record_wait_outcome( + let recorded = record_wait_outcome( paths, &mut record, launch_status, readiness == EndpointReadiness::Serving, - )?; + ); + // After a stop, the stop is what the user needs to read; a record that + // could not be rewritten is best-effort there, as `settle_ended_launch` is. + if startup_stop.is_none() { + recorded?; + } let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); record_cli_audit_event( paths, diff --git a/docs/architecture.md b/docs/architecture.md index 7b54a80d3c..dab3eb7520 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -94,7 +94,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) — full domain extraction. - [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`stop_recorded_processes`, which hands the recorded PIDs to the routine [`proc_lifecycle.rs`](../crates/rocm-core/src/proc_lifecycle.rs) shares with `rocmd`, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which passes it to [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `run_attached_service` as the stop for a Ctrl-C, the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. This module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which does not import it back. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. - [`serve_port.rs`](../apps/rocm/src/serve_port.rs) — full domain extraction: the central `rocm serve` port policy (automatic vs explicit request, reservation by live managed services, OS preflight lease), used by [`service_launch.rs`](../apps/rocm/src/service_launch.rs); the CLI restart path shares the `rocm-core` lease primitives (`lease_loopback_port_or_bind_error`, `is_canonical_loopback_host`) rather than this module. -- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` and `rocm logs --service ` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. +- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` and `rocm logs --service ` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. - [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, since split into a directory of child modules, owns no code of its own beyond its declarations: spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind, divided among the children listed below. The file declares the children, a `test_support` module compiled only for tests (the fixtures more than one child's tests share), and `pub(crate) use` re-exports of the items other modules call, so [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs) and [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) still reach them as `crate::service_launch::name`; [`main.rs`](../apps/rocm/src/main.rs) does not. An item that sibling children share is `pub(super)` and reached through `super::`; everything else is private to the child that owns it. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path. This module does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it: `run_attached_service` takes the stop for a Ctrl-C as a parameter, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) passes `run_internal_sandbox_tool`. - [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: bind-host validation and the endpoint API key policy. `validate_bind_host` refuses a non-loopback host without `--allow-public-bind` (`is_loopback_host` is the inverse of `rocm_engine_protocol::is_public_bind_host`, which `rocmd` shares); `resolve_endpoint_auth` decides the key a launch uses (none for a loopback bind unless a key is required, otherwise the supplied key trimmed or a generated one, refusing an empty key or one with a control character); `drop_orphaned_endpoint_key_on_already_running` clears a key staged for an attempt that spawned nothing; `ensure_public_bind_engine_supported` refuses a public bind with Windows managed Lemonade, whose key cannot be enforced; and `ensure_public_service_has_endpoint_key` is the guard a respawn runs so a public (or `--require-api-key`) service whose key file is gone is not reopened without authentication. All six are `pub(crate)` and re-exported by the parent: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads `ensure_public_service_has_endpoint_key`, [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) reads `is_loopback_host`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads the rest. - [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the part of a managed launch that decides before anything spawns. `resolve_managed_port_in_transaction` refreshes liveness, answers an equivalent live service before leasing any port (`ManagedPortDecision`), and otherwise leases a concrete port against both the ports live records still reserve (`live_managed_services`, `reserved_service_ports`) and the real OS through [`serve_port.rs`](../apps/rocm/src/serve_port.rs); a caller must hold the launch lock across it and the record write that follows. `abandon_failed_managed_launch` undoes a failed attempt before that lock is released: a spawned child is terminated first, through the same `rocm_core::terminate_recorded_pid_within` routine a stop uses (so on Windows its descendants are taken down too), and its record kept, marked stopped, while an attempt that never spawned has every artifact removed. A spawned child whose start token was not captured, on a host that records tokens (Linux and Windows, which records the process creation time), is not signalled (`launch_cleanup_refusal`): it was gone before it could be captured and its pid may belong to another process. `managed_service_reuse_refusal` (with `ManagedReuseRefusal`) is the one answer to whether a live matching service can serve a request (recipe, `--require-api-key`, device policy), and `no_usable_gpu_refusal` words the no-usable-GPU refusal once for `serve()`'s pre-flight and for the spawn. `managed_service_reuse_refusal` and `no_usable_gpu_refusal` are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), and `ManagedReuseRefusal` is `pub(crate)` too because the first returns it (only [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) matches on it); `ManagedPortDecision`, `live_managed_services`, `resolve_managed_port_in_transaction` and `abandon_failed_managed_launch` are `pub(super)` for [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs). From ad1b07cd37499935a2192133e2cec86f05619d8c Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Wed, 11 Nov 2026 12:16:05 +0200 Subject: [PATCH 05/14] fix(serve): judge silence by its own length and check a spared pid's identity The unreachable silent limit counted the whole wait, so a long startup that went quiet for a moment could stop at once. It now counts from the last change of what the wait observes, as the load limit does and as the README describes. rocmd's live-startup guard trusted a bare pid for up to 30 minutes. It now checks the recorded start time with identity_state, so a recycled pid does not keep a dead service from being recovered. The event re-check that authorizes a restart now uses the same guarded reasons as the scan, so an event raised before a startup was spared cannot restart it. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 26 +++-- apps/rocmd/src/watchers/server_recover.rs | 127 +++++++++++++++++----- docs/architecture.md | 4 +- docs/testing.md | 3 +- 4 files changed, 120 insertions(+), 40 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index d2b88a7204..e1783d9005 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -534,7 +534,7 @@ pub(crate) fn decide_serve_startup( ); } if view.phase == ServeStartupPhase::Unreachable - && timing.elapsed >= policy.bounds.silent_unreachable + && timing.since_progress >= policy.bounds.silent_unreachable { return stop_decision( model, @@ -1279,7 +1279,10 @@ mod tests { "vllm", EndpointReadiness::Unreachable, &alive(), - at(SILENT_UNREACHABLE.as_millis() as u64, 0), + at( + SILENT_UNREACHABLE.as_millis() as u64, + SILENT_UNREACHABLE.as_millis() as u64, + ), &interactive(), )); assert!(stop.user_message("qwen", "vllm", "svc-1").contains(&hint)); @@ -1639,12 +1642,13 @@ mod tests { ); } - /// The two silent limits read different clocks: the unreachable one - /// counts from the start of the wait, the load one from the last change - /// of the log line. Each row moves one clock past its limit and holds the - /// other below it, so swapping the fields in either check fails a row. + /// Both silent limits count the silence, from the last change of what + /// the wait observes, not the whole wait: a long startup that goes quiet + /// is judged on how long it has been quiet. Each row moves one clock past + /// the limit and holds the other below it, so a check reading `elapsed` + /// fails a row. #[test] - fn silent_limits_read_their_own_clocks() { + fn silent_limits_count_from_the_last_change() { let decide = |log_line: Option<&str>, elapsed: Duration, since: Duration| { let mut signals = alive(); signals.log_line = log_line.map(str::to_owned); @@ -1661,13 +1665,13 @@ mod tests { ) }; let short = Duration::from_secs(1); - // Unreachable, no log: the time since the start decides. - let stop = assert_stop(decide(None, SILENT_UNREACHABLE, short)); - assert_eq!(stop.reason, ServeStartupStopReason::Silent); + // Unreachable, no log: only now silent, after a long startup. assert!(matches!( - decide(None, short, SILENT_UNREACHABLE), + decide(None, SILENT_UNREACHABLE * 3, short), ServeStartupDecision::Continue(_) )); + let stop = assert_stop(decide(None, SILENT_UNREACHABLE, SILENT_UNREACHABLE)); + assert_eq!(stop.reason, ServeStartupStopReason::Silent); // Loading: the time since the line last changed decides, however long // the wait has run. let load = Some("Loading safetensors checkpoint shards: 40%"); diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index 0c57bba560..aac129792c 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -12,8 +12,8 @@ use crate::persistence::load_service_record; use crate::supervise::restart_managed_service; use anyhow::{Context, Result}; use rocm_core::{ - AppPaths, AutomationRuntimeState, AutomationTriggerEvent, ManagedServiceRecord, WatcherMode, - format_host_port, process_is_running, unix_time_millis, + AppPaths, AutomationRuntimeState, AutomationTriggerEvent, IdentityState, ManagedServiceRecord, + ProcessIdentity, WatcherMode, format_host_port, unix_time_millis, }; #[cfg(test)] use serde_json::json; @@ -96,10 +96,14 @@ fn service_record_matches_recovery_event( return false; } match event.kind.as_str() { + // The same guarded reasons as the scan, so a live startup the scan + // spared cannot be restarted by an event raised before it was spared. "service.manifest_recoverable" => { - manifest_service_recovery_reason(record, unix_time_millis()).is_some() + stale_manifest_recovery_reason(record, unix_time_millis()).is_some() + } + "service.endpoint_recoverable" => { + unreachable_endpoint_recovery_reason(record, unix_time_millis()).is_some() } - "service.endpoint_recoverable" => endpoint_service_recovery_reason(record).is_some(), "service.healthcheck_recoverable" => crate::engine_ipc::engine_healthcheck_response( paths, &record.engine, @@ -326,17 +330,13 @@ pub(crate) fn find_recoverable_service( { continue; } - if let Some(reason) = manifest_service_recovery_reason(&record, now) { - // `rocm services` persists an engine's own `starting` state into - // the record, so a live server still downloading under the - // interactive serve wait (or left by a Ctrl-C of it) can read - // `starting` past the stale-start window. It is not stale until - // the startup deadline. - if record.status == "starting" && live_startup_still_in_progress(&record, now) { - continue; - } + if let Some(reason) = stale_manifest_recovery_reason(&record, now) { return Ok(Some((record, reason))); } + if manifest_service_recovery_reason(&record, now).is_some() { + // A live startup the guard above spared. + continue; + } if matches!(record.status.as_str(), "ready" | "running") { let Ok(healthcheck) = crate::engine_ipc::engine_healthcheck_response( paths, @@ -367,8 +367,9 @@ pub(crate) fn find_recoverable_service( /// /// A live `running` process that has not opened its port is still the /// interactive serve wait's job: restarting it kills a download the CLI is -/// reporting. A dead pid is not spared. One function for both places that -/// fall back to the endpoint, so neither can drop the guard. +/// reporting. A dead pid is not spared. One function for every place that +/// falls back to the endpoint, scan and event re-check, so none can drop the +/// guard. fn unreachable_endpoint_recovery_reason( record: &ManagedServiceRecord, now_unix_ms: u128, @@ -407,10 +408,39 @@ fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u1 if age_ms >= rocm_core::SERVE_STARTUP_DEADLINE.as_millis() { return false; } - [Some(record.supervisor_pid), record.engine_pid] - .into_iter() - .flatten() - .any(|pid| pid != 0 && process_is_running(pid)) + // A recorded pid alone could have been recycled within the window, so the + // recorded start time must match too. A record with none is the + // best-effort `Matches` `identity_state` gives a legacy record. + [ + Some((record.supervisor_pid, record.supervisor_start_ticks)), + record + .engine_pid + .map(|pid| (pid, record.engine_start_ticks)), + ] + .into_iter() + .flatten() + .any(|(pid, start_ticks)| { + pid != 0 + && rocm_core::identity_state(&ProcessIdentity { pid, start_ticks }) + == IdentityState::Matches + }) +} + +/// Why a stale or failed manifest should be recovered, or `None`. +/// +/// `rocm services` persists an engine's own `starting` state into the +/// record, so a live server still downloading under the interactive serve +/// wait (or left by a Ctrl-C of it) can read `starting` past the stale-start +/// window. It is not stale until the startup deadline. Shared by the scan and +/// the event re-check, so neither can drop the guard. +fn stale_manifest_recovery_reason( + record: &ManagedServiceRecord, + now_unix_ms: u128, +) -> Option { + if record.status == "starting" && live_startup_still_in_progress(record, now_unix_ms) { + return None; + } + manifest_service_recovery_reason(record, now_unix_ms) } fn endpoint_service_recovery_reason(record: &ManagedServiceRecord) -> Option { @@ -1168,6 +1198,18 @@ mod tests { Ok(()) } + fn recovery_event(record: &ManagedServiceRecord, kind: &str) -> AutomationTriggerEvent { + AutomationTriggerEvent { + at_unix_ms: 1, + kind: kind.to_owned(), + source: "managed_service".to_owned(), + watcher_hint: Some("server-recover".to_owned()), + service_id: Some(record.service_id.clone()), + reason: None, + payload: json!({}), + } + } + /// A live process that has not opened its port yet is the interactive /// `rocm serve` wait, not a crash. Endpoint recovery must leave it alone /// until [`rocm_core::SERVE_STARTUP_DEADLINE`]. A dead pid at the same age @@ -1200,6 +1242,18 @@ mod tests { live.write()?; let found_inside = find_recoverable_service(&paths)?; + let event_inside = service_record_matches_recovery_event( + &paths, + &live, + &recovery_event(&live, "service.endpoint_recoverable"), + ); + + // The pid is alive but the recorded start time is not its own: a + // recycled pid is not the startup, so it is recovered. + live.supervisor_start_ticks = Some(u64::MAX); + live.write()?; + let found_recycled = find_recoverable_service(&paths)?; + live.supervisor_start_ticks = None; live.supervisor_pid = u32::MAX; live.engine_pid = None; @@ -1218,14 +1272,26 @@ mod tests { found_inside.is_none(), "a live process inside the startup deadline must not be restarted: {found_inside:?}" ); - assert_eq!( - found_dead.map(|(record, reason)| (record.service_id, reason)), - Some(( - "svc-live-startup".to_owned(), - "endpoint_status_unreachable".to_owned() - )), - "a dead pid is not a startup still in progress" + assert!( + !event_inside, + "an endpoint event raised earlier must not restart it either" ); + for (found, why) in [ + ( + found_recycled, + "a recycled pid is not a startup still in progress", + ), + (found_dead, "a dead pid is not a startup still in progress"), + ] { + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-startup".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + "{why}" + ); + } assert_eq!( found_at_deadline.map(|(record, reason)| (record.service_id, reason)), Some(( @@ -1269,6 +1335,11 @@ mod tests { assert!(now.saturating_sub(live.created_at_unix_ms) >= SERVER_TRANSIENT_STALE_MS); live.write()?; let found_inside = find_recoverable_service(&paths)?; + let event_inside = service_record_matches_recovery_event( + &paths, + &live, + &recovery_event(&live, "service.manifest_recoverable"), + ); live.supervisor_pid = u32::MAX; live.write()?; @@ -1284,6 +1355,10 @@ mod tests { found_inside.is_none(), "a live starting process inside the startup deadline must not be restarted: {found_inside:?}" ); + assert!( + !event_inside, + "a manifest event raised earlier must not restart it either" + ); for (found, why) in [ (found_dead, "a dead pid is a stale start"), (found_at_deadline, "a live start past the deadline is stale"), diff --git a/docs/architecture.md b/docs/architecture.md index dab3eb7520..7b202e8ae9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -89,7 +89,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): `rocm runtimes uninstall` and the storage pruning that removes a registered runtime. `plan_runtime_uninstall` builds the plan (`RuntimeUninstallPlan`, with the `InstallRootDecision` for its folder), `print_runtime_uninstall_plan` shows it, `confirm_and_revalidate_runtime_uninstall` asks and then checks that nothing changed while the prompt was open (`revalidate_runtime_uninstall_plan`), and `apply_runtime_uninstall` unregisters the runtime, clears the active and default state that named it and removes its install root when `should_remove_runtime_install_root` says ROCm CLI owns it (`local_runtime_manifest_matches`, `ensure_runtime_install_root_is_safe_to_remove`). `RuntimeUninstallPlan` also records whether Lemonade's install record points inside the runtime's folder (`lemonade_install_inside`, from `rocm_engine_lemonade::recorded_runtime_dir`), which `print_runtime_uninstall_plan` reports, and whether the folder holds a Lemonade tree at all (`holds_lemonade_tree`), which decides whether an unreadable record is worth reporting; `apply_runtime_uninstall` clears that record once the folder is removed (`RuntimeUninstallResult::lemonade_record`, a `RecordCleanup` that also carries a failed deletion or an unreadable record, worded by `lemonade_record_report_line`). `uninstall_runtime` is the non-interactive entry point. It reads the registry, the active-runtime marker and the path comparison from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and does not depend on `runtimes_cmd.rs`, which calls into it from `runtimes()` and re-exports `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line`. The planning and apply entry points, `RuntimeUninstallPlan`, `RuntimeUninstallConfirmation` and `RuntimeUninstallResult` (with its fields) are `pub(crate)` for `runtimes()`; `revalidate_runtime_uninstall_plan` and the folder-safety helpers are private. Its tests cover planning, revalidation, the folder-removal rules and uninstalling a read-only imported runtime; the uninstall tests that first activate a runtime stay with the activation tests in `runtimes_cmd.rs`. - [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) — mechanical relocation, owns types (Phase 6c, ROCMAI-89): `runtimes()` and its activate/rollback handlers (the activation transaction, including `write_active_runtime_marker`), plus the runtime list rendering; the import and adopt handlers it dispatches to live in [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) and the uninstall plan, confirmation and apply in [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs); registry selection and manifest validation moved to [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), which this module re-exports for the callers that import them from here. `RuntimesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) (part of the `Command`/`Cli` clap definition, the same way `EnginesCommand` does). `RuntimeActivationResult` is `pub(crate)`, read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary`; `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line` (from `runtime_uninstall.rs`) and `ActiveRuntimeMarker` and `active_runtime_marker_path` (from `runtime_registry.rs`) are re-exported for [`storage.rs`](../apps/rocm/src/storage.rs) and [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs); `recover_setup_runtime_registration`, `newest_installed_runtime_manifest` and `setup_runtime_paths` (from `runtime_adopt.rs`) and `normalize_path_for_compare` (from `runtime_registry.rs`) are re-exported for the modules that import them from here, such as [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs), [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) and [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs). [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) is deliberately not one of them: it imports from `runtime_adopt.rs` and `runtime_registry.rs` directly, because going through this module would lead back to it via `runtime_services.rs`. The dependency on the sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) above runs the other way: that module calls into this one (`activate_runtime`, `newest_installed_runtime_manifest`, `recover_setup_runtime_registration`), while this module reaches it only from its test module. - [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) — mechanical relocation, owns types (Phase 6a): `serve()`, plus the engine selection that runs ahead of it (`select_serve_engine`, `recipe_preferred_engine`, `serve_model_ref_for_engine`) and the AIM pre-flight that runs inside it: the Hugging Face token check, the GPU-family profile check and pin, the multi-GPU refusal (`aim_gpu_preflight`), and the board note. The engine-recipe overrides `serve()` applies to the launch hint live in [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs); this module imports them by name. It also holds `collect_serve_notes`, the GPU/device warnings folded into the interactive summary, which was relocated here from `main.rs` with its two tests. Its tests cover engine selection, the AIM pre-flight and the reuse pre-gate; the AIM tests build their hint with `protocol_engine_recipe_hint` and `engine_recipe_with_extra_env` from `serve_recipe.rs`, which `serve()` also calls. -- [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) — interactive `rocm serve` startup status: the stage line (download, load, health) and when that wait stops. [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs)'s `wait_for_launch_startup` calls it for a fresh launch without `--wait`; the historical piped readiness budget stays a separate policy on the same waiter. +- [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) — interactive `rocm serve` startup status: the stage line (download, load, health) and when that wait stops. [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs)'s `wait_for_launch_startup` calls it for a fresh launch without `--wait`, [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) turns its stop into the typed `ServeStartupFailed`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) picks the policy, draws its view on the spinner and supervises only a stop that left the server running; the historical piped readiness budget stays a separate policy on the same waiter. - [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs) — mechanical relocation, owns a type (split out of [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which imports the entry points by name): the pure transforms `serve()` applies to the selected engine's launch recipe hint. `protocol_engine_recipe_hint` maps the catalog's engine record onto the protocol hint, and `render_serve_engine_recipe_lines` prints it for the serve plan. The explicit controls are then layered on top: `--tool-call-parser` (`engine_recipe_with_tool_call_override`), `--gpu-memory-utilization` (`parse_gpu_memory_utilization`, `engine_recipe_with_gpu_memory_utilization_override`), `--temperature`/`--top-p`/`--max-tokens` (`ServeGenerationDefaults`, `engine_recipe_with_generation_defaults`) and `--engine-env` (`engine_recipe_with_extra_env`, `engine_env_ignored_note`, with the env names it refuses to override in `RESERVED_ENGINE_ENV_KEYS`); `engine_recipe_enables_tool_choice` reads the result back. The per-flag rewriters (`set_vllm_tool_call_parser`, `set_vllm_gpu_memory_utilization`, `set_vllm_override_generation_config`, `set_lemonade_generation_defaults`) are private. The entry points above, and `ServeGenerationDefaults`, are `pub(crate)` for `serve()` and for the AIM tests in `serve_cmd.rs`; `RESERVED_ENGINE_ENV_KEYS` is private. Among first-party crates the module depends only on `rocm_core` and `rocm_engine_protocol`, and nothing in it calls back into `serve_cmd.rs`. Its tests cover each override on its own. - [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) — full domain extraction. - [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`stop_recorded_processes`, which hands the recorded PIDs to the routine [`proc_lifecycle.rs`](../crates/rocm-core/src/proc_lifecycle.rs) shares with `rocmd`, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which passes it to [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `run_attached_service` as the stop for a Ctrl-C, the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. This module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which does not import it back. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. @@ -170,7 +170,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses (its engine child runs on the `rocm` binary that `DaemonHost::rocm_binary` finds, before the first write, since `rocmd` has no engine commands, and it polls readiness through the same host), and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor; the binary that runs `supervise` comes from [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), and a restart that cannot find it fails before it writes the record). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and `stop_requested` from [`watchers.rs`](../apps/rocmd/src/watchers.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. - [`watchers.rs`](../apps/rocmd/src/watchers.rs) — the module root of the built-in watchers: the registry (`reconcile_watcher_snapshots`) and the event collector and dispatcher (`evaluate_watchers`, `collect_automation_events_with_gpu_snapshot`, which takes the GPU snapshot as a closure so tests can inject one, and `evaluate_watchers_for_events`), which route each event to its watcher's handler. `find_recoverable_service` and `stop_requested` are re-exported from here because [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s and [`service.rs`](../apps/rocmd/src/service.rs)'s tests reach them by the `crate::watchers::` path. Each watcher's handler lives in the child below that carries its name, together with the policy they share, and a child reaches only [`policy.rs`](../apps/rocmd/src/watchers/policy.rs) and, for the thermal-protect watcher, [`gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) among the others. - [`watchers/policy.rs`](../apps/rocmd/src/watchers/policy.rs) — what a watcher does in each mode and the proposals it queues: `WatcherPolicyAction` and `watcher_policy_action` (observe, queue a proposal or run contained, chosen by the watcher mode), `queue_proposal` and `queue_proposal_with_arguments` with the action-to-tool and action-to-arguments tables behind them, and `pending_proposal_exists`, the check the thermal-protect and server-recover handlers use so they do not queue a duplicate of a proposal that is still pending. It uses no other watcher module. - - [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) — the `server-recover` watcher: finding a managed service that needs recovery (`find_recoverable_service`, from a failed or stale manifest, an unreachable endpoint or a failing engine healthcheck, skipping a service whose stop request still stands via `stop_requested`, and sparing a live `running` or `starting` service younger than `rocm_core::SERVE_STARTUP_DEADLINE` whose port is not open yet via `live_startup_still_in_progress`), the recovery event kinds and plain-language reasons, the backoff that spaces recovery attempts (`server_recover_due`), and the handler that restarts the service or queues a restart proposal by watcher mode. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s `restart_managed_service`. + - [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) — the `server-recover` watcher: finding a managed service that needs recovery (`find_recoverable_service`, from a failed or stale manifest, an unreachable endpoint or a failing engine healthcheck, skipping a service whose stop request still stands via `stop_requested`, and sparing a live `running` or `starting` service younger than `rocm_core::SERVE_STARTUP_DEADLINE` whose port is not open yet via `live_startup_still_in_progress`, which checks the recorded start time with `rocm_core::identity_state` so a recycled pid is not spared; the scan and the event re-check share the guarded reasons), the recovery event kinds and plain-language reasons, the backoff that spaces recovery attempts (`server_recover_due`), and the handler that restarts the service or queues a restart proposal by watcher mode. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s `restart_managed_service`. - [`watchers/gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) — the `gpu-metrics` watcher and the GPU readings it works from: the telemetry-event handler (`handle_gpu_metrics_event`, which records telemetry and takes no action), the interval check (`gpu_metrics_due`), the one-line snapshot summary, and the reading of amd-smi monitor entries into thermal-pressure and memory-pressure events (`gpu_pressure_events`, `gpu_pressure_event`) with the thresholds they use. It uses [`persistence.rs`](../apps/rocmd/src/persistence.rs) and no other watcher module, and [`gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) reads its interval and thresholds. - [`watchers/gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) — the `gpu-thermal-protect` watcher: the pressure-event handler (`handle_gpu_thermal_protect_event`), which records the event and, outside observe mode, picks the one running managed service as the target and queues a reviewed stop request unless one is already pending, and the interval check (`gpu_thermal_protect_due`). It uses [`gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) for the interval and thresholds, [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`common.rs`](../apps/rocmd/src/common.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). - [`watchers/therock_update.rs`](../apps/rocmd/src/watchers/therock_update.rs) — the `therock-update` watcher: the schedule check (`therock_update_due`, every six hours) and the handler, which in observe mode records a reminder, in propose mode also queues an update-check proposal, and in contained mode runs the read-only `check_updates` sandbox tool and records a notification when an update or repair is available, never applying one. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`sandbox.rs`](../apps/rocmd/src/sandbox.rs), [`sandbox_tool.rs`](../apps/rocmd/src/sandbox_tool.rs), [`common.rs`](../apps/rocmd/src/common.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). diff --git a/docs/testing.md b/docs/testing.md index 97f6facde5..1fe259bca1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2002,7 +2002,8 @@ previous request returned (`InferenceProbeThrottle`, tested under a fake clock by `inference_probe_backs_off_to_the_retry_interval_after_listing`). `rocmd` does not restart a live `running` service whose endpoint does not answer, or a live `starting` one past the stale-start window, while it is younger than that -deadline; a recoverable engine healthcheck is still restarted. +deadline (a pid whose recorded start time does not match is not live); a +recoverable engine healthcheck is still restarted. Piped output keeps the 45-second status budget. Like `--wait`, the interactive wait first prints the service id, `rocm logs --service --follow` and a public bind's key (`announces_wait` in From eb3ce2a0d8adc0517862a67f04114df535b94bf9 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Fri, 13 Nov 2026 11:53:24 +0200 Subject: [PATCH 06/14] test(serve): add a @requires-gpu PTY scenario for interactive serve startup Scenario serve-55 runs `rocm serve` under a real terminal, checks the wait line names a log command that actually runs, and that the serve exits 0 with the endpoint listing the model. Also documents that the inference probe runs inline in the startup wait. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 5 ++ docs/testing.md | 9 ++- .../features/model_serving.feature | 24 ++++-- tests/e2e-cucumber/tests/e2e/serving_steps.rs | 73 +++++++++++++++++++ 4 files changed, 101 insertions(+), 10 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index e1783d9005..0a7248ad10 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -648,6 +648,11 @@ impl ServeStartupStop { /// "serving" still means an inference request came back. Once the model is /// listed, the inference request is spaced by the policy's /// [`InferenceProbeBackoff`] rather than sent every poll. +/// +/// The probe runs inline, so a poll that sends one can take up to the probe +/// timeout: the spinner does not tick and an engine failure is not read until +/// it returns. The backoff bounds how often that happens; it is not moved off +/// the polling thread. #[allow(clippy::too_many_arguments)] pub(crate) fn wait_for_serve_startup( engine_name: &str, diff --git a/docs/testing.md b/docs/testing.md index b425a1de60..092e2057ec 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2057,10 +2057,11 @@ guidance (a `--wait` that sees the server exit now records the event too); when the engine failed but its process is still up, the message names the stop command. Only a stall or the deadline leaves the server running for the background helper to supervise. -There is no Gherkin scenario for this yet. The harness can drive a PTY -(`download_progress_pty.feature`), but interactive `rocm serve` needs a live -engine: the no-GPU mock lane stubs the HTTP endpoint and never runs -`rocm serve`, so a scenario would be `@requires-gpu`. `decide_serve_startup` and `wait_for_serve_startup` in +Scenario `serve-55` in `model_serving.feature` drives it under a PTY. It +needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes only; +it checks the wait line, that the named `rocm logs --service` command runs, +and that the serve exits 0 with the model listed. The stall, deadline and +engine-failed branches need a controllable engine, so `decide_serve_startup` and `wait_for_serve_startup` in [serve_progress.rs](../apps/rocm/src/serve_progress.rs) are the state matrix, covered by: diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index c127384159..248f7d1c6e 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -2,12 +2,12 @@ Feature: Model serving # Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows # download, load, and health on the status line until the server is ready or - # the startup wait fails. There is no scenario for it yet: the harness can - # drive a PTY (download_progress_pty.feature), but interactive `rocm serve` - # needs a live engine, and the no-GPU mock lane stubs the HTTP endpoint - # without running `rocm serve`, so one would be `@requires-gpu`. The state matrix is - # covered without a GPU by `decide_serve_startup` and `wait_for_serve_startup` - # in apps/rocm/src/serve_progress.rs. + # the startup wait fails. Scenario serve-55 drives it under a real PTY; it + # needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes + # only. The stall, deadline and engine-failed branches need a controllable + # engine, so the no-GPU lane cannot reach them; `decide_serve_startup` and + # `wait_for_serve_startup` in apps/rocm/src/serve_progress.rs are their + # state matrix. # Per-platform expectations (pass / xfail / skip) are resolved at runtime from # the host capability probe + expectations.toml, keyed by the @id tag — not by @@ -867,3 +867,15 @@ Feature: Model serving When the user sends a streaming request to the Anthropic messages endpoint Then the response is delivered as Anthropic-style server-sent events And the streamed content reassembles into a complete reply + + # Interactive `rocm serve` under a real terminal: while it waits it names the + # command that follows the engine's log, then it exits 0 once the endpoint + # lists the model. The named command is run, not just read: a hint that + # points at a command that does not work would still pass a string check. + @id:serve-interactive-startup-progress @requires-gpu + Scenario: serve-55 - Interactive serve shows how to follow startup and exits when the model is served + Given a managed runtime is active + When the user serves a model on GPU under a real terminal + Then the terminal names the command that follows the server's log + And the named log command runs successfully + And the serve exits cleanly and the endpoint lists the model diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index faddf85a07..ef78d56ebe 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -613,6 +613,79 @@ fn claim_relaunch() -> bool { .is_ok() } +#[when("the user serves a model on GPU under a real terminal")] +async fn user_serves_gpu_model_under_pty(world: &mut E2eWorld) { + let (model, engine, _) = host_serve_target(); + // Same free-port precondition as `serve_gpu_model`: a leaked server from a + // prior scenario would answer for this one. + let _ = ensure_serve_port_free().await; + // Summary mode (the interactive status line) is `stdout is a terminal` and + // neither `--verbose` nor `--foreground`, so the PTY is the whole setup. + let session = TuiSession::spawn(world, &["serve", model, "--engine", engine, "--managed"]) + .unwrap_or_else(|e| panic!("failed to spawn `rocm serve` under a pty: {e}")); + world.tui = Some(session); +} + +/// The id the wait line's `rocm logs --service --follow` names. +fn followed_service_id(screen: &str) -> Option { + let rest = screen.split("rocm logs --service ").nth(1)?; + let id = rest.split_whitespace().next()?; + Some(id.trim_matches('\'').to_owned()) +} + +#[then("the terminal names the command that follows the server's log")] +async fn terminal_names_log_command(world: &mut E2eWorld) { + let timeout = Duration::from_secs(serve_timeout_for(world)); + let session = world.tui.as_mut().expect("no pty session for `rocm serve`"); + session + .wait_for_screen("rocm logs --service ", timeout) + .await + .unwrap_or_else(|e| panic!("the wait line never named a log command: {e}")); + let screen = session.screen_text(); + assert!( + screen.contains("Ctrl-C stops waiting"), + "the wait line does not say Ctrl-C stops waiting, not the server:\n{screen}" + ); + followed_service_id(&screen) + .unwrap_or_else(|| panic!("no service id after `rocm logs --service`:\n{screen}")); +} + +#[then("the named log command runs successfully")] +async fn named_log_command_runs(world: &mut E2eWorld) { + let screen = world + .tui + .as_ref() + .expect("no pty session for `rocm serve`") + .screen_text(); + let id = followed_service_id(&screen) + .unwrap_or_else(|| panic!("no service id after `rocm logs --service`:\n{screen}")); + // Without `--follow`, which never returns. The same id and subcommand the + // hint names, so a hint that points at nothing fails here. + let (stdout, stderr, rc) = crate::run_rocm(world, &["logs", "--service", &id]); + assert_eq!( + rc, 0, + "`rocm logs --service {id}` failed.\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); +} + +#[then("the serve exits cleanly and the endpoint lists the model")] +async fn serve_exits_and_endpoint_lists_model(world: &mut E2eWorld) { + let timeout = Duration::from_secs(serve_timeout_for(world)); + let (_, _, ready_substr) = host_serve_target(); + world + .tui + .as_mut() + .expect("no pty session for `rocm serve`") + .assert_exits_cleanly("the interactive serve", timeout) + .await; + assert!( + model_is_ready(&MODELS_URL, Some(ready_substr), 30).await, + "serve exited 0 but {} does not list {ready_substr}", + MODELS_URL.as_str() + ); + world.endpoint = Some(SERVE_BASE_URL.to_string()); +} + #[given("a model is being served on GPU")] async fn setup_gpu_model(world: &mut E2eWorld) { serve_gpu_model(world).await; From e4933a9731113b2f1c5c494d8754661914ee677c Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 16 Nov 2026 13:30:45 +0200 Subject: [PATCH 07/14] fix(serve): record an unlisted stalled launch as starting; cover startup exit on the mock lane A stall or deadline stop now records the status that matches how far the endpoint got (`starting` when it never listed the model), instead of `running`, which elsewhere means the model is listed. When the engine reports failed but its process is still up, the message now says a running `rocm daemon` with `server-recover` may restart it. Scenario serve-60 drives an engine that exits during interactive startup on the no-GPU mock lane: an `e2e-oom-fault-injection` build swaps the managed child for a stand-in when ROCM_E2E_SIMULATE_STARTUP_EXIT=1. It checks the non-zero exit and message, runs the `rocm logs --service` command the failure names, and checks the record is settled `stopped`. The docs say the silent limits arm while the endpoint has not listed the model, which is what the code checks, rather than "the port stays closed". Signed-off-by: Roman Inflianskas --- README.md | 6 +- apps/rocm/src/serve_cmd.rs | 5 +- apps/rocm/src/serve_progress.rs | 12 +- apps/rocm/src/service_launch.rs | 2 +- apps/rocm/src/service_launch/readiness.rs | 30 +++++ apps/rocm/src/service_launch/spawn.rs | 22 ++++ apps/rocm/src/service_launch/start.rs | 21 ++-- docs/architecture.md | 4 +- docs/manual-testing.md | 2 +- docs/testing.md | 12 +- .../features/model_serving.feature | 24 +++- tests/e2e-cucumber/tests/e2e/serving_steps.rs | 114 ++++++++++++++++++ 12 files changed, 227 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 6b0fc91422..ae957fa8f5 100644 --- a/README.md +++ b/README.md @@ -1261,9 +1261,9 @@ and a quick smoke test (time to first token and approximate tokens per second). A model that is still loading is not reported as a finished "not ready" summary. The wait fails, and `rocm serve` exits non-zero, when the engine reports failure or its process exits, when a download percent stops -advancing for 3 minutes, when the port stays closed with no engine log for -2 minutes, when a load log line stops changing for 5 minutes with the port -still closed, or after 30 minutes in all. A stall leaves the server running: +advancing for 3 minutes, when the endpoint has not listed the model and there is no engine log for +2 minutes, when a load log line stops changing for 5 minutes with the endpoint +not yet listing the model, or after 30 minutes in all. A stall leaves the server running: the message names `rocm logs --service ` and `rocm services stop --yes`, and the background helper does not restart a live server for being slow until those 30 minutes pass. Engine log diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 54d7473faa..0791d051de 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -865,8 +865,8 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { // (probe returns `None`). The E2E-only // backend-failure scenario bypasses this host precondition so the black-box // test reaches Lemonade's backend boundary without real GPU hardware, and the - // E2E-only OOM-launch fault injection likewise stands in for the GPU it does - // not have. + // E2E-only OOM-launch fault injection and startup-exit stand-in engine + // likewise stand in for the GPU they do not have. // // The scripted free-space override opens the same waiver: it exists to refuse a // Lemonade install on a drive that cannot hold it, which `rocm serve` only @@ -881,6 +881,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { if !cpu_only && !scripted_backend_failure && !e2e_simulate_oom_launch() + && !crate::service_launch::e2e_simulate_startup_exit() && !reuse_existing && host_has_no_usable_gpu { diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index 0a7248ad10..ee1dfb85f2 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -48,10 +48,10 @@ pub(crate) const DOWNLOAD_STALL: Duration = Duration::from_mins(3); /// Interactive summary mode follows the stage instead. pub(crate) const PIPED_SERVE_STATUS_BUDGET: Duration = Duration::from_secs(45); -/// Port down and no meaningful log line. +/// Endpoint not listing the model and no meaningful log line. pub(crate) const SILENT_UNREACHABLE: Duration = Duration::from_mins(2); -/// A load line that stops changing while the port is still down. +/// A load line that stops changing while the endpoint has not listed the model. pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5); const POLL: Duration = Duration::from_millis(250); @@ -628,7 +628,8 @@ impl ServeStartupStop { let stop = service_stop_hint(service_id); let _ = write!( message, - "Its process is still up (service {service_id}). The log is `{logs}`; `{stop}` stops it." + "Its process is still up (service {service_id}). The log is `{logs}`; `{stop}` stops it. \ + A running `rocm daemon` with `server-recover` enabled may restart it first." ); } else { let _ = write!( @@ -1719,6 +1720,11 @@ mod tests { alive, "{reason:?} alive={alive}: {message}" ); + assert_eq!( + message.contains("may restart it first"), + alive, + "{reason:?} alive={alive}: {message}" + ); assert_eq!( message.contains("Its process is still up"), alive, diff --git a/apps/rocm/src/service_launch.rs b/apps/rocm/src/service_launch.rs index ae5c2fdb5e..036e4f87c7 100644 --- a/apps/rocm/src/service_launch.rs +++ b/apps/rocm/src/service_launch.rs @@ -68,7 +68,7 @@ pub(crate) use helper_autostart::{ ensure_background_helper_running, ensure_background_helper_running_quiet, }; pub(crate) use launch_transaction::{managed_service_reuse_refusal, no_usable_gpu_refusal}; -pub(crate) use readiness::e2e_simulate_oom_launch; +pub(crate) use readiness::{e2e_simulate_oom_launch, e2e_simulate_startup_exit}; #[cfg(not(windows))] pub(crate) use spawn::{ attach_background_stdio, detach_background_command, managed_engine_startup_failure_detail, diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index c8befa5246..adc9f54829 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -223,6 +223,36 @@ pub(crate) const fn e2e_simulate_oom_launch() -> bool { false } +/// Test-only switch for the mock lane, compiled in only with +/// `e2e-oom-fault-injection` and read only on Linux: a managed launch spawns a +/// stand-in engine that writes one log line and exits during startup, in place +/// of the real `rocm` serve child, so the interactive startup wait's +/// process-exited path runs end to end on a host with no GPU. Everything else +/// about the launch (record, log, wait, settling) is the production path. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +pub(crate) fn e2e_simulate_startup_exit() -> bool { + std::env::var_os(SIMULATE_STARTUP_EXIT_ENV).is_some_and(|value| value == "1") +} + +/// Release builds, and hosts other than Linux, carry no such switch. +#[cfg(not(all(feature = "e2e-oom-fault-injection", target_os = "linux")))] +pub(crate) const fn e2e_simulate_startup_exit() -> bool { + false +} + +/// The variable [`e2e_simulate_startup_exit`] reads; the e2e step spells the +/// same name, since the shipped crate must not depend on the harness. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +const SIMULATE_STARTUP_EXIT_ENV: &str = "ROCM_E2E_SIMULATE_STARTUP_EXIT"; + +/// What the stand-in engine runs: one meaningful log line, then an exit after +/// the spawn's immediate-exit check, so the launch is published and the +/// startup wait (not the spawn) is what sees the process end. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +pub(super) const SIMULATED_STARTUP_EXIT_SCRIPT: &str = + "echo 'e2e stand-in engine: loading model weights'; sleep 2; \ + echo 'e2e stand-in engine: exiting during startup'; exit 1"; + /// Test-only (see [`e2e_simulate_oom_launch`]): fabricate the on-disk state of a /// managed vLLM launch that spawned, OOM'd, and never became ready, then return /// the launch report the summary path would see for it — `already_running: false` diff --git a/apps/rocm/src/service_launch/spawn.rs b/apps/rocm/src/service_launch/spawn.rs index 9b3252efc0..ea6f2ac3b3 100644 --- a/apps/rocm/src/service_launch/spawn.rs +++ b/apps/rocm/src/service_launch/spawn.rs @@ -392,6 +392,23 @@ fn test_spawn_hook() -> Option>> { TEST_SPAWN_HOOK.with(|hook| hook.borrow().clone()) } +/// The stand-in engine behind [`super::readiness::e2e_simulate_startup_exit`]: +/// a shell that appends to the launch's own log and exits during startup. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +fn simulated_startup_exit_spawner(log_path: &Path) -> Box> { + let log_path = log_path.to_path_buf(); + Box::new(move |_command| { + let log = fs::OpenOptions::new().append(true).open(&log_path)?; + let stderr = log.try_clone()?; + ProcessCommand::new("sh") + .args(["-c", super::readiness::SIMULATED_STARTUP_EXIT_SCRIPT]) + .stdin(Stdio::null()) + .stdout(log) + .stderr(stderr) + .spawn() + }) +} + #[cfg(not(windows))] type SpawnOverride<'a> = dyn Fn(&mut ProcessCommand) -> io::Result + 'a; @@ -519,6 +536,11 @@ fn launch_managed_engine_child_locked( let hook = test_spawn_hook(); #[cfg(all(test, not(windows)))] let spawn_override = spawn_override.or(hook.as_deref()); + #[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] + let stand_in = super::readiness::e2e_simulate_startup_exit() + .then(|| simulated_startup_exit_spawner(&record.log_path)); + #[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] + let spawn_override = spawn_override.or(stand_in.as_deref()); let mut child = match spawn_override { Some(spawn) => spawn(&mut command), None => command.spawn(), diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 0c4cde39ff..07da253e46 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -175,13 +175,12 @@ pub(crate) fn start_managed_service( } .into()); } - // A stall or the deadline left the server up: `running`, which `rocmd` - // leaves alone until the startup deadline. - let launch_status = if startup_stop.is_some() { - "running" - } else { - status_for_readiness(readiness) - }; + // A stall or the deadline left the server up. Its status follows how far + // the endpoint got, as for any other launch: `running` only when the model + // is listed, `starting` when it never opened, so `rocm services`, dash and + // chat do not show a server that has not listed its model as up. + // `rocmd` leaves a live `starting` record alone until the startup deadline. + let launch_status = status_for_readiness(readiness); let recorded = record_wait_outcome( paths, &mut record, @@ -577,8 +576,8 @@ mod tests { } /// A fresh launch whose wait stops on a live server: the wait is announced - /// first, the server is left running and recorded `running` without an - /// inference verification, and the typed failure names that service's + /// first, the server is left running and recorded `starting` (its endpoint + /// never listed the model) without an inference verification, and the typed failure names that service's /// logs and stop commands and asks `serve` to supervise it. #[cfg(not(windows))] #[test] @@ -596,7 +595,9 @@ mod tests { } assert_eq!(launch.announced, [service_id], "the wait is announced once"); let record = launch.record?; - assert_eq!(record.status, "running", "the live server stays recorded"); + // The endpoint never listed the model, so the record says `starting`, not + // `running`, which elsewhere means "lists the model". + assert_eq!(record.status, "starting", "the live server stays recorded"); assert_eq!(record.inference_verified_at_unix_ms, None); Ok(()) } diff --git a/docs/architecture.md b/docs/architecture.md index d941111891..d481506fcf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -121,8 +121,8 @@ The `rocm` binary: the clap command tree, a handler module for most commands, an | [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) | Bind-host validation and the endpoint API key checks that gate a public bind (`validate_bind_host`, `resolve_endpoint_auth`), including on a respawn. | relocation | | [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) | Starting the background automation helper when a watcher needs it, with the autostart claim that keeps concurrent callers from spawning two. | relocation | | [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) | The locked launch decision: port allocation against live records and the OS, failed-launch cleanup, and whether a live service may answer a `serve` request. The cleanup terminates a spawned child through the same `rocm_core::terminate_recorded_pid_within` routine a stop uses, so on Windows its descendants are taken down too. A spawned child whose start token was not captured, on a host that records tokens (Linux, and Windows, which records the process creation time), is not signalled (`launch_cleanup_refusal`): it was gone before it could be captured and its pid may belong to another process. | relocation | -| [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) | Waiting on a launched or live managed service: the readiness budget, the interactive startup wait it runs through [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) (`wait_for_launch_startup`), process liveness, how a wait ends (`settle_ended_launch`), and the OOM fault-injection seam for end-to-end tests. | relocation | -| [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) | Spawning the detached managed engine child both serve paths share: process and stdio plumbing and the serialized spawn transaction (`spawn_managed_engine_child`). | relocation | +| [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) | Waiting on a launched or live managed service: the readiness budget, the interactive startup wait it runs through [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) (`wait_for_launch_startup`), process liveness, how a wait ends (`settle_ended_launch`), and the OOM and startup-exit fault-injection seams for end-to-end tests. | relocation | +| [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) | Spawning the detached managed engine child both serve paths share: process and stdio plumbing, the serialized spawn transaction (`spawn_managed_engine_child`), and the stand-in engine an `e2e-oom-fault-injection` build spawns for `ROCM_E2E_SIMULATE_STARTUP_EXIT`. | relocation | | [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) | The background launch path (`start_managed_service`), which returns its launch report or the typed `ServeStartupFailed`, and the plain line-by-line rendering of that report. | relocation | | [`service_launch/test_support.rs`](../apps/rocm/src/service_launch/test_support.rs) | Test fixtures shared by more than one `service_launch/` child. | flat | | [`service_records.rs`](../apps/rocm/src/service_records.rs) | Removing a managed-service record and every file it owns, and the `rocm services prune` plan, render and apply steps with their age gate. `remove_managed_service_record_with` takes `--force` and the read of each recorded PID: `--force` removes a record only when none of its recorded processes is confirmed running, one could not be queried and a stop was requested, and reports which. | relocation | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 0943302649..b62cb579a9 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -574,7 +574,7 @@ download percent, a load line, or a wait for the model to serve. A model that is still loading after 45 seconds (the old wait, which piped output still uses) keeps that line; it is not replaced by a finished "not ready yet" summary. The wait fails with a non-zero exit only when the engine fails or -exits, a download percent stops for 3 minutes, the port stays closed with no +exits, a download percent stops for 3 minutes, the endpoint has not listed the model with no engine log for 2 minutes, a load line stops changing for 5 minutes, or 30 minutes pass. When the endpoint answers, the command prints the deployment summary and returns to the shell. If the wait diff --git a/docs/testing.md b/docs/testing.md index d83d48f8d6..c041bc8661 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2218,8 +2218,8 @@ serve or that wait fails. A download that is still moving is not cut off at 45 seconds, and a failure names `rocm logs --service ` and `rocm services stop --yes` instead of printing `Deployment summary (not ready yet)`. The wait fails when the engine fails or -exits, a download percent stops for 3 minutes (`DOWNLOAD_STALL`), the port -stays closed with no log for 2 minutes (`SILENT_UNREACHABLE`), a load line +exits, a download percent stops for 3 minutes (`DOWNLOAD_STALL`), the endpoint +has not listed the model, with no log, for 2 minutes (`SILENT_UNREACHABLE`), a load line stops changing for 5 minutes (`SILENT_LOAD`), or after 30 minutes (`rocm_core::SERVE_STARTUP_DEADLINE`). Engine log lines are masked of credentials before they are shown (`redact_log_line`). Once the endpoint lists the model, @@ -2246,7 +2246,13 @@ deadline leaves the server running for the background helper to supervise. Scenario `serve-59` in `model_serving.feature` drives it under a PTY. It needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes only; it checks the wait line, that the named `rocm logs --service` command runs, -and that the serve exits 0 with the model listed. The stall, deadline and +and that the serve exits 0 with the model listed. Scenario `serve-60` is the +failure side on the no-GPU mock lane: with `ROCM_E2E_SIMULATE_STARTUP_EXIT=1`, +an `e2e-oom-fault-injection` build (Linux only) spawns a stand-in engine that +logs one line and exits during startup, in place of the real child. It checks +the non-zero exit and the "server process exited" message, runs the +`rocm logs --service` command the failure names, and checks the record is +`stopped` (`cargo xtask e2e -- -n "serve-60"`). The stall, deadline and engine-failed branches need a controllable engine, so `decide_serve_startup` and `wait_for_serve_startup` in [serve_progress.rs](../apps/rocm/src/serve_progress.rs) are the state matrix, covered by: diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 2b749f42e2..47d5bdfde1 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -4,8 +4,9 @@ Feature: Model serving # download, load, and health on the status line until the server is ready or # the startup wait fails. Scenario serve-59 drives it under a real PTY; it # needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes - # only. The stall, deadline and engine-failed branches need a controllable - # engine, so the no-GPU lane cannot reach them; `decide_serve_startup` and + # only. serve-60 covers an engine that exits during startup on the no-GPU + # mock lane with a stand-in engine. The stall, deadline and engine-failed + # branches need a controllable engine, so no lane reaches them; `decide_serve_startup` and # `wait_for_serve_startup` in apps/rocm/src/serve_progress.rs are their # state matrix. @@ -961,3 +962,22 @@ Feature: Model serving Then the terminal names the command that follows the server's log And the named log command runs successfully And the serve exits cleanly and the endpoint lists the model + + # The failure side of serve-59, on the mock lane. A real engine cannot start on + # a host with no GPU, so the `e2e-oom-fault-injection` build swaps the managed + # child for a stand-in that writes one log line and exits during startup + # (`ROCM_E2E_SIMULATE_STARTUP_EXIT`); the record, the log, the startup wait and + # the settling are the production path. The hook is absent from shipped + # binaries, so @requires-oom-fault-injection keeps this off the self-hosted + # lanes. The log command the failure names is run, and the record is checked, + # so the message cannot claim a log or a settled launch that is not there. + @id:serve-startup-exit-names-log @requires-no-gpu @requires-os:linux @requires-oom-fault-injection + Scenario: serve-60 - An engine that exits during interactive startup fails the serve and names its log + Given a managed engine will exit while it starts + When the user serves a model with that engine under a real terminal + Then the serve fails saying the server process exited + And the failure names the command that shows the server's log + And that log command shows the engine's last line + And the failed launch is recorded as stopped + And no background automation daemon was left behind + diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 9c72725ea4..3efa08477b 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -695,6 +695,120 @@ async fn serve_exits_and_endpoint_lists_model(world: &mut E2eWorld) { world.endpoint = Some(SERVE_BASE_URL.to_string()); } +#[given("a managed engine will exit while it starts")] +async fn arm_startup_exit_stand_in(world: &mut E2eWorld) { + // Consumed by `e2e_simulate_startup_exit` in + // `apps/rocm/src/service_launch/readiness.rs`; spelled out on both sides + // for the same reason as `ROCM_E2E_SIMULATE_OOM_LAUNCH`. A typo does not + // arm the stand-in, the no-GPU pre-flight refuses, and the Then steps fail. + world + .command_env + .push(("ROCM_E2E_SIMULATE_STARTUP_EXIT", "1".into())); +} + +#[when("the user serves a model with that engine under a real terminal")] +async fn serve_with_startup_exit_stand_in(world: &mut E2eWorld) { + let mut session = TuiSession::spawn( + world, + &[ + "serve", + OOM_LAUNCH_MODEL, + "--engine", + "vllm", + "--env-id", + "e2e-startup-exit", + ], + ) + .unwrap_or_else(|e| panic!("failed to spawn `rocm serve` under a pty: {e}")); + let code = session + .wait_for_exit_code(interactive_summary_timeout()) + .await + .unwrap_or_else(|e| panic!("`rocm serve` did not exit: {e}")); + world.tui = Some(session); + world.cli_rc = Some(code); +} + +/// The final screen with all whitespace removed: the 80-column PTY wraps the +/// failure message across rows. +fn startup_exit_screen(world: &E2eWorld) -> (String, String) { + let screen = world + .tui + .as_ref() + .expect("no pty session for `rocm serve`") + .screen_text(); + (screen_without_whitespace(&screen), screen) +} + +/// The id in the failure's "The log is `rocm logs --service `". +fn failed_launch_service_id(world: &E2eWorld) -> String { + let (flat, screen) = startup_exit_screen(world); + let marker = "Thelogis`rocmlogs--service"; + let rest = flat + .split(marker) + .nth(1) + .unwrap_or_else(|| panic!("the failure does not name `rocm logs --service`:\n{screen}")); + let id = rest.split('`').next().unwrap_or_default().trim_matches('\''); + assert!(!id.is_empty(), "no service id in the log hint:\n{screen}"); + id.to_owned() +} + +#[then("the serve fails saying the server process exited")] +async fn serve_fails_with_process_exited(world: &mut E2eWorld) { + let (flat, screen) = startup_exit_screen(world); + let code = world.cli_rc; + assert!( + code.is_some_and(|code| code != 0), + "an engine that exited during startup must fail the serve (exit {code:?}):\n{screen}" + ); + assert!( + flat.contains("failed:theserverprocessexited"), + "the failure must say the server process exited:\n{screen}" + ); + assert!( + flat.contains("Theserverdidnotstart"), + "the failure must say the server did not start, not that it is still running:\n{screen}" + ); +} + +#[then("the failure names the command that shows the server's log")] +async fn failure_names_log_command(world: &mut E2eWorld) { + failed_launch_service_id(world); +} + +#[then("that log command shows the engine's last line")] +async fn log_command_shows_engine_line(world: &mut E2eWorld) { + let id = failed_launch_service_id(world); + let (stdout, stderr, rc) = crate::run_rocm(world, &["logs", "--service", &id]); + assert_eq!( + rc, 0, + "`rocm logs --service {id}` failed.\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); + assert!( + stdout.contains("e2e stand-in engine: exiting during startup"), + "`rocm logs --service {id}` does not show the engine's last line:\n{stdout}" + ); +} + +#[then("the failed launch is recorded as stopped")] +async fn failed_launch_recorded_stopped(world: &mut E2eWorld) { + let id = failed_launch_service_id(world); + let (stdout, stderr, rc) = crate::run_rocm(world, &["services", "list", "--all", "--json"]); + assert_eq!(rc, 0, "`rocm services list --all --json` failed:\n{stderr}"); + let records: serde_json::Value = serde_json::from_str(&stdout) + .unwrap_or_else(|e| panic!("services list --json did not parse: {e}\n{stdout}")); + let status = records + .as_array() + .into_iter() + .flatten() + .find(|record| record.get("service_id").and_then(serde_json::Value::as_str) == Some(&id)) + .and_then(|record| record.get("status").and_then(serde_json::Value::as_str)) + .unwrap_or_else(|| panic!("no record for {id}:\n{stdout}")) + .to_owned(); + // `stopped`, not `failed`: `rocmd` restarts a `failed` record with no stop + // marker, behind the error the user just read. + assert_eq!(status, "stopped", "{stdout}"); +} + #[given("a model is being served on GPU")] async fn setup_gpu_model(world: &mut E2eWorld) { serve_gpu_model(world).await; From ced7f76175b8c3b984f67943f8c6b5395d207cec Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 16 Nov 2026 14:18:51 +0200 Subject: [PATCH 08/14] fix(serve): say a stopped wait's server may be restarted after the startup deadline A stall or deadline stop leaves the server running, and rocmd spares a live startup only until SERVE_STARTUP_DEADLINE. The message now says a running daemon with server-recover may restart it after that. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index ee1dfb85f2..bfb8feb147 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -621,7 +621,9 @@ impl ServeStartupStop { let stop = service_stop_hint(service_id); let _ = write!( message, - "The server is still running (service {service_id}). Follow it with `{logs}`, or stop it with `{stop}`." + "The server is still running (service {service_id}). Follow it with `{logs}`, or stop it with `{stop}`. \ + If it is still not answering {minutes} minutes after it started, a running `rocm daemon` \ + with `server-recover` enabled may restart it." ); } else if self.process_alive { // The engine said failed but its process has not exited yet. @@ -1742,7 +1744,22 @@ mod tests { ServeStartupStopReason::SilentLoad, ServeStartupStopReason::Deadline, ] { - assert!(stop(reason, true).leaves_server_running(), "{reason:?}"); + let stop = stop(reason, true); + assert!(stop.leaves_server_running(), "{reason:?}"); + // The restart this names is real: `rocmd` spares a live startup only + // until the deadline, which + // `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one` + // and `recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline` + // in apps/rocmd/src/watchers/server_recover.rs prove. + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!( + message.contains(&format!( + "{minutes} minutes after it started, a running `rocm daemon` with \ + `server-recover` enabled may restart it" + )), + "{reason:?}: {message}" + ); } } From f04ebd134b7c24c10f108b657af0f932aff6769f Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 16 Nov 2026 15:04:54 +0200 Subject: [PATCH 09/14] fix(serve): follow the startup only when stderr is a terminal too The interactive wait was chosen by stdout alone, while its status line and failures go to stderr, so `rocm serve m 2>log` in a terminal could wait up to 30 minutes with nothing drawn. A redirected stderr now keeps the piped 45-second status budget. Also applies rustfmt to the previous commit. Signed-off-by: Roman Inflianskas --- README.md | 2 +- apps/rocm/src/cli.rs | 2 +- apps/rocm/src/serve_cmd.rs | 16 ++++----- apps/rocm/src/serve_progress.rs | 35 +++++++++++++++++++ apps/rocm/src/service_launch/readiness.rs | 3 +- docs/manual-testing.md | 2 +- docs/testing.md | 2 +- tests/e2e-cucumber/tests/e2e/serving_steps.rs | 6 +++- 8 files changed, 52 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index ae957fa8f5..3acdfa5782 100644 --- a/README.md +++ b/README.md @@ -1270,7 +1270,7 @@ restart a live server for being slow until those 30 minutes pass. Engine log lines shown on the status line have credentials (the endpoint key, `--api-key` and token values, `hf_` tokens) masked. Once the server lists the model, the wait checks whether it can answer with a one-token request, first right away and -then at growing intervals up to every 15 seconds. Piped output (not a terminal) keeps +then at growing intervals up to every 15 seconds. Piped output, or a stderr that is not a terminal (`2>log`), keeps the 45-second wait and reports the server as it stood then. Control returns to your shell with the server still running; manage it later with `rocm services` (below). diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 63941600ed..4345b6a772 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -555,7 +555,7 @@ rocm serve qwen --verbose --device gpu_required")] /// (vLLM and AIM after 5 minutes by default, raised with /// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), /// with no CLI-side stall limit or deadline. Without it, a terminal gives up when - /// startup progress stalls or after 30 minutes, and piped output returns + /// startup progress stalls or after 30 minutes, and piped output (or a redirected stderr) returns /// `starting` after 45 seconds. Also waits on a matching server that is already starting. Fails with a /// pointer to the logs if the server exits first. Cannot be /// combined with --foreground or --verbose, which already stay attached. diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 0791d051de..9629e0ef84 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -1138,17 +1138,15 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(starting.clone())); spinner.borrow_mut().tick(); let mut shown = starting; - // Interactive summary follows download / load / health until ready or - // a bounded failure. Piped output keeps the historical 45-second + // Interactive summary (stdout and stderr both terminals) follows + // download / load / health until ready or a bounded failure. Piped + // output, or a redirected stderr, keeps the historical 45-second // status budget so a serve that never answers does not block CI. // `--wait` keeps waiting past either, until the server answers or dies. - let policy = if summary_mode { - serve_progress::ServeStartupPolicy::interactive() - } else { - serve_progress::ServeStartupPolicy::status_budget( - serve_progress::PIPED_SERVE_STATUS_BUDGET, - ) - }; + let policy = serve_progress::ServeStartupPolicy::for_background_serve( + summary_mode, + std::io::IsTerminal::is_terminal(&std::io::stderr()), + ); // The key is shown when the wait begins, since Ctrl-C or a failed wait // would otherwise leave a running authenticated server whose key was never // printed. It is then not repeated in the summary below. diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index bfb8feb147..7c776de69d 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -173,6 +173,22 @@ impl ServeStartupPolicy { } } + /// The policy for a background `rocm serve`: follow the startup only when + /// the summary goes to a terminal *and* stderr, where the status line and + /// any startup failure are written, is one too. With stderr redirected + /// (`rocm serve m 2>log`) nothing would be drawn for up to the full + /// deadline, so that case keeps the piped status budget. + pub(crate) const fn for_background_serve( + summary_on_terminal: bool, + stderr_is_terminal: bool, + ) -> Self { + if summary_on_terminal && stderr_is_terminal { + Self::interactive() + } else { + Self::status_budget(PIPED_SERVE_STATUS_BUDGET) + } + } + /// Piped / captured output. `timeout` is the historical readiness budget. pub(crate) const fn status_budget(timeout: Duration) -> Self { Self { @@ -1691,6 +1707,25 @@ mod tests { )); } + /// Only a terminal on both streams follows the startup; a redirected + /// stderr would hide the status line for up to the whole deadline. + #[test] + fn a_background_serve_follows_the_startup_only_with_both_streams_on_a_terminal() { + let interactive = ServeStartupPolicy::interactive(); + let piped = ServeStartupPolicy::status_budget(PIPED_SERVE_STATUS_BUDGET); + assert_eq!( + ServeStartupPolicy::for_background_serve(true, true), + interactive + ); + for (summary, stderr) in [(true, false), (false, true), (false, false)] { + assert_eq!( + ServeStartupPolicy::for_background_serve(summary, stderr), + piped, + "summary on terminal={summary} stderr terminal={stderr}" + ); + } + } + #[test] fn only_a_stall_or_the_deadline_leaves_the_server_running() { let stop = |reason, process_alive| ServeStartupStop { diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index adc9f54829..67ab020f77 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -249,8 +249,7 @@ const SIMULATE_STARTUP_EXIT_ENV: &str = "ROCM_E2E_SIMULATE_STARTUP_EXIT"; /// the spawn's immediate-exit check, so the launch is published and the /// startup wait (not the spawn) is what sees the process end. #[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] -pub(super) const SIMULATED_STARTUP_EXIT_SCRIPT: &str = - "echo 'e2e stand-in engine: loading model weights'; sleep 2; \ +pub(super) const SIMULATED_STARTUP_EXIT_SCRIPT: &str = "echo 'e2e stand-in engine: loading model weights'; sleep 2; \ echo 'e2e stand-in engine: exiting during startup'; exit 1"; /// Test-only (see [`e2e_simulate_oom_launch`]): fabricate the on-disk state of a diff --git a/docs/manual-testing.md b/docs/manual-testing.md index b62cb579a9..113df493b0 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -571,7 +571,7 @@ Expected result: The default command, without `--foreground` or `--verbose`, leaves the server in the background. While it starts, the status line follows the engine: a download percent, a load line, or a wait for the model to serve. A model that -is still loading after 45 seconds (the old wait, which piped output still +is still loading after 45 seconds (the old wait, which piped output or a redirected stderr still uses) keeps that line; it is not replaced by a finished "not ready yet" summary. The wait fails with a non-zero exit only when the engine fails or exits, a download percent stops for 3 minutes, the endpoint has not listed the model with no diff --git a/docs/testing.md b/docs/testing.md index c041bc8661..27f145b4a1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2231,7 +2231,7 @@ restart a live `running` service whose endpoint does not answer, or a live `starting` one past the stale-start window, while it is younger than that deadline (a pid whose recorded start time does not match is not live); a recoverable engine healthcheck is still restarted. -Piped output keeps the 45-second status budget. Like `--wait`, the +Piped output, or a redirected stderr (`ServeStartupPolicy::for_background_serve`), keeps the 45-second status budget. Like `--wait`, the interactive wait first prints the service id, `rocm logs --service --follow` and a public bind's key (`announces_wait` in [start.rs](../apps/rocm/src/service_launch/start.rs)), since it can fail diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 3efa08477b..7900817abe 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -747,7 +747,11 @@ fn failed_launch_service_id(world: &E2eWorld) -> String { .split(marker) .nth(1) .unwrap_or_else(|| panic!("the failure does not name `rocm logs --service`:\n{screen}")); - let id = rest.split('`').next().unwrap_or_default().trim_matches('\''); + let id = rest + .split('`') + .next() + .unwrap_or_default() + .trim_matches('\''); assert!(!id.is_empty(), "no service id in the log hint:\n{screen}"); id.to_owned() } From 3d41093473da3dff557b8a3d7b8118e0264f630b Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 16 Nov 2026 15:08:56 +0200 Subject: [PATCH 10/14] test(serve): end model_serving.feature with a single newline Signed-off-by: Roman Inflianskas --- tests/e2e-cucumber/features/model_serving.feature | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 47d5bdfde1..77fef89f33 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -980,4 +980,3 @@ Feature: Model serving And that log command shows the engine's last line And the failed launch is recorded as stopped And no background automation daemon was left behind - From 7fa1193775cedf0fe343d970ec6a22140f999aa2 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Tue, 17 Nov 2026 14:41:26 +0200 Subject: [PATCH 11/14] fix(serve): judge a finished download as loading; quote the engine line by decision A download line at 100% followed by a quiet engine was held to the 3-minute download stall; it is now judged by the 5-minute load limit. A stop quoted the log by comparing the label with this module's own fallback strings. The view now carries the engine line it shows, set where the label is chosen, so an engine line that reads like a fallback label is still quoted, and stop_decision no longer needs model and engine. A start with no log line yet now gets as long as a quiet load (5 minutes, was 2), so a slow pre-spawn step is not reported as silent sooner. A rocmd test now shows that a `failed` record whose process is still up is recovered, which the engine-failed message warns about. Signed-off-by: Roman Inflianskas --- README.md | 4 +- apps/rocm/src/serve_progress.rs | 143 +++++++++++++++++----- apps/rocm/src/service_launch/start.rs | 1 + apps/rocmd/src/watchers/server_recover.rs | 46 +++++++ docs/manual-testing.md | 4 +- docs/testing.md | 4 +- 6 files changed, 168 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 3acdfa5782..f2c96a35c2 100644 --- a/README.md +++ b/README.md @@ -1261,8 +1261,8 @@ and a quick smoke test (time to first token and approximate tokens per second). A model that is still loading is not reported as a finished "not ready" summary. The wait fails, and `rocm serve` exits non-zero, when the engine reports failure or its process exits, when a download percent stops -advancing for 3 minutes, when the endpoint has not listed the model and there is no engine log for -2 minutes, when a load log line stops changing for 5 minutes with the endpoint +advancing for 3 minutes (a line at 100% counts as loading), when the endpoint has not listed the model and there is no engine log for +5 minutes, when a load log line stops changing for 5 minutes with the endpoint not yet listing the model, or after 30 minutes in all. A stall leaves the server running: the message names `rocm logs --service ` and `rocm services stop --yes`, and the background helper does not diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index 7c776de69d..6185935808 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -49,7 +49,11 @@ pub(crate) const DOWNLOAD_STALL: Duration = Duration::from_mins(3); pub(crate) const PIPED_SERVE_STATUS_BUDGET: Duration = Duration::from_secs(45); /// Endpoint not listing the model and no meaningful log line. -pub(crate) const SILENT_UNREACHABLE: Duration = Duration::from_mins(2); +/// +/// As long as [`SILENT_LOAD`]: a start that has written nothing yet (a slow +/// pre-spawn step, such as a container engine pulling its image without a +/// progress line) is given no less time than a quiet load. +pub(crate) const SILENT_UNREACHABLE: Duration = SILENT_LOAD; /// A load line that stops changing while the endpoint has not listed the model. pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5); @@ -237,6 +241,10 @@ pub(crate) enum ServeStartupPhase { pub(crate) struct ServeStartupView { pub phase: ServeStartupPhase, pub label: String, + /// The engine-log line `label` shows, when it shows one. Set where the + /// label is chosen, so a stop quotes the engine only when the label is + /// its line, never one of this module's own fallback labels. + pub log_line: Option, } #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -351,6 +359,13 @@ fn line_is_download(line: &str) -> bool { lower.contains("download") || lower.contains("fetching ") || lower.contains("pulling ") } +/// A download line still in progress. One whose last percent is 100 is a +/// finished download: what follows is loading, so a quiet engine after it +/// is judged by the load limit, not the download stall. +fn line_is_active_download(line: &str) -> bool { + line_is_download(line) && last_percent(line) != Some("100") +} + /// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent /// does not is still stalled. fn download_fingerprint(line: &str) -> String { @@ -386,7 +401,7 @@ fn progress_token(readiness: EndpointReadiness, signals: &ServeStartupSignals) - if let Some(line) = signals .log_line .as_deref() - .filter(|line| line_is_download(line)) + .filter(|line| line_is_active_download(line)) { return format!("download:{}", download_fingerprint(line)); } @@ -413,12 +428,14 @@ fn classify( return ServeStartupView { phase: ServeStartupPhase::Failed, label, + log_line: signals.log_line.clone(), }; } if readiness == EndpointReadiness::Serving { return ServeStartupView { phase: ServeStartupPhase::Serving, label: starting_label(model, engine), + log_line: None, }; } // Listing wins over a stale download line. vLLM advertises the model @@ -431,43 +448,35 @@ fn classify( .log_line .clone() .unwrap_or_else(|| waiting_label(model)), + log_line: signals.log_line.clone(), }; } if let Some(line) = signals .log_line .as_ref() - .filter(|line| line_is_download(line)) + .filter(|line| line_is_active_download(line)) { return ServeStartupView { phase: ServeStartupPhase::Downloading, label: line.clone(), + log_line: Some(line.clone()), }; } if let Some(line) = &signals.log_line { return ServeStartupView { phase: ServeStartupPhase::Loading, label: line.clone(), + log_line: Some(line.clone()), }; } ServeStartupView { phase: ServeStartupPhase::Unreachable, label: starting_label(model, engine), - } -} - -fn quoted_log(model: &str, engine: &str, view: &ServeStartupView) -> Option { - if view.label == starting_label(model, engine) || view.label == waiting_label(model) { - return None; - } - match view.label.as_str() { - "the engine reported failed" | "the server process exited" => None, - _ => Some(view.label.clone()), + log_line: None, } } fn stop_decision( - model: &str, - engine: &str, view: &ServeStartupView, reason: ServeStartupStopReason, process_alive: bool, @@ -476,7 +485,7 @@ fn stop_decision( ServeStartupDecision::Stop(ServeStartupStop { phase: view.phase, reason, - log_line: quoted_log(model, engine, view), + log_line: view.log_line.clone(), label: view.label.clone(), process_alive, readiness, @@ -502,8 +511,6 @@ pub(crate) fn decide_serve_startup( if signals.engine_failed { return stop_decision( - model, - engine, &view, ServeStartupStopReason::EngineFailed, process_alive, @@ -512,8 +519,6 @@ pub(crate) fn decide_serve_startup( } if !process_alive { return stop_decision( - model, - engine, &view, ServeStartupStopReason::ProcessExited, process_alive, @@ -526,8 +531,6 @@ pub(crate) fn decide_serve_startup( if timing.elapsed >= policy.bounds.deadline { return if policy.stop_on_give_up { stop_decision( - model, - engine, &view, ServeStartupStopReason::Deadline, process_alive, @@ -541,8 +544,6 @@ pub(crate) fn decide_serve_startup( && timing.since_progress >= policy.bounds.download_stall { return stop_decision( - model, - engine, &view, ServeStartupStopReason::DownloadStalled, process_alive, @@ -553,8 +554,6 @@ pub(crate) fn decide_serve_startup( && timing.since_progress >= policy.bounds.silent_unreachable { return stop_decision( - model, - engine, &view, ServeStartupStopReason::Silent, process_alive, @@ -565,8 +564,6 @@ pub(crate) fn decide_serve_startup( && timing.since_progress >= policy.bounds.silent_load { return stop_decision( - model, - engine, &view, ServeStartupStopReason::SilentLoad, process_alive, @@ -1116,6 +1113,94 @@ mod tests { assert_eq!(view.label, "Loading safetensors checkpoint shards: 80%"); } + /// A download line at 100% is a finished download: a quiet engine after + /// it is loading, so the download stall must not fire at 3 minutes; the + /// load limit decides, at 5. + /// A start with no log line yet is not held to a shorter limit than a + /// quiet load: neither has shown a hang the other has not. + #[test] + fn a_start_with_no_log_gets_as_long_as_a_quiet_load() { + assert!(SILENT_UNREACHABLE >= SILENT_LOAD); + let just_under = SILENT_LOAD.as_millis() as u64 - 1_000; + assert_continue(decide_serve_startup( + "qwen", + "aim", + EndpointReadiness::Unreachable, + &alive(), + at(just_under, just_under), + &interactive(), + )); + } + + #[test] + fn a_finished_download_line_is_judged_as_loading() { + let line = "Downloading model.safetensors: 100%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let past_download_stall = DOWNLOAD_STALL.as_millis() as u64 + 1_000; + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + assert_eq!(view.log_line.as_deref(), Some(line)); + let past_load = SILENT_LOAD.as_millis() as u64 + 1_000; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_load + 1_000, past_load), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + // An unfinished percent is still a download. + signals.log_line = Some("Downloading model.safetensors: 99%".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + } + + /// A stop quotes the engine only when its label is the engine's line, + /// decided where the label is chosen: an engine line that happens to read + /// like one of the fallback labels is still quoted, and a fallback label + /// is never quoted. + #[test] + fn a_stop_quotes_the_engine_line_not_a_fallback_label() { + let mut signals = alive(); + signals.process_alive = false; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(stop.label, "the server process exited"); + assert_eq!(stop.log_line, None); + signals.log_line = Some("the server process exited".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(stop.log_line.as_deref(), Some("the server process exited")); + } + #[test] fn listing_keeps_a_stale_download_line_off_the_download_stall() { // vLLM often still has "download" as the last log line after it has @@ -1757,6 +1842,8 @@ mod tests { alive, "{reason:?} alive={alive}: {message}" ); + // True of rocmd: `a_failed_record_with_a_live_process_is_still_recovered` + // in apps/rocmd/src/watchers/server_recover.rs. assert_eq!( message.contains("may restart it first"), alive, diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 07da253e46..2246919399 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -65,6 +65,7 @@ pub(crate) fn start_managed_service( let steady = ServeStartupView { phase: ServeStartupPhase::Unreachable, label: serve_progress::starting_label(requested_model, engine), + log_line: None, }; #[cfg(feature = "e2e-oom-fault-injection")] if e2e_simulate_oom_launch() { diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index aac129792c..459a480b05 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -843,6 +843,52 @@ mod tests { fs::remove_dir_all(root).ok(); } + /// An engine that reported `failed` while its process is still up is a + /// recovery candidate: no liveness guard covers `failed`. This is what the + /// interactive serve's engine-failed message warns about when it says a + /// running daemon with `server-recover` may restart the server first + /// (`only_a_stall_or_the_deadline_leaves_the_server_running` in + /// apps/rocm/src/serve_progress.rs pins that sentence). + #[test] + fn a_failed_record_with_a_live_process_is_still_recovered() -> Result<()> { + let (root, paths) = temp_app_paths("recoverable-failed-live"); + paths.ensure()?; + let mut failed = ManagedServiceRecord::new( + &paths, + "svc-failed-live", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + failed.status = "failed".to_owned(); + failed.created_at_unix_ms = unix_time_millis(); + failed.write()?; + let found = find_recoverable_service(&paths)?; + let event = service_record_matches_recovery_event( + &paths, + &failed, + &recovery_event(&failed, "service.manifest_recoverable"), + ); + fs::remove_dir_all(root).ok(); + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-failed-live".to_owned(), + "manifest_status_failed".to_owned() + )), + "a live `failed` record inside the startup deadline is restarted" + ); + assert!(event, "the event re-check authorizes the same restart"); + Ok(()) + } + #[test] fn find_recoverable_service_prefers_failed_managed_manifest() -> Result<()> { let (root, paths) = temp_app_paths("recoverable-service"); diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 113df493b0..4f8b67ab85 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -574,8 +574,8 @@ download percent, a load line, or a wait for the model to serve. A model that is still loading after 45 seconds (the old wait, which piped output or a redirected stderr still uses) keeps that line; it is not replaced by a finished "not ready yet" summary. The wait fails with a non-zero exit only when the engine fails or -exits, a download percent stops for 3 minutes, the endpoint has not listed the model with no -engine log for 2 minutes, a load line stops changing for 5 minutes, or 30 +exits, a download percent below 100 stops for 3 minutes, the endpoint has not listed the model with no +engine log for 5 minutes, a load line stops changing for 5 minutes, or 30 minutes pass. When the endpoint answers, the command prints the deployment summary and returns to the shell. If the wait fails and the process is still up, the message names `rocm logs --service diff --git a/docs/testing.md b/docs/testing.md index 27f145b4a1..71df7fed7d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2218,8 +2218,8 @@ serve or that wait fails. A download that is still moving is not cut off at 45 seconds, and a failure names `rocm logs --service ` and `rocm services stop --yes` instead of printing `Deployment summary (not ready yet)`. The wait fails when the engine fails or -exits, a download percent stops for 3 minutes (`DOWNLOAD_STALL`), the endpoint -has not listed the model, with no log, for 2 minutes (`SILENT_UNREACHABLE`), a load line +exits, a download percent below 100 stops for 3 minutes (`DOWNLOAD_STALL`; a line at 100% is judged as loading), the endpoint +has not listed the model, with no log, for 5 minutes (`SILENT_UNREACHABLE`, equal to `SILENT_LOAD`), a load line stops changing for 5 minutes (`SILENT_LOAD`), or after 30 minutes (`rocm_core::SERVE_STARTUP_DEADLINE`). Engine log lines are masked of credentials before they are shown (`redact_log_line`). Once the endpoint lists the model, From 4c6cf192b1f73d4f7127eff9eff43a085c3e1cf2 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Tue, 17 Nov 2026 14:56:08 +0200 Subject: [PATCH 12/14] docs(serve): name where the code makes two startup-wait comments true The SILENT_UNREACHABLE comment no longer gives a container image pull as its example, which writes its own lines to the log, and names the test that pins the limit; the start.rs note about rocmd's guard names the function. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 7 ++++--- apps/rocm/src/service_launch/start.rs | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index 00cf591963..3b115b9702 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -50,9 +50,10 @@ pub(crate) const PIPED_SERVE_STATUS_BUDGET: Duration = Duration::from_secs(45); /// Endpoint not listing the model and no meaningful log line. /// -/// As long as [`SILENT_LOAD`]: a start that has written nothing yet (a slow -/// pre-spawn step, such as a container engine pulling its image without a -/// progress line) is given no less time than a quiet load. +/// As long as [`SILENT_LOAD`]: a start that has written no meaningful line +/// to its log yet (a slow step before the engine logs anything) is given no +/// less time than a quiet load. Pinned by +/// `a_start_with_no_log_gets_as_long_as_a_quiet_load`. pub(crate) const SILENT_UNREACHABLE: Duration = SILENT_LOAD; /// A load line that stops changing while the endpoint has not listed the model. diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 2246919399..b83d1cce9c 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -180,7 +180,8 @@ pub(crate) fn start_managed_service( // the endpoint got, as for any other launch: `running` only when the model // is listed, `starting` when it never opened, so `rocm services`, dash and // chat do not show a server that has not listed its model as up. - // `rocmd` leaves a live `starting` record alone until the startup deadline. + // `rocmd` leaves a live `starting` record alone until the startup deadline + // (`live_startup_still_in_progress` in apps/rocmd/src/watchers/server_recover.rs). let launch_status = status_for_readiness(readiness); let recorded = record_wait_outcome( paths, From 3cc61aa3198e83aae21a80331165d04888dbc230 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Wed, 18 Nov 2026 11:26:11 +0200 Subject: [PATCH 13/14] fix(serve): address the fourth review round on the startup wait - A decimal percent (`100.0%`) is read whole, so a finished download in that format is judged as loading too; the download fingerprint also tells `12.3%` from `12.4%`. - The smoke-test note builds its logs hint with `service_logs_hint`, so its id is shell-quoted like the summary's other notes. - A `--wait` that sees the server die is now tested for its audit event, and an engine that reports `failed` while its process is alive is tested through `start_managed_service`: it settles `failed`, names the stop command, is audited and is not supervised. - Settling an ended launch leaves the manifest alone when its recorded process is alive again, which means `rocmd` restarted it in between. - rocmd's live-startup guard spares a pid whose identity cannot be read (`Indeterminate`), as proc_lifecycle does; its comments name the code they rely on; a test pins that `ready` is not spared; and a `continue` that did nothing independent is replaced by a comment. - serve-60 gets back its last step, lost when this branch's scenarios were inserted after it, and the e2e README's fault-injection row names the startup-exit stand-in engine. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_progress.rs | 37 ++++- apps/rocm/src/serve_summary.rs | 13 +- apps/rocm/src/service_launch/readiness.rs | 54 ++++++- apps/rocm/src/service_launch/start.rs | 56 +++++++- apps/rocmd/src/watchers/server_recover.rs | 136 ++++++++++++++++-- tests/e2e-cucumber/README.md | 2 +- .../features/model_serving.feature | 1 + 7 files changed, 272 insertions(+), 27 deletions(-) diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs index 3b115b9702..f6b6e3c2a8 100644 --- a/apps/rocm/src/serve_progress.rs +++ b/apps/rocm/src/serve_progress.rs @@ -364,7 +364,10 @@ fn line_is_download(line: &str) -> bool { /// finished download: what follows is loading, so a quiet engine after it /// is judged by the load limit, not the download stall. fn line_is_active_download(line: &str) -> bool { - line_is_download(line) && last_percent(line) != Some("100") + line_is_download(line) + && !last_percent(line) + .and_then(|pct| pct.parse::().ok()) + .is_some_and(|pct| pct >= 100.0) } /// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent @@ -377,12 +380,15 @@ fn last_percent(line: &str) -> Option<&str> { let bytes = line.as_bytes(); let mut search_end = bytes.len(); while let Some(rel) = line[..search_end].rfind('%') { + // Digits with an optional fraction (`100.0%`), so a decimal percent is + // read whole rather than as the digits after its point. let mut start = rel; - while start > 0 && bytes[start - 1].is_ascii_digit() { + while start > 0 && (bytes[start - 1].is_ascii_digit() || bytes[start - 1] == b'.') { start -= 1; } - if start < rel { - return Some(&line[start..rel]); + let number = line[start..rel].trim_start_matches('.'); + if number.bytes().any(|byte| byte.is_ascii_digit()) { + return Some(number); } search_end = rel; } @@ -1159,7 +1165,28 @@ mod tests { &interactive(), )); assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); - // An unfinished percent is still a download. + // A decimal percent is read whole: `100.0%` is finished too. + signals.log_line = Some("Downloading model.safetensors: 100.0%".to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + // An unfinished percent is still a download, decimal or not. + signals.log_line = Some("Downloading model.safetensors: 99.9%".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); signals.log_line = Some("Downloading model.safetensors: 99%".to_owned()); let stop = assert_stop(decide_serve_startup( "qwen", diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index 3cb3612f75..c6aa72ee33 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -278,8 +278,8 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { // empty, hedged: the reason may be rocm's own plumbing, not the engine. let _ = writeln!( out, - " note: the smoke test could not complete, so no metrics were measured ({reason}); the server may not be able to serve requests - check `rocm logs --service {id}`", - id = summary.service_id + " note: the smoke test could not complete, so no metrics were measured ({reason}); the server may not be able to serve requests - check `{logs}`", + logs = crate::service_hints::service_logs_hint(&summary.service_id) ); } for note in &summary.notes { @@ -1290,6 +1290,15 @@ mod tests { // The hedge: the note must not claim the server is broken, only that it // may be unable to serve. assert!(rendered.contains("; the server may not be able to serve requests")); + // The id goes through the shared hint, so one that needs quoting is + // quoted here as in the summary's other notes. + let mut hostile = failed_summary("connection closed mid-response"); + hostile.service_id = "svc $(id)".to_owned(); + let rendered = render_summary(&hostile); + assert!( + rendered.contains("check `rocm logs --service 'svc $(id)'`"), + "{rendered}" + ); } #[test] diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index 2bd8fe18c7..73f7ec2c82 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -100,13 +100,25 @@ fn service_still_live(paths: &AppPaths, service_id: &str, pid_alive: impl Fn(u32 /// Record a server that died mid-start as stopped, leaving a manifest that a /// `services stop` already settled untouched. A `failed` state the engine /// reported is persisted rather than overwritten, so the reason survives. -fn settle_exited_service(paths: &AppPaths, service_id: &str) { +/// +/// The manifest is re-read here, after the caller decided the server ended. +/// If it now describes a live process again, `rocmd` restarted the service in +/// between (`restart_managed_service` in apps/rocmd/src/supervise.rs records +/// the new pid through `record_spawned_supervisor`), and that restart is left alone rather than written back to +/// `stopped`. +fn settle_exited_service(paths: &AppPaths, service_id: &str, pid_alive: &dyn Fn(u32) -> bool) { let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { return; }; let on_disk = latest.status.clone(); let _ = latest.refresh_from_engine_state(); if crate::managed_service_status::managed_service_is_live(&latest) { + if crate::managed_service_status::recorded_service_pids(&latest) + .into_iter() + .any(pid_alive) + { + return; + } latest.status = "stopped".to_owned(); } else if latest.status == on_disk { return; @@ -126,7 +138,7 @@ pub(super) fn died_during_wait( ) -> Option { if !wait_until_ready || readiness == EndpointReadiness::Serving - || service_still_live(paths, service_id, pid_alive) + || service_still_live(paths, service_id, &pid_alive) { return None; } @@ -138,7 +150,8 @@ pub(super) fn died_during_wait( paths, service_id, "exited during --wait", - message + message, + &pid_alive ))) } @@ -157,8 +170,9 @@ pub(super) fn settle_ended_launch( service_id: &str, why: &str, mut message: String, + pid_alive: &dyn Fn(u32) -> bool, ) -> String { - settle_exited_service(paths, service_id); + settle_exited_service(paths, service_id, pid_alive); crate::record_cli_audit_event( paths, "service", @@ -650,9 +664,14 @@ mod tests { fn settle_exited_service_demotes_a_live_manifest_and_spares_a_settled_one() { let (root, paths) = test_paths("wait-settle"); paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, 0, "starting"); + let mut record = record_tracking_pid(&paths, 4_242, "starting"); record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); + // A manifest whose recorded process is alive again was restarted by + // `rocmd` after the caller decided the server ended: left alone. + settle_exited_service(&paths, &record.service_id, &|_| true); + let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); + assert_eq!(after.status, "starting"); + settle_exited_service(&paths, &record.service_id, &|_| false); let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); assert_eq!(after.status, "stopped"); @@ -661,7 +680,7 @@ mod tests { // be invisible). record.status = "failed".to_owned(); record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); + settle_exited_service(&paths, &record.service_id, &|_| false); let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); assert_eq!(after.status, "failed"); let _ = fs::remove_dir_all(root); @@ -728,6 +747,18 @@ mod tests { .status }; + let launch_errors = || { + rocm_core::load_recent_audit_events(&paths, 20) + .unwrap_or_default() + .into_iter() + .filter(|event| { + event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(id.as_str()) + }) + .collect::>() + }; + // Not waiting, already serving, or still alive: no failure, no change. let pending = EndpointReadiness::Unreachable; assert!(died_during_wait(&paths, &id, false, pending, |_| false).is_none()); @@ -736,6 +767,7 @@ mod tests { ); assert!(died_during_wait(&paths, &id, true, pending, |_| true).is_none()); assert_eq!(status(), "starting"); + assert!(launch_errors().is_empty(), "{:?}", launch_errors()); // Dead server: the error names the logs command and the record is settled. let error = died_during_wait(&paths, &id, true, pending, |_| false).expect("died"); @@ -749,6 +781,14 @@ mod tests { "{message}" ); assert_eq!(status(), "stopped"); + // The `--wait` death is audited like the interactive wait's. + let errors = launch_errors(); + assert!( + errors + .iter() + .any(|event| event.message.contains("exited during --wait")), + "{errors:?}" + ); let _ = fs::remove_dir_all(root); } diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index b83d1cce9c..8c70e38462 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -171,7 +171,7 @@ pub(crate) fn start_managed_service( let message = stop.user_message(requested_model, engine, service_id); let why = format!("{:?}", stop.reason); return Err(serve_progress::ServeStartupFailed { - message: settle_ended_launch(paths, service_id, &why, message), + message: settle_ended_launch(paths, service_id, &why, message, &service_pid_is_alive), server_running: false, } .into()); @@ -512,10 +512,14 @@ mod tests { paths.ensure()?; let child = std::rc::Rc::new(std::cell::RefCell::new(None::)); let spawned = child.clone(); + // The engine-state file the launch's record reads, for a child that + // plays an engine reporting its own status. + let engine_state = paths.service_engine_state_path("lemonade", service_id); TEST_SPAWN_HOOK.with(|hook| { *hook.borrow_mut() = Some(std::rc::Rc::new(move |_command| { let process = std::process::Command::new(argv[0]) .args(&argv[1..]) + .env("TEST_ENGINE_STATE", &engine_state) .spawn()?; *spawned.borrow_mut() = Some(process.id()); Ok(process) @@ -644,6 +648,56 @@ mod tests { Ok(()) } + /// An engine that reports `failed` while its process is still up ends the + /// launch through the same settling path as an exit: the typed failure + /// does not ask `serve` to supervise it, names the stop command, the + /// record keeps the engine's own `failed` (which `rocmd` may restart: + /// `a_failed_record_with_a_live_process_is_still_recovered` in + /// apps/rocmd/src/watchers/server_recover.rs), and the end is audited. + #[cfg(not(windows))] + #[test] + fn an_engine_that_fails_while_alive_is_settled_failed_and_not_supervised() -> Result<()> { + let service_id = "lemonade-startup-engine-failed"; + let launch = launch_hooked_child( + service_id, + &[ + "sh", + "-c", + "mkdir -p \"$(dirname \"$TEST_ENGINE_STATE\")\" && \ + printf '{\"status\":\"failed\"}' > \"$TEST_ENGINE_STATE\" && sleep 30", + ], + Duration::from_secs(5), + )?; + assert!(launch.child_alive_after, "the process stays up"); + let failure = startup_failure(launch.result); + assert!(!failure.server_running, "{}", failure.message); + assert!( + failure.message.contains("the engine reported failed"), + "{}", + failure.message + ); + assert!( + failure + .message + .contains(&crate::service_hints::service_stop_hint(service_id)), + "{}", + failure.message + ); + assert_eq!(launch.record?.status, "failed"); + assert!( + launch + .audit + .iter() + .any(|event| event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(service_id) + && event.message.contains("EngineFailed")), + "the ended launch is audited: {:?}", + launch.audit + ); + Ok(()) + } + /// The launch lock must still be held when the child is spawned: it is what /// serializes allocation against publication. Observed through the spawn /// hook, since releasing it early changes nothing else a test can see. diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index 459a480b05..05371a427f 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -333,10 +333,10 @@ pub(crate) fn find_recoverable_service( if let Some(reason) = stale_manifest_recovery_reason(&record, now) { return Ok(Some((record, reason))); } - if manifest_service_recovery_reason(&record, now).is_some() { - // A live startup the guard above spared. - continue; - } + // A manifest reason the guard above spared (a live startup) needs no + // `continue`: `manifest_service_recovery_reason` only answers for + // `failed`, `exited`, `unreachable` and a stale `starting`/`recovering`, + // none of which the `ready`/`running` check below takes. if matches!(record.status.as_str(), "ready" | "running") { let Ok(healthcheck) = crate::engine_ipc::engine_healthcheck_response( paths, @@ -366,8 +366,15 @@ pub(crate) fn find_recoverable_service( /// recovered, or `None` when it should not. /// /// A live `running` process that has not opened its port is still the -/// interactive serve wait's job: restarting it kills a download the CLI is -/// reporting. A dead pid is not spared. One function for every place that +/// interactive serve wait's job (`wait_for_launch_startup` in +/// apps/rocm/src/service_launch/readiness.rs): restarting it kills a download +/// the CLI is reporting. `running` is covered, not only `starting`, because +/// the CLI records `running` once the endpoint lists the model +/// (`status_for_readiness` in apps/rocm/src/main.rs) and the engines write +/// their own state `running` before their port opens (`write_running_state` +/// in engines/lemonade/src/state.rs, called before the server is ready). `ready` is not covered: a +/// verified service that stops answering is a regression, not a startup. +/// A dead pid is not spared. One function for every place that /// falls back to the endpoint, scan and event re-check, so none can drop the /// guard. fn unreachable_endpoint_recovery_reason( @@ -401,6 +408,21 @@ pub(crate) const fn stop_requested(record: &ManagedServiceRecord) -> bool { /// or load before the port accepts connections. Endpoint-unreachable /// recovery during it restarts the process out from under that wait. fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u128) -> bool { + live_startup_still_in_progress_with(record, now_unix_ms, rocm_core::identity_state) +} + +/// [`live_startup_still_in_progress`] with the identity check injected, so a +/// test can give every answer a read can give. +/// +/// A pid whose identity cannot be read (`IdentityState::Indeterminate`) is +/// treated as possibly the startup and spared, as `proc_lifecycle` treats it +/// as possibly alive and does not signal it: a failure to look is not an +/// exit. Only `Recycled` and `Gone` are not the startup. +fn live_startup_still_in_progress_with( + record: &ManagedServiceRecord, + now_unix_ms: u128, + identity: impl Fn(&ProcessIdentity) -> IdentityState, +) -> bool { let started_at = record .last_restart_unix_ms .unwrap_or(record.created_at_unix_ms); @@ -421,17 +443,22 @@ fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u1 .flatten() .any(|(pid, start_ticks)| { pid != 0 - && rocm_core::identity_state(&ProcessIdentity { pid, start_ticks }) - == IdentityState::Matches + && matches!( + identity(&ProcessIdentity { pid, start_ticks }), + IdentityState::Matches | IdentityState::Indeterminate + ) }) } /// Why a stale or failed manifest should be recovered, or `None`. /// /// `rocm services` persists an engine's own `starting` state into the -/// record, so a live server still downloading under the interactive serve -/// wait (or left by a Ctrl-C of it) can read `starting` past the stale-start -/// window. It is not stale until the startup deadline. Shared by the scan and +/// record (`ManagedServiceRecord::refresh_from_engine_state` in +/// crates/rocm-core/src/service_record.rs), and the interactive serve wait +/// records a launch that never listed the model as `starting` +/// (`status_for_readiness` in apps/rocm/src/main.rs), so a live server still +/// downloading under that wait (or left by a Ctrl-C of it) can read +/// `starting` past the stale-start window. It is not stale until the startup deadline. Shared by the scan and /// the event re-check, so neither can drop the guard. fn stale_manifest_recovery_reason( record: &ManagedServiceRecord, @@ -1349,6 +1376,93 @@ mod tests { Ok(()) } + /// The guard covers `running` (and `starting`) but not `ready`: a service + /// that was verified serving and then stopped answering is recovered at + /// once, even with a live process inside the startup deadline. + #[test] + fn recovery_does_not_spare_a_ready_service_that_stopped_answering() -> Result<()> { + let (root, paths) = temp_app_paths("recover-ready-live"); + paths.ensure()?; + let mut ready = ManagedServiceRecord::new( + &paths, + "svc-ready-live", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + ready.status = "ready".to_owned(); + ready.created_at_unix_ms = unix_time_millis(); + ready.write()?; + let found = find_recoverable_service(&paths)?; + let event = service_record_matches_recovery_event( + &paths, + &ready, + &recovery_event(&ready, "service.endpoint_recoverable"), + ); + fs::remove_dir_all(root).ok(); + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-ready-live".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + ); + assert!(event, "the event re-check authorizes the same restart"); + Ok(()) + } + + /// Every identity answer: the startup is spared when its pid matches or + /// cannot be read, and not when it was recycled or is gone, or past the + /// deadline whatever the answer. + #[test] + fn the_live_startup_guard_spares_a_matching_or_unreadable_pid_only() { + let (root, paths) = temp_app_paths("recover-identity-answers"); + let mut record = ManagedServiceRecord::new( + &paths, + "svc-identity", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + 11435, + "managed", + 4242, + None, + None, + None, + ); + let now = unix_time_millis(); + record.created_at_unix_ms = now; + for (answer, spared) in [ + (IdentityState::Matches, true), + (IdentityState::Indeterminate, true), + (IdentityState::Recycled, false), + (IdentityState::Gone, false), + ] { + assert_eq!( + live_startup_still_in_progress_with(&record, now, |_| answer), + spared, + "{answer:?}" + ); + assert!( + !live_startup_still_in_progress_with( + &record, + now + rocm_core::SERVE_STARTUP_DEADLINE.as_millis(), + |_| answer + ), + "{answer:?} past the deadline" + ); + } + fs::remove_dir_all(root).ok(); + } + /// `rocm services` writes an engine's own `starting` state into the /// record, so a live startup can read `starting` past the five-minute /// stale-start window. That record is spared until diff --git a/tests/e2e-cucumber/README.md b/tests/e2e-cucumber/README.md index d871ab21d1..ed10ae737e 100644 --- a/tests/e2e-cucumber/README.md +++ b/tests/e2e-cucumber/README.md @@ -145,7 +145,7 @@ Scenarios carry stable-id and capability tags: | `@requires-case-sensitive-fs` | Premise is two files whose names differ only in letter case. Resolves to **skip** where the scenarios' temp root folds case. The capability probe's `case_sensitive_fs` writes two such names in its own temp dir, which follows `$TMPDIR` like every scenario's isolated root, and checks they stayed two files. `@requires-os:linux` cannot express this: `$TMPDIR` on Linux, WSL2 included, can sit on a casefolded directory or a mounted Windows drive. | | `@requires-engine:` | Pins the serve engine. Resolves to skip where that engine can't start (e.g. vLLM on a lemonade-only Strix host). | | `@requires-os:` | Premise is OS-specific; skip on other OSes. | -| `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch so `rocm serve`'s summary can be verified without a GPU: one that runs out of GPU memory (the OOM guidance), or, in its `ready` mode, one that becomes ready and then fails its smoke test. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | +| `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch so `rocm serve`'s summary can be verified without a GPU: one that runs out of GPU memory (the OOM guidance), or, in its `ready` mode, one that becomes ready and then fails its smoke test. On Linux the same feature also carries a stand-in engine (`ROCM_E2E_SIMULATE_STARTUP_EXIT=1`) that logs a line and exits during startup, so the interactive startup wait's failure path runs for real. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | | `@serve-timeout:` | Lengthen the serve-readiness wait for a genuinely slow serve (e.g. a large model). | | `@nightly` | Expensive scenario skipped by default; included when `E2E_INCLUDE_NIGHTLY=1`. | | `@lifecycle` | Expensive, OS-mutating release-lifecycle scenario (packaging + real installer + install/uninstall). Skipped by default; included when `E2E_INCLUDE_LIFECYCLE=1`. `E2E_ONLY_LIFECYCLE=1` selects only this set without bypassing expectation resolution. A caller can additionally fold specific non-`@lifecycle` scenarios into that restricted set via `E2E_ALSO_RUN_IDS` (comma-separated `@id`s), without tagging them `@lifecycle` — which would also pull them out of the full-suite lanes that don't set `E2E_INCLUDE_LIFECYCLE`. `ci.yml`'s Windows lane uses this to fold in the hermetic comfyui-01/02/03 scenarios, the native-Windows vLLM refusal scenarios serve-33/34/35, and the side-by-side runtime scenarios runtime-lifecycle-19/20/21. | diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 9f67e07c5e..29485d262c 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -989,6 +989,7 @@ Feature: Model serving And the engine received exactly one chat request And the deployment summary names the logs command for that service And the launch's service record is left as ready + And no background automation daemon was left behind # Interactive `rocm serve` under a real terminal: while it waits it names the # command that follows the engine's log, then it exits 0 once the endpoint From 0c2d0746e45d996b03884a135ebc060f69a55fe6 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Wed, 18 Nov 2026 11:49:57 +0200 Subject: [PATCH 14/14] test(serve): do not print a rendered summary in a failed assertion The rendered deployment summary can carry an endpoint key; CodeQL flagged the new hostile-id assertion for printing it on failure. The asserts beside it already leave it out. Signed-off-by: Roman Inflianskas --- apps/rocm/src/serve_summary.rs | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index c6aa72ee33..82482dcdc4 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -1295,10 +1295,9 @@ mod tests { let mut hostile = failed_summary("connection closed mid-response"); hostile.service_id = "svc $(id)".to_owned(); let rendered = render_summary(&hostile); - assert!( - rendered.contains("check `rocm logs --service 'svc $(id)'`"), - "{rendered}" - ); + // No `{rendered}` in the failure message: the rendered summary can + // carry an endpoint key, so it is not printed, like the asserts above. + assert!(rendered.contains("check `rocm logs --service 'svc $(id)'`")); } #[test]