diff --git a/README.md b/README.md index 9a05b8190..0229b41a1 100644 --- a/README.md +++ b/README.md @@ -1405,10 +1405,16 @@ Likewise, an explicit `--port` that differs from the port the running service is already on is refused rather than silently ignored; omit `--port` to reuse the running service, or stop it first to serve on the new port. -By default the server runs in the background under ROCm CLI's supervision and -prints a deployment summary: a progress indicator while it starts, then a table -with the status, the full inference endpoint, the API-qualified model name, and a -quick smoke test (time to first token and approximate tokens per second). On an +By default the server runs in the background under ROCm CLI's supervision. +For a fresh launch in a terminal it first prints the service id, how to follow its log +(`rocm logs --service --follow`) and, for a public bind, the +API key. While it starts, the status line follows the engine: a download, a weight +load, or the latest log line, until the server is ready or that wait fails +(the engine exits, a download stops advancing, or the startup deadline +passes). When the server is ready it prints a deployment summary: a table +with the status, the full inference endpoint, the API-qualified model name, +and a quick smoke test (time to first token and approximate tokens per +second). On an interactive terminal, if the smoke test cannot complete, the summary is headed `(smoke test failed)`, shows the status as `started, smoke test failed` instead of `ready`, and a note gives the reason and points at @@ -1416,9 +1422,22 @@ of `ready`, and a note gives the reason and points at still exits 0. Which failures are detected, and which still read as `ready`, is listed under [Serve Smoke Test Failure](https://github.com/ROCm/rocm-cli/blob/main/docs/testing.md#serve-smoke-test-failure) -in the testing guide. Control -returns to your shell with the server still running; manage it later with `rocm -services` (below). +in the testing guide. A model that is still loading is not reported as a finished "not +ready" summary. The wait fails, and `rocm serve` exits non-zero, when the +engine reports failure or its process exits, when a download percent stops +advancing for 3 minutes (a line at 100% counts as loading), when the endpoint has not listed the model and there is no engine log for +5 minutes, when a load log line stops changing for 5 minutes with the endpoint +not yet listing the model, or after 30 minutes in all. A stall leaves the server running: +the message names `rocm logs --service ` and +`rocm services stop --yes`, and the background helper does not +restart a live server for being slow until those 30 minutes pass. Engine log +lines shown on the status line have credentials (the endpoint key, `--api-key` +and token values, `hf_` tokens) masked. Once the server lists the model, the wait +checks whether it can answer with a one-token request, first right away and +then at growing intervals up to every 15 seconds. Piped output, or a stderr that is not a terminal (`2>log`), keeps +the 45-second wait and reports the server as it stood then. Control returns +to your shell with the server still running; manage it later with +`rocm services` (below). `--verbose` (or `--foreground`) instead attaches to the server in the current terminal and streams every engine log line. Use it to debug a startup problem. @@ -1436,8 +1455,8 @@ can: `--managed` is the explicit form of the default background behavior. `--wait` keeps `rocm serve` waiting until the server answers requests or its engine gives up (vLLM and AIM after 5 minutes by default; raise it with -`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), instead -of returning `starting` after 45 seconds; if the server exits first (crashes, is killed, or is stopped with +`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), with +no CLI-side stall limit or deadline, instead of the default wait above; if the server exits first (crashes, is killed, or is stopped with `rocm services stop`), it fails with an error that points at the server's logs instead of waiting forever. Ctrl-C ends the wait and leaves the server starting in the background; a generated API key (public bind) is printed when the wait diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index c50a1af20..c82349605 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -512,8 +512,13 @@ rocm model --verbose" /// Picks an engine and ROCm runtime automatically unless overridden. An /// Anthropic `/v1/messages` endpoint is published beside the OpenAI /// one (see `rocm services list`). By default the - /// server runs as a managed background service and prints a deployment summary - /// (status, endpoint, model, and smoke-test metrics); use --verbose to stream engine + /// server runs as a managed background service. For a fresh launch in a terminal it + /// first prints the service id and how to follow its log; while it starts, the status line + /// follows what the engine is doing (download, load, or the latest log line) until + /// the server is ready or that wait fails. A ready server then prints a deployment + /// summary (status, endpoint, model, and smoke-test metrics). A wait that fails + /// while the server is still running names `rocm logs` and `rocm services stop` + /// instead of a finished "not ready" summary. Use --verbose to stream engine /// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the /// server running, or Ctrl-C to stop it. If that stop answers that it could not confirm /// the server exited, it says so on stderr and exits with an error; if the stop itself @@ -602,8 +607,9 @@ rocm serve qwen --verbose --device gpu_required")] /// Keep waiting until the server answers requests or its engine gives up /// (vLLM and AIM after 5 minutes by default, raised with /// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), - /// instead of returning `starting` after 45 seconds. - /// Also waits on a matching server that is already starting. Fails with a + /// with no CLI-side stall limit or deadline. Without it, a terminal gives up when + /// startup progress stalls or after 30 minutes, and piped output (or a redirected stderr) returns + /// `starting` after 45 seconds. Also waits on a matching server that is already starting. Fails with a /// pointer to the logs if the server exits first. Cannot be /// combined with --foreground or --verbose, which already stay attached. #[arg(long, conflicts_with_all = ["foreground", "verbose"])] diff --git a/apps/rocm/src/engine_runtime_select.rs b/apps/rocm/src/engine_runtime_select.rs index 88d101918..5aaa26f58 100644 --- a/apps/rocm/src/engine_runtime_select.rs +++ b/apps/rocm/src/engine_runtime_select.rs @@ -72,7 +72,9 @@ //! `service_launch.rs` and `service_control.rs`'s `restart_internal_managed_service_with`. //! `wait_for_service_http_ready` is `pub(crate)`, read from `service_control.rs`'s //! `restart_internal_managed_service_with`; `wait_for_service_http_ready_with_progress` -//! is `pub(crate)`, read from `service_launch.rs`. `http_post_local_service_json` +//! is `pub(crate)`, read from `service_launch.rs`. `service_http_lists_model` and +//! `service_http_inference_answers`, the two halves of its readiness probe, are +//! `pub(crate)`, read from `serve_progress.rs`'s `wait_for_serve_startup`. `http_post_local_service_json` //! is `pub(crate)`, read from `service_control.rs`'s `unload_lemonade_service_model`. //! `service_model_names_match` is `pub(crate)`, read from `chat/turn.rs`. //! `service_http_readiness_paths`, `http_get_local_service`, @@ -348,43 +350,14 @@ pub(crate) fn wait_for_service_http_ready_with_progress( on_tick: &mut dyn FnMut(Duration), ) -> EndpointReadiness { let start = std::time::Instant::now(); - let endpoint = format_http_base_url(host, port); let mut best = EndpointReadiness::Unreachable; while start.elapsed() < timeout { // Re-evaluated each pass: a container engine reports the name its server // answers to only after it starts, so a name fixed up front can be wrong. - let canonical_model_id = model_name(); - let canonical_model_id = canonical_model_id.as_str(); - let listed = service_http_readiness_paths(engine).iter().any(|path| { - http_get_local_service( - host, - port, - path, - endpoint_api_key, - Duration::from_millis(750), - ) - .is_ok_and(|(status, body)| { - service_http_readiness_response_ready( - engine, - path, - status, - &body, - canonical_model_id, - ) - }) - }); - if listed { - best = EndpointReadiness::Listing; - if rocm_core::openai_chat_completion_probe( - &endpoint, - canonical_model_id, - endpoint_api_key, - rocm_core::INFERENCE_PROBE_TIMEOUT, - ) - .unwrap_or(false) - { - return EndpointReadiness::Serving; - } + match probe_service_http_readiness(engine, host, port, &model_name(), endpoint_api_key) { + EndpointReadiness::Serving => return EndpointReadiness::Serving, + EndpointReadiness::Listing => best = EndpointReadiness::Listing, + EndpointReadiness::Unreachable => {} } on_tick(start.elapsed()); thread::sleep(Duration::from_millis(250)); @@ -392,6 +365,70 @@ pub(crate) fn wait_for_service_http_ready_with_progress( best } +/// One readiness probe: [`EndpointReadiness::Serving`] only after an inference +/// request comes back, [`EndpointReadiness::Listing`] when the model is +/// advertised but that request does not, otherwise unreachable. +/// +/// Probes inference on every call. [`crate::serve_progress::wait_for_serve_startup`] +/// spaces those probes out instead, through [`service_http_lists_model`] and +/// [`service_http_inference_answers`], the same two checks, so the interactive +/// status line and this wait cannot disagree about "ready". +fn probe_service_http_readiness( + engine: &str, + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> EndpointReadiness { + if !service_http_lists_model(engine, host, port, canonical_model_id, endpoint_api_key) { + return EndpointReadiness::Unreachable; + } + if service_http_inference_answers(host, port, canonical_model_id, endpoint_api_key) { + return EndpointReadiness::Serving; + } + EndpointReadiness::Listing +} + +/// The cheap half of a readiness probe: the endpoint answers and advertises +/// the model (a short GET per readiness path). +pub(crate) fn service_http_lists_model( + engine: &str, + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> bool { + service_http_readiness_paths(engine).iter().any(|path| { + http_get_local_service( + host, + port, + path, + endpoint_api_key, + Duration::from_millis(750), + ) + .is_ok_and(|(status, body)| { + service_http_readiness_response_ready(engine, path, status, &body, canonical_model_id) + }) + }) +} + +/// The expensive half: a real one-token chat request, which can wait up to +/// [`rocm_core::INFERENCE_PROBE_TIMEOUT`] on a model that is still loading. +pub(crate) fn service_http_inference_answers( + host: &str, + port: u16, + canonical_model_id: &str, + endpoint_api_key: Option<&str>, +) -> bool { + rocm_core::openai_chat_completion_probe( + &format_http_base_url(host, port), + canonical_model_id, + endpoint_api_key, + rocm_core::INFERENCE_PROBE_TIMEOUT, + ) + .unwrap_or(false) +} + fn service_http_readiness_paths(engine: &str) -> &'static [&'static str] { match engine { "lemonade" => &["/v1/health", "/v1/models"], diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 712208d80..2c6f1b4c2 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -61,6 +61,7 @@ mod runtime_uninstall; mod runtimes_cmd; mod serve_cmd; mod serve_port; +mod serve_progress; mod serve_recipe; mod serve_summary; mod service_control; diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index f359699db..5876cdcb2 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -67,6 +67,7 @@ use crate::gpu_policy::{ }; use crate::managed_service_status::{find_live_managed_service, live_managed_services_for_model}; use crate::serve_port; +use crate::serve_progress::{self, ServeStartupView}; use crate::serve_recipe::{ ServeGenerationDefaults, engine_env_ignored_note, engine_recipe_enables_tool_choice, engine_recipe_with_extra_env, engine_recipe_with_generation_defaults, @@ -864,8 +865,8 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { // (probe returns `None`). The E2E-only // backend-failure scenario bypasses this host precondition so the black-box // test reaches Lemonade's backend boundary without real GPU hardware, and the - // E2E-only launch fault injection (either mode) likewise stands in for the - // GPU it does not have. + // E2E-only launch fault injection (either mode) and the startup-exit + // stand-in engine likewise stand in for the GPU they do not have. // // The scripted free-space override opens the same waiver: it exists to refuse a // Lemonade install on a drive that cannot hold it, which `rocm serve` only @@ -880,6 +881,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { if !cpu_only && !scripted_backend_failure && !e2e_simulate_oom_launch() + && !crate::service_launch::e2e_simulate_startup_exit() && !reuse_existing && host_has_no_usable_gpu { @@ -1132,10 +1134,19 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { } if background { - let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(format!( - "Starting {model} on {selected_engine}…" - ))); + let starting = serve_progress::starting_label(&model, &selected_engine); + let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(starting.clone())); spinner.borrow_mut().tick(); + let mut shown = starting; + // Interactive summary (stdout and stderr both terminals) follows + // download / load / health until ready or a bounded failure. Piped + // output, or a redirected stderr, keeps the historical 45-second + // status budget so a serve that never answers does not block CI. + // `--wait` keeps waiting past either, until the server answers or dies. + let policy = serve_progress::ServeStartupPolicy::for_background_serve( + summary_mode, + std::io::IsTerminal::is_terminal(&std::io::stderr()), + ); // The key is shown when the wait begins, since Ctrl-C or a failed wait // would otherwise leave a running authenticated server whose key was never // printed. It is then not repeated in the summary below. @@ -1158,6 +1169,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { require_api_key, gpu_refusal_waived_for_reuse, wait, + &policy, // Named with the id the launch resolved to: an already-starting // server keeps its own, not the one generated above. The spinner // has no id on it, and a non-TTY stderr gets no spinner at all. @@ -1165,7 +1177,8 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { spinner.borrow_mut().clear(); eprintln!( "Waiting for {waiting_on}; Ctrl-C stops waiting, the server keeps running. \ - Follow: rocm logs --service {waiting_on} --follow" + Follow: {} --follow", + crate::service_hints::service_logs_hint(waiting_on) ); if let Some(config) = wait_begin_client_config( waiting_on, @@ -1181,13 +1194,26 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { print!("{config}"); } }, - &mut |_elapsed| spinner.borrow_mut().tick(), + &mut |view: &ServeStartupView| { + let mut spinner = spinner.borrow_mut(); + if view.label == shown { + spinner.tick(); + } else { + shown.clone_from(&view.label); + spinner.set_label(view.label.clone()); + } + }, ); // `Spinner` has no `Drop`: clear its row so `Error:` is not glued to it. let report = match launched { Ok(report) => report, Err(error) => { spinner.borrow_mut().clear(); + if let Some(warning) = supervise_after_launch_error(&error, || { + ensure_background_helper_running_quiet(summary_mode) + }) { + eprintln!("{warning}"); + } return Err(error); } }; @@ -1409,6 +1435,35 @@ fn collect_serve_notes( notes } +/// Start supervision after `start_managed_service` failed, when that failure +/// left a launched server behind. Returns the warning to print when the +/// helper itself would not start. +/// +/// Only a startup stop that left the server running +/// ([`serve_progress::ServeStartupFailed`] with `server_running`) leaves a +/// process to supervise. A stop on an engine that failed or a process that +/// exited has already been settled and reported as failed, so the CLI does not +/// start the helper for it. Other errors either come before the child +/// exists (the claiming record is written first, as `starting`, and `rocmd` +/// treats it as any stale start), or from a child that already exited or an +/// I/O error writing its record, which did not start the helper before this +/// wait existed either. The startup failure stays the error the user reads; a +/// helper failure is a warning beside it. +fn supervise_after_launch_error( + error: &anyhow::Error, + start_helper: impl FnOnce() -> Result<()>, +) -> Option { + error + .downcast_ref::() + .filter(|stop| stop.server_running) + .and_then(|_| start_helper().err()) + .map(|helper| { + format!( + "warning: the background helper did not start, so nothing restarts this server if it fails: {helper:#}" + ) + }) +} + #[cfg(test)] mod tests { use super::*; @@ -2267,4 +2322,62 @@ mod tests { "non-vLLM engines must not be told to pass a vLLM-only flag: {lemonade:?}" ); } + + #[test] + fn a_startup_stop_starts_supervision_and_keeps_its_own_error() { + use std::cell::Cell; + let stop = || -> anyhow::Error { + serve_progress::ServeStartupFailed { + message: "qwen on vllm stopped making download progress".to_owned(), + server_running: true, + } + .into() + }; + // Row: startup stop, helper starts. Helper called, nothing to warn. + let called = Cell::new(false); + let warning = super::supervise_after_launch_error(&stop(), || { + called.set(true); + Ok(()) + }); + assert!(called.get()); + assert_eq!(warning, None); + + // Row: startup stop, helper fails. The warning names the helper + // error; the startup failure is still what the caller returns. + let warning = super::supervise_after_launch_error(&stop(), || anyhow::bail!("no daemon")) + .expect("a helper that failed is reported"); + assert!( + warning.starts_with("warning: the background helper did not start"), + "{warning}" + ); + assert!(warning.contains("no daemon"), "{warning}"); + + // Row: the server ended (engine failed, process exited). It was + // settled and reported as failed, so no helper is started to restart + // it behind that error. + let called = Cell::new(false); + let ended: anyhow::Error = serve_progress::ServeStartupFailed { + message: "qwen on vllm failed: the server process exited".to_owned(), + server_running: false, + } + .into(); + let warning = super::supervise_after_launch_error(&ended, || { + called.set(true); + Ok(()) + }); + assert!(!called.get(), "an ended launch starts no helper"); + assert_eq!(warning, None); + + // Row: any other launch error. No helper. + let called = Cell::new(false); + let warning = super::supervise_after_launch_error(&anyhow::anyhow!("spawn failed"), || { + called.set(true); + Ok(()) + }); + assert!( + !called.get(), + "a launch that never ran a server starts no helper" + ); + assert_eq!(warning, None); + } } diff --git a/apps/rocm/src/serve_progress.rs b/apps/rocm/src/serve_progress.rs new file mode 100644 index 000000000..f6b6e3c2a --- /dev/null +++ b/apps/rocm/src/serve_progress.rs @@ -0,0 +1,2210 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Interactive `rocm serve` startup progress. +//! +//! Summary mode used to tick one static "Starting …" label and, after 45 +//! seconds, print a finished "not ready yet" summary while the server kept +//! loading. This module decides the status line from the endpoint and the +//! engine log, and when that wait is allowed to stop. +//! +//! State matrix for the wait. Every pending row has an exit when nothing +//! new arrives; a download that stops advancing does not spin until the +//! overall deadline. +//! +//! | State | What we observe | Status line | Exit if data never arrives | +//! | --- | --- | --- | --- | +//! | unreachable | port down, no meaningful log, process alive | `Starting {model} on {engine}…` | silent limit, or the deadline | +//! | downloading | latest log line is a download, the endpoint is not listing the model, and progress is still moving | that log line | stall limit once the line (or its percent) freezes, or the deadline | +//! | loading | latest log line is anything else meaningful, port still down | that log line | silent-load limit once the line freezes, or the deadline | +//! | listing | HTTP lists the model, inference has not answered | the log line, or `Waiting for {model} to serve…` | the overall deadline (the server is answering) | +//! | serving | inference answered | wait returns ready | immediate | +//! | failed | engine state `failed`, or every recorded pid is dead | last log line, or a fixed failure line | immediate | +//! +//! A log line that still says "download" does not pull a listed model back +//! into the download stall. The server has answered; the exit stays the +//! deadline. +//! +//! A piped serve keeps the historical status budget and returns the best +//! readiness instead of this failure. That path has no status line; holding +//! it for a full model load would block CI and the assistant on a server +//! that never answers. + +use std::fmt::Write as _; +use std::path::Path; +use std::thread; +use std::time::{Duration, Instant}; + +use rocm_core::EndpointReadiness; + +use crate::engine_runtime_select::{service_http_inference_answers, service_http_lists_model}; +use crate::service_hints::{service_logs_hint, service_stop_hint}; + +/// How long a download percent (or line) may stay unchanged before it is hung. +pub(crate) const DOWNLOAD_STALL: Duration = Duration::from_mins(3); + +/// Piped / captured `rocm serve` returns the best readiness after this long. +/// Interactive summary mode follows the stage instead. +pub(crate) const PIPED_SERVE_STATUS_BUDGET: Duration = Duration::from_secs(45); + +/// Endpoint not listing the model and no meaningful log line. +/// +/// As long as [`SILENT_LOAD`]: a start that has written no meaningful line +/// to its log yet (a slow step before the engine logs anything) is given no +/// less time than a quiet load. Pinned by +/// `a_start_with_no_log_gets_as_long_as_a_quiet_load`. +pub(crate) const SILENT_UNREACHABLE: Duration = SILENT_LOAD; + +/// A load line that stops changing while the endpoint has not listed the model. +pub(crate) const SILENT_LOAD: Duration = Duration::from_mins(5); + +const POLL: Duration = Duration::from_millis(250); +const LOG_TAIL_LINES: usize = 40; +const LOG_TAIL_BYTES: u64 = 64 * 1024; +const MAX_LABEL_CHARS: usize = 160; + +/// How the interactive wait treats time, and whether giving up is an error. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupPolicy { + pub bounds: ServeStartupBounds, + pub poll: Duration, + /// Stop with an error at the deadline and the stall limits. + /// + /// `false` for piped output: the deadline returns the best readiness, the + /// way the 45-second wait did. + pub stop_on_give_up: bool, + /// Read engine-failed and process-liveness. + /// + /// Off for the piped path so a serve that exits in CI still returns a + /// readiness status. + pub watch_engine: bool, + /// How far apart inference probes are once the endpoint lists the model. + pub inference_backoff: InferenceProbeBackoff, +} + +/// Spacing between inference probes while the endpoint lists the model but +/// cannot answer yet. +/// +/// Listing is a short GET; the inference probe is a real generation request +/// that can hold the engine for up to [`rocm_core::INFERENCE_PROBE_TIMEOUT`]. +/// The first probe runs as soon as the model is listed, then the gap starts at +/// `first` and doubles up to `max`. A zero `first` probes every poll. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct InferenceProbeBackoff { + pub first: Duration, + pub max: Duration, +} + +impl InferenceProbeBackoff { + /// Probe on every poll: the piped status budget, bounded at 45 seconds. + pub(crate) const EVERY_POLL: Self = Self { + first: Duration::ZERO, + max: Duration::ZERO, + }; +} + +/// First gap between inference probes on the interactive wait. +pub(crate) const INFERENCE_PROBE_FIRST_GAP: Duration = Duration::from_secs(1); + +/// When the next inference probe may run. See [`InferenceProbeBackoff`]. +#[derive(Debug)] +pub(crate) struct InferenceProbeThrottle { + backoff: InferenceProbeBackoff, + gap: Duration, + next_at: Option, +} + +impl InferenceProbeThrottle { + pub(crate) const fn new(backoff: InferenceProbeBackoff) -> Self { + Self { + backoff, + gap: backoff.first, + next_at: None, + } + } + + /// Readiness for one poll. `probe` runs only when the model is listed and + /// the gap since the last failed probe has passed; the next gap is counted + /// from when that probe returned, read from `clock`. + pub(crate) fn readiness( + &mut self, + listed: bool, + clock: &mut dyn FnMut() -> Instant, + probe: impl FnOnce() -> bool, + ) -> EndpointReadiness { + if !listed { + return EndpointReadiness::Unreachable; + } + if self.next_at.is_some_and(|at| clock() < at) { + return EndpointReadiness::Listing; + } + if probe() { + return EndpointReadiness::Serving; + } + let returned = clock(); + self.next_at = Some(returned.checked_add(self.gap).unwrap_or(returned)); + self.gap = self.gap.saturating_mul(2).min(self.backoff.max); + EndpointReadiness::Listing + } +} + +/// The clocks [`decide_serve_startup`] compares against. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupBounds { + pub deadline: Duration, + pub download_stall: Duration, + pub silent_unreachable: Duration, + pub silent_load: Duration, +} + +impl ServeStartupPolicy { + /// Terminal summary: follow the stage until ready or a bounded failure. + pub(crate) const fn interactive() -> Self { + Self { + bounds: ServeStartupBounds { + deadline: rocm_core::SERVE_STARTUP_DEADLINE, + download_stall: DOWNLOAD_STALL, + silent_unreachable: SILENT_UNREACHABLE, + silent_load: SILENT_LOAD, + }, + poll: POLL, + stop_on_give_up: true, + watch_engine: true, + inference_backoff: InferenceProbeBackoff { + first: INFERENCE_PROBE_FIRST_GAP, + max: rocm_core::INFERENCE_PROBE_RETRY_INTERVAL, + }, + } + } + + /// The policy for a background `rocm serve`: follow the startup only when + /// the summary goes to a terminal *and* stderr, where the status line and + /// any startup failure are written, is one too. With stderr redirected + /// (`rocm serve m 2>log`) nothing would be drawn for up to the full + /// deadline, so that case keeps the piped status budget. + pub(crate) const fn for_background_serve( + summary_on_terminal: bool, + stderr_is_terminal: bool, + ) -> Self { + if summary_on_terminal && stderr_is_terminal { + Self::interactive() + } else { + Self::status_budget(PIPED_SERVE_STATUS_BUDGET) + } + } + + /// Piped / captured output. `timeout` is the historical readiness budget. + pub(crate) const fn status_budget(timeout: Duration) -> Self { + Self { + bounds: ServeStartupBounds { + deadline: timeout, + download_stall: Duration::MAX, + silent_unreachable: Duration::MAX, + silent_load: Duration::MAX, + }, + poll: POLL, + stop_on_give_up: false, + watch_engine: false, + inference_backoff: InferenceProbeBackoff::EVERY_POLL, + } + } +} + +/// One sample of engine and process state, separate from the HTTP probe. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupSignals { + pub engine_failed: bool, + pub process_alive: bool, + /// Latest meaningful log line, already one sanitized line. + pub log_line: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ServeStartupTiming { + pub elapsed: Duration, + pub since_progress: Duration, +} + +/// Lifecycle row the status line is on. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ServeStartupPhase { + Unreachable, + Downloading, + Loading, + Listing, + Serving, + Failed, +} + +/// What the spinner should show for this sample. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupView { + pub phase: ServeStartupPhase, + pub label: String, + /// The engine-log line `label` shows, when it shows one. Set where the + /// label is chosen, so a stop quotes the engine only when the label is + /// its line, never one of this module's own fallback labels. + pub log_line: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ServeStartupStopReason { + EngineFailed, + ProcessExited, + DownloadStalled, + Silent, + SilentLoad, + Deadline, +} + +/// The wait stopped before the server could serve. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ServeStartupStop { + pub phase: ServeStartupPhase, + pub reason: ServeStartupStopReason, + /// Spinner label for this sample. + /// + /// May contain the spinner ellipsis. The failure text quotes + /// [`Self::log_line`] instead, which is ASCII. + pub label: String, + pub log_line: Option, + pub process_alive: bool, + pub readiness: EndpointReadiness, +} + +/// The error `start_managed_service` returns when the wait stopped. +/// +/// `Display` is [`ServeStartupStop::user_message`], plus memory guidance when +/// the server did not start. Only a stop that left the server running +/// (`server_running`) has a process for the caller to supervise; a spawn +/// failure or a server that ended has none. +#[derive(Debug)] +pub(crate) struct ServeStartupFailed { + pub message: String, + pub server_running: bool, +} + +impl std::fmt::Display for ServeStartupFailed { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(&self.message) + } +} + +impl std::error::Error for ServeStartupFailed {} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum ServeStartupDecision { + Continue(ServeStartupView), + Ready, + /// Piped path: the status budget elapsed. Hand back the best readiness. + BudgetElapsed(EndpointReadiness), + Stop(ServeStartupStop), +} + +pub(crate) fn starting_label(model: &str, engine: &str) -> String { + format!("Starting {model} on {engine}…") +} + +pub(crate) fn waiting_label(model: &str) -> String { + format!("Waiting for {model} to serve…") +} + +/// Latest meaningful engine-log line, sanitized to one ASCII line. +/// +/// Carriage-return progress bars keep only their last segment. ANSI and +/// other control bytes are stripped. Blank lines and lines with no words +/// are skipped so a rule of dashes cannot cover a real stage. Credentials +/// are masked ([`redact_log_line`]) before the line is truncated, so a key +/// cut at the length limit cannot show its prefix. +pub(crate) fn latest_meaningful_log_line( + path: &Path, + endpoint_api_key: Option<&str>, +) -> Option { + let lines = rocm_core::tail_lines(path, LOG_TAIL_LINES, LOG_TAIL_BYTES).ok()?; + lines.into_iter().rev().find_map(|line| { + let sanitized = sanitize_log_line(&line, endpoint_api_key); + is_meaningful(&sanitized).then_some(sanitized) + }) +} + +pub(crate) fn sanitize_log_line(raw: &str, endpoint_api_key: Option<&str>) -> String { + let segment = raw + .rsplit(['\r', '\n']) + .find(|part| !part.trim().is_empty()) + .unwrap_or(raw); + let stripped = rocm_core::terminal::strip_terminal_control_sequences(segment); + let ascii: String = stripped.chars().filter(char::is_ascii).collect(); + let collapsed = ascii.split_whitespace().collect::>().join(" "); + truncate_label(&redact_log_line(&collapsed, endpoint_api_key)) +} + +fn truncate_label(line: &str) -> String { + if line.chars().count() <= MAX_LABEL_CHARS { + return line.to_owned(); + } + let mut out: String = line + .chars() + .take(MAX_LABEL_CHARS.saturating_sub(3)) + .collect(); + out.push_str("..."); + out +} + +fn is_meaningful(line: &str) -> bool { + line.chars().filter(char::is_ascii_alphabetic).count() >= 4 && line.len() >= 6 +} + +fn line_is_download(line: &str) -> bool { + let lower = line.to_ascii_lowercase(); + lower.contains("download") || lower.contains("fetching ") || lower.contains("pulling ") +} + +/// A download line still in progress. One whose last percent is 100 is a +/// finished download: what follows is loading, so a quiet engine after it +/// is judged by the load limit, not the download stall. +fn line_is_active_download(line: &str) -> bool { + line_is_download(line) + && !last_percent(line) + .and_then(|pct| pct.parse::().ok()) + .is_some_and(|pct| pct >= 100.0) +} + +/// Last `NN%` in the line, so a bar whose ETA keeps moving but whose percent +/// does not is still stalled. +fn download_fingerprint(line: &str) -> String { + last_percent(line).map_or_else(|| line.to_owned(), |pct| format!("pct:{pct}")) +} + +fn last_percent(line: &str) -> Option<&str> { + let bytes = line.as_bytes(); + let mut search_end = bytes.len(); + while let Some(rel) = line[..search_end].rfind('%') { + // Digits with an optional fraction (`100.0%`), so a decimal percent is + // read whole rather than as the digits after its point. + let mut start = rel; + while start > 0 && (bytes[start - 1].is_ascii_digit() || bytes[start - 1] == b'.') { + start -= 1; + } + let number = line[start..rel].trim_start_matches('.'); + if number.bytes().any(|byte| byte.is_ascii_digit()) { + return Some(number); + } + search_end = rel; + } + None +} + +fn progress_token(readiness: EndpointReadiness, signals: &ServeStartupSignals) -> String { + if signals.engine_failed { + return "engine-failed".to_owned(); + } + if !signals.process_alive { + return "process-exited".to_owned(); + } + if readiness == EndpointReadiness::Listing { + return format!("listing:{}", signals.log_line.as_deref().unwrap_or("")); + } + if let Some(line) = signals + .log_line + .as_deref() + .filter(|line| line_is_active_download(line)) + { + return format!("download:{}", download_fingerprint(line)); + } + if let Some(line) = &signals.log_line { + return format!("log:{line}"); + } + "unreachable".to_owned() +} + +fn classify( + model: &str, + engine: &str, + readiness: EndpointReadiness, + signals: &ServeStartupSignals, +) -> ServeStartupView { + if signals.engine_failed || !signals.process_alive { + let label = signals.log_line.clone().unwrap_or_else(|| { + if signals.engine_failed { + "the engine reported failed".to_owned() + } else { + "the server process exited".to_owned() + } + }); + return ServeStartupView { + phase: ServeStartupPhase::Failed, + label, + log_line: signals.log_line.clone(), + }; + } + if readiness == EndpointReadiness::Serving { + return ServeStartupView { + phase: ServeStartupPhase::Serving, + label: starting_label(model, engine), + log_line: None, + }; + } + // Listing wins over a stale download line. vLLM advertises the model + // before inference answers, and the last log line often still says + // "download". That must not arm the download stall. + if readiness == EndpointReadiness::Listing { + return ServeStartupView { + phase: ServeStartupPhase::Listing, + label: signals + .log_line + .clone() + .unwrap_or_else(|| waiting_label(model)), + log_line: signals.log_line.clone(), + }; + } + if let Some(line) = signals + .log_line + .as_ref() + .filter(|line| line_is_active_download(line)) + { + return ServeStartupView { + phase: ServeStartupPhase::Downloading, + label: line.clone(), + log_line: Some(line.clone()), + }; + } + if let Some(line) = &signals.log_line { + return ServeStartupView { + phase: ServeStartupPhase::Loading, + label: line.clone(), + log_line: Some(line.clone()), + }; + } + ServeStartupView { + phase: ServeStartupPhase::Unreachable, + label: starting_label(model, engine), + log_line: None, + } +} + +fn stop_decision( + view: &ServeStartupView, + reason: ServeStartupStopReason, + process_alive: bool, + readiness: EndpointReadiness, +) -> ServeStartupDecision { + ServeStartupDecision::Stop(ServeStartupStop { + phase: view.phase, + reason, + log_line: view.log_line.clone(), + label: view.label.clone(), + process_alive, + readiness, + }) +} + +/// Decide the next status line and whether the wait stops. +/// +/// `readiness` is the best probe so far, never a downgrade: one refused +/// connection after the model has listed does not put the line back on +/// "Starting" or arm the silent-unreachable exit. +pub(crate) fn decide_serve_startup( + model: &str, + engine: &str, + readiness: EndpointReadiness, + signals: &ServeStartupSignals, + timing: ServeStartupTiming, + policy: &ServeStartupPolicy, +) -> ServeStartupDecision { + let signals = effective_signals(signals.clone(), policy); + let view = classify(model, engine, readiness, &signals); + let process_alive = signals.process_alive; + + if signals.engine_failed { + return stop_decision( + &view, + ServeStartupStopReason::EngineFailed, + process_alive, + readiness, + ); + } + if !process_alive { + return stop_decision( + &view, + ServeStartupStopReason::ProcessExited, + process_alive, + readiness, + ); + } + if readiness == EndpointReadiness::Serving || view.phase == ServeStartupPhase::Serving { + return ServeStartupDecision::Ready; + } + if timing.elapsed >= policy.bounds.deadline { + return if policy.stop_on_give_up { + stop_decision( + &view, + ServeStartupStopReason::Deadline, + process_alive, + readiness, + ) + } else { + ServeStartupDecision::BudgetElapsed(readiness) + }; + } + if view.phase == ServeStartupPhase::Downloading + && timing.since_progress >= policy.bounds.download_stall + { + return stop_decision( + &view, + ServeStartupStopReason::DownloadStalled, + process_alive, + readiness, + ); + } + if view.phase == ServeStartupPhase::Unreachable + && timing.since_progress >= policy.bounds.silent_unreachable + { + return stop_decision( + &view, + ServeStartupStopReason::Silent, + process_alive, + readiness, + ); + } + if view.phase == ServeStartupPhase::Loading + && timing.since_progress >= policy.bounds.silent_load + { + return stop_decision( + &view, + ServeStartupStopReason::SilentLoad, + process_alive, + readiness, + ); + } + ServeStartupDecision::Continue(view) +} + +impl ServeStartupStop { + /// The wait gave up on a server that is still up: a stall or the deadline. + /// + /// Engine failure and a dead process end the launch instead. Those are + /// settled like a `--wait` that saw the server exit: a dead process is + /// recorded `stopped`; a record the engine already marked `failed` keeps + /// that status, and `rocmd`'s recovery treats it like any `failed` service + /// (its message names the stop command while its process is still up); a + /// stop another terminal recorded is left alone. The CLI does not start the + /// background helper for either. A server left running is recorded + /// `running`, which `rocmd` leaves alone until + /// [`rocm_core::SERVE_STARTUP_DEADLINE`] (see + /// `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one`), + /// so it does not kill the server this message says is still running. + /// [`decide_serve_startup`] returns `ProcessExited` before it checks any + /// stall or the deadline, so those stops always have `process_alive`. + pub(crate) const fn leaves_server_running(&self) -> bool { + self.process_alive + && !matches!( + self.reason, + ServeStartupStopReason::EngineFailed | ServeStartupStopReason::ProcessExited + ) + } + + /// Text the CLI prints when the wait stops. + /// + /// A log line is quoted only when [`Self::label`] is that line. The + /// spinner's own "Starting" and "Waiting" labels are not quoted. + pub(crate) fn user_message(&self, model: &str, engine: &str, service_id: &str) -> String { + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + let headline = match self.reason { + ServeStartupStopReason::EngineFailed => { + format!("{model} on {engine} failed: the engine reported failed") + } + ServeStartupStopReason::ProcessExited => { + format!("{model} on {engine} failed: the server process exited") + } + ServeStartupStopReason::DownloadStalled => { + format!("{model} on {engine} stopped making download progress") + } + ServeStartupStopReason::Silent => { + format!("{model} on {engine} produced no log and never answered") + } + ServeStartupStopReason::SilentLoad => { + format!("{model} on {engine} stopped making progress while loading") + } + ServeStartupStopReason::Deadline => { + format!("{model} on {engine} did not become ready within {minutes} minutes") + } + }; + let mut message = headline; + if let Some(line) = &self.log_line { + message.push_str(": "); + message.push_str(line); + } + message.push('\n'); + let logs = service_logs_hint(service_id); + if self.leaves_server_running() { + let stop = service_stop_hint(service_id); + let _ = write!( + message, + "The server is still running (service {service_id}). Follow it with `{logs}`, or stop it with `{stop}`. \ + If it is still not answering {minutes} minutes after it started, a running `rocm daemon` \ + with `server-recover` enabled may restart it." + ); + } else if self.process_alive { + // The engine said failed but its process has not exited yet. + let stop = service_stop_hint(service_id); + let _ = write!( + message, + "Its process is still up (service {service_id}). The log is `{logs}`; `{stop}` stops it. \ + A running `rocm daemon` with `server-recover` enabled may restart it first." + ); + } else { + let _ = write!( + message, + "The server did not start (service {service_id}). The log is `{logs}`." + ); + } + message + } +} + +/// Poll until the endpoint can serve, reporting each stage through `on_view`. +/// +/// `observe` returns the model name the probe should ask for (a container +/// engine may not know it until it starts) and the latest engine/process +/// sample. The same HTTP checks as the historical readiness wait are used, so +/// "serving" still means an inference request came back. Once the model is +/// listed, the inference request is spaced by the policy's +/// [`InferenceProbeBackoff`] rather than sent every poll. +/// +/// The probe runs inline, so a poll that sends one can take up to the probe +/// timeout: the spinner does not tick and an engine failure is not read until +/// it returns. The backoff bounds how often that happens; it is not moved off +/// the polling thread. +#[allow(clippy::too_many_arguments)] +pub(crate) fn wait_for_serve_startup( + engine_name: &str, + model: &str, + host: &str, + port: u16, + endpoint_api_key: Option<&str>, + policy: &ServeStartupPolicy, + observe: &mut dyn FnMut() -> (String, ServeStartupSignals), + on_view: &mut dyn FnMut(&ServeStartupView), +) -> Result { + let started = Instant::now(); + let mut progress_at = started; + let mut token = String::new(); + let mut best = EndpointReadiness::Unreachable; + let mut throttle = InferenceProbeThrottle::new(policy.inference_backoff); + + // One observe, one probe, one decision and at most one status-line update + // per poll, so the spinner advances once per tick. + loop { + let (probe_model, signals) = observe(); + let mut signals = effective_signals(signals, policy); + signals.log_line = signals + .log_line + .map(|line| redact_log_line(&line, endpoint_api_key)); + // A failed engine or a dead process is decided without waiting on a + // probe that cannot succeed. + if !signals.engine_failed && signals.process_alive { + let listed = + service_http_lists_model(engine_name, host, port, &probe_model, endpoint_api_key); + match throttle.readiness(listed, &mut Instant::now, || { + service_http_inference_answers(host, port, &probe_model, endpoint_api_key) + }) { + EndpointReadiness::Serving => best = EndpointReadiness::Serving, + EndpointReadiness::Listing => best = EndpointReadiness::Listing, + EndpointReadiness::Unreachable => {} + } + } + let next_token = progress_token(best, &signals); + if next_token != token { + token = next_token; + progress_at = Instant::now(); + } + let timing = ServeStartupTiming { + elapsed: started.elapsed(), + since_progress: progress_at.elapsed(), + }; + match decide_serve_startup(model, engine_name, best, &signals, timing, policy) { + ServeStartupDecision::Continue(view) => on_view(&view), + ServeStartupDecision::Ready => return Ok(EndpointReadiness::Serving), + ServeStartupDecision::BudgetElapsed(readiness) => return Ok(readiness), + ServeStartupDecision::Stop(stop) => return Err(stop), + } + + let remaining = policy.bounds.deadline.saturating_sub(started.elapsed()); + if remaining.is_zero() { + continue; + } + thread::sleep(policy.poll.min(remaining)); + } +} + +/// The signals the policy looks at. +/// +/// The piped path ignores engine failure and process liveness so a serve that +/// exits in CI still returns a readiness status. +const fn effective_signals( + mut signals: ServeStartupSignals, + policy: &ServeStartupPolicy, +) -> ServeStartupSignals { + if !policy.watch_engine { + signals.engine_failed = false; + signals.process_alive = true; + } + signals +} + +const REDACTED: &str = "[redacted]"; + +/// Mask credentials in an engine log line before it reaches the status line +/// or the failure text. +/// +/// Engines echo their launch arguments (vLLM prints its non-default args), +/// and those can carry the endpoint key or a hub token. The line is already +/// one ASCII line with single spaces. Masked: the endpoint key wherever it +/// appears, the value after a secret-named flag or key (`--api-key X`, +/// `api_key=X`, `'api_key': 'X'`, `"api_key":"X"`, `Authorization: Bearer X`), +/// and Hugging Face `hf_` tokens. A list value (`'api_key': ['a', 'b']`) has +/// only its first item masked. +pub(crate) fn redact_log_line(line: &str, endpoint_api_key: Option<&str>) -> String { + let line = match endpoint_api_key.filter(|key| !key.is_empty()) { + Some(key) => line.replace(key, REDACTED), + None => line.to_owned(), + }; + let mut out: Vec = Vec::new(); + let mut mask_next = false; + for word in line.split(' ') { + if mask_next { + mask_next = false; + if word.eq_ignore_ascii_case("bearer") || word.eq_ignore_ascii_case("basic") { + out.push(word.to_owned()); + mask_next = true; + continue; + } + out.push(mask_value(word)); + continue; + } + if let Some((name, value)) = word.split_once('=') + && is_secret_name(name) + { + out.push(if value.is_empty() { + format!("{name}=") + } else { + format!("{name}={}", mask_value(value)) + }); + continue; + } + // Compact JSON or YAML: `"api_key":"X"`. + if let Some((name, value)) = word.split_once(':') + && !value.is_empty() + && is_secret_name(name) + { + out.push(format!("{name}:{}", mask_compact_value(value))); + continue; + } + if names_a_following_secret(word) { + mask_next = true; + out.push(word.to_owned()); + continue; + } + if looks_like_hf_token(word) { + out.push(REDACTED.to_owned()); + continue; + } + out.push(word.to_owned()); + } + out.join(" ") +} + +/// Whether `word` names a secret (`--api-key`, `HF_TOKEN`, `'api_key':`, +/// `Authorization:`, `Bearer`). +fn is_secret_name(word: &str) -> bool { + let name = word + .trim_start_matches('-') + .trim_matches(|c: char| matches!(c, '\'' | '"' | ':' | ',' | '{' | '(')) + .to_ascii_lowercase() + .replace('-', "_"); + if name.is_empty() { + return false; + } + name == "authorization" + || name == "bearer" + || ["api_key", "apikey", "token", "password", "secret", "private_key"] + .iter() + .any(|suffix| name.ends_with(suffix)) + // `secret_key`, `token_value`: a secret word anywhere in a `_` name. + || name + .split('_') + .any(|part| matches!(part, "secret" | "token" | "password" | "passwd")) +} + +/// A flag (`--api-key`), a key ending in `:` (`'api_key':`, +/// `Authorization:`), or `Bearer`, whose next word is the secret. A bare word +/// such as "token" in prose does not mask what follows it. +fn names_a_following_secret(word: &str) -> bool { + if word.eq_ignore_ascii_case("bearer") { + return true; + } + let unquoted = word.trim_end_matches(['\'', '"', ',']); + (word.starts_with('-') || unquoted.ends_with(':')) && is_secret_name(word) +} + +/// Keep the quoting and trailing punctuation around a masked value, so a +/// Python-dict log line stays readable. +fn mask_value(word: &str) -> String { + if word.contains(REDACTED) { + return word.to_owned(); + } + let body_start = word + .find(|c: char| !matches!(c, '\'' | '"' | '(' | '[')) + .unwrap_or(word.len()); + // The end of the last body character, which may be wider than one byte. + let body_end = word + .char_indices() + .rev() + .find(|(_, c)| !matches!(c, '\'' | '"' | ',' | ')' | ']' | '}')) + .map_or(body_start, |(index, c)| index + c.len_utf8()); + if body_end <= body_start { + return word.to_owned(); + } + format!("{}{REDACTED}{}", &word[..body_start], &word[body_end..]) +} + +/// Mask one compact value (`"X","port":8000}` or `X,port:8000}`) and keep +/// what follows it, so the rest of a one-word JSON object stays readable. +fn mask_compact_value(value: &str) -> String { + if let Some(quote) = value.chars().next().filter(|c| matches!(c, '"' | '\'')) { + let rest = &value[1..]; + return match rest.find(quote) { + Some(end) if end > 0 => format!("{quote}{REDACTED}{}", &rest[end..]), + _ => value.to_owned(), + }; + } + let end = value.find([',', '}', ']']).unwrap_or(value.len()); + if end == 0 { + return value.to_owned(); + } + format!("{REDACTED}{}", &value[end..]) +} + +fn looks_like_hf_token(word: &str) -> bool { + let body = word.trim_matches(|c: char| !c.is_ascii_alphanumeric() && c != '_'); + body.strip_prefix("hf_") + .is_some_and(|rest| rest.len() >= 20 && rest.chars().all(|c| c.is_ascii_alphanumeric())) +} + +#[cfg(test)] +mod tests { + use std::io::{Read, Write}; + use std::net::TcpListener; + use std::thread; + use std::time::{Duration, Instant}; + + use super::*; + use crate::service_hints::{service_logs_hint, service_stop_hint}; + use crate::test_support::unique_suffix; + + fn alive() -> ServeStartupSignals { + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: None, + } + } + + fn at(elapsed_ms: u64, since_ms: u64) -> ServeStartupTiming { + ServeStartupTiming { + elapsed: Duration::from_millis(elapsed_ms), + since_progress: Duration::from_millis(since_ms), + } + } + + fn interactive() -> ServeStartupPolicy { + ServeStartupPolicy::interactive() + } + + fn assert_continue(decision: ServeStartupDecision) -> ServeStartupView { + match decision { + ServeStartupDecision::Continue(view) => view, + other => panic!("expected the wait to keep going, got {other:?}"), + } + } + + fn assert_stop(decision: ServeStartupDecision) -> ServeStartupStop { + match decision { + ServeStartupDecision::Stop(stop) => stop, + other => panic!("expected the wait to stop, got {other:?}"), + } + } + + #[test] + fn unreachable_keeps_the_starting_label_until_the_silent_limit() { + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Unreachable); + assert_eq!(view.label, starting_label("qwen", "vllm")); + } + + #[test] + fn unreachable_with_no_log_stops_when_nothing_arrives() { + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at( + SILENT_UNREACHABLE.as_millis() as u64, + SILENT_UNREACHABLE.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::Silent); + assert_eq!(stop.phase, ServeStartupPhase::Unreachable); + assert_eq!(stop.label, starting_label("qwen", "vllm")); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!( + message.contains("produced no log and never answered"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + assert!(message.contains(&service_stop_hint("svc-1")), "{message}"); + assert!( + !message.contains("Deployment summary"), + "a silent start must not become a finished summary: {message}" + ); + // The stop command is the one the approval gate runs. That it stops + // a live service is `@id:service-stop-yes-stops-a-running-service` + // in tests/e2e-cucumber/features/service_stop.feature. The logs + // command is `services_text::render_service_logs_text`, proven to + // tail the service log by `render_service_logs_text_tails_manifest_log`. + } + + #[test] + fn downloading_past_the_old_budget_keeps_reporting_the_log_line() { + let line = "Downloading model.safetensors: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + // 50s is past the old 45s budget. Progress moved 1s ago, so this is + // not a hung download. + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(50_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Downloading); + assert_eq!(view.label, line); + } + + #[test] + fn hung_download_stops_and_the_message_quotes_the_stalled_line() { + let line = "Downloading model.safetensors: 12%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at( + DOWNLOAD_STALL.as_millis() as u64 + 1_000, + DOWNLOAD_STALL.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!( + message.contains("stopped making download progress"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + assert!( + !message.contains("not ready yet"), + "a hung download is a failure, not a finished summary: {message}" + ); + assert!( + Duration::from_millis(DOWNLOAD_STALL.as_millis() as u64) + < rocm_core::SERVE_STARTUP_DEADLINE, + "the stall must fire before the overall deadline" + ); + } + + #[test] + fn a_percent_that_does_not_change_is_the_download_fingerprint() { + assert_eq!( + download_fingerprint("Downloading model.safetensors: 12%| ETA 0:40"), + "pct:12" + ); + assert_eq!( + download_fingerprint("Downloading model.safetensors: 12%| ETA 0:01"), + "pct:12", + "a moving ETA at the same percent is not progress" + ); + assert_ne!( + download_fingerprint("Downloading model.safetensors: 13%"), + download_fingerprint("Downloading model.safetensors: 12%") + ); + } + + #[test] + fn loading_past_the_old_budget_keeps_reporting_the_log_line() { + let line = "Loading safetensors checkpoint shards: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(50_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + assert_eq!(view.label, line); + } + + #[test] + fn loading_with_no_new_log_stops() { + let line = "Loading safetensors checkpoint shards: 40%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at( + SILENT_LOAD.as_millis() as u64 + 1_000, + SILENT_LOAD.as_millis() as u64, + ), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!( + message.contains("stopped making progress while loading"), + "{message}" + ); + assert!(message.contains("The server is still running"), "{message}"); + } + + #[test] + fn listing_past_the_old_budget_keeps_waiting() { + let mut signals = alive(); + signals.log_line = Some("Loading safetensors checkpoint shards: 80%".to_owned()); + // The log has been quiet for longer than the load stall. Listing + // still waits: the server is answering, and a GPU load can sit on + // one line. The exit is the overall deadline, not the stall. + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &signals, + at(50_000, SILENT_LOAD.as_millis() as u64), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, "Loading safetensors checkpoint shards: 80%"); + } + + /// A download line at 100% is a finished download: a quiet engine after + /// it is loading, so the download stall must not fire at 3 minutes; the + /// load limit decides, at 5. + /// A start with no log line yet is not held to a shorter limit than a + /// quiet load: neither has shown a hang the other has not. + #[test] + fn a_start_with_no_log_gets_as_long_as_a_quiet_load() { + assert!(SILENT_UNREACHABLE >= SILENT_LOAD); + let just_under = SILENT_LOAD.as_millis() as u64 - 1_000; + assert_continue(decide_serve_startup( + "qwen", + "aim", + EndpointReadiness::Unreachable, + &alive(), + at(just_under, just_under), + &interactive(), + )); + } + + #[test] + fn a_finished_download_line_is_judged_as_loading() { + let line = "Downloading model.safetensors: 100%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let past_download_stall = DOWNLOAD_STALL.as_millis() as u64 + 1_000; + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + assert_eq!(view.log_line.as_deref(), Some(line)); + let past_load = SILENT_LOAD.as_millis() as u64 + 1_000; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_load + 1_000, past_load), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + // A decimal percent is read whole: `100.0%` is finished too. + signals.log_line = Some("Downloading model.safetensors: 100.0%".to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Loading); + // An unfinished percent is still a download, decimal or not. + signals.log_line = Some("Downloading model.safetensors: 99.9%".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + signals.log_line = Some("Downloading model.safetensors: 99%".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(past_download_stall + 1_000, past_download_stall), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + } + + /// A stop quotes the engine only when its label is the engine's line, + /// decided where the label is chosen: an engine line that happens to read + /// like one of the fallback labels is still quoted, and a fallback label + /// is never quoted. + #[test] + fn a_stop_quotes_the_engine_line_not_a_fallback_label() { + let mut signals = alive(); + signals.process_alive = false; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(stop.label, "the server process exited"); + assert_eq!(stop.log_line, None); + signals.log_line = Some("the server process exited".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(stop.log_line.as_deref(), Some("the server process exited")); + } + + #[test] + fn listing_keeps_a_stale_download_line_off_the_download_stall() { + // vLLM often still has "download" as the last log line after it has + // started advertising the model. That line must not arm the stall: + // the server has answered, and the exit is the overall deadline. + let line = "Downloading model.safetensors: 100%"; + let mut signals = alive(); + signals.log_line = Some(line.to_owned()); + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &signals, + at(50_000, DOWNLOAD_STALL.as_millis() as u64), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, line); + } + + #[test] + fn listing_without_a_log_says_it_is_waiting_to_serve() { + let view = assert_continue(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &alive(), + at(1_000, 1_000), + &interactive(), + )); + assert_eq!(view.phase, ServeStartupPhase::Listing); + assert_eq!(view.label, waiting_label("qwen")); + } + + #[test] + fn listing_at_the_deadline_stops_without_a_not_ready_summary() { + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Listing, + &alive(), + at(rocm_core::SERVE_STARTUP_DEADLINE.as_millis() as u64, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + assert_eq!(stop.label, waiting_label("qwen")); + let message = stop.user_message("qwen", "vllm", "svc-1"); + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + assert!(message.contains(&format!("{minutes} minutes")), "{message}"); + assert!(message.contains("The server is still running"), "{message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + assert!( + !message.contains("not ready yet") && !message.contains("Deployment summary"), + "{message}" + ); + } + + #[test] + fn serving_is_ready() { + let decision = decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Serving, + &alive(), + at(1_000, 0), + &interactive(), + ); + assert!(matches!(decision, ServeStartupDecision::Ready)); + } + + #[test] + fn engine_failed_stops_immediately_and_quotes_the_log() { + let mut signals = alive(); + signals.engine_failed = true; + signals.log_line = Some("CUDA out of memory".to_owned()); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(0, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::EngineFailed); + assert_eq!(stop.label, "CUDA out of memory"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains("the engine reported failed"), "{message}"); + assert!(message.contains("CUDA out of memory"), "{message}"); + } + + #[test] + fn process_exited_stops_without_claiming_the_server_is_running() { + let mut signals = alive(); + signals.process_alive = false; + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(0, 0), + &interactive(), + )); + assert_eq!(stop.reason, ServeStartupStopReason::ProcessExited); + assert!(!stop.process_alive); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains("the server process exited"), "{message}"); + assert!(message.contains("The server did not start"), "{message}"); + assert!( + !message.contains("still running"), + "a dead process must not be described as still running: {message}" + ); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + } + + #[test] + fn piped_serve_returns_at_the_status_budget_instead_of_following_a_download() { + let mut signals = alive(); + signals.log_line = Some("Downloading model.safetensors: 40%".to_owned()); + let policy = ServeStartupPolicy::status_budget(Duration::from_secs(45)); + let decision = decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + at(45_000, 0), + &policy, + ); + assert_eq!( + decision, + ServeStartupDecision::BudgetElapsed(EndpointReadiness::Unreachable) + ); + } + + #[test] + fn sanitize_log_line_keeps_the_latest_progress_segment_and_drops_ansi() { + let raw = "\u{1b}[32mold line\rDownloading model.safetensors: 12%\u{1b}[0m"; + assert_eq!( + sanitize_log_line(raw, None), + "Downloading model.safetensors: 12%" + ); + } + + #[test] + fn latest_meaningful_log_line_skips_blank_and_rule_lines() { + let path = std::env::temp_dir().join(format!("serve-progress-{}.log", unique_suffix())); + std::fs::write( + &path, + "Loading safetensors checkpoint shards: 40%\n\n----------\n", + ) + .unwrap(); + let line = latest_meaningful_log_line(&path, None); + let _ = std::fs::remove_file(&path); + assert_eq!( + line.as_deref(), + Some("Loading safetensors checkpoint shards: 40%") + ); + } + + #[test] + fn failure_message_names_the_same_logs_command_as_the_summary() { + let hint = service_logs_hint("svc-1"); + let summary = crate::serve_summary::DeploymentSummary { + engine: "vllm".to_owned(), + requested_model: "qwen".to_owned(), + api_model: "qwen".to_owned(), + chat_endpoint: "http://127.0.0.1:1/v1/chat/completions".to_owned(), + anthropic_endpoint: None, + service_id: "svc-1".to_owned(), + status: "starting".to_owned(), + already_running: false, + metrics: crate::serve_summary::SmokeMetrics::default(), + smoke_failure: None, + api_key: None, + notes: Vec::new(), + }; + let rendered = crate::serve_summary::render_summary(&summary); + // The message names only the hint: the rendered summary can carry an + // endpoint key. + assert!( + rendered.contains(&hint), + "the deployment summary must name `{hint}`" + ); + let stop = assert_stop(decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &alive(), + at( + SILENT_UNREACHABLE.as_millis() as u64, + SILENT_UNREACHABLE.as_millis() as u64, + ), + &interactive(), + )); + assert!(stop.user_message("qwen", "vllm", "svc-1").contains(&hint)); + } + + fn short_policy(deadline: Duration, download_stall: Duration) -> ServeStartupPolicy { + ServeStartupPolicy { + bounds: ServeStartupBounds { + deadline, + download_stall, + silent_unreachable: Duration::from_secs(30), + silent_load: Duration::from_secs(30), + }, + poll: Duration::from_millis(20), + stop_on_give_up: true, + watch_engine: true, + inference_backoff: InferenceProbeBackoff::EVERY_POLL, + } + } + + /// The interactive wait's inference probe, under a clock that moves only + /// when the test says: a model listed but never answering is probed at 0, + /// 1, 3, 7 and 15 seconds, then every 15, never once per 250 ms poll. Each + /// probe takes the full 8-second timeout, and the next gap counts from + /// when it returned. + #[test] + fn inference_probe_backs_off_to_the_retry_interval_after_listing() { + use std::cell::Cell; + let start = Instant::now(); + let now = Cell::new(start); + let mut clock = || now.get(); + let mut throttle = + InferenceProbeThrottle::new(ServeStartupPolicy::interactive().inference_backoff); + let mut probed_at = Vec::new(); + // Ten minutes of 250 ms polls. + while now.get() - start < Duration::from_mins(10) { + let readiness = throttle.readiness(true, &mut clock, || { + probed_at.push((now.get() - start).as_secs()); + now.set(now.get() + rocm_core::INFERENCE_PROBE_TIMEOUT); + false + }); + assert_eq!(readiness, EndpointReadiness::Listing); + now.set(now.get() + POLL); + } + let gaps: Vec = probed_at + .windows(2) + .map(|pair| pair[1] - pair[0] - rocm_core::INFERENCE_PROBE_TIMEOUT.as_secs()) + .collect(); + assert_eq!(&gaps[..5], &[1, 2, 4, 8, 15], "probed at {probed_at:?}"); + assert!( + gaps[4..].iter().all(|gap| *gap == 15), + "the gap must stay at the retry interval: {gaps:?}" + ); + // 2400 polls; one probe each would be 2400 eight-second requests. + assert!(probed_at.len() < 30, "probed {} times", probed_at.len()); + } + + /// Not listed, the probe never runs; listed, the first probe is not + /// delayed, and a probe that answers is ready at once. + #[test] + fn inference_probe_runs_at_once_when_listed_and_not_before() { + let now = Instant::now(); + let mut clock = || now; + let mut throttle = + InferenceProbeThrottle::new(ServeStartupPolicy::interactive().inference_backoff); + assert_eq!( + throttle.readiness(false, &mut clock, || panic!("not listed, no probe")), + EndpointReadiness::Unreachable + ); + assert_eq!( + throttle.readiness(true, &mut clock, || true), + EndpointReadiness::Serving + ); + } + + /// The same backoff through the real wait: an endpoint that lists the model + /// and refuses every chat request gets a handful of inference requests over + /// a second of 10 ms polls, not one per poll. + #[test] + fn wait_spaces_inference_probes_while_the_model_is_listed() { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let chat_requests = Arc::new(AtomicUsize::new(0)); + let counted = Arc::clone(&chat_requests); + let _server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + counted.fetch_add(1, Ordering::SeqCst); + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } else { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + let mut policy = short_policy(Duration::from_secs(1), Duration::from_secs(30)); + policy.poll = Duration::from_millis(10); + policy.inference_backoff = InferenceProbeBackoff { + first: Duration::from_millis(100), + max: Duration::from_millis(200), + }; + let mut polls = 0_usize; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &policy, + &mut || { + polls += 1; + ("qwen".to_owned(), alive()) + }, + &mut |_| {}, + ); + let stop = result.expect_err("a model that never answers hits the deadline"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + let probes = chat_requests.load(Ordering::SeqCst); + // 0, 100, 300, 500, 700, 900 ms: at most a few more under a slow host, + // where an unthrottled wait sends one per poll. + assert!( + (1..=8).contains(&probes), + "{probes} inference requests over {polls} polls" + ); + } + + #[test] + fn wait_follows_a_download_until_the_server_serves() { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let started = Instant::now(); + let serve_after = Duration::from_millis(400); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let inference_ready = started.elapsed() >= serve_after; + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + if inference_ready { + ( + "HTTP/1.1 200 OK", + r#"{"choices":[{"message":{"content":"ok"}}]}"#, + ) + } else { + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } + } else if inference_ready { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + } else { + // Still downloading: the endpoint does not list the model. + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + + let mut views = Vec::new(); + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: Some("Downloading model.safetensors: 40%".to_owned()), + }, + ) + }, + &mut |view| views.push(view.clone()), + ); + drop(server); + + assert_eq!(result.unwrap(), EndpointReadiness::Serving); + assert!( + started.elapsed() >= serve_after, + "the wait returned before the server could serve" + ); + assert!( + views + .iter() + .any(|view| view.phase == ServeStartupPhase::Downloading + && view.label == "Downloading model.safetensors: 40%"), + "the status line must show the download while the wait continues: {views:?}" + ); + } + + #[test] + fn wait_stops_a_hung_download_before_the_deadline() { + let line = "Downloading model.safetensors: 12%"; + let stall = Duration::from_millis(200); + let deadline = Duration::from_secs(5); + let started = Instant::now(); + let mut labels = Vec::new(); + // The endpoint never lists the model, and answers at once. + let port = not_listing_port(); + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(deadline, stall), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + log_line: Some(line.to_owned()), + }, + ) + }, + &mut |view| labels.push(view.label.clone()), + ); + let elapsed = started.elapsed(); + let stop = result.expect_err("a frozen download must fail the wait"); + assert_eq!(stop.reason, ServeStartupStopReason::DownloadStalled); + assert_eq!(stop.label, line); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(message.contains(line), "{message}"); + assert!(message.contains("still running"), "{message}"); + assert!( + labels.iter().any(|label| label == line), + "the spinner must have shown the stalled line: {labels:?}" + ); + assert!( + elapsed < Duration::from_secs(3), + "stall fired at {elapsed:?}, not before the {deadline:?} deadline" + ); + } + + #[test] + fn wait_reports_serving_once_inference_answers_from_the_first_probe() { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let body = if request.starts_with("POST /v1/chat/completions ") { + r#"{"choices":[{"message":{"content":"ok"}}]}"# + } else { + r#"{"data":[{"id":"qwen"}]}"# + }; + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + let started = Instant::now(); + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || ("qwen".to_owned(), alive()), + &mut |_| {}, + ); + drop(server); + assert_eq!(result.unwrap(), EndpointReadiness::Serving); + assert!( + started.elapsed() < SILENT_UNREACHABLE, + "a server that is already serving must not wait out the silent limit" + ); + } + + /// A local endpoint that answers every request with 503 at once: the + /// model is never listed. Not a closed port: on Windows a refused loopback + /// connect takes about two seconds, which outlasts these tests' limits. + fn not_listing_port() -> u16 { + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let _ = stream.read(&mut buffer); + let body = r#"{"error":"loading"}"#; + let _ = write!( + stream, + "HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + port + } + + /// Piped output returns a readiness status at the budget even when the + /// engine reports failure or the process is gone: that is the historical + /// 45-second contract scripts and CI read, and it has no stop. + #[test] + fn piped_wait_ignores_engine_failure_and_a_dead_process() { + let port = not_listing_port(); + let mut policy = ServeStartupPolicy::status_budget(Duration::from_millis(200)); + policy.poll = Duration::from_millis(20); + let mut polls = 0; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &policy, + &mut || { + polls += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: true, + process_alive: false, + log_line: None, + }, + ) + }, + &mut |_| {}, + ); + assert_eq!(result, Ok(EndpointReadiness::Unreachable)); + assert!( + polls > 1, + "the piped wait must run to its budget, not stop at once" + ); + } + + /// Both silent limits count the silence, from the last change of what + /// the wait observes, not the whole wait: a long startup that goes quiet + /// is judged on how long it has been quiet. Each row moves one clock past + /// the limit and holds the other below it, so a check reading `elapsed` + /// fails a row. + #[test] + fn silent_limits_count_from_the_last_change() { + let decide = |log_line: Option<&str>, elapsed: Duration, since: Duration| { + let mut signals = alive(); + signals.log_line = log_line.map(str::to_owned); + decide_serve_startup( + "qwen", + "vllm", + EndpointReadiness::Unreachable, + &signals, + ServeStartupTiming { + elapsed, + since_progress: since, + }, + &interactive(), + ) + }; + let short = Duration::from_secs(1); + // Unreachable, no log: only now silent, after a long startup. + assert!(matches!( + decide(None, SILENT_UNREACHABLE * 3, short), + ServeStartupDecision::Continue(_) + )); + let stop = assert_stop(decide(None, SILENT_UNREACHABLE, SILENT_UNREACHABLE)); + assert_eq!(stop.reason, ServeStartupStopReason::Silent); + // Loading: the time since the line last changed decides, however long + // the wait has run. + let load = Some("Loading safetensors checkpoint shards: 40%"); + let stop = assert_stop(decide(load, SILENT_LOAD, SILENT_LOAD)); + assert_eq!(stop.reason, ServeStartupStopReason::SilentLoad); + assert!(matches!( + decide(load, SILENT_LOAD * 2, short), + ServeStartupDecision::Continue(_) + )); + } + + /// Only a terminal on both streams follows the startup; a redirected + /// stderr would hide the status line for up to the whole deadline. + #[test] + fn a_background_serve_follows_the_startup_only_with_both_streams_on_a_terminal() { + let interactive = ServeStartupPolicy::interactive(); + let piped = ServeStartupPolicy::status_budget(PIPED_SERVE_STATUS_BUDGET); + assert_eq!( + ServeStartupPolicy::for_background_serve(true, true), + interactive + ); + for (summary, stderr) in [(true, false), (false, true), (false, false)] { + assert_eq!( + ServeStartupPolicy::for_background_serve(summary, stderr), + piped, + "summary on terminal={summary} stderr terminal={stderr}" + ); + } + } + + #[test] + fn only_a_stall_or_the_deadline_leaves_the_server_running() { + let stop = |reason, process_alive| ServeStartupStop { + phase: ServeStartupPhase::Loading, + reason, + label: String::new(), + log_line: None, + process_alive, + readiness: EndpointReadiness::Unreachable, + }; + // The engine said failed, or the process is gone: the launch is over, + // so the message does not say the server is still running. (That the + // CLI does not supervise it is + // `a_startup_stop_starts_supervision_and_keeps_its_own_error`.) + for (reason, alive) in [ + (ServeStartupStopReason::EngineFailed, true), + (ServeStartupStopReason::EngineFailed, false), + (ServeStartupStopReason::ProcessExited, false), + ] { + let stop = stop(reason, alive); + assert!(!stop.leaves_server_running(), "{reason:?}"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!(!message.contains("still running"), "{reason:?}: {message}"); + assert!(message.contains(&service_logs_hint("svc-1")), "{message}"); + // A failed engine whose process is still up gets the command that + // stops it; one that is gone does not. + assert_eq!( + message.contains(&service_stop_hint("svc-1")), + alive, + "{reason:?} alive={alive}: {message}" + ); + // True of rocmd: `a_failed_record_with_a_live_process_is_still_recovered` + // in apps/rocmd/src/watchers/server_recover.rs. + assert_eq!( + message.contains("may restart it first"), + alive, + "{reason:?} alive={alive}: {message}" + ); + assert_eq!( + message.contains("Its process is still up"), + alive, + "{reason:?} alive={alive}: {message}" + ); + assert_eq!( + message.contains("The server did not start"), + !alive, + "{reason:?} alive={alive}: {message}" + ); + } + for reason in [ + ServeStartupStopReason::DownloadStalled, + ServeStartupStopReason::Silent, + ServeStartupStopReason::SilentLoad, + ServeStartupStopReason::Deadline, + ] { + let stop = stop(reason, true); + assert!(stop.leaves_server_running(), "{reason:?}"); + // The restart this names is real: `rocmd` spares a live startup only + // until the deadline, which + // `recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one` + // and `recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline` + // in apps/rocmd/src/watchers/server_recover.rs prove. + let minutes = rocm_core::SERVE_STARTUP_DEADLINE.as_secs() / 60; + let message = stop.user_message("qwen", "vllm", "svc-1"); + assert!( + message.contains(&format!( + "{minutes} minutes after it started, a running `rocm daemon` with \ + `server-recover` enabled may restart it" + )), + "{reason:?}: {message}" + ); + } + } + + #[test] + fn wait_stops_at_once_when_the_process_has_exited() { + let started = Instant::now(); + let mut views = 0; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + not_listing_port(), + None, + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: false, + log_line: Some("Traceback: boom".to_owned()), + }, + ) + }, + &mut |_| views += 1, + ); + let stop = result.expect_err("a dead process cannot become ready"); + assert_eq!(stop.reason, ServeStartupStopReason::ProcessExited); + assert_eq!(views, 0, "no spinner frame for a server that is gone"); + assert!(started.elapsed() < Duration::from_secs(2)); + } + + #[test] + fn wait_hands_back_listing_at_the_deadline_with_a_stop() { + // Row: listing, inference never answers. The interactive wait ends in + // a deadline stop (not a finished summary) carrying `Listing`; the + // piped budget returns `Listing` as `Ok`. + let listener = TcpListener::bind(("127.0.0.1", 0)).unwrap(); + let port = listener.local_addr().unwrap().port(); + let server = thread::spawn(move || { + while let Ok((mut stream, _)) = listener.accept() { + stream.set_read_timeout(Some(Duration::from_secs(2))).ok(); + let mut buffer = [0_u8; 2048]; + let Ok(read) = stream.read(&mut buffer) else { + continue; + }; + let request = String::from_utf8_lossy(&buffer[..read]); + let (status_line, body) = if request.starts_with("POST /v1/chat/completions ") { + ("HTTP/1.1 503 Service Unavailable", r#"{"error":"loading"}"#) + } else { + ("HTTP/1.1 200 OK", r#"{"data":[{"id":"qwen"}]}"#) + }; + let _ = write!( + stream, + "{status_line}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } + }); + + let mut labels = Vec::new(); + let interactive_result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &short_policy(Duration::from_millis(1_500), Duration::from_secs(30)), + &mut || ("qwen".to_owned(), alive()), + &mut |view| labels.push(view.label.clone()), + ); + let mut budget = ServeStartupPolicy::status_budget(Duration::from_millis(1_500)); + budget.poll = Duration::from_millis(20); + let piped_result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + port, + None, + &budget, + &mut || ("qwen".to_owned(), alive()), + &mut |_| {}, + ); + drop(server); + + let stop = interactive_result.expect_err("the interactive wait must end in a stop"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + assert_eq!(stop.readiness, EndpointReadiness::Listing); + assert!( + labels.iter().any(|label| *label == waiting_label("qwen")), + "once the model is listed the line says it is waiting to serve: {labels:?}" + ); + assert_eq!(piped_result.unwrap(), EndpointReadiness::Listing); + } + + #[test] + fn wait_keeps_a_changing_download_alive_past_the_stall_limit() { + // A percent that keeps advancing re-arms the stall clock, so a long + // healthy download is not mistaken for a hung one. + let stall = Duration::from_millis(300); + let mut step = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + not_listing_port(), + None, + &short_policy(Duration::from_millis(1_200), stall), + &mut || { + step += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: false, + process_alive: true, + // A new percent on every sample re-arms the stall clock. + // Wall-clock buckets can sit still long enough to look hung. + log_line: Some(format!("Downloading model.safetensors: {step}%")), + }, + ) + }, + &mut |_| {}, + ); + let stop = result.expect_err("the deadline still ends the wait"); + assert_eq!( + stop.reason, + ServeStartupStopReason::Deadline, + "a moving download must hit the deadline, not the stall limit" + ); + assert!(step > 3); + } + + #[test] + fn wait_observes_and_updates_the_status_line_once_per_poll() { + // Row: unreachable, refused port. One observe and one status-line + // update per poll, so the spinner does not tick twice per tick. + let poll = Duration::from_millis(20); + let deadline = Duration::from_millis(400); + let mut policy = short_policy(deadline, Duration::from_secs(30)); + policy.poll = poll; + let mut observes = 0_u32; + let mut views = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + not_listing_port(), + None, + &policy, + &mut || { + observes += 1; + ("qwen".to_owned(), alive()) + }, + &mut |_| views += 1, + ); + let stop = result.expect_err("nothing answers, so the deadline ends the wait"); + assert_eq!(stop.reason, ServeStartupStopReason::Deadline); + // The last observe ends in the stop, which draws no frame. + assert_eq!(views + 1, observes, "one frame per observed poll"); + let polls = u32::try_from(deadline.as_millis() / poll.as_millis()).unwrap(); + assert!( + observes <= polls + 2, + "{observes} observes in {polls} polls: the engine is sampled more than once per poll" + ); + } + + #[test] + fn redact_log_line_masks_keys_tokens_and_secret_flags() { + let key = "sk-endpoint-0123456789"; + for (raw, expected) in [ + ( + "launch --secret-key abc123 --private-key def456 --port 8000", + "launch --secret-key [redacted] --private-key [redacted] --port 8000", + ), + // A value ending in a multi-byte character is cut at a character + // boundary, not one byte past the start of that character. + ( + "launch --api-key clé --port 8000", + "launch --api-key [redacted] --port 8000", + ), + ( + "launch --api-key 'clé', --port 8000", + "launch --api-key '[redacted]', --port 8000", + ), + ( + "non-default args: {'model': 'qwen', 'api_key': 'sk-endpoint-0123456789', 'port': 8000}", + "non-default args: {'model': 'qwen', 'api_key': '[redacted]', 'port': 8000}", + ), + ( + "launch --api-key abc123 --port 8000", + "launch --api-key [redacted] --port 8000", + ), + ( + "env HF_TOKEN=hf_abcdefghijklmnopqrstuvwx set", + "env HF_TOKEN=[redacted] set", + ), + ( + "using hf_abcdefghijklmnopqrstuvwx for the hub", + "using [redacted] for the hub", + ), + ( + "header Authorization: Bearer abc.def.ghi sent", + "header Authorization: Bearer [redacted] sent", + ), + ( + r#"config {"api_key":"abc123","port":8000}"#, + r#"config {"api_key":"[redacted]","port":8000}"#, + ), + ( + "header Authorization: Basic dXNlcjpwYXNz sent", + "header Authorization: Basic [redacted] sent", + ), + ( + "serving with key sk-endpoint-0123456789 on 0.0.0.0", + "serving with key [redacted] on 0.0.0.0", + ), + ] { + assert_eq!(redact_log_line(raw, Some(key)), expected, "{raw}"); + } + // Ordinary progress lines, including prose about tokens, are untouched. + for line in [ + "Loading safetensors checkpoint shards: 40%", + "Downloading tokenizer.json: 100%", + "the token budget is 8192 tokens", + "max_num_batched_tokens=8192", + ] { + assert_eq!(redact_log_line(line, Some(key)), line); + } + } + + #[test] + fn a_secret_in_the_engine_log_never_reaches_the_status_line_or_the_failure() { + let key = "sk-endpoint-0123456789"; + let mut labels = Vec::new(); + let mut samples = 0_u32; + let result = wait_for_serve_startup( + "vllm", + "qwen", + "127.0.0.1", + not_listing_port(), + Some(key), + &short_policy(Duration::from_secs(5), Duration::from_secs(30)), + &mut || { + // A live sample first, so the status line draws the line, + // then the failure that ends the wait. + samples += 1; + ( + "qwen".to_owned(), + ServeStartupSignals { + engine_failed: samples > 1, + process_alive: true, + log_line: Some(format!( + "error: bad args --api-key {key} --hf-token hf_abcdefghijklmnopqrstuvwx" + )), + }, + ) + }, + &mut |view| labels.push(view.label.clone()), + ); + let stop = result.expect_err("a failed engine ends the wait"); + let message = stop.user_message("qwen", "vllm", "svc-1"); + for text in [&stop.label, &message] { + assert!( + !text.contains(key), + "endpoint key leaked: {}", + text.replace(key, "") + ); + assert!( + !text.contains("hf_abcdefghijklmnopqrstuvwx"), + "hub token leaked" + ); + } + assert!( + message.contains("--api-key [redacted]"), + "the line is still quoted, masked" + ); + assert!(!labels.is_empty(), "the live sample drew a status line"); + for label in &labels { + assert!(!label.contains(key), "endpoint key on the status line"); + assert!(label.contains("--api-key [redacted]"), "{label}"); + } + } + + #[test] + fn a_key_cut_at_the_length_limit_is_masked_before_truncation() { + let key = "sk-endpoint-0123456789abcdef"; + // Put the key across the MAX_LABEL_CHARS boundary. + let pad = "x".repeat(MAX_LABEL_CHARS - 25); + let raw = format!("args {pad} {key} trailing words"); + let line = sanitize_log_line(&raw, Some(key)); + assert!( + !line.contains("sk-endpoint"), + "key prefix survived truncation: {line}" + ); + assert!(line.chars().count() <= MAX_LABEL_CHARS); + } +} diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index c30bcc654..82482dcdc 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -5,7 +5,8 @@ //! Engine-agnostic deployment summary for `rocm serve`. //! //! By default `rocm serve ` no longer streams raw engine logs. Instead it -//! shows an animated status indicator while the server starts, runs a small +//! shows a status line that follows the engine (a download, a load, or the +//! latest log line) while the server starts, runs a small //! inference smoke test, and then prints a compact summary table: deployment //! status, the full inference endpoint (with port), the API-qualified model name, //! and the measured time-to-first-token / throughput. @@ -168,10 +169,11 @@ pub(crate) fn format_tps(tps: Option) -> String { /// Render the deployment summary table as plain text (returned rather than /// printed so it is unit-testable and identical across engines). pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { - // The server answered its health check within the startup window. A launch - // that timed out lands here with `status == "starting"`; the heading and a - // note must make that visibly different from a healthy deployment so the - // summary is never mistaken for success. + // A ready server answered an inference request. Any other status on a fresh + // launch gets a distinct heading so it cannot be read as success. + // Interactive `rocm serve` does not render this heading for a startup that + // is still loading: that wait ends in `ServeStartupStop::user_message` + // (`listing_at_the_deadline_stops_without_a_not_ready_summary`). let ready = summary.status == "ready"; let smoke_failed = summary.smoke_failure.is_some() && !summary.already_running; let heading = if summary.already_running { @@ -229,7 +231,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ), ( "logs", - format!("rocm logs --service {}", summary.service_id), + crate::service_hints::service_logs_hint(&summary.service_id), ), ], ] @@ -266,8 +268,8 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { }; let _ = writeln!( out, - " note: {explanation} - follow `rocm logs --service {} --follow` or check `rocm services list`", - summary.service_id + " note: {explanation} - follow `{} --follow` or check `rocm services list`", + crate::service_hints::service_logs_hint(&summary.service_id) ); } if let (true, Some(reason)) = (smoke_failed, summary.smoke_failure.as_deref()) { @@ -276,8 +278,8 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { // empty, hedged: the reason may be rocm's own plumbing, not the engine. let _ = writeln!( out, - " note: the smoke test could not complete, so no metrics were measured ({reason}); the server may not be able to serve requests - check `rocm logs --service {id}`", - id = summary.service_id + " note: the smoke test could not complete, so no metrics were measured ({reason}); the server may not be able to serve requests - check `{logs}`", + logs = crate::service_hints::service_logs_hint(&summary.service_id) ); } for note in &summary.notes { @@ -1288,6 +1290,14 @@ mod tests { // The hedge: the note must not claim the server is broken, only that it // may be unable to serve. assert!(rendered.contains("; the server may not be able to serve requests")); + // The id goes through the shared hint, so one that needs quoting is + // quoted here as in the summary's other notes. + let mut hostile = failed_summary("connection closed mid-response"); + hostile.service_id = "svc $(id)".to_owned(); + let rendered = render_summary(&hostile); + // No `{rendered}` in the failure message: the rendered summary can + // carry an endpoint key, so it is not printed, like the asserts above. + assert!(rendered.contains("check `rocm logs --service 'svc $(id)'`")); } #[test] diff --git a/apps/rocm/src/service_hints.rs b/apps/rocm/src/service_hints.rs index 6556c2e70..bc81279a4 100644 --- a/apps/rocm/src/service_hints.rs +++ b/apps/rocm/src/service_hints.rs @@ -53,6 +53,15 @@ pub(crate) fn service_stop_hint(service_id: &str) -> String { service_action_hint("stop", service_id) } +/// The directly-runnable `rocm logs --service ` hint. +/// +/// `rocm logs --service` prints `services_text::render_service_logs_text` for +/// that service. Shared so a startup failure and the deployment summary cannot +/// name two different log commands. +pub(crate) fn service_logs_hint(service_id: &str) -> String { + format!("rocm logs --service {}", shell_quote(service_id)) +} + /// The directly-runnable ` services stop --yes` hint for a /// service on a remote machine, where `remote_cli` is that machine's own /// `rocm` binary name (not necessarily `"rocm"` — it is whatever the remote @@ -119,6 +128,10 @@ mod tests { remote_service_stop_hint("rocm", "vllm-qwen-8000"), "rocm services stop vllm-qwen-8000 --yes" ); + assert_eq!( + service_logs_hint("vllm-qwen-8000"), + "rocm logs --service vllm-qwen-8000" + ); } proptest! { @@ -144,6 +157,19 @@ mod tests { hint ); } + // Mutation: printing `service_id` bare in the logs hint fails this. + let logs = service_logs_hint(&service_id); + prop_assert_eq!( + shlex::split(&logs), + Some(vec![ + "rocm".to_owned(), + "logs".to_owned(), + "--service".to_owned(), + service_id, + ]), + "{}", + logs + ); } } } diff --git a/apps/rocm/src/service_launch.rs b/apps/rocm/src/service_launch.rs index 7d0e49106..e08e6f77b 100644 --- a/apps/rocm/src/service_launch.rs +++ b/apps/rocm/src/service_launch.rs @@ -68,7 +68,7 @@ pub(crate) use helper_autostart::{ ensure_background_helper_running, ensure_background_helper_running_quiet, }; pub(crate) use launch_transaction::{managed_service_reuse_refusal, no_usable_gpu_refusal}; -pub(crate) use readiness::e2e_simulate_oom_launch; +pub(crate) use readiness::{e2e_simulate_oom_launch, e2e_simulate_startup_exit}; #[cfg(not(windows))] pub(crate) use spawn::{ attach_background_stdio, detach_background_command, managed_engine_startup_failure_detail, diff --git a/apps/rocm/src/service_launch/attach.rs b/apps/rocm/src/service_launch/attach.rs index d76a90e8d..f12da3a5b 100644 --- a/apps/rocm/src/service_launch/attach.rs +++ b/apps/rocm/src/service_launch/attach.rs @@ -105,7 +105,7 @@ pub(crate) fn run_attached_service( println!(" service_id: {}", report.service_id); println!(" endpoint: {}", report.endpoint_url); println!(" status: {}", report.status); - println!(" logs: rocm logs {}", report.service_id); + println!("{}", logs_row("logs", &report.service_id)); println!( " stop: {}", service_hints::service_stop_hint(&report.service_id) @@ -169,7 +169,7 @@ pub(crate) fn run_attached_service( println!(" service_id: {service_id}"); println!(" endpoint: {endpoint}"); println!(" list: rocm services"); - println!(" logs: rocm logs {service_id}"); + println!("{}", logs_row("logs", &service_id)); println!(" stop: {}", service_hints::service_stop_hint(&service_id)); Ok(()) } @@ -189,12 +189,22 @@ pub(crate) fn run_attached_service( AttachOutcome::ServerExited => { println!("server process exited"); println!(" service_id: {service_id}"); - println!(" recent logs: rocm logs {service_id}"); + println!("{}", logs_row("recent logs", &service_id)); Ok(()) } } } +/// The ` {label}: rocm logs --service ` row of the attached path's +/// reports. A bare `rocm logs ` would be a free-text search, not that +/// service's log. +fn logs_row(label: &str, service_id: &str) -> String { + format!( + " {label}: {}", + service_hints::service_logs_hint(service_id) + ) +} + /// Restores cooked terminal mode when dropped, so [`stream_attached_logs`] leaves /// the terminal usable on every exit path (normal return, `?` error, or panic). struct RawModeGuard; @@ -433,6 +443,26 @@ fn finish_attached_stop( #[cfg(test)] mod tests { use super::*; + + /// Every `logs` row the attached path prints names a command that selects + /// that service's log. `rocm logs ` parses as a free-text search. + #[test] + fn attached_logs_rows_select_the_service_log() { + use clap::Parser as _; + for label in ["logs", "recent logs"] { + let row = logs_row(label, "svc-1"); + let command = row + .strip_prefix(&format!(" {label}: ")) + .expect("row starts with its label"); + let parsed = crate::cli::Cli::try_parse_from(command.split_whitespace()) + .expect("the printed command parses"); + let Some(crate::cli::Command::Logs { service, query, .. }) = parsed.command else { + panic!("`{command}` is not `rocm logs`"); + }; + assert_eq!(service.as_deref(), Some("svc-1"), "`{command}`"); + assert!(query.is_empty(), "`{command}` searches for {query:?}"); + } + } use crate::endpoint_keys; use crate::tests::test_paths; use rocm_core::ManagedServiceRecord; diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs index c10a29eea..73f7ec2c8 100644 --- a/apps/rocm/src/service_launch/readiness.rs +++ b/apps/rocm/src/service_launch/readiness.rs @@ -19,6 +19,7 @@ use super::spawn::ManagedLaunchReport; use crate::device_policy_name; use crate::endpoint_keys; use crate::engine_runtime_select::wait_for_service_http_ready_with_progress; +use crate::serve_progress; use crate::status_for_readiness; #[cfg(feature = "e2e-oom-fault-injection")] use anyhow::Context; @@ -31,8 +32,9 @@ use rocm_engine_protocol::{DevicePolicy, ResolveModelResponse}; use std::fmt::Write as _; use std::time::Duration; -/// How long one readiness wait polls before `rocm serve` reports `starting`. -const READINESS_WAIT: Duration = Duration::from_secs(45); +/// How long one `--wait` readiness pass polls: the same budget after which a +/// piped `rocm serve` reports `starting`. +const READINESS_WAIT: Duration = crate::serve_progress::PIPED_SERVE_STATUS_BUDGET; /// `rocm serve --wait`: keep waiting past [`READINESS_WAIT`] (a first start /// downloads the model, which can take far longer) until the server answers or @@ -98,13 +100,25 @@ fn service_still_live(paths: &AppPaths, service_id: &str, pid_alive: impl Fn(u32 /// Record a server that died mid-start as stopped, leaving a manifest that a /// `services stop` already settled untouched. A `failed` state the engine /// reported is persisted rather than overwritten, so the reason survives. -fn settle_exited_service(paths: &AppPaths, service_id: &str) { +/// +/// The manifest is re-read here, after the caller decided the server ended. +/// If it now describes a live process again, `rocmd` restarted the service in +/// between (`restart_managed_service` in apps/rocmd/src/supervise.rs records +/// the new pid through `record_spawned_supervisor`), and that restart is left alone rather than written back to +/// `stopped`. +fn settle_exited_service(paths: &AppPaths, service_id: &str, pid_alive: &dyn Fn(u32) -> bool) { let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { return; }; let on_disk = latest.status.clone(); let _ = latest.refresh_from_engine_state(); if crate::managed_service_status::managed_service_is_live(&latest) { + if crate::managed_service_status::recorded_service_pids(&latest) + .into_iter() + .any(pid_alive) + { + return; + } latest.status = "stopped".to_owned(); } else if latest.status == on_disk { return; @@ -124,18 +138,49 @@ pub(super) fn died_during_wait( ) -> Option { if !wait_until_ready || readiness == EndpointReadiness::Serving - || service_still_live(paths, service_id, pid_alive) + || service_still_live(paths, service_id, &pid_alive) { return None; } - settle_exited_service(paths, service_id); - let mut message = format!( - "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `rocm logs --service {service_id}` for its output" + let message = format!( + "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `{}` for its output", + crate::service_hints::service_logs_hint(service_id) + ); + Some(anyhow!(settle_ended_launch( + paths, + service_id, + "exited during --wait", + message, + &pid_alive + ))) +} + +/// Settle a launch whose server ended before it became ready, record it in +/// the audit log (`why` says how it ended), and return `message` with the +/// memory guidance a plain `serve` would print for it. +/// +/// Shared by a `--wait` that saw the server exit and the interactive wait's +/// engine-failed or process-exited stop: the record is settled by +/// [`settle_exited_service`] (`stopped`, or the engine's own `failed`, never +/// over a stop already recorded), and a vLLM launch that ran out of VRAM gets +/// the same note either way. "starting" is the status the note keys on for +/// "never became ready", whatever the settled record now says. +pub(super) fn settle_ended_launch( + paths: &AppPaths, + service_id: &str, + why: &str, + mut message: String, + pid_alive: &dyn Fn(u32) -> bool, +) -> String { + settle_exited_service(paths, service_id, pid_alive); + crate::record_cli_audit_event( + paths, + "service", + "managed_service_launch", + "error", + format!("managed service ended before it became ready: {why}"), + Some(service_id), ); - // A plain `serve` adds the memory guidance for a vLLM launch that ran out of - // VRAM; the wait's failure is the same launch, so it gets the same note. - // "starting" is the status the note keys on for "never became ready", which is - // what a `--wait` death is, whatever the settled record now says. if let Ok(record) = ManagedServiceRecord::read_manifest(paths, service_id) { for note in crate::serve_summary::append_oom_serve_note( Vec::new(), @@ -147,7 +192,7 @@ pub(super) fn died_during_wait( let _ = write!(message, "\n note: {note}"); } } - Some(anyhow!(message)) + message } /// Record the outcome of a finished wait on the manifest as it is now, changing @@ -206,6 +251,35 @@ pub(crate) const fn e2e_simulate_oom_launch() -> bool { false } +/// Test-only switch for the mock lane, compiled in only with +/// `e2e-oom-fault-injection` and read only on Linux: a managed launch spawns a +/// stand-in engine that writes one log line and exits during startup, in place +/// of the real `rocm` serve child, so the interactive startup wait's +/// process-exited path runs end to end on a host with no GPU. Everything else +/// about the launch (record, log, wait, settling) is the production path. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +pub(crate) fn e2e_simulate_startup_exit() -> bool { + std::env::var_os(SIMULATE_STARTUP_EXIT_ENV).is_some_and(|value| value == "1") +} + +/// Release builds, and hosts other than Linux, carry no such switch. +#[cfg(not(all(feature = "e2e-oom-fault-injection", target_os = "linux")))] +pub(crate) const fn e2e_simulate_startup_exit() -> bool { + false +} + +/// The variable [`e2e_simulate_startup_exit`] reads; the e2e step spells the +/// same name, since the shipped crate must not depend on the harness. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +const SIMULATE_STARTUP_EXIT_ENV: &str = "ROCM_E2E_SIMULATE_STARTUP_EXIT"; + +/// What the stand-in engine runs: one meaningful log line, then an exit after +/// the spawn's immediate-exit check, so the launch is published and the +/// startup wait (not the spawn) is what sees the process end. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +pub(super) const SIMULATED_STARTUP_EXIT_SCRIPT: &str = "echo 'e2e stand-in engine: loading model weights'; sleep 2; \ + echo 'e2e stand-in engine: exiting during startup'; exit 1"; + /// Test-only (see [`e2e_simulate_oom_launch`]): fabricate the on-disk state of a /// managed vLLM launch, then return the launch report the summary path would see /// for it. `already_running` is false either way (this invocation launched it) @@ -313,12 +387,11 @@ pub(super) fn simulate_oom_managed_launch( }) } -/// Probe `record`'s endpoint until it answers: once within the readiness -/// budget, or (with `wait_until_ready`) for as long as the server stays alive. +/// Probe `record`'s endpoint until it answers (`--wait`), for as long as the +/// server stays alive. Each pass is one [`READINESS_WAIT`]-long readiness wait. pub(super) fn wait_for_record_ready( paths: &AppPaths, record: &ManagedServiceRecord, - wait_until_ready: bool, endpoint_api_key: Option<&str>, on_wait_tick: &mut dyn FnMut(Duration), ) -> EndpointReadiness { @@ -337,14 +410,57 @@ pub(super) fn wait_for_record_ready( on_tick, ) }; - if wait_until_ready { - wait_past_readiness_budget( - || probe(&mut *on_wait_tick), - || service_still_live(paths, &record.service_id, service_pid_is_alive), - ) - } else { - probe(on_wait_tick) - } + wait_past_readiness_budget( + || probe(&mut *on_wait_tick), + || service_still_live(paths, &record.service_id, service_pid_is_alive), + ) +} + +/// One sample of a freshly launched service for the startup wait: the model +/// name its endpoint answers to and the engine/process signals. +/// +/// Read from the manifest on disk, like [`service_still_live`], so a +/// `rocm services stop` from another terminal is seen. Liveness is +/// [`service_pid_is_alive`]: the supervisor is an unreaped child of this +/// process, so a dead one still passes `process_is_running`. +fn observe_launch( + paths: &AppPaths, + held: &ManagedServiceRecord, + endpoint_api_key: Option<&str>, + pid_alive: impl Fn(u32) -> bool, +) -> (String, serve_progress::ServeStartupSignals) { + let mut latest = ManagedServiceRecord::read_manifest(paths, &held.service_id) + .unwrap_or_else(|_| held.clone()); + let _ = latest.refresh_from_engine_state(); + let pids = crate::managed_service_status::recorded_service_pids(&latest); + let signals = serve_progress::ServeStartupSignals { + engine_failed: latest.status == "failed", + process_alive: pids.is_empty() || pids.into_iter().any(pid_alive), + log_line: serve_progress::latest_meaningful_log_line(&latest.log_path, endpoint_api_key), + }; + (latest.served_model_name().to_owned(), signals) +} + +/// Follow a fresh launch (without `--wait`) until it serves, the policy's +/// budget runs out, or the wait stops. See [`serve_progress`]. +pub(super) fn wait_for_launch_startup( + paths: &AppPaths, + record: &ManagedServiceRecord, + requested_model: &str, + endpoint_api_key: Option<&str>, + policy: &serve_progress::ServeStartupPolicy, + on_view: &mut dyn FnMut(&serve_progress::ServeStartupView), +) -> Result { + serve_progress::wait_for_serve_startup( + &record.engine, + requested_model, + &record.host, + record.port, + endpoint_api_key, + policy, + &mut || observe_launch(paths, record, endpoint_api_key, service_pid_is_alive), + on_view, + ) } /// An equivalent service is already live. Without `--wait`, or once it is @@ -368,8 +484,7 @@ pub(super) fn wait_on_existing_service( // The key the caller generated belongs to a launch that did not happen; the // existing service answers to its own. let stored_key = endpoint_keys::endpoint_api_key(paths, &report.service_id); - let readiness = - wait_for_record_ready(paths, &record, true, stored_key.as_deref(), on_wait_tick); + let readiness = wait_for_record_ready(paths, &record, stored_key.as_deref(), on_wait_tick); if let Some(error) = died_during_wait( paths, &report.service_id, @@ -396,6 +511,42 @@ mod tests { use crate::tests::test_paths; use std::fs; + /// The startup wait's sample comes from the manifest on disk: the + /// engine's own `failed`, liveness by pid (supervisor or engine), and the + /// name the endpoint answers to. + #[test] + fn observe_launch_reads_failure_and_liveness_from_the_manifest() -> anyhow::Result<()> { + let (root, paths) = test_paths("observe-launch"); + paths.ensure()?; + let mut held = reusable_record(&paths); + held.supervisor_pid = 41; + held.engine_pid = Some(42); + held.write()?; + let sample = |pid_alive: fn(u32) -> bool| observe_launch(&paths, &held, None, pid_alive); + + let (model, signals) = sample(|pid| pid == 42); + assert_eq!(model, held.served_model_name()); + assert!(!signals.engine_failed); + assert!(signals.process_alive, "a live engine pid is alive"); + assert!( + sample(|pid| pid == 41).1.process_alive, + "a live supervisor is alive" + ); + assert!( + !sample(|_| false).1.process_alive, + "every recorded pid dead" + ); + + // Another writer's `failed` on disk is what the wait sees, not the + // copy it holds. + let mut on_disk = ManagedServiceRecord::read_manifest(&paths, &held.service_id)?; + on_disk.status = "failed".to_owned(); + on_disk.write()?; + assert!(sample(|_| true).1.engine_failed); + let _ = fs::remove_dir_all(root); + Ok(()) + } + #[test] fn wait_past_readiness_budget_keeps_probing_a_live_starting_server() { let mut readings = vec![ @@ -513,9 +664,14 @@ mod tests { fn settle_exited_service_demotes_a_live_manifest_and_spares_a_settled_one() { let (root, paths) = test_paths("wait-settle"); paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, 0, "starting"); + let mut record = record_tracking_pid(&paths, 4_242, "starting"); record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); + // A manifest whose recorded process is alive again was restarted by + // `rocmd` after the caller decided the server ended: left alone. + settle_exited_service(&paths, &record.service_id, &|_| true); + let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); + assert_eq!(after.status, "starting"); + settle_exited_service(&paths, &record.service_id, &|_| false); let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); assert_eq!(after.status, "stopped"); @@ -524,7 +680,7 @@ mod tests { // be invisible). record.status = "failed".to_owned(); record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); + settle_exited_service(&paths, &record.service_id, &|_| false); let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); assert_eq!(after.status, "failed"); let _ = fs::remove_dir_all(root); @@ -591,6 +747,18 @@ mod tests { .status }; + let launch_errors = || { + rocm_core::load_recent_audit_events(&paths, 20) + .unwrap_or_default() + .into_iter() + .filter(|event| { + event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(id.as_str()) + }) + .collect::>() + }; + // Not waiting, already serving, or still alive: no failure, no change. let pending = EndpointReadiness::Unreachable; assert!(died_during_wait(&paths, &id, false, pending, |_| false).is_none()); @@ -599,6 +767,7 @@ mod tests { ); assert!(died_during_wait(&paths, &id, true, pending, |_| true).is_none()); assert_eq!(status(), "starting"); + assert!(launch_errors().is_empty(), "{:?}", launch_errors()); // Dead server: the error names the logs command and the record is settled. let error = died_during_wait(&paths, &id, true, pending, |_| false).expect("died"); @@ -612,6 +781,14 @@ mod tests { "{message}" ); assert_eq!(status(), "stopped"); + // The `--wait` death is audited like the interactive wait's. + let errors = launch_errors(); + assert!( + errors + .iter() + .any(|event| event.message.contains("exited during --wait")), + "{errors:?}" + ); let _ = fs::remove_dir_all(root); } diff --git a/apps/rocm/src/service_launch/spawn.rs b/apps/rocm/src/service_launch/spawn.rs index 9b3252efc..ea6f2ac3b 100644 --- a/apps/rocm/src/service_launch/spawn.rs +++ b/apps/rocm/src/service_launch/spawn.rs @@ -392,6 +392,23 @@ fn test_spawn_hook() -> Option>> { TEST_SPAWN_HOOK.with(|hook| hook.borrow().clone()) } +/// The stand-in engine behind [`super::readiness::e2e_simulate_startup_exit`]: +/// a shell that appends to the launch's own log and exits during startup. +#[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] +fn simulated_startup_exit_spawner(log_path: &Path) -> Box> { + let log_path = log_path.to_path_buf(); + Box::new(move |_command| { + let log = fs::OpenOptions::new().append(true).open(&log_path)?; + let stderr = log.try_clone()?; + ProcessCommand::new("sh") + .args(["-c", super::readiness::SIMULATED_STARTUP_EXIT_SCRIPT]) + .stdin(Stdio::null()) + .stdout(log) + .stderr(stderr) + .spawn() + }) +} + #[cfg(not(windows))] type SpawnOverride<'a> = dyn Fn(&mut ProcessCommand) -> io::Result + 'a; @@ -519,6 +536,11 @@ fn launch_managed_engine_child_locked( let hook = test_spawn_hook(); #[cfg(all(test, not(windows)))] let spawn_override = spawn_override.or(hook.as_deref()); + #[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] + let stand_in = super::readiness::e2e_simulate_startup_exit() + .then(|| simulated_startup_exit_spawner(&record.log_path)); + #[cfg(all(feature = "e2e-oom-fault-injection", target_os = "linux"))] + let spawn_override = spawn_override.or(stand_in.as_deref()); let mut child = match spawn_override { Some(spawn) => spawn(&mut command), None => command.spawn(), diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs index 06cb93697..8c70e3846 100644 --- a/apps/rocm/src/service_launch/start.rs +++ b/apps/rocm/src/service_launch/start.rs @@ -7,13 +7,14 @@ use super::endpoint_auth::drop_orphaned_endpoint_key_on_already_running; use super::readiness::{ - died_during_wait, record_wait_outcome, service_pid_is_alive, wait_for_record_ready, - wait_on_existing_service, + died_during_wait, record_wait_outcome, service_pid_is_alive, settle_ended_launch, + wait_for_launch_startup, wait_for_record_ready, wait_on_existing_service, }; #[cfg(feature = "e2e-oom-fault-injection")] use super::readiness::{e2e_simulate_oom_launch, simulate_oom_managed_launch}; use super::spawn::{ManagedLaunchReport, ManagedSpawn, spawn_managed_engine_child}; use crate::serve_port; +use crate::serve_progress::{self, ServeStartupPhase, ServeStartupPolicy, ServeStartupView}; use crate::{record_cli_audit_event, status_for_readiness}; use anyhow::Result; use rocm_core::format_http_base_url; @@ -22,6 +23,7 @@ use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse} use std::fmt::Write as _; #[cfg(windows)] use std::thread; +#[cfg(windows)] use std::time::Duration; /// Launch a managed service in the background: run the locked allocation and @@ -55,9 +57,16 @@ pub(crate) fn start_managed_service( require_api_key: bool, gpu_refusal_waived_for_reuse: bool, wait_until_ready: bool, + policy: &ServeStartupPolicy, on_wait_begin: &mut dyn FnMut(&str, u16), - on_wait_tick: &mut dyn FnMut(Duration), + on_view: &mut dyn FnMut(&ServeStartupView), ) -> Result { + // `--wait` keeps the label it started with: its wait reports no stage. + let steady = ServeStartupView { + phase: ServeStartupPhase::Unreachable, + label: serve_progress::starting_label(requested_model, engine), + log_line: None, + }; #[cfg(feature = "e2e-oom-fault-injection")] if e2e_simulate_oom_launch() { return simulate_oom_managed_launch( @@ -104,7 +113,9 @@ pub(crate) fn start_managed_service( report, wait_until_ready, on_wait_begin, - on_wait_tick, + &mut |_elapsed| { + on_view(&steady); + }, ); } ManagedSpawn::Spawned { record, child_pid } => (*record, child_pid), @@ -121,16 +132,27 @@ pub(crate) fn start_managed_service( #[cfg(windows)] thread::sleep(Duration::from_millis(200)); - if wait_until_ready { + if announces_wait(wait_until_ready, policy) { on_wait_begin(service_id, port); } - let readiness = wait_for_record_ready( - paths, - &record, - wait_until_ready, - endpoint_api_key, - on_wait_tick, - ); + let (readiness, startup_stop) = if wait_until_ready { + let readiness = wait_for_record_ready(paths, &record, endpoint_api_key, &mut |_elapsed| { + on_view(&steady); + }); + (readiness, None) + } else { + match wait_for_launch_startup( + paths, + &record, + requested_model, + endpoint_api_key, + policy, + on_view, + ) { + Ok(readiness) => (readiness, None), + Err(stop) => (stop.readiness, Some(stop)), + } + }; if let Some(error) = died_during_wait( paths, service_id, @@ -140,13 +162,38 @@ pub(crate) fn start_managed_service( ) { return Err(error); } + if let Some(stop) = startup_stop + .as_ref() + .filter(|stop| !stop.leaves_server_running()) + { + // The engine failed or the process ended: settled the way a `--wait` + // that saw the server exit is, with the same memory guidance. + let message = stop.user_message(requested_model, engine, service_id); + let why = format!("{:?}", stop.reason); + return Err(serve_progress::ServeStartupFailed { + message: settle_ended_launch(paths, service_id, &why, message, &service_pid_is_alive), + server_running: false, + } + .into()); + } + // A stall or the deadline left the server up. Its status follows how far + // the endpoint got, as for any other launch: `running` only when the model + // is listed, `starting` when it never opened, so `rocm services`, dash and + // chat do not show a server that has not listed its model as up. + // `rocmd` leaves a live `starting` record alone until the startup deadline + // (`live_startup_still_in_progress` in apps/rocmd/src/watchers/server_recover.rs). let launch_status = status_for_readiness(readiness); - record_wait_outcome( + let recorded = record_wait_outcome( paths, &mut record, launch_status, readiness == EndpointReadiness::Serving, - )?; + ); + // After a stop, the stop is what the user needs to read; a record that + // could not be rewritten is best-effort there, as `settle_ended_launch` is. + if startup_stop.is_none() { + recorded?; + } let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); record_cli_audit_event( paths, @@ -159,6 +206,13 @@ pub(crate) fn start_managed_service( ), Some(service_id), ); + if let Some(stop) = startup_stop { + return Err(serve_progress::ServeStartupFailed { + message: stop.user_message(requested_model, engine, service_id), + server_running: true, + } + .into()); + } Ok(ManagedLaunchReport { service_id: service_id.to_owned(), endpoint_url, @@ -170,6 +224,17 @@ pub(crate) fn start_managed_service( }) } +/// Whether a fresh launch announces its wait (the server id, how to follow its +/// log, and a public bind's key) before waiting. +/// +/// `--wait` does, and so does the interactive wait, which can run for +/// [`rocm_core::SERVE_STARTUP_DEADLINE`] and can end in an error that prints no +/// summary: without the announcement a public bind's key would never be shown. +/// The piped status budget returns a report either way and prints the key there. +const fn announces_wait(wait_until_ready: bool, policy: &ServeStartupPolicy) -> bool { + wait_until_ready || policy.stop_on_give_up +} + /// Reproduce the original plain, line-by-line managed-launch output. Used for /// non-interactive output (piped, CI, the chat assistant's `serve --managed`), /// where the animated summary is inappropriate. The interactive path renders the @@ -263,6 +328,21 @@ mod tests { use std::fs; #[cfg(not(windows))] use std::io; + #[cfg(not(windows))] + use std::time::Duration; + + /// A wait that can end in an error without a summary announces itself, so + /// a public bind's key is printed before it. The piped status budget + /// always returns a report, and the summary prints the key there. + #[test] + fn a_wait_that_can_fail_without_a_summary_is_announced() { + let interactive = ServeStartupPolicy::interactive(); + let piped = ServeStartupPolicy::status_budget(serve_progress::PIPED_SERVE_STATUS_BUDGET); + assert!(announces_wait(false, &interactive)); + assert!(announces_wait(true, &interactive)); + assert!(announces_wait(true, &piped)); + assert!(!announces_wait(false, &piped)); + } #[test] fn wait_begin_shows_the_key_only_for_the_launch_it_belongs_to() { @@ -369,8 +449,9 @@ mod tests { false, false, false, + &ServeStartupPolicy::status_budget(Duration::ZERO), &mut |_, _| {}, - &mut |_elapsed| {}, + &mut |_view| {}, ); let Err(error) = result else { let _ = fs::remove_dir_all(root); @@ -392,6 +473,231 @@ mod tests { Ok(()) } + /// What a fresh launch through the spawn hook left behind. + #[cfg(not(windows))] + struct HookedLaunch { + result: Result, + record: Result, + announced: Vec, + child_alive_after: bool, + audit: Vec, + } + + /// Launch `service_id` with `argv` as the engine child, under a short + /// interactive policy: nothing listens on the port, so the wait ends on + /// the silent limit or on the child's exit. + #[cfg(not(windows))] + fn launch_hooked_child( + service_id: &str, + argv: &'static [&'static str], + silent_unreachable: Duration, + ) -> Result { + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + let (root, paths) = test_paths(service_id); + paths.ensure()?; + let child = std::rc::Rc::new(std::cell::RefCell::new(None::)); + let spawned = child.clone(); + // The engine-state file the launch's record reads, for a child that + // plays an engine reporting its own status. + let engine_state = paths.service_engine_state_path("lemonade", service_id); + TEST_SPAWN_HOOK.with(|hook| { + *hook.borrow_mut() = Some(std::rc::Rc::new(move |_command| { + let process = std::process::Command::new(argv[0]) + .args(&argv[1..]) + .env("TEST_ENGINE_STATE", &engine_state) + .spawn()?; + *spawned.borrow_mut() = Some(process.id()); + Ok(process) + })); + }); + let mut policy = ServeStartupPolicy::interactive(); + policy.poll = Duration::from_millis(20); + policy.bounds.silent_unreachable = silent_unreachable; + policy.bounds.deadline = Duration::from_secs(10); + let mut announced = Vec::new(); + let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + let result = start_managed_service( + &paths, + "lemonade", + service_id, + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Auto, + &resolve.device_policy, + &[], + None, + None, + None, + None, + launch_lock, + false, + false, + false, + &policy, + &mut |id, _port| announced.push(id.to_owned()), + &mut |_view| {}, + ); + TEST_SPAWN_HOOK.with(|hook| *hook.borrow_mut() = None); + let record = ManagedServiceRecord::read_manifest(&paths, service_id); + let audit = rocm_core::load_recent_audit_events(&paths, 20).unwrap_or_default(); + let pid = *child.borrow(); + let child_alive_after = pid.is_some_and(super::service_pid_is_alive); + if let Some(pid) = pid { + let _ = rocm_core::terminate_process(pid); + } + let _ = fs::remove_dir_all(root); + Ok(HookedLaunch { + result, + record, + announced, + child_alive_after, + audit, + }) + } + + #[cfg(not(windows))] + fn startup_failure(result: Result) -> serve_progress::ServeStartupFailed { + let Err(error) = result else { + panic!("the startup wait must stop"); + }; + error + .downcast::() + .expect("the stop is the typed startup failure") + } + + /// A fresh launch whose wait stops on a live server: the wait is announced + /// first, the server is left running and recorded `starting` (its endpoint + /// never listed the model) without an inference verification, and the typed failure names that service's + /// logs and stop commands and asks `serve` to supervise it. + #[cfg(not(windows))] + #[test] + fn a_startup_stop_records_the_live_server_and_returns_the_typed_failure() -> Result<()> { + let service_id = "lemonade-startup-stop"; + let launch = launch_hooked_child(service_id, &["sleep", "30"], Duration::from_millis(300))?; + assert!(launch.child_alive_after, "the wait must not end the server"); + let failure = startup_failure(launch.result); + assert!(failure.server_running, "{}", failure.message); + for hint in [ + crate::service_hints::service_logs_hint(service_id), + crate::service_hints::service_stop_hint(service_id), + ] { + assert!(failure.message.contains(&hint), "{}", failure.message); + } + assert_eq!(launch.announced, [service_id], "the wait is announced once"); + let record = launch.record?; + // The endpoint never listed the model, so the record says `starting`, not + // `running`, which elsewhere means "lists the model". + assert_eq!(record.status, "starting", "the live server stays recorded"); + assert_eq!(record.inference_verified_at_unix_ms, None); + Ok(()) + } + + /// A launch whose server exits during the wait is over: settled `stopped` + /// the way a `--wait` that saw it exit is, not `failed`, which `rocmd` + /// would restart behind the error the user just read; and the typed + /// failure does not ask `serve` to supervise it. + #[cfg(not(windows))] + #[test] + fn a_server_that_exits_during_the_wait_is_settled_stopped_and_not_supervised() -> Result<()> { + let service_id = "lemonade-startup-exit"; + // The silent limit is far past the child's life, so the exit is what + // ends the wait. + let launch = launch_hooked_child(service_id, &["sleep", "0.3"], Duration::from_secs(5))?; + let failure = startup_failure(launch.result); + assert!(!failure.server_running, "{}", failure.message); + assert!( + failure.message.contains("the server process exited"), + "{}", + failure.message + ); + assert!( + !failure + .message + .contains(&crate::service_hints::service_stop_hint(service_id)), + "{}", + failure.message + ); + assert_eq!(launch.record?.status, "stopped"); + assert!( + launch + .audit + .iter() + .any(|event| event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(service_id) + && event.message.contains("ProcessExited")), + "the ended launch is audited: {:?}", + launch.audit + ); + Ok(()) + } + + /// An engine that reports `failed` while its process is still up ends the + /// launch through the same settling path as an exit: the typed failure + /// does not ask `serve` to supervise it, names the stop command, the + /// record keeps the engine's own `failed` (which `rocmd` may restart: + /// `a_failed_record_with_a_live_process_is_still_recovered` in + /// apps/rocmd/src/watchers/server_recover.rs), and the end is audited. + #[cfg(not(windows))] + #[test] + fn an_engine_that_fails_while_alive_is_settled_failed_and_not_supervised() -> Result<()> { + let service_id = "lemonade-startup-engine-failed"; + let launch = launch_hooked_child( + service_id, + &[ + "sh", + "-c", + "mkdir -p \"$(dirname \"$TEST_ENGINE_STATE\")\" && \ + printf '{\"status\":\"failed\"}' > \"$TEST_ENGINE_STATE\" && sleep 30", + ], + Duration::from_secs(5), + )?; + assert!(launch.child_alive_after, "the process stays up"); + let failure = startup_failure(launch.result); + assert!(!failure.server_running, "{}", failure.message); + assert!( + failure.message.contains("the engine reported failed"), + "{}", + failure.message + ); + assert!( + failure + .message + .contains(&crate::service_hints::service_stop_hint(service_id)), + "{}", + failure.message + ); + assert_eq!(launch.record?.status, "failed"); + assert!( + launch + .audit + .iter() + .any(|event| event.action == "managed_service_launch" + && event.level == "error" + && event.service_id.as_deref() == Some(service_id) + && event.message.contains("EngineFailed")), + "the ended launch is audited: {:?}", + launch.audit + ); + Ok(()) + } + /// The launch lock must still be held when the child is spawned: it is what /// serializes allocation against publication. Observed through the spawn /// hook, since releasing it early changes nothing else a test can see. @@ -466,8 +772,9 @@ mod tests { false, false, false, + &ServeStartupPolicy::status_budget(Duration::ZERO), &mut |_, _| {}, - &mut |_elapsed| {}, + &mut |_view| {}, ) .map(|_| ()) }; diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index aac0a9e9d..05371a427 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -12,8 +12,8 @@ use crate::persistence::load_service_record; use crate::supervise::restart_managed_service; use anyhow::{Context, Result}; use rocm_core::{ - AppPaths, AutomationRuntimeState, AutomationTriggerEvent, ManagedServiceRecord, WatcherMode, - format_host_port, unix_time_millis, + AppPaths, AutomationRuntimeState, AutomationTriggerEvent, IdentityState, ManagedServiceRecord, + ProcessIdentity, WatcherMode, format_host_port, unix_time_millis, }; #[cfg(test)] use serde_json::json; @@ -96,10 +96,14 @@ fn service_record_matches_recovery_event( return false; } match event.kind.as_str() { + // The same guarded reasons as the scan, so a live startup the scan + // spared cannot be restarted by an event raised before it was spared. "service.manifest_recoverable" => { - manifest_service_recovery_reason(record, unix_time_millis()).is_some() + stale_manifest_recovery_reason(record, unix_time_millis()).is_some() + } + "service.endpoint_recoverable" => { + unreachable_endpoint_recovery_reason(record, unix_time_millis()).is_some() } - "service.endpoint_recoverable" => endpoint_service_recovery_reason(record).is_some(), "service.healthcheck_recoverable" => crate::engine_ipc::engine_healthcheck_response( paths, &record.engine, @@ -326,16 +330,20 @@ pub(crate) fn find_recoverable_service( { continue; } - if let Some(reason) = manifest_service_recovery_reason(&record, now) { + if let Some(reason) = stale_manifest_recovery_reason(&record, now) { return Ok(Some((record, reason))); } + // A manifest reason the guard above spared (a live startup) needs no + // `continue`: `manifest_service_recovery_reason` only answers for + // `failed`, `exited`, `unreachable` and a stale `starting`/`recovering`, + // none of which the `ready`/`running` check below takes. if matches!(record.status.as_str(), "ready" | "running") { let Ok(healthcheck) = crate::engine_ipc::engine_healthcheck_response( paths, &record.engine, &record.service_id, ) else { - if let Some(reason) = endpoint_service_recovery_reason(&record) { + if let Some(reason) = unreachable_endpoint_recovery_reason(&record, now) { return Ok(Some((record, reason))); } continue; @@ -346,7 +354,7 @@ pub(crate) fn find_recoverable_service( format!("healthcheck_status_{}", healthcheck.status), ))); } - if let Some(reason) = endpoint_service_recovery_reason(&record) { + if let Some(reason) = unreachable_endpoint_recovery_reason(&record, now) { return Ok(Some((record, reason))); } } @@ -354,6 +362,31 @@ pub(crate) fn find_recoverable_service( Ok(None) } +/// Why a `ready`/`running` service whose endpoint does not answer should be +/// recovered, or `None` when it should not. +/// +/// A live `running` process that has not opened its port is still the +/// interactive serve wait's job (`wait_for_launch_startup` in +/// apps/rocm/src/service_launch/readiness.rs): restarting it kills a download +/// the CLI is reporting. `running` is covered, not only `starting`, because +/// the CLI records `running` once the endpoint lists the model +/// (`status_for_readiness` in apps/rocm/src/main.rs) and the engines write +/// their own state `running` before their port opens (`write_running_state` +/// in engines/lemonade/src/state.rs, called before the server is ready). `ready` is not covered: a +/// verified service that stops answering is a regression, not a startup. +/// A dead pid is not spared. One function for every place that +/// falls back to the endpoint, scan and event re-check, so none can drop the +/// guard. +fn unreachable_endpoint_recovery_reason( + record: &ManagedServiceRecord, + now_unix_ms: u128, +) -> Option { + if record.status == "running" && live_startup_still_in_progress(record, now_unix_ms) { + return None; + } + endpoint_service_recovery_reason(record) +} + /// Whether an operator asked for this service to stop and that stop is still /// standing. /// @@ -368,6 +401,75 @@ pub(crate) const fn stop_requested(record: &ManagedServiceRecord) -> bool { record.stop_requested_unix_ms.is_some() } +/// A recorded process is alive and the service is younger than +/// [`rocm_core::SERVE_STARTUP_DEADLINE`]. +/// +/// That is the window interactive `rocm serve` spends following a download +/// or load before the port accepts connections. Endpoint-unreachable +/// recovery during it restarts the process out from under that wait. +fn live_startup_still_in_progress(record: &ManagedServiceRecord, now_unix_ms: u128) -> bool { + live_startup_still_in_progress_with(record, now_unix_ms, rocm_core::identity_state) +} + +/// [`live_startup_still_in_progress`] with the identity check injected, so a +/// test can give every answer a read can give. +/// +/// A pid whose identity cannot be read (`IdentityState::Indeterminate`) is +/// treated as possibly the startup and spared, as `proc_lifecycle` treats it +/// as possibly alive and does not signal it: a failure to look is not an +/// exit. Only `Recycled` and `Gone` are not the startup. +fn live_startup_still_in_progress_with( + record: &ManagedServiceRecord, + now_unix_ms: u128, + identity: impl Fn(&ProcessIdentity) -> IdentityState, +) -> bool { + let started_at = record + .last_restart_unix_ms + .unwrap_or(record.created_at_unix_ms); + let age_ms = now_unix_ms.saturating_sub(started_at); + if age_ms >= rocm_core::SERVE_STARTUP_DEADLINE.as_millis() { + return false; + } + // A recorded pid alone could have been recycled within the window, so the + // recorded start time must match too. A record with none is the + // best-effort `Matches` `identity_state` gives a legacy record. + [ + Some((record.supervisor_pid, record.supervisor_start_ticks)), + record + .engine_pid + .map(|pid| (pid, record.engine_start_ticks)), + ] + .into_iter() + .flatten() + .any(|(pid, start_ticks)| { + pid != 0 + && matches!( + identity(&ProcessIdentity { pid, start_ticks }), + IdentityState::Matches | IdentityState::Indeterminate + ) + }) +} + +/// Why a stale or failed manifest should be recovered, or `None`. +/// +/// `rocm services` persists an engine's own `starting` state into the +/// record (`ManagedServiceRecord::refresh_from_engine_state` in +/// crates/rocm-core/src/service_record.rs), and the interactive serve wait +/// records a launch that never listed the model as `starting` +/// (`status_for_readiness` in apps/rocm/src/main.rs), so a live server still +/// downloading under that wait (or left by a Ctrl-C of it) can read +/// `starting` past the stale-start window. It is not stale until the startup deadline. Shared by the scan and +/// the event re-check, so neither can drop the guard. +fn stale_manifest_recovery_reason( + record: &ManagedServiceRecord, + now_unix_ms: u128, +) -> Option { + if record.status == "starting" && live_startup_still_in_progress(record, now_unix_ms) { + return None; + } + manifest_service_recovery_reason(record, now_unix_ms) +} + fn endpoint_service_recovery_reason(record: &ManagedServiceRecord) -> Option { (!wait_for_port(&record.host, record.port, ENDPOINT_HEALTH_TIMEOUT)) .then(|| "endpoint_status_unreachable".to_owned()) @@ -768,6 +870,52 @@ mod tests { fs::remove_dir_all(root).ok(); } + /// An engine that reported `failed` while its process is still up is a + /// recovery candidate: no liveness guard covers `failed`. This is what the + /// interactive serve's engine-failed message warns about when it says a + /// running daemon with `server-recover` may restart the server first + /// (`only_a_stall_or_the_deadline_leaves_the_server_running` in + /// apps/rocm/src/serve_progress.rs pins that sentence). + #[test] + fn a_failed_record_with_a_live_process_is_still_recovered() -> Result<()> { + let (root, paths) = temp_app_paths("recoverable-failed-live"); + paths.ensure()?; + let mut failed = ManagedServiceRecord::new( + &paths, + "svc-failed-live", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + failed.status = "failed".to_owned(); + failed.created_at_unix_ms = unix_time_millis(); + failed.write()?; + let found = find_recoverable_service(&paths)?; + let event = service_record_matches_recovery_event( + &paths, + &failed, + &recovery_event(&failed, "service.manifest_recoverable"), + ); + fs::remove_dir_all(root).ok(); + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-failed-live".to_owned(), + "manifest_status_failed".to_owned() + )), + "a live `failed` record inside the startup deadline is restarted" + ); + assert!(event, "the event re-check authorizes the same restart"); + Ok(()) + } + #[test] fn find_recoverable_service_prefers_failed_managed_manifest() -> Result<()> { let (root, paths) = temp_app_paths("recoverable-service"); @@ -1122,4 +1270,268 @@ mod tests { assert_eq!(record.service_id, "svc-backoff-b"); Ok(()) } + + fn recovery_event(record: &ManagedServiceRecord, kind: &str) -> AutomationTriggerEvent { + AutomationTriggerEvent { + at_unix_ms: 1, + kind: kind.to_owned(), + source: "managed_service".to_owned(), + watcher_hint: Some("server-recover".to_owned()), + service_id: Some(record.service_id.clone()), + reason: None, + payload: json!({}), + } + } + + /// A live process that has not opened its port yet is the interactive + /// `rocm serve` wait, not a crash. Endpoint recovery must leave it alone + /// until [`rocm_core::SERVE_STARTUP_DEADLINE`]. A dead pid at the same age + /// is still recovered: the guard is the live process, not the clock alone. + #[test] + fn recovery_leaves_a_live_startup_alone_and_still_restarts_a_dead_one() -> Result<()> { + let (root, paths) = temp_app_paths("recover-live-startup"); + paths.ensure()?; + + let mut live = ManagedServiceRecord::new( + &paths, + "svc-live-startup", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + live.status = "running".to_owned(); + let now = unix_time_millis(); + let deadline_ms = rocm_core::SERVE_STARTUP_DEADLINE.as_millis(); + // Two seconds inside the bound, so the check below cannot cross it. + // A shorter window than the shared deadline would recover this. + live.created_at_unix_ms = now.saturating_sub(deadline_ms.saturating_sub(2_000)); + live.write()?; + + let found_inside = find_recoverable_service(&paths)?; + let event_inside = service_record_matches_recovery_event( + &paths, + &live, + &recovery_event(&live, "service.endpoint_recoverable"), + ); + + // The pid is alive but the recorded start time is not its own: a + // recycled pid is not the startup, so it is recovered. + live.supervisor_start_ticks = Some(u64::MAX); + live.write()?; + let found_recycled = find_recoverable_service(&paths)?; + live.supervisor_start_ticks = None; + + live.supervisor_pid = u32::MAX; + live.engine_pid = None; + live.write()?; + let found_dead = find_recoverable_service(&paths)?; + + live.supervisor_pid = std::process::id(); + // Age equal to the deadline is no longer the interactive wait. + // A longer window than the shared deadline would spare this. + live.created_at_unix_ms = now.saturating_sub(deadline_ms); + live.write()?; + let found_at_deadline = find_recoverable_service(&paths)?; + fs::remove_dir_all(root).ok(); + + assert!( + found_inside.is_none(), + "a live process inside the startup deadline must not be restarted: {found_inside:?}" + ); + assert!( + !event_inside, + "an endpoint event raised earlier must not restart it either" + ); + for (found, why) in [ + ( + found_recycled, + "a recycled pid is not a startup still in progress", + ), + (found_dead, "a dead pid is not a startup still in progress"), + ] { + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-startup".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + "{why}" + ); + } + assert_eq!( + found_at_deadline.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-startup".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + "at the startup deadline a live process with a closed port is recoverable" + ); + Ok(()) + } + + /// The guard covers `running` (and `starting`) but not `ready`: a service + /// that was verified serving and then stopped answering is recovered at + /// once, even with a live process inside the startup deadline. + #[test] + fn recovery_does_not_spare_a_ready_service_that_stopped_answering() -> Result<()> { + let (root, paths) = temp_app_paths("recover-ready-live"); + paths.ensure()?; + let mut ready = ManagedServiceRecord::new( + &paths, + "svc-ready-live", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + ready.status = "ready".to_owned(); + ready.created_at_unix_ms = unix_time_millis(); + ready.write()?; + let found = find_recoverable_service(&paths)?; + let event = service_record_matches_recovery_event( + &paths, + &ready, + &recovery_event(&ready, "service.endpoint_recoverable"), + ); + fs::remove_dir_all(root).ok(); + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-ready-live".to_owned(), + "endpoint_status_unreachable".to_owned() + )), + ); + assert!(event, "the event re-check authorizes the same restart"); + Ok(()) + } + + /// Every identity answer: the startup is spared when its pid matches or + /// cannot be read, and not when it was recycled or is gone, or past the + /// deadline whatever the answer. + #[test] + fn the_live_startup_guard_spares_a_matching_or_unreadable_pid_only() { + let (root, paths) = temp_app_paths("recover-identity-answers"); + let mut record = ManagedServiceRecord::new( + &paths, + "svc-identity", + "vllm", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + 11435, + "managed", + 4242, + None, + None, + None, + ); + let now = unix_time_millis(); + record.created_at_unix_ms = now; + for (answer, spared) in [ + (IdentityState::Matches, true), + (IdentityState::Indeterminate, true), + (IdentityState::Recycled, false), + (IdentityState::Gone, false), + ] { + assert_eq!( + live_startup_still_in_progress_with(&record, now, |_| answer), + spared, + "{answer:?}" + ); + assert!( + !live_startup_still_in_progress_with( + &record, + now + rocm_core::SERVE_STARTUP_DEADLINE.as_millis(), + |_| answer + ), + "{answer:?} past the deadline" + ); + } + fs::remove_dir_all(root).ok(); + } + + /// `rocm services` writes an engine's own `starting` state into the + /// record, so a live startup can read `starting` past the five-minute + /// stale-start window. That record is spared until + /// [`rocm_core::SERVE_STARTUP_DEADLINE`] like a `running` one; a dead pid + /// at the same age, or a live one past the deadline, is still stale. + #[test] + fn recovery_leaves_a_live_starting_record_alone_until_the_startup_deadline() -> Result<()> { + let (root, paths) = temp_app_paths("recover-live-starting"); + paths.ensure()?; + + let mut live = ManagedServiceRecord::new( + &paths, + "svc-live-starting", + "lemonade", + "qwen", + "Qwen/Qwen3.5", + "127.0.0.1", + free_loopback_port(), + "managed", + std::process::id(), + None, + None, + None, + ); + live.status = "starting".to_owned(); + let now = unix_time_millis(); + let deadline_ms = rocm_core::SERVE_STARTUP_DEADLINE.as_millis(); + // Past the stale-start window, two seconds inside the startup deadline. + live.created_at_unix_ms = now.saturating_sub(deadline_ms.saturating_sub(2_000)); + assert!(now.saturating_sub(live.created_at_unix_ms) >= SERVER_TRANSIENT_STALE_MS); + live.write()?; + let found_inside = find_recoverable_service(&paths)?; + let event_inside = service_record_matches_recovery_event( + &paths, + &live, + &recovery_event(&live, "service.manifest_recoverable"), + ); + + live.supervisor_pid = u32::MAX; + live.write()?; + let found_dead = find_recoverable_service(&paths)?; + + live.supervisor_pid = std::process::id(); + live.created_at_unix_ms = now.saturating_sub(deadline_ms); + live.write()?; + let found_at_deadline = find_recoverable_service(&paths)?; + fs::remove_dir_all(root).ok(); + + assert!( + found_inside.is_none(), + "a live starting process inside the startup deadline must not be restarted: {found_inside:?}" + ); + assert!( + !event_inside, + "a manifest event raised earlier must not restart it either" + ); + for (found, why) in [ + (found_dead, "a dead pid is a stale start"), + (found_at_deadline, "a live start past the deadline is stale"), + ] { + assert_eq!( + found.map(|(record, reason)| (record.service_id, reason)), + Some(( + "svc-live-starting".to_owned(), + "manifest_status_starting_stale".to_owned() + )), + "{why}" + ); + } + Ok(()) + } } diff --git a/crates/rocm-core/src/endpoint_readiness.rs b/crates/rocm-core/src/endpoint_readiness.rs index 52895a8be..ed28c87c0 100644 --- a/crates/rocm-core/src/endpoint_readiness.rs +++ b/crates/rocm-core/src/endpoint_readiness.rs @@ -386,6 +386,12 @@ pub fn managed_service_endpoint_model_ready( openai_models_endpoint_has_model(&record.endpoint_url, expected, endpoint_api_key, timeout) } +/// Interactive `rocm serve` startup deadline. +/// +/// `rocmd`'s server-recover watcher uses the same bound so it does not restart +/// a live process that has not opened its port yet. +pub const SERVE_STARTUP_DEADLINE: Duration = Duration::from_mins(30); + /// How far along a managed service's endpoint is. /// /// The middle state is the one that matters: an engine lists a model within diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 67f48850f..cdaf02110 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -124,7 +124,7 @@ pub use download::{ pub use endpoint_key::{endpoint_api_key_has_forbidden_chars, generate_endpoint_api_key}; pub use endpoint_readiness::{ EndpointReadiness, EndpointReadinessOutcome, INFERENCE_PROBE_TIMEOUT, - JSON_STATE_LOCK_FILE_NAME, JsonStateFileRefused, JsonStateRefusal, + JSON_STATE_LOCK_FILE_NAME, JsonStateFileRefused, JsonStateRefusal, SERVE_STARTUP_DEADLINE, engine_state_inference_verified, json_state_lock_path, managed_service_endpoint_model_ready, managed_service_endpoint_readiness, merge_json_state_file, openai_chat_completion_probe, openai_chat_completion_status, openai_models_endpoint_has_model, write_json_state_file, diff --git a/docs/architecture.md b/docs/architecture.md index d2d498bf7..45a15d301 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -111,18 +111,19 @@ The `rocm` binary: the clap command tree, a handler module for most commands, an | [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) | `rocm runtimes` (`runtimes`): activation and rollback, which write the active-runtime marker through `runtime_registry.rs`, and the runtime list (`runtimes_listing`, printed as text or `--json` through `JsonOrText`). | relocation | | [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) | `rocm serve` (`serve`): engine selection (`select_serve_engine`), the AIM pre-flight, and the GPU and device notes folded into the summary. | relocation | | [`serve_port.rs`](../apps/rocm/src/serve_port.rs) | The `rocm serve` port policy: turns an automatic or explicit `--port` request into one leased port before any manifest, log, key or child exists. | domain | +| [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) | Interactive `rocm serve` startup status: the stage line (download, load, health) and when that wait stops: ready, an engine failure, an exited process, a stalled download, too long a silence, or `SERVE_STARTUP_DEADLINE`. | domain | | [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs) | The pure transforms `serve()` applies to an engine's launch recipe hint: the catalog mapping, then `--tool-call-parser`, `--gpu-memory-utilization`, generation defaults and `--engine-env`. | relocation | | [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) | The engine-agnostic `rocm serve` deployment summary: a status indicator while the server starts, an inference smoke test, and the endpoint and timing table. | domain | | [`service_control.rs`](../apps/rocm/src/service_control.rs) | Stopping and restarting a managed local server (`rocm services stop` and `restart`, runtime moves) and the internal sandbox-tool runner (`run_internal_sandbox_tool`). Its stop path, `stop_recorded_processes`, hands the recorded PIDs to the routine [`proc_lifecycle.rs`](../crates/rocm-core/src/proc_lifecycle.rs) shares with `rocmd`. | relocation | -| [`service_hints.rs`](../apps/rocm/src/service_hints.rs) | The shared builder of the hint text that tells a user how to stop, restart or remove a service with `--yes`, plus the forced-removal hint (`service_forced_remove_hint`, `rocm services remove --force --yes`) and the command word, verb and past tense for a service tool. | flat | +| [`service_hints.rs`](../apps/rocm/src/service_hints.rs) | The shared builder of the hint text that tells a user how to stop, restart or remove a service with `--yes`, plus the forced-removal hint (`service_forced_remove_hint`, `rocm services remove --force --yes`), the `rocm logs --service ` hint and the command word, verb and past tense for a service tool. | flat | | [`service_launch.rs`](../apps/rocm/src/service_launch.rs) | Front of the `service_launch/` children: their declarations and the `pub(crate) use` re-exports other modules call, with no code of its own. | relocation | | [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) | The attached serve path (`--verbose`/`--foreground`): spawn the managed child and stream its log in the terminal until the user detaches or stops it. | relocation | | [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) | Bind-host validation and the endpoint API key checks that gate a public bind (`validate_bind_host`, `resolve_endpoint_auth`), including on a respawn. | relocation | | [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) | Starting the background automation helper when a watcher needs it, with the autostart claim that keeps concurrent callers from spawning two. | relocation | | [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) | The locked launch decision: port allocation against live records and the OS, failed-launch cleanup, and whether a live service may answer a `serve` request. The cleanup terminates a spawned child through the same `rocm_core::terminate_recorded_pid_within` routine a stop uses, so on Windows its descendants are taken down too. A spawned child whose start token was not captured, on a host that records tokens (Linux, and Windows, which records the process creation time), is not signalled (`launch_cleanup_refusal`): it was gone before it could be captured and its pid may belong to another process. | relocation | -| [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) | Waiting on a launched or live managed service: the readiness budget, process liveness, how a wait ends, and the fault-injection seam (an OOM launch, or a ready launch) for end-to-end tests. | relocation | -| [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) | Spawning the detached managed engine child both serve paths share: process and stdio plumbing and the serialized spawn transaction (`spawn_managed_engine_child`). | relocation | -| [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) | The background launch path (`start_managed_service`) and the plain line-by-line rendering of its launch report. | relocation | +| [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) | Waiting on a launched or live managed service: the readiness budget, the interactive startup wait it runs through [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) (`wait_for_launch_startup`), process liveness, how a wait ends (`settle_ended_launch`), and the fault-injection seams for end-to-end tests (an OOM launch, a ready launch, or an engine that exits during startup). | relocation | +| [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) | Spawning the detached managed engine child both serve paths share: process and stdio plumbing, the serialized spawn transaction (`spawn_managed_engine_child`), and the stand-in engine an `e2e-oom-fault-injection` build spawns for `ROCM_E2E_SIMULATE_STARTUP_EXIT`. | relocation | +| [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) | The background launch path (`start_managed_service`), which returns its launch report or the typed `ServeStartupFailed`, and the plain line-by-line rendering of that report. | relocation | | [`service_launch/test_support.rs`](../apps/rocm/src/service_launch/test_support.rs) | Test fixtures shared by more than one `service_launch/` child. | flat | | [`service_records.rs`](../apps/rocm/src/service_records.rs) | Removing a managed-service record and every file it owns — including any copy of its endpoint key an interrupted save left behind, found with one listing of the services folder per command (`staged_endpoint_keys_by_service`) — and the `rocm services prune` plan, render and apply steps with their age gate. `remove_managed_service_record_with` takes `--force` and the read of each recorded PID: `--force` removes a record only when none of its recorded processes is confirmed running, one could not be queried and a stop was requested, and reports which. | relocation | | [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) | `rocm services` (`services`) for `list`, `logs`, `stop`, `restart`, `remove` and `prune`, and the approval-gated stop and restart path. | relocation | @@ -187,6 +188,7 @@ Boundaries: **Services and uninstall** +- [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs) decides when the interactive startup wait stops; [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) samples the manifest for it, [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) turns its stop into `ServeStartupFailed`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) picks the policy, draws its view on the spinner and supervises only a stop that left the server running. - `service_launch` does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it (the Ctrl-C stop is passed into `run_attached_service`), and [`services_text.rs`](../apps/rocm/src/services_text.rs) does not call back into [`service_control.rs`](../apps/rocm/src/service_control.rs). - [`background_helper.rs`](../apps/rocm/src/background_helper.rs) imports nothing from the service modules, so its helper note can be read without importing the spawner. - [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) depend on each other; the plan half is separate because [`storage.rs`](../apps/rocm/src/storage.rs) shares it. @@ -194,7 +196,7 @@ Boundaries: **Shared sources and contracts** -- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) is the shared builder of the `rocm services stop|restart|remove --yes` hint text, so the callers that use it cannot drift apart. Those callers are [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_port.rs`](../apps/rocm/src/serve_port.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), [`services_text.rs`](../apps/rocm/src/services_text.rs) and, under `service_launch/`, [`attach.rs`](../apps/rocm/src/service_launch/attach.rs) and [`spawn.rs`](../apps/rocm/src/service_launch/spawn.rs); the parent [`service_launch.rs`](../apps/rocm/src/service_launch.rs) does not call it. +- [`service_hints.rs`](../apps/rocm/src/service_hints.rs) is the shared builder of the `rocm services stop|restart|remove --yes` and `rocm logs --service ` hint text, so the callers that use it cannot drift apart. Those callers are [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`serve_port.rs`](../apps/rocm/src/serve_port.rs), [`serve_progress.rs`](../apps/rocm/src/serve_progress.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), [`services_text.rs`](../apps/rocm/src/services_text.rs) and, under `service_launch/`, [`attach.rs`](../apps/rocm/src/service_launch/attach.rs), [`readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) and [`spawn.rs`](../apps/rocm/src/service_launch/spawn.rs); the parent [`service_launch.rs`](../apps/rocm/src/service_launch.rs) does not call it. - `DEFAULT_ENGINE_MARKER` in [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) is shared by both engine-inventory renderers and their legend line, so the rendered marker and the legend cannot drift apart. - [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) is the CLI's quoter; `rocm-dash-tui` keeps a copy (`quote_display_arg`) because it may not depend on `rocm-core`, and a property test here holds the two equal. - The local-model thinking rule [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) applies lives in `rocm-dash-core`'s [`chat_template.rs`](../crates/rocm-dash-core/src/chat_template.rs), because the dashboard's chat client must decide it identically. @@ -234,7 +236,7 @@ The `rocmd` binary and the daemon library `rocm daemon` embeds: the watcher loop | [`watchers/gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) | The `gpu-metrics` watcher: records GPU telemetry and reads amd-smi monitor entries into thermal- and memory-pressure events, with the interval and thresholds they use. | flat | | [`watchers/gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) | The `gpu-thermal-protect` watcher: on GPU pressure, picks the one running managed service and queues a reviewed stop request unless one is pending. | flat | | [`watchers/policy.rs`](../apps/rocmd/src/watchers/policy.rs) | What a watcher does in each mode (`watcher_policy_action`: observe, propose or run contained) and the proposals it queues (`queue_proposal`, `pending_proposal_exists`). | flat | -| [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) | The `server-recover` watcher: finds a managed service that needs recovery (`find_recoverable_service`), spaces attempts with a backoff, and restarts it or queues a restart proposal. | flat | +| [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) | The `server-recover` watcher: finds a managed service that needs recovery (`find_recoverable_service`), sparing a live one younger than `rocm_core::SERVE_STARTUP_DEADLINE` whose port is not open yet (`live_startup_still_in_progress`, which checks the recorded start time so a recycled pid is not spared), spaces attempts with a backoff, and restarts it or queues a restart proposal. | flat | | [`watchers/therock_update.rs`](../apps/rocmd/src/watchers/therock_update.rs) | The `therock-update` watcher: the six-hourly update check, with a reminder, a proposal or a read-only contained run depending on mode. | flat | | [`webhook.rs`](../apps/rocmd/src/webhook.rs) | The local webhook source: its axum routes, request validation and watcher-kind allow-list. | flat | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index a9e8a851e..f7c82e145 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -434,6 +434,16 @@ Expected result: - The OpenAI-compatible endpoint answers a simple chat request. - `rocm services list` shows an `anthropic endpoint:` URL, and `POST /v1/messages` with `{"model":"x","max_tokens":16,"messages":[{"role":"user","content":"Hello"}]}` returns an Anthropic `message` that names the served model. The same request with `"stream":true` answers with `text/event-stream` (`message_start` through `message_stop`, text in `content_block_delta` events). A request with `tools` and `"tool_choice":{"type":"tool","name":""}` returns a `tool_use` content block naming that tool with its arguments and `"stop_reason":"tool_use"`; sending the call back as a `tool_result` block gets the model's answer. Combining `"stream":true` with `tools` is refused with an `invalid_request_error`. +The same model without `--foreground` or `--verbose` is the interactive +summary. While the server starts, the status line follows the engine (a +download percent, a load line, or a wait for the model to serve) and keeps +updating past the first minute. It does not finish with +`Deployment summary (not ready yet)` while the server is still loading. A +ready server then prints the deployment summary. If that wait fails and the +process is still up, the message names `rocm logs --service ` and +`rocm services stop --yes`, and `rocm services` still lists the +server. + If the llama.cpp server exits at start with an access violation (`0xC0000005` in `MSVCP140.dll`), the Visual C++ runtime on the machine is probably older than 14.40. Run `rocm diagnose` and expect `fix-21-msvc-redist-outdated`. `rocm fix @@ -619,6 +629,20 @@ Expected result: and how to free it (`rocm services stop --yes`), without spawning an engine process or writing a new record. +The default command, without `--foreground` or `--verbose`, leaves the server +in the background. While it starts, the status line follows the engine: a +download percent, a load line, or a wait for the model to serve. A model that +is still loading after 45 seconds (the old wait, which piped output or a redirected stderr still +uses) keeps that line; it is not replaced by a finished "not ready yet" +summary. The wait fails with a non-zero exit only when the engine fails or +exits, a download percent below 100 stops for 3 minutes, the endpoint has not listed the model with no +engine log for 5 minutes, a load line stops changing for 5 minutes, or 30 +minutes pass. When the endpoint answers, the +command prints the deployment summary and returns to the shell. If the wait +fails and the process is still up, the message names `rocm logs --service +` and `rocm services stop --yes`, and `rocm services` +still lists the server. + ### Windows display driver check On a Windows machine with an AMD GPU, check what the CLI says about the display diff --git a/docs/testing.md b/docs/testing.md index 5435b6db6..44c1838a2 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2561,6 +2561,56 @@ cargo test -p rocm --bin rocm serve_engine_selection These tests verify that explicit/configured engines still win and shared model recipes only choose a preferred engine when no user/default engine is set. +Interactive `rocm serve` (a terminal, without `--verbose` or `--foreground`) +keeps a status line through download, load, and health until the server can +serve or that wait fails. A download that is still moving is not cut off at +45 seconds, and a failure names `rocm logs --service ` and +`rocm services stop --yes` instead of printing +`Deployment summary (not ready yet)`. The wait fails when the engine fails or +exits, a download percent below 100 stops for 3 minutes (`DOWNLOAD_STALL`; a line at 100% is judged as loading), the endpoint +has not listed the model, with no log, for 5 minutes (`SILENT_UNREACHABLE`, equal to `SILENT_LOAD`), a load line +stops changing for 5 minutes (`SILENT_LOAD`), or after 30 minutes +(`rocm_core::SERVE_STARTUP_DEADLINE`). Engine log lines are masked of +credentials before they are shown (`redact_log_line`). Once the endpoint lists the model, +the one-token inference request is not sent every poll: it backs off from 1 +second to `INFERENCE_PROBE_RETRY_INTERVAL` (15 seconds), timed from when the +previous request returned (`InferenceProbeThrottle`, tested under a fake clock +by `inference_probe_backs_off_to_the_retry_interval_after_listing`). `rocmd` does not +restart a live `running` service whose endpoint does not answer, or a live +`starting` one past the stale-start window, while it is younger than that +deadline (a pid whose recorded start time does not match is not live); a +recoverable engine healthcheck is still restarted. +Piped output, or a redirected stderr (`ServeStartupPolicy::for_background_serve`), keeps the 45-second status budget. Like `--wait`, the +interactive wait first prints the service id, `rocm logs --service + --follow` and a public bind's key (`announces_wait` in +[start.rs](../apps/rocm/src/service_launch/start.rs)), since it can fail +without printing a summary; `--wait` keeps its own wait, with no CLI-side +stall limit or deadline. A wait that ends because the engine failed or the +process exited settles the record the way a `--wait` does (`stopped`, or the +engine's own `failed`), records an audit event and adds the same memory +guidance (a `--wait` that sees the server exit now records the event too); +when the engine failed but its process is still up, the message names the +stop command. Only a stall or the +deadline leaves the server running for the background helper to supervise. +Scenario `serve-63` in `model_serving.feature` drives it under a PTY. It +needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes only; +it checks the wait line, that the named `rocm logs --service` command runs, +and that the serve exits 0 with the model listed. Scenario `serve-64` is the +failure side on the no-GPU mock lane: with `ROCM_E2E_SIMULATE_STARTUP_EXIT=1`, +an `e2e-oom-fault-injection` build (Linux only) spawns a stand-in engine that +logs one line and exits during startup, in place of the real child. It checks +the non-zero exit and the "server process exited" message, runs the +`rocm logs --service` command the failure names, and checks the record is +`stopped` (`cargo xtask e2e -- -n "serve-64"`). The stall, deadline and +engine-failed branches need a controllable engine, so `decide_serve_startup` and `wait_for_serve_startup` in +[serve_progress.rs](../apps/rocm/src/serve_progress.rs) are the state matrix, +covered by: + +```bash +cargo test -p rocm --bin rocm serve_progress +cargo test -p rocmd recovery_leaves_a_live +``` + Public-endpoint authentication focused tests: ```bash diff --git a/tests/e2e-cucumber/README.md b/tests/e2e-cucumber/README.md index c68381db7..967e98ba3 100644 --- a/tests/e2e-cucumber/README.md +++ b/tests/e2e-cucumber/README.md @@ -196,7 +196,7 @@ Scenarios carry stable-id and capability tags: | `@requires-case-sensitive-fs` | Premise is two files whose names differ only in letter case. Resolves to **skip** where the scenarios' temp root folds case. The capability probe's `case_sensitive_fs` writes two such names in its own temp dir, which follows `$TMPDIR` like every scenario's isolated root, and checks they stayed two files. `@requires-os:linux` cannot express this: `$TMPDIR` on Linux, WSL2 included, can sit on a casefolded directory or a mounted Windows drive. | | `@requires-engine:` | Pins the serve engine. Resolves to skip where that engine can't start (e.g. vLLM on a lemonade-only Strix host). | | `@requires-os:` | Premise is OS-specific; skip on other OSes. | -| `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch so `rocm serve`'s summary can be verified without a GPU: one that runs out of GPU memory (the OOM guidance), or, in its `ready` mode, one that becomes ready and then fails its smoke test. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | +| `@requires-oom-fault-injection` | Needs the `rocm` binary under test to carry the test-only `e2e-oom-fault-injection` hook, which fabricates a managed launch so `rocm serve`'s summary can be verified without a GPU: one that runs out of GPU memory (the OOM guidance), or, in its `ready` mode, one that becomes ready and then fails its smoke test. On Linux the same feature also carries a stand-in engine (`ROCM_E2E_SIMULATE_STARTUP_EXIT=1`) that logs a line and exits during startup, so the interactive startup wait's failure path runs for real. Unlike every other capability this one **cannot be probed** — no product command exposes it — so `xtask e2e` reports it via `ROCM_E2E_OOM_FAULT_INJECTION=1`, set only when xtask compiled the binary itself with the feature. A prebuilt `ROCM_CLI_BINARY` (the self-hosted lanes' shipping release build) has the hook compiled out, so these scenarios resolve to **skip** there and run on the mock lane. | | `@serve-timeout:` | Lengthen the serve-readiness wait for a genuinely slow serve (e.g. a large model). | | `@nightly` | Expensive scenario skipped by default; included when `E2E_INCLUDE_NIGHTLY=1`. | | `@lifecycle` | Expensive, OS-mutating release-lifecycle scenario (packaging + real installer + install/uninstall). Skipped by default; included when `E2E_INCLUDE_LIFECYCLE=1`. `E2E_ONLY_LIFECYCLE=1` selects only this set without bypassing expectation resolution. A caller can additionally fold specific non-`@lifecycle` scenarios into that restricted set via `E2E_ALSO_RUN_IDS` (comma-separated `@id`s), without tagging them `@lifecycle` — which would also pull them out of the full-suite lanes that don't set `E2E_INCLUDE_LIFECYCLE`. `ci.yml`'s Windows lane uses this to fold in the hermetic comfyui-01/02/03 scenarios, the native-Windows vLLM refusal scenarios serve-33/34/35, and the side-by-side runtime scenarios runtime-lifecycle-19/20/21. | diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index d6db3d214..c406ca8d9 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -1,5 +1,15 @@ Feature: Model serving + # Interactive `rocm serve` (a terminal, not --verbose or --foreground) follows + # download, load, and health on the status line until the server is ready or + # the startup wait fails. Scenario serve-63 drives it under a real PTY; it + # needs a live engine, so it is `@requires-gpu` and runs on the GPU lanes + # only. serve-64 covers an engine that exits during startup on the no-GPU + # mock lane with a stand-in engine. The stall, deadline and engine-failed + # branches need a controllable engine, so no lane reaches them; `decide_serve_startup` and + # `wait_for_serve_startup` in apps/rocm/src/serve_progress.rs are their + # state matrix. + # Per-platform expectations (pass / xfail / skip) are resolved at runtime from # the host capability probe + expectations.toml, keyed by the @id tag — not by # @expected-failure tags. Engine requirements are declared via @requires-engine @@ -1019,3 +1029,33 @@ Feature: Model serving When the user lists the engines of that install Then vLLM is reported as installed but not usable And the report names the extension that failed to import + + # Interactive `rocm serve` under a real terminal: while it waits it names the + # command that follows the engine's log, then it exits 0 once the endpoint + # lists the model. The named command is run, not just read: a hint that + # points at a command that does not work would still pass a string check. + @id:serve-interactive-startup-progress @requires-gpu + Scenario: serve-63 - Interactive serve shows how to follow startup and exits when the model is served + Given a managed runtime is active + When the user serves a model on GPU under a real terminal + Then the terminal names the command that follows the server's log + And the named log command runs successfully + And the serve exits cleanly and the endpoint lists the model + + # The failure side of serve-63, on the mock lane. A real engine cannot start on + # a host with no GPU, so the `e2e-oom-fault-injection` build swaps the managed + # child for a stand-in that writes one log line and exits during startup + # (`ROCM_E2E_SIMULATE_STARTUP_EXIT`); the record, the log, the startup wait and + # the settling are the production path. The hook is absent from shipped + # binaries, so @requires-oom-fault-injection keeps this off the self-hosted + # lanes. The log command the failure names is run, and the record is checked, + # so the message cannot claim a log or a settled launch that is not there. + @id:serve-startup-exit-names-log @requires-no-gpu @requires-os:linux @requires-oom-fault-injection + Scenario: serve-64 - An engine that exits during interactive startup fails the serve and names its log + Given a managed engine will exit while it starts + When the user serves a model with that engine under a real terminal + Then the serve fails saying the server process exited + And the failure names the command that shows the server's log + And that log command shows the engine's last line + And the failed launch is recorded as stopped + And no background automation daemon was left behind diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index c649d4de6..0b5fc5eca 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -639,6 +639,197 @@ fn claim_relaunch() -> bool { .is_ok() } +#[when("the user serves a model on GPU under a real terminal")] +async fn user_serves_gpu_model_under_pty(world: &mut E2eWorld) { + let (model, engine, _) = host_serve_target(); + // Same free-port precondition as `serve_gpu_model`: a leaked server from a + // prior scenario would answer for this one. + let _ = ensure_serve_port_free().await; + // Summary mode (the interactive status line) is `stdout is a terminal` and + // neither `--verbose` nor `--foreground`, so the PTY is the whole setup. + let session = TuiSession::spawn(world, &["serve", model, "--engine", engine, "--managed"]) + .unwrap_or_else(|e| panic!("failed to spawn `rocm serve` under a pty: {e}")); + world.tui = Some(session); +} + +/// The id the wait line's `rocm logs --service --follow` names. +fn followed_service_id(screen: &str) -> Option { + let rest = screen.split("rocm logs --service ").nth(1)?; + let id = rest.split_whitespace().next()?; + Some(id.trim_matches('\'').to_owned()) +} + +#[then("the terminal names the command that follows the server's log")] +async fn terminal_names_log_command(world: &mut E2eWorld) { + let timeout = Duration::from_secs(serve_timeout_for(world)); + let session = world.tui.as_mut().expect("no pty session for `rocm serve`"); + session + .wait_for_screen("rocm logs --service ", timeout) + .await + .unwrap_or_else(|e| panic!("the wait line never named a log command: {e}")); + let screen = session.screen_text(); + assert!( + screen.contains("Ctrl-C stops waiting"), + "the wait line does not say Ctrl-C stops waiting, not the server:\n{screen}" + ); + followed_service_id(&screen) + .unwrap_or_else(|| panic!("no service id after `rocm logs --service`:\n{screen}")); +} + +#[then("the named log command runs successfully")] +async fn named_log_command_runs(world: &mut E2eWorld) { + let screen = world + .tui + .as_ref() + .expect("no pty session for `rocm serve`") + .screen_text(); + let id = followed_service_id(&screen) + .unwrap_or_else(|| panic!("no service id after `rocm logs --service`:\n{screen}")); + // Without `--follow`, which never returns. The same id and subcommand the + // hint names, so a hint that points at nothing fails here. + let (stdout, stderr, rc) = crate::run_rocm(world, &["logs", "--service", &id]); + assert_eq!( + rc, 0, + "`rocm logs --service {id}` failed.\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); +} + +#[then("the serve exits cleanly and the endpoint lists the model")] +async fn serve_exits_and_endpoint_lists_model(world: &mut E2eWorld) { + let timeout = Duration::from_secs(serve_timeout_for(world)); + let (_, _, ready_substr) = host_serve_target(); + world + .tui + .as_mut() + .expect("no pty session for `rocm serve`") + .assert_exits_cleanly("the interactive serve", timeout) + .await; + assert!( + model_is_ready(&MODELS_URL, Some(ready_substr), 30).await, + "serve exited 0 but {} does not list {ready_substr}", + MODELS_URL.as_str() + ); + world.endpoint = Some(SERVE_BASE_URL.to_string()); +} + +#[given("a managed engine will exit while it starts")] +async fn arm_startup_exit_stand_in(world: &mut E2eWorld) { + // Consumed by `e2e_simulate_startup_exit` in + // `apps/rocm/src/service_launch/readiness.rs`; spelled out on both sides + // for the same reason as `ROCM_E2E_SIMULATE_OOM_LAUNCH`. A typo does not + // arm the stand-in, the no-GPU pre-flight refuses, and the Then steps fail. + world + .command_env + .push(("ROCM_E2E_SIMULATE_STARTUP_EXIT", "1".into())); +} + +#[when("the user serves a model with that engine under a real terminal")] +async fn serve_with_startup_exit_stand_in(world: &mut E2eWorld) { + let mut session = TuiSession::spawn( + world, + &[ + "serve", + OOM_LAUNCH_MODEL, + "--engine", + "vllm", + "--env-id", + "e2e-startup-exit", + ], + ) + .unwrap_or_else(|e| panic!("failed to spawn `rocm serve` under a pty: {e}")); + let code = session + .wait_for_exit_code(interactive_summary_timeout()) + .await + .unwrap_or_else(|e| panic!("`rocm serve` did not exit: {e}")); + world.tui = Some(session); + world.cli_rc = Some(code); +} + +/// The final screen with all whitespace removed: the 80-column PTY wraps the +/// failure message across rows. +fn startup_exit_screen(world: &E2eWorld) -> (String, String) { + let screen = world + .tui + .as_ref() + .expect("no pty session for `rocm serve`") + .screen_text(); + (screen_without_whitespace(&screen), screen) +} + +/// The id in the failure's "The log is `rocm logs --service `". +fn failed_launch_service_id(world: &E2eWorld) -> String { + let (flat, screen) = startup_exit_screen(world); + let marker = "Thelogis`rocmlogs--service"; + let rest = flat + .split(marker) + .nth(1) + .unwrap_or_else(|| panic!("the failure does not name `rocm logs --service`:\n{screen}")); + let id = rest + .split('`') + .next() + .unwrap_or_default() + .trim_matches('\''); + assert!(!id.is_empty(), "no service id in the log hint:\n{screen}"); + id.to_owned() +} + +#[then("the serve fails saying the server process exited")] +async fn serve_fails_with_process_exited(world: &mut E2eWorld) { + let (flat, screen) = startup_exit_screen(world); + let code = world.cli_rc; + assert!( + code.is_some_and(|code| code != 0), + "an engine that exited during startup must fail the serve (exit {code:?}):\n{screen}" + ); + assert!( + flat.contains("failed:theserverprocessexited"), + "the failure must say the server process exited:\n{screen}" + ); + assert!( + flat.contains("Theserverdidnotstart"), + "the failure must say the server did not start, not that it is still running:\n{screen}" + ); +} + +#[then("the failure names the command that shows the server's log")] +async fn failure_names_log_command(world: &mut E2eWorld) { + failed_launch_service_id(world); +} + +#[then("that log command shows the engine's last line")] +async fn log_command_shows_engine_line(world: &mut E2eWorld) { + let id = failed_launch_service_id(world); + let (stdout, stderr, rc) = crate::run_rocm(world, &["logs", "--service", &id]); + assert_eq!( + rc, 0, + "`rocm logs --service {id}` failed.\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); + assert!( + stdout.contains("e2e stand-in engine: exiting during startup"), + "`rocm logs --service {id}` does not show the engine's last line:\n{stdout}" + ); +} + +#[then("the failed launch is recorded as stopped")] +async fn failed_launch_recorded_stopped(world: &mut E2eWorld) { + let id = failed_launch_service_id(world); + let (stdout, stderr, rc) = crate::run_rocm(world, &["services", "list", "--all", "--json"]); + assert_eq!(rc, 0, "`rocm services list --all --json` failed:\n{stderr}"); + let records: serde_json::Value = serde_json::from_str(&stdout) + .unwrap_or_else(|e| panic!("services list --json did not parse: {e}\n{stdout}")); + let status = records + .as_array() + .into_iter() + .flatten() + .find(|record| record.get("service_id").and_then(serde_json::Value::as_str) == Some(&id)) + .and_then(|record| record.get("status").and_then(serde_json::Value::as_str)) + .unwrap_or_else(|| panic!("no record for {id}:\n{stdout}")) + .to_owned(); + // `stopped`, not `failed`: `rocmd` restarts a `failed` record with no stop + // marker, behind the error the user just read. + assert_eq!(status, "stopped", "{stdout}"); +} + #[given("a model is being served on GPU")] async fn setup_gpu_model(world: &mut E2eWorld) { serve_gpu_model(world).await; diff --git a/xtask/src/ascii_output_contract.rs b/xtask/src/ascii_output_contract.rs index d4d802aef..e77fcefd3 100644 --- a/xtask/src/ascii_output_contract.rs +++ b/xtask/src/ascii_output_contract.rs @@ -189,8 +189,13 @@ mod tests { SPINNER_LABEL, ), ( - "apps/rocm/src/serve_cmd.rs", - "Starting {model} on {selected_engine}…", + "apps/rocm/src/serve_progress.rs", + "Starting {model} on {engine}…", + SPINNER_LABEL, + ), + ( + "apps/rocm/src/serve_progress.rs", + "Waiting for {model} to serve…", SPINNER_LABEL, ), (