Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
c44e0b0
fix(serve): keep startup progress until the server is ready or stalled
rominf Nov 10, 2026
06dfc79
fix(serve): space inference probes and settle an ended startup
rominf Nov 10, 2026
802eee2
test(serve): give the startup-wait tests an endpoint that answers at …
rominf Nov 11, 2026
9e8419b
fix(serve): quote the logs hint and cut redacted values on a char bou…
rominf Nov 11, 2026
ad1b07c
fix(serve): judge silence by its own length and check a spared pid's …
rominf Nov 11, 2026
2ffa4f9
Merge origin/main into fix/serve-startup-progress
rominf Nov 12, 2026
eb3ce2a
test(serve): add a @requires-gpu PTY scenario for interactive serve s…
rominf Nov 13, 2026
0493acf
Merge origin/main into fix/serve-startup-progress
rominf Nov 16, 2026
e4933a9
fix(serve): record an unlisted stalled launch as starting; cover star…
rominf Nov 16, 2026
ced7f76
fix(serve): say a stopped wait's server may be restarted after the st…
rominf Nov 16, 2026
f04ebd1
fix(serve): follow the startup only when stderr is a terminal too
rominf Nov 16, 2026
3d41093
test(serve): end model_serving.feature with a single newline
rominf Nov 16, 2026
7fa1193
fix(serve): judge a finished download as loading; quote the engine li…
rominf Nov 17, 2026
d6d2e90
Merge origin/main into fix/serve-startup-progress
rominf Nov 17, 2026
4c6cf19
docs(serve): name where the code makes two startup-wait comments true
rominf Nov 17, 2026
3cc61aa
fix(serve): address the fourth review round on the startup wait
rominf Nov 18, 2026
36bd652
Merge origin/main into fix/serve-startup-progress
rominf Nov 18, 2026
0c2d074
test(serve): do not print a rendered summary in a failed assertion
rominf Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1405,20 +1405,39 @@ Likewise, an explicit `--port` that differs from the port the running service
is already on is refused rather than silently ignored; omit `--port` to reuse
the running service, or stop it first to serve on the new port.

By default the server runs in the background under ROCm CLI's supervision and
prints a deployment summary: a progress indicator while it starts, then a table
with the status, the full inference endpoint, the API-qualified model name, and a
quick smoke test (time to first token and approximate tokens per second). On an
By default the server runs in the background under ROCm CLI's supervision.
For a fresh launch in a terminal it first prints the service id, how to follow its log
(`rocm logs --service <service-id> --follow`) and, for a public bind, the
API key. While it starts, the status line follows the engine: a download, a weight
load, or the latest log line, until the server is ready or that wait fails
(the engine exits, a download stops advancing, or the startup deadline
passes). When the server is ready it prints a deployment summary: a table
with the status, the full inference endpoint, the API-qualified model name,
and a quick smoke test (time to first token and approximate tokens per
second). On an
interactive terminal, if the smoke test cannot complete, the summary is headed
`(smoke test failed)`, shows the status as `started, smoke test failed` instead
of `ready`, and a note gives the reason and points at
`rocm logs --service <id>`. ROCm CLI does not stop the server, and the command
still exits 0. Which failures are detected, and which still read as `ready`, is
listed under
[Serve Smoke Test Failure](https://github.com/ROCm/rocm-cli/blob/main/docs/testing.md#serve-smoke-test-failure)
in the testing guide. Control
returns to your shell with the server still running; manage it later with `rocm
services` (below).
in the testing guide. A model that is still loading is not reported as a finished "not
ready" summary. The wait fails, and `rocm serve` exits non-zero, when the
engine reports failure or its process exits, when a download percent stops
advancing for 3 minutes (a line at 100% counts as loading), when the endpoint has not listed the model and there is no engine log for
5 minutes, when a load log line stops changing for 5 minutes with the endpoint
not yet listing the model, or after 30 minutes in all. A stall leaves the server running:
the message names `rocm logs --service <service-id>` and
`rocm services stop <service-id> --yes`, and the background helper does not
restart a live server for being slow until those 30 minutes pass. Engine log
lines shown on the status line have credentials (the endpoint key, `--api-key`
and token values, `hf_` tokens) masked. Once the server lists the model, the wait
checks whether it can answer with a one-token request, first right away and
then at growing intervals up to every 15 seconds. Piped output, or a stderr that is not a terminal (`2>log`), keeps
the 45-second wait and reports the server as it stood then. Control returns
to your shell with the server still running; manage it later with
`rocm services` (below).

`--verbose` (or `--foreground`) instead attaches to the server in the current
terminal and streams every engine log line. Use it to debug a startup problem.
Expand All @@ -1436,8 +1455,8 @@ can:
`--managed` is the explicit form of the default background behavior.
`--wait` keeps `rocm serve` waiting until the server answers requests or its
engine gives up (vLLM and AIM after 5 minutes by default; raise it with
`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), instead
of returning `starting` after 45 seconds; if the server exits first (crashes, is killed, or is stopped with
`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), with
no CLI-side stall limit or deadline, instead of the default wait above; if the server exits first (crashes, is killed, or is stopped with
`rocm services stop`), it fails with an error that points at the server's logs
instead of waiting forever. Ctrl-C ends the wait and leaves the server starting
in the background; a generated API key (public bind) is printed when the wait
Expand Down
14 changes: 10 additions & 4 deletions apps/rocm/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -512,8 +512,13 @@ rocm model --verbose"
/// Picks an engine and ROCm runtime automatically unless overridden. An
/// Anthropic `/v1/messages` endpoint is published beside the OpenAI
/// one (see `rocm services list`). By default the
/// server runs as a managed background service and prints a deployment summary
/// (status, endpoint, model, and smoke-test metrics); use --verbose to stream engine
/// server runs as a managed background service. For a fresh launch in a terminal it
/// first prints the service id and how to follow its log; while it starts, the status line
/// follows what the engine is doing (download, load, or the latest log line) until
/// the server is ready or that wait fails. A ready server then prints a deployment
/// summary (status, endpoint, model, and smoke-test metrics). A wait that fails
/// while the server is still running names `rocm logs` and `rocm services stop`
/// instead of a finished "not ready" summary. Use --verbose to stream engine
/// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the
/// server running, or Ctrl-C to stop it. If that stop answers that it could not confirm
/// the server exited, it says so on stderr and exits with an error; if the stop itself
Expand Down Expand Up @@ -602,8 +607,9 @@ rocm serve qwen --verbose --device gpu_required")]
/// Keep waiting until the server answers requests or its engine gives up
/// (vLLM and AIM after 5 minutes by default, raised with
/// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`),
/// instead of returning `starting` after 45 seconds.
/// Also waits on a matching server that is already starting. Fails with a
/// with no CLI-side stall limit or deadline. Without it, a terminal gives up when
/// startup progress stalls or after 30 minutes, and piped output (or a redirected stderr) returns
/// `starting` after 45 seconds. Also waits on a matching server that is already starting. Fails with a
/// pointer to the logs if the server exits first. Cannot be
/// combined with --foreground or --verbose, which already stay attached.
#[arg(long, conflicts_with_all = ["foreground", "verbose"])]
Expand Down
105 changes: 71 additions & 34 deletions apps/rocm/src/engine_runtime_select.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@
//! `service_launch.rs` and `service_control.rs`'s `restart_internal_managed_service_with`.
//! `wait_for_service_http_ready` is `pub(crate)`, read from `service_control.rs`'s
//! `restart_internal_managed_service_with`; `wait_for_service_http_ready_with_progress`
//! is `pub(crate)`, read from `service_launch.rs`. `http_post_local_service_json`
//! is `pub(crate)`, read from `service_launch.rs`. `service_http_lists_model` and
//! `service_http_inference_answers`, the two halves of its readiness probe, are
//! `pub(crate)`, read from `serve_progress.rs`'s `wait_for_serve_startup`. `http_post_local_service_json`
//! is `pub(crate)`, read from `service_control.rs`'s `unload_lemonade_service_model`.
//! `service_model_names_match` is `pub(crate)`, read from `chat/turn.rs`.
//! `service_http_readiness_paths`, `http_get_local_service`,
Expand Down Expand Up @@ -348,50 +350,85 @@ pub(crate) fn wait_for_service_http_ready_with_progress(
on_tick: &mut dyn FnMut(Duration),
) -> EndpointReadiness {
let start = std::time::Instant::now();
let endpoint = format_http_base_url(host, port);
let mut best = EndpointReadiness::Unreachable;
while start.elapsed() < timeout {
// Re-evaluated each pass: a container engine reports the name its server
// answers to only after it starts, so a name fixed up front can be wrong.
let canonical_model_id = model_name();
let canonical_model_id = canonical_model_id.as_str();
let listed = service_http_readiness_paths(engine).iter().any(|path| {
http_get_local_service(
host,
port,
path,
endpoint_api_key,
Duration::from_millis(750),
)
.is_ok_and(|(status, body)| {
service_http_readiness_response_ready(
engine,
path,
status,
&body,
canonical_model_id,
)
})
});
if listed {
best = EndpointReadiness::Listing;
if rocm_core::openai_chat_completion_probe(
&endpoint,
canonical_model_id,
endpoint_api_key,
rocm_core::INFERENCE_PROBE_TIMEOUT,
)
.unwrap_or(false)
{
return EndpointReadiness::Serving;
}
match probe_service_http_readiness(engine, host, port, &model_name(), endpoint_api_key) {
EndpointReadiness::Serving => return EndpointReadiness::Serving,
EndpointReadiness::Listing => best = EndpointReadiness::Listing,
EndpointReadiness::Unreachable => {}
}
on_tick(start.elapsed());
thread::sleep(Duration::from_millis(250));
}
best
}

/// One readiness probe: [`EndpointReadiness::Serving`] only after an inference
/// request comes back, [`EndpointReadiness::Listing`] when the model is
/// advertised but that request does not, otherwise unreachable.
///
/// Probes inference on every call. [`crate::serve_progress::wait_for_serve_startup`]
/// spaces those probes out instead, through [`service_http_lists_model`] and
/// [`service_http_inference_answers`], the same two checks, so the interactive
/// status line and this wait cannot disagree about "ready".
fn probe_service_http_readiness(
engine: &str,
host: &str,
port: u16,
canonical_model_id: &str,
endpoint_api_key: Option<&str>,
) -> EndpointReadiness {
if !service_http_lists_model(engine, host, port, canonical_model_id, endpoint_api_key) {
return EndpointReadiness::Unreachable;
}
if service_http_inference_answers(host, port, canonical_model_id, endpoint_api_key) {
return EndpointReadiness::Serving;
}
EndpointReadiness::Listing
}

/// The cheap half of a readiness probe: the endpoint answers and advertises
/// the model (a short GET per readiness path).
pub(crate) fn service_http_lists_model(
engine: &str,
host: &str,
port: u16,
canonical_model_id: &str,
endpoint_api_key: Option<&str>,
) -> bool {
service_http_readiness_paths(engine).iter().any(|path| {
http_get_local_service(
host,
port,
path,
endpoint_api_key,
Duration::from_millis(750),
)
.is_ok_and(|(status, body)| {
service_http_readiness_response_ready(engine, path, status, &body, canonical_model_id)
})
})
}

/// The expensive half: a real one-token chat request, which can wait up to
/// [`rocm_core::INFERENCE_PROBE_TIMEOUT`] on a model that is still loading.
pub(crate) fn service_http_inference_answers(
host: &str,
port: u16,
canonical_model_id: &str,
endpoint_api_key: Option<&str>,
) -> bool {
rocm_core::openai_chat_completion_probe(
&format_http_base_url(host, port),
canonical_model_id,
endpoint_api_key,
rocm_core::INFERENCE_PROBE_TIMEOUT,
)
.unwrap_or(false)
}

fn service_http_readiness_paths(engine: &str) -> &'static [&'static str] {
match engine {
"lemonade" => &["/v1/health", "/v1/models"],
Expand Down
1 change: 1 addition & 0 deletions apps/rocm/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ mod runtime_uninstall;
mod runtimes_cmd;
mod serve_cmd;
mod serve_port;
mod serve_progress;
mod serve_recipe;
mod serve_summary;
mod service_control;
Expand Down
Loading
Loading