Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a789a8a
ROCMAI-791: add rocm logs --follow to watch a starting server's output
juhovainio Nov 3, 2026
82d482a
ROCMAI-791: add rocm serve --wait to wait out a first-time model down…
juhovainio Nov 3, 2026
2f6d0b5
ROCMAI-791: end serve --wait when the server dies; harden --follow ou…
juhovainio Nov 4, 2026
b9deedf
ROCMAI-791: test the serve --wait server-died path; keep a failed state
juhovainio Nov 4, 2026
8fedcfa
ROCMAI-791: fix CI: const fn on non-Linux zombie probe, feature-quali…
juhovainio Nov 4, 2026
d0062bf
ROCMAI-791: keep the --follow hint under the newest log line on a ter…
juhovainio Nov 4, 2026
567774f
ROCMAI-791: merge main; renumber the --wait scenario to serve-42, res…
juhovainio Nov 5, 2026
486dcc3
ROCMAI-791: order serve-42 after serve-41 so scenario names stay sequ…
juhovainio Nov 5, 2026
f14f485
ROCMAI-791: wait on an already-starting server, tighten follow output
juhovainio Nov 5, 2026
125ba11
ROCMAI-791: name the real service id in the --wait banner; strip esca…
juhovainio Nov 5, 2026
d05bc07
ROCMAI-791: say --wait ends when the engine gives up, not only when t…
juhovainio Nov 5, 2026
2aeeea4
ROCMAI-791: follow/wait review items: key before wait, fresh record w…
juhovainio Nov 5, 2026
84b539f
ROCMAI-791: keep the dummy api key out of wait-begin test failure mes…
juhovainio Nov 5, 2026
0476f4d
Merge remote-tracking branch 'origin/main' into fix-857
juhovainio Nov 6, 2026
58c683f
Merge remote-tracking branch 'origin/main' into fix-857
juhovainio Nov 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -842,7 +842,7 @@ rocm serve <model> [--engine lemonade|vllm|aim]
[--gpu auto|<index>]
[--runtime-id KEY | --env-id ID]
[--host HOST] [--port PORT]
[--verbose] [--foreground | --managed]
[--verbose] [--foreground | --managed] [--wait]
[--no-smoke-test]
[--allow-public-bind] [--require-api-key] [--api-key KEY]
[--temperature TEMP] [--top-p PROB] [--max-tokens N]
Expand Down Expand Up @@ -912,6 +912,17 @@ can:
directly and exits 0; if that signal fails too, it says so on stderr.

`--managed` is the explicit form of the default background behavior.
`--wait` keeps `rocm serve` waiting until the server answers requests or its
engine gives up (vLLM and AIM after 5 minutes by default; raise it with
`ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), instead
of returning `starting` after 45 seconds; if the server exits first (crashes, is killed, or is stopped with
`rocm services stop`), it fails with an error that points at the server's logs
instead of waiting forever. Ctrl-C ends the wait and leaves the server starting
in the background; a generated API key (public bind) is printed when the wait
begins, so interrupting the wait does not lose it. It cannot be combined with `--foreground` or `--verbose`, and
when the model is already being served and still starting, it waits on that
server instead (an already-serving one is reported at once). Watch the download with
`rocm logs --service <id> --follow` from another terminal.
`--no-smoke-test` skips the post-startup inference probe.

Which model form to pass depends on the engine your GPU selects. The Lemonade
Expand Down Expand Up @@ -1082,7 +1093,7 @@ Manage background servers started with `--managed`:

```
rocm services list [--all] [--json]
rocm services logs <service-id>
rocm services logs <service-id> [--follow]
rocm services stop <service-id> --yes
rocm services restart <service-id> --yes
rocm services remove <service-id> --yes
Expand Down Expand Up @@ -1383,13 +1394,23 @@ provider settings are left untouched.
### Logs and cleanup

```
rocm logs [--service <service-id>] [--search TERM ...] [QUERY ...]
rocm logs [--service <service-id> [--follow]] [--search TERM ...] [QUERY ...]
rocm services logs <service-id> [--follow]

rocm uninstall [--yes] [--dry-run]
[--keep-binaries] [--keep-config] [--keep-data] [--keep-cache]
[--force-dev-binaries]
```

`--follow` (`-f`) keeps printing a server's new log output, such as a model
download's progress, until the server stops (a server that ends `failed` makes the command exit
non-zero), and prints `Service <id> is ready.` when it finishes starting. Ctrl-C
ends the follow and leaves the server running. On a terminal, a `Ctrl-C stops following, the server keeps running. Following <id>`
line stays on the row under the newest output. Piped or redirected output keeps
stdout to the log text alone: the hint (once), `is ready.` and `stopped following.`
lines go to stderr. A server that is already `failed` when the follow starts exits
non-zero too, after printing its log.

`rocm uninstall` removes four categories of files that ROCm CLI manages on this
computer: its binaries, configuration, data, and caches. It lists what it will
do and asks `[y/N]` (default: no) before doing it. Use `--dry-run` to only list,
Expand Down
34 changes: 34 additions & 0 deletions apps/rocm/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,15 @@ rocm serve qwen --verbose --device gpu_required")]
/// Keep the server managed by ROCm CLI.
#[arg(long)]
managed: bool,
/// Keep waiting until the server answers requests or its engine gives up
/// (vLLM and AIM after 5 minutes by default, raised with
/// `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`),
/// instead of returning `starting` after 45 seconds.
/// Also waits on a matching server that is already starting. Fails with a
/// pointer to the logs if the server exits first. Cannot be
/// combined with --foreground or --verbose, which already stay attached.
#[arg(long, conflicts_with_all = ["foreground", "verbose"])]
wait: bool,
/// Stream every engine log line in this terminal instead of a deployment summary.
#[arg(long, conflicts_with = "managed")]
verbose: bool,
Expand Down Expand Up @@ -552,11 +561,16 @@ rocm serve qwen --verbose --device gpu_required")]
#[command(after_help = "EXAMPLES:\n \
rocm logs\n \
rocm logs --service <service-id>\n \
rocm logs --service <service-id> --follow\n \
rocm logs --search error timeout")]
Logs {
/// Show logs for a managed service id.
#[arg(long)]
service: Option<String>,
/// Keep printing new output for --service until the server stops (Ctrl-C ends the
/// follow and leaves the server running). Use it to watch a model download.
#[arg(long, short = 'f', requires = "service")]
follow: bool,
/// Search recent logs for one or more words.
#[arg(long, num_args = 1.., value_name = "QUERY")]
search: Vec<String>,
Expand Down Expand Up @@ -1040,6 +1054,10 @@ pub(crate) enum ServicesCommand {
Logs {
/// Service id from `rocm services list`.
service_id: String,
/// Keep printing new output until the server stops (Ctrl-C ends the follow and
/// leaves the server running). Use it to watch a model download.
#[arg(long, short = 'f')]
follow: bool,
},
/// Stop a local model server.
///
Expand Down Expand Up @@ -1909,6 +1927,22 @@ mod tests {
Cli::try_parse_from(argv)
}

#[test]
fn serve_wait_parses_and_conflicts_with_attached_modes() {
let cli = parse_serve(&["--wait"]).expect("--wait parses");
assert!(matches!(
cli.command,
Some(Command::Serve { wait: true, .. })
));
let cli = parse_serve(&[]).expect("default parses");
assert!(matches!(
cli.command,
Some(Command::Serve { wait: false, .. })
));
assert!(parse_serve(&["--wait", "--verbose"]).is_err());
assert!(parse_serve(&["--wait", "--foreground"]).is_err());
}

#[test]
fn serve_parses_verbose_and_no_smoke_test_flags() {
let cli = parse_serve(&["--verbose", "--no-smoke-test"]).expect("flags parse");
Expand Down
2 changes: 1 addition & 1 deletion apps/rocm/src/cli_progress.rs
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ impl Spinner {
/// CJK glyph occupies two terminal columns) to fit within `max_width`
/// columns, appending an ellipsis when it doesn't already fit, so a repaint
/// can never wrap to a second terminal row.
fn truncate_to_width(line: &str, max_width: usize) -> String {
pub(crate) fn truncate_to_width(line: &str, max_width: usize) -> String {
if max_width == 0 {
return String::new();
}
Expand Down
11 changes: 9 additions & 2 deletions apps/rocm/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ use crate::service_control::run_internal_sandbox_tool;
use crate::services_cmd::run_approved_service_action;
use crate::services_cmd::services;
use crate::services_text::{
render_internal_status_text, render_logs_browser_text, render_logs_text,
follow_service_logs, render_internal_status_text, render_logs_browser_text, render_logs_text,
render_service_logs_text,
};
use crate::setup_cmd::{render_setup_status_text, setup};
Expand Down Expand Up @@ -822,6 +822,7 @@ fn dispatch(cli: Cli) -> Result<()> {
port,
foreground,
managed,
wait,
verbose,
no_smoke_test,
allow_public_bind,
Expand All @@ -844,6 +845,7 @@ fn dispatch(cli: Cli) -> Result<()> {
port,
foreground,
managed,
wait,
verbose,
no_smoke_test,
allow_public_bind,
Expand All @@ -863,6 +865,7 @@ fn dispatch(cli: Cli) -> Result<()> {
Some(Command::Config { command }) => config(command),
Some(Command::Logs {
service,
follow,
search,
query,
}) => {
Expand All @@ -878,7 +881,11 @@ fn dispatch(cli: Cli) -> Result<()> {
);
}
if let Some(service_id) = service {
print!("{}", render_service_logs_text(&paths, &service_id)?);
if follow {
follow_service_logs(&paths, &service_id)?;
} else {
print!("{}", render_service_logs_text(&paths, &service_id)?);
}
} else {
let query = if search.is_empty() {
(!query.is_empty()).then(|| query.join(" "))
Expand Down
55 changes: 48 additions & 7 deletions apps/rocm/src/serve_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ use crate::service_launch::{
ensure_background_helper_running_quiet, ensure_public_bind_engine_supported,
managed_service_reuse_refusal, no_usable_gpu_refusal, print_managed_launch_plain,
resolve_endpoint_auth, run_attached_service, start_managed_service, validate_bind_host,
wait_begin_client_config,
};
use crate::{
GatedEngineAction, cli_progress, device_policy_name, ensure_engine_allowed_on_os,
Expand Down Expand Up @@ -482,6 +483,7 @@ pub(crate) struct ServeArgs {
pub(crate) port: u16,
pub(crate) foreground: bool,
pub(crate) managed: bool,
pub(crate) wait: bool,
pub(crate) verbose: bool,
pub(crate) no_smoke_test: bool,
pub(crate) allow_public_bind: bool,
Expand All @@ -507,6 +509,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
port,
foreground,
managed,
wait,
verbose,
no_smoke_test,
allow_public_bind,
Expand Down Expand Up @@ -1113,10 +1116,15 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
}

if background {
let mut spinner =
cli_progress::Spinner::new(format!("Starting {model} on {selected_engine}…"));
spinner.tick();
let report = start_managed_service(
let spinner = std::cell::RefCell::new(cli_progress::Spinner::new(format!(
"Starting {model} on {selected_engine}…"
)));
spinner.borrow_mut().tick();
// The key is shown when the wait begins, since Ctrl-C or a failed wait
// would otherwise leave a running authenticated server whose key was never
// printed. It is then not repeated in the summary below.
let key_shown_before_wait = std::cell::Cell::new(false);
let launched = start_managed_service(
&selected_engine,
&service_id,
&model,
Expand All @@ -1132,8 +1140,41 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
launch_lock,
require_api_key,
gpu_refusal_waived_for_reuse,
&mut |_elapsed| spinner.tick(),
)?;
wait,
// Named with the id the launch resolved to: an already-starting
// server keeps its own, not the one generated above. The spinner
// has no id on it, and a non-TTY stderr gets no spinner at all.
&mut |waiting_on| {
spinner.borrow_mut().clear();
eprintln!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: with --allow-public-bind / --require-api-key the generated key is printed ("shown only now") only after start_managed_service returns. Now that the wait is unbounded, Ctrl-C (which the banner invites) or a failed wait leaves a running authenticated server whose key was never shown, and no command prints it. Printing the client config before the wait (e.g. from this callback) would avoid that.

"Waiting for {waiting_on}; Ctrl-C stops waiting, the server keeps running. \
Follow: rocm logs --service {waiting_on} --follow"
);
if let Some(config) = wait_begin_client_config(
waiting_on,
&service_id,
&host,
port,
endpoint_auth.as_deref(),
) {
key_shown_before_wait.set(true);
// Same intentional one-time key display as
// `print_managed_launch_plain`; see its rationale.
// codeql[rust/cleartext-logging]
print!("{config}");
Comment thread
juhovainio marked this conversation as resolved.
Dismissed
}
},
&mut |_elapsed| spinner.borrow_mut().tick(),
);
// `Spinner` has no `Drop`: clear its row so `Error:` is not glued to it.
let report = match launched {
Ok(report) => report,
Err(error) => {
spinner.borrow_mut().clear();
return Err(error);
}
};
let mut spinner = spinner.into_inner();
// Not when the launch was fabricated. `simulate_oom_managed_launch`
// writes a `starting` record for a process that never existed, so the
// helper would be left watching a pid that will never appear, with
Expand Down Expand Up @@ -1168,7 +1209,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
}
// Safe to move `endpoint_auth` here: this branch always returns, so the
// fall-through (attached) path below never observes it moved.
let launched_key = if report.already_running {
let launched_key = if report.already_running || key_shown_before_wait.get() {
None
} else {
endpoint_auth
Expand Down
2 changes: 1 addition & 1 deletion apps/rocm/src/serve_summary.rs
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String {
};
let _ = writeln!(
out,
" note: {explanation} - check `rocm logs --service {}` or `rocm services list`",
" note: {explanation} - follow `rocm logs --service {} --follow` or check `rocm services list`",
summary.service_id
);
}
Expand Down
Loading
Loading