Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -858,9 +858,10 @@ own process/container (repeatable); currently only `aim` consumes these (e.g.
argv and in `rocm services list --json`, so do not put secrets there; pass a
bare `KEY` (e.g. `--engine-env HF_TOKEN`) to forward the value from your
current environment without storing it. A service restarted by the daemon
after a crash inherits the daemon's environment, not yours, so a forwarded
secret must also be set there. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the
5-minute readiness wait for a cold-start model download.
after a crash (with the `server-recover` automation enabled) inherits the
daemon's environment, not yours, so a forwarded secret must also be set
there. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the 5-minute readiness wait
for a cold-start model download.

`--gpu-memory-utilization` (a fraction above 0.0 and up to 1.0 of each GPU's
total VRAM) and `--tool-call-parser` (for example `hermes`, `llama3_json`, or
Expand Down Expand Up @@ -1240,6 +1241,14 @@ apply changes automatically. `list` shows the available watchers and their IDs.
`server-recover` does: it restarts a managed service that failed. The other
watchers stay review-gated or record only in this mode.

Enabled watchers run in a background helper process (`rocm daemon`), which
`enable` starts, and `rocm serve` starts too while any watcher is on. All
watchers are off by default, so serving a model starts no helper. The helper
exits on its own within a few seconds once no watcher is enabled; disable each
enabled watcher to stop it. `rocm daemon --status` shows whether it is running.
The helper is not tied to a service: while a watcher is enabled it outlives
`rocm services stop`, and that command, like `rocm uninstall`, says so.

### Configuration

Show or change ROCm CLI's saved settings: the default engine and runtime,
Expand Down
2 changes: 1 addition & 1 deletion apps/rocm/src/automations.rs
Original file line number Diff line number Diff line change
Expand Up @@ -436,7 +436,7 @@ pub(crate) fn render_daemon_text(paths: &AppPaths, config: &RocmCliConfig) -> St
let _ = writeln!(output, "Startup: on demand");
let _ = writeln!(
output,
"Why it runs: automation checks and local model servers use it when needed."
"Why it runs: enabled automation checks (server recovery is one) need it; it stops on its own when none is on."
);
let _ = writeln!(
output,
Expand Down
31 changes: 19 additions & 12 deletions apps/rocm/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -694,21 +694,28 @@ rocm logs --search error timeout")]
#[arg(value_name = "QUERY")]
query: Vec<String>,
},
/// Run the background supervisor that powers automations, managed servers, and the dashboard.
/// Run the background supervisor that runs enabled automation watchers.
///
/// The daemon is a multi-role helper that, while running:
/// - executes enabled automation watchers (update checks, driver-plan
/// checks, artifact prefetch) on a 5s tick in a sandboxed subprocess
/// The daemon runs enabled automation watchers, and only while at least one
/// is enabled (all are off by default). While running it:
/// - executes the enabled watchers (update checks, driver-plan checks,
/// artifact prefetch) on a 5s tick in a sandboxed subprocess
/// - health-checks and auto-recovers managed local model servers
/// (Lemonade, vLLM)
/// - collects GPU thermal/VRAM metrics every 60s for the TUI dashboard
/// - listens on a local webhook port for automation events from other
/// `rocm` commands
/// (Lemonade, vLLM) when the `server-recover` watcher is enabled
/// - records GPU metrics roughly every 60s when the `gpu-metrics` or
/// `gpu-thermal-protect` watcher is enabled
/// - opens no webhook listener: that exists only when `rocmd run` is
/// started directly with `--local-webhook-port`, which `rocm daemon`
/// does not pass
///
/// It is normally started on demand by `rocm automations enable` and
/// `rocm serve --managed`, so you rarely need to launch it yourself. Run it
/// directly only when you want to observe its behavior in the foreground,
/// e.g. for debugging.
/// The dashboard does not need it: `rocm dash` has its own embedded daemon.
///
/// It is started on demand by `rocm automations enable`, and by
/// `rocm serve --managed` when an automation watcher is enabled, so you
/// rarely need to launch it yourself. It exits on its own once no watcher
/// is enabled: `rocm automations disable <watcher>` for each enabled one
/// stops it. Run it directly only when you want to observe its behavior in
/// the foreground, e.g. for debugging.
#[command(verbatim_doc_comment)]
Daemon {
/// Print a one-shot snapshot of automation/watcher status, then exit without starting the supervisor loop.
Expand Down
Loading
Loading