From 1eb574a160bd4ad561d4aff31d8f2d441f5d5ddf Mon Sep 17 00:00:00 2001 From: Eugene Volen Date: Tue, 3 Nov 2026 09:42:00 +0200 Subject: [PATCH] ROCMAI-827: start the background helper only while an automation needs it `rocm serve` started the background automation helper (`rocm daemon`) on every launch, and the helper's loop ran until it was signalled. With every automation watcher off, which is the default, the helper has no work at all: crash recovery is the opt-in `server-recover` watcher. So a serve left a detached no-op process behind long after `rocm services stop` stopped its service, and nothing ever stopped it. - The autostart shared by `serve` and `automations enable` now spawns the helper only when an automation watcher is enabled. The predicate, automation_daemon_enabled(), is "any watcher is enabled", which is what the spawn gate, the exit and the user-facing wording all say; the stored daemon_enabled flag alone no longer counts. - The helper's loop exits on its own once no watcher is enabled, which makes disabling the last watcher the way to stop it. It exits by itself, so no process is signalled and no PID identity check is involved. - The exit is decided under the autostart lock: the helper re-reads the config after taking it, and only if still no watcher is enabled publishes "stopped" and clears its own spawn claim before releasing it. The autostart path has no lock-free "already running" read. Together with enable's existing order (save the config, then ensure the helper), an enable that lands while the helper is about to exit either makes it stay or finds it stopped and starts a new one, instead of leaving a watcher enabled with no daemon. - `rocm daemon` help no longer lists GPU metrics and the webhook listener as unconditional work. The `LemonadeServer.exe --silent` seen next to it in the report is a separate Lemonade Server install, started at logon from its own Startup shortcut. rocm-cli neither starts nor stops it. Tests: unit tests on both halves (autostart gate with an injected spawn; loop exit, persistence over several ticks, the exit window reproduced deterministically, and the lock and claim handling), mock-lane scenario automations-11 (a live helper exits after its last watcher is disabled), and GPU-lane scenario serve-40 (serving with no watcher enabled starts no helper). Review round 2: the helper takes the autostart lock with a non-blocking try-lock polled on the async runtime (bounded; it stays up if the lock is stuck) instead of a blocking acquire, and the lock tests wait on observed events instead of fixed sleeps. `rocm services stop` and the `rocm uninstall` plan say when the helper outlives them and how to end it (disable the enabled watchers). `rocmd run` help and status text no longer promise a persistent loop, docs/automations.md and docs/manual-testing.md cover the lifecycle, and the serve scenario is serve-40, the next free number on main. Review round 3: the helper note is derived from both whether a watcher is enabled and whether the helper is running (four cases, exact text pinned), the uninstall suffix is added only where it is true, an unreadable config no longer fails a serve after launch, the idle-exit lock wait is its own select arm so shutdown and webhook dispatch stay live, the serve scenario also checks the autostart claim, and the e2e World stops a helper left behind by a failed step. Signed-off-by: Eugene Volen --- README.md | 15 +- apps/rocm/src/automations.rs | 2 +- apps/rocm/src/main.rs | 31 +- apps/rocm/src/service_launch.rs | 463 ++++++++++++++-- apps/rocm/src/services_cmd.rs | 9 +- apps/rocm/src/services_text.rs | 16 + apps/rocm/src/uninstall_plan.rs | 81 +++ apps/rocmd/src/cli.rs | 5 +- apps/rocmd/src/daemon.rs | 517 +++++++++++++++++- crates/rocm-core/src/config.rs | 9 +- crates/rocm-core/src/process.rs | 26 +- docs/architecture.md | 2 +- docs/automations.md | 5 +- docs/manual-testing.md | 31 ++ docs/testing.md | 30 + .../e2e-cucumber/features/automations.feature | 19 + .../features/model_serving.feature | 13 + tests/e2e-cucumber/tests/e2e.rs | 15 + .../tests/e2e/automations_steps.rs | 126 +++++ tests/e2e-cucumber/tests/e2e/serving_steps.rs | 67 ++- 20 files changed, 1402 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index 87c0c35491..1147f3ed41 100644 --- a/README.md +++ b/README.md @@ -858,9 +858,10 @@ own process/container (repeatable); currently only `aim` consumes these (e.g. argv and in `rocm services list --json`, so do not put secrets there; pass a bare `KEY` (e.g. `--engine-env HF_TOKEN`) to forward the value from your current environment without storing it. A service restarted by the daemon -after a crash inherits the daemon's environment, not yours, so a forwarded -secret must also be set there. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the -5-minute readiness wait for a cold-start model download. +after a crash (with the `server-recover` automation enabled) inherits the +daemon's environment, not yours, so a forwarded secret must also be set +there. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the 5-minute readiness wait +for a cold-start model download. `--gpu-memory-utilization` (a fraction above 0.0 and up to 1.0 of each GPU's total VRAM) and `--tool-call-parser` (for example `hermes`, `llama3_json`, or @@ -1240,6 +1241,14 @@ apply changes automatically. `list` shows the available watchers and their IDs. `server-recover` does: it restarts a managed service that failed. The other watchers stay review-gated or record only in this mode. +Enabled watchers run in a background helper process (`rocm daemon`), which +`enable` starts, and `rocm serve` starts too while any watcher is on. All +watchers are off by default, so serving a model starts no helper. The helper +exits on its own within a few seconds once no watcher is enabled; disable each +enabled watcher to stop it. `rocm daemon --status` shows whether it is running. +The helper is not tied to a service: while a watcher is enabled it outlives +`rocm services stop`, and that command, like `rocm uninstall`, says so. + ### Configuration Show or change ROCm CLI's saved settings: the default engine and runtime, diff --git a/apps/rocm/src/automations.rs b/apps/rocm/src/automations.rs index 13cd50d1eb..bbed155f45 100644 --- a/apps/rocm/src/automations.rs +++ b/apps/rocm/src/automations.rs @@ -436,7 +436,7 @@ pub(crate) fn render_daemon_text(paths: &AppPaths, config: &RocmCliConfig) -> St let _ = writeln!(output, "Startup: on demand"); let _ = writeln!( output, - "Why it runs: automation checks and local model servers use it when needed." + "Why it runs: enabled automation checks (server recovery is one) need it; it stops on its own when none is on." ); let _ = writeln!( output, diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index b153d74d4b..14418f0935 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -694,21 +694,28 @@ rocm logs --search error timeout")] #[arg(value_name = "QUERY")] query: Vec, }, - /// Run the background supervisor that powers automations, managed servers, and the dashboard. + /// Run the background supervisor that runs enabled automation watchers. /// - /// The daemon is a multi-role helper that, while running: - /// - executes enabled automation watchers (update checks, driver-plan - /// checks, artifact prefetch) on a 5s tick in a sandboxed subprocess + /// The daemon runs enabled automation watchers, and only while at least one + /// is enabled (all are off by default). While running it: + /// - executes the enabled watchers (update checks, driver-plan checks, + /// artifact prefetch) on a 5s tick in a sandboxed subprocess /// - health-checks and auto-recovers managed local model servers - /// (Lemonade, vLLM) - /// - collects GPU thermal/VRAM metrics every 60s for the TUI dashboard - /// - listens on a local webhook port for automation events from other - /// `rocm` commands + /// (Lemonade, vLLM) when the `server-recover` watcher is enabled + /// - records GPU metrics roughly every 60s when the `gpu-metrics` or + /// `gpu-thermal-protect` watcher is enabled + /// - opens no webhook listener: that exists only when `rocmd run` is + /// started directly with `--local-webhook-port`, which `rocm daemon` + /// does not pass /// - /// It is normally started on demand by `rocm automations enable` and - /// `rocm serve --managed`, so you rarely need to launch it yourself. Run it - /// directly only when you want to observe its behavior in the foreground, - /// e.g. for debugging. + /// The dashboard does not need it: `rocm dash` has its own embedded daemon. + /// + /// It is started on demand by `rocm automations enable`, and by + /// `rocm serve --managed` when an automation watcher is enabled, so you + /// rarely need to launch it yourself. It exits on its own once no watcher + /// is enabled: `rocm automations disable ` for each enabled one + /// stops it. Run it directly only when you want to observe its behavior in + /// the foreground, e.g. for debugging. #[command(verbatim_doc_comment)] Daemon { /// Print a one-shot snapshot of automation/watcher status, then exit without starting the supervisor loop. diff --git a/apps/rocm/src/service_launch.rs b/apps/rocm/src/service_launch.rs index b7acc45c66..57d405e5d2 100644 --- a/apps/rocm/src/service_launch.rs +++ b/apps/rocm/src/service_launch.rs @@ -58,7 +58,7 @@ use crate::managed_service_status::existing_live_managed_service; use crate::service_hints; #[cfg(not(windows))] use crate::services_text::read_optional_tail_lines; -use crate::services_text::{render_service_action_result, run_internal_sandbox_tool}; +use crate::services_text::run_internal_sandbox_tool; use crate::{SandboxToolArg, device_policy_name, record_cli_audit_event, status_for_readiness}; pub(crate) fn validate_bind_host(host: &str, allow_public_bind: bool) -> Result<()> { @@ -837,12 +837,77 @@ fn background_helper_already_running(paths: &AppPaths) -> Result { .is_some_and(|state| state.running && rocm_core::process_is_running(state.daemon_pid))) } +/// What the user should be told about the background automation helper +/// (`rocm daemon`) after stopping a service or when planning an uninstall. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct HelperNote { + /// The sentence to print. + pub(crate) text: String, + /// The helper is running right now. + pub(crate) running: bool, + /// An automation watcher is enabled, so the helper stays up on its own. + pub(crate) outlives: bool, +} + +/// The note for one combination of the two facts the helper itself acts on: +/// whether a watcher is enabled +/// ([`rocm_core::RocmCliConfig::automation_daemon_enabled`] gates both the spawn +/// and the helper's exit) and whether it is running. `None` for neither. +fn helper_note_for(enabled: &[&str], running: bool) -> Option { + let text = match (enabled.is_empty(), running) { + (true, false) => return None, + (true, true) => "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds".to_owned(), + (false, true) => { + let disable = enabled + .iter() + .map(|id| format!("`rocm automations disable {id}`")) + .collect::>() + .join(", "); + format!( + "the background automation helper (`rocm daemon`) keeps running while an automation is enabled ({}); to stop it, run {disable}, after which it exits on its own within a few seconds", + enabled.join(", ") + ) + } + (false, false) => format!( + "an automation is enabled ({}) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it", + enabled.join(", ") + ), + }; + Some(HelperNote { + text, + running, + outlives: !enabled.is_empty() && running, + }) +} + +/// [`helper_note_for`] over the facts on disk. An unreadable config counts as +/// "no watcher enabled". +pub(crate) fn background_helper_note(paths: &AppPaths) -> Option { + let config = rocm_core::RocmCliConfig::load(paths).unwrap_or_default(); + let enabled: Vec<&str> = config + .automations + .watchers + .iter() + .filter(|(_, watcher)| watcher.enabled) + .map(|(id, _)| id.as_str()) + .collect(); + let running = background_helper_already_running(paths).unwrap_or(false); + helper_note_for(&enabled, running) +} + /// Shared daemon-lifecycle entrypoint: ensures the background automation helper -/// (`rocm daemon`) is running, spawning it detached if not. Liveness is read from -/// the file-based automation runtime state. Intentionally `pub(crate)` — reused by -/// both the `serve --managed` path and `automations enable`. Only the spawn result -/// itself (`command.spawn()` / `spawn_detached_no_inherit`) is logged rather than -/// propagated; setup errors (path discovery, stdio attach) still return `Err`. +/// (`rocm daemon`) is running when an automation watcher needs it, spawning it +/// detached if not. Liveness is read from the file-based automation runtime +/// state. Intentionally `pub(crate)` — reused by both the `serve --managed` path +/// and `automations enable`. Only the spawn result itself (`command.spawn()` / +/// `spawn_detached_no_inherit`) is logged rather than propagated; setup errors +/// (path discovery, config load, stdio attach) still return `Err`. +/// +/// The daemon's only work is running enabled automation watchers (server +/// recovery included — `server-recover` is one of them and is off by default), +/// and it exits on its own once none is enabled. So with no watcher enabled +/// nothing is spawned: a `serve` would otherwise leave a no-op daemon running +/// long after its service was stopped (ROCMAI-827). pub(crate) fn ensure_background_helper_running() -> Result<()> { ensure_background_helper_running_quiet(false) } @@ -852,18 +917,52 @@ pub(crate) fn ensure_background_helper_running() -> Result<()> { /// the daemon-spawn note does not appear above the deployment summary table. pub(crate) fn ensure_background_helper_running_quiet(quiet: bool) -> Result<()> { let paths = AppPaths::discover()?; - if background_helper_already_running(&paths)? { + ensure_background_helper_running_with(&paths, quiet, spawn_background_helper, &|| {}) +} + +/// The body of [`ensure_background_helper_running_quiet`], with the actual +/// process spawn injected so the decision around it is testable without +/// launching a daemon. `spawn` returns `Err` for a setup failure (propagated) +/// and `Ok(Err(..))` for a failed spawn (reported, not fatal). `on_lock_busy` is +/// called each time the autostart lock is found held by another process, which +/// lets a test wait for this call to be genuinely queued behind it. +fn ensure_background_helper_running_with( + paths: &AppPaths, + quiet: bool, + spawn: impl FnOnce(&AppPaths) -> Result>, + on_lock_busy: &dyn Fn(), +) -> Result<()> { + // An unreadable config is "no watcher": this also runs after `serve` has + // already launched its child, where failing would report a launch that + // happened as an error. + if !rocm_core::RocmCliConfig::load(paths).is_ok_and(|config| config.automation_daemon_enabled()) + { return Ok(()); } - // The check above and the spawn below are a TOCTOU window: two concurrent - // callers (e.g. two `rocm serve`) can both read "not running" and each spawn - // a daemon. Serialize the decision on a lock file and re-check under it — the - // first holder spawns, later holders observe the now-running daemon and - // return without spawning. The unlocked pre-check above keeps the common - // already-running case lock-free. - let _autostart_lock = rocm_core::FileLock::acquire(paths.automation_autostart_lock_path())?; - if background_helper_already_running(&paths)? { + // Two concurrent callers (e.g. two `rocm serve`) can both read "not running" + // and each spawn a daemon. Serialize the decision on a lock file — the first + // holder spawns, later holders observe the now-running daemon and return + // without spawning. + // + // The "already running" read is deliberately made only under the lock, with + // no lock-free fast path. The daemon exits by itself once the last watcher + // is disabled, and it decides that under this same lock and publishes + // `running = false` before releasing it. Caller order is: save the enabling + // config, then come here. A lock-free read could still see the daemon that is + // about to exit as running, spawn nothing, and leave a watcher enabled with + // no daemon; under the lock the daemon has either seen the new config and + // stays, or has already published that it stopped. + let _autostart_lock = loop { + if let Some(lock) = + rocm_core::FileLock::try_acquire(paths.automation_autostart_lock_path())? + { + break lock; + } + on_lock_busy(); + std::thread::sleep(std::time::Duration::from_millis(10)); + }; + if background_helper_already_running(paths)? { return Ok(()); } @@ -883,25 +982,7 @@ pub(crate) fn ensure_background_helper_running_quiet(quiet: bool) -> Result<()> return Ok(()); } - let exe = managed_service_launcher_path() - .context("failed to resolve current rocm executable path")?; - let args = vec!["daemon".to_owned()]; - #[cfg(windows)] - let spawn_result = { - let env_values = app_path_env_var_values(&paths, None); - let env_refs = app_path_env_var_refs(&env_values); - rocm_core::spawn_detached_no_inherit(&exe, &args, &env_refs) - }; - #[cfg(not(windows))] - let spawn_result = { - let mut command = managed_service_process_command(&exe, &args); - command.stdin(Stdio::null()); - attach_background_stdio(&mut command, None)?; - detach_background_command(&mut command); - apply_app_path_env(&mut command, &paths); - command.spawn().map(|child| child.id()) - }; - match spawn_result { + match spawn(paths)? { Ok(daemon_pid) => { // Record the claim before returning (and thus releasing the lock) so a // concurrent holder in the spawn→publish window defers. Best-effort: a @@ -925,6 +1006,34 @@ pub(crate) fn ensure_background_helper_running_quiet(quiet: bool) -> Result<()> Ok(()) } +/// Spawn `rocm daemon` detached from this process. The outer `Result` is a setup +/// failure (resolving the executable, attaching stdio); the inner one is the +/// spawn itself. +fn spawn_background_helper(paths: &AppPaths) -> Result> { + let exe = managed_service_launcher_path() + .context("failed to resolve current rocm executable path")?; + let args = vec!["daemon".to_owned()]; + #[cfg(windows)] + let spawn_result = { + let env_values = app_path_env_var_values(paths, None); + let env_refs = app_path_env_var_refs(&env_values); + rocm_core::spawn_detached_no_inherit(&exe, &args, &env_refs) + }; + #[cfg(not(windows))] + let spawn_result = { + let mut command = managed_service_process_command(&exe, &args); + command.stdin(Stdio::null()); + attach_background_stdio(&mut command, None)?; + detach_background_command(&mut command); + apply_app_path_env(&mut command, paths); + command + .spawn() + .map(|child| child.id()) + .map_err(anyhow::Error::from) + }; + Ok(spawn_result) +} + /// How long an autostart claim is honoured before it is treated as stale even if /// its recorded PID is still alive. Comfortably longer than a cold daemon boot /// (clap parse → runtime build → config load → state publish) yet short enough @@ -1066,12 +1175,17 @@ pub(crate) fn run_attached_service( } }; - // The child is a managed service that outlives this session once detached, so - // it needs the same supervision the background path gives it: the daemon - // health-checks and auto-recovers managed servers, reconciles a self-exited - // server's record, and feeds the dashboard. Match the background ordering - // (spawn, then ensure the helper) and keep it quiet so no status line breaks - // into the log stream. + // The child is a managed service that outlives this session once detached. + // The background helper adds one thing for it: with the `server-recover` + // watcher enabled, it health-checks managed servers and restarts failed ones. + // Every duty of the helper is watcher-gated, so with no watcher enabled no + // helper is started. Nothing else depends on it: a self-exited server's + // record is settled by its own supervisor (`supervise_service` writes the + // exit via `record_supervised_exit` in `apps/rocmd/src/supervise.rs`) and on + // every read (`refresh_managed_service_runtime_liveness`), and the dashboard + // has its own embedded daemon. Match the background ordering (spawn, then + // ensure the helper) and keep it quiet so no status line breaks into the + // log stream. ensure_background_helper_running_quiet(true)?; // The resolution detail (model, engine, runtime, GPU, warnings) was already @@ -1118,7 +1232,14 @@ pub(crate) fn run_attached_service( Some(service_id.clone()), true, ) { - Ok(result) => print!("{}", render_service_action_result("stop_server", &result)), + Ok(result) => print!( + "{}", + crate::services_text::render_service_action_result_with_helper_note( + &paths, + "stop_server", + &result + ) + ), Err(error) => { // Best-effort direct signal so Ctrl-C never leaves the model // orphaned when the sandbox stop path fails. @@ -1930,6 +2051,266 @@ mod tests { let _ = fs::remove_dir_all(&root); } + /// ROCMAI-827: `rocm serve` started the background daemon on every launch, + /// and with no automation watcher enabled that daemon had nothing to do yet + /// kept running after every service was stopped. With the default config + /// (no watcher enabled) the helper must not be spawned at all. + #[test] + fn background_helper_not_spawned_when_no_automation_is_enabled() { + let (root, paths) = test_paths("helper-no-automation"); + paths.ensure().expect("create app dirs"); + // The stored flag alone is not "a watcher is enabled": nothing would run. + let mut config = rocm_core::RocmCliConfig::default(); + config.automations.daemon_enabled = true; + config.save(&paths).expect("save default config"); + + let mut spawned = false; + ensure_background_helper_running_with( + &paths, + true, + |_| { + spawned = true; + Ok(Ok(4242)) + }, + &|| {}, + ) + .expect("ensure helper"); + + let claim_written = paths.automation_autostart_claim_path().exists(); + let _ = fs::remove_dir_all(&root); + assert!( + !spawned, + "no watcher enabled ⇒ the daemon has no work and must not be started" + ); + assert!(!claim_written, "no spawn ⇒ no autostart claim"); + } + + /// The other half of the gate: once a watcher is enabled the helper is still + /// started (exactly once) and its autostart claim recorded. + #[test] + fn background_helper_spawned_when_an_automation_is_enabled() { + let (root, paths) = test_paths("helper-automation-enabled"); + paths.ensure().expect("create app dirs"); + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = true; + config.save(&paths).expect("save config"); + + let mut spawns = 0; + ensure_background_helper_running_with( + &paths, + true, + |_| { + spawns += 1; + Ok(Ok(4242)) + }, + &|| {}, + ) + .expect("ensure helper"); + + let claim = read_autostart_claim(&paths.automation_autostart_claim_path()); + let _ = fs::remove_dir_all(&root); + assert_eq!(spawns, 1, "an enabled watcher needs the daemon"); + assert_eq!(claim.map(|claim| claim.daemon_pid), Some(4242)); + } + + /// ROCMAI-827 race, enable side: the daemon that is about to exit still has + /// `running = true` and a live pid in its state when `automations enable` + /// (config already saved) asks for a helper. It decides under the autostart + /// lock and publishes "stopped" before releasing it, so the enable must wait + /// on that lock and then start a new helper. A lock-free "already running" + /// read would return here without spawning, leaving a watcher with no daemon. + #[test] + fn background_helper_is_started_when_the_exiting_daemon_publishes_stopped() { + let (root, paths) = test_paths("helper-exit-window"); + paths.ensure().expect("create app dirs"); + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = true; + config.save(&paths).expect("save config"); + // The old daemon: alive (this process stands in) and still `running`. + runtime_state(true, std::process::id()) + .write(&paths) + .expect("write runtime state"); + + // The exiting daemon holds the autostart lock while it decides. + let exiting_daemon_lock = + rocm_core::FileLock::acquire(paths.automation_autostart_lock_path()) + .expect("hold the lock"); + let enable_paths = paths.clone(); + let queued_flag = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let queued = queued_flag.clone(); + let enable = std::thread::spawn(move || { + let mut spawns = 0; + ensure_background_helper_running_with( + &enable_paths, + true, + |_| { + spawns += 1; + Ok(Ok(4242)) + }, + &|| queued.store(true, std::sync::atomic::Ordering::SeqCst), + ) + .expect("ensure helper"); + spawns + }); + // Wait until the enable path has actually found the lock held and queued + // behind it, rather than assuming how long that takes. + let deadline = std::time::Instant::now() + std::time::Duration::from_mins(1); + // A caller that skipped the lock finishes without ever queuing. + while !queued_flag.load(std::sync::atomic::Ordering::SeqCst) && !enable.is_finished() { + assert!(std::time::Instant::now() < deadline, "enable never queued"); + std::thread::yield_now(); + } + runtime_state(false, std::process::id()) + .write(&paths) + .expect("daemon publishes stopped"); + drop(exiting_daemon_lock); + + let spawns = enable.join().expect("enable thread"); + let _ = fs::remove_dir_all(&root); + assert_eq!( + spawns, 1, + "the enable must start a helper once the old one has published it stopped" + ); + } + + fn stop_result() -> serde_json::Value { + serde_json::json!({"output": {"result": { + "service": {"service_id": "svc-a", "status": "stopped"}, + "signaled_pids": [1] + }}}) + } + + /// Put the on-disk facts in one of the four combinations: whether a watcher + /// is enabled, and whether the helper is running (this process stands in for + /// a live daemon). + fn set_helper_facts(paths: &AppPaths, watcher_enabled: bool, helper_running: bool) { + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = watcher_enabled; + config.save(paths).expect("save config"); + runtime_state(helper_running, std::process::id()) + .write(paths) + .expect("write runtime state"); + } + + /// ROCMAI-827: the helper note is derived from both facts, so it cannot + /// disagree with the state it describes. Each combination: the exact wording, + /// and the running/outlives flags the uninstall plan acts on. + #[test] + fn helper_note_follows_watcher_enabled_and_helper_running() { + let (root, paths) = test_paths("helper-note-combos"); + paths.ensure().expect("create app dirs"); + + set_helper_facts(&paths, false, false); + let neither = background_helper_note(&paths); + set_helper_facts(&paths, false, true); + let idle_running = background_helper_note(&paths).expect("note"); + set_helper_facts(&paths, true, false); + let enabled_dead = background_helper_note(&paths).expect("note"); + set_helper_facts(&paths, true, true); + let enabled_running = background_helper_note(&paths).expect("note"); + let _ = fs::remove_dir_all(&root); + + assert_eq!(neither, None); + assert_eq!( + idle_running.text, + "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds" + ); + assert!(idle_running.running && !idle_running.outlives); + assert_eq!( + enabled_dead.text, + "an automation is enabled (server-recover) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it" + ); + assert!(!enabled_dead.running && !enabled_dead.outlives); + assert_eq!( + enabled_running.text, + "the background automation helper (`rocm daemon`) keeps running while an automation is enabled (server-recover); to stop it, run `rocm automations disable server-recover`, after which it exits on its own within a few seconds" + ); + assert!(enabled_running.running && enabled_running.outlives); + } + + /// `rocm services stop` prints the note for a stop only, and only the variant + /// that matches the state. + #[test] + fn stop_output_carries_the_helper_note_that_matches_the_state() { + use crate::services_text::render_service_action_result_with_helper_note as render; + + let (root, paths) = test_paths("stop-helper-note"); + paths.ensure().expect("create app dirs"); + set_helper_facts(&paths, false, false); + let quiet = render(&paths, "stop_server", &stop_result()); + set_helper_facts(&paths, true, true); + let outlives = render(&paths, "stop_server", &stop_result()); + let restart = render(&paths, "restart_server", &stop_result()); + set_helper_facts(&paths, true, false); + let not_running = render(&paths, "stop_server", &stop_result()); + let _ = fs::remove_dir_all(&root); + + assert!(!quiet.contains("background automation helper"), "{quiet}"); + assert!( + outlives.contains("stopped processes: 1") + && outlives.contains("keeps running while an automation is enabled") + && outlives.contains("`rocm automations disable server-recover`"), + "{outlives}" + ); + assert!( + not_running.contains("is not running") && !not_running.contains("keeps running"), + "{not_running}" + ); + assert!( + !restart.contains("background automation helper"), + "{restart}" + ); + } + + /// The command the note names must exist and accept exactly those arguments, + /// and its effect (the helper exiting once the last watcher is off) is + /// `automations-11` in `features/automations.feature` plus the rocmd + /// `automation_loop_` tests. + #[test] + fn the_command_named_by_the_helper_note_parses() { + use clap::Parser as _; + + let (root, paths) = test_paths("helper-note-command"); + paths.ensure().expect("create app dirs"); + set_helper_facts(&paths, true, true); + let note = background_helper_note(&paths).expect("a note"); + let _ = fs::remove_dir_all(&root); + + let command = note + .text + .split('`') + .find(|part| part.starts_with("rocm automations disable")) + .expect("the note names the disable command"); + let argv: Vec<&str> = command.split_whitespace().collect(); + assert!(crate::Cli::try_parse_from(argv).is_ok(), "{command}"); + } + + /// `serve` calls the helper autostart after its child is already running. An + /// unreadable config must not turn that launch into an error: it reads as "no + /// watcher enabled" and nothing is spawned. + #[test] + fn background_helper_autostart_treats_an_unreadable_config_as_no_watcher() { + let (root, paths) = test_paths("helper-bad-config"); + paths.ensure().expect("create app dirs"); + fs::write(paths.config_path(), b"{ this is not json").expect("write bad config"); + assert!(rocm_core::RocmCliConfig::load(&paths).is_err()); + + let mut spawned = false; + let result = ensure_background_helper_running_with( + &paths, + true, + |_| { + spawned = true; + Ok(Ok(4242)) + }, + &|| {}, + ); + let _ = fs::remove_dir_all(&root); + + assert!(result.is_ok(), "{result:?}"); + assert!(!spawned); + } + #[test] fn autostart_spawn_in_flight_defers_only_for_a_live_recent_claim() { let claim = AutostartClaim { diff --git a/apps/rocm/src/services_cmd.rs b/apps/rocm/src/services_cmd.rs index 9a8fd95579..1481cd6c2a 100644 --- a/apps/rocm/src/services_cmd.rs +++ b/apps/rocm/src/services_cmd.rs @@ -36,7 +36,7 @@ use crate::service_records::{ prune_managed_service_records, remove_managed_service_record, service_prune_min_age_hours, }; use crate::services_text::{ - render_service_action_result, render_service_logs_text, render_services_json, + render_service_action_result_with_helper_note, render_service_logs_text, render_services_json, render_services_text, run_internal_sandbox_tool, validate_service_id, }; @@ -134,7 +134,10 @@ pub(crate) fn run_approved_service_action( } let sandbox_tool = sandbox_tool_arg_from_service_tool(tool)?; let result = run_internal_sandbox_tool(paths, sandbox_tool, Some(service_id.to_owned()), true)?; - print!("{}", render_service_action_result(tool, &result)); + print!( + "{}", + render_service_action_result_with_helper_note(paths, tool, &result) + ); record_cli_audit_event( paths, "service", @@ -163,7 +166,7 @@ mod tests { assert!(error.contains("requires --yes")); assert!(error.contains("rocm services stop svc-qwen --yes")); - let rendered = render_service_action_result( + let rendered = crate::services_text::render_service_action_result( "stop_server", &serde_json::json!({ "output": { diff --git a/apps/rocm/src/services_text.rs b/apps/rocm/src/services_text.rs index 81f87d3f00..21284cb006 100644 --- a/apps/rocm/src/services_text.rs +++ b/apps/rocm/src/services_text.rs @@ -534,6 +534,22 @@ fn render_service_logs_text_with_options( Ok(output) } +/// [`render_service_action_result`], plus — for a stop — the note about the +/// background automation helper when it outlives the stopped service. +pub(crate) fn render_service_action_result_with_helper_note( + paths: &AppPaths, + tool: &str, + value: &serde_json::Value, +) -> String { + let mut text = render_service_action_result(tool, value); + if tool == "stop_server" + && let Some(note) = crate::service_launch::background_helper_note(paths) + { + let _ = writeln!(text, " note: {}", note.text); + } + text +} + pub(crate) fn render_service_action_result(tool: &str, value: &serde_json::Value) -> String { let output = value.get("output").unwrap_or(value); let action = service_action_past_tense(tool); diff --git a/apps/rocm/src/uninstall_plan.rs b/apps/rocm/src/uninstall_plan.rs index 266185b235..e450fde189 100644 --- a/apps/rocm/src/uninstall_plan.rs +++ b/apps/rocm/src/uninstall_plan.rs @@ -134,6 +134,19 @@ pub(crate) fn build_uninstall_plan( )); } + // Only a helper that is running can be left behind. "Uninstall does not stop + // it" is said only where that is the whole story: with a watcher enabled the + // helper stays up; an idle one is already on its way out. + if let Some(note) = crate::service_launch::background_helper_note(paths) + && note.running + { + plan.warnings.push(if note.outlives { + format!("{}; uninstall does not stop it", note.text) + } else { + note.text + }); + } + // Remote sessions are worse than local ones to drop silently. The model runs // on someone else's machine and its endpoint is published there, so removing // the record here does not stop either — it only destroys the last thing that @@ -650,4 +663,72 @@ mod tests { ); Ok(()) } + + fn set_helper_facts(paths: &AppPaths, watcher_enabled: bool, helper_running: bool) { + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = watcher_enabled; + config.save(paths).expect("save config"); + rocm_core::AutomationRuntimeState { + running: helper_running, + automations_enabled: true, + daemon_pid: std::process::id(), + started_at_unix_ms: 1, + last_tick_unix_ms: 1, + local_webhook_endpoint: None, + active_watchers: Vec::new(), + } + .write(paths) + .expect("write runtime state"); + } + + /// ROCMAI-827: uninstall does not stop the automation helper, and the plan + /// says so only where it is true. The exact warning for each state: a helper + /// that stays up (a watcher is enabled) is named with the command that ends it + /// and "uninstall does not stop it"; one that is idle and about to exit says + /// only that, since uninstall leaving it is not the whole story; none running + /// is not mentioned. + #[test] + fn plan_warns_about_a_helper_uninstall_leaves_running_only_when_there_is_one() { + let (root, paths) = test_paths("uninstall-helper-warning"); + paths.ensure().expect("create app dirs"); + let options = UninstallOptions { + keep_binaries: true, + ..UninstallOptions::default() + }; + let helper_warnings = |plan: &UninstallPlan| -> Vec { + plan.warnings + .iter() + .filter(|w| w.contains("background automation helper")) + .cloned() + .collect() + }; + + set_helper_facts(&paths, false, false); + let none = helper_warnings(&build_uninstall_plan(&paths, &options).expect("plan")); + set_helper_facts(&paths, true, false); + let enabled_dead = helper_warnings(&build_uninstall_plan(&paths, &options).expect("plan")); + set_helper_facts(&paths, false, true); + let idle = helper_warnings(&build_uninstall_plan(&paths, &options).expect("plan")); + set_helper_facts(&paths, true, true); + let outlives = helper_warnings(&build_uninstall_plan(&paths, &options).expect("plan")); + let _ = std::fs::remove_dir_all(&root); + + assert!(none.is_empty(), "{none:?}"); + assert!( + enabled_dead.is_empty(), + "nothing is running to leave: {enabled_dead:?}" + ); + assert_eq!( + idle, + [ + "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds" + ] + ); + assert_eq!( + outlives, + [ + "the background automation helper (`rocm daemon`) keeps running while an automation is enabled (server-recover); to stop it, run `rocm automations disable server-recover`, after which it exits on its own within a few seconds; uninstall does not stop it" + ] + ); + } } diff --git a/apps/rocmd/src/cli.rs b/apps/rocmd/src/cli.rs index d0f4aa9fea..dac75e120c 100644 --- a/apps/rocmd/src/cli.rs +++ b/apps/rocmd/src/cli.rs @@ -21,7 +21,10 @@ pub(crate) struct Cli { #[derive(Subcommand, Debug)] pub(crate) enum Command { Run { - #[arg(long, help = "Enable the persistent watcher loop.")] + #[arg( + long, + help = "Run the watcher loop. It exits on its own once no automation watcher is enabled." + )] automations_enabled: bool, #[arg( long, diff --git a/apps/rocmd/src/daemon.rs b/apps/rocmd/src/daemon.rs index 9369b8f487..22571dd7e5 100644 --- a/apps/rocmd/src/daemon.rs +++ b/apps/rocmd/src/daemon.rs @@ -7,16 +7,78 @@ use rocm_core::{ AppPaths, AutomationRuntimeState, RocmCliConfig, WatcherRuntimeSnapshot, builtin_watchers, unix_time_millis, }; +use std::fs; use std::io::{self, Write}; use std::time::Duration; use tokio::time::{self, MissedTickBehavior}; const WATCHER_TICK_INTERVAL: Duration = Duration::from_secs(5); +/// A pending [`acquire_autostart_lock`]. +type AutostartLockWait<'a> = std::pin::Pin< + Box>> + Send + 'a>, +>; + +/// Pause between non-blocking attempts at the autostart lock. +const AUTOSTART_LOCK_POLL: Duration = Duration::from_millis(10); +/// The autostart lock is held only for a config read and a spawn, so a wait this +/// long means the holder is stuck; the exit attempt then gives up and the loop +/// stays up. +const AUTOSTART_LOCK_WAIT: Duration = Duration::from_secs(30); + +/// Seams for the automation loop's tests: the tick period, and two callbacks +/// that let a test act at an exact point of a tick instead of racing a real +/// 5-second clock. Production passes [`LoopHooks::default`]. +#[derive(Clone, Copy)] +pub(crate) struct LoopHooks<'a> { + pub(crate) tick_interval: Duration, + /// Called at the start of every tick, before the config is read. + pub(crate) on_tick: Option<&'a (dyn Fn() + Sync)>, + /// Called when a tick has read a config with no watcher enabled, before the + /// exit decision is confirmed: the window an `automations enable` can land in. + pub(crate) on_idle_observed: Option<&'a (dyn Fn() + Sync)>, + /// Called each time the autostart lock is found held by someone else, so a + /// test can wait for the loop to be genuinely waiting on it. + pub(crate) on_lock_busy: Option<&'a (dyn Fn() + Sync)>, + /// How long one exit attempt waits for the autostart lock before giving up + /// (the loop stays up and decides again on a later tick). The wait runs as its + /// own select arm, so it never delays webhook dispatch or shutdown. + pub(crate) lock_wait: Duration, + /// Stands in for Ctrl-C / SIGTERM, which a test cannot raise. + pub(crate) stop: Option<&'a tokio::sync::Notify>, +} + +impl Default for LoopHooks<'_> { + fn default() -> Self { + Self { + tick_interval: WATCHER_TICK_INTERVAL, + on_tick: None, + on_idle_observed: None, + on_lock_busy: None, + lock_wait: AUTOSTART_LOCK_WAIT, + stop: None, + } + } +} pub(crate) async fn run_daemon( paths: &AppPaths, automations_enabled: bool, local_webhook_port: Option, +) -> Result<()> { + run_daemon_with( + paths, + automations_enabled, + local_webhook_port, + LoopHooks::default(), + ) + .await +} + +pub(crate) async fn run_daemon_with( + paths: &AppPaths, + automations_enabled: bool, + local_webhook_port: Option, + hooks: LoopHooks<'_>, ) -> Result<()> { if local_webhook_port.is_some() && !automations_enabled { bail!("local webhook source requires --automations-enabled"); @@ -45,7 +107,7 @@ pub(crate) async fn run_daemon( println!( " lifecycle: {}", if automations_enabled { - "persistent" + "while an automation watcher is enabled" } else { "on-demand" } @@ -71,7 +133,7 @@ pub(crate) async fn run_daemon( if !automations_enabled { println!( - " note: rerun with --automations-enabled to keep rocmd alive for watcher execution" + " note: rerun with --automations-enabled to run the watcher loop (it exits once no watcher is enabled)" ); return Ok(()); } @@ -93,21 +155,81 @@ pub(crate) async fn run_daemon( state.last_tick_unix_ms = unix_time_millis(); state.write(paths)?; - let shutdown = shutdown_signal(); + let shutdown = async { + match hooks.stop { + Some(stop) => stop.notified().await, + None => shutdown_signal().await, + } + }; tokio::pin!(shutdown); - let mut ticker = time::interval(WATCHER_TICK_INTERVAL); + // An idle tick's pending wait for the autostart lock; see its select arm. + let mut exit_attempt: Option> = None; + let mut ticker = time::interval(hooks.tick_interval); ticker.set_missed_tick_behavior(MissedTickBehavior::Delay); loop { tokio::select! { _ = ticker.tick() => { + if let Some(on_tick) = hooks.on_tick { + on_tick(); + } let config = RocmCliConfig::load(paths)?; + // Persistent only while a watcher is enabled: once none is (none + // was, or the last one was disabled) there is no work left, so + // exit rather than idle forever as a detached orphan + // (ROCMAI-827). This read is only a hint. `automations enable` + // saves its config and then asks the autostart path whether a + // daemon is running, so a watcher enabled after it would find + // this process still "running" and spawn nothing. The decision + // is therefore confirmed under the autostart lock (the + // `exit_attempt` arm below), and "stopped" is published before + // the lock is released: an enable either lands first (the + // confirmation sees it and the loop stays) or finds the helper + // stopped and starts a fresh one. + if config.automation_daemon_enabled() { + // Work again: abandon any exit that was waiting for the lock. + exit_attempt = None; + } else if exit_attempt.is_none() { + if let Some(on_idle_observed) = hooks.on_idle_observed { + on_idle_observed(); + } + exit_attempt = Some(Box::pin(acquire_autostart_lock(paths, &hooks))); + continue; + } else { + // Still waiting for the lock from an earlier tick. + continue; + } crate::watchers::reconcile_watcher_snapshots(&config, &mut state); crate::watchers::evaluate_watchers(paths, &config, &mut state)?; state.last_tick_unix_ms = unix_time_millis(); state.write(paths)?; } + // The wait for the autostart lock is its own arm, not an await inside + // the tick arm: while another process holds the lock (up to + // `hooks.lock_wait`) the webhook and shutdown arms below stay + // pollable. It is polled with try-lock, so no thread is parked either. + attempt = async { exit_attempt.as_mut().expect("guarded by the arm condition").await }, if exit_attempt.is_some() => { + exit_attempt = None; + // `None`: the lock stayed held for the whole bounded wait. Stay + // up; the next tick decides again. + if let Some(_autostart_lock) = attempt? { + let config = RocmCliConfig::load(paths)?; + if !config.automation_daemon_enabled() { + println!(" exiting: no automation watcher is enabled"); + io::stdout() + .flush() + .context("failed to flush rocmd exit note")?; + // Our own spawn claim would otherwise make the next + // autostart defer to this still-alive process. + release_own_autostart_claim(paths); + if let Some(task) = local_webhook_task.as_ref() { + task.abort(); + } + return record_daemon_stop(paths, &mut state); + } + } + } event = crate::webhook::receive_local_webhook_event(&mut local_webhook_receiver) => { if let Some(event) = event { let config = RocmCliConfig::load(paths)?; @@ -163,23 +285,66 @@ pub(crate) async fn run_daemon( } } + record_daemon_stop(paths, &mut state)?; + if let Some(task) = local_webhook_task { + task.abort(); + } + Ok(()) +} + +/// Take the autostart lock without blocking the runtime. `None` once +/// `hooks.lock_wait` has passed with the lock still held by someone else. +async fn acquire_autostart_lock( + paths: &AppPaths, + hooks: &LoopHooks<'_>, +) -> Result> { + let deadline = time::Instant::now() + hooks.lock_wait; + loop { + if let Some(lock) = + rocm_core::FileLock::try_acquire(paths.automation_autostart_lock_path())? + { + return Ok(Some(lock)); + } + if let Some(on_lock_busy) = hooks.on_lock_busy { + on_lock_busy(); + } + if time::Instant::now() >= deadline { + return Ok(None); + } + time::sleep(AUTOSTART_LOCK_POLL).await; + } +} + +/// Publish that the loop is no longer running. +fn record_daemon_stop(paths: &AppPaths, state: &mut AutomationRuntimeState) -> Result<()> { state.running = false; state.last_tick_unix_ms = unix_time_millis(); state.local_webhook_endpoint = None; crate::persistence::record_event( paths, - &mut state, + state, "rocmd", "info", "daemon_stop", "rocmd automation supervisor stopped", None, )?; - state.write(paths)?; - if let Some(task) = local_webhook_task { - task.abort(); + state.write(paths) +} + +/// Remove the autostart claim (`" "`) if it names this process. +/// Best-effort: a leftover claim only delays a restart until its TTL. +fn release_own_autostart_claim(paths: &AppPaths) { + let claim_path = paths.automation_autostart_claim_path(); + let own = std::fs::read_to_string(&claim_path).is_ok_and(|text| { + text.split_whitespace() + .next() + .and_then(|pid| pid.parse::().ok()) + == Some(std::process::id()) + }); + if own { + let _ = fs::remove_file(claim_path); } - Ok(()) } pub(crate) fn print_status(paths: &AppPaths) -> Result<()> { @@ -187,7 +352,9 @@ pub(crate) fn print_status(paths: &AppPaths) -> Result<()> { println!("rocmd status"); println!(" config dir: {}", paths.config_dir.display()); println!(" data dir: {}", paths.data_dir.display()); - println!(" policy: on-demand by default, persistent only with background features"); + println!( + " policy: on-demand; the watcher loop runs only while an automation watcher is enabled" + ); println!( " automations desired: {}", if config.automation_daemon_enabled() { @@ -294,6 +461,7 @@ async fn shutdown_signal() { mod tests { use super::*; use crate::test_support::temp_app_paths; + use std::thread; #[tokio::test] async fn local_webhook_requires_enabled_automation_loop() { @@ -302,4 +470,333 @@ mod tests { assert!(error.to_string().contains("requires --automations-enabled")); } + + /// ROCMAI-827: the automation loop used to run forever even with nothing to + /// do, so a daemon `rocm serve` autostarted stayed alive after every service + /// was stopped. With no automation watcher enabled it now exits by itself and + /// records that it is no longer running. + #[tokio::test] + async fn automation_loop_exits_when_no_watcher_is_enabled() -> Result<()> { + let (root, paths) = temp_app_paths("loop-exits-without-watchers"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + + let finished = + tokio::time::timeout(Duration::from_secs(30), run_daemon(&paths, true, None)).await; + let state = AutomationRuntimeState::load(&paths)?; + let _ = fs::remove_dir_all(root); + + finished.expect("the loop must exit on its own with no watcher enabled")?; + let state = state.expect("the loop records its runtime state"); + assert!(!state.running, "an exited loop must not read as running"); + Ok(()) + } + + fn set_watcher(paths: &AppPaths, enabled: bool) { + let mut config = RocmCliConfig::load(paths).expect("load config"); + config.watcher_config_mut("server-recover").enabled = enabled; + config.automations.daemon_enabled = enabled; + config.save(paths).expect("save config"); + } + + fn fast_ticks<'a>( + on_tick: &'a (dyn Fn() + Sync), + on_idle_observed: Option<&'a (dyn Fn() + Sync)>, + ) -> LoopHooks<'a> { + LoopHooks { + tick_interval: Duration::from_millis(20), + on_tick: Some(on_tick), + on_idle_observed, + ..LoopHooks::default() + } + } + + /// The loop's other half: while a watcher is enabled it keeps running across + /// ticks, not just the first one (an interval fires immediately), and it still + /// exits once the watcher is disabled. The test disables the watcher only + /// after several ticks, so an exit check that misfired on a later tick (or a + /// loop that stopped after the first) would show in the tick count. + #[tokio::test] + async fn automation_loop_keeps_running_while_automations_are_enabled() -> Result<()> { + use std::sync::atomic::{AtomicU32, Ordering}; + + const TICKS_WITH_WATCHER: u32 = 4; + let (root, paths) = temp_app_paths("loop-persists-with-automations"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + set_watcher(&paths, true); + + let ticks = AtomicU32::new(0); + let on_tick = || { + if ticks.fetch_add(1, Ordering::SeqCst) + 1 > TICKS_WITH_WATCHER { + set_watcher(&paths, false); + } + }; + let hooks = fast_ticks(&on_tick, None); + let finished = time::timeout( + Duration::from_secs(30), + run_daemon_with(&paths, true, None, hooks), + ) + .await; + let state = AutomationRuntimeState::load(&paths)?; + let _ = fs::remove_dir_all(root); + + finished.expect("the loop must exit once the watcher is disabled")?; + assert_eq!( + ticks.load(Ordering::SeqCst), + TICKS_WITH_WATCHER + 1, + "the loop must run every tick while the watcher is enabled, and stop on the first tick after it is disabled" + ); + assert!(!state.expect("runtime state").running); + Ok(()) + } + + /// ROCMAI-827 race: the loop read "no watcher enabled", and an + /// `automations enable` saved its config before the loop acted on that. The + /// enable's autostart check would see this process as running and spawn + /// nothing, so exiting here would leave a watcher enabled with no daemon. The + /// exit must be re-decided against the current config and not happen. + #[tokio::test] + async fn automation_loop_stays_when_a_watcher_is_enabled_inside_the_exit_window() -> Result<()> + { + use std::sync::atomic::{AtomicU32, Ordering}; + + let (root, paths) = temp_app_paths("loop-exit-window"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + set_watcher(&paths, true); + + let ticks = AtomicU32::new(0); + let window_hits = AtomicU32::new(0); + // Tick 2: the last watcher is disabled, so the loop observes "idle". + // That observation is the window: the watcher is enabled again before + // the loop confirms the exit. Tick 6 disables it for good (no window + // re-enable) so the loop can end and the test can finish. + let on_tick = || { + let tick = ticks.fetch_add(1, Ordering::SeqCst) + 1; + if tick == 2 || tick == 6 { + set_watcher(&paths, false); + } + }; + let on_idle = || { + if window_hits.fetch_add(1, Ordering::SeqCst) == 0 { + set_watcher(&paths, true); + } + }; + let hooks = fast_ticks(&on_tick, Some(&on_idle)); + let finished = time::timeout( + Duration::from_secs(30), + run_daemon_with(&paths, true, None, hooks), + ) + .await; + let enabled_when_loop_ended = RocmCliConfig::load(&paths)?.automation_daemon_enabled(); + let _ = fs::remove_dir_all(root); + + finished.expect("the loop must end once the watcher stays disabled")?; + assert_eq!( + window_hits.load(Ordering::SeqCst), + 2, + "exactly the window and the real exit" + ); + assert_eq!( + ticks.load(Ordering::SeqCst), + 6, + "the loop must survive the tick where a watcher was re-enabled in the window" + ); + assert!(!enabled_when_loop_ended); + Ok(()) + } + + /// Poll `condition` (yielding, no fixed pause) until it holds, so a test waits + /// for an observed event rather than assuming how long it takes. + fn wait_until(what: &str, condition: impl Fn() -> bool) { + let deadline = std::time::Instant::now() + Duration::from_mins(1); + while !condition() { + assert!( + std::time::Instant::now() < deadline, + "timed out waiting for {what}" + ); + thread::yield_now(); + } + } + + /// Run the loop on its own thread and runtime, next to a task that counts + /// ticks of its own 1 ms timer. Returns the thread and that counter. + fn spawn_daemon_beside_a_counter( + paths: &AppPaths, + hooks: LoopHooks<'static>, + ) -> ( + thread::JoinHandle>, + std::sync::Arc, + ) { + let counter = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)); + let beat = counter.clone(); + let paths = paths.clone(); + let daemon = thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("runtime") + .block_on(async move { + tokio::spawn(async move { + loop { + time::sleep(Duration::from_millis(1)).await; + beat.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + } + }); + run_daemon_with(&paths, true, None, hooks).await + }) + }); + (daemon, counter) + } + + /// The exit is published under the autostart lock: while another caller holds + /// it (an `automations enable` mid-decision) the loop does not report + /// "stopped" or exit, and it clears its own spawn claim when it does. The wait + /// is non-blocking: a task sharing the loop's single-threaded runtime keeps + /// running meanwhile (a blocking acquire would stall it, and with it the + /// webhook task). No fixed pause: the test waits until the loop has actually + /// found the lock held. + #[test] + fn automation_loop_exit_waits_for_the_autostart_lock_and_clears_its_claim() -> Result<()> { + use std::sync::atomic::{AtomicU32, Ordering}; + + static BUSY: AtomicU32 = AtomicU32::new(0); + static ON_BUSY: fn() = || { + BUSY.fetch_add(1, Ordering::SeqCst); + }; + BUSY.store(0, Ordering::SeqCst); + + let (root, paths) = temp_app_paths("loop-exit-lock"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + std::fs::write( + paths.automation_autostart_claim_path(), + format!("{} 1", std::process::id()), + )?; + + let guard = rocm_core::FileLock::acquire(paths.automation_autostart_lock_path())?; + let (daemon, beat) = spawn_daemon_beside_a_counter( + &paths, + LoopHooks { + tick_interval: Duration::from_millis(20), + on_lock_busy: Some(&ON_BUSY), + ..LoopHooks::default() + }, + ); + wait_until("the loop to find the lock held", || { + BUSY.load(Ordering::SeqCst) >= 3 + }); + let before = beat.load(Ordering::SeqCst); + wait_until("the runtime to keep running while waiting", || { + beat.load(Ordering::SeqCst) >= before + 3 + }); + let finished_while_locked = daemon.is_finished(); + let running_while_locked = + AutomationRuntimeState::load(&paths)?.is_some_and(|state| state.running); + drop(guard); + daemon.join().expect("daemon thread")?; + let state = AutomationRuntimeState::load(&paths)?; + let claim_left = paths.automation_autostart_claim_path().exists(); + let _ = fs::remove_dir_all(root); + + assert!(!finished_while_locked, "exit must wait for the lock"); + assert!( + running_while_locked, + "still published as running until then" + ); + assert!(!state.expect("runtime state").running); + assert!(!claim_left, "the exiting helper must clear its own claim"); + Ok(()) + } + + /// A holder that never lets go must not wedge the loop: after the bounded wait + /// the tick gives up on exiting, the loop stays up and tries again on later + /// ticks, and once the lock is free it exits. + #[test] + fn automation_loop_gives_up_a_stuck_lock_wait_and_exits_once_it_is_free() -> Result<()> { + use std::sync::atomic::{AtomicU32, Ordering}; + + static TICKS: AtomicU32 = AtomicU32::new(0); + static ON_TICK: fn() = || { + TICKS.fetch_add(1, Ordering::SeqCst); + }; + TICKS.store(0, Ordering::SeqCst); + + let (root, paths) = temp_app_paths("loop-exit-stuck-lock"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + + let guard = rocm_core::FileLock::acquire(paths.automation_autostart_lock_path())?; + let (daemon, _beat) = spawn_daemon_beside_a_counter( + &paths, + LoopHooks { + tick_interval: Duration::from_millis(20), + on_tick: Some(&ON_TICK), + lock_wait: Duration::from_millis(30), + ..LoopHooks::default() + }, + ); + // Several ticks have each waited out the bound and given up. + wait_until("several timed-out exit attempts", || { + TICKS.load(Ordering::SeqCst) >= 6 + }); + let finished_while_stuck = daemon.is_finished(); + drop(guard); + daemon.join().expect("daemon thread")?; + let state = AutomationRuntimeState::load(&paths)?; + let _ = fs::remove_dir_all(root); + + assert!(!finished_while_stuck, "a stuck lock must not make it exit"); + assert!(!state.expect("runtime state").running); + Ok(()) + } + + /// The wait for the autostart lock is its own select arm: while it is waiting + /// on a held lock, a shutdown request is still acted on at once instead of + /// after the (up to 30 s) wait. The lock stays held until the loop has + /// returned, so the exit cannot have come from the lock being released. + #[test] + fn automation_loop_shuts_down_while_waiting_for_a_held_autostart_lock() -> Result<()> { + use std::sync::atomic::{AtomicU32, Ordering}; + + static BUSY: AtomicU32 = AtomicU32::new(0); + static ON_BUSY: fn() = || { + BUSY.fetch_add(1, Ordering::SeqCst); + }; + static STOP: tokio::sync::Notify = tokio::sync::Notify::const_new(); + BUSY.store(0, Ordering::SeqCst); + + let (root, paths) = temp_app_paths("loop-shutdown-while-locked"); + paths.ensure()?; + RocmCliConfig::default().save(&paths)?; + + let guard = rocm_core::FileLock::acquire(paths.automation_autostart_lock_path())?; + let (daemon, _beat) = spawn_daemon_beside_a_counter( + &paths, + LoopHooks { + tick_interval: Duration::from_millis(20), + on_lock_busy: Some(&ON_BUSY), + stop: Some(&STOP), + // Longer than the test's own deadline: only a wait that really + // runs beside the shutdown arm lets the loop stop with the lock held. + lock_wait: Duration::from_mins(10), + ..LoopHooks::default() + }, + ); + wait_until("the loop to be waiting on the held lock", || { + BUSY.load(Ordering::SeqCst) >= 2 + }); + STOP.notify_one(); + wait_until("the loop to shut down with the lock still held", || { + daemon.is_finished() + }); + drop(guard); + daemon.join().expect("daemon thread")?; + let state = AutomationRuntimeState::load(&paths)?; + let _ = fs::remove_dir_all(root); + + assert!(!state.expect("runtime state").running); + Ok(()) + } } diff --git a/crates/rocm-core/src/config.rs b/crates/rocm-core/src/config.rs index b1efe4b1ab..33622e5ebc 100644 --- a/crates/rocm-core/src/config.rs +++ b/crates/rocm-core/src/config.rs @@ -594,8 +594,15 @@ impl RocmCliConfig { .or_default() } + /// Whether any automation watcher is enabled, i.e. whether the background + /// helper (`rocm daemon`) has work. This is the single predicate behind the + /// helper's autostart, its self-exit and the "background checks" display, so + /// they cannot disagree. The stored `automations.daemon_enabled` flag is + /// deliberately not consulted: `automations enable`/`disable` keep it in + /// step with the watchers, and on its own it would describe a daemon with + /// nothing to run. pub fn automation_daemon_enabled(&self) -> bool { - self.automations.daemon_enabled || self.automations.watchers.values().any(|cfg| cfg.enabled) + self.automations.watchers.values().any(|cfg| cfg.enabled) } pub fn watcher_enabled(&self, watcher: &BuiltinWatcherSpec) -> bool { diff --git a/crates/rocm-core/src/process.rs b/crates/rocm-core/src/process.rs index 462bbc7547..aafbb4cd1b 100644 --- a/crates/rocm-core/src/process.rs +++ b/crates/rocm-core/src/process.rs @@ -409,7 +409,27 @@ impl FileLock { /// The lock file itself carries no data; it is a rendezvous point, so an /// existing file is reused (never truncated) and its contents are ignored. pub fn acquire(path: impl Into) -> Result { - let path = path.into(); + let (file, path) = Self::open(path.into())?; + file.lock() + .with_context(|| format!("failed to acquire lock {}", path.display()))?; + Ok(Self { file, path }) + } + + /// As [`acquire`](Self::acquire), but never blocks: `Ok(None)` means another + /// holder has the lock right now. For callers that must not park a thread + /// (an async loop) and retry on their own schedule. + pub fn try_acquire(path: impl Into) -> Result> { + let (file, path) = Self::open(path.into())?; + match file.try_lock() { + Ok(()) => Ok(Some(Self { file, path })), + Err(fs::TryLockError::WouldBlock) => Ok(None), + Err(fs::TryLockError::Error(error)) => { + Err(error).with_context(|| format!("failed to try-acquire lock {}", path.display())) + } + } + } + + fn open(path: PathBuf) -> Result<(fs::File, PathBuf)> { if let Some(parent) = path.parent() { fs::create_dir_all(parent) .with_context(|| format!("failed to create lock directory {}", parent.display()))?; @@ -421,9 +441,7 @@ impl FileLock { .truncate(false) .open(&path) .with_context(|| format!("failed to open lock file {}", path.display()))?; - file.lock() - .with_context(|| format!("failed to acquire lock {}", path.display()))?; - Ok(Self { file, path }) + Ok((file, path)) } /// The lock file backing this guard. diff --git a/docs/architecture.md b/docs/architecture.md index 7782fed8c5..a2523c7e33 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -115,7 +115,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`cli.rs`](../apps/rocmd/src/cli.rs) — the `Cli`/`Command` clap definitions, the bridge-snapshot diagnostic printer, the pretty-JSON printer for the subcommands that print a result (`print_json`), and the top-level dispatch in `run_cli`/`run_bin_cli`/`run_from_args`. - [`common.rs`](../apps/rocmd/src/common.rs) — helpers the other modules call into: the read-only `rocm update` message, the `rocm`-subprocess capture wrapper (`run_rocm_capture_for_paths`) the sandbox and MCP clusters both call (it only chooses the child's `ROCM_CLI_*_DIR` environment; `CommandCapture` and the spawn itself live in [`command_capture.rs`](../crates/rocm-core/src/command_capture.rs), shared with `rocm`'s chat tool bridge, and the bounded-execution half in [`process.rs`](../crates/rocm-core/src/process.rs)), and small arg/JSON-payload utilities (`payload_string`, for example, is read by both [`watchers.rs`](../apps/rocmd/src/watchers.rs) and [`webhook.rs`](../apps/rocmd/src/webhook.rs)). -- [`daemon.rs`](../apps/rocmd/src/daemon.rs) — the foreground `rocmd run` loop (`run_daemon`: evaluate the watchers each tick, dispatch local webhook events, write the runtime state, stop on a signal) and `rocmd status` (`print_status`), with the watcher tick interval. It uses [`watchers.rs`](../apps/rocmd/src/watchers.rs), [`webhook.rs`](../apps/rocmd/src/webhook.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs) calls it. +- [`daemon.rs`](../apps/rocmd/src/daemon.rs) — the foreground `rocmd run` loop (`run_daemon`: evaluate the watchers each tick, dispatch local webhook events, write the runtime state, stop on a signal; with no watcher enabled it exits by itself, deciding that under the autostart lock and publishing "stopped" and clearing its spawn claim before releasing it — `record_daemon_stop`, `release_own_autostart_claim`) and `rocmd status` (`print_status`), with the watcher tick interval, which `LoopHooks` makes injectable for tests. It uses [`watchers.rs`](../apps/rocmd/src/watchers.rs), [`webhook.rs`](../apps/rocmd/src/webhook.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs) calls it. - [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) — talking to an engine and guarding its endpoint key: `engine_request` (the stdio request to an engine process), the healthcheck call and its readiness and recoverability checks, and the endpoint-key helpers `apply_endpoint_key_env` and `ensure_public_service_has_endpoint_key`. It uses no other module of the crate; [`service.rs`](../apps/rocmd/src/service.rs), [`supervise.rs`](../apps/rocmd/src/supervise.rs) and [`watchers.rs`](../apps/rocmd/src/watchers.rs) call it. - [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs) — the host snapshot: `build_bridge_snapshot` (the bridge snapshot printed by `rocmd bridge-snapshot` and returned by the `bridge_snapshot` MCP tool, assembled from the examine summary, the GPU snapshot, the automation runtime state and recent events, the engine inventory and the managed services), the amd-smi GPU snapshot (`gather_gpu_snapshot_for_config`, which honours the telemetry setting, over time-bounded amd-smi probes) and `bridge_engine_inventory`. It uses only [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs), [`mcp.rs`](../apps/rocmd/src/mcp.rs) and [`watchers.rs`](../apps/rocmd/src/watchers.rs) call it. - [`mcp.rs`](../apps/rocmd/src/mcp.rs) — the MCP stdio server, tool schema table, tool dispatch, and the argv-building and result-shaping helpers behind the MCP tools (the `rocm` subprocess itself is spawned through [`common.rs`](../apps/rocmd/src/common.rs), via `run_rocm_capture` here). diff --git a/docs/automations.md b/docs/automations.md index e77ddec78b..d17a2c348e 100644 --- a/docs/automations.md +++ b/docs/automations.md @@ -12,7 +12,10 @@ grant a new action, choose a tool, override watcher mode, or run shell. ## Local Webhook Source -Start the persistent watcher loop with a loopback-only webhook source: +Start the watcher loop with a loopback-only webhook source. The loop runs only +while at least one watcher is enabled and exits on its own within a few seconds +otherwise, so enable one first (for example `rocm automations enable +gpu-metrics`): ```bash rocmd run --automations-enabled --local-webhook-port 18080 diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 1e2e593fb0..1185621dbe 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1027,3 +1027,34 @@ Expected result: service read `failed` rather than `stopped`, so rocmd's recovery restarts it. - On native Windows, the same command fails explicitly instead of falling back to another engine. + +## 11. Background Automation Helper + +The helper (`rocm daemon`) runs only while an automation watcher is enabled. +Start from a fresh config (all watchers off) and use a scratch +`ROCM_CLI_CONFIG_DIR`/`ROCM_CLI_DATA_DIR` if you do not want to touch yours. + +```bash +rocm automations enable gpu-metrics --mode observe +rocm daemon --status +rocm automations disable gpu-metrics +``` + +Expected result: + +- Enabling prints `helper: started background automation daemon`, and a + `rocm daemon` process (Windows: `rocm.exe daemon`; Linux: `ps`) is running. + `rocm automations` reports the background service as `running`. +- After disabling the only watcher, the process exits by itself within a few + seconds (the loop ticks every 5 seconds) and `rocm automations` reports it + `stopped`. Nothing needs to be killed. +- With no watcher enabled (the default), `rocm serve` of a model starts no + `rocm daemon` process at all, and `rocm services stop --yes` + prints no note about a helper. +- With a watcher enabled, `rocm services stop --yes` leaves the + helper running and says so: its `note:` line names the enabled watcher and + the `rocm automations disable ` command that ends it. `rocm uninstall + --dry-run` carries the same warning, ending `uninstall does not stop it`. +- Re-enabling a watcher right after disabling the last one (within the few + seconds before the helper exits) still ends with a helper running: enable + waits for an exiting helper to finish, then starts a new one. diff --git a/docs/testing.md b/docs/testing.md index 4a4de17a3d..5cede211ba 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1561,6 +1561,36 @@ so it carries `@requires-dir-permissions` and resolves to a skip for a user the permission bits do not bind (the self-hosted WSL2 guest runs as root) rather than pass without testing the fallback. +Background helper (`rocm daemon`) lifecycle tests: + +```bash +cargo nextest run -p rocm -E 'test(background_helper_not_spawned_when_no_automation_is_enabled) | test(background_helper_spawned_when_an_automation_is_enabled) | test(background_helper_is_started_when_the_exiting_daemon_publishes_stopped) | test(helper_note) | test(stop_output_carries) | test(plan_warns_about_a_helper) | test(unreadable_config)' +cargo nextest run -p rocmd -E 'test(automation_loop_)' +``` + +The first command checks that the autostart shared by `serve` and +`automations enable` spawns the helper only while an automation watcher is +enabled, and that an enable racing the helper's exit still gets a helper (the +decision is made under the autostart lock). The second drives the helper's loop +with a fast injected tick: it exits on its own once no watcher is enabled, +keeps running across several ticks while one is, stays when a watcher is +re-enabled inside the exit window, and publishes "stopped" and clears its spawn +claim under the autostart lock. The lock wait is its own select arm polled with +try-lock, so it neither parks a thread nor delays webhook dispatch or shutdown +(a test shuts the loop down while the lock is held) and gives up a stuck wait. +The note that `services stop` and `rocm uninstall` print about the helper is +derived from both whether a watcher is enabled and whether the helper is running: +`helper_note_follows_watcher_enabled_and_helper_running` pins the exact text for +all four combinations, `stop_output_carries_the_helper_note_...` and +`plan_warns_about_a_helper_...` pin what each surface prints for them, and +`background_helper_autostart_treats_an_unreadable_config_as_no_watcher` keeps a +bad config from failing a `serve` after its launch. The user-visible side is `automations-11` in +[automations.feature](../tests/e2e-cucumber/features/automations.feature) +(mock lane: enable starts a live helper, disabling the last watcher makes it +exit) and `serve-40` in +[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature) +(GPU lane: serving with no watcher enabled starts no helper). + Natural-language plan handoff tests: ```bash diff --git a/tests/e2e-cucumber/features/automations.feature b/tests/e2e-cucumber/features/automations.feature index fc08c39d1c..b18d398fe2 100644 --- a/tests/e2e-cucumber/features/automations.feature +++ b/tests/e2e-cucumber/features/automations.feature @@ -151,3 +151,22 @@ Feature: Automation watchers Then the CLI confirms the permissions mode And the CLI prints no warning about the audit log And the audit log keeps its earlier record, drops the torn line and records the permissions change + + # The background helper's lifecycle (ROCMAI-827). Enabling a watcher starts a + # detached `rocm daemon`; it used to run forever after that, so a helper + # started by `serve` or `automations enable` outlived every service and every + # watcher. It now exits on its own once no watcher is enabled. This drives the + # real spawn — no suppression marker — and checks the `helper:` line the + # enable prints together with the process it claims to have started, then + # that disabling the only watcher makes that process exit. The other half, + # that `serve` with no watcher enabled starts no helper at all, needs a real + # serve and is `serve-stop-leaves-no-background-helper` on the GPU lane. + # Config-only — no GPU or network — so it runs on the mock lane every PR. + @id:automations-helper-stops-when-no-watcher-is-enabled + Scenario: automations-11 - The background helper stops on its own once no watcher is enabled + Given no automation watcher is enabled + When the user enables an automation watcher in observe mode + Then the CLI reports starting the background helper and it is running + When the user disables the watcher + Then the CLI confirms the watcher is disabled + And the background helper stops on its own diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 2536900ffa..243ef74500 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -619,3 +619,16 @@ Feature: Model serving Then serving is refused before any engine starts And the CLI says the image needs eight GPUs and multi-GPU serving is not supported And Docker was never called + + + # ROCMAI-827: `rocm serve` started the background automation helper + # (`rocm daemon`) on every launch, even with every automation watcher off — + # when it has nothing to do — and that helper outlived the service. With no + # watcher enabled, serve now starts no helper; the scenario's fresh config + # has none enabled. The helper's own lifecycle once a watcher is on is + # `automations-helper-stops-when-no-watcher-is-enabled` on the mock lane. + @id:serve-stop-leaves-no-background-helper @requires-gpu @requires-engine:lemonade + Scenario: serve-40 - Serving a model with no automation enabled starts no background helper + Given a managed runtime is active + When the user serves a GGUF model on lemonade with no automation enabled + Then serving started no background helper diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 2d56ac2835..91b2241b28 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -169,6 +169,10 @@ pub struct E2eWorld { /// field rather than borrowing `model_name`, which means a served model and /// has nothing to do with a user's error report. pub skill_symptom: Option, + /// The background helper (`rocm daemon`) a scenario saw running, captured + /// while it was known live, so `Drop` can stop it by a verified identity if + /// an earlier step failed before the scenario disabled the watcher. + pub helper_identity: Option, /// A live process a scenario recorded as a managed server's own, so a stop /// has something real to terminate. Held here so `Drop` can kill and reap /// it whatever the scenario's outcome, and so the stop's effect on it can @@ -338,6 +342,7 @@ impl Default for E2eWorld { lifecycle: None, skill_reference: None, skill_symptom: None, + helper_identity: None, recorded_process: None, dash_report_refused: None, } @@ -674,6 +679,16 @@ impl Drop for E2eWorld { let _ = process.kill(); let _ = process.wait(); } + // A scenario that enabled a watcher started a real detached `rocm daemon`, + // which only exits once the watcher is disabled. If a step failed first, + // nothing else would stop it, and it would keep polling a config dir + // that is about to be deleted. + if self.isolated_root.is_some() { + e2e::automations_steps::stop_leftover_helper( + &self.app_paths(), + self.helper_identity.take(), + ); + } if let Some(mock) = self.mock.take() { mock.stop(); } diff --git a/tests/e2e-cucumber/tests/e2e/automations_steps.rs b/tests/e2e-cucumber/tests/e2e/automations_steps.rs index 0204fe8bc4..0dd72bdd6f 100644 --- a/tests/e2e-cucumber/tests/e2e/automations_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/automations_steps.rs @@ -121,6 +121,13 @@ async fn daemon_spawn_in_flight(world: &mut E2eWorld) { plant_inflight_autostart_claim(world); } +#[given("no automation watcher is enabled")] +async fn no_watcher_enabled(_world: &mut E2eWorld) { + // The World already gives each scenario an isolated, empty config dir, where + // every watcher is off. Deliberately plants NO spawn-suppression marker: the + // scenario drives the real daemon spawn. +} + #[given("a pending artifact-download proposal approved for a size just under 1 MiB")] async fn pending_artifact_download_proposal(world: &mut E2eWorld) { append_automation_proposal( @@ -547,6 +554,125 @@ async fn no_second_daemon(world: &mut E2eWorld) { ); } +/// How long the helper gets to publish its runtime state after the spawn, or +/// to exit after its last watcher is disabled (it re-reads the config on a 5 s +/// tick). Generous because the mock lane runs many scenarios at once. +const HELPER_LIFECYCLE_TIMEOUT: std::time::Duration = std::time::Duration::from_mins(1); + +/// The helper's recorded pid and `running` flag, read from the runtime state +/// the daemon itself publishes — the same file `rocm daemon --status` reads. +fn helper_runtime_state(world: &E2eWorld) -> Option<(u32, bool)> { + rocm_core::AutomationRuntimeState::load(&world.app_paths()) + .ok() + .flatten() + .map(|state| (state.daemon_pid, state.running)) +} + +/// Teardown for a scenario that started a real helper: stop it if it is still +/// alive, whatever step failed. The helper is found from the runtime state it +/// published (when it says `running`) or, if it never got that far, from the +/// autostart claim the spawn writes synchronously (honoured for the CLI's own +/// 30 s). It is signalled only through `terminate_verified`, against the identity +/// captured when the scenario saw it live, or captured now where the state or +/// claim names a live pid — never by a bare pid. Best-effort: teardown must not +/// panic. +pub(crate) fn stop_leftover_helper( + paths: &rocm_core::AppPaths, + seen: Option, +) { + let from_state = rocm_core::AutomationRuntimeState::load(paths) + .ok() + .flatten() + .filter(|state| state.running) + .map(|state| state.daemon_pid); + let from_claim = || { + let text = std::fs::read_to_string(paths.automation_autostart_claim_path()).ok()?; + let mut parts = text.split_whitespace(); + let pid: u32 = parts.next()?.parse().ok()?; + let at_ms: u128 = parts.next()?.parse().ok()?; + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .ok()? + .as_millis(); + (now_ms.saturating_sub(at_ms) < 30_000).then_some(pid) + }; + let identity = seen.or_else(|| { + from_state + .or_else(from_claim) + // Scenarios that suppress the spawn plant a state and claim naming + // this very harness; it is never the helper. + .filter(|pid| *pid != std::process::id()) + .filter(|pid| rocm_core::process_is_running(*pid)) + .map(rocm_core::ProcessIdentity::capture) + }); + if let Some(identity) = identity { + let _ = rocm_core::terminate_verified( + &identity, + rocm_core::KillScope::Single, + std::time::Duration::from_secs(5), + true, + ); + } +} + +#[then("the CLI reports starting the background helper and it is running")] +async fn helper_started_and_running(world: &mut E2eWorld) { + let out = ok_output(world); + assert!( + out.contains("helper: started background automation daemon"), + "expected the enable to report starting the helper, got: +{}", + combined(world) + ); + // The claim is checked against the process: the helper publishes a running + // state under its own pid, and that pid is a live process. + let deadline = std::time::Instant::now() + HELPER_LIFECYCLE_TIMEOUT; + loop { + if let Some((pid, true)) = helper_runtime_state(world) + && rocm_core::process_is_running(pid) + { + world.helper_identity = Some(rocm_core::ProcessIdentity::capture(pid)); + let status = crate::run_rocm_ok(world, &["daemon", "--status"]); + assert!( + status.contains("Status: running"), + "expected `rocm daemon --status` to report the helper running, got: +{status}" + ); + return; + } + assert!( + std::time::Instant::now() < deadline, + "the helper never published a running state under a live pid; state: {:?}", + helper_runtime_state(world) + ); + tokio::time::sleep(std::time::Duration::from_millis(250)).await; + } +} + +#[then("the background helper stops on its own")] +async fn helper_stops_on_its_own(world: &mut E2eWorld) { + let (pid, _) = helper_runtime_state(world).expect("the helper recorded no runtime state"); + let deadline = std::time::Instant::now() + HELPER_LIFECYCLE_TIMEOUT; + loop { + let running = helper_runtime_state(world).is_some_and(|(_, running)| running); + if !running && !rocm_core::process_is_running(pid) { + let status = crate::run_rocm_ok(world, &["daemon", "--status"]); + assert!( + status.contains("Status: not running"), + "expected `rocm daemon --status` to report the helper stopped, got: +{status}" + ); + return; + } + assert!( + std::time::Instant::now() < deadline, + "the helper (pid {pid}) was still running {}s after its last watcher was disabled", + HELPER_LIFECYCLE_TIMEOUT.as_secs() + ); + tokio::time::sleep(std::time::Duration::from_millis(250)).await; + } +} + #[then("the CLI confirms the watcher is disabled")] async fn confirm_disabled(world: &mut E2eWorld) { let out = ok_output(world); diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 49ab899ce3..d5d40e0c04 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -173,14 +173,21 @@ fn serve_failure_evidence( /// dropped before the panic, which is why #260 could not be root-caused from a /// CI artifact. A non-zero exit is reported the same way rather than through /// `run_rocm_ok`, since the engine log explains those launches too. -async fn serve_and_wait(world: &mut E2eWorld, args: &[&str], model: &str, ready_substr: &str) { +/// +/// Returns the serve command's stdout, for a scenario that checks what it printed. +async fn serve_and_wait( + world: &mut E2eWorld, + args: &[&str], + model: &str, + ready_substr: &str, +) -> String { let timeout_secs = serve_timeout_for(world); let device_state = ensure_serve_port_free().await; let (stdout, stderr, rc) = crate::run_rocm(world, args); if rc == 0 && model_is_ready(&MODELS_URL, Some(ready_substr), timeout_secs).await { world.endpoint = Some(SERVE_BASE_URL.to_string()); world.model_name = Some(model.to_string()); - return; + return stdout; } let invocation = format!("`rocm {}`", args.join(" ")); let headline = if rc == 0 { @@ -643,6 +650,62 @@ async fn setup_lemonade_model(world: &mut E2eWorld) { .await; } +/// ROCMAI-827: serve the same small GGUF model as `setup_lemonade_model`, from +/// the scenario's fresh config (every automation watcher off), keeping what the +/// serve printed so the Then can check it against the helper's runtime state. +#[when("the user serves a GGUF model on lemonade with no automation enabled")] +async fn serve_lemonade_without_automations(world: &mut E2eWorld) { + let stdout = serve_and_wait( + world, + &[ + "serve", + "Qwen3-0.6B-GGUF", + "--engine", + "lemonade", + "--managed", + ], + "Qwen3-0.6B-GGUF", + "Qwen3-0.6B", + ) + .await; + world.cli_output = Some(stdout); +} + +#[then("serving started no background helper")] +async fn serving_started_no_background_helper(world: &mut E2eWorld) { + let out = world.cli_output.clone().unwrap_or_default(); + assert!( + !out.contains("background automation daemon"), + "with no watcher enabled, serve must not start (or try to start) the helper, got: +{out}" + ); + // The printed line is checked together with the state: the helper never + // published a runtime state, and its status reports it not running. + // A helper spawned a moment ago may not have published its runtime state + // yet, so the state alone can pass vacuously. The spawn records its + // autostart claim synchronously, before the CLI returns, so a claim file is + // proof a helper was started. + let claim = world.app_paths().automation_autostart_claim_path(); + assert!( + !claim.exists(), + "serve spawned a background helper (autostart claim {} exists: {:?})", + claim.display(), + std::fs::read_to_string(&claim) + ); + let state = rocm_core::AutomationRuntimeState::load(&world.app_paths()) + .expect("failed to read the helper's runtime state"); + assert!( + state.as_ref().is_none_or(|state| !state.running), + "serve left a background helper recorded running: {state:?}" + ); + let status = crate::run_rocm_ok(world, &["daemon", "--status"]); + assert!( + status.contains("Status: not running"), + "expected `rocm daemon --status` to report no helper, got: +{status}" + ); +} + #[given("a canonical Hugging Face GGUF checkpoint is being served on lemonade")] async fn setup_lemonade_hf_checkpoint_model(world: &mut E2eWorld) { // Forces the `owner/repo:variant` direct-serve path (`serve_hf_checkpoint`,