Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1132,6 +1132,14 @@ rocm services remove <service-id> --yes
rocm services prune [--older-than-hours <n> | --any-age] [--dry-run] [--yes]
```

On Linux, managed supervisor and engine PIDs are recorded with their process
start times. A live numeric PID counts as the recorded service only when a
recorded start time still matches; a mismatch means the PID was recycled, so
`services list` persists the record as stopped without touching the unrelated
process now using that PID. If the start time cannot currently be read, the
service remains conservatively live. Records created before start times were
stored retain their legacy PID-only behavior.

`stop` reports a server stopped only once every recorded process is confirmed
gone. If it cannot confirm that, it prints `Local server stop not confirmed` on
stderr and exits with an error. It keeps the record, which still names the
Expand Down
2 changes: 2 additions & 0 deletions apps/rocm/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1048,6 +1048,8 @@ pub(crate) enum ComfyuiCommand {
pub(crate) enum ServicesCommand {
/// Show currently running local model servers, and count the records that
/// are no longer running.
///
/// Recycled PIDs with mismatched recorded start times count as no longer running.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this is --help text that users read, and "recycled PIDs with mismatched recorded start times" is internal wording; it is also Linux-only behaviour (the README says so). Something like "A record whose process ID now belongs to a different process is shown as not running (Linux)." would read better.

List {
/// Include failed, stopped, and old service records.
#[arg(short, long)]
Expand Down
210 changes: 193 additions & 17 deletions apps/rocm/src/managed_service_status.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ use std::time::Duration;
use anyhow::{Context, Result};
use rocm_core::{
AppPaths, EndpointReadiness, EndpointReadinessOutcome, ManagedServiceRecord,
managed_service_endpoint_readiness, process_is_running,
managed_service_endpoint_readiness,
};

use crate::endpoint_keys;
Expand Down Expand Up @@ -178,16 +178,40 @@ pub(crate) fn live_managed_services_for_model(

const SERVICE_LIVENESS_CHECK_TIMEOUT: Duration = Duration::from_millis(750);

/// The real process ids a record tracks.
/// The real processes a record tracks, each paired with its own start-ticks.
///
/// The launcher pid is always recorded; `engine_pid` is adopted once the engine
/// state file reports one. A `0` pid is a placeholder, never a real process.
pub(crate) fn recorded_service_pids(record: &ManagedServiceRecord) -> Vec<u32> {
[record.engine_pid, Some(record.supervisor_pid)]
.into_iter()
.flatten()
.filter(|pid| *pid != 0)
.collect()
/// When both name the same pid, the supervisor's known start-ticks win a
/// conflict, and either known token wins over `None`.
pub(crate) fn recorded_service_pids(
record: &ManagedServiceRecord,
) -> Vec<rocm_core::ProcessIdentity> {
let mut identities: Vec<rocm_core::ProcessIdentity> = Vec::new();
for (pid, ticks) in [
(Some(record.supervisor_pid), record.supervisor_start_ticks),
(record.engine_pid, record.engine_start_ticks),
] {
let Some(pid) = pid.filter(|pid| *pid != 0) else {
continue;
};
match identities.iter_mut().find(|identity| identity.pid == pid) {
Some(existing) if existing.start_ticks.is_none() => existing.start_ticks = ticks,
Some(_) => {}
None => identities.push(rocm_core::ProcessIdentity::new(pid, ticks)),
}
}
identities
}

/// Whether a recorded process may still be the one the record started.
/// Unverifiable identities are conservatively alive; legacy records without
/// start-ticks retain plain PID liveness.
fn recorded_service_process_is_alive(identity: &rocm_core::ProcessIdentity) -> bool {
matches!(
rocm_core::identity_state(identity),
rocm_core::IdentityState::Matches | rocm_core::IdentityState::Indeterminate
)
}

/// Complete a stop that was requested but could not confirm termination, once
Expand All @@ -212,7 +236,7 @@ fn settle_pending_stop_key_cleanup(paths: &AppPaths, record: &mut ManagedService
}
if recorded_service_pids(record)
.iter()
.any(|pid| process_is_running(*pid))
.any(recorded_service_process_is_alive)
{
return false;
}
Expand Down Expand Up @@ -271,7 +295,7 @@ pub(crate) fn refresh_managed_service_runtime_liveness(

let tracked_pids = recorded_service_pids(record);
let has_tracked_pid = !tracked_pids.is_empty();
let has_live_pid = tracked_pids.iter().any(|pid| process_is_running(*pid));
let has_live_pid = tracked_pids.iter().any(recorded_service_process_is_alive);
if has_tracked_pid && !has_live_pid {
// Reconcile the status only. The endpoint key is *not* dropped here:
// dead pids alone do not distinguish a stopped service from a crashed
Expand Down Expand Up @@ -570,6 +594,148 @@ mod tests {
Ok(())
}

/// Run the liveness refresh on a `starting` record whose supervisor and
/// engine pid are both a live `sleep` child, recorded with `ticks(real)`.
#[cfg(target_os = "linux")]
fn refresh_starting_record_for_live_child(
name: &str,
ticks: impl Fn(u64) -> Option<u64>,
) -> (ManagedServiceRecord, bool) {
let mut child = std::process::Command::new("sleep")
.arg("60")
.spawn()
.expect("spawn child");
let pid = child.id();
let (root, paths) = test_paths(name);
let real = rocm_core::process_start_ticks(pid).expect("start-ticks");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-liveness",
"vllm",
"m",
"m",
"127.0.0.1",
9,
"managed",
pid,
None,
None,
None,
);
record.engine_pid = Some(pid);
record.supervisor_start_ticks = ticks(real);
record.engine_start_ticks = ticks(real);

let changed = refresh_managed_service_runtime_liveness(&paths, &mut record);

let _ = child.kill();
let _ = child.wait();
let _ = fs::remove_dir_all(root);
(record, changed)
}

#[cfg(target_os = "linux")]
#[test]
fn liveness_refresh_demotes_record_whose_pid_was_recycled() {
let (record, changed) =
refresh_starting_record_for_live_child("liveness-recycled", |real| {
Some(real.wrapping_add(1))
});
assert_eq!(record.status, "stopped");
assert!(changed);
}

#[cfg(target_os = "linux")]
#[test]
fn liveness_refresh_keeps_record_whose_pid_identity_matches_not_recycled() {
let (record, changed) =
refresh_starting_record_for_live_child("liveness-not-recycled", Some);
assert_eq!(record.status, "starting");
assert!(!changed);
}

#[cfg(target_os = "linux")]
#[test]
fn liveness_refresh_keeps_legacy_record_without_start_ticks_not_recycled() {
let (record, changed) =
refresh_starting_record_for_live_child("liveness-legacy-not-recycled", |_| None);
assert_eq!(record.status, "starting");
assert!(!changed);
}

#[cfg(target_os = "linux")]
#[test]
fn liveness_refresh_demotes_recycled_supervisor_when_engine_ticks_unknown() {
let mut child = std::process::Command::new("sleep")
.arg("60")
.spawn()
.expect("spawn child");
let pid = child.id();
let (root, paths) = test_paths("liveness-recycled-engine-unknown");
let real = rocm_core::process_start_ticks(pid).expect("start-ticks");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-liveness",
"vllm",
"m",
"m",
"127.0.0.1",
9,
"managed",
pid,
None,
None,
None,
);
record.engine_pid = Some(pid);
record.supervisor_start_ticks = Some(real.wrapping_add(1));

let changed = refresh_managed_service_runtime_liveness(&paths, &mut record);

let _ = child.kill();
let _ = child.wait();
let _ = fs::remove_dir_all(root);
assert_eq!(record.status, "stopped");
assert!(changed);
}

#[cfg(target_os = "linux")]
#[test]
fn liveness_refresh_prefers_supervisor_token_when_known_tokens_conflict() {
let mut child = std::process::Command::new("sleep")
.arg("60")
.spawn()
.expect("spawn child");
let pid = child.id();
let (root, paths) = test_paths("liveness-conflicting-known-tokens");
let real = rocm_core::process_start_ticks(pid).expect("start-ticks");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-liveness",
"vllm",
"m",
"m",
"127.0.0.1",
9,
"managed",
pid,
None,
None,
None,
);
record.engine_pid = Some(pid);
record.supervisor_start_ticks = Some(real.wrapping_add(1));
record.engine_start_ticks = Some(real);

let changed = refresh_managed_service_runtime_liveness(&paths, &mut record);

let _ = child.kill();
let _ = child.wait();
let _ = fs::remove_dir_all(root);
assert_eq!(record.status, "stopped");
assert!(changed);
}

/// A public record with a dead pid and a stored endpoint key, for the
/// liveness-refresh cases below. The port is one nothing listens on, so the
/// refresh's endpoint probe fails and it falls through to the pid check.
Expand Down Expand Up @@ -625,16 +791,20 @@ mod tests {
let _ = fs::remove_dir_all(root);
}

#[cfg(target_os = "linux")]
#[test]
fn unconfirmed_stop_clears_the_endpoint_key_once_the_processes_are_gone() {
// The other half: a stop that could not confirm termination leaves the
// key in place (the engine may still be alive and enforcing it) and
// records the intent. Once the processes are observed gone, the deferred
// cleanup runs, so no plaintext secret is stranded for a service the
// operator did ask to stop.
fn unconfirmed_stop_clears_the_endpoint_key_once_the_recorded_identity_is_recycled() {
let mut child = std::process::Command::new("sleep")
.arg("60")
.spawn()
.expect("spawn child");
let pid = child.id();
let (root, paths) = test_paths("liveness-pending-stop-clears-key");
let service_id = "svc-pending-stop";
let mut record = dead_public_service_with_key(&paths, service_id, 11983);
let real = rocm_core::process_start_ticks(pid).expect("start-ticks");
record.supervisor_pid = pid;
record.supervisor_start_ticks = Some(real.wrapping_add(1));
record.stop_requested_unix_ms = Some(1);

let changed = refresh_managed_service_runtime_liveness(&paths, &mut record);
Expand All @@ -644,12 +814,18 @@ mod tests {
assert_eq!(
endpoint_keys::endpoint_api_key(&paths, service_id),
None,
"a requested stop must still drop the key"
"a requested stop must drop the key once its identity is recycled"
);
assert_eq!(
record.stop_requested_unix_ms, None,
"the marker is consumed, so the cleanup does not run again"
);
assert!(
rocm_core::process_is_running(pid),
"the unrelated process holding the recycled pid remains alive"
);
let _ = child.kill();
let _ = child.wait();
let _ = fs::remove_dir_all(root);
}

Expand Down
7 changes: 5 additions & 2 deletions apps/rocm/src/service_launch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1078,8 +1078,11 @@ fn record_still_live(record: &ManagedServiceRecord, pid_alive: impl Fn(u32) -> b
if !crate::managed_service_status::managed_service_is_live(record) {
return false;
}
let pids = crate::managed_service_status::recorded_service_pids(record);
pids.is_empty() || pids.into_iter().any(pid_alive)
let identities = crate::managed_service_status::recorded_service_pids(record);
identities.is_empty()
|| identities
.into_iter()
.any(|identity| pid_alive(identity.pid))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: record_still_live now builds the identities and then drops the start-ticks (pid_alive(identity.pid)). It only ends the startup wait early, so the risk is small, but it is a second bare-PID check that the description does not list as an intentional exception (it names only the prune guard). Either pass the identity through or name it in the description.

}

/// [`record_still_live`] over the manifest on disk, so a `rocm services stop`
Expand Down
2 changes: 1 addition & 1 deletion apps/rocm/src/service_records.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ fn managed_service_record_is_in_use(record: &ManagedServiceRecord) -> bool {
managed_service_is_live(record)
|| recorded_service_pids(record)
.iter()
.any(|pid| process_is_running(*pid))
.any(|identity| process_is_running(identity.pid))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the prune/remove guard still uses bare process_is_running, so for a record whose PID was recycled the commands now disagree: services list shows it as not running, while services prune skips it as "still running, left alone" and services remove refuses it ("cannot be removed while it is running") with a stop hint that has nothing to stop. That is issue #614, and the description's "over-reporting in use is harmless" is easier to disprove now that list is corrected. recorded_service_pids already returns the identities, so using identity_state here (Matches/Indeterminate = in use) would keep the "a live server must not be deleted" intent from the comment above. Otherwise please link #614 as the follow-up and soften "harmless".

}

/// Remove one non-running local server record and everything it owns.
Expand Down
6 changes: 6 additions & 0 deletions docs/manual-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,12 @@ Expected result:
block at all.
- `rocm services list --all` shows saved history, including the failed or
stopped attempts the default view hides.
- On Linux, listing validates a recorded supervisor or engine PID against its
saved process start time. If the numeric PID now belongs to a newer process,
the record is persisted as `stopped`, disappears from the default live-only
list, and the newer process is left alone. If the start time cannot be read,
the service remains conservatively live; older records with no saved start
time keep their PID-only behavior.
- `rocm services logs <service-id> --follow` prints the recent output, then new
output as it arrives (a download bar redraws on one terminal row), and returns
by itself with `Service <id> is <status>; stopped following.` once the server
Expand Down
18 changes: 18 additions & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -1812,6 +1812,24 @@ a hooks build, as `cargo xtask e2e` makes: `cargo build -p rocm -p rocmd
--features rocm/e2e-test-hooks`, then the command above with `-n
"service-stop"`.

`features/service_record_cleanup.feature` (`service-cleanup-08` through
`service-cleanup-10`, Linux, mock lane) covers service liveness when a recorded
PID has been recycled. The scenarios record a real live decoy as the supervisor,
as the engine, and as both through a known supervisor token plus a legacy
tokenless engine alias.
Each stores a deliberately wrong start time, runs the real `rocm services
list`, and checks that the live-only output hides the record, the manifest is
persisted as `stopped`, and the decoy remains alive. Run only these scenarios
with:

```bash
ROCM_CLI_BINARY=target/debug/rocm cargo test -p e2e-cucumber --test e2e -- -n "service-cleanup-(08|09|10)"
```

These scenarios exercise liveness reconciliation, not record pruning or the
existing bare-PID prune behavior. Unit tests cover matching tokens remaining
live and tokenless legacy records retaining PID-only behavior.

Manual restricted-tool smoke:

```bash
Expand Down
Loading