Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

325 changes: 325 additions & 0 deletions apps/rocmd/src/watchers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -854,6 +854,23 @@ fn pending_stop_proposal_exists(paths: &AppPaths, service_id: &str) -> Result<bo
}))
}

/// Reads the whole proposals store rather than a recent window: a pending
/// proposal is never dropped, so it can be arbitrarily old, and a window that
/// missed it would let the watcher queue a duplicate once the original aged
/// out of that window. Compaction bounds the reviewed proposals to a fixed
/// count; pending ones are kept.
fn pending_restart_proposal_exists(paths: &AppPaths, service_id: &str) -> Result<bool> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dedupe only scans the 100 most recent proposals. A pending restart older than that is missed and the flood returns. Also, this watcher fix is outside ROCMAI-729's scope and has no Gherkin scenario for the new restart_proposal_already_pending event; please add one, state the justification in the PR text, or split it out.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e5b2361. The check now reads the whole proposals store (a pending proposal is never dropped, and compaction keeps the reviewed ones to 200), and a new test queues 101 newer proposals behind the pending restart; it fails against the 100-record window (a second restart is queued) and passes now. I also made an unreadable proposals store a reported restart_proposal_check_failed error event rather than an error out of the watcher, since that would end the daemon loop.

On scope and the scenario: I kept the fix here because the flood was also pushing older pending proposals out of the window other dedupe checks read. There is no scenario because the e2e harness has no step that drives a rocmd watcher tick against an unhealthy managed service, so it is covered by unit tests against the real proposals store; the PR text says so now. gpu-thermal-protect's pending_stop_proposal_exists has the same 100-record window and error propagation. I left it as it was, as outside this change, and noted that in the Scope section; it would be a separate change.

Ok(rocm_core::load_automation_proposals(paths)?
.into_iter()
.any(|proposal| {
proposal.status == "pending"
&& proposal.watcher_id == "server-recover"
&& proposal.service_id.as_deref() == Some(service_id)
&& (proposal.action == "queue_restart_proposal"
|| proposal.tool.as_deref() == Some("restart_server"))
}))
}

fn handle_cache_warm_event(
paths: &AppPaths,
mode: WatcherMode,
Expand Down Expand Up @@ -1225,6 +1242,43 @@ fn handle_server_recover_event_with_record(
Some(record.service_id.clone()),
),
WatcherPolicyAction::QueueProposal => {
// An unreadable proposals store is reported and skipped, never
// propagated: the daemon loop exits on an error out of
// `evaluate_watchers`, so a store that stays corrupt would take it
// down on every recovery tick.
match pending_restart_proposal_exists(paths, &record.service_id) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds user-visible events (restart_proposal_already_pending, restart_proposal_check_failed) and suppresses repeat restart proposals, but only unit tests cover it. User-observable behavior needs a scenario in tests/e2e-cucumber/features/, or a stated reason it cannot have one. It is also outside ROCMAI-729's scope: consider splitting it out, and mention it in docs/automations.md.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept in this PR, with the two things you asked for in lieu of a scenario: a note in docs/automations.md (a8b8669) naming the two events and the behaviour, and the reason there is no scenario stated in the PR text. The e2e harness has no step that drives a rocmd watcher tick against an unhealthy managed service, so the dedupe is covered by the four server_recover_* unit tests in this file, which run the handler against the real proposals store. I kept it here rather than splitting because the flood it stops was also pushing older pending proposals out of the window the other dedupe checks read, and because the store-size caps in this PR are only meaningful once something stops queueing a new proposal every recovery tick. Happy to split it out if you would still rather.

Ok(true) => {
return crate::persistence::record_event(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: while the restart stays pending this records an info event on every recovery tick (~30s). Previously each tick also logged an event, so this is not a regression, but the events store is now capped, so these entries will push gpu-metrics history out and crowd the "recent background activity" listing. Consider skipping or rate-limiting the event when a restart is already pending.

paths,
state,
"server-recover",
"info",
"restart_proposal_already_pending",
&format!(
"managed service {} needs recovery ({recovery_reason_display}); a restart request awaiting review is already queued",
record.service_id,
),
Some(record.service_id.clone()),
);
}
Ok(false) => {}
Err(error) => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trade-off worth stating in the PR: with a corrupt proposals store this arm logs an error event every tick and never queues a restart, where before the append path did not parse the store and queueing kept working. Skipping rather than propagating is right (an Err ends the daemon loop); just make the behavior explicit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stated in the PR text (the server-recover paragraph under "What changes") and in the new docs/automations.md note (a8b8669): while the proposals store stays unreadable, each recovery check logs a restart_proposal_check_failed error event and no restart is queued, because it cannot tell whether one is already pending. The error is reported rather than raised so the daemon loop keeps running.

return crate::persistence::record_event(
paths,
state,
"server-recover",
"error",
"restart_proposal_check_failed",
&format!(
"managed service {} needs recovery ({recovery_reason_display}); \
not queueing a restart request because the proposals store could \
not be checked for one already pending: {error:#}",
record.service_id,
),
Some(record.service_id.clone()),
);
}
}
let message = format!(
"managed service {} needs recovery ({recovery_reason_display}); queueing restart proposal",
record.service_id,
Expand Down Expand Up @@ -2234,6 +2288,277 @@ mod tests {
Ok(())
}

#[test]
fn server_recover_does_not_duplicate_pending_restart_proposals() -> Result<()> {
let (root, paths) = temp_app_paths("server-recover-dedupe");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-unhealthy",
"vllm",
"tiny",
"Tiny/Test",
"127.0.0.1",
11436,
"managed",
123,
None,
None,
None,
);
record.status = "unreachable".to_owned();
record.write()?;
let mut state = test_runtime_state(vec![test_watcher_snapshot(
"server-recover",
WatcherMode::Propose,
None,
)]);
let event = AutomationTriggerEvent {
at_unix_ms: 1,
kind: "service.manifest_recoverable".to_owned(),
source: "managed_service".to_owned(),
watcher_hint: Some("server-recover".to_owned()),
service_id: Some(record.service_id.clone()),
reason: Some("manifest_status_failed".to_owned()),
payload: json!({}),
};

handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
)?;
handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
)?;
let events = rocm_core::load_recent_automation_events(&paths, 2)?;
let proposals = rocm_core::load_recent_automation_proposals(&paths, 10)?;
fs::remove_dir_all(root).ok();

assert_eq!(proposals.len(), 1);
assert_eq!(proposals[0].tool.as_deref(), Some("restart_server"));
// The wording describes a proposal still awaiting review; the single
// pending proposal asserted here is what makes that true.
assert_eq!(proposals[0].status, "pending");
assert!(events.iter().any(|event| {
event.action == "restart_proposal_already_pending"
&& event.message.contains("a restart request awaiting review")
}));
Ok(())
}

#[test]
fn server_recover_dedupe_sees_a_pending_restart_older_than_the_recent_window() -> Result<()> {
let (root, paths) = temp_app_paths("server-recover-dedupe-old");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-unhealthy",
"vllm",
"tiny",
"Tiny/Test",
"127.0.0.1",
11436,
"managed",
123,
None,
None,
None,
);
record.status = "unreachable".to_owned();
record.write()?;
let mut state = test_runtime_state(vec![test_watcher_snapshot(
"server-recover",
WatcherMode::Propose,
None,
)]);
let event = AutomationTriggerEvent {
at_unix_ms: 1,
kind: "service.manifest_recoverable".to_owned(),
source: "managed_service".to_owned(),
watcher_hint: Some("server-recover".to_owned()),
service_id: Some(record.service_id.clone()),
reason: Some("manifest_status_failed".to_owned()),
payload: json!({}),
};

handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
)?;
// Enough newer proposals from other watchers that the restart proposal
// is no longer among the 100 most recent.
for index in 0..101 {
queue_proposal_with_arguments(
&paths,
"gpu-thermal-protect",
"queue_stop_server_proposal",
"Stop another server",
"unrelated proposal",
Some(format!("svc-other-{index}")),
json!({}),
)?;
}
handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
)?;
let restarts = rocm_core::load_automation_proposals(&paths)?
.into_iter()
.filter(|proposal| proposal.tool.as_deref() == Some("restart_server"))
.count();
fs::remove_dir_all(root).ok();

assert_eq!(restarts, 1);
Ok(())
}

#[test]
fn server_recover_dedupe_ignores_proposals_that_do_not_block_a_restart() -> Result<()> {
let (root, paths) = temp_app_paths("server-recover-dedupe-filters");
let mut record = ManagedServiceRecord::new(
&paths,
"svc-unhealthy",
"vllm",
"tiny",
"Tiny/Test",
"127.0.0.1",
11436,
"managed",
123,
None,
None,
None,
);
record.status = "unreachable".to_owned();
record.write()?;
// Each of these matches the restart check on all but one field, so
// none of them may suppress a new restart for `svc-unhealthy`.
for (watcher_id, service_id, status) in [
("server-recover", "svc-unhealthy", "approved"),
("server-recover", "svc-other", "pending"),
("another-watcher", "svc-unhealthy", "pending"),
] {
append_automation_proposal(
&paths,
&AutomationProposalRecord {
at_unix_ms: unix_time_millis(),
proposal_id: String::new(),
watcher_id: watcher_id.to_owned(),
action: "queue_restart_proposal".to_owned(),
title: "Restart".to_owned(),
message: "unrelated".to_owned(),
status: status.to_owned(),
service_id: Some(service_id.to_owned()),
tool: Some("restart_server".to_owned()),
arguments: json!({}),
reviewed_at_unix_ms: None,
},
)?;
}
let mut state = test_runtime_state(vec![test_watcher_snapshot(
"server-recover",
WatcherMode::Propose,
None,
)]);
let event = AutomationTriggerEvent {
at_unix_ms: 1,
kind: "service.manifest_recoverable".to_owned(),
source: "managed_service".to_owned(),
watcher_hint: Some("server-recover".to_owned()),
service_id: Some(record.service_id.clone()),
reason: Some("manifest_status_failed".to_owned()),
payload: json!({}),
};

handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
)?;
let queued = rocm_core::load_automation_proposals(&paths)?
.into_iter()
.filter(|proposal| {
proposal.watcher_id == "server-recover"
&& proposal.status == "pending"
&& proposal.service_id.as_deref() == Some("svc-unhealthy")
})
.count();
fs::remove_dir_all(root).ok();

assert_eq!(queued, 1);
Ok(())
}

#[test]
fn server_recover_reports_an_unreadable_proposals_store_instead_of_failing() -> Result<()> {
let (root, paths) = temp_app_paths("server-recover-corrupt-store");
paths.ensure()?;
fs::write(paths.automation_proposals_path(), "not a proposal\n")?;
let mut record = ManagedServiceRecord::new(
&paths,
"svc-unhealthy",
"vllm",
"tiny",
"Tiny/Test",
"127.0.0.1",
11436,
"managed",
123,
None,
None,
None,
);
record.status = "unreachable".to_owned();
record.write()?;
let mut state = test_runtime_state(vec![test_watcher_snapshot(
"server-recover",
WatcherMode::Propose,
None,
)]);
let event = AutomationTriggerEvent {
at_unix_ms: 1,
kind: "service.manifest_recoverable".to_owned(),
source: "managed_service".to_owned(),
watcher_hint: Some("server-recover".to_owned()),
service_id: Some(record.service_id.clone()),
reason: Some("manifest_status_failed".to_owned()),
payload: json!({}),
};

let outcome = handle_server_recover_event_with_record(
&paths,
WatcherMode::Propose,
&mut state,
&event,
&mut record,
);
let events = rocm_core::load_recent_automation_events(&paths, 1)?;
let stored = fs::read_to_string(paths.automation_proposals_path())?;
fs::remove_dir_all(root).ok();

// An error out of a watcher ends the daemon loop, so a corrupt store
// must surface as an event and a skipped tick, not as `Err`.
outcome?;
assert_eq!(stored, "not a proposal\n", "nothing may be queued");
assert!(events.iter().any(|event| {
event.level == "error" && event.action == "restart_proposal_check_failed"
}));
Ok(())
}

#[test]
fn local_webhook_gpu_metrics_event_uses_existing_read_only_policy() -> Result<()> {
let (root, paths) = temp_app_paths("local-webhook-gpu-metrics");
Expand Down
3 changes: 3 additions & 0 deletions crates/rocm-core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ libc.workspace = true
rand.workspace = true
regex = "1"
rsa.workspace = true
# File identity (inode / Windows file index), so the lock-free JSONL readers in
# `automation.rs` can tell when a store rotation renamed a file under them.
same-file = "1"
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true
Expand Down
Loading
Loading