Skip to content

fix(core): stop managed services through one rocm-core routine shared by rocm and rocmd, with Windows identity, liveness and tree-kill hardening; adds services remove --force, System32 tool resolution (incl. ComfyUI) and a stricter failed-launch cleanup - #960

Merged
jussielo-amd merged 40 commits into
mainfrom
split-managed-service-stop-routine
Nov 10, 2026

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes ROCMAI-874.

Summary

Besides the shared stop routine, this PR contains changes a reader should know about up front, named here because they are not obvious from the title alone: rocm services remove --force (with prune listing the record and naming that command), the ROCM_E2E_UNQUERYABLE_PIDS e2e hook, System32 resolution of the Windows tools, including ComfyUI's own tasklist and taskkill calls, and a changed failed-launch cleanup on Windows. They are kept in one PR because they share the one routine and its Windows tests, and none of them stands alone without a Windows host to show it: remove --force and the e2e hook exist only because the routine now reports a process it cannot query as unverified, and without them that record could never be cleared or tested (the hook is what lets service-unqueryable-01 and -02 plant such a process on Linux); the Windows start token is what lets the routine's identity gate refuse a recycled PID before taskkill /T takes its tree, so shipping the tree kill without it would widen the very exposure the ticket is about; System32 resolution is the same routine's taskkill and tasklist calls resolved from the system directory, applied to ComfyUI's calls that go through the same helper so there is one rule for it; and the failed-launch cleanup is the third caller of the termination, which would otherwise keep a Windows tree-kill gap that the other two callers have just lost. Splitting them would leave the first PR with either a tree kill that can hit a recycled PID or a state no test can reach. Each is described below with the test that covers it.

What was and was not executed. Nothing in this PR has been run on a Windows host by me, and this text does not claim a Windows result for this head. From .github/workflows/ci.yml: the windows-debug job, when Rust files change, runs clippy, the build, cargo test --workspace --all-targets and the harness self-tests. That test run includes two #[cfg(windows)] unit tests in proc_lifecycle.rs (terminate_recorded_pid_leaves_a_recycled_pid_alone_and_stops_the_verified_one and terminate_recorded_pid_stops_a_process_recorded_without_a_token) that start a real ping process and stop it through terminate_recorded_pid, so the PID hold, the creation-time read, the identity gate (a wrong token leaves the process running) and the System32 taskkill /T /F run on a real process there. It does not run any scenario (the e2e target is excluded from cargo test): the service-stop-* and service-unqueryable-* scenarios do not run on that job, so service-stop-07 (@requires-os:windows) and the Windows half of service-stop-08 can only run through the harness on a Windows host, and docs/manual-testing.md has a Windows host check that selects only the scenarios that apply there (service-stop-0[178] and service-unqueryable-01). Not exercised by any CI job: the end-to-end tree kill of a parent with a child, the failure cases that cannot be staged (an OpenProcess access-denied process, taskkill exiting 128, a failing or stalled taskkill), and the launch-cleanup branch; the ping those unit tests stop has no child of its own, so they do not show a descendant being taken down.

rocm services stop (and rocm services restart and the sandbox stop_server tool, which stop first) and rocmd's stop_server each carried their own copy of the code that terminates the processes a managed-service record names: a private MANAGED_STOP_GRACE of 10 seconds that was kept in step with the other copy only by a comment, and a private terminate_recorded_service_pids. The two copies had drifted. rocmd reported the stopping process's own PID as a self row and rocm silently dropped it, so the same record produced a different pid_outcomes list from each binary. On Windows only rocmd ran taskkill /PID <pid> /T /F before terminating the recorded process, even though the engine runs one level below that PID and the verified termination cannot walk a process tree there, so rocm services stop could leave the engine behind. rocmd also cleared the recorded PIDs and start-time tokens from the record once a stop was confirmed, while rocm left the record naming processes that were already gone.

This change moves one routine and one constant into rocm-core's proc_lifecycle.rs, next to terminate_verified, and deletes both private copies. terminate_recorded_service_pids walks the processes a record names (supervisor first, then engine, each paired with its own start-time token, a PID recorded under both roles visited once) and returns one RecordedPidStop per PID; terminate_recorded_pid terminates one of them and, on Windows, kills its descendants first; MANAGED_STOP_GRACE is the single grace period. The per-PID termination is passed in, so each caller's tests drive its own stop through the real routine with the signal replaced, and the Windows steps sit behind a function that takes the tree kill, the PID hold, the identity read and the termination as arguments so its tests run on every host. rocm keeps only a thin stop_recorded_processes that adds the existing e2e-test-hooks seam. Each caller has a source check that no private copy of the routine, the constant or a taskkill call has come back; those checks pin names only, so they are a tripwire for a copy returning under the same name, not coverage of what either stop does.

Intended user-visible differences, where the two behaviours disagreed and the more complete one (rocmd's) won:

  • A confirmed rocm services stop now clears the supervisor and engine PIDs and start-time tokens from the record (one ManagedServiceRecord::clear_recorded_processes, called by both binaries), so a later stop of the same record reaches no stale PID. An unconfirmed stop still keeps them, because the record is the only thing that names a survivor.
  • rocm services stop|restart, the sandbox stop_server/restart_server tools and the attached rocm serve Ctrl-C stop now report the stopping process's own PID as a self row, like rocmd, and on Windows take the engine's subtree down with the recorded process.
  • On Windows the start-time identity token is now the process creation time (GetProcessTimes), where it was always absent. A record written by this version therefore refutes a PID that an unrelated process has since taken: the stop reports identity_mismatch, skips the tree kill and leaves that process alone, instead of killing a stranger and its subtree. A record written before this has no token and stays best-effort. A recorded token that can no longer be read back from a running process reads as unverified and is not signalled. That is one of two routes to an unverified stop on Windows; the other is a PID that cannot be queried at all (next bullet). The advice a stop prints for an unverified process names tasklist /FI "PID eq <pid>" on Windows, where it used to name ps -p, which Windows does not have.
  • On Windows the stop does not report success on three paths where it cannot tell. A live process the stop may not open is reported unverified rather than already gone, with or without a recorded token, because a failed query is no longer read as an exit (ProcessProbe; Linux already counted a live process it may not signal as running). A recorded token that can still be read and differs outranks the failed query, but on Windows the token and the query are read with the same access right, so that only reaches a process that opens and then fails the exit-code query. The same rule holds for what the stop leaves behind: the later liveness refresh (the deferred endpoint-key cleanup and the reconciliation of the record to stopped) and the in-use check behind rocm services remove and prune take a recorded process for gone only when probe_process confirms it, so an unverified stop's key and stop request survive a service listing. When that refresh does complete the pending stop (every recorded process confirmed gone), it also forgets the recorded processes, only for a record that named at least one (a starting record with a stale marker has its key and marker cleaned but is not read as stopped), and reads the record as stopped whatever it read before (a failed record too, as a confirmed stop sets it, so a stop still finalizing covers it), as a stop that confirmed the exit itself does, so a later rocm services stop visits no PID: a PID recorded without a token cannot be refuted, and on Windows the tree kill would take whatever process now holds the number. A process that may still be running stays named. A pending stop of a container-backed engine (engine_stop_decides_confirmation, today aim) is never completed this way, is not read as stopped by the rest of the liveness refresh, and counts as in use for remove and prune (container_stop_unresolved; a_pending_container_stop_survives_refresh_remove_and_prune runs a refresh, remove --yes and prune --yes over it, for a record reading ready and one already reading stopped): its supervisor can be gone while the container still runs, so only the engine Stop may confirm it, and its key, marker and recorded processes stay for the next services stop (a_pending_stop_of_a_container_engine_is_not_completed_by_a_gone_supervisor; the situation is reachable because an aim launch records the supervisor and docker run -d backgrounds the container itself). Both stops (rocm services stop and rocmd's) decide whether their verdict still covers the record they re-read with ManagedServiceRecord::is_covered_by_stop_of: the same processes, or none recorded because a listing completed this same stop first (status stopped, nothing named), so (with an empty stopping snapshot too: an empty record is that stop only while it carries the stop's marker, reads stopped, or a listing consumed that very marker, which the refresh records as the marker's value in settled_stop_unix_ms because it leaves a starting or failed record at its status; an empty starting relaunch has cleared the marker without leaving the value, and a restart wipes it, so it is not covered) a listing that persists the cleanup between the termination and the final read does not turn a successful stop into an unconfirmed one or make services restart refuse, including for a stop that began on a record naming no process. A restart is still not covered, whether it recorded new processes or is mid-launch with none yet (starting), and so is the empty stopped record a rocm services restart leaves between its own confirmed stop and the publication of its child: the restart stamps a relaunch generation (relaunch_started_unix_ms) on it before it restores the endpoint key or spawns, a stop covers a record only in the generation it read, and publishing the child clears the stamp, so an older stop still finalizing in that window neither claims the record nor deletes the restored key (the restart's own writes also refuse a record whose generation moved since it read it; a restart that fails after stamping leaves a stamp that every later stop's snapshot carries too, so it blocks nothing). The cost is stated rather than hidden: if a stopped service's recorded PID is later taken by a process the account cannot open, the record stays unverified for good (it never reconciles to stopped, remove and prune refuse it, and a stop keeps reporting unverified). The way out is rocm services remove <service-id> --force --yes (for a record that still reads ready and that nobody has stopped, --force is refused and names rocm services stop first, which reports the process unverified and records the request; that order is intended, because --force never removes a record nobody tried to stop, and a_ready_record_held_by_an_unqueryable_process_is_cleared_by_stop_then_force runs the two commands), deliberately narrow: it removes the record only when a stop was requested, no recorded process is confirmed running, and at least one could not be queried; it says which process and that, if it is still running, it keeps running; it never goes past a process rocm can see running; the plain refusal names this exact command, and prune has no --force, but when it leaves such a record alone it lists it with the exact rocm services remove <id> --force --yes command (a test runs the named command and checks it clears the record), and a forced removal that is missing --yes is told to add --yes, not to drop --force. It is user-observable, so service-unqueryable-01 drives the stop (unverified, with the check named), the refusal naming --force, and the forced removal through the real binary, and service-unqueryable-02 (Linux) shows --force refused for a real running process; both use the e2e-test-hooks variable ROCM_E2E_UNQUERYABLE_PIDS, which production builds never read. I looked at reading the creation time through a process snapshot instead, to tell a reused PID apart without opening the process: NtQuerySystemInformation(SystemProcessInformation) carries a create time without a per-process open, but it is only semi-documented, its buffer layout is not something I could check without a Windows host, and a wrong token would flag a real engine as recycled, so I did not do it; the Toolhelp snapshot has no creation time. The PID is held open from before the identity check until the stop returns, so the process whose creation time was compared is the one the kill reaches. And a taskkill that failed to start, did not exit cleanly, or did not finish within MANAGED_STOP_GRACE (it is killed and reaped at that deadline, so a stalled helper cannot hang the stop) leaves the stop unconfirmed, with the record, the stop request and the endpoint key kept, because only the recorded process's own exit is watched on Windows. When this stop's own wait saw the root exit (or it was already gone) the outcome is descendants_unconfirmed, printed as not confirmed: process <pid> (<role>) exited, but a process under it may still be running; timed_out, whose text says something outlived a deadline, is kept for a root that did. A taskkill that exits 128 (process not found) is judged by the root's own re-read instead. If the root is still running, the kill counts as failed (descendants_unconfirmed when the termination then saw the root exit). If the root exited by itself between the identity check and the kill, the outcome is exited_before_tree_kill: taskkill /T reaches children only through a live parent, so a child the root left running is out of its reach, and nothing recorded what was under the root. That is not a stop on its own. Across the record, terminate_recorded_service_pids turns it into already_gone only when another recorded process is confirmed gone, which relies on the engine being visited as its own role (it is), and both callers fold every row, so the record, PIDs and key are cleared only when every recorded role is confirmed. A root that is the only recorded process, or whose engine is running or unconfirmed, stays unconfirmed and prints not confirmed: process <pid> (<role>) was already gone when the processes under it were to be taken down; the next liveness refresh still completes the deferred cleanup once the recorded PID is seen gone, and that now includes forgetting the recorded PID. The shared taskkill, and ComfyUI's taskkill and tasklist, are started from the system directory (windows_system_tool, which reads it with GetSystemDirectoryW, falling back to the bare name when it cannot be read) so neither a PATH entry nor a SystemRoot variable can substitute them. Independent verification of descendant exit there is not part of this change.
  • On Windows the identity read that gates the tree kill is also the stop's verdict: when it refuses (gone, recycled, unverifiable) the root is not handed to the termination, which re-reads the identity and could find a process that was unverifiable a moment ago verifiable, then stop the root alone with the tree kill skipped. The same applies to the PID hold: the hold is an OpenProcess handle that keeps Windows from giving the PID to another process, and when it cannot be taken but the identity reads as matching, nothing reserves the PID between the read and taskkill, which takes a whole tree by PID, so the stop is reported unverified and neither the tree kill nor the termination runs (a gone, recycled or unreadable PID keeps its own refusal; where there is no separate tree-kill step the missing hold changes nothing). A sibling whose PID a stranger now holds (identity_mismatch) does not vouch for what was under a root that exited by itself; only a stop, an already-gone process or another such root does.
  • The stop's identity decision is made in one function (classify_live_identity, with refusal_before_signalling), so a table test covers every combination of recorded and live token on every host.
  • The failed-launch cleanup in service_launch/launch_transaction.rs (abandon_failed_managed_launch) said Windows records no start token and only refused a child without one on Linux. Windows records one now, so a child whose token was not captured is refused there too, through a pure launch_cleanup_refusal tested on every host, instead of falling through to a forced tree kill of a pid that may have been recycled. The comments that repeated the old claim were corrected. The cleanup also terminates the child through the same routine a stop uses (rocm_core::terminate_recorded_pid_within, the routine with the root's grace chosen by the caller, so the cleanup keeps its 2 second grace) rather than terminate_verified alone: on Windows terminate_verified cannot walk a process tree, so a failed launch could leave the engine one level below the supervisor running, which is the defect the stop had. The identity rule is unchanged and decided first (launch_cleanup_refusal). What the cleanup asks of the termination is tested through an injected function on every host (abandoned_launch_asks_the_termination_for_the_spawned_child_with_its_token); that the production wiring reaches the Windows tree kill is not observable on Linux, where the termination walks the tree itself, so it rests on the routine's own tests and review, and no test runs a failed launch against a real Windows child. The tradeoff is deliberate and written in the code: a child whose creation time could not be read at capture is not killed, so if OpenProcess failed for a child that is in fact alive (it is this account's own fresh child, so that is not expected), a failed launch leaves its engine running and the audit event says unverified; the alternative is a forced tree kill of a PID that may now belong to another process. A handle-based identity would need the spawn primitive to return its handle, which it does not, so it is not done here.
  • The three destructive decisions made from a recorded process's liveness (the deferred key cleanup, the reconciliation to stopped, and the in-use check behind remove and prune) share one reading, recorded_pids_may_be_running, so they cannot drift apart on what counts as gone.

The commits are in reading order: characterisation tests that pin what each copy did before the change, the merge into the shared routine, the record clearing, the Linux scenario for it, the Windows creation-time identity, the identity-verdict table, a teardown for the Windows scenario's processes, the renumbering of the last scenario so the feature's ids stay sequential, the Windows hardening described in the last bullet above, the follow-up that makes the liveness refresh and the removal guard read a process that cannot be queried as possibly running and bounds the taskkill by a deadline, a further one that treats taskkill exit 128 as a stop when the root is gone, lets a readable token mismatch outrank a failed query, and documents the two Windows routes to an unverified stop and the way out, one that verifies the launch cleanup's token on Windows and corrects the comments that said Windows records none, one that gives a root seen to exit under a failed tree kill its own outcome, resolves the Windows tools under System32 and shares the may-be-running reading, and one that settles a root that exited alone across the record and adds the narrow remove --force; one that refuses an unverifiable first identity read before the tree kill can be skipped, restricts the corroboration, makes prune name remove --force, and corrects the claims about where the Windows paths run; one that keeps the marker's value when a listing consumes it off a record naming no process, so a stop that began on such a starting or failed record is still confirmed (settled_stop_unix_ms, cleared by a restart); one that makes the rocmd test of that run its listing in the engine stop callback, where a record naming no process can reach it; and one that stamps a relaunch generation on the empty stopped record a restart leaves before it publishes, so an older stop finalizing in that window cannot claim it or delete the restored key; the merges of main are separate commits.

Scenarios: service-stop-06 (rocm services stop of a real recorded process confirms the stop and leaves no process in the record; Linux, mock lane), service-stop-07 (Windows only: stopping a server takes down the recorded process and the process it started, where a tasklist that does not exit successfully now fails the scenario instead of reading as an exit; no CI job runs scenarios on Windows, and it was not run on a Windows host), and service-stop-08 (stopping a server whose recorded PID now belongs to a different, running process leaves that process alone and still clears the record; runs on every OS). Scope of the Windows rule beyond the two stops: the unqueryable-process rule lives in identity_state, so it reaches every caller of terminate_verified; the PID hold and the tree kill live only in terminate_recorded_pid and reach only the rocm and rocmd stops. I read each caller. The lemonade engine's stop_service (and vLLM's, which native Windows does not run) now answers stopped: false and writes no terminal state for an unqueryable recorded PID, where it would have claimed a stop; the CLI's own stop reaches the same verdict for the same PID. rocmd's terminate_recovery_supervisor signals a child it just spawned and still holds, so the PID is openable and reserved. The launch cleanup is the bullet above. I also read every consumer of the outcome strings and every remaining process_is_running call site: the stop text in services_text.rs is the only renderer of per-process outcomes (it now prints descendants_unconfirmed and exited_before_tree_kill lines); rocmd's MCP and sandbox answers pass pid_outcomes through and key on the stopped boolean, as do its watchers; the remaining process_is_running callers watch a child this command just started (launch waits, attached log streams), the helper daemon's own PID, or ComfyUI, which has its own check, and none deletes a key or a record on a failed query. None needed a further change. The scenario teardown also ends the process that 07's recorded process started when it is still that process, so a failed 07 does not leave it running behind a dead parent; that decision (leftover_action) has a table test that runs on every host. I was told that service-stop-07 failed once in the original combined PR on a self-hosted Windows lane and passed on a re-run; I have not seen that result myself, and the scenario and the pure identity-verdict tests are unchanged. service-unqueryable-01 and -02 (their own feature file; -02 Linux-only) cover the unqueryable-process stop, the refusal naming --force, prune naming it, and the forced removal through the e2e hook; they ran on Linux only, and -01 carries no OS tag but no CI job runs it on Windows. The other new Windows failure paths (a failed, stalled or not-found taskkill) have no scenario: they cannot be staged from outside the binary, so they are covered by unit tests with the system reads injected.

Four gaps in the scenario coverage, stated plainly. service-stop-07 records no token, so the path a record written by this version takes on Windows (a token recorded, matching, then the tree kill) has no scenario; it rests on what_a_stop_does_for_each_recorded_and_live_token and the other injected tests, and on the Windows-only unit tests that stop a real process with its true token. And the self row that rocm now puts in pid_outcomes has no scenario, deliberately: it appears only in the JSON answer of the hidden sandbox-run stop_server and the sandbox tool, never in the text of rocm services stop, and it names the stopping process's own PID, which a scenario cannot plant in a record before the process that will read it exists. a_stop_reports_its_own_pid_as_self and a_self_row_adds_no_line_to_an_unconfirmed_report cover the answer and the report. Third, the check: line an unverified stop prints (ps -p <pid> -o pid,args, or tasklist /FI "PID eq <pid>" on Windows) is a diagnostic, not a remediation: it shows whether the process is still running, and what clears the condition is the process ending (after which a stop or a listing confirms and drops the key, asserted by a_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried) or, for a record held by a process rocm cannot open, rocm services remove <id> --force --yes, which the refusal and prune both name. The command's wording is pinned by the_process_check_command_is_the_one_the_host_has, and service-stop-07 uses the same tasklist filter, in CSV form because the default output is localised, to see whether a process is running; no test follows the printed command on a Windows host and reads its default output. Fourth, the interleavings of a listing with a stop that began on a record naming no process, and of an older stop with a restart between its own stop and its publication (the stop's key deletion is what the generation stamp prevents): it depends on a listing landing between the stop's termination and its final read, which a scenario cannot place, so it is covered by a_listing_clearing_the_marker_of_an_empty_stop_does_not_make_it_unconfirmed (a real refresh persisted at that point), a_relaunch_after_a_listing_consumed_the_marker_is_still_not_the_stop, the is_covered_by_stop_of table and rocmd's a_stop_of_an_empty_record_whose_marker_a_listing_consumed_is_still_confirmed; an_older_stop_finalizing_before_a_restart_publishes_does_not_delete_its_key runs the second as an outer stop whose termination step runs a whole restart with its spawn failed on purpose, so the record is left as it is at the spawn; none of the existing service-stop scenarios covers either. I have not seen any Windows result for this head, and this text does not claim one.

Split from #929 at the reviewer's request; the other parts are #961 (log-tail routine), #959 (engine IPC client) and the respawn guard, which follows once this and #959 have merged.

Test plan

Ran locally on Linux, in one cargo target directory. The first bullets were run on an earlier head; the bullet headed "On the current head" gives the figures for the head after the last fix and the last merge of main:

  • cargo test -q --locked -p rocm-core (1351 passed, 6 ignored), including proc_lifecycle (54 tests), process::tests and service_record.
  • cargo test -q --locked -p rocmd (237 passed, 1 ignored; one run of the whole suite showed supervise::tests::recovery_ignores_its_own_live_looking_record_on_its_own_port failing once, and it passed alone and in two further whole-suite runs, so it looks timing-dependent and is not in a file this change touches).
  • cargo test -p rocm --bin rocm filtered to service_control (46 passed), services_text (39 passed), service_launch (72 passed, 1 ignored), managed_service_status (15 passed) and service_records (31 passed) on the final head, and comfyui (105 passed, 4 ignored), cli (97 passed, 1 ignored), help (24 passed) and advised (20 passed, 1 ignored) on the head before the last merge. I did not run the whole rocm test binary or the workspace suite.
  • The real e2e harness against a debug rocm/rocmd built with --features rocm/e2e-test-hooks, -n "service-stop-0[1-68]": 7 scenarios passed (01 to 06 and 08; 07 is Windows-only and was not selected); -n service-unqueryable-0: 2 passed (Linux); -n service-cleanup: 7 passed. cargo test -p e2e-cucumber --test feature_naming: 4 passed.
  • cargo fmt --all --check, cargo clippy --locked -p rocm-core -p rocmd -p rocm --all-targets -- -D warnings, cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings, cargo xtask manifest --check, cargo xtask check-crate-edges, cargo xtask check-architecture-doc, cargo xtask lychee (local links) and cargo test -q --locked -p xtask prepush (98 passed) all exit 0. cargo xtask check-module-size reports no file over the limit (412 files checked; this change adds nothing to services_text.rs, which stays at 1457 of 1500 production lines, or to process.rs, 1411), now that main has split service_launch.rs into service_launch/.
  • Windows target (x86_64-pc-windows-msvc): cargo clippy with -D warnings for rocm-core, rocmd and rocm (all targets) and e2e-cucumber (all targets) exits 0 on the final head, merge included. This is compile and lint only.
  • Leak scan of the diff against origin/main: the only hits are existing internal_* identifiers; no conflict markers.
  • On the current head (the two fixes after the merge of main that brought in refactor(engine-ipc): share one engine stdio client between rocm and rocmd #959): cargo fmt --all --check; cargo clippy -D warnings --all-targets for rocm-core, rocm, rocmd, e2e-cucumber, rocm-engine-protocol and xtask; the whole workspace for x86_64-pc-windows-msvc with -D warnings (compile and lint only); cargo xtask check-module-size and check-architecture-doc; cargo test -p xtask prepush (98 passed); cargo test --no-fail-fast for rocm-core --lib (1449 passed, 6 ignored), rocmd (278 passed, 1 ignored) and rocm --bins (1828 passed, 8 ignored), all whole suites. -n service-stop through the real harness: 7 of 8 scenarios passed; service-stop-07 (Windows only) fails on this WSL host in its first step because powershell is not on PATH here, before it reaches any code of this change. No Windows host was used.
  • Evidence that the tests fail without the change: the first commit pins what the two private copies on main did (the grace, the visiting order, the token pairing, the zero-PID case, the per-outcome classification) with no production change, and the merge into the shared routine has to edit those tests; against the pre-change rocm copy, a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing fails (its record keeps naming the stopped PIDs) while an_unconfirmed_stop_keeps_the_recorded_processes_for_the_next_stop passes. The tests of the new API (the Windows identity table, ProcessProbe, TreeKill, the hold, is_covered_by_stop_of, tasklist_reports_running) call functions that do not exist on main, so they cannot run against a copy of it; for those the mutation checks below are the evidence that each test goes red when the branch it covers is broken.
  • A tree kill that did not complete (descendants_unconfirmed, exited_before_tree_kill) is a warning for one run, and the printed line now says so: rocm does not keep it, so the next services stop or listing finds the root gone and reports the stop confirmed, with no new evidence about what was under it. The test asserts the line and then runs that second stop and asserts it is confirmed, so the sentence is checked against the state; a durable marker in the record was the alternative and is not done.
  • The clap value enums moved out of cli.rs into cli_args.rs as a pure move (re-exported, no other change), because cli.rs is now checked by a blocking size check and main keeps growing it; it is at about 1348 production lines, services_text.rs stays at 1457 and process.rs is at 1437 with the system-directory read (it was 1411).
  • Mutation checks, each restored afterwards: removing the clear_recorded_processes call from rocm's confirmed branch turns a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing red; mapping an unreadable live token to a match in classify_live_identity turns what_a_stop_does_for_each_recorded_and_live_token and a_recorded_token_refutes_a_live_one_only_when_both_are_known_and_differ red; reading an unqueryable PID as gone turns a_pid_that_cannot_be_queried_is_unverified_not_already_gone and a_stop_of_a_live_process_it_cannot_query_is_not_confirmed red; reading every failed open as gone turns a_failed_open_is_gone_only_for_no_such_process red; letting a failed tree kill keep the root's verdict turns a_tree_kill_that_did_not_complete_leaves_the_stop_unconfirmed red; accepting any taskkill exit code as clean turns only_a_clean_taskkill_exit_is_a_completed_tree_kill_and_128_is_not_found red; dropping the not-found handling turns that test and a_tree_kill_that_found_no_process_is_a_stop_when_the_root_is_gone red; dropping the rule that a readable token mismatch outranks a failed query turns a_readable_token_mismatch_outranks_a_failed_query red; reading a process that cannot be queried as not in use in the removal guard also turns a_record_held_by_an_unqueryable_process_can_be_cleared_by_hand_and_prune red; releasing the PID hold immediately turns five ordering tests red, including the_pid_is_held_before_its_identity_is_read_and_released_after_the_stop; removing the forgetting of the recorded processes from the deferred cleanup turns a_stop_completed_by_a_later_listing_leaves_no_pid_for_the_next_stop, a_completed_pending_stop_reads_stopped_whatever_live_status_it_had and a_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried red, forgetting them before the confirmed-gone check turns that last test and two neighbours red, dropping the status it sets turns the status test red, and reading a failed tasklist as a missing row turns a_failed_run_is_no_answer_whatever_it_printed red; reading the stop's re-read as a relaunch whenever the recorded processes differ (the previous comparison) turns a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it, a_restart_is_not_refused_because_a_listing_completed_its_stop and a_stop_completed_by_a_concurrent_listing_is_still_confirmed red, covering a record that names no process whatever its status turns the two mid-launch tests and the table red, and covering every record turns the relaunch tests red; making the failed-launch cleanup drop the spawn token from the identity it asks the termination about, or treat a host as one that records no tokens, turns abandoned_launch_asks_the_termination_for_the_spawned_child_with_its_token and abandoned_launch_never_signals_a_pid_that_no_longer_matches_the_spawn_token red; setting stopped in the deferred cleanup only for a live status turns a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it (its failed case) and a_completed_pending_stop_reads_stopped_whatever_live_status_it_had red; removing the container-engine guard from the deferred cleanup turns a_pending_stop_of_a_container_engine_is_not_completed_by_a_gone_supervisor red; dropping the stop-requested condition of --force turns a_ready_record_held_by_an_unqueryable_process_is_cleared_by_stop_then_force and force_is_refused_for_a_running_process_and_without_a_requested_stop red; removing the container check from the remove/prune guard, or from the refresh's status reconciliation, turns a_pending_container_stop_survives_refresh_remove_and_prune red; removing the no-process condition from the deferred cleanup turns a_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stopped red; accepting an empty stopping snapshot's equal empty record without the marker turns the is_covered_by_stop_of table and stop_does_not_confirm_over_an_empty_relaunch_of_an_empty_record red; and dropping the printed "rocm does not keep this warning" sentence turns the unconfirmed-tree-kill report test red; leaving no stamp when the refresh consumes a marker off an empty record turns a_listing_clearing_the_marker_of_an_empty_stop_does_not_make_it_unconfirmed (its starting and failed cases) and a_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stopped red while stop_does_not_confirm_over_an_empty_relaunch_of_an_empty_record stays green, keeping the stamp across a restart turns a_relaunch_after_a_listing_consumed_the_marker_is_still_not_the_stop and restarting_forgets_a_settled_stop red, and matching an absent stamp against an absent marker turns the is_covered_by_stop_of table red; dropping the settled-stamp clause from is_covered_by_stop_of turns a_stop_of_an_empty_record_whose_marker_a_listing_consumed_is_still_confirmed in rocmd red for both statuses (its listing now runs in the engine stop callback, with the interleaving asserted); dropping the generation check from is_covered_by_stop_of, or having the restart not stamp, turns an_older_stop_finalizing_before_a_restart_publishes_does_not_delete_its_key red; removing the failed-hold refusal turns a_tree_kill_is_not_run_for_a_pid_that_could_not_be_held_even_when_its_identity_matches red, and applying it also where there is no tree-kill step turns an_unheld_pid_that_is_gone_recycled_or_unverifiable_keeps_its_refusal and without_a_tree_kill_step_an_unheld_pid_is_still_terminated red; reading a process that cannot be queried as gone in the deferred key cleanup turns a_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried and an_unverified_stop_keeps_its_key_and_marker_through_a_later_liveness_refresh red, the same in the status reconciliation turns the first red, and in the removal guard turns a_record_is_in_use_unless_every_recorded_process_is_confirmed_gone red; not killing and reaping a stalled taskkill at its deadline turns a_helper_that_stalls_is_killed_and_reaped_at_its_deadline red; mapping a root seen to exit under a failed tree kill back to timed_out turns the three tree-kill outcome tests red, and dropping the printed line turns a_stop_with_an_unconfirmed_tree_kill_says_what_happened_and_keeps_the_service red; ignoring the system directory in system_tool_path turns a_system_tool_is_resolved_under_the_system_directory red; reading an unqueryable process as not running in the shared helper turns recorded_pids_may_be_running_unless_every_recorded_process_is_confirmed_gone and the earlier tests of the three decisions red; an unverifiable first read followed by a verifiable one turning into a root-only stop (reverting the early refusal) turns an_unverifiable_first_read_is_not_overtaken_by_a_verifiable_second_read and four neighbours red; accepting identity_mismatch as corroboration turns an_identity_mismatch_sibling_does_not_corroborate_a_root_that_exited_alone red; reading STILL_ACTIVE as gone turns an_exit_code_query_is_running_gone_or_unqueryable red; dropping --force from the missing---yes hint turns a_forced_removal_missing_yes_names_the_command_that_completes_it red, and prune not naming the forced removal turns prune_names_the_forced_removal_for_a_record_held_by_an_unqueryable_process and service-unqueryable-01 red; dropping the launch-cleanup refusal turns launch_cleanup_refuses_a_child_without_a_token_only_where_tokens_exist red; confirming a root that exited alone without a corroborating process turns the lone-root and engine-not-gone tests red in rocm-core, rocm and rocmd; dropping the stop-requested condition of --force turns force_is_refused_for_a_running_process_and_without_a_requested_stop red, dropping the running-process check turns force_is_refused_when_one_process_is_unqueryable_and_another_is_running red, and making --force do nothing turns service-unqueryable-01 red.

Not run: nothing on a Windows host, so the Windows code paths themselves (the creation-time read, taskkill resolved under System32, the Windows launch-cleanup refusal, whether taskkill really exits 128 for a missing process on the Windows builds in use, the process handle that holds the PID, the OpenProcess error codes, the real taskkill and its failures and deadline, service-stop-07 and the Windows half of service-stop-08, and the Windows branch of the process-check advice) have not been executed on a Windows host. The windows-debug job's unit tests do run the happy paths of the PID hold, the creation-time read and the System32 taskkill against a real ping process (the two #[cfg(windows)] tests named above), without a result for this head being claimed here; no CI job runs the failure side (OpenProcess access denied, taskkill exit 128, a failing or stalled taskkill), the tree kill of a parent with a child, or the launch-cleanup of a real child. What was moved out of the FFI so that it is unit-tested on Linux and, as plain cargo test, on the Windows unit-test job (not yet seen to pass there): the OpenProcess failure-code mapping, the exit-code query mapping, the taskkill result and arguments, the System32 path building, the FILETIME join, and the deadline wait (a long-sleeping child on Unix, ping on Windows). The ordering of the PID hold, the identity read, the tree kill and the termination, the identity verdicts, the liveness gate, the liveness refresh and removal guard are tested through injected functions that also compile for the Windows target. cargo test --workspace --all-targets, scripts/smoke_local.py and the coverage floors are left to CI.

…its recorded PIDs

rocm (service_control.rs) and rocmd (service.rs) each carry their own
MANAGED_STOP_GRACE and terminate_recorded_service_pids, and the two have
diverged. Pin what each copy does today, with no production change, so the
change that merges them has to edit the tests that describe the difference:
the grace value (10 s in both), the order the PIDs are visited
(supervisor, then engine), the per-PID start-time pairing and the
de-duplication of a PID recorded under both roles, zero PIDs, and the
per-outcome signalled/forced/stopped classification.

The two divergences are pinned as such: rocm drops the stopping process's own
PID from the report while rocmd reports it as `self`, and the Windows
`taskkill /T /F` step exists only in rocmd. The latter is a private
cfg(windows) block with no injection point, so it cannot be observed from a
test yet; only its portable half (an absent PID reaches terminate_verified
unchanged) is pinned here.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…by rocm and rocmd

`rocm services stop` and `rocmd`'s `stop_server` each carried their own
MANAGED_STOP_GRACE (10 s, kept in step only by a comment) and their own
terminate_recorded_service_pids, and the copies had diverged. Move one routine
and one constant into rocm-core's proc_lifecycle.rs, beside terminate_verified,
and delete both private copies. The termination of each PID stays injectable.

rocmd's behaviour wins, because it is the more complete one:

- The stopping process's own PID is reported as `self` (never signalled,
  counted as stopped). rocm used to drop it, so `pid_outcomes` listed one
  entry fewer than rocmd's for the same record.
- On Windows a verified root is preceded by `taskkill /PID <pid> /T /F`,
  because terminate_verified cannot walk a process tree there and the engine
  runs one level below the recorded PID. rocm never did this, so on Windows
  `rocm services stop|restart`, the sandbox `stop_server`/`restart_server`
  tool and the attached `rocm serve` Ctrl-C stop now take the engine's
  subtree down as `rocmd` already did. A root the tree kill already took is
  reported `forced`, not `already_gone`.

The Windows step sits behind terminate_recorded_pid_with, which takes the
tree-kill, the identity read and the termination as arguments, so its tests run
on every host. Each caller's tests drive its own stop through the shared
routine, and each carries a source check that no private copy has come back.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocmd`'s stop clears the supervisor and engine PIDs and their start-time
tokens from the record once every recorded process is confirmed gone;
`rocm services stop` did not. Its record kept naming the processes it had just
stopped, so a later stop of the same record reached the termination again.
A PID recorded without a start-time token (every one on Windows) cannot be
refuted once the OS has recycled it, and the Windows tree kill that the shared
routine added for the CLI would then take the unrelated process and everything
under it.

The clearing becomes one `ManagedServiceRecord::clear_recorded_processes` in
`rocm-core`, called from the confirmed branch of both stops; `rocmd`'s four
field assignments are replaced by the call, so the two leave the same record
behind. An unconfirmed stop still keeps the PIDs, since the record is the only
thing that names a survivor.

`a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing`
runs the real termination routine with the per-PID signal injected: after the
first stop the persisted record names no process, and a second stop signals
nothing and still reports stopped. Before the change the record kept PID 4242
and the second stop visited it again. The unconfirmed direction and the
clearing method have their own tests.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ices stop

No scenario ran `rocm services stop` against a real recorded process. service-stop-01
plants PID 0, so it never signals anything, and service-stop-02 drives the
daemon's tool, not the CLI's stop; the Windows tree kill the CLI now shares with
the daemon (`taskkill /PID <pid> /T /F` before the verified termination) had no
live coverage at all, only unit tests with the signalling injected.

service-stop-06 (Linux) runs `rocm services stop --yes` against a recorded
process the scenario starts, and checks the persisted state with the report: the
process exited, the record reads stopped and names no process, and the endpoint
key file is gone. It needs no new step: it reuses the steps of -02 and -03.

service-stop-07 (`@requires-os:windows`, no GPU) records a PowerShell parent
that has started a `ping` child, with the PID alone and no start-time token as
Windows records it, runs `rocm services stop --yes`, and asserts that both
processes are gone together with the report (`Local server stopped`,
`stopped processes: 1`) and the record. Without the tree-kill step the stop
terminates only the recorded process, the child keeps running, and the scenario
fails on it. The Given checks both processes are running first, so the Then
cannot pass for the wrong reason.

The scenario was written on Linux and has not been run on Windows; its tag
keeps it off every other host (the suite's host filter skips it here, which a
run without a name filter confirms). A name filter bypasses that filter, so the
documented command for the stop scenarios is now `-n "service-stop-0[1-6]"`.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Rust 1.96 clippy::duration_suboptimal_units flags Duration::from_secs(60) in the service-stop step. The deadline stays 60 seconds; only the unit it is spelled in changes. The e2e harness target sets test = false, so the workspace clippy run skips it and only the dedicated e2e-cucumber --test e2e clippy step lints it.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ndows, so a recycled PID is refused before the tree kill

process_start_ticks read /proc and was None on Windows, so no recorded identity could be refuted there and the identity gate in front of `taskkill /T /F` only turned away a PID that was not running at all. A service that died by itself, whose PID an unrelated process then took, still passed the gate, and the stop took that process and its whole subtree. Clearing the record on a confirmed stop closed the repeat-stop route but not this one.

On Windows the token is now the process creation time from GetProcessTimes, a FILETIME that only has to be equal to the value read when the PID was recorded. A record carrying one refutes a recycled PID exactly as on Linux: terminate_recorded_pid reads it back as Recycled, skips the tree kill and reports identity_mismatch. A record written before this has no token and stays best-effort. A recorded token that can no longer be read (the process is more privileged) reads as Indeterminate, and the stop reports it unverified instead of signalling it.

That outcome is now reachable on Windows, and its advice named `ps -p`, which Windows does not have; the check it names is now `tasklist /FI "PID eq <pid>"` there.

The FILETIME join is a pure function tested on every host. The refusal runs against the test process's own PID with a wrong token on Linux and Windows, and against a bystander process on Windows, where a stop that went ahead would run taskkill. Scenario service-stop-09 stops a server whose recorded PID belongs to a running bystander under a token no live process has and checks the bystander is still running; it runs on every OS.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… covers for every recorded and live token

A Windows lane run showed service-stop-07, which records a PowerShell parent with no start-time token, ending with the recorded process still running after a stop that reported success. Reading the stop path, a record with no token already gets the whole stop (identity_state reads it as a match, so the tree kill and the verified termination run); I could not run Windows code here and did not find a path in the creation-time change that skips it. What the change did leave was that this decision was spread over identity_state and terminate_verified with the system reads mixed in.

The decision now lives in two pure functions, classify_live_identity (a process is refused only when a token was recorded and the live one is provably different; no recorded token is a match; a recorded token that cannot be read back is indeterminate) and refusal_before_signalling, which identity_state and terminate_verified use. A table test drives terminate_recorded_pid_with through them for no token (the shape service-stop-07 plants), a matching token, a differing token and an unreadable one, asserting whether the tree kill runs and the verdict label together; each row was shown red by a targeted mutation of the classification. A Windows-only test stops a real process recorded without a token.

The step that checks the recorded process has exited now allows its handle a few seconds to signal, since the stop confirms an exit from the OS exit code; a survivor still fails it.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…, even when the scenario fails

service-stop-07 records a PowerShell parent that has started a ping child. If a stop kills the parent but leaves the child, which is the regression the scenario exists to catch, the world's teardown only killed the recorded parent and a later stop cannot recover a child from a dead parent, so ping ran on for its five minutes. The Given step now keeps the child's identity (its PID with its creation time where the host reads one) and teardown, which runs when a step fails too, ends it when it is still that process.

Whether to end it is a pure function, stray_process::leftover_action, with a table test that runs on every host: only a process that is still the recorded one is ended, never one that is gone, whose PID has been reused, or whose identity cannot be confirmed; each row was shown red by a targeted mutation. The Windows clippy check compiles the teardown; the scenario itself has not run on Windows here.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd marked this pull request as ready for review November 9, 2026 07:02
@jussielo-amd
jussielo-amd requested a review from a team as a code owner November 9, 2026 07:02
@jussielo-amd
jussielo-amd requested review from johnl-amd and a balanced review from Copilot November 9, 2026 07:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical Windows lifecycle flaws can falsely confirm stops while processes remain alive and can kill unrelated processes after PID reuse.

3 open issues
What changed in this PR

This PR consolidates managed-service process termination in rocm-core, giving rocm and rocmd shared stop behavior while adding Windows process creation-time checks.

Changes:

  • Shares PID termination, stop outcomes, and the grace period between both binaries.
  • Clears recorded process identities after confirmed stops and adds Windows-specific process-check advice.
  • Adds regression tests and end-to-end scenarios for record cleanup, Windows subtree termination, and recycled PIDs.
File Description
tests/​e2e-cucumber/​tests/​e2e/​service_stop_steps.rs Adds process fixtures and stop-result assertions.
tests/​e2e-cucumber/​tests/​e2e.rs Tracks descendant identities for scenario cleanup.
tests/​e2e-cucumber/​src/​stray_process.rs Adds a tested teardown decision helper.
tests/​e2e-cucumber/​src/​lib.rs Exports the teardown helper module.
tests/​e2e-cucumber/​features/​service_stop.feature Adds record-clearing, Windows tree-stop, and recycled-PID scenarios.
docs/​testing.md Documents stop coverage and platform-specific execution.
docs/​architecture.md Updates ownership of the shared stop routine.
crates/​rocm-core/​src/​service_record.rs Adds shared clearing of recorded process identities.
crates/​rocm-core/​src/​proc_lifecycle.rs Centralizes termination and adds Windows creation-time checks.
crates/​rocm-core/​src/​lib.rs Exports shared stop APIs and types.
apps/​rocmd/​src/​service.rs Replaces private stop logic with shared routines.
apps/​rocm/​src/​services_text.rs Provides platform-specific process-check commands.
apps/​rocm/​src/​service_launch.rs Adapts tests to shared outcomes and platform-specific advice.
apps/​rocm/​src/​service_control.rs Uses shared termination and clears identities after confirmed stops.

Review effort: Balanced

Comment thread crates/rocm-core/src/proc_lifecycle.rs Resolved
Comment thread crates/rocm-core/src/proc_lifecycle.rs Resolved
Comment thread crates/rocm-core/src/proc_lifecycle.rs Resolved
…he feature's ids stay sequential

The scenario was numbered 09 to leave room for a restart scenario that belongs to a separate change, but the suite requires each feature's scenario names to be indexed sequentially in declaration order. Renumber it to 08 and update the references to it in the feature comment, docs/testing.md (including the name filter that selects the stop scenarios) and the rocmd stop test's comment.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ueried, or when its tree kill did not complete, and hold the PID across the stop

On Windows three things let a stop report success while a process of the service might still run.

A live process the stop may not open made OpenProcess fail, and process_is_running read every failure as "not running", so the identity gate classified it Gone and the stop reported it already gone. The liveness gate now keeps a failed query apart from an exit: ProcessProbe::Unqueryable is Indeterminate, so the stop reports it unverified and keeps the record and the endpoint key. Only the OpenProcess code for "no such process" counts as gone. process_is_running itself is unchanged.

The creation-time comparison closed its handle before taskkill ran by PID, leaving a window for the process to exit and an unrelated one to take the PID. The PID is now held open from before the identity read until the stop returns; Windows does not hand a PID to another process while a handle to it is open.

The taskkill result was discarded, and terminate_verified can only watch the root on Windows, so a descendant taskkill did not take down went unnoticed. The tree kill now returns whether taskkill started and exited cleanly, and a failed one turns any verdict that would confirm the stop into timed_out, which keeps the record, the stop request and the endpoint key.

The decisions sit behind pure functions and injected steps so the tests run on every host: identity_state_with, classify_open_failure, classify_taskkill, after_tree_kill and terminate_recorded_pid_with with an injected hold. The Windows API calls themselves (the handle, the error codes, a real taskkill failing) run only on the Windows lane.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Windows stop handling can hang indefinitely on a stalled taskkill, and subsequent liveness refreshes can delete endpoint state for an inaccessible process that is still running.

4 open issues
2 resolved since last review

Review effort: Balanced

Comment thread crates/rocm-core/src/proc_lifecycle.rs Outdated
) -> IdentityState {
match probe(id.pid) {
ProcessProbe::Gone => IdentityState::Gone,
ProcessProbe::Unqueryable => IdentityState::Indeterminate,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The finding is right. After a stop reports an inaccessible Windows process as unverified, it deliberately leaves the stop request and the endpoint key on disk so a later confirmed exit can finish the cleanup. But settle_pending_stop_key_cleanup, which runs at the start of every liveness refresh (for example when services are listed), still decided "gone" from process_is_running, and on Windows that returns false for any process the caller cannot open. So the next listing would delete the key, clear the stop request and, a few lines later, reconcile the record to stopped, undoing exactly what the unverified verdict was protecting. Fixed in 4ebcdf6.

The cleanup, the reconciliation to stopped, and the in-use check behind rocm services remove and prune (which has the same shape and could delete a record, its log and its key under a live process) now read each recorded PID through probe_process and treat anything but a confirmed exit as possibly running, using a new ProcessProbe::may_be_running. Only gone may delete the key and clear the marker; unqueryable keeps both and keeps the record live. Each function takes the probe as an argument so the tests run on every host. The regression tests are a_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried, which refreshes a pending-stop record with an unqueryable probe and asserts the key file, the stop marker and the non-stopped status are kept, then with a confirmed-gone probe and asserts the key is dropped, the marker cleared and the record stopped; an_unverified_stop_keeps_its_key_and_marker_through_a_later_liveness_refresh, which runs a real unverified stop and then the refresh over the record it wrote; and a_record_is_in_use_unless_every_recorded_process_is_confirmed_gone for the removal guard. Reading unqueryable as not running again (the old behaviour) turns the first two red in the cleanup and the first red in the reconciliation, and turns the third red in the removal guard; I restored each afterwards. The real OpenProcess access-denied case can only be exercised on the Windows lane.

I swept every caller of process_is_running and the places that decide "gone" from a boolean. Fixed, as the same stop and cleanup family: the key cleanup and the reconciliation in managed_service_status.rs, and the removal guard in service_records.rs. Left alone, with reasons: ensure_spawned_child_survives_startup, the attached log streams and service_pid_is_alive in service_launch.rs watch a child this very command just started with the same account, and are not a decision about a recorded service from an earlier command; background_helper.rs checks the helper's own daemon PID and at worst starts a second helper, with no key or record involved; the ComfyUI stop in comfyui/process.rs has its own tasklist-based check, which lists a process whatever its access rights, and manages a different state file (its taskkill call has no deadline either, which is a sibling of the second finding that I did not change here); the remaining hits in rocmd and rocm-dash-collectors are tests or a test helper. process_is_running itself is unchanged and its documentation now points at probe_process.

Comment thread crates/rocm-core/src/proc_lifecycle.rs Resolved
…and bound the Windows tree kill with a deadline

The previous change made a stop report a Windows process it cannot query as unverified and keep the record, the stop request and the endpoint key. The liveness refresh that runs when services are listed then undid that: it still called process_is_running, which reads a process it cannot open as not running, so it deleted the key, cleared the stop request and reconciled the record to stopped while the process might be alive. The same call decided whether a record may be removed or pruned.

All three now read each recorded PID through probe_process and treat anything but a confirmed exit as possibly running (ProcessProbe::may_be_running, exported from rocm-core): the deferred key cleanup, the reconciliation to stopped, and the in-use check behind services remove and prune. Each takes the probe as an argument, so the tests drive an unqueryable process on every host. process_is_running itself is unchanged; the launch-time callers that watch a child this process just started keep it.

The taskkill that takes a Windows tree down ran through Command::status, which waits without limit and runs before the stop's bounded waits, so a stalled taskkill hung the stop. It now runs under a deadline of MANAGED_STOP_GRACE: a child still running at the deadline is killed and reaped, the run reports a timeout, and that is a failed tree kill, which the stop already treats as unconfirmed. The wait is a plain helper over std::process, tested on every host with a long-sleeping child (ping on Windows) under a short deadline; only the taskkill wiring is Windows-only.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Windows runtime confirmation is still needed for process creation-time checks, PID-reuse protection, and taskkill tree termination because static review and injected-function tests cannot validate these OS-specific safety guarantees.

2 open issues
2 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical stop-handling gaps can let refresh followed by remove/prune delete records for still-running containers and let stop confirmation overwrite a concurrent relaunch’s status and delete its endpoint key.

4 open issues
1 resolved since last review

Review effort: Balanced

Comment thread apps/rocm/src/managed_service_status.rs Outdated
Comment on lines +249 to +250
if rocm_core::engine_stop_decides_confirmation(&record.engine) {
return false;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right: the guard I added in the last round protected only the cleanup, and the rest of the refresh and the removal guard still read the supervisor's PID as evidence. For an unresolved container stop (a stop was requested, and the engine is one whose Stop decides confirmation, today aim) the record is now protected in all three places through one predicate, container_stop_unresolved: the cleanup does not complete, the refresh returns before it would reconcile the status from the PIDs (whether or not the endpoint answers), and the in-use check behind remove and prune counts the record as in use. The way out is the one the stop already has: services stop again, which asks the engine. a_pending_container_stop_survives_refresh_remove_and_prune runs a refresh, remove --yes and prune --yes over an aim record with the supervisor confirmed gone, for a record reading ready and one already reading stopped (which only the in-use guard protects), and asserts nothing was deleted, with a process engine in the same state removed as before. Removing the check from the in-use guard or from the refresh turns it red. Fixed in 2ca770d.

Comment thread crates/rocm-core/src/service_record.rs Resolved

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the full diff. Consolidating the stop routine into rocm-core with the injectable Windows steps is the right shape, and the new scenarios assert the printed messages together with on-disk state. I have a few things I'd like addressed or answered before this merges, left inline where they have a line to anchor to:

  • An "unconfirmed" stop (ExitedBeforeTreeKill / DescendantsUnconfirmed) doesn't stay unconfirmed: the next rocm services stop, or a plain listing, sees the root gone and confirms it with no new evidence about the descendants. The "a process under it may still be running" warning is only informational as written.
  • windows_system_tool builds its path from the SystemRoot environment variable, which undercuts the goal of the tools not being substitutable through the environment.
  • The pending-stop settle path now forces stopped for any record with a stop marker whose PIDs are all gone, including records that name no process. That shape has no test.
  • Scope: the routine merge, remove --force, the ComfyUI System32 change, the Windows creation-time token and the failed-launch cleanup are all in one ~5.8k-line PR, and go beyond what ROCMAI-874 asks for. The ComfyUI System32 change in particular could stand alone. Please say whether you'd rather split it.
  • Windows tree kill, the taskkill failure paths, launch cleanup of a real child and service-stop-07 run in no CI job. The answer to the ticket's "which lane runs this on Windows" is currently "none", so a manual Windows host check from docs/manual-testing.md should be recorded before merge.

I didn't build or run the tests, so the mutation/regression claims in the description are unverified on my side.

// deny a stop this command did perform, so it counts as forced; but when
// the kill found no such process the root went by itself, and
// `AlreadyGone` is the truth.
let outcome = match (tree, terminate(identity)) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two verdicts are only good for one run: the next rocm services stop reads this root as gone and reports it confirmed, and a plain listing completes the pending stop the same way, neither with new evidence about the descendants. Either say so in the printed report and docs, or keep a durable marker so the "may still be running" warning isn't cleared by one more command.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that the warning lasts one run. I took the smaller option, saying so, rather than a durable marker, which would need a new field in the record and a rule for clearing it that the ticket does not ask for. The printed not confirmed: lines for descendants_unconfirmed and exited_before_tree_kill now end with "rocm does not keep this warning, so a later stop or listing will find the process gone and report the stop confirmed", and docs/testing.md says the same. The test asserts the message and the state together: the report test checks the line and that the record, marker and key are kept, then runs the next stop over the same record with the root gone and asserts it is reported confirmed, so the sentence is checked against what actually happens. Dropping the sentence turns it red. If you would rather have the marker I can add it, but it changes the record format and the restart and recovery paths, so I would do it as its own change.

/// So a directory earlier on `PATH` cannot stand in for it; the bare name when
/// `SystemRoot` is not set. `name` includes the extension (`taskkill.exe`).
#[must_use]
pub fn windows_system_tool(name: &str) -> PathBuf {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds the path from the SystemRoot environment variable, so a controlled environment can still redirect taskkill/tasklist. GetSystemDirectoryW (or GetWindowsDirectoryW) doesn't read the environment and would match the stated goal that nothing the caller controls can substitute these tools.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, SystemRoot is a variable a caller controls. windows_system_tool now takes the directory from GetSystemDirectoryW, which does not read the environment, behind the same seam: the pure system_tool_path takes the directory, so the path logic stays unit-tested on every host (a_system_tool_is_resolved_under_the_system_directory, red if the directory is ignored), and the Windows call is the only part that is not, with its SAFETY comment. A length of zero or one that does not fit the buffer falls back to the bare name rather than trusting a truncated path. Off Windows there is no system directory and the answer is the bare name, as before. The Windows-target clippy of the whole workspace is clean, and no test on this machine runs the Windows call, which the PR body says; the Windows job's tests that start taskkill do run it. Fixed in 2ca770d.

// stop still finalizing would report itself unconfirmed. Setting it also makes
// the caller's liveness gate return at once, so the one write the caller makes
// covers the key cleanup, the cleared processes and the status together.
"stopped".clone_into(&mut record.status);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now sets stopped for any record with a stop marker whose recorded PIDs are all confirmed gone, including a record that names no process at all (e.g. a starting record with a stale marker). Before, only the key and marker were cleaned. Is that intended? The tests cover live statuses with PIDs; consider limiting it to records that named at least one process, or add a test for the no-PID case.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intended, thank you. The cleanup now goes on to forget the processes and read the record as stopped only for a record that named at least one process; a record that named none, such as a starting one with a stale marker, has its key and marker cleaned as before and is otherwise left as it was, because it may be a launch that has not recorded its supervisor yet. a_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stopped is the no-PID test: it asserts the marker and key are gone and the status is still starting. Removing the condition turns it red. Fixed in 2ca770d.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 08c5f445

Review — needs work

Full review of the whole change, fanned out across 8 independent file-coupling groups given the size (30 files, +5786/-555 lines): the shared rocm-core stop routine, its process/identity support layer, the rocm stop/status caller, the rocm records/remove/prune/CLI-text cluster, launch-cleanup + ComfyUI's System32 calls, the rocmd/supervise/vLLM callers, the e2e-cucumber suite, and docs/README. Implements a shared rocm-core termination routine with Windows identity/liveness/tree-kill hardening, services remove --force, System32 tool resolution, and stricter failed-launch cleanup, as described; claims to close ROCMAI-874 (an internal AMD tracker ticket — named here but not retrieved or verified, per this repo's convention for externally-visible review automation).

Blocking

  • Required clippy CI check is currently failing at this exact head on the module-size gate — apps/rocm/src/cli.rs:1502 production lines, 2 over the repo's 1,500-line cap (cargo xtask check-module-size, run id 129880496283, step output: error: 1 source file(s) over the limit of 1500 production lines (of 412 checked): apps/rocm/src/cli.rs: 1502 production lines (2 over), exit code 1). This also means the PR's own Test Plan text ("cargo xtask check-module-size reports no file over the limit... 412 files checked") no longer holds at this head — it was evidently true on an earlier commit (the file sat at ~1499 lines per independent review) and was pushed over by 2 lines since, likely by the final merge main commit.
    Confidence 100 · architectural · Fix: split apps/rocm/src/cli.rs (a pure move, tests included) per the repo's own documented remedy, or trim 2 production lines back under the cap.

  • service-stop-07 (the Windows process-tree-kill scenario) has no CI lane that actually executes it, despite meeting the project's own established criteria for folding such a scenario into the required job — tests/e2e-cucumber/features/service_stop.feature:162-171 (@id:service-stop-takes-down-the-whole-process-tree-on-windows, @requires-os:windows, "No GPU, no engine, no network"). The required windows-release job already folds comparably hermetic, @requires-os:windows, no-e2e-test-hooks scenarios into its otherwise-lifecycle-only run via E2E_ALSO_RUN_IDS (.github/workflows/ci.yml:1009-1010: comfyui-uv-install-failure-names-log, serve-vllm-refused-on-native-windows, etc.), with an in-repo comment explaining exactly why ("the self-hosted Windows lane that otherwise runs them... is optional and not a required check"). service-stop-07 matches that profile exactly but was not added, and no reason is given. As a result the PR's central new safety mechanism — the identity-gated Windows subtree kill the whole PR exists to harden — has zero automated execution anywhere in CI; only two #[cfg(windows)] unit tests in proc_lifecycle.rs exercise the underlying primitives against a real process with no child.
    Confidence 80 · architectural · Fix: add service-stop-takes-down-the-whole-process-tree-on-windows to the windows-release job's E2E_ALSO_RUN_IDS.

  • Two module doc-comments misdescribe the production call graph after this PR's own _with refactor — apps/rocm/src/service_records.rs:19-20 states "services_cmd.rs's services() calls remove_managed_service_record..."; apps/rocm/src/services_text.rs:51-52 states "service_records.rs reads validate_service_id (in... remove_managed_service_record) and load_managed_service (in remove_managed_service_record and apply_service_prune_plan)." Verified directly: remove_managed_service_record and apply_service_prune_plan are now #[cfg(test)]-only wrappers (service_records.rs:251-252, :711-712) that cannot run in a release build; services_cmd.rs:41,82 confirms the real production caller is remove_managed_service_record_with, and the real validate_service_id/load_managed_service calls live in remove_managed_service_record_with/apply_service_prune_plan_with (service_records.rs:270-271,735). These docs make a concrete, checkable false claim about what calls what, introduced by this PR's own rename/split and never updated.
    Confidence 85 · mechanical · Fix: update both doc comments (and service_hints.rs:43's "see remove_managed_service_record" pointer) to name the _with functions.

  • ComfyUI's two Windows process-tool call sites have zero test coverage pinning them to the new System32 resolution — apps/rocm/src/comfyui/process.rs:420-422 (windows_process_is_running) and :470-471 (terminate_windows_process_tree) both correctly call rocm_core::windows_system_tool("tasklist.exe"/"taskkill.exe") (verified: both call sites fixed, not just one). But the file's only test, running_state_requires_saved_process_and_reachable_port (:504-526), never touches either function. A future edit that silently reverted either call site back to a bare Command::new("tasklist")/Command::new("taskkill") — reopening exactly the PATH-hijack exposure this PR closes — would go undetected, and this is cheaply testable (e.g. asserting on Command::get_program()) without a real Windows host.
    Confidence 75 · mechanical · Fix: a small test per call site (or a shared helper tested once) asserting the built Command's program path runs through windows_system_tool.

  • A new bare-backtick file citation in docs/manual-testing.md violates this repo's own markdown-link convention — docs/manual-testing.md:625: two `#[cfg(windows)]` tests in `proc_lifecycle.rs`. Confirmed this is the sole bare-backtick .rs citation in the file; every other .rs reference (e.g. lines 486, 1469, 1502) is a proper relative link. AGENTS.md §5 requires citing a tracked repo file as a relative markdown link specifically so the lychee link checker can catch future drift.
    Confidence 75 · mechanical · Fix: [proc_lifecycle.rs](../crates/rocm-core/src/proc_lifecycle.rs).

Non-blocking

None at ≥70 confidence. (Several lower-confidence candidates — a taskkill-kill-failure edge case in wait_for_exit_code, a blocking call inside rocmd's tokio runtime without spawn_blocking, a narrow self-healing stop_requested_unix_ms race, a windows_system_tool PATH fallback when SystemRoot is unset — were investigated and dropped: each is either traced to a pre-existing, unworsened pattern, or remains speculative with no demonstrable harm.)

Decisions for the author

  • rocmd's terminate_recovery_supervisor (apps/rocmd/src/supervise.rs:787-797) still calls terminate_verified directly rather than the new terminate_recorded_pid/terminate_recorded_pid_within — tradeoff/pre-existing. Confirmed byte-for-byte unchanged by this PR, so on Windows a recovery-supervisor cleanup still only kills the launcher root, not its subtree — the exact gap this PR just fixed for the two main stop paths. Not introduced or widened here, but worth a follow-up given the sibling fix just landed.
  • TREE_KILL_DEADLINE ignores a caller's shorter grace — tradeoff. terminate_recorded_pid_within's Windows descendant taskkill is always bounded by MANAGED_STOP_GRACE (10s) even when the failed-launch cleanup passes a nominal 2s grace. Documented as intentional (one well-tested deadline vs. a path that could silently take 5x longer than its parameter name suggests) — confirm this is the intended tradeoff.
  • Add service_forced_remove_hint to the sibling hostile-input proptest (apps/rocm/src/service_hints.rs:44-49 vs. the a_hint_denotes_the_argv_it_names test at :124-147) — non-blocking-improvement. Reuses an already-tested quoting call, so risk is low, but it's the one hint-builder the module's own anti-drift guard doesn't cover.
  • named_processes's "processes X and Y" plural branch (apps/rocm/src/service_records.rs:213-224) is never exercised by this feature's own tests (the one two-PID test short-circuits via a Running probe) — non-blocking-improvement, low risk given the trivial join(" and ") logic.
  • No Windows-host test runs abandon_failed_managed_launch against a real spawned child — answered-by-intent. The PR's own text states plainly: "no test runs a failed launch against a real Windows child... it rests on the routine's own tests and review," arguing this is impractical without Windows CI hardware running the e2e target. Accepted as an honestly-argued, acknowledged limitation rather than a fresh finding.
  • This is a very large, bundled PR (30 files, 31 commits) — tradeoff. The description pre-justifies bundling the shared routine, --force, System32 resolution, and launch-cleanup hardening together ("none of them stands alone... without a Windows host to show it"). Accepted as reasoned, but it is worth recording that a change this size strains what any single review pass — automated or human — can cover with full confidence; this review required an 8-way fan-out to attempt full coverage.

Positive signals

  • The Windows identity gate and tree-kill state machine (classify_live_identity, refusal_before_signalling, classify_taskkill, wait_for_exit_code) are split into pure/injectable functions exercised by unit tests on every host, including explicit stalled-helper and deadline paths — unusually thorough compliance with this repo's "platform-gated code gets a test that runs on its gate" rule.
  • windows_system_tool resolves taskkill/tasklist under %SystemRoot%\System32 everywhere in the repo, including both of ComfyUI's call sites — confirmed no remaining bare-name invocation anywhere, closing a real PATH-hijack exposure.
  • This PR fixes a genuine pre-existing Windows safety bug: the remove/prune in-use guard previously mapped an access-denied (Unqueryable) PID to "not running," which could have deleted a record (and its endpoint key) out from under a still-serving process it merely couldn't query. The new recorded_pids_may_be_running reasoning is confirmed used identically by all three destructive call sites (deferred cleanup, status reconciliation, removal guard) with no drifted fourth copy.
  • --force's narrow gating (stop requested, nothing confirmed running, at least one unqueryable) was traced branch-by-branch with no inverted boolean, and --yes is independently and unconditionally still required alongside it.
  • Extensive mutation-style evidence in the PR's own test plan (each guard tied to a named test that goes red when it's removed), independently corroborated by this review's own code-reading.

Deployment notes

  • rocm services stop (and restart, and the sandbox stop tool) now perform a Windows process-subtree kill (taskkill /T) they never did before, and now clear recorded PIDs/start-time tokens from the record on a confirmed stop. Operators who scripted around the old behavior (engine process left running after a stop; stale PIDs persisting in the record) should expect different behavior after upgrading.
  • A record whose recorded PID becomes permanently unqueryable (e.g. a permissions change) now stays unverified indefinitely — it never reconciles to stopped, and remove/prune refuse it — until an operator runs rocm services remove <id> --force --yes. This is a deliberate, documented tradeoff, not a regression, but it is a new persistent-state outcome operators should know about.

What this covered

Fanned out across 8 independent, blind sub-reviewers, each reading its assigned files completely at head commit 08c5f445aea874193501b1ec73a4a7c4be967892 (base c4e1f72635e42b90f9917a0dbbb7aa00bcc344e5, a clean three-dot diff with no stacked-branch contamination — merge-base verified equal to the PR's reported base): (1) crates/rocm-core/src/proc_lifecycle.rs; (2) process.rs/service_record.rs/lib.rs; (3) apps/rocm/src/service_control.rs/managed_service_status.rs; (4) service_records.rs/services_text.rs/service_hints.rs/services_cmd.rs/cli.rs; (5) service_launch/launch_transaction.rs/attach.rs/comfyui/process.rs; (6) apps/rocmd/src/service.rs/supervise.rs/engines/vllm/src/state.rs; (7) the full tests/e2e-cucumber suite; (8) README.md/docs/architecture.md/docs/manual-testing.md/docs/testing.md. All 8 reported real, substantive findings; none degraded. The consolidating pass independently re-verified the highest-value candidates directly against the source and against live CI logs/checks rather than accepting self-reported confidence scores, which is how the module-size CI failure and the service-stop-07 CI-gap finding were confirmed. Design review (Step 7) was done by the consolidating pass directly (reasoning through blast-radius, tradeoff and completeness questions) rather than via separate sub-workers, given the scale of fan-out already performed.

Did not run: the "prior changes" pass (review comments on earlier changes to these 30 files) — impractical to search exhaustively here and no standing objections were supplied. History and code-comments passes were performed per-file by each sub-worker rather than as a separate pass. Not reconciled against this PR's existing discussion or other reviewers' comments (a Copilot review and a human CHANGES_REQUESTED review exist at this head) — this review was read blind and does not know what they say.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

cli.rs is checked by check-module-size, which blocks, and main keeps adding lines to it, so the services help text this branch adds repeatedly put it over the 1500 line limit. Trimming the help text again would only buy a few lines until the next merge.

The clap value enums behind --framework, --format, --provider, --device and the similar arguments (FrameworkArg, InstallFormat, Provider, WatcherModeArg, TelemetryModeArg, DevicePolicyArg, SandboxToolArg, PermissionsModeArg) and their conversions are self-contained, so they move as they are to cli_args.rs, and cli.rs re-exports each one so every module keeps reaching them as crate::cli::Provider and so on. Nothing else changes: attributes, help text, aliases and declaration order are as before. cli.rs goes from over the limit to about 1348 production lines, leaving real headroom, and docs/architecture.md has the new file's entry.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ell an empty relaunch from the stop, and resolve system tools without the environment

An unresolved stop of a container-backed engine (a stop was requested and the engine Stop decides confirmation, today aim) is now protected everywhere, not only in the deferred cleanup. The liveness refresh does not read such a record as stopped from the supervisor's PID, and the in-use check behind services remove and prune counts it as in use, so a listing followed by remove or prune no longer deletes a record and key whose container may still be running. One predicate, container_stop_unresolved, is the single reading.

is_covered_by_stop_of no longer treats an empty relaunch as the stop. When the stopping snapshot names no process, a record naming none is covered only while it still carries the stop's marker or reads stopped; an empty starting or recovering relaunch has cleared the marker, so the stop does not confirm over it, overwrite its status or delete its key.

The deferred cleanup goes on to forget the recorded processes and read the record as stopped only for a record that named at least one process. A starting record with a stale marker gets its key and marker cleaned, as before, and is otherwise left alone.

The two unconfirmed tree-kill lines now say that rocm does not keep the warning, so a later stop or listing finds the process gone and reports the stop confirmed, and the test runs that later stop to check the sentence.

windows_system_tool reads the system directory with GetSystemDirectoryW instead of building it from the SystemRoot environment variable, which a caller can redirect. The pure path logic keeps its seam and its test.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd requested a balanced review from Copilot November 10, 2026 15:37
@jussielo-amd
jussielo-amd dismissed juhovainio’s stale review November 10, 2026 15:38

Fixed in 2ca770d and answered inline (container stop guard, empty relaunch record, honest unconfirmed-stop message, GetSystemDirectoryW, no-process stop marker), with cli.rs back under the size limit in bd168e9. The ComfyUI change stays in this PR (no split). Still open and said so in the PR body: the unconfirmed-stop warning is not kept across runs, and the Windows host check from docs/manual-testing.md has not been run.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

# Conflicts:
#	tests/e2e-cucumber/src/lib.rs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A concurrent listing can erase evidence of an in-flight stop with no recorded processes, causing stop confirmation to fail and blocking restart even when no relaunch occurred.

4 open issues
1 resolved since last review

Review effort: Balanced

Comment thread crates/rocm-core/src/service_record.rs Resolved
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…g consumes it off an empty record

A stop that begins on a record naming no process writes its marker and has
nothing to signal. A concurrent listing finds the marker with no process to
wait for and clears it, but leaves a starting or failed record at its status,
so the stop's re-read saw an empty record with no marker, which is also what a
relaunch that has not recorded its supervisor yet looks like. The stop
reported itself unconfirmed, and so did the restart built on it.

The refresh now keeps the consumed marker's value in settled_stop_unix_ms, and
is_covered_by_stop_of accepts an empty record whose stamp equals the stop's own
request stamp. A restart clears the stamp, so a real empty relaunch is still
not covered.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An older stop can still delete the replacement service’s endpoint key before its PIDs are published, and the stop-marker regression test never exercises its intended interleaving.

5 open issues
1 resolved since last review

Review effort: Balanced

Comment thread crates/rocm-core/src/service_record.rs Resolved
Comment thread apps/rocmd/src/service.rs Resolved
…k, where an empty record can reach it

The test of a stop whose marker a concurrent listing consumed put the
simulated listing in the per-PID termination callback. The record names no
process, so that callback never ran and the test passed whether or not the
settled stamp was honoured. The rewrite now happens in the engine stop
callback, which does run for a record naming no process, the termination
callback is unreachable, and the test asserts that the interleaving occurred.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…start leaves before it publishes

Between its own confirmed stop and the publication of its child, a restart
leaves an empty stopped record, restores the endpoint key and spawns. That
record is also what an older stop's own completion looks like, so an older stop
still finalizing took it for its own, wrote stopped over it and deleted the key
the restart had just restored for its child.

The restart now stamps a relaunch generation (relaunch_started_unix_ms) on the
record before it restores the key or spawns. A stop covers a record only in the
generation it read, and publishing the child clears the stamp. The restart's own
writes also refuse a record whose generation moved since it read it.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for a critical stop/relaunch race that can delete the replacement generation’s restored endpoint while its publication guard still accepts the record.

3 open issues
2 resolved since last review

Review effort: Balanced

@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit eb2a819 Nov 10, 2026
41 checks passed
@jussielo-amd
jussielo-amd deleted the split-managed-service-stop-routine branch November 10, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants