Repository navigation
fix(core): stop managed services through one rocm-core routine shared by rocm and rocmd, with Windows identity, liveness and tree-kill hardening; adds services remove --force, System32 tool resolution (incl. ComfyUI) and a stricter failed-launch cleanup - #960
Conversation
…its recorded PIDs rocm (service_control.rs) and rocmd (service.rs) each carry their own MANAGED_STOP_GRACE and terminate_recorded_service_pids, and the two have diverged. Pin what each copy does today, with no production change, so the change that merges them has to edit the tests that describe the difference: the grace value (10 s in both), the order the PIDs are visited (supervisor, then engine), the per-PID start-time pairing and the de-duplication of a PID recorded under both roles, zero PIDs, and the per-outcome signalled/forced/stopped classification. The two divergences are pinned as such: rocm drops the stopping process's own PID from the report while rocmd reports it as `self`, and the Windows `taskkill /T /F` step exists only in rocmd. The latter is a private cfg(windows) block with no injection point, so it cannot be observed from a test yet; only its portable half (an absent PID reaches terminate_verified unchanged) is pinned here. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…by rocm and rocmd `rocm services stop` and `rocmd`'s `stop_server` each carried their own MANAGED_STOP_GRACE (10 s, kept in step only by a comment) and their own terminate_recorded_service_pids, and the copies had diverged. Move one routine and one constant into rocm-core's proc_lifecycle.rs, beside terminate_verified, and delete both private copies. The termination of each PID stays injectable. rocmd's behaviour wins, because it is the more complete one: - The stopping process's own PID is reported as `self` (never signalled, counted as stopped). rocm used to drop it, so `pid_outcomes` listed one entry fewer than rocmd's for the same record. - On Windows a verified root is preceded by `taskkill /PID <pid> /T /F`, because terminate_verified cannot walk a process tree there and the engine runs one level below the recorded PID. rocm never did this, so on Windows `rocm services stop|restart`, the sandbox `stop_server`/`restart_server` tool and the attached `rocm serve` Ctrl-C stop now take the engine's subtree down as `rocmd` already did. A root the tree kill already took is reported `forced`, not `already_gone`. The Windows step sits behind terminate_recorded_pid_with, which takes the tree-kill, the identity read and the termination as arguments, so its tests run on every host. Each caller's tests drive its own stop through the shared routine, and each carries a source check that no private copy has come back. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocmd`'s stop clears the supervisor and engine PIDs and their start-time tokens from the record once every recorded process is confirmed gone; `rocm services stop` did not. Its record kept naming the processes it had just stopped, so a later stop of the same record reached the termination again. A PID recorded without a start-time token (every one on Windows) cannot be refuted once the OS has recycled it, and the Windows tree kill that the shared routine added for the CLI would then take the unrelated process and everything under it. The clearing becomes one `ManagedServiceRecord::clear_recorded_processes` in `rocm-core`, called from the confirmed branch of both stops; `rocmd`'s four field assignments are replaced by the call, so the two leave the same record behind. An unconfirmed stop still keeps the PIDs, since the record is the only thing that names a survivor. `a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing` runs the real termination routine with the per-PID signal injected: after the first stop the persisted record names no process, and a second stop signals nothing and still reports stopped. Before the change the record kept PID 4242 and the second stop visited it again. The unconfirmed direction and the clearing method have their own tests. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ices stop No scenario ran `rocm services stop` against a real recorded process. service-stop-01 plants PID 0, so it never signals anything, and service-stop-02 drives the daemon's tool, not the CLI's stop; the Windows tree kill the CLI now shares with the daemon (`taskkill /PID <pid> /T /F` before the verified termination) had no live coverage at all, only unit tests with the signalling injected. service-stop-06 (Linux) runs `rocm services stop --yes` against a recorded process the scenario starts, and checks the persisted state with the report: the process exited, the record reads stopped and names no process, and the endpoint key file is gone. It needs no new step: it reuses the steps of -02 and -03. service-stop-07 (`@requires-os:windows`, no GPU) records a PowerShell parent that has started a `ping` child, with the PID alone and no start-time token as Windows records it, runs `rocm services stop --yes`, and asserts that both processes are gone together with the report (`Local server stopped`, `stopped processes: 1`) and the record. Without the tree-kill step the stop terminates only the recorded process, the child keeps running, and the scenario fails on it. The Given checks both processes are running first, so the Then cannot pass for the wrong reason. The scenario was written on Linux and has not been run on Windows; its tag keeps it off every other host (the suite's host filter skips it here, which a run without a name filter confirms). A name filter bypasses that filter, so the documented command for the stop scenarios is now `-n "service-stop-0[1-6]"`. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Rust 1.96 clippy::duration_suboptimal_units flags Duration::from_secs(60) in the service-stop step. The deadline stays 60 seconds; only the unit it is spelled in changes. The e2e harness target sets test = false, so the workspace clippy run skips it and only the dedicated e2e-cucumber --test e2e clippy step lints it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ndows, so a recycled PID is refused before the tree kill process_start_ticks read /proc and was None on Windows, so no recorded identity could be refuted there and the identity gate in front of `taskkill /T /F` only turned away a PID that was not running at all. A service that died by itself, whose PID an unrelated process then took, still passed the gate, and the stop took that process and its whole subtree. Clearing the record on a confirmed stop closed the repeat-stop route but not this one. On Windows the token is now the process creation time from GetProcessTimes, a FILETIME that only has to be equal to the value read when the PID was recorded. A record carrying one refutes a recycled PID exactly as on Linux: terminate_recorded_pid reads it back as Recycled, skips the tree kill and reports identity_mismatch. A record written before this has no token and stays best-effort. A recorded token that can no longer be read (the process is more privileged) reads as Indeterminate, and the stop reports it unverified instead of signalling it. That outcome is now reachable on Windows, and its advice named `ps -p`, which Windows does not have; the check it names is now `tasklist /FI "PID eq <pid>"` there. The FILETIME join is a pure function tested on every host. The refusal runs against the test process's own PID with a wrong token on Linux and Windows, and against a bystander process on Windows, where a stop that went ahead would run taskkill. Scenario service-stop-09 stops a server whose recorded PID belongs to a running bystander under a token no live process has and checks the bystander is still running; it runs on every OS. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… covers for every recorded and live token A Windows lane run showed service-stop-07, which records a PowerShell parent with no start-time token, ending with the recorded process still running after a stop that reported success. Reading the stop path, a record with no token already gets the whole stop (identity_state reads it as a match, so the tree kill and the verified termination run); I could not run Windows code here and did not find a path in the creation-time change that skips it. What the change did leave was that this decision was spread over identity_state and terminate_verified with the system reads mixed in. The decision now lives in two pure functions, classify_live_identity (a process is refused only when a token was recorded and the live one is provably different; no recorded token is a match; a recorded token that cannot be read back is indeterminate) and refusal_before_signalling, which identity_state and terminate_verified use. A table test drives terminate_recorded_pid_with through them for no token (the shape service-stop-07 plants), a matching token, a differing token and an unreadable one, asserting whether the tree kill runs and the verdict label together; each row was shown red by a targeted mutation of the classification. A Windows-only test stops a real process recorded without a token. The step that checks the recorded process has exited now allows its handle a few seconds to signal, since the stop confirms an exit from the OS exit code; a survivor still fails it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…, even when the scenario fails service-stop-07 records a PowerShell parent that has started a ping child. If a stop kills the parent but leaves the child, which is the regression the scenario exists to catch, the world's teardown only killed the recorded parent and a later stop cannot recover a child from a dead parent, so ping ran on for its five minutes. The Given step now keeps the child's identity (its PID with its creation time where the host reads one) and teardown, which runs when a step fails too, ends it when it is still that process. Whether to end it is a pure function, stray_process::leftover_action, with a table test that runs on every host: only a process that is still the recorded one is ended, never one that is gone, whose PID has been reused, or whose identity cannot be confirmed; each row was shown red by a targeted mutation. The Windows clippy check compiles the teardown; the scenario itself has not run on Windows here. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical Windows lifecycle flaws can falsely confirm stops while processes remain alive and can kill unrelated processes after PID reuse.
3 open issues
What changed in this PR
This PR consolidates managed-service process termination in rocm-core, giving rocm and rocmd shared stop behavior while adding Windows process creation-time checks.
Changes:
- Shares PID termination, stop outcomes, and the grace period between both binaries.
- Clears recorded process identities after confirmed stops and adds Windows-specific process-check advice.
- Adds regression tests and end-to-end scenarios for record cleanup, Windows subtree termination, and recycled PIDs.
| File | Description |
|---|---|
tests/e2e-cucumber/tests/e2e/service_stop_steps.rs |
Adds process fixtures and stop-result assertions. |
tests/e2e-cucumber/tests/e2e.rs |
Tracks descendant identities for scenario cleanup. |
tests/e2e-cucumber/src/stray_process.rs |
Adds a tested teardown decision helper. |
tests/e2e-cucumber/src/lib.rs |
Exports the teardown helper module. |
tests/e2e-cucumber/features/service_stop.feature |
Adds record-clearing, Windows tree-stop, and recycled-PID scenarios. |
docs/testing.md |
Documents stop coverage and platform-specific execution. |
docs/architecture.md |
Updates ownership of the shared stop routine. |
crates/rocm-core/src/service_record.rs |
Adds shared clearing of recorded process identities. |
crates/rocm-core/src/proc_lifecycle.rs |
Centralizes termination and adds Windows creation-time checks. |
crates/rocm-core/src/lib.rs |
Exports shared stop APIs and types. |
apps/rocmd/src/service.rs |
Replaces private stop logic with shared routines. |
apps/rocm/src/services_text.rs |
Provides platform-specific process-check commands. |
apps/rocm/src/service_launch.rs |
Adapts tests to shared outcomes and platform-specific advice. |
apps/rocm/src/service_control.rs |
Uses shared termination and clears identities after confirmed stops. |
Review effort: Balanced
…he feature's ids stay sequential The scenario was numbered 09 to leave room for a restart scenario that belongs to a separate change, but the suite requires each feature's scenario names to be indexed sequentially in declaration order. Renumber it to 08 and update the references to it in the feature comment, docs/testing.md (including the name filter that selects the stop scenarios) and the rocmd stop test's comment. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ueried, or when its tree kill did not complete, and hold the PID across the stop On Windows three things let a stop report success while a process of the service might still run. A live process the stop may not open made OpenProcess fail, and process_is_running read every failure as "not running", so the identity gate classified it Gone and the stop reported it already gone. The liveness gate now keeps a failed query apart from an exit: ProcessProbe::Unqueryable is Indeterminate, so the stop reports it unverified and keeps the record and the endpoint key. Only the OpenProcess code for "no such process" counts as gone. process_is_running itself is unchanged. The creation-time comparison closed its handle before taskkill ran by PID, leaving a window for the process to exit and an unrelated one to take the PID. The PID is now held open from before the identity read until the stop returns; Windows does not hand a PID to another process while a handle to it is open. The taskkill result was discarded, and terminate_verified can only watch the root on Windows, so a descendant taskkill did not take down went unnoticed. The tree kill now returns whether taskkill started and exited cleanly, and a failed one turns any verdict that would confirm the stop into timed_out, which keeps the record, the stop request and the endpoint key. The decisions sit behind pure functions and injected steps so the tests run on every host: identity_state_with, classify_open_failure, classify_taskkill, after_tree_kill and terminate_recorded_pid_with with an injected hold. The Windows API calls themselves (the handle, the error codes, a real taskkill failing) run only on the Windows lane. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Windows stop handling can hang indefinitely on a stalled taskkill, and subsequent liveness refreshes can delete endpoint state for an inaccessible process that is still running.
4 open issues
2 resolved since last review
Review effort: Balanced
| ) -> IdentityState { | ||
| match probe(id.pid) { | ||
| ProcessProbe::Gone => IdentityState::Gone, | ||
| ProcessProbe::Unqueryable => IdentityState::Indeterminate, |
There was a problem hiding this comment.
The finding is right. After a stop reports an inaccessible Windows process as unverified, it deliberately leaves the stop request and the endpoint key on disk so a later confirmed exit can finish the cleanup. But settle_pending_stop_key_cleanup, which runs at the start of every liveness refresh (for example when services are listed), still decided "gone" from process_is_running, and on Windows that returns false for any process the caller cannot open. So the next listing would delete the key, clear the stop request and, a few lines later, reconcile the record to stopped, undoing exactly what the unverified verdict was protecting. Fixed in 4ebcdf6.
The cleanup, the reconciliation to stopped, and the in-use check behind rocm services remove and prune (which has the same shape and could delete a record, its log and its key under a live process) now read each recorded PID through probe_process and treat anything but a confirmed exit as possibly running, using a new ProcessProbe::may_be_running. Only gone may delete the key and clear the marker; unqueryable keeps both and keeps the record live. Each function takes the probe as an argument so the tests run on every host. The regression tests are a_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried, which refreshes a pending-stop record with an unqueryable probe and asserts the key file, the stop marker and the non-stopped status are kept, then with a confirmed-gone probe and asserts the key is dropped, the marker cleared and the record stopped; an_unverified_stop_keeps_its_key_and_marker_through_a_later_liveness_refresh, which runs a real unverified stop and then the refresh over the record it wrote; and a_record_is_in_use_unless_every_recorded_process_is_confirmed_gone for the removal guard. Reading unqueryable as not running again (the old behaviour) turns the first two red in the cleanup and the first red in the reconciliation, and turns the third red in the removal guard; I restored each afterwards. The real OpenProcess access-denied case can only be exercised on the Windows lane.
I swept every caller of process_is_running and the places that decide "gone" from a boolean. Fixed, as the same stop and cleanup family: the key cleanup and the reconciliation in managed_service_status.rs, and the removal guard in service_records.rs. Left alone, with reasons: ensure_spawned_child_survives_startup, the attached log streams and service_pid_is_alive in service_launch.rs watch a child this very command just started with the same account, and are not a decision about a recorded service from an earlier command; background_helper.rs checks the helper's own daemon PID and at worst starts a second helper, with no key or record involved; the ComfyUI stop in comfyui/process.rs has its own tasklist-based check, which lists a process whatever its access rights, and manages a different state file (its taskkill call has no deadline either, which is a sibling of the second finding that I did not change here); the remaining hits in rocmd and rocm-dash-collectors are tests or a test helper. process_is_running itself is unchanged and its documentation now points at probe_process.
…and bound the Windows tree kill with a deadline The previous change made a stop report a Windows process it cannot query as unverified and keep the record, the stop request and the endpoint key. The liveness refresh that runs when services are listed then undid that: it still called process_is_running, which reads a process it cannot open as not running, so it deleted the key, cleared the stop request and reconciled the record to stopped while the process might be alive. The same call decided whether a record may be removed or pruned. All three now read each recorded PID through probe_process and treat anything but a confirmed exit as possibly running (ProcessProbe::may_be_running, exported from rocm-core): the deferred key cleanup, the reconciliation to stopped, and the in-use check behind services remove and prune. Each takes the probe as an argument, so the tests drive an unqueryable process on every host. process_is_running itself is unchanged; the launch-time callers that watch a child this process just started keep it. The taskkill that takes a Windows tree down ran through Command::status, which waits without limit and runs before the stop's bounded waits, so a stalled taskkill hung the stop. It now runs under a deadline of MANAGED_STOP_GRACE: a child still running at the deadline is killed and reaped, the run reports a timeout, and that is a failed tree kill, which the stop already treats as unconfirmed. The wait is a plain helper over std::process, tested on every host with a long-sleeping child (ping on Windows) under a short deadline; only the taskkill wiring is Windows-only. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Windows runtime confirmation is still needed for process creation-time checks, PID-reuse protection, and taskkill tree termination because static review and injected-function tests cannot validate these OS-specific safety guarantees.
2 open issues
2 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Critical stop-handling gaps can let refresh followed by remove/prune delete records for still-running containers and let stop confirmation overwrite a concurrent relaunch’s status and delete its endpoint key.
4 open issues
1 resolved since last review
Review effort: Balanced
| if rocm_core::engine_stop_decides_confirmation(&record.engine) { | ||
| return false; |
There was a problem hiding this comment.
Right: the guard I added in the last round protected only the cleanup, and the rest of the refresh and the removal guard still read the supervisor's PID as evidence. For an unresolved container stop (a stop was requested, and the engine is one whose Stop decides confirmation, today aim) the record is now protected in all three places through one predicate, container_stop_unresolved: the cleanup does not complete, the refresh returns before it would reconcile the status from the PIDs (whether or not the endpoint answers), and the in-use check behind remove and prune counts the record as in use. The way out is the one the stop already has: services stop again, which asks the engine. a_pending_container_stop_survives_refresh_remove_and_prune runs a refresh, remove --yes and prune --yes over an aim record with the supervisor confirmed gone, for a record reading ready and one already reading stopped (which only the in-use guard protects), and asserts nothing was deleted, with a process engine in the same state removed as before. Removing the check from the in-use guard or from the refresh turns it red. Fixed in 2ca770d.
juhovainio
left a comment
There was a problem hiding this comment.
I read through the full diff. Consolidating the stop routine into rocm-core with the injectable Windows steps is the right shape, and the new scenarios assert the printed messages together with on-disk state. I have a few things I'd like addressed or answered before this merges, left inline where they have a line to anchor to:
- An "unconfirmed" stop (
ExitedBeforeTreeKill/DescendantsUnconfirmed) doesn't stay unconfirmed: the nextrocm services stop, or a plain listing, sees the root gone and confirms it with no new evidence about the descendants. The "a process under it may still be running" warning is only informational as written. windows_system_toolbuilds its path from theSystemRootenvironment variable, which undercuts the goal of the tools not being substitutable through the environment.- The pending-stop settle path now forces
stoppedfor any record with a stop marker whose PIDs are all gone, including records that name no process. That shape has no test. - Scope: the routine merge,
remove --force, the ComfyUI System32 change, the Windows creation-time token and the failed-launch cleanup are all in one ~5.8k-line PR, and go beyond what ROCMAI-874 asks for. The ComfyUI System32 change in particular could stand alone. Please say whether you'd rather split it. - Windows tree kill, the
taskkillfailure paths, launch cleanup of a real child andservice-stop-07run in no CI job. The answer to the ticket's "which lane runs this on Windows" is currently "none", so a manual Windows host check fromdocs/manual-testing.mdshould be recorded before merge.
I didn't build or run the tests, so the mutation/regression claims in the description are unverified on my side.
| // deny a stop this command did perform, so it counts as forced; but when | ||
| // the kill found no such process the root went by itself, and | ||
| // `AlreadyGone` is the truth. | ||
| let outcome = match (tree, terminate(identity)) { |
There was a problem hiding this comment.
These two verdicts are only good for one run: the next rocm services stop reads this root as gone and reports it confirmed, and a plain listing completes the pending stop the same way, neither with new evidence about the descendants. Either say so in the printed report and docs, or keep a durable marker so the "may still be running" warning isn't cleared by one more command.
There was a problem hiding this comment.
You are right that the warning lasts one run. I took the smaller option, saying so, rather than a durable marker, which would need a new field in the record and a rule for clearing it that the ticket does not ask for. The printed not confirmed: lines for descendants_unconfirmed and exited_before_tree_kill now end with "rocm does not keep this warning, so a later stop or listing will find the process gone and report the stop confirmed", and docs/testing.md says the same. The test asserts the message and the state together: the report test checks the line and that the record, marker and key are kept, then runs the next stop over the same record with the root gone and asserts it is reported confirmed, so the sentence is checked against what actually happens. Dropping the sentence turns it red. If you would rather have the marker I can add it, but it changes the record format and the restart and recovery paths, so I would do it as its own change.
| /// So a directory earlier on `PATH` cannot stand in for it; the bare name when | ||
| /// `SystemRoot` is not set. `name` includes the extension (`taskkill.exe`). | ||
| #[must_use] | ||
| pub fn windows_system_tool(name: &str) -> PathBuf { |
There was a problem hiding this comment.
This builds the path from the SystemRoot environment variable, so a controlled environment can still redirect taskkill/tasklist. GetSystemDirectoryW (or GetWindowsDirectoryW) doesn't read the environment and would match the stated goal that nothing the caller controls can substitute these tools.
There was a problem hiding this comment.
Right, SystemRoot is a variable a caller controls. windows_system_tool now takes the directory from GetSystemDirectoryW, which does not read the environment, behind the same seam: the pure system_tool_path takes the directory, so the path logic stays unit-tested on every host (a_system_tool_is_resolved_under_the_system_directory, red if the directory is ignored), and the Windows call is the only part that is not, with its SAFETY comment. A length of zero or one that does not fit the buffer falls back to the bare name rather than trusting a truncated path. Off Windows there is no system directory and the answer is the bare name, as before. The Windows-target clippy of the whole workspace is clean, and no test on this machine runs the Windows call, which the PR body says; the Windows job's tests that start taskkill do run it. Fixed in 2ca770d.
| // stop still finalizing would report itself unconfirmed. Setting it also makes | ||
| // the caller's liveness gate return at once, so the one write the caller makes | ||
| // covers the key cleanup, the cleared processes and the status together. | ||
| "stopped".clone_into(&mut record.status); |
There was a problem hiding this comment.
This now sets stopped for any record with a stop marker whose recorded PIDs are all confirmed gone, including a record that names no process at all (e.g. a starting record with a stale marker). Before, only the key and marker were cleaned. Is that intended? The tests cover live statuses with PIDs; consider limiting it to records that named at least one process, or add a test for the no-PID case.
There was a problem hiding this comment.
Not intended, thank you. The cleanup now goes on to forget the processes and read the record as stopped only for a record that named at least one process; a record that named none, such as a starting one with a stale marker, has its key and marker cleaned as before and is otherwise left as it was, because it may be a launch that has not recorded its supervisor yet. a_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stopped is the no-PID test: it asserts the marker and key are gone and the status is still starting. Removing the condition turns it red. Fixed in 2ca770d.
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 08c5f445
Review — needs work
Full review of the whole change, fanned out across 8 independent file-coupling groups given the size (30 files, +5786/-555 lines): the shared rocm-core stop routine, its process/identity support layer, the rocm stop/status caller, the rocm records/remove/prune/CLI-text cluster, launch-cleanup + ComfyUI's System32 calls, the rocmd/supervise/vLLM callers, the e2e-cucumber suite, and docs/README. Implements a shared rocm-core termination routine with Windows identity/liveness/tree-kill hardening, services remove --force, System32 tool resolution, and stricter failed-launch cleanup, as described; claims to close ROCMAI-874 (an internal AMD tracker ticket — named here but not retrieved or verified, per this repo's convention for externally-visible review automation).
Blocking
-
Required
clippyCI check is currently failing at this exact head on the module-size gate —apps/rocm/src/cli.rs:1502 production lines, 2 over the repo's 1,500-line cap (cargo xtask check-module-size, run id 129880496283, step output:error: 1 source file(s) over the limit of 1500 production lines (of 412 checked): apps/rocm/src/cli.rs: 1502 production lines (2 over), exit code 1). This also means the PR's own Test Plan text ("cargo xtask check-module-sizereports no file over the limit... 412 files checked") no longer holds at this head — it was evidently true on an earlier commit (the file sat at ~1499 lines per independent review) and was pushed over by 2 lines since, likely by the finalmerge maincommit.
Confidence 100 · architectural · Fix: splitapps/rocm/src/cli.rs(a pure move, tests included) per the repo's own documented remedy, or trim 2 production lines back under the cap. -
service-stop-07(the Windows process-tree-kill scenario) has no CI lane that actually executes it, despite meeting the project's own established criteria for folding such a scenario into the required job —tests/e2e-cucumber/features/service_stop.feature:162-171(@id:service-stop-takes-down-the-whole-process-tree-on-windows,@requires-os:windows, "No GPU, no engine, no network"). The requiredwindows-releasejob already folds comparably hermetic,@requires-os:windows, no-e2e-test-hooksscenarios into its otherwise-lifecycle-only run viaE2E_ALSO_RUN_IDS(.github/workflows/ci.yml:1009-1010:comfyui-uv-install-failure-names-log,serve-vllm-refused-on-native-windows, etc.), with an in-repo comment explaining exactly why ("the self-hosted Windows lane that otherwise runs them... is optional and not a required check").service-stop-07matches that profile exactly but was not added, and no reason is given. As a result the PR's central new safety mechanism — the identity-gated Windows subtree kill the whole PR exists to harden — has zero automated execution anywhere in CI; only two#[cfg(windows)]unit tests inproc_lifecycle.rsexercise the underlying primitives against a real process with no child.
Confidence 80 · architectural · Fix: addservice-stop-takes-down-the-whole-process-tree-on-windowsto thewindows-releasejob'sE2E_ALSO_RUN_IDS. -
Two module doc-comments misdescribe the production call graph after this PR's own
_withrefactor —apps/rocm/src/service_records.rs:19-20states "services_cmd.rs'sservices()callsremove_managed_service_record...";apps/rocm/src/services_text.rs:51-52states "service_records.rsreadsvalidate_service_id(in...remove_managed_service_record) andload_managed_service(inremove_managed_service_recordandapply_service_prune_plan)." Verified directly:remove_managed_service_recordandapply_service_prune_planare now#[cfg(test)]-only wrappers (service_records.rs:251-252,:711-712) that cannot run in a release build;services_cmd.rs:41,82confirms the real production caller isremove_managed_service_record_with, and the realvalidate_service_id/load_managed_servicecalls live inremove_managed_service_record_with/apply_service_prune_plan_with(service_records.rs:270-271,735). These docs make a concrete, checkable false claim about what calls what, introduced by this PR's own rename/split and never updated.
Confidence 85 · mechanical · Fix: update both doc comments (andservice_hints.rs:43's "seeremove_managed_service_record" pointer) to name the_withfunctions. -
ComfyUI's two Windows process-tool call sites have zero test coverage pinning them to the new System32 resolution —
apps/rocm/src/comfyui/process.rs:420-422(windows_process_is_running) and:470-471(terminate_windows_process_tree) both correctly callrocm_core::windows_system_tool("tasklist.exe"/"taskkill.exe")(verified: both call sites fixed, not just one). But the file's only test,running_state_requires_saved_process_and_reachable_port(:504-526), never touches either function. A future edit that silently reverted either call site back to a bareCommand::new("tasklist")/Command::new("taskkill")— reopening exactly the PATH-hijack exposure this PR closes — would go undetected, and this is cheaply testable (e.g. asserting onCommand::get_program()) without a real Windows host.
Confidence 75 · mechanical · Fix: a small test per call site (or a shared helper tested once) asserting the builtCommand's program path runs throughwindows_system_tool. -
A new bare-backtick file citation in
docs/manual-testing.mdviolates this repo's own markdown-link convention —docs/manual-testing.md:625:two `#[cfg(windows)]` tests in `proc_lifecycle.rs`. Confirmed this is the sole bare-backtick.rscitation in the file; every other.rsreference (e.g. lines 486, 1469, 1502) is a proper relative link. AGENTS.md §5 requires citing a tracked repo file as a relative markdown link specifically so thelycheelink checker can catch future drift.
Confidence 75 · mechanical · Fix:[proc_lifecycle.rs](../crates/rocm-core/src/proc_lifecycle.rs).
Non-blocking
None at ≥70 confidence. (Several lower-confidence candidates — a taskkill-kill-failure edge case in wait_for_exit_code, a blocking call inside rocmd's tokio runtime without spawn_blocking, a narrow self-healing stop_requested_unix_ms race, a windows_system_tool PATH fallback when SystemRoot is unset — were investigated and dropped: each is either traced to a pre-existing, unworsened pattern, or remains speculative with no demonstrable harm.)
Decisions for the author
rocmd'sterminate_recovery_supervisor(apps/rocmd/src/supervise.rs:787-797) still callsterminate_verifieddirectly rather than the newterminate_recorded_pid/terminate_recorded_pid_within— tradeoff/pre-existing. Confirmed byte-for-byte unchanged by this PR, so on Windows a recovery-supervisor cleanup still only kills the launcher root, not its subtree — the exact gap this PR just fixed for the two main stop paths. Not introduced or widened here, but worth a follow-up given the sibling fix just landed.TREE_KILL_DEADLINEignores a caller's shortergrace— tradeoff.terminate_recorded_pid_within's Windows descendanttaskkillis always bounded byMANAGED_STOP_GRACE(10s) even when the failed-launch cleanup passes a nominal 2s grace. Documented as intentional (one well-tested deadline vs. a path that could silently take 5x longer than its parameter name suggests) — confirm this is the intended tradeoff.- Add
service_forced_remove_hintto the sibling hostile-input proptest (apps/rocm/src/service_hints.rs:44-49vs. thea_hint_denotes_the_argv_it_namestest at:124-147) — non-blocking-improvement. Reuses an already-tested quoting call, so risk is low, but it's the one hint-builder the module's own anti-drift guard doesn't cover. named_processes's "processes X and Y" plural branch (apps/rocm/src/service_records.rs:213-224) is never exercised by this feature's own tests (the one two-PID test short-circuits via aRunningprobe) — non-blocking-improvement, low risk given the trivialjoin(" and ")logic.- No Windows-host test runs
abandon_failed_managed_launchagainst a real spawned child — answered-by-intent. The PR's own text states plainly: "no test runs a failed launch against a real Windows child... it rests on the routine's own tests and review," arguing this is impractical without Windows CI hardware running the e2e target. Accepted as an honestly-argued, acknowledged limitation rather than a fresh finding. - This is a very large, bundled PR (30 files, 31 commits) — tradeoff. The description pre-justifies bundling the shared routine,
--force, System32 resolution, and launch-cleanup hardening together ("none of them stands alone... without a Windows host to show it"). Accepted as reasoned, but it is worth recording that a change this size strains what any single review pass — automated or human — can cover with full confidence; this review required an 8-way fan-out to attempt full coverage.
Positive signals
- The Windows identity gate and tree-kill state machine (
classify_live_identity,refusal_before_signalling,classify_taskkill,wait_for_exit_code) are split into pure/injectable functions exercised by unit tests on every host, including explicit stalled-helper and deadline paths — unusually thorough compliance with this repo's "platform-gated code gets a test that runs on its gate" rule. windows_system_toolresolvestaskkill/tasklistunder%SystemRoot%\System32everywhere in the repo, including both of ComfyUI's call sites — confirmed no remaining bare-name invocation anywhere, closing a real PATH-hijack exposure.- This PR fixes a genuine pre-existing Windows safety bug: the
remove/prunein-use guard previously mapped an access-denied (Unqueryable) PID to "not running," which could have deleted a record (and its endpoint key) out from under a still-serving process it merely couldn't query. The newrecorded_pids_may_be_runningreasoning is confirmed used identically by all three destructive call sites (deferred cleanup, status reconciliation, removal guard) with no drifted fourth copy. --force's narrow gating (stop requested, nothing confirmed running, at least one unqueryable) was traced branch-by-branch with no inverted boolean, and--yesis independently and unconditionally still required alongside it.- Extensive mutation-style evidence in the PR's own test plan (each guard tied to a named test that goes red when it's removed), independently corroborated by this review's own code-reading.
Deployment notes
rocm services stop(andrestart, and the sandbox stop tool) now perform a Windows process-subtree kill (taskkill /T) they never did before, and now clear recorded PIDs/start-time tokens from the record on a confirmed stop. Operators who scripted around the old behavior (engine process left running after a stop; stale PIDs persisting in the record) should expect different behavior after upgrading.- A record whose recorded PID becomes permanently unqueryable (e.g. a permissions change) now stays
unverifiedindefinitely — it never reconciles tostopped, andremove/prunerefuse it — until an operator runsrocm services remove <id> --force --yes. This is a deliberate, documented tradeoff, not a regression, but it is a new persistent-state outcome operators should know about.
What this covered
Fanned out across 8 independent, blind sub-reviewers, each reading its assigned files completely at head commit 08c5f445aea874193501b1ec73a4a7c4be967892 (base c4e1f72635e42b90f9917a0dbbb7aa00bcc344e5, a clean three-dot diff with no stacked-branch contamination — merge-base verified equal to the PR's reported base): (1) crates/rocm-core/src/proc_lifecycle.rs; (2) process.rs/service_record.rs/lib.rs; (3) apps/rocm/src/service_control.rs/managed_service_status.rs; (4) service_records.rs/services_text.rs/service_hints.rs/services_cmd.rs/cli.rs; (5) service_launch/launch_transaction.rs/attach.rs/comfyui/process.rs; (6) apps/rocmd/src/service.rs/supervise.rs/engines/vllm/src/state.rs; (7) the full tests/e2e-cucumber suite; (8) README.md/docs/architecture.md/docs/manual-testing.md/docs/testing.md. All 8 reported real, substantive findings; none degraded. The consolidating pass independently re-verified the highest-value candidates directly against the source and against live CI logs/checks rather than accepting self-reported confidence scores, which is how the module-size CI failure and the service-stop-07 CI-gap finding were confirmed. Design review (Step 7) was done by the consolidating pass directly (reasoning through blast-radius, tradeoff and completeness questions) rather than via separate sub-workers, given the scale of fan-out already performed.
Did not run: the "prior changes" pass (review comments on earlier changes to these 30 files) — impractical to search exhaustively here and no standing objections were supplied. History and code-comments passes were performed per-file by each sub-worker rather than as a separate pass. Not reconciled against this PR's existing discussion or other reviewers' comments (a Copilot review and a human CHANGES_REQUESTED review exist at this head) — this review was read blind and does not know what they say.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
cli.rs is checked by check-module-size, which blocks, and main keeps adding lines to it, so the services help text this branch adds repeatedly put it over the 1500 line limit. Trimming the help text again would only buy a few lines until the next merge. The clap value enums behind --framework, --format, --provider, --device and the similar arguments (FrameworkArg, InstallFormat, Provider, WatcherModeArg, TelemetryModeArg, DevicePolicyArg, SandboxToolArg, PermissionsModeArg) and their conversions are self-contained, so they move as they are to cli_args.rs, and cli.rs re-exports each one so every module keeps reaching them as crate::cli::Provider and so on. Nothing else changes: attributes, help text, aliases and declaration order are as before. cli.rs goes from over the limit to about 1348 production lines, leaving real headroom, and docs/architecture.md has the new file's entry. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ell an empty relaunch from the stop, and resolve system tools without the environment An unresolved stop of a container-backed engine (a stop was requested and the engine Stop decides confirmation, today aim) is now protected everywhere, not only in the deferred cleanup. The liveness refresh does not read such a record as stopped from the supervisor's PID, and the in-use check behind services remove and prune counts it as in use, so a listing followed by remove or prune no longer deletes a record and key whose container may still be running. One predicate, container_stop_unresolved, is the single reading. is_covered_by_stop_of no longer treats an empty relaunch as the stop. When the stopping snapshot names no process, a record naming none is covered only while it still carries the stop's marker or reads stopped; an empty starting or recovering relaunch has cleared the marker, so the stop does not confirm over it, overwrite its status or delete its key. The deferred cleanup goes on to forget the recorded processes and read the record as stopped only for a record that named at least one process. A starting record with a stale marker gets its key and marker cleaned, as before, and is otherwise left alone. The two unconfirmed tree-kill lines now say that rocm does not keep the warning, so a later stop or listing finds the process gone and reports the stop confirmed, and the test runs that later stop to check the sentence. windows_system_tool reads the system directory with GetSystemDirectoryW instead of building it from the SystemRoot environment variable, which a caller can redirect. The pure path logic keeps its seam and its test. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Fixed in 2ca770d and answered inline (container stop guard, empty relaunch record, honest unconfirmed-stop message, GetSystemDirectoryW, no-process stop marker), with cli.rs back under the size limit in bd168e9. The ComfyUI change stays in this PR (no split). Still open and said so in the PR body: the unconfirmed-stop warning is not kept across runs, and the Windows host check from docs/manual-testing.md has not been run.
Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # tests/e2e-cucumber/src/lib.rs
There was a problem hiding this comment.
🟡 Changes recommended
A concurrent listing can erase evidence of an in-flight stop with no recorded processes, causing stop confirmation to fail and blocking restart even when no relaunch occurred.
4 open issues
1 resolved since last review
Review effort: Balanced
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…g consumes it off an empty record A stop that begins on a record naming no process writes its marker and has nothing to signal. A concurrent listing finds the marker with no process to wait for and clears it, but leaves a starting or failed record at its status, so the stop's re-read saw an empty record with no marker, which is also what a relaunch that has not recorded its supervisor yet looks like. The stop reported itself unconfirmed, and so did the restart built on it. The refresh now keeps the consumed marker's value in settled_stop_unix_ms, and is_covered_by_stop_of accepts an empty record whose stamp equals the stop's own request stamp. A restart clears the stamp, so a real empty relaunch is still not covered. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
An older stop can still delete the replacement service’s endpoint key before its PIDs are published, and the stop-marker regression test never exercises its intended interleaving.
5 open issues
1 resolved since last review
Review effort: Balanced
…k, where an empty record can reach it The test of a stop whose marker a concurrent listing consumed put the simulated listing in the per-PID termination callback. The record names no process, so that callback never ran and the test passed whether or not the settled stamp was honoured. The rewrite now happens in the engine stop callback, which does run for a record naming no process, the termination callback is unreachable, and the test asserts that the interleaving occurred. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…start leaves before it publishes Between its own confirmed stop and the publication of its child, a restart leaves an empty stopped record, restores the endpoint key and spawns. That record is also what an older stop's own completion looks like, so an older stop still finalizing took it for its own, wrote stopped over it and deleted the key the restart had just restored for its child. The restart now stamps a relaunch generation (relaunch_started_unix_ms) on the record before it restores the key or spawns. A stop covers a record only in the generation it read, and publishing the child clears the stamp. The restart's own writes also refuse a record whose generation moved since it read it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed for a critical stop/relaunch race that can delete the replacement generation’s restored endpoint while its publication guard still accepts the record.
3 open issues
2 resolved since last review
Review effort: Balanced


Closes ROCMAI-874.
Summary
Besides the shared stop routine, this PR contains changes a reader should know about up front, named here because they are not obvious from the title alone:
rocm services remove --force(withprunelisting the record and naming that command), theROCM_E2E_UNQUERYABLE_PIDSe2e hook, System32 resolution of the Windows tools, including ComfyUI's owntasklistandtaskkillcalls, and a changed failed-launch cleanup on Windows. They are kept in one PR because they share the one routine and its Windows tests, and none of them stands alone without a Windows host to show it:remove --forceand the e2e hook exist only because the routine now reports a process it cannot query as unverified, and without them that record could never be cleared or tested (the hook is what letsservice-unqueryable-01and-02plant such a process on Linux); the Windows start token is what lets the routine's identity gate refuse a recycled PID beforetaskkill /Ttakes its tree, so shipping the tree kill without it would widen the very exposure the ticket is about; System32 resolution is the same routine'staskkillandtasklistcalls resolved from the system directory, applied to ComfyUI's calls that go through the same helper so there is one rule for it; and the failed-launch cleanup is the third caller of the termination, which would otherwise keep a Windows tree-kill gap that the other two callers have just lost. Splitting them would leave the first PR with either a tree kill that can hit a recycled PID or a state no test can reach. Each is described below with the test that covers it.What was and was not executed. Nothing in this PR has been run on a Windows host by me, and this text does not claim a Windows result for this head. From
.github/workflows/ci.yml: thewindows-debugjob, when Rust files change, runs clippy, the build,cargo test --workspace --all-targetsand the harness self-tests. That test run includes two#[cfg(windows)]unit tests inproc_lifecycle.rs(terminate_recorded_pid_leaves_a_recycled_pid_alone_and_stops_the_verified_oneandterminate_recorded_pid_stops_a_process_recorded_without_a_token) that start a realpingprocess and stop it throughterminate_recorded_pid, so the PID hold, the creation-time read, the identity gate (a wrong token leaves the process running) and the System32taskkill /T /Frun on a real process there. It does not run any scenario (the e2e target is excluded fromcargo test): theservice-stop-*andservice-unqueryable-*scenarios do not run on that job, soservice-stop-07(@requires-os:windows) and the Windows half ofservice-stop-08can only run through the harness on a Windows host, anddocs/manual-testing.mdhas a Windows host check that selects only the scenarios that apply there (service-stop-0[178]andservice-unqueryable-01). Not exercised by any CI job: the end-to-end tree kill of a parent with a child, the failure cases that cannot be staged (anOpenProcessaccess-denied process,taskkillexiting 128, a failing or stalledtaskkill), and the launch-cleanup branch; thepingthose unit tests stop has no child of its own, so they do not show a descendant being taken down.rocm services stop(androcm services restartand the sandboxstop_servertool, which stop first) androcmd'sstop_servereach carried their own copy of the code that terminates the processes a managed-service record names: a privateMANAGED_STOP_GRACEof 10 seconds that was kept in step with the other copy only by a comment, and a privateterminate_recorded_service_pids. The two copies had drifted.rocmdreported the stopping process's own PID as aselfrow androcmsilently dropped it, so the same record produced a differentpid_outcomeslist from each binary. On Windows onlyrocmdrantaskkill /PID <pid> /T /Fbefore terminating the recorded process, even though the engine runs one level below that PID and the verified termination cannot walk a process tree there, sorocm services stopcould leave the engine behind.rocmdalso cleared the recorded PIDs and start-time tokens from the record once a stop was confirmed, whilerocmleft the record naming processes that were already gone.This change moves one routine and one constant into
rocm-core'sproc_lifecycle.rs, next toterminate_verified, and deletes both private copies.terminate_recorded_service_pidswalks the processes a record names (supervisor first, then engine, each paired with its own start-time token, a PID recorded under both roles visited once) and returns oneRecordedPidStopper PID;terminate_recorded_pidterminates one of them and, on Windows, kills its descendants first;MANAGED_STOP_GRACEis the single grace period. The per-PID termination is passed in, so each caller's tests drive its own stop through the real routine with the signal replaced, and the Windows steps sit behind a function that takes the tree kill, the PID hold, the identity read and the termination as arguments so its tests run on every host.rocmkeeps only a thinstop_recorded_processesthat adds the existinge2e-test-hooksseam. Each caller has a source check that no private copy of the routine, the constant or ataskkillcall has come back; those checks pin names only, so they are a tripwire for a copy returning under the same name, not coverage of what either stop does.Intended user-visible differences, where the two behaviours disagreed and the more complete one (
rocmd's) won:rocm services stopnow clears the supervisor and engine PIDs and start-time tokens from the record (oneManagedServiceRecord::clear_recorded_processes, called by both binaries), so a later stop of the same record reaches no stale PID. An unconfirmed stop still keeps them, because the record is the only thing that names a survivor.rocm services stop|restart, the sandboxstop_server/restart_servertools and the attachedrocm serveCtrl-C stop now report the stopping process's own PID as aselfrow, likerocmd, and on Windows take the engine's subtree down with the recorded process.GetProcessTimes), where it was always absent. A record written by this version therefore refutes a PID that an unrelated process has since taken: the stop reportsidentity_mismatch, skips the tree kill and leaves that process alone, instead of killing a stranger and its subtree. A record written before this has no token and stays best-effort. A recorded token that can no longer be read back from a running process reads as unverified and is not signalled. That is one of two routes to an unverified stop on Windows; the other is a PID that cannot be queried at all (next bullet). The advice a stop prints for an unverified process namestasklist /FI "PID eq <pid>"on Windows, where it used to nameps -p, which Windows does not have.ProcessProbe; Linux already counted a live process it may not signal as running). A recorded token that can still be read and differs outranks the failed query, but on Windows the token and the query are read with the same access right, so that only reaches a process that opens and then fails the exit-code query. The same rule holds for what the stop leaves behind: the later liveness refresh (the deferred endpoint-key cleanup and the reconciliation of the record tostopped) and the in-use check behindrocm services removeandprunetake a recorded process for gone only whenprobe_processconfirms it, so an unverified stop's key and stop request survive a service listing. When that refresh does complete the pending stop (every recorded process confirmed gone), it also forgets the recorded processes, only for a record that named at least one (astartingrecord with a stale marker has its key and marker cleaned but is not read as stopped), and reads the record as stopped whatever it read before (afailedrecord too, as a confirmed stop sets it, so a stop still finalizing covers it), as a stop that confirmed the exit itself does, so a laterrocm services stopvisits no PID: a PID recorded without a token cannot be refuted, and on Windows the tree kill would take whatever process now holds the number. A process that may still be running stays named. A pending stop of a container-backed engine (engine_stop_decides_confirmation, todayaim) is never completed this way, is not read asstoppedby the rest of the liveness refresh, and counts as in use forremoveandprune(container_stop_unresolved;a_pending_container_stop_survives_refresh_remove_and_pruneruns a refresh,remove --yesandprune --yesover it, for a record readingreadyand one already readingstopped): its supervisor can be gone while the container still runs, so only the engineStopmay confirm it, and its key, marker and recorded processes stay for the nextservices stop(a_pending_stop_of_a_container_engine_is_not_completed_by_a_gone_supervisor; the situation is reachable because anaimlaunch records the supervisor anddocker run -dbackgrounds the container itself). Both stops (rocm services stopandrocmd's) decide whether their verdict still covers the record they re-read withManagedServiceRecord::is_covered_by_stop_of: the same processes, or none recorded because a listing completed this same stop first (statusstopped, nothing named), so (with an empty stopping snapshot too: an empty record is that stop only while it carries the stop's marker, readsstopped, or a listing consumed that very marker, which the refresh records as the marker's value insettled_stop_unix_msbecause it leaves astartingorfailedrecord at its status; an emptystartingrelaunch has cleared the marker without leaving the value, and a restart wipes it, so it is not covered) a listing that persists the cleanup between the termination and the final read does not turn a successful stop into an unconfirmed one or makeservices restartrefuse, including for a stop that began on a record naming no process. A restart is still not covered, whether it recorded new processes or is mid-launch with none yet (starting), and so is the emptystoppedrecord arocm services restartleaves between its own confirmed stop and the publication of its child: the restart stamps a relaunch generation (relaunch_started_unix_ms) on it before it restores the endpoint key or spawns, a stop covers a record only in the generation it read, and publishing the child clears the stamp, so an older stop still finalizing in that window neither claims the record nor deletes the restored key (the restart's own writes also refuse a record whose generation moved since it read it; a restart that fails after stamping leaves a stamp that every later stop's snapshot carries too, so it blocks nothing). The cost is stated rather than hidden: if a stopped service's recorded PID is later taken by a process the account cannot open, the record stays unverified for good (it never reconciles tostopped,removeandprunerefuse it, and a stop keeps reportingunverified). The way out isrocm services remove <service-id> --force --yes(for a record that still readsreadyand that nobody has stopped,--forceis refused and namesrocm services stopfirst, which reports the process unverified and records the request; that order is intended, because--forcenever removes a record nobody tried to stop, anda_ready_record_held_by_an_unqueryable_process_is_cleared_by_stop_then_forceruns the two commands), deliberately narrow: it removes the record only when a stop was requested, no recorded process is confirmed running, and at least one could not be queried; it says which process and that, if it is still running, it keeps running; it never goes past a processrocmcan see running; the plain refusal names this exact command, andprunehas no--force, but when it leaves such a record alone it lists it with the exactrocm services remove <id> --force --yescommand (a test runs the named command and checks it clears the record), and a forced removal that is missing--yesis told to add--yes, not to drop--force. It is user-observable, soservice-unqueryable-01drives the stop (unverified, with the check named), the refusal naming--force, and the forced removal through the real binary, andservice-unqueryable-02(Linux) shows--forcerefused for a real running process; both use thee2e-test-hooksvariableROCM_E2E_UNQUERYABLE_PIDS, which production builds never read. I looked at reading the creation time through a process snapshot instead, to tell a reused PID apart without opening the process:NtQuerySystemInformation(SystemProcessInformation)carries a create time without a per-process open, but it is only semi-documented, its buffer layout is not something I could check without a Windows host, and a wrong token would flag a real engine as recycled, so I did not do it; the Toolhelp snapshot has no creation time. The PID is held open from before the identity check until the stop returns, so the process whose creation time was compared is the one the kill reaches. And ataskkillthat failed to start, did not exit cleanly, or did not finish withinMANAGED_STOP_GRACE(it is killed and reaped at that deadline, so a stalled helper cannot hang the stop) leaves the stop unconfirmed, with the record, the stop request and the endpoint key kept, because only the recorded process's own exit is watched on Windows. When this stop's own wait saw the root exit (or it was already gone) the outcome isdescendants_unconfirmed, printed asnot confirmed: process <pid> (<role>) exited, but a process under it may still be running;timed_out, whose text says something outlived a deadline, is kept for a root that did. Ataskkillthat exits 128 (process not found) is judged by the root's own re-read instead. If the root is still running, the kill counts as failed (descendants_unconfirmedwhen the termination then saw the root exit). If the root exited by itself between the identity check and the kill, the outcome isexited_before_tree_kill:taskkill /Treaches children only through a live parent, so a child the root left running is out of its reach, and nothing recorded what was under the root. That is not a stop on its own. Across the record,terminate_recorded_service_pidsturns it intoalready_goneonly when another recorded process is confirmed gone, which relies on the engine being visited as its own role (it is), and both callers fold every row, so the record, PIDs and key are cleared only when every recorded role is confirmed. A root that is the only recorded process, or whose engine is running or unconfirmed, stays unconfirmed and printsnot confirmed: process <pid> (<role>) was already gone when the processes under it were to be taken down; the next liveness refresh still completes the deferred cleanup once the recorded PID is seen gone, and that now includes forgetting the recorded PID. The sharedtaskkill, and ComfyUI'staskkillandtasklist, are started from the system directory (windows_system_tool, which reads it withGetSystemDirectoryW, falling back to the bare name when it cannot be read) so neither aPATHentry nor aSystemRootvariable can substitute them. Independent verification of descendant exit there is not part of this change.OpenProcesshandle that keeps Windows from giving the PID to another process, and when it cannot be taken but the identity reads as matching, nothing reserves the PID between the read andtaskkill, which takes a whole tree by PID, so the stop is reported unverified and neither the tree kill nor the termination runs (a gone, recycled or unreadable PID keeps its own refusal; where there is no separate tree-kill step the missing hold changes nothing). A sibling whose PID a stranger now holds (identity_mismatch) does not vouch for what was under a root that exited by itself; only a stop, an already-gone process or another such root does.classify_live_identity, withrefusal_before_signalling), so a table test covers every combination of recorded and live token on every host.service_launch/launch_transaction.rs(abandon_failed_managed_launch) said Windows records no start token and only refused a child without one on Linux. Windows records one now, so a child whose token was not captured is refused there too, through a purelaunch_cleanup_refusaltested on every host, instead of falling through to a forced tree kill of a pid that may have been recycled. The comments that repeated the old claim were corrected. The cleanup also terminates the child through the same routine a stop uses (rocm_core::terminate_recorded_pid_within, the routine with the root's grace chosen by the caller, so the cleanup keeps its 2 second grace) rather thanterminate_verifiedalone: on Windowsterminate_verifiedcannot walk a process tree, so a failed launch could leave the engine one level below the supervisor running, which is the defect the stop had. The identity rule is unchanged and decided first (launch_cleanup_refusal). What the cleanup asks of the termination is tested through an injected function on every host (abandoned_launch_asks_the_termination_for_the_spawned_child_with_its_token); that the production wiring reaches the Windows tree kill is not observable on Linux, where the termination walks the tree itself, so it rests on the routine's own tests and review, and no test runs a failed launch against a real Windows child. The tradeoff is deliberate and written in the code: a child whose creation time could not be read at capture is not killed, so ifOpenProcessfailed for a child that is in fact alive (it is this account's own fresh child, so that is not expected), a failed launch leaves its engine running and the audit event saysunverified; the alternative is a forced tree kill of a PID that may now belong to another process. A handle-based identity would need the spawn primitive to return its handle, which it does not, so it is not done here.stopped, and the in-use check behindremoveandprune) share one reading,recorded_pids_may_be_running, so they cannot drift apart on what counts as gone.The commits are in reading order: characterisation tests that pin what each copy did before the change, the merge into the shared routine, the record clearing, the Linux scenario for it, the Windows creation-time identity, the identity-verdict table, a teardown for the Windows scenario's processes, the renumbering of the last scenario so the feature's ids stay sequential, the Windows hardening described in the last bullet above, the follow-up that makes the liveness refresh and the removal guard read a process that cannot be queried as possibly running and bounds the
taskkillby a deadline, a further one that treatstaskkillexit 128 as a stop when the root is gone, lets a readable token mismatch outrank a failed query, and documents the two Windows routes to an unverified stop and the way out, one that verifies the launch cleanup's token on Windows and corrects the comments that said Windows records none, one that gives a root seen to exit under a failed tree kill its own outcome, resolves the Windows tools underSystem32and shares the may-be-running reading, and one that settles a root that exited alone across the record and adds the narrowremove --force; one that refuses an unverifiable first identity read before the tree kill can be skipped, restricts the corroboration, makesprunenameremove --force, and corrects the claims about where the Windows paths run; one that keeps the marker's value when a listing consumes it off a record naming no process, so a stop that began on such astartingorfailedrecord is still confirmed (settled_stop_unix_ms, cleared by a restart); one that makes therocmdtest of that run its listing in the engine stop callback, where a record naming no process can reach it; and one that stamps a relaunch generation on the emptystoppedrecord a restart leaves before it publishes, so an older stop finalizing in that window cannot claim it or delete the restored key; the merges ofmainare separate commits.Scenarios:
service-stop-06(rocm services stopof a real recorded process confirms the stop and leaves no process in the record; Linux, mock lane),service-stop-07(Windows only: stopping a server takes down the recorded process and the process it started, where atasklistthat does not exit successfully now fails the scenario instead of reading as an exit; no CI job runs scenarios on Windows, and it was not run on a Windows host), andservice-stop-08(stopping a server whose recorded PID now belongs to a different, running process leaves that process alone and still clears the record; runs on every OS). Scope of the Windows rule beyond the two stops: the unqueryable-process rule lives inidentity_state, so it reaches every caller ofterminate_verified; the PID hold and the tree kill live only interminate_recorded_pidand reach only therocmandrocmdstops. I read each caller. The lemonade engine'sstop_service(and vLLM's, which native Windows does not run) now answersstopped: falseand writes no terminal state for an unqueryable recorded PID, where it would have claimed a stop; the CLI's own stop reaches the same verdict for the same PID.rocmd'sterminate_recovery_supervisorsignals a child it just spawned and still holds, so the PID is openable and reserved. The launch cleanup is the bullet above. I also read every consumer of the outcome strings and every remainingprocess_is_runningcall site: the stop text inservices_text.rsis the only renderer of per-process outcomes (it now printsdescendants_unconfirmedandexited_before_tree_killlines);rocmd's MCP and sandbox answers passpid_outcomesthrough and key on thestoppedboolean, as do its watchers; the remainingprocess_is_runningcallers watch a child this command just started (launch waits, attached log streams), the helper daemon's own PID, or ComfyUI, which has its own check, and none deletes a key or a record on a failed query. None needed a further change. The scenario teardown also ends the process that 07's recorded process started when it is still that process, so a failed 07 does not leave it running behind a dead parent; that decision (leftover_action) has a table test that runs on every host. I was told thatservice-stop-07failed once in the original combined PR on a self-hosted Windows lane and passed on a re-run; I have not seen that result myself, and the scenario and the pure identity-verdict tests are unchanged.service-unqueryable-01and-02(their own feature file;-02Linux-only) cover the unqueryable-process stop, the refusal naming--force,prunenaming it, and the forced removal through the e2e hook; they ran on Linux only, and-01carries no OS tag but no CI job runs it on Windows. The other new Windows failure paths (a failed, stalled or not-foundtaskkill) have no scenario: they cannot be staged from outside the binary, so they are covered by unit tests with the system reads injected.Four gaps in the scenario coverage, stated plainly.
service-stop-07records no token, so the path a record written by this version takes on Windows (a token recorded, matching, then the tree kill) has no scenario; it rests onwhat_a_stop_does_for_each_recorded_and_live_tokenand the other injected tests, and on the Windows-only unit tests that stop a real process with its true token. And theselfrow thatrocmnow puts inpid_outcomeshas no scenario, deliberately: it appears only in the JSON answer of the hiddensandbox-run stop_serverand the sandbox tool, never in the text ofrocm services stop, and it names the stopping process's own PID, which a scenario cannot plant in a record before the process that will read it exists.a_stop_reports_its_own_pid_as_selfanda_self_row_adds_no_line_to_an_unconfirmed_reportcover the answer and the report. Third, thecheck:line an unverified stop prints (ps -p <pid> -o pid,args, ortasklist /FI "PID eq <pid>"on Windows) is a diagnostic, not a remediation: it shows whether the process is still running, and what clears the condition is the process ending (after which a stop or a listing confirms and drops the key, asserted bya_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queried) or, for a record held by a processrocmcannot open,rocm services remove <id> --force --yes, which the refusal andpruneboth name. The command's wording is pinned bythe_process_check_command_is_the_one_the_host_has, andservice-stop-07uses the sametasklistfilter, in CSV form because the default output is localised, to see whether a process is running; no test follows the printed command on a Windows host and reads its default output. Fourth, the interleavings of a listing with a stop that began on a record naming no process, and of an older stop with a restart between its own stop and its publication (the stop's key deletion is what the generation stamp prevents): it depends on a listing landing between the stop's termination and its final read, which a scenario cannot place, so it is covered bya_listing_clearing_the_marker_of_an_empty_stop_does_not_make_it_unconfirmed(a real refresh persisted at that point),a_relaunch_after_a_listing_consumed_the_marker_is_still_not_the_stop, theis_covered_by_stop_oftable androcmd'sa_stop_of_an_empty_record_whose_marker_a_listing_consumed_is_still_confirmed;an_older_stop_finalizing_before_a_restart_publishes_does_not_delete_its_keyruns the second as an outer stop whose termination step runs a whole restart with its spawn failed on purpose, so the record is left as it is at the spawn; none of the existingservice-stopscenarios covers either. I have not seen any Windows result for this head, and this text does not claim one.Split from #929 at the reviewer's request; the other parts are #961 (log-tail routine), #959 (engine IPC client) and the respawn guard, which follows once this and #959 have merged.
Test plan
Ran locally on Linux, in one cargo target directory. The first bullets were run on an earlier head; the bullet headed "On the current head" gives the figures for the head after the last fix and the last merge of
main:cargo test -q --locked -p rocm-core(1351 passed, 6 ignored), includingproc_lifecycle(54 tests),process::testsandservice_record.cargo test -q --locked -p rocmd(237 passed, 1 ignored; one run of the whole suite showedsupervise::tests::recovery_ignores_its_own_live_looking_record_on_its_own_portfailing once, and it passed alone and in two further whole-suite runs, so it looks timing-dependent and is not in a file this change touches).cargo test -p rocm --bin rocmfiltered toservice_control(46 passed),services_text(39 passed),service_launch(72 passed, 1 ignored),managed_service_status(15 passed) andservice_records(31 passed) on the final head, andcomfyui(105 passed, 4 ignored),cli(97 passed, 1 ignored),help(24 passed) andadvised(20 passed, 1 ignored) on the head before the last merge. I did not run the wholerocmtest binary or the workspace suite.rocm/rocmdbuilt with--features rocm/e2e-test-hooks,-n "service-stop-0[1-68]": 7 scenarios passed (01 to 06 and 08; 07 is Windows-only and was not selected);-n service-unqueryable-0: 2 passed (Linux);-n service-cleanup: 7 passed.cargo test -p e2e-cucumber --test feature_naming: 4 passed.cargo fmt --all --check,cargo clippy --locked -p rocm-core -p rocmd -p rocm --all-targets -- -D warnings,cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings,cargo xtask manifest --check,cargo xtask check-crate-edges,cargo xtask check-architecture-doc,cargo xtask lychee(local links) andcargo test -q --locked -p xtask prepush(98 passed) all exit 0.cargo xtask check-module-sizereports no file over the limit (412 files checked; this change adds nothing toservices_text.rs, which stays at 1457 of 1500 production lines, or toprocess.rs, 1411), now that main has splitservice_launch.rsintoservice_launch/.x86_64-pc-windows-msvc):cargo clippywith-D warningsforrocm-core,rocmdandrocm(all targets) ande2e-cucumber(all targets) exits 0 on the final head, merge included. This is compile and lint only.origin/main: the only hits are existinginternal_*identifiers; no conflict markers.mainthat brought in refactor(engine-ipc): share one engine stdio client between rocm and rocmd #959):cargo fmt --all --check;cargo clippy -D warnings --all-targetsforrocm-core,rocm,rocmd,e2e-cucumber,rocm-engine-protocolandxtask; the whole workspace forx86_64-pc-windows-msvcwith-D warnings(compile and lint only);cargo xtask check-module-sizeandcheck-architecture-doc;cargo test -p xtask prepush(98 passed);cargo test --no-fail-fastforrocm-core --lib(1449 passed, 6 ignored),rocmd(278 passed, 1 ignored) androcm --bins(1828 passed, 8 ignored), all whole suites.-n service-stopthrough the real harness: 7 of 8 scenarios passed;service-stop-07(Windows only) fails on this WSL host in its first step becausepowershellis not onPATHhere, before it reaches any code of this change. No Windows host was used.rocmcopy,a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothingfails (its record keeps naming the stopped PIDs) whilean_unconfirmed_stop_keeps_the_recorded_processes_for_the_next_stoppasses. The tests of the new API (the Windows identity table,ProcessProbe,TreeKill, the hold,is_covered_by_stop_of,tasklist_reports_running) call functions that do not exist on main, so they cannot run against a copy of it; for those the mutation checks below are the evidence that each test goes red when the branch it covers is broken.descendants_unconfirmed,exited_before_tree_kill) is a warning for one run, and the printed line now says so: rocm does not keep it, so the nextservices stopor listing finds the root gone and reports the stop confirmed, with no new evidence about what was under it. The test asserts the line and then runs that second stop and asserts it is confirmed, so the sentence is checked against the state; a durable marker in the record was the alternative and is not done.cli.rsintocli_args.rsas a pure move (re-exported, no other change), becausecli.rsis now checked by a blocking size check and main keeps growing it; it is at about 1348 production lines,services_text.rsstays at 1457 andprocess.rsis at 1437 with the system-directory read (it was 1411).clear_recorded_processescall fromrocm's confirmed branch turnsa_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothingred; mapping an unreadable live token to a match inclassify_live_identityturnswhat_a_stop_does_for_each_recorded_and_live_tokenanda_recorded_token_refutes_a_live_one_only_when_both_are_known_and_differred; reading an unqueryable PID as gone turnsa_pid_that_cannot_be_queried_is_unverified_not_already_goneanda_stop_of_a_live_process_it_cannot_query_is_not_confirmedred; reading every failed open as gone turnsa_failed_open_is_gone_only_for_no_such_processred; letting a failed tree kill keep the root's verdict turnsa_tree_kill_that_did_not_complete_leaves_the_stop_unconfirmedred; accepting anytaskkillexit code as clean turnsonly_a_clean_taskkill_exit_is_a_completed_tree_kill_and_128_is_not_foundred; dropping the not-found handling turns that test anda_tree_kill_that_found_no_process_is_a_stop_when_the_root_is_gonered; dropping the rule that a readable token mismatch outranks a failed query turnsa_readable_token_mismatch_outranks_a_failed_queryred; reading a process that cannot be queried as not in use in the removal guard also turnsa_record_held_by_an_unqueryable_process_can_be_cleared_by_hand_and_prunered; releasing the PID hold immediately turns five ordering tests red, includingthe_pid_is_held_before_its_identity_is_read_and_released_after_the_stop; removing the forgetting of the recorded processes from the deferred cleanup turnsa_stop_completed_by_a_later_listing_leaves_no_pid_for_the_next_stop,a_completed_pending_stop_reads_stopped_whatever_live_status_it_hadanda_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queriedred, forgetting them before the confirmed-gone check turns that last test and two neighbours red, dropping the status it sets turns the status test red, and reading a failedtasklistas a missing row turnsa_failed_run_is_no_answer_whatever_it_printedred; reading the stop's re-read as a relaunch whenever the recorded processes differ (the previous comparison) turnsa_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it,a_restart_is_not_refused_because_a_listing_completed_its_stopanda_stop_completed_by_a_concurrent_listing_is_still_confirmedred, covering a record that names no process whatever its status turns the two mid-launch tests and the table red, and covering every record turns the relaunch tests red; making the failed-launch cleanup drop the spawn token from the identity it asks the termination about, or treat a host as one that records no tokens, turnsabandoned_launch_asks_the_termination_for_the_spawned_child_with_its_tokenandabandoned_launch_never_signals_a_pid_that_no_longer_matches_the_spawn_tokenred; settingstoppedin the deferred cleanup only for a live status turnsa_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it(itsfailedcase) anda_completed_pending_stop_reads_stopped_whatever_live_status_it_hadred; removing the container-engine guard from the deferred cleanup turnsa_pending_stop_of_a_container_engine_is_not_completed_by_a_gone_supervisorred; dropping the stop-requested condition of--forceturnsa_ready_record_held_by_an_unqueryable_process_is_cleared_by_stop_then_forceandforce_is_refused_for_a_running_process_and_without_a_requested_stopred; removing the container check from the remove/prune guard, or from the refresh's status reconciliation, turnsa_pending_container_stop_survives_refresh_remove_and_prunered; removing the no-process condition from the deferred cleanup turnsa_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stoppedred; accepting an empty stopping snapshot's equal empty record without the marker turns theis_covered_by_stop_oftable andstop_does_not_confirm_over_an_empty_relaunch_of_an_empty_recordred; and dropping the printed "rocm does not keep this warning" sentence turns the unconfirmed-tree-kill report test red; leaving no stamp when the refresh consumes a marker off an empty record turnsa_listing_clearing_the_marker_of_an_empty_stop_does_not_make_it_unconfirmed(itsstartingandfailedcases) anda_stale_stop_marker_on_a_record_naming_no_process_is_cleaned_without_reading_it_stoppedred whilestop_does_not_confirm_over_an_empty_relaunch_of_an_empty_recordstays green, keeping the stamp across a restart turnsa_relaunch_after_a_listing_consumed_the_marker_is_still_not_the_stopandrestarting_forgets_a_settled_stopred, and matching an absent stamp against an absent marker turns theis_covered_by_stop_oftable red; dropping the settled-stamp clause fromis_covered_by_stop_ofturnsa_stop_of_an_empty_record_whose_marker_a_listing_consumed_is_still_confirmedinrocmdred for both statuses (its listing now runs in the engine stop callback, with the interleaving asserted); dropping the generation check fromis_covered_by_stop_of, or having the restart not stamp, turnsan_older_stop_finalizing_before_a_restart_publishes_does_not_delete_its_keyred; removing the failed-hold refusal turnsa_tree_kill_is_not_run_for_a_pid_that_could_not_be_held_even_when_its_identity_matchesred, and applying it also where there is no tree-kill step turnsan_unheld_pid_that_is_gone_recycled_or_unverifiable_keeps_its_refusalandwithout_a_tree_kill_step_an_unheld_pid_is_still_terminatedred; reading a process that cannot be queried as gone in the deferred key cleanup turnsa_pending_stop_keeps_its_key_and_marker_while_a_process_cannot_be_queriedandan_unverified_stop_keeps_its_key_and_marker_through_a_later_liveness_refreshred, the same in the status reconciliation turns the first red, and in the removal guard turnsa_record_is_in_use_unless_every_recorded_process_is_confirmed_gonered; not killing and reaping a stalledtaskkillat its deadline turnsa_helper_that_stalls_is_killed_and_reaped_at_its_deadlinered; mapping a root seen to exit under a failed tree kill back totimed_outturns the three tree-kill outcome tests red, and dropping the printed line turnsa_stop_with_an_unconfirmed_tree_kill_says_what_happened_and_keeps_the_servicered; ignoring the system directory insystem_tool_pathturnsa_system_tool_is_resolved_under_the_system_directoryred; reading an unqueryable process as not running in the shared helper turnsrecorded_pids_may_be_running_unless_every_recorded_process_is_confirmed_goneand the earlier tests of the three decisions red; an unverifiable first read followed by a verifiable one turning into a root-only stop (reverting the early refusal) turnsan_unverifiable_first_read_is_not_overtaken_by_a_verifiable_second_readand four neighbours red; acceptingidentity_mismatchas corroboration turnsan_identity_mismatch_sibling_does_not_corroborate_a_root_that_exited_alonered; readingSTILL_ACTIVEas gone turnsan_exit_code_query_is_running_gone_or_unqueryablered; dropping--forcefrom the missing---yeshint turnsa_forced_removal_missing_yes_names_the_command_that_completes_itred, and prune not naming the forced removal turnsprune_names_the_forced_removal_for_a_record_held_by_an_unqueryable_processandservice-unqueryable-01red; dropping the launch-cleanup refusal turnslaunch_cleanup_refuses_a_child_without_a_token_only_where_tokens_existred; confirming a root that exited alone without a corroborating process turns the lone-root and engine-not-gone tests red inrocm-core,rocmandrocmd; dropping the stop-requested condition of--forceturnsforce_is_refused_for_a_running_process_and_without_a_requested_stopred, dropping the running-process check turnsforce_is_refused_when_one_process_is_unqueryable_and_another_is_runningred, and making--forcedo nothing turnsservice-unqueryable-01red.Not run: nothing on a Windows host, so the Windows code paths themselves (the creation-time read,
taskkillresolved underSystem32, the Windows launch-cleanup refusal, whethertaskkillreally exits 128 for a missing process on the Windows builds in use, the process handle that holds the PID, theOpenProcesserror codes, the realtaskkilland its failures and deadline,service-stop-07and the Windows half ofservice-stop-08, and the Windows branch of the process-check advice) have not been executed on a Windows host. Thewindows-debugjob's unit tests do run the happy paths of the PID hold, the creation-time read and the System32taskkillagainst a realpingprocess (the two#[cfg(windows)]tests named above), without a result for this head being claimed here; no CI job runs the failure side (OpenProcessaccess denied,taskkillexit 128, a failing or stalledtaskkill), the tree kill of a parent with a child, or the launch-cleanup of a real child. What was moved out of the FFI so that it is unit-tested on Linux and, as plaincargo test, on the Windows unit-test job (not yet seen to pass there): theOpenProcessfailure-code mapping, the exit-code query mapping, thetaskkillresult and arguments, the System32 path building, the FILETIME join, and the deadline wait (a long-sleeping child on Unix,pingon Windows). The ordering of the PID hold, the identity read, the tree kill and the termination, the identity verdicts, the liveness gate, the liveness refresh and removal guard are tested through injected functions that also compile for the Windows target.cargo test --workspace --all-targets,scripts/smoke_local.pyand the coverage floors are left to CI.