Repository navigation
Conversation
On Windows, spawn_managed_engine_child launched the detached __engine-serve-http child with spawn_detached_no_inherit, which leaves its stdout/stderr unattached to anything. Linux attaches them to the service log via attach_background_stdio, so a crash's reason lands there; on Windows it reached nothing. A dying engine therefore left the log silent past its last explicit progress line, and both the managed and --verbose/--foreground serve paths only ever saw the child's pid disappear, which they treated as a benign stop: "server process exited" (or an unreadable "starting" status), exit 0, and the engine's own state file stuck at whatever it last wrote. Fix the Windows spawn (both the launch and the restart path) to attach the service log via the existing spawn_hidden_console_with_log primitive, mirroring Unix. As a platform-generic backstop — covering a gap in the attach, and any engine code path that returns an error without updating its own state file — run_builtin_engine_serve_http now logs the full error chain to the service log and this process's own structured log, and marks the engine state file `failed`, on any error exit. Neither platform's rocm serve actually inspected the child's fate after spawning it, though: a readiness wait that times out is indistinguishable from one that will never succeed because the child already died, and the attached log-streaming loop treated every self-exit as a graceful stop. Add finalize_managed_launch_readiness (background path) and an equivalent check in run_attached_service's ServerExited arm: both now confirm the child is still alive before accepting a non-Serving readiness result, and fail the launch with a log-tail-carrying error (and a failed engine state file) when it is not. A readiness timeout on a still-alive child keeps reporting its previous status with exit 0 unchanged, since a large model can legitimately take longer than the readiness budget. pid_is_running is injected into finalize_managed_launch_readiness so the dead-child branch is unit-testable without spawning and killing a real process, the same seam autostart_spawn_in_flight already uses nearby for an identical reason. New tests cover both liveness outcomes plus the untouched Serving-trusts-itself case, and assert the failure message and the resulting process exit code together, reading back the on-disk manifest and engine state file to confirm both land on `failed`. Not covered by a new e2e scenario: exercising this live (rather than via the injected-liveness unit tests) needs a real __engine-serve-http spawn that survives past both platforms' early-startup liveness checks before dying, without GPU hardware. The one close precedent in this codebase (a scripted engine death at startup) needed its own preflight- and engine-preparation- bypass plumbing built up over several review rounds; replicating an equivalent for a later-than-startup death is a separate, larger effort left to a follow-up rather than folded into this fix. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…(ROCMAI-541) Round-2 review fixes on top of 5b4fe55: B1 — the Unix liveness check was inert: the managed engine is a direct child (setsid only, no double-fork), and process_is_running's kill(pid, 0) probe succeeds for a zombie this process has not reaped yet. A dead-but-unreaped child was therefore still reported "running" on Linux, reproducing the original symptom despite the round-1 fix. Added rocm_core::ManagedChildLiveness, which retains the spawned std::process::Child on Unix and calls try_wait() (reaping it) instead of signal-probing; Windows keeps using process_is_running, which already reads the real exit code via OpenProcess/GetExitCodeProcess and needs no parent-side reaping. Threaded through finalize_managed_launch_readiness (launch path), run_attached_service and stream_attached_logs(_no_tty) (attached path), and the restart path's early-death check and (new) record_restart_readiness liveness gate. B2 — AttachOutcome::Stop only ever fires from this attached session's own keystroke reader; a stop triggered by `rocm services stop` from another shell, `rocm dash`, or rocmd's restart-recovery kills the watched pid directly, landing in ServerExited instead — which the round-1 fix treated unconditionally as a crash. This made every piped/CI `serve --verbose` (no keystroke capture at all) fail on an ordinary stop. ServerExited now checks the manifest's stop_requested_unix_ms marker (set before any such stop signals anything) and status=="stopped" before concluding a crash; when either is present it keeps the old benign-stop behavior (exit 0, no `failed` stamp). Fixing this also surfaced a related gap: the crash branch only marked the engine state file `failed`, never the manifest itself: now does both. B3 — added e2e coverage per AGENTS.md's Gherkin requirement for exit code / output changes: serve-25/26 in model_serving.feature exercise a real managed child dying after startup, and a real external stop in the same window, via a new scripted-engine test hook (ROCM_E2E_SCRIPTED_ENGINE[_DEATH_MS], e2e-test-hooks-gated) that stands in for the real engine dispatch and needs no GPU. Confirmed serve-25 hangs (the exact ROCMAI-541 symptom) when the B1 fix is reverted. N1 — removed the vacuous `exit_code_for(...) == ExitCode::FAILURE` assertion from finalize_managed_launch_readiness's dead-child test: it could not fail regardless of correctness (every non-FixExitCode/ ClapExitCode error maps to FAILURE there), and the test never goes through the real run()/exit_code_for call chain it implied. N2 — extended the same liveness fix to the restart path: record_restart_readiness now takes a liveness check and fails closed (marking both manifest and engine state `failed`) instead of recording `starting` forever when a restarted engine dies during its own readiness wait; the Unix early-death branch now also marks the engine state file, matching the launch path. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…(ROCMAI-541) Round-3 review fixes on top of 0df6166: B1 — managed_service_stop_was_requested was only ever consulted on the attached (--verbose/--foreground) ServerExited path. The background launch path's own readiness-wait failure branch (finalize_managed_launch_readiness) and the restart path's (record_restart_readiness) both still treated a dead-and-not-Serving child as a crash unconditionally, so a `rocm services stop`/`rocm dash` stop landing during either's 45s readiness wait still got stamped `failed` on both files and a non-zero exit, exactly like a crash. Both now check the stop marker first and return cleanly (exit 0, no `failed` stamp) when it is set, mirroring the attach path. finalize_managed_launch_readiness now also refreshes its record from disk in that case and reports to its caller (via a new `bool` return) whether to skip its own follow-up bookkeeping, so that caller's subsequent write can no longer clobber a stop marker written since the record was captured at spawn time. Fixed the same class of bug at its root in the crash branch too: it was writing `record.write()` (the whole, stale, spawn-time snapshot) instead of merging just the `failed` field, which could just as easily have erased a marker concurrently written by the stop itself. B2 — the Unix branch of spawn_managed_engine_child (an engine that dies before the readiness wait even starts) never touched the manifest or the engine state file on its bail-out, leaving both at their just-created `starting`/pid-0 defaults forever — this is ROCMAI-541's original, literal symptom, on the one code path round 1 and round 2 had not actually reached despite both rounds' commit messages implying otherwise (they fixed the restart path's equivalent branch). Now marks both `failed` via merge, same as every other failure branch in this file. N1 — corrected two comments that mischaracterized rocmd's restart-recovery watcher as killing the engine it supervises directly; per apps/rocmd/src/watchers.rs, it only ever fires once the service is already dead/unreachable on its own and reacts by clearing stop_requested_unix_ms and re-exec'ing a fresh supervisor, never signalling the old engine. N2 — guarded run_attached_service's ServerExited `failed` manifest merge on the manifest still naming the exact engine pid (+ start-time token) this session spawned and watched, so a reaped child's stale write can no longer clobber a manifest a successful restart has since legitimately replaced. N3 — added a unit test for ManagedChildLiveness's Unix arm: spawns a real short-lived child, polls until is_running() reads false, and asserts a second call still reads false (pins the no-double-reap contract the type exists for). N4 — dropped finalize_managed_launch_readiness's vestigial child_pid parameter; its only caller already ignored it in favor of the liveness closure. N5 — serve-25 now captures its service_id from the CLI's own printed output at launch time instead of re-deriving "the newest record" in its Then step; serve-26's attach-wait loop now calls assert_scripted_hook_was_reached on timeout, so a binary built without the e2e-test-hooks feature fails with a clear diagnostic instead of a generic 30s timeout panic. Left as-is per this round's explicit scope: the Windows ManagedChildLiveness PID-reuse race (a known fork against unmerged PR #437) and the serve-25/26 scenario id collision with that PR's own serve-25 — both cross-PR sequencing calls for a human, not this round. Verified on the MI300X box: fmt --check, build and clippy -D warnings (both with and without --features rocm/e2e-test-hooks), full workspace test suite, and cargo xtask e2e -n 'serve-2[2-6]' (serve-22/25/26 pass; serve-23/24 fail on pre-existing environment gaps unrelated to this change — missing libdrm_amdgpu.so.1 / a vLLM import error in this sandbox). Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…, not engine identity (ROCMAI-541) Round-4 review fix on top of 7ad214c: Blocking — round 3's N2 guard compared the manifest's engine_pid/ engine_start_ticks against run_attached_service's watched child_pid before stamping failed on a confirmed crash. refresh_from_engine_state overwrites engine_pid with whatever the engine itself reports as its server_pid, which for an engine like Lemonade that spawns its own llama-server child is *that* child's pid, not the __engine-serve-http process this loop actually watches — so for the single most important case this whole ticket is about (a Lemonade-backed server that was running and then died), the comparison was false and the manifest never got stamped failed, silently undoing the earlier fix for that case. It also didn't stop the stale-write problem it was built for: the engine-state-file write right below it was unconditionally unguarded, and refresh_from_engine_state would propagate that status straight back into a newer manifest anyway. serve-25 didn't catch this because the scripted test engine never writes a server_pid, so engine_pid == child_pid happened to hold there by coincidence, not by exercising the real-engine case. Fixed by guarding on supervisor_pid/supervisor_start_ticks instead — recorded for this exact child at spawn and never touched by refresh_from_engine_state (confirmed by reading it end to end: it only ever adopts server_pid/pid into engine_pid/engine_start_ticks), but rewritten by every legitimate restart path via record_supervisor_identity. Applied the same guard to BOTH the manifest write and the engine-state-file write, where previously only the manifest had one. N1 — spawn_managed_engine_child's and start_managed_service's "normal" (non-dead-child) record.write() calls are now merges of only the fields each site actually owns (status/supervisor+engine identity at spawn; status/inference_verified_at_unix_ms at readiness) instead of a whole-file write of the in-memory spawn-time snapshot, closing the gap round 3's commit message overstated as "fixed at its root": that round only fixed the dead-child branches, not these. N2 — corrected a doc comment citing the old serve-scripted-death/ serve-scripted-stop scenario ids; the real ones are serve-scripted-death-after-startup and serve-scripted-external-stop-while-attached. N3 — serve-26's two remaining steps (another_shell_stops_scripted_service, assert_scripted_stop_not_marked_failed) now read off the service_id captured the moment user_attaches_scripted_serve_in_background first observes the record running, instead of re-scanning the services dir for "the newest record" afterward. Removed the now-unused newest-record scan helper. N4 — checked: server-recover defaults to disabled per scenario (fresh isolated config.json each run, and WatcherUserConfig::enabled is only true when a scenario explicitly opts in via `rocm automations enable`), so serve-25's manifest-failed assertion isn't exposed to that race in this harness. No change needed. Verified on the MI300X box: fmt --check, build and clippy -D warnings (both with and without --features rocm/e2e-test-hooks), full workspace test suite (one unrelated pre-existing flake in cli_progress::tests::animated_spinner_keeps_ticking_without_progress_calls under parallel load, passes in isolation), and cargo xtask e2e -n 'serve-2[2-6]' (serve-22/25/26 pass; serve-23/24 fail on the same pre-existing environment gap as before — missing libdrm_amdgpu.so.1 — unrelated to this change). Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The branch predates the main.rs modularization, so its changes had to be relocated rather than merged in place: - `restart_internal_managed_service_with`, `record_restarted_child` and `record_restart_readiness` (plus their tests) moved to `services_text.rs`; the Windows log attach, the `ManagedChildLiveness` threading and the restart-path liveness gate are reapplied there. - `run_builtin_engine_serve_http` moved to `engine_ipc.rs`; the failure capture (`record_engine_serve_http_failure`) and the scripted-engine E2E hook (`run_scripted_e2e_engine_serve_http`) are reapplied there, and the new `aim` arm added since the branch is covered too. - `serve_cmd.rs`'s no-GPU refusal was rewritten by #703, so the `scripted_engine_death` bypass is expressed against `reuse_existing` / `host_has_no_usable_gpu` / `no_usable_gpu_refusal` instead of the old inline bail. - the branch's serve-25/26 scenario numbers now collide with scenarios that landed meanwhile, so they are renumbered serve-33/34; their `@id:` tags were already unique and are unchanged. Not yet verified — the workspace does not build on Darwin, so this merge still needs a build/clippy/test run on the Linux dev box. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ayout Fallout from moving the fix off main.rs, caught by the first build on the Linux box: - `read_managed_service_manifest` now lives in `services_text.rs` and was private, so `service_launch.rs`'s five `crate::`-rooted calls could not resolve. Re-pointed at `crate::services_text::` and made it `pub(crate)`. - dropped a duplicate `crate::apply_app_path_env` import left over from the pre-refactor path; `engine_runtime_select::apply_app_path_env` is the one that exists. - gated `engine_ipc.rs`'s `Duration` import on `e2e-test-hooks`, since only the scripted-engine hook uses it. Plus the rustfmt reflow the longer module paths force. Verified on the MI300X box: build (with and without rocm/e2e-test-hooks), clippy --workspace --all-targets and clippy -p e2e-cucumber --test e2e both -D warnings, fmt --check, and the full workspace suite (0 failures). The nine ROCMAI-541 unit tests run and pass. serve-33/34 pass under HIP_VISIBLE_DEVICES="" (which puts the harness on its no-GPU mock premise), and serve-33 fails when the ServerExited crash branch is reverted to the pre-fix `Ok(())`, so it discriminates rather than passing vacuously. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Review findings on the merged ROCMAI-541 fix. The Windows spawn reused `spawn_hidden_console_with_log`, which swaps `DETACHED_PROCESS | CREATE_NO_WINDOW` for `CREATE_NEW_CONSOLE`. That buys nothing here and costs a `conhost` per engine: a managed engine is a background service meant to outlive the CLI, and the console was incidental to the only thing wanted from that helper, the log attach. `STARTF_USESTDHANDLES` is set independently of the creation flags, so the log can be attached while staying detached. Adds `spawn_detached_with_log`: `spawn_detached_no_inherit`'s flags plus the log. The handle duplication both variants need moves into a shared `spawn_windows_with_log`, so `spawn_hidden_console_with_log` keeps its exact previous behaviour — it still has a caller, the lemond daemon in `engines/lemonade/src/process.rs`, which does want a console. Blanket `bInheritHandles` is unchanged and still inherent to passing std handles this way; the new function's doc comment says so, and scoping it would need `STARTUPINFOEXW` with an explicit handle list. Also: document serve-33/34 in docs/testing.md, which AGENTS.md requires for an observable-behaviour change, including how to run them on a GPU host; and correct a comment still placing `run_scripted_e2e_engine_serve_http` in main.rs rather than engine_ipc.rs. The Windows paths here are `cfg(windows)` and cannot be compiled on the Linux box (the MSVC target's C dependencies need `lib.exe`), so they are checked by CI's windows-build-and-test job, not locally. Linux re-verified: fmt, clippy -D warnings, full suite, and serve-33/34 still pass. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Thanks for tackling this. I read the full diff against ROCMAI-541 and have four things I'd want addressed before merge; details are inline or below.
Blocking
- CI hasn't run on this head commit (only the docs check shows up) and the PR conflicts with main. The Windows spawn/liveness paths, the changed
spawn_hidden_console_with_logpath used by the lemond daemon, and the removal of#[cfg(not(windows))]on theread_optional_tail_linesimport are all unverified. Needs a rebase and green required checks. - serve-33 pins the wrapper text ("exited unexpectedly") and the non-zero exit, but not the engine's own failure reason. That string is emitted even when the log tail is empty, so the scenario would stay green if the reason never reached the log or the CLI output, which is the core ask of the ticket. Please assert the scripted death message in the output and the service log too.
- The Windows log attach (the headline fix) has no test that runs on Windows; the new liveness test is Unix-only and serve-33 would not catch child output still going nowhere. A pure helper or injectable seam that the Windows job can exercise would do it.
- Docs aren't synced:
rocm serve --verbosenow exits non-zero with a new message where it used to print "server process exited" and exit 0. README, the--helptext and docs/manual-testing.md still describe the old behavior.
Non-blocking / questions
- On the non-verbose path a crash is only reported after the full 45s readiness wait, and that path only has a unit test that doesn't assert the exit code.
engine_ipc.rs: the wrapper stampsfailedon anyErrfrom the engine entry. If an engine returnsErrwhen stopped viarocm services stop, an intentional stop would readfailed; I couldn't tell from the diff whether the stop marker is consulted on this path.managed_service_stop_was_requestedtreatsstatus == "stopped"as a requested stop. Please confirm nothing other than an operator stop (liveness refresh, rocmd reconciliation) can writestoppedbefore that check, or a crash would be reported as a clean exit.- The error chain can end up in the service log more than once (explicit append, tracing, and the CLI's own
Error:print now that stdio is attached). - The "mark failed" merge pair is repeated in five places and the stop-vs-crash decision in three; a helper each would help.
- Mutation evidence in the PR text covers serve-33 only; serve-34, the restart-path tests and the stop-marker tests should be shown to fail with their target broken too.
- At about 2100 lines, with stop-marker, restart-path, zombie-reaping and merge-vs-write conversions beyond the ticket, this would be easier to review split up.
- docs/testing.md cites
engine_ipc.rsandserve_cmd.rsas bare backtick paths; AGENTS asks for relative links.
| assert_scripted_hook_was_reached(&output); | ||
| assert_ne!(rc, 0, "expected a non-zero exit, got 0:\n{output}"); | ||
| assert!( | ||
| output.contains("exited unexpectedly"), |
There was a problem hiding this comment.
This asserts only "exited unexpectedly", which managed_engine_exit_failure_detail emits even when the log tail is empty. Please also assert that the scripted death message ("scripted e2e engine death after startup") appears in the output and the service log; otherwise the ticket's core ask (reason reaches log and CLI) isn't covered and the Windows attach could regress unnoticed.
| /// ROCMAI-541 acceptance criterion 2: a managed engine that died before | ||
| /// ever answering HTTP must fail `rocm serve` with both a non-zero exit | ||
| /// code and a message naming it, not just the unreadable "starting" | ||
| /// status the readiness timeout alone used to leave behind. `pid_is_running` |
There was a problem hiding this comment.
Stale doc: it says pid_is_running is injected as |_| false, but the function takes child_is_alive: impl FnMut() -> bool and the test passes || false.
| ); | ||
| if proceed_with_launch_bookkeeping { | ||
| if readiness == EndpointReadiness::Serving { | ||
| // Latch the verification the wait just performed, so the readiness |
There was a problem hiding this comment.
The liveness check only runs after the 45s readiness wait returns, so a crashed engine makes non-verbose rocm serve hang the full 45s before failing, and this path has no scenario (the unit test doesn't assert the exit code). Consider feeding liveness into the wait loop, or covering it with a scenario.
| /// [`spawn_hidden_console_with_log`] already uses; scoping it would need | ||
| /// `STARTUPINFOEXW` with an explicit handle list. | ||
| #[cfg(windows)] | ||
| pub fn spawn_detached_with_log( |
There was a problem hiding this comment.
Passing std handles makes bInheritHandles true for every managed engine (previously none were inherited). A background rocm serve whose stdout/stderr is a captured pipe may now keep that pipe open for the engine's lifetime. Please verify on Windows with piped output; scoping the inheritance would need a STARTUPINFOEX handle list.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 46b7fe2
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (merge-base c39ee0c6 through head 46b7fe23, all 7 commits). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees server process exited", status In Progress). The description's account of the diff is accurate and unusually self-aware — it already names the main-branch scenario-number collision and the relationship to sibling PR #437 — so there is no description/scope mismatch to report.
Blocking
-
The PR has a real merge conflict with current
main, from an independent scenario-number collision —tests/e2e-cucumber/features/model_serving.feature:538,554
This PR addsScenario: serve-33/serve-34.mainhas independently addedserve-33throughserve-39for unrelated work (a native-Windows platform gate and gated-AIM scenarios) using the same numbers. Confirmed by actually dry-running the merge (git merge --no-commit --no-ff origin/mainfrom the PR head) —CONFLICT (content): Merge conflict in tests/e2e-cucumber/features/model_serving.feature— and by GitHub's ownmergeable_state: "dirty"on the live PR.
Confidence 100 · architectural · Fix: rebase onto currentmainand renumber this PR's scenarios to the next free IDs. -
No required CI check has executed for this commit. Only a Read-the-Docs build shows up (
get_check_runsreturns 0 check runs;get_statusshows a singlesuccessstatus, the docs build).build-and-test,clippy,prek, and the requiredwindows-build-and-testlane have not run at all, so none of the Windows spawn/liveness changes, thespawn_hidden_console_with_logrefactor, or the#[cfg(not(windows))]removal onread_optional_tail_lines's import have been verified by CI at this commit.
Confidence 100 · architectural · Fix: resolve the conflict above, push, and let the required checks run green before this is re-reviewable with confidence. -
README.md and docs/manual-testing.md were not updated for the new
rocm serve --verbose/--foregroundcrash behavior, violating the project's own same-change doc-sync rule (AGENTS.md §5). The change makes a crashing managed engine exit non-zero with a new message and afailedmanifest/engine-state stamp, where it previously printedserver process exitedand exited 0. Grepping both files for either string returns nothing, and the one passage in README.md that already documents the--verbose/--foregroundattach behavior (README.md:891-893, Ctrl-D/Ctrl-C) says nothing about what happens on a crash. Onlydocs/testing.mdwas touched.
Confidence 85 · mechanical · Fix: add a sentence to README.md and docs/manual-testing.md describing the new exit code/message/failed-stamp behavior on an attached crash. -
The headline Windows fix (log attachment) has no test coverage reachable from any required CI check — neither unit nor e2e.
crates/rocm-core/src/process.rs's newspawn_detached_with_log(line 101),spawn_windows_with_log(line 138) and theManagedChildLiveness::Windowsarm (line 461) have no#[cfg(windows)]test exercising them — the file's existing Windows tests (WindowsFixture-based, further down) are all pre-existing and unrelated; one of this PR's own doc comments admits "this repo's test environment has no Windows runner to exercise it on." Separately, the new e2e scenarios (model_serving.feature:538,554) depend on theROCM_E2E_SCRIPTED_ENGINE/e2e-test-hooksmechanism and are tagged@requires-no-gpuonly (no Windows tag or inclusion list). The one required Windows job that runs e2e at all (.github/workflows/ci.yml:932-954, insidewindows-release) is explicitlyE2E_ONLY_LIFECYCLE=1withE2E_ALSO_RUN_IDSnaming three unrelated comfyui scenarios, and is "Deliberately NOT built withrocm/e2e-test-hooks" per its own comment — so even if these two scenarios were added to that list, the scripted-engine hook they depend on would not compile in. This also means the PR description's claim that "CI'swindows-build-and-testjob is the real check on" the Windows paths overstates what that job actually exercises: it will confirm the code compiles and existing tests still pass, not that the new log-attach logic works.
Confidence 85 · logic · Fix: add an injectable/pure-helper Windows unit test forspawn_detached_with_log/ManagedChildLiveness::Windows, and either get serve-33/34 running on a required Windows lane withe2e-test-hooks, or add a dedicated Windows-only scenario/unit test. -
managed_service_stop_was_requested's reliance onrecord.status == "stopped"is not sound proof of an intentional stop, and this PR is the first to build a safety contract on it.apps/rocm/src/service_launch.rs:321-325treatsstop_requested_unix_ms.is_some() || record.status == "stopped"as "this exit was not a crash." Butapps/rocm/src/managed_service_status.rs:274-277(pre-existing, unrelated to this PR) writesstatus = "stopped"the instant a tracked pid is found dead during an ordinary status read (rocm services list/status,rocm dash, a daemon poll) — its own comment says so explicitly: "dead pids alone do not distinguish a stopped service from a crashed one." Nostop_requested_unix_msis set on that path. If a concurrent status read lands during this PR's new 45s launch/restart readiness windows (or the long-lived attach loop) right as a genuine crash happens, that reconciliation can flipstatusto"stopped"first, and this PR's new guard would then read the exit as a clean stop — silently swallowing the crash this whole PR exists to surface. The new tests only cover the case wherestop_requested_unix_msis set together withstatus: "stopped", neverstatus == "stopped"alone.
Confidence 85 · logic · Fix: only truststop_requested_unix_msas proof of an intentional stop; have the liveness-reconciliation path use a distinct status (e.g."crashed"/"unreachable") for a bare dead-pid discovery instead of reusing"stopped". -
serve-33's "engine state... reads failed" assertion doesn't test the code path it's meant to cover.
tests/e2e-cucumber/tests/e2e/serving_steps.rs:2344-2355asserts bothrecord["status"] == "failed"(the manifest) andengine_state["status"] == "failed". The engine-state file, however, is already stamped"failed"unconditionally byengine_ipc.rs'srecord_engine_serve_http_failure, which runs inside the detached child itself the instant the scripted engine errors — independent ofservice_launch.rs's ownServerExitedhandling and itsstill_watching_this_supervisorguard. Breaking or deleting the CLI-side (service_launch.rs) engine-state write would leave this assertion green; only the manifest-status half actually depends on the new parent-side logic.
Confidence 70 · logic · Fix: assert something that only the parent-side guarded write can produce (e.g. a supervisor-identity-mismatch case), or move the engine-state check to a dedicated unit test ofrun_attached_service's guard.
Non-blocking
None scoring above the reporting threshold — see Decisions for the author for lower-confidence/judgment items.
Decisions for the author
ManagedChildLiveness::Windowsreopens the child by bare PID on every liveness check (crates/rocm-core/src/process.rs:461-479), across windows up to 45s (launch/restart readiness) or the whole attached session — a PID-reuse race. This isn't a new hole (Windows'sprocess_is_runninghas always had this property, and PR #499's own history already accepts the equivalent gap for the stop path), but this PR is the first to key the crash/starting decision on it in three places. This PR's own commit7ad214c2explicitly flags it as deferred: "Left as-is per this round's explicit scope: the Windows ManagedChildLiveness PID-reuse race (a known fork against unmerged PR #437)... cross-PR sequencing calls for a human, not this round." PR #437 uses a race-free held-handle design for the same problem. Tradeoff: merge now with this narrow, already-accepted-elsewhere residual gap and track reconciliation with #437 separately, vs. hold this PR until it adopts #437's approach. Since this reasoning lives only in a commit message and not in the PR description, it's worth carrying forward into the description either way so a future reader doesn't have to dig for it.- Scope/size — at ~1624 insertions across 10 files, bundling the narrow Windows-log-attach fix with a broader stop-vs-crash redesign touching three call sites (launch, restart, attach) makes this harder to review and rebase (as the conflict above shows) than necessary. Consider whether the log-attach fix and the liveness/stop-vs-crash logic could land as two PRs. (non-blocking-improvement)
- Duplicated "mark failed" merge pattern across roughly six call sites (
engine_ipc.rs,services_text.rs×2,service_launch.rs×3) with no shared helper is itself not a defect, but it's the kind of duplication that produced the asymmetry in the blocking status-check finding above — factoring it into one helper that always checks the stop marker first would make that class of bug harder to reintroduce. (non-blocking-improvement)
Positive signals
- Doc comments throughout are unusually rigorous — nearly every non-obvious decision names the exact race or prior bug it guards against (e.g. why
merge_json_state_fileis used instead ofrecord.write(), whysupervisor_pid/supervisor_start_ticksrather thanengine_pidis the right identity to gate on). - Mutation-style self-verification was actually performed and documented, not just asserted: the PR text states serve-33 was confirmed to fail when the crash branch is reverted to the pre-fix
Ok(()), matching AGENTS.md's own "shown to fail when the code it covers is broken" rule. - Commit history is clean and well-labeled (round-by-round
B1/B2/B3/N1-N5commits, each individually signed off), not a trail of "wip"/"fix" noise. - The
spawn_hidden_console_with_log→ sharedspawn_windows_with_logrefactor is a careful, verified-safe extraction: the Lemonade daemon's existing caller is confirmed unchanged in behavior.
Deployment notes
rocm serve --verbose/--foreground changes observable behavior on a managed-engine crash: it now exits non-zero with a new message and stamps the manifest/engine-state failed, where it previously exited 0 and printed server process exited. Any external automation or script keyed on the old string or the old exit code will break — this should be called out explicitly once the doc-sync finding above is addressed.
What this covered
Read completely: apps/rocm/src/engine_ipc.rs, apps/rocm/src/serve_cmd.rs (diff + surrounding context), apps/rocm/src/service_launch.rs, apps/rocm/src/services_text.rs, crates/rocm-core/src/lib.rs, crates/rocm-core/src/process.rs, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs — via four independent background review passes (engine_ipc/services_text/serve_cmd; e2e tests/docs; AGENTS.md agent-instruction adherence; git history/blame) plus my own direct, complete read of service_launch.rs/process.rs/lib.rs, README.md/docs/manual-testing.md, and .github/workflows/ci.yml's Windows e2e configuration. Also ran a local dry-run merge against origin/main to confirm the conflict, and checked the live PR's CI/check status directly via the GitHub API. The root AGENTS.md was loaded and applied as the agent-instruction file.
Fan-out ran at reduced parallelism for one group: the shared environment's subagent-concurrency ceiling was saturated by other sessions for part of this review, so the "core launch/liveness" file group (service_launch.rs/process.rs/lib.rs) was reviewed directly by this driving context rather than by an independent worker — all other passes ran as independent background workers as intended.
Disclosure on the mandatory pre-review claim-check: resolving the live PR state required fetching existing reviews and the PR body, which necessarily exposed this context to the full text of an existing human review (by a different reviewer, state CHANGES_REQUESTED, same commit) and the PR's own description before all findings above were settled. No finding above was derived from that review's content — each was independently re-derived and verified from the code, CI configuration, and git history as cited inline. Several findings overlap with that existing review's points (CI/merge-conflict status, the Windows test gap, doc sync, the stop-vs-crash status ambiguity, double-logging, the repeated merge pattern, PR size) — that is convergent independent discovery, not copying. The PR description was read in full as part of this same required call; per method it was used only at the end, to state the ticket relationship and check for a scope mismatch, not to shape findings. Not reconciled against that review's thread or any other discussion.
Did not run: cargo build/test/clippy myself (reasoned from code, the PR's own stated verification, and CI configuration rather than executing the suite in this sandbox).
🤖 Generated with Claude Code
main moved 9 commits since the first merge, and the resulting conflict meant GitHub could not build refs/pull/854/merge — so no pull_request workflow ran at all, which is why only the docs check reported. Merging unblocks CI. ROCMAI-823 took serve-33/34 meanwhile, so these are renumbered serve-40/41 once more; their @id: tags are unchanged and remain the stable reference. Review fixes: - serve-40 asserted only the wrapper text ("exited unexpectedly") and a non-zero exit. `managed_engine_exit_failure_detail` emits that wrapper even when the log tail is EMPTY, so the scenario would have stayed green in exactly the failure this ticket is about. It now pins the engine's own death message in both the service log and the CLI output. - the Windows log attach — the headline fix — had no test on the platform the bug was reported on: the liveness test is Unix-only and the scripted-engine scenarios run the Unix branch. Adds a cfg(windows) test spawning a child through `spawn_detached_with_log` and asserting BOTH streams reach the log, stderr included, since that is where an engine's error chain goes. - README's --verbose section said nothing about what a crash does; it now states the non-zero exit and the `failed` stamp, and that a deliberate stop still exits 0. - extracted `mark_service_failed` for the manifest+engine-state pair that was repeated at three sites, with the merge-not-write reasoning stated once. - docs/testing.md: relative links instead of bare backtick paths. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Thanks — this was a genuinely useful read, and point 2 in particular found a real hole. Pushed as Fixed2. serve-40 didn't pin the failure reason. You were right, and for the reason you gave: I checked that the new assertion actually discriminates rather than just being longer. Breaking only the error-message tail was not enough to fail it — on Linux 3. No Windows-runnable test for the log attach. Added one in 4. Documented the behavior change. README's Non-blocking, also done: extracted Mutation evidence, as asked. Each test fails when its own target is broken:
Two corrections1a. CI wasn't a separate problem from the conflict — it was caused by it. With the PR unmergeable GitHub can't build 1b. The pub(crate) fn managed_engine_exit_failure_detail(service_id: &str, log_path: &Path) -> String {
let recent_lines = read_optional_tail_lines(log_path, 80, "service log");Restoring the gate would fail the Windows build on an unresolved import. The gate was correct before only because the sole Windows-side caller didn't exist yet. (The still-gated 4a. The docs weren't stale. Open questions, answered
On splittingFair, and I'd push back only partly. The stop-vs-crash and zombie-reaping work isn't scope creep: the first version of this fix turned every ordinary piped |
…ture
The new `a_detached_child_writes_both_streams_into_the_log` failed on the
Windows lane, in the fixture rather than the code under test. Its temp dir
embedded `std::thread::current().id()`, whose Debug form is `ThreadId(2)`,
and `quote_windows_arg` only quotes an argument containing whitespace or a
quote — so the path reached cmd bare and `(` ended the token early:
'C:\...\rocm-core-detached-log-664-ThreadId' is not recognized as an
internal or external command
Dropped the thread id; the pid alone is unique, since only this test uses
the name.
The failure was still informative: cmd's complaint reached the log file via
stderr, which is exactly the attachment the test exists to prove. What it
caught was a malformed fixture command, not a broken attach.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · a849ce1
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (base 87a7c7a7 through head a849ce1, 9 commits; the 11-file, 1745/89-line diff matches the PR's own additions/deletions). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees server process exited", status In Progress). All three of the ticket's acceptance criteria (error chain reaches the service log on Windows, rocm serve reports failure with a non-zero exit, the engine state file is stamped failed on any error exit) are addressed by this change; the PR does not claim to close the ticket, and the extra stop-vs-crash/restart-path work is additive to the literal AC text.
Blocking
-
The PR currently conflicts with
main, from a fresh scenario-number collision.tests/e2e-cucumber/features/model_serving.feature: this PR'sScenario: serve-40/serve-41collide with an independently-addedserve-40frommain's current tip (commitdb7e5408, "ROCMAI-827: start the background helper only while an automation needs it", #838), which uses the same number for unrelated work. Confirmed by directly dry-runninggit merge --no-commit --no-ff origin/mainfrom the PR head —CONFLICT (content): Merge conflict in tests/e2e-cucumber/features/model_serving.feature— and by the live PR's ownmergeable_state: "dirty".
Confidence 100 · architectural · Fix: rebase onto currentmainand renumber this PR's two scenarios to the next free IDs. -
The "child died before its readiness wait even started" branches on both the launch and restart paths stamp
failedwithout first checking whether a stop was requested, contradicting this file's own documented contract.apps/rocm/src/service_launch.rs(spawn_managed_engine_child's immediate post-spawn liveness check) andapps/rocm/src/services_text.rs(restart_internal_managed_service_with's equivalent branch, ~line 1085) both callmark_service_failed/an unconditioned engine-state merge the instant a freshly-spawned child is found dead within its first ~200ms — neither consultsmanaged_service_stop_was_requestedfirst.mark_service_failed's own doc comment inservice_launch.rsstates the precondition explicitly: "Callers reach this only aftermanaged_service_stop_was_requestedhas already ruled out an intentional/external stop." Every sibling failure branch this PR adds (finalize_managed_launch_readiness,record_restart_readiness, the attachedServerExitedarm) does consult it first; these two do not, so an externalrocm services stoplanding in that narrow post-spawn window can be reported as a crash. This is corroborated directly from this PR's own git history: commit7ad214c2's message describes adding the stop-check explicitly tofinalize_managed_launch_readiness/record_restart_readiness(its "B1") as a separate, later fix from the one that touched this exact branch ("B2" — which only fixed the manifest/engine-state write mechanism, not the missing stop-check), so the gap was never actually closed for this branch in either round. On the restart side specifically, the bug compounds:update_restarting_record's CAS guard (names_same_processes+ status match) would detect a concurrent stop (the status would already differ) and returnOk(None), but thatOptionreturn value is discarded — so the engine-state file is still stampedfailedunconditionally and the restart command stillbail!s with a failure message regardless of what the CAS detected. No test exercises either branch against a concurrent stop (unlike the readiness-wait branches, which have dedicated tests for exactly that race).
Confidence 85 · logic · Fix: have both branches callmanaged_service_stop_was_requestedbefore marking failed, the same way their sibling branches do; have the restart branch checkupdate_restarting_record's return value before writing the engine-state file or bailing. -
ManagedChildLiveness::Windows— the liveness arm that is the actual crash-detection primitive on Windows — has no test reachable by any required CI check. Its own test module says so directly: the only test exercisingManagedChildLiveness(managed_child_liveness_unix_reports_dead_after_exit_and_does_not_double_reap) is#[cfg(not(windows))], and its doc comment states the Windows arm "is not exercised here." Separately verified against.github/workflows/ci.yml: the one required Windows e2e lane (windows-release's "Acceptance install lifecycle" step) runs withE2E_ONLY_LIFECYCLE=1and a fixedE2E_ALSO_RUN_IDSallowlist of unrelated scenarios, and its own comment states it is "Deliberately NOT built withrocm/e2e-test-hooks" — so the new scripted-engine scenarios this PR adds could not run there even if added to that list; their compiled-in test seam does not exist in that binary. By contrast, the sibling Windows fix in the same file —spawn_detached_with_log, the log-attachment half of this same ticket — does have a dedicated#[cfg(windows)] #[test](a_detached_child_writes_both_streams_into_the_log), which runs and passed in the "Windows build and test (debug)" CI job on this commit. So a Windows CI runner capable of exercisingcfg(windows)unit tests is demonstrably available and used for one half of the fix, but not the other. This is exactly the case AGENTS.md §3 names: "Platform-gated code gets a test that runs on its gate... If a path stays untested, say which and why in the PR text" — the PR body's "CI'swindows-build-and-testjob is the real check on" the Windows paths overstates what that job verifies for this specific arm (compiles and doesn't regress existing tests, not that the new liveness logic is correct).
Confidence 85 · architectural · Fix: add a#[cfg(windows)] #[test]forManagedChildLiveness::Windows(a real detached child, killed and polled for liveness, analogous to the existing Unix test), or at minimum say in the PR text that this specific arm stays unverified and why. -
docs/manual-testing.mdwas not updated for the newrocm serve --verbose/--foregroundcrash behavior, though its own Section 3 walks through exactly that scenario. The change makes an attached managed engine that dies on its own exit non-zero with a crash message and afailedmanifest/engine-state stamp, where it previously exited 0 and printed "server process exited." README.md was correctly updated (new paragraph after the existing Ctrl-D/Ctrl-C bullets) anddocs/testing.mddocuments the new e2e coverage, butdocs/manual-testing.md's "3. Lemonade GPU Verification" section — which already manually walks a tester through this exact--foregroundattached session and its Ctrl+D/Ctrl+C behavior — says nothing about what happens if the engine dies on its own, so a manual QA pass following this checklist would not catch a regression here. AGENTS.md §5 is explicit: "when a command's ... observable behavior changes, update README.md, its --help/doc comment, docs/testing.md, and docs/manual-testing.md in the same change — do not leave user-facing docs for a follow-up." (The--helptext for--verbose/--foregroundmakes no claim about exit behavior either way, so no change is needed there.)
Confidence 85 · mechanical · Fix: add a step to Section 3 (e.g., kill the Lemonade child while attached) naming the expected non-zero exit, crash message, andfailedstatus.
Non-blocking
- The new scripted-death e2e fixture uses a fixed 1.5s sleep with no explicit synchronization against the ~200ms early-startup liveness-check window it is meant to clear.
tests/e2e-cucumber/tests/e2e/serving_steps.rs's scripted-engine death timer is a bare wall-clock delay, not gated on any observable signal that the liveness check actually completed; on a slow/loaded CI runner this margin (currently ~7.5x) could in principle narrow enough to land the death inside the startup window instead of after it, producing an intermittent failure in a different code path than intended. No actual flakiness was observed or reproduced; this is a structural risk rather than a confirmed one.
Confidence 70 · logic · Fix: have the scripted engine wait on an explicit signal (a file/env marker the real liveness check writes) rather than a timer, or make the margin generous and document why it's believed sufficient.
Decisions for the author
ManagedChildLiveness::Windowsre-derives liveness from a bare PID on every check (a PID-reuse race) across windows up to 45s or a whole attached session — tradeoff. This is not a new hole (process_is_runninghas always had this property), and this PR's own commit7ad214c2documents it as a deliberate, already-known deferral: "Left as-is per this round's explicit scope: the WindowsManagedChildLivenessPID-reuse race (a known fork against unmerged PR #437)... cross-PR sequencing calls for a human, not this round." That reasoning currently lives only in a commit message; carrying it into the PR description would save a future reader from digging for it.- The "mark failed" merge pattern is duplicated across roughly six call sites with no shared helper — non-blocking-improvement. Not a defect by itself, but it's the kind of duplication that let the blocking stop-check asymmetry above exist; one helper that always checks the stop marker first would make that class of bug harder to reintroduce.
- PR scope/size — non-blocking-improvement. At ~1745 insertions bundling the narrow Windows-log-attach fix with a broader stop-vs-crash redesign across three call sites, this is harder to review and rebase (as the live conflict above shows) than the two pieces would be split.
Positive signals
- Doc comments throughout are unusually rigorous about why, not just what — e.g.
mark_service_failed's comment states its own precondition and the reason merges are used instead of whole-record writes (to avoid erasing a concurrently-written stop marker), which is exactly the contract the blocking finding above traces a violation of. record_engine_serve_http_failure's tracing call and the explicit service-log append do not actually double-log:apps/rocm/src/logging.rswires the process'stracingsubscriber to a dedicated, file-only, non-blocking appender (~/.rocm/logs/rocm-cli.log.*) that explicitly never touches stdout/stderr, so the two writes land in two different files rather than duplicating in the onerocm logstails.- Mutation-style self-verification was actually performed and documented: the PR text states the crash scenario was confirmed to fail when the fix is reverted to the pre-fix
Ok(()). - Commit history is clean, round-labeled (B1/B2/B3/N1-N5), and individually signed off rather than a "wip"/"fix" trail — and one of those commit messages is itself the clearest evidence for the blocking stop-check gap above.
Deployment notes
rocm serve --verbose/--foreground changes observable behavior on a managed-engine crash: it now exits non-zero with a new message and stamps the manifest/engine-state failed, where it previously exited 0 and printed "server process exited." Any external automation keyed on the old string or exit code will break — worth calling out explicitly once the doc-sync finding above is addressed.
What this covered
Read completely: apps/rocm/src/engine_ipc.rs, apps/rocm/src/serve_cmd.rs, apps/rocm/src/service_launch.rs, apps/rocm/src/services_text.rs, crates/rocm-core/src/lib.rs, crates/rocm-core/src/process.rs, docs/testing.md, docs/manual-testing.md, README.md, tests/e2e-cucumber/features/model_serving.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs — via three independent background review passes grouped by coupling (engine_ipc/serve_cmd/lib.rs; service_launch/process/services_text; e2e-tests/docs/README), each given the full files plus an exact diff patch, followed by my own direct agent-instruction-adherence pass against the root AGENTS.md and a git-history/blame pass (git log, reading the PR's own round-by-round commit messages) over the same files. Verified via direct code tracing rather than assumption: the merge conflict (live git merge --no-commit --no-ff dry-run against current origin/main), the CAS/discard bug in update_restarting_record's caller, the Windows CI job's actual e2e configuration (.github/workflows/ci.yml), and the tracing-subscriber's file-only destination (apps/rocm/src/logging.rs, read for this purpose though outside the PR's diff). Did not run cargo build/test/clippy myself; relied on the GitHub Checks API, which shows all 25 check runs for this commit completed with conclusion success (one skipped, the non-required skill-evals job) — no failing required check to report.
Did not run as separate passes: "prior changes" (no prior-PR review comments were read, by design — only commit_id/login/state metadata was used for the mandatory claim-check); "code comments" was folded into the coupled-group passes above rather than run as a standalone pass.
Disclosure: the mandatory pre-review claim-check (get_reviews) and the PR-metadata fetch needed to read mergeable_state both returned full review/PR bodies as a side effect, which is more than the task intended me to see before findings were settled — this included an existing human reviewer's CHANGES_REQUESTED review and a prior automated review at an older commit. No finding above was taken from either; each was independently re-derived and directly verified against the code, git history, and CI configuration as cited inline (the live merge-conflict re-check in particular shows the conflict is a new, different collision than any earlier round would have seen, since main has moved again). Several findings likely overlap with points raised in those earlier texts — that reflects the same defects being independently reachable from the code, not reliance on having read them. Not reconciled against either review's thread.
🤖 Generated with Claude Code
Review of PR 854 (automated pass) found that the stop-vs-crash guard rests on a status that cannot carry it, and it is right. `managed_service_stop_was_requested` treated `status == "stopped"` as proof of an intentional stop. But `refresh_managed_service_runtime_liveness` reconciles ANY record whose tracked pids are all dead to `"stopped"`, with no marker — `dash.rs` calls it "demoting a crashed server's stale `running` record to `stopped`". So an ordinary concurrent `rocm services`/`rocm dash`/daemon read landing between a crash and the check could relabel that crash as a clean stop and have it reported as one: the exact failure this change exists to prevent. Adds `StopWatch`, sampled on every liveness poll on all three paths (attach, launch readiness, restart readiness). `stop_requested_unix_ms` is the only sound evidence of an intentional stop, but it is transient — written before a stop signals and cleared once it confirms — so a watcher reading it only after the child is gone can miss it. Latching it while the child is still being watched catches any stop in flight, including an *unconfirmed* one, which re-stamps the marker and leaves the status alone. The restart path switches to `wait_for_service_http_ready_with_progress` purely for its per-tick hook; nothing is rendered. That left the non-progress `wait_for_service_http_ready` wrapper with no production caller, so it is removed and its three tests call the progress form directly. The latch does NOT close the race, and the status term therefore stays. Verified empirically: with the term removed, the external-stop scenario goes red, because a stop can be requested, signal, and confirm — clearing its own marker — well inside one 200ms poll. A confirmed stop leaves no other trace, so status is the only evidence it happened. Closing this properly means making the two distinguishable on disk (a distinct status for a bare dead-pid reconciliation, or a durable marker for a confirmed stop), which changes bookkeeping shared with `rocmd` and the dashboard. That is deliberately out of scope here and is now documented at the guard with the reasoning and the evidence, rather than left as an unstated assumption. Also from the same review: - docs/manual-testing.md gains an attached-crash section (4b) covering the new exit code, message and `failed` stamp, with the deliberate-stop control. - serve-40's two assertions now say which writer each one pins: the manifest is parent-side only, while the engine-state file is stamped by the child's own backstop and would stay green if the parent write were deleted. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Addressing the automated pass ( The one that mattered — and where I was wrong@juhovainio asked whether anything other than an operator stop can write
What I changed. Added What I could not close, and why. The latch narrows the race; it does not close it. I removed the A stop can be requested, signal, and confirm — clearing its own marker — well inside one 200 ms poll, and a confirmed stop leaves no other trace. So the status term stays. The residual race is now documented at the guard with that evidence and the two ways to close it properly (a distinct status for a bare dead-pid reconciliation, or a durable marker for a confirmed stop) — both of which change bookkeeping shared with I'd rather ship this as a documented, narrowed, pre-existing race than either leave it as an unstated assumption or expand this PR into shared status semantics. Already fixed before this pass was writtenThe review ran against
Also addressed now
On splittingKeeping it as one PR. The stop-vs-crash work isn't adjacent scope — it exists because the headline fix would otherwise turn every ordinary piped Verificationfmt, |
main took serve-40/41 (ROCMAI-827 #838 and the OOM-guidance scenario), so these are renumbered serve-42/43 once more; the @id: tags are unchanged and remain the stable reference. Review finding, and a real one: the "child died before its readiness wait even started" branches on both the launch and restart paths stamped `failed` without first asking whether a stop had been requested — contradicting `mark_service_failed`'s own documented precondition, and unlike every sibling branch (`finalize_managed_launch_readiness`, `record_restart_readiness`, the attached `ServerExited` arm), which all consult it. A `rocm services stop` landing in that ~200ms post-spawn window killed the child we had just spawned and was then reported as a crash. The launch branch's own comment already anticipated a concurrent stop writing its marker in "this same narrow window"; it simply never read it. Both branches now check first and report the stop instead. On the restart side the same branch discarded `update_restarting_record`'s return value, so a CAS that detected a concurrent writer (returning `None`, having deliberately written nothing) was followed by an unconditional engine-state stamp anyway — and since `refresh_from_engine_state` reads that file straight back into the manifest, the clobber the CAS had just prevented came back through the other door. The engine-state write is now gated on the manifest write having actually landed. Also adds the Windows half of the liveness coverage: `managed_child_liveness_windows_tracks_a_real_child_through_termination` spawns a real child, asserts it reads as running, terminates it, and asserts it stops reading as running and stays that way. `ManagedChildLiveness::Windows` is the crash-detection primitive on the reported platform and cannot be constructed off Windows, so the Unix test could not reach it and the scripted-engine scenarios run the Unix arm — it had nothing exercising it, while its sibling `spawn_detached_with_log` did. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Second automated pass (against The real find: the post-spawn branches never checked for a stopThis one was correct and I'd missed it twice. The "child died before its readiness wait even started" branches — on both the launch and restart paths — called What makes it a fair catch: the launch branch's own comment already anticipated "a The compounding bug on the restart side was right too. That branch discarded Windows liveness coverageAdded Already landed before this passThe review ran against
Scenario renumbering, again
Not takenThe 1.5 s scripted-death timer. I left it. The margin is ~7.5x over the ~200 ms startup window, and gating the scripted engine on a signal the real liveness check writes would couple the fixture to the very code path under test. If it ever flakes, the fix is a longer margin, not tighter coupling — I'd rather not pay that coupling for a risk neither of us has observed. Verificationfmt, One process note on this round: the first attempt at this merge truncated its own conflict output and I missed three of the five conflicts, then |
…zero README and docs/manual-testing.md already describe the ROCMAI-541 behaviour; the `--help` long description still stopped at the Ctrl-C stop path, so the one place a user looks first never mentioned that an unexpected engine exit is now a loud non-zero failure rather than "server process exited" / exit 0. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com> # Conflicts: # apps/rocm/src/engine_runtime_select.rs # apps/rocm/src/main.rs # apps/rocm/src/service_launch.rs # apps/rocm/src/services_text.rs # crates/rocm-core/src/lib.rs # tests/e2e-cucumber/features/model_serving.feature
|
Merged Your blocking items1. CI hadn't run / conflicts. Merged
2. serve-33 didn't assert the engine's own failure reason. Already addressed in 3. No Windows-runnable test for the log attach. Also in 4. Docs not synced. README and Verification (Linux MI300X; the Mac cannot build
|
Addressed in 550f9a1: merged main (66 behind, service_launch.rs and the restart path both relocated/rewritten on main), and synced the --help text that items 2-4 had left behind. Details in the PR comment.
nowycondro
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (base e2e5c5bb3d3…head caebb946eb8). Implements the ROCMAI-541 acceptance criterion ("the engine state file is marked failed on any error exit") across every exit point of a managed engine child; ticket ROCMAI-541 is named but not retrievable from this environment (no tracker access here), so this is judged against the PR description alone.
Blocking
-
The Windows post-spawn early-exit check still uses the crash-prone probe this very PR replaces everywhere else —
apps/rocm/src/service_launch/spawn.rs:559-577(theensure_spawned_child_survives_startupcall), contrastapps/rocm/src/service_control.rs:705-711(ensure_restarted_child_survives_startup).
The Windows launch path passesrocm_core::process_is_runningas theis_runningcheck (spawn.rs:563).process_is_running's own doc comment (crates/rocm-core/src/process.rs:606-611) warns: "a live process this one may not open reads as not running here; a decision that must not take it for gone... asksprobe_processinstead."ensure_spawned_child_survives_startupreturning false here is exactly such a decision: it feeds straight intosettle_startup_death, which (absent a recorded stop) callsmark_service_failedand bails — i.e. a transientOpenProcessfailure on a perfectly healthy new engine gets reported as a crash. This PR introducesManagedChildLivenessspecifically to fix this class of false negative (Windows::is_runningusesprobe_process(..).may_be_running(), treatingUnqueryableas "may still be running"), and the sibling restart path got the fix —service_control.rs:710passes|| liveness.is_running(), with a comment explicitly contrasting itself against "a bareprocess_is_runningpid probe." The launch path even constructs the neededManagedChildLiveness::from_pid(child_pid)two lines later (spawn.rs:580) — it is simply not used for the check itself. This is new code in this PR (the PR's own table lists "Windows post-spawn early exit" as one of three paths "added in review"), unverified on Windows per the author's own disclosure, and currently reintroduces ROCMAI-541's symptom on the one platform the ticket is about.
Confidence 90 · logic · Fix: constructlivenessbefore the check (as the restart path does) and pass|| liveness.is_running()instead of the raw function. -
The PR description names the wrong regression scenarios — description body ("Tests" and "Also in this diff" sections) vs.
tests/e2e-cucumber/features/model_serving.feature:926-951anddocs/testing.md's new "Managed-engine crash vs. stop" section.
The description repeatedly says "serve-55 and serve-56 cover the attached--verbosepath" and that mutation-testing "turn[s] serve-56 red".serve-55/serve-56are pre-existing, unrelated scenarios ("The recommended small assistant serves from its short name alone" and "Serving a Hugging Face repo with no quantization variant is refused",model_serving.feature:884,903). The scenarios this PR actually adds areserve-57/serve-58(@id:serve-scripted-death-after-startup,@id:serve-scripted-external-stop-while-attached), which the PR's owndocs/testing.mdaddition correctly names. A reviewer or future maintainer following the description to the named scenarios lands on the wrong tests entirely.
Confidence 95 · mechanical · Fix: correct every "serve-55"/"serve-56" reference in the PR description to "serve-57"/"serve-58". -
A stop can still lose its own evidence through an untouched helper, reopening the crash-vs-stop confusion this PR closes elsewhere —
apps/rocm/src/managed_service_status.rs:264-286(settle_pending_stop_key_cleanup), consumed byapps/rocm/src/service_launch/readiness.rs:499-503(managed_service_stop_was_requested) andapps/rocm/src/service_control.rs:270(stop_internal_managed_service_confirmed_with).
This PR's new contract (stated directly inmanaged_service_stop_was_requested's doc) is thatstop_requested_unix_ms/settled_stop_unix_msare the only evidence of a deliberate stop, and that every deliberate stop sets one of them.settle_pending_stop_key_cleanup's non-empty-recorded-process branch computesconsumed_marker = record.stop_requested_unix_ms.take()and then discards it — only the empty-process branch a few lines below assigns it tosettled_stop_unix_ms. This function is unmodified by this PR but runs ahead ofstop_internal_managed_service_confirmed_with's own re-read whenever a concurrentrocm services/rocm dash/rocmdliveness refresh observes the signalled PIDs already dead first — a racestop_internal_managed_service_confirmed_with's own comment explicitly anticipates ("a record that reached 'stopped' by another route still gets its deferred cleanup", "load_managed_service's liveness refresh would act on this stop's own marker before the verdict below has been reached"). When that race lands, the record ends upstatus: "stopped"with both markers unset, so a latermanaged_service_stop_was_requestedread for the same service (e.g. the attachedServerExitedcheck) sees no evidence at all and would misreport the stop as a crash — the original symptom, via a path this PR didn't touch.
Confidence 72 · architectural · Fix: have the non-empty-process branch move the marker tosettled_stop_unix_mstoo, the same way the empty-process branch andstop_internal_managed_service_confirmed_withdo. -
A required-looking check fails at this head commit —
E2E consolidated report (self-hosted), FAILURE, run 43446100233.
The job log shows this is not a test-content failure: thee2e-gpu(MI300X) ande2e-gpu-strix-windowslanes were themselvescancelled, and the consolidator errors because a cancelled lane uploaded no report ("lane e2e-gpu ran (needs result: cancelled) but uploaded no e2e-gpu-report").Windows build and test (debug)— the one check the author names as covering the Windows paths this PR touches — is green. Still, this is a red check on the current head and needs a rerun (or triage of why the self-hosted lanes were cancelled) before merge.
Confidence 100 · architectural · Fix: rerun the self-hosted E2E workflow; confirm the consolidated report goes green once the GPU/Windows lanes actually execute.
Non-blocking
- The doc comment for the PR's central function is attached to the wrong one —
apps/rocm/src/service_launch/readiness.rs:579-647. A ~55-line doc block describing a "failure branch" that "marks the engine's own state filefailed" and returnstrue/falseis clearly written forfinalize_managed_launch_readiness(line 648), but a missing blank line before the next paragraph attaches the whole block tostill_watching_this_supervisor(line 634) instead. The single most intricate function in this diff ends up with no rustdoc at all, while the guard function carries doc text describing different behavior.
Confidence 85 · mechanical · Fix: insert the blank line/doc split so each function keeps its own comment. - The new "move, don't drop" invariant isn't directly asserted in
service_control.rs's own primary tests —apps/rocm/src/service_control.rs(services_stop_reports_a_confirmed_stop_as_stopped,a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing). Both exercise the real confirmed-stop path whose new line isrecord.settled_stop_unix_ms = record.stop_requested_unix_ms.take(), but neither assertssettled_stop_unix_ms.is_some()afterward. The mirrored change inapps/rocmd/src/service.rsdid get that assertion added to its sibling test (service.rs:662).
Confidence 70 · mechanical · Fix: add the samesettled_stop_unix_ms.is_some()assertion to theservice_control.rstests.
Decisions for the author
docs/architecture.md's entries forreadiness.rs,process.rsandengine_ipc.rsstill describe their pre-PR contents — answered-by-intent. The PR's "Known limitations" section states: "docs/architecture.mdgainedmanaged_child.rs, but thereadiness.rs,process.rsandengine_ipc.rsentries still describe their older contents." Independently checked: the entries still read as generically true (not contradicted by the new code) but don't mention the new crash/stop-verdict logic or the scripted-engine hook. Would otherwise be a documentation-accuracy finding; confirm this is still the intended follow-up scope.
Positive signals
serve-57/serve-58deliberately pin two independent writers offailed(the parent-side manifest write vs. the in-childrecord_engine_serve_http_failurebackstop) with comments explaining why each assertion would or wouldn't survive the other writer being deleted — unusually resistant to a shallow partial fix passing the test.- The
cli.rsregression test for therocm logs→rocm services logsremediation-hint fix positively proves the old text was wrong (rocm logs <id>parses asLogs { service: None, .. }, a search query) rather than only asserting the new text is right. settled_stop_unix_ms's doc comment (crates/rocm-core/src/service_record.rs) adds a "Writers" table naming all four call sites that touch the field; independently verified against the actual code at all four and found accurate.
Deployment notes
- Exit-code/behavior change for scripts: background launches,
rocm serve --wait, androcm services restartnow exit non-zero and leave the recordfailedon a genuine crash where they previously exited 0 (or, for--wait, leftstopped/starting). Anything scripting around the old exit codes needs to adjust. - Windows engines are now logged (
spawn_detached_with_log), which the author flags as forcing blanket handle inheritance in the child, matching the existingspawn_hidden_console_with_logmodel rather than a new exposure. - New
rocm-corepublic API (ManagedChildLiveness,spawn_detached_with_log) — any out-of-tree consumer of this crate should note the surface addition.
What this covered
Every file in the diff (23 files, +3448/-150) read in full against e2e5c5bb3d34605888fcab61f4511ab24f215c3c…caebb946eb8390f1fc284f84bd94bd8bd9a186af: crates/rocm-core/src/{managed_child.rs,process.rs,lib.rs,service_record.rs}, apps/rocm/src/service_launch/{spawn.rs,attach.rs,start.rs} and service_launch.rs, apps/rocm/src/service_launch/readiness.rs, apps/rocm/src/service_control.rs + apps/rocmd/src/{service.rs,watchers/server_recover.rs}, apps/rocm/src/{cli.rs,engine_ipc.rs,engine_runtime_select.rs,serve_cmd.rs}, the e2e-cucumber test additions, and README.md/docs/architecture.md/docs/manual-testing.md/docs/testing.md — split across independent blind passes grouped by coupling, each reading surrounding/caller code beyond the diff, plus a consolidation pass that re-traced the two highest-severity candidates (the Windows liveness asymmetry and the stop-marker race) directly against source before scoring.
Did not run: a dedicated "prior changes"/version-history pass (git-log/blame context was read incidentally while tracing call sites, not as a separate pass); the workspace test/clippy suite locally (relied on the PR's reported CI results per-check, see Blocking). This PR has several earlier automated-review rounds at prior commits; per method this round was produced blind and was not reconciled against their content — it may re-find things they already raised, or diverge from them, as an independent check against the current head rather than a continuation.
🤖 Automated by an AI agent
… crash Two paths could still report a healthy or deliberately-stopped managed engine as crashed, which is ROCMAI-541's own symptom surviving inside its fix. The Windows post-spawn early exit was handed `rocm_core::process_is_running` while the sibling restart path already asked the `ManagedChildLiveness` it held. That probe's own doc comment warns that a live process this one may not open reads as not running there, and this check's `false` goes straight to `settle_startup_death`, which stamps the manifest and the engine state file `failed` and aborts the launch -- so a transient `OpenProcess` denial on a perfectly healthy new engine was a reported crash. The launch already constructed the right value two lines below the check; it is now built before it and used for it. `ensure_spawned_child_survives_startup` takes the answer instead of the pid, so passing the raw probe is a type error rather than a silent one, and `rocm-core` grows an all-platform `pid_child_is_running` seam -- only its call site is `#[cfg(windows)]` -- so the Linux lane pins the one answer the two probes disagree about. `settle_pending_stop_key_cleanup` consumed a stop's marker and discarded it on its non-empty-process branch, leaving `status: "stopped"` as the only trace. That is not evidence: the same function writes that status for a record whose pids are merely gone. It runs ahead of `stop_internal_managed_service_confirmed_with`'s re-read whenever a concurrent `rocm services`/`rocm dash`/`rocmd` liveness refresh sees the signalled pids dead first -- the race that function's comment anticipates -- so the record ended with both markers unset and the next `managed_service_stop_was_requested` called the operator's own stop a crash. The marker now moves into `settled_stop_unix_ms` on both branches, as the confirmed-stop path and `rocmd` already do. Both fixes are pinned by tests that fail when the fix is reverted. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Addressed all three actionable blocking findings in 1. The Windows post-spawn early exit ran on the raw probeCorrect — and the shape of the bug was the inconsistency itself. The restart path got the fix ( Fixed as suggested:
Stated plainly: the call-site change itself is inside a 2. Wrong scenario numbers in the descriptionCorrect, and this is the second time this PR's renumbering has been missed in prose — a Swept for the rest rather than fixing only what was reported: 3.
|
main added serve-57 (registered vision GGUF retains the Lemonade router) and serve-58 (simple registered GGUF served directly by the GPU backend) to model_serving.feature, colliding with this branch's serve-57/serve-58. Keep both pairs: main's retain 57/58 and this branch's scripted-engine regressions move to serve-59 and serve-60 in the same relative order, so feature_naming's position-equals-index rule still holds with no gaps. The @id: tags are untouched, since expectations.toml and the step bindings key on those rather than on the numbers. Swept the numbered references and repointed them at 59/60: the two cross-references inside the scenarios' own comments, the two bullets and the repro command in docs/testing.md, and the doc comment in service_launch/readiness.rs. The repro filter becomes `-n "serve-(59|60)"`; left as `serve-5[78]` it would have matched main's two new @requires-gpu scenarios, which the documented HIP_VISIBLE_DEVICES="" run skips, so the command would have run nothing and looked clean. Merge only: no review feedback is addressed here, and the three review fixes at f8f09e2 are carried through unchanged. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Merged
|
main |
this PR (before) | this PR (after) | |
|---|---|---|---|
| 57 | serve-lemonade-complex-registered-inference — registered vision GGUF retains the Lemonade router |
serve-scripted-death-after-startup |
— |
| 58 | serve-lemonade-simple-registered-direct-route — simple registered GGUF served directly by the GPU backend |
serve-scripted-external-stop-while-attached |
— |
| 59 | — | — | serve-scripted-death-after-startup |
| 60 | — | — | serve-scripted-external-stop-while-attached |
Resolved by keeping both pairs: main's retain 57/58 and sit first, this PR's move to 59/60 in the same relative order. feature_naming's position-equals-index rule holds with no gaps — proven by gate 5 below, not by eye.
@id: tags are unchanged. expectations.toml and the step bindings key on those, not on the numbers, so nothing downstream moved.
Every numbered reference updated
Swept with grep -rn "serve-5\|serve-6" (note: not serve-5[0-9], which silently misses a bracketed filter regex like serve-5[78] — that is how a stale one survived a previous round). Each hit was classified as main's or this PR's before editing; main's new scenarios are referenced only by @id: anywhere else in the tree, so every numbered serve-57/serve-58 hit was this PR's.
model_serving.feature— the two scenario titles, plus the two cross-references inside serve-60's own comment block (AttachOutcome::ServerExitedarmserve-59, and "likeserve-59's crash").docs/testing.md— the two bullets under "Managed-engine crash vs. stop (ROCMAI-541)".docs/testing.md— the repro filter, nowHIP_VISIBLE_DEVICES="" cargo xtask e2e -- -n "serve-(59|60)". This one mattered: left asserve-5[78]it would now matchmain's two new scenarios, which are@requires-gpuand therefore skipped under the documentedHIP_VISIBLE_DEVICES=""— the documented command would have run zero scenarios and exited clean. I checked the new regex against every scenario name in every.featurefile: it matches exactly serve-59 and serve-60 and nothing else. Alternation is already the house style here (cf.update-(09|10)).apps/rocm/src/service_launch/readiness.rs:484— doc comment,(serve-58)→(serve-60).- The PR description, corrected to 57/58 last round, now says 59/60 (8 occurrences, including its own
-n '(serve-59|serve-60)'command line).
The three review fixes survived
main has not touched spawn.rs, service_control.rs or managed_service_status.rs since the merge base, so no hunk competed with them. git diff f8f09e2dd over those three files is empty — the Windows launch-path liveness fix, the settle_pending_stop_key_cleanup marker move and their tests are byte-identical. readiness.rs differs by the one-line comment renumber above and nothing else.
Gates (Linux devbox — this Mac cannot build the workspace)
Run one at a time, each from a clean sync of the merge result:
| gate | result |
|---|---|
cargo fmt --all --check |
pass |
cargo clippy --workspace --all-targets -- -D warnings |
pass |
cargo clippy --workspace --all-targets --features e2e-test-hooks -- -D warnings |
pass |
cargo clippy -p e2e-cucumber --test e2e -- -D warnings |
pass (run separately; the --all-targets sweep false-greens the e2e step files) |
cargo test -p e2e-cucumber --test feature_naming |
4 passed — the renumbering proof |
cargo xtask check-architecture-doc / check-crate-edges / check-module-size |
pass |
cargo test -p rocm --bin rocm |
1894 passed, 0 failed, 8 ignored |
No flakes hit.
Why this keeps happening
This is the third collision on the same two scenarios (55/56 → 56/57 → 57/58 → 59/60). The cause is structural, not bad luck: both these scenarios and everyone else's are appended to the end of model_serving.feature, which is a hot file, and the index is positional — so any two branches that append concurrently collide, and the loser has to renumber and chase every prose reference to the old numbers. The first two rounds each left stale prose behind that a reviewer had to catch. Worth considering whether the positional-index rule could key on @id: instead, or whether new scenarios should be inserted by topic rather than appended.
juhovainio
left a comment
There was a problem hiding this comment.
(Correction: the previous review posted by this same automated process on this pull request, consisting only of the word "placeholder", was posted in error due to a tool-usage mistake and should be disregarded. This review below is the intended one.)
Review — needs work
Full review of the whole change (all 24 touched files, read completely at the diffed commits, not just the hunks). Implements ROCMAI-541 (an internal AMD tracker identifier; not retrievable from here).
Blocking
-
A crash in the fresh-launch startup window is overwritten back to
stoppedby its own cleanup path —apps/rocm/src/service_launch/spawn.rs:87-99(settle_startup_death) and:422-433(spawn_managed_engine_child'sErrarm), composed withapps/rocm/src/service_launch/launch_transaction.rs:159-211(abandon_failed_managed_launch_with) andcrates/rocm-core/src/service_record.rs:633-636(mark_launch_abandoned).
settle_startup_deathcorrectly merge-writes"failed"into both the manifest and the engine-state file and returnsCrash, which becomes anErrbubbled up throughlaunch_managed_engine_child_locked.spawn_managed_engine_child'sErrarm then unconditionally callsabandon_failed_managed_launch(paths, &record)with the stale in-memoryrecordcaptured before the crash (status still"starting"). That cleanup clones it, callsmark_launch_abandoned()(sets status"stopped"— by its own doc, deliberately, to keeprocmd's recovery scan from respawning it), and does an unconditional whole-fileabandoned.write(), clobbering the"failed"that was merged into the manifest moments earlier. The engine-state file is untouched by this cleanup and keeps"failed", leaving a permanently self-inconsistent record thatrefresh_from_engine_statecan never reconcile (it only re-reads the engine-state file when the manifest status isstarting/running/recovering/ready). This is exactly the "Unix post-spawn early exit" row the PR's own description lists as fixed, and the mirror-image sibling path this PR added for restarts (service_control.rs'ssettle_restart_startup_death/update_restarting_record) explicitly uses a CAS-style guarded write to avoid this same clobber — the fresh-launch path has no equivalent guard. The existing test (a_child_lost_in_the_startup_window_is_marked_failed_unless_a_stop_asked_for_it,spawn.rs:803) callssettle_startup_deathdirectly and never goes throughspawn_managed_engine_child's fullErrpath, so it passes despite the clobber.
Confidence 90 · architectural · Fix: either make theCrashverdict skip/short-circuit the record-clobbering write (merge rather thanmark_launch_abandoned+write()), or giveabandon_failed_managed_launchthe same guarded-write disciplineupdate_restarting_recorduses. Add a test that drives a crashing child through the fullspawn_managed_engine_childcall and asserts the final on-disk manifest status is"failed". -
The stop's own durable evidence (
settled_stop_unix_ms) can be clobbered back to unset by the explicit stop path itself —apps/rocm/src/service_control.rs:270, insidestop_internal_managed_service_confirmed_with(:188).
record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();runs on a record freshly re-read from disk afterterminate(). If a concurrentrocm services/rocm dash/rocmdlisting's liveness refresh (managed_service_status.rs:286-287) observes the same signalled PIDs already dead first and performs this exact same move itself (stop_requested_unix_ms → None,settled_stop_unix_ms → Some(ts)), then by the time this function's own re-read happens,stop_requested_unix_msis alreadyNone. The line above still runs unconditionally:.take()on an already-Nonefield returnsNone, and thatNoneis assigned intosettled_stop_unix_ms, erasing theSome(ts)the concurrent refresh had just correctly written. The on-disk record ends up with both fields unset — the exact "crash indistinguishable from a stop" bug class this PR exists to close, reachable through the explicit stop path's own confirmation step rather than the liveness-refresh path it was written to guard against.
Confidence 85 · logic (race condition) · Fix: only move the marker when one is actually present, e.g.if let Some(ts) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(ts); }, leaving an already-settled value alone. -
No test anywhere in
service_control.rsassertssettled_stop_unix_ms's actual value after a confirmed stop —apps/rocm/src/service_control.rs(wholemod tests).
settled_stop_unix_msis this PR's central fix for distinguishing a stop from a crash, but every confirmed-stop test in this file (services_stop_reports_a_confirmed_stop_as_stopped,a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing,stop_final_write_keeps_what_others_wrote_while_it_ran,a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it) only checksstop_requested_unix_ms == None, neversettled_stop_unix_ms. Reverting line 270 to the oldrecord.stop_requested_unix_ms = None;would not fail a single test in this file — behaviour this change introduces, untested, and the direct reason the previous finding shipped undetected.
Confidence 90 · test-quality · Fix: addsettled_stop_unix_msassertions to the confirmed-stop tests, and extenda_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_itto asserton_disk.settled_stop_unix_ms == Some(_)(which would currently fail given the finding above). -
settle_exited_service's "was this deliberate" verdict and the record it stamps come from two independent, unsynchronized reads —apps/rocm/src/service_launch/readiness.rs:107-145.
let deliberate = managed_service_stop_was_requested(paths, service_id);performs its own file read;ManagedServiceRecord::read_manifest(paths, service_id)immediately after performs a second, independent read. A stop marker written in the gap between the two pinsdeliberate = falsefrom the stale first read while the second read could already show the record live, stamping"failed"for a death that was in fact just requested. New code (the whole--waitcrash-vs-stop distinction this PR adds), with no comment acknowledging this particular window, unlike the sibling checks elsewhere in this same file that discuss exactly this class of race at length.
Confidence 75 · logic · Fix: derivedeliberatefrom the same manifest snapshot already read forlatest/on_disk, instead of a separate call tomanaged_service_stop_was_requested. -
StopWatch's doc comments describe a "status term" inmanaged_service_stop_was_requestedthat this same PR deleted —apps/rocm/src/service_launch/readiness.rs:546("That case is caught by the status term inmanaged_service_stop_was_requested") and:567("The status term still backs the one-shot check at the end of a wait"), versus the actual implementation at:499-505:record.stop_requested_unix_ms.is_some() || record.settled_stop_unix_ms.is_some()— no status term anywhere. The PR's own stated purpose is removing exactly thatstatus == "stopped"term as unsound; both the function and theStopWatchdoc that still refers to it are new lines in this diff, so this is a self-contradiction introduced within one change, not inherited drift. A maintainer trustingStopWatch's doc would believe a safety net exists that was deleted by the very same PR — precisely the kind of comment that later causes a regression when someone "cleans up" what looks like dead code on its word.
Confidence 85 · mechanical (doc/comment accuracy in new code) · Fix: updateStopWatch's doc to name the actual residual-race closer (settled_stop_unix_msdurability, or whatever the intended remaining gap is), not a "status term" that no longer exists. -
A test asserting a guard's effect can't actually fail if that guard is deleted —
apps/rocm/src/service_launch/readiness.rs:1409-1416(testa_wait_death_is_a_crash_unless_a_stop_asked_for_it, third arm), exercising theelse if latest.status == on_disk { return deliberate; }guard at:140-142. In this armis_live(&latest)is alreadyfalse, so neither branch of the precedingif is_livemutateslatest.status; whether the guard early-returns or is deleted and falls through towrite(), the write isstatus: "failed"either way, and the test's only assertion (after.status == "failed") passes identically both ways. Per this project's own test-quality bar, mutating a single branch inside the change and staying green means the test does not cover that branch.
Confidence 80 · logic/test-quality · Fix: assert something the guard actually controls — e.g. that an unrelated field written to disk between the two reads survives the write (demonstrating the guard avoids a stale-snapshot overwrite), rather than a result identical either way. -
A GPU-refusal waiver derived variable was not updated for the new
scripted_engine_deathbypass it claims to fully mirror —apps/rocm/src/serve_cmd.rs:906, with the exemption added at:885-889. The no-GPU bail gained a third bypass term (scripted_engine_death), butlet gpu_refusal_waived_for_reuse = reuse_existing && !scripted_backend_failure;'s own comment ("the scripted backend failure is the one remaining term") is now false — it omits!scripted_engine_death. Traced the consumer atapps/rocm/src/service_launch/spawn.rs:367:if gpu_refusal_waived_for_reuse { bail!(...) }, fired when "nothing live to reuse" is discovered at spawn time. Ifreuse_existingandscripted_engine_deathwere ever both true (not exercised by the current e2e scenarios, which always do a fresh launch), the flag would wrongly readtrueand hard-bail instead of reaching the scripted-death path, defeating the feature for that combination.
Confidence 78 · logic · Fix: add&& !scripted_engine_deathto the expression and correct the comment's "one remaining term" claim. -
The remediation-command test proves the named commands parse, not that they clear the condition that printed them —
apps/rocm/src/cli.rs:2676(the_log_commands_the_crash_messages_name_actually_parse). This project's own standard for a remediation hint requires the named command to "actually clear the condition that printed it," asserted together with the resulting state in the same test. This test (a real improvement over nothing) only assertsrocm services logs <id>/rocm logs --service <id>parse via clap; the only other related check found (readiness.rs:1498-area) only asserts the printed message contains the command text, not that running it against afailedrecord prints that record's log. Could not confirm or rule out stronger coverage intests/e2e-cucumber/features/logs.feature(out of this review's deepest scope).
Confidence 70 · test-quality · Fix: either point to an existing e2e scenario that closes this gap in a comment near the test, or add one that seeds afailedrecord with a known log tail and asserts the named command actually prints it.
Non-blocking
None beyond the items below.
Decisions for the author
- Several of the blocking findings above share a root enabler worth addressing as a class, not just as point-fixes — non-blocking-improvement. The PR's "shape of the fix" is seven independent entry points that each owe the same verdict, and its own "Known limitations" already names the risk ("No systematic entry-point parity test. Three of the gaps above were the same shape and each was found by review rather than a test.") Two of the findings above (the fresh-launch clobber, and the explicit-stop-path clobber) are exactly that pattern recurring a fourth and fifth time, in each case because a call site did a plain/unconditional write where a sibling call site (the restart path's CAS guard,
mark_service_failed's merge-write) got the same problem right. Worth considering a single shared "stamp this verdict without clobbering what's already there" primitive used by every entry point, plus a test that drives each real entry point end to end (as the existinga_child_lost_in_the_startup_window...test does not, per the first finding) rather than unit-testing eachsettle_*/mark_*helper in isolation. engine_ipc.rs's in-child stop-detection (engine_stop_was_deliberate_in, checking the engine's own state file plusstop_requested_unix_ms) is a separate, independently-built mechanism from the rest of the PR's central one (managed_service_stop_was_requested, checkingstop_requested_unix_ms/settled_stop_unix_ms) — tradeoff. Investigated for a concrete bug (whether the early, pre-confirmationstop_requested_unix_mssignal it relies on could misclassify a genuine unrelated crash as deliberate during the window between the stop marker being written and the engine actually being signalled) and could not establish one with confidence: the window is plausibly intentional given how the Lemonade/vLLM adapters report a stop-triggered shutdown as anErr. Flagging only because it is a second, parallel implementation of "was this exit requested" with different inputs than the rest of the PR's carefully-documented mechanism; worth the author's own judgment on whether to unify them.
Positive signals
crates/rocm-core/src/managed_child.rs'spid_child_is_runningtest seam lets the Unix-zombie-vs-Windows-Unqueryabledistinction — the entire point ofManagedChildLiveness— be exercised on every platform via dependency injection, not just on a real Windows CI run.- The scripted-engine e2e scenarios (serve-59/serve-60,
tests/e2e-cucumber/tests/e2e/serving_steps.rs) drive a real spawned child through the actualAttachOutcome::ServerExitedproduction path and assert message text, exit code, and both the manifest's and engine-state's status together — the comments in these tests explicitly reason through "would this still fail if the production write were reverted," exactly the standard this review applies. apps/rocm/src/cli.rs:2676's remediation test deliberately also asserts that the old, brokenrocm logs <id>spelling still parses (so the original defect was silent), which is a sharper regression guard than simply testing the new spelling in isolation.docs/architecture.md:329's entry for the newmanaged_child.rsexplicitly cites PR #437 by number for the known Windows PID-reuse limitation it knowingly leaves unresolved — genuine prior-art awareness rather than a silently deferred gap.- The xtask check (
assert_prebuilt_e2e_lanes_enable_test_hooks/ the shipped-release-build scan inxtask/src/workflow_contract.rs) that the PR claims keeps thee2e-test-hooksfeature out of shipped binaries was read directly and is a real, reasonably robust static check on CI workflow text, not a vague claim.
Deployment notes
- This PR changes recorded service state semantics (
stoppedvsfailed) thatrocmd's recovery scan reads to decide whether to respawn a service; the PR's own text flags that whetherrocmdshould respawn afailedrecord is still an open question, pinned by a test rather than resolved. Operators relying on auto-recovery should be aware a crash during the fresh-launch startup window will currently still readstopped(see the first finding above), notfailed, so it will not be picked up by that recovery path despite the PR's stated intent. - Windows paths (
spawn_detached_with_log, the Windows restart early-exit path,ManagedChildLiveness::Windows) are explicitly disclosed by the author as unverified on real Windows hardware; onlywindows-debugCI exercises them.
What this covered
Every file in the 24-file diff was read completely at the reviewed commits (6f3185d97503554ec387f3a44d2979d457f3a4c9…7c0c281260e79e6f9e40f4c225067b62d75a4dab), plus the surrounding/callee code needed to judge each change in place (launch_transaction.rs, managed_service_status.rs, service_record.rs, engine_runtime_select.rs, and others not themselves in the diff). The review fanned out across independent workers grouped by coupling: core liveness/spawn (managed_child.rs, process.rs, lib.rs, service_record.rs, spawn.rs, start.rs, service_launch.rs), readiness+attach (readiness.rs, attach.rs), service_control+status (service_control.rs, managed_service_status.rs), CLI+rocmd recovery (cli.rs, engine_runtime_select.rs, serve_cmd.rs, rocmd/service.rs, rocmd/watchers/server_recover.rs), a history/prior-art pass (git log/blame plus PRs #437 and #960), and a docs pass (docs/architecture.md, docs/manual-testing.md, docs/testing.md, README.md) — all run as genuinely independent background workers blind to each other's results, each subsequently spot-verified directly against the source by the coordinating reviewer before being included above.
A shared concurrency limit on background workers (outside this review's control) meant two planned passes — the e2e-test-hooks/scripted-engine-hook group (engine_ipc.rs, the Cucumber step definitions and feature file) and the dedicated AGENTS.md-compliance pass — could not be scheduled as independent background workers after repeated attempts; both were instead conducted directly by the coordinating reviewer (same method, not independently blind) and are reflected above with that caveat. Module-size (all changed files comfortably under the 1,500-production-line cap), markdown-link discipline, and crate-layering (rocmd still never imports from rocm) were all checked directly and found clean. CI on the reviewed commit was a mix of completed/in-progress at review time (clippy, build-and-test, Windows build/test, and the Linux/Windows/Strix-Halo E2E lanes visible so far were green; a couple of GPU-hardware E2E lanes were still queued/in-progress) — not re-polled to completion as part of this review. Not reconciled against any discussion on this pull request.
🤖 Automated by an AI agent
…cleans up after A crash in the fresh-launch startup window was reported to the user as a clean stop, and the record was left permanently self-contradictory. `settle_startup_death` merges `failed` into both the manifest and the engine state file and returns `Crash`, which bubbles up as `Err`. `spawn_managed_engine_child`'s `Err` arm then handed cleanup the in-memory record captured *before* the spawn (still `starting`), marked it abandoned and wrote the whole file — putting `stopped` back over the `failed` recorded moments earlier, while the engine state file kept it. `refresh_from_engine_state` will not re-read an engine state file behind a `stopped` manifest, so the two could never agree again. Cleanup now applies its verdict to the manifest as it reads on disk and leaves a record that is already not live alone, so a recorded cause (and any stop marker a concurrent `services stop` wrote) survives. A merge rather than the restart path's CAS: declining to write would strand the record at `starting`, still reserving its port. This also makes the fresh-launch path agree with the two siblings that already leave a crash at `failed`, with the recovery consequence noted at the call site. The explicit stop path's own confirmation step could likewise erase `settled_stop_unix_ms`: `= stop_requested_unix_ms.take()` assigns `None` across when a concurrent liveness refresh already moved the marker, ending the record with both fields unset. Guarded, and asserted by the confirmed-stop tests, which previously checked only that the marker was gone. `settle_exited_service` took its deliberate/crash verdict from a separate file read to the one it stamped; both now come from one snapshot through a shared predicate. `StopWatch`'s doc no longer credits a `status` term this change removes. Signed-off-by: Roman Sirokov <roman@flowrl.com> Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Thanks — this was an accurate review. Five findings addressed in Verified on Linux (the workstation here cannot build the workspace). Gates: Finding 1 — fresh-launch crash overwritten back to
|
|
Two review rounds landed here at older heads — nowycondro's round
Non-blocking:
juhovainio's round
So: one blocking item genuinely open (juhovainio #6), everything else addressed. The doc-block move is not pushed yet; it goes up with that fix. One thing worth flagging that neither round could have seen: this PR's serve-59/60 and #678's new serve-59 now both claim 59 on top of the same |
Review finding: `a_wait_death_is_a_crash_unless_a_stop_asked_for_it`'s third
arm claimed to cover `settle_exited_service`'s `latest.status == on_disk`
early return, but asserted only that a record already reading `failed` still
read `failed` afterwards. Delete the guard and the write that follows stamps
`failed` over `failed`, so the assertion passed either way — the arm could
not fail for the reason it existed.
What the guard controls is not the status but whether the stale snapshot is
written back at all. The arm now plants a key on disk that
`ManagedServiceRecord` does not model: a read drops it and `write()`
re-serializes without it, so it survives exactly when the early return skips
the write. That turns the concurrent overwrite the guard exists to prevent
into something observable without racing for it.
Confirmed by deleting the guard: the new assertion fails with
`left: None, right: Some("concurrent")`, where before the whole arm stayed
green.
Also moves the ~46-line doc block that describes the failure branch, the
`failed` stamp and the true/false return onto
`finalize_managed_launch_readiness`, which it was written for. A missing
blank line had attached it to `still_watching_this_supervisor` — which has
its own doc — leaving the more intricate function with no rustdoc at all.
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Follow-up: the one blocking item I left open is now fixed, pushed as juhovainio #6 — the third arm of The fix follows your suggestion. What the Mutation evidence — deleted the guard: and the arm goes red on precisely the new assertion: Restored, and green again. Before this change, that same deletion left the whole arm passing. The same commit carries the doc-block move from nowycondro's non-blocking item: the ~46 lines describing the failure branch, the Verified on Linux: That clears every blocking finding from both rounds. The serve-59/60 collision with #678 noted above is still the one thing to watch at merge time. |
juhovainio
left a comment
There was a problem hiding this comment.
I went through this fresh at the current head. I couldn't demonstrate a hard blocker from the diff, but a few things are worth settling before merge, so I'm requesting changes. Inline comments cover the specifics.
--waitrace: when a concurrent liveness reconcile has already writtenstopped,--waitreports the crash but never stampsfailed. That contradicts the README and the "mark failed on any error exit" criterion for that path, and the unit test doesn't cover that arm.- Windows handle inheritance:
spawn_detached_with_lognow inherits handles for every managed engine, which can keep a captured parent pipe open. This was raised in earlier rounds and is still unverified; the Windows CI jobs were still pending when I looked. - Recovery loop: a crashed fresh launch now stays
failed, which makes it a rocmd recovery candidate, so an engine that crashes on every start can be respawned repeatedly. The PR lists this as an open question, but it ships in this change, so it should be decided here. - Docs drift: the
managed_child.rstype doc is stale, and the PR body itself says threedocs/architecture.mdentries (readiness.rs,process.rs,engine_ipc.rs) are out of date; they should be fixed in the same change. - Tests/scenarios: serve-60 only asserts "not failed" rather than
stopped, and background launch, restart,--waitand the new log-hint text are covered by unit tests only, with no scenario or stated reason.
AC1 and AC2 of ROCMAI-541 look met; AC3 is met except for the --wait gap above. This is a diff-only read; I did not build or run anything.
| &serde_json::json!({ "status": "failed" }), | ||
| ); | ||
| } | ||
| } else if latest.status == on_disk { |
There was a problem hiding this comment.
When the record already reads non-live (e.g. a concurrent rocm services reconcile wrote stopped with no stop marker), this branch returns deliberate (false) without stamping anything. The --wait caller then reports a crash while the manifest stays stopped and the engine state is untouched, which contradicts the README's "a --wait death leaves the record failed". The attach path stamps via mark_service_failed regardless of status; consider doing the same here (guarded on !deliberate) and adding a test arm for a pre-reconciled stopped record.
| program, | ||
| args, | ||
| env_overrides, | ||
| DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW | CREATE_UNICODE_ENVIRONMENT, |
There was a problem hiding this comment.
Handing std handles to CreateProcessW makes this spawn use bInheritHandles=TRUE for every managed engine. A parent whose own stdout/stderr is a redirected pipe (CI wrapper, script capturing rocm serve) can have that pipe held open by the long-lived engine, so the caller never sees EOF. This was raised in earlier rounds and is still unaddressed or untested; either scope inheritance with an explicit handle list or add a Windows test that a captured-pipe parent returns.
| /// its *real* wait status rather than merely probing whether the pid still | ||
| /// exists. | ||
| /// | ||
| /// On Windows there is no equivalent gap: [`process_is_running`] there opens a |
There was a problem hiding this comment.
Stale: this says the Windows variant defers to process_is_running, but is_running now goes through probe_process via pid_child_is_running, precisely to avoid that function's collapse of unqueryable into gone. Please update the sentence.
| "an externally stopped attach must exit 0, got {rc}:\n{combined}" | ||
| ); | ||
| assert!( | ||
| combined.contains("server process exited"), |
There was a problem hiding this comment.
The stop outcome is pinned by wording here and by assert_ne!(.., "failed") in the final step. Please assert the positive state (stopped) too, so the scenario cannot pass for any non-failed record.
| // `rocmd`'s own restart-recovery watcher needs no such carve-out | ||
| // here. Only when neither is present is this an unexplained exit | ||
| // — i.e. a crash. | ||
| if stop_seen_while_attached || managed_service_stop_was_requested(paths, &service_id) { |
There was a problem hiding this comment.
Background launch, rocm services restart and --wait also changed exit code/message but have no scenario (only serve-59/60 cover attach). Per AGENTS.md section 3, either add scenarios or state in the PR why they cannot run on the no-GPU lane.
Three conflicts. tests/e2e-cucumber/features/model_serving.feature: main added its own `serve-59` (the endpoint-key ownership scenario). Main's keeps 59; this branch's two become `serve-60` and `serve-61` and move after it, so each number still equals its position. Four prose references followed the renumber: two scenario lists and the repro filter in `docs/testing.md` (now `serve-(60|61)`), the `(serve-60)` note in `service_launch/readiness.rs`, and the feature file's own "exactly like serve-59's crash" line. The `@id:` tags are the binding key and are unchanged, so no step binding moved. docs/architecture.md: this branch adds a `managed_child.rs` row while main rewrote the neighbouring `managed_runtime.rs` description to mention the loader shims. Both are kept, and main's newer description is the one that survives — taking this branch's side wholesale would have silently reverted that doc update. tests/e2e-cucumber/tests/e2e/serving_steps.rs: two pairs of independent step definitions appended at the same place. Both sides of each pair are kept. In both cases the split left each side's function body unterminated and sharing one tail, so the tail is reproduced for each rather than closing only whichever block happened to come last. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Merged
The
Verified on Linux before pushing: On the red CI:
That is a prewarmed-runtime image problem, not a code one. The strongest evidence is that it reproduces identically on #1095, whose diff touches only |
|
Heads-up on a scenario-number collision between this PR and a sibling, so whoever merges second is not surprised by a red After
Each is correct within its own branch, because the rule is that a scenario's number equals its position in its file. The collision only exists across them, so whichever merges first forces the other to renumber. That is cheap and safe: The No action needed now; I will renumber whichever branch lands second. |
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (25 files, +4105/-180 against base 9101286a, head e00fac0f). Implements ROCMAI-541 ("a managed rocm serve that crashes must say so"); the ticket itself is not retrievable from here (no tracker access), so this is judged against the PR's own description of the acceptance criteria. The description's "shape of the fix" table and its claims about what was added in review were checked line-by-line against the diff and held up, with one material omission: apps/rocmd/src/service.rs received the identical "moved, not dropped" stop-evidence fix applied to service_control.rs, but is named nowhere in the table or in "the evidence a stop leaves" narrative — and that undocumented instance is exactly where the fix's guard was dropped (see Blocking #2).
Reconciliation note (done after blind findings were locked): this PR carries a standing CHANGES_REQUESTED review (juhovainio, at commit 75f27a79, the last substantive commit before the current head — the only commit since is a merge from main, not new work responding to it). Three of its five points are independently reconfirmed by this blind review: the --wait race (Blocking #1), the stale managed_child.rs type doc (Blocking #3), and the weak assertion in the external-stop control scenario (Non-blocking, serving_steps.rs:4346). A fourth point (recovery respawn loop) is mitigated by an existing backoff guard in apps/rocmd/src/watchers/server_recover.rs that I confirmed independently, though the underlying design question ("should a failed record be a recovery candidate at all") is still left open by this PR, as the reviewer notes. A fifth point (Windows handle inheritance in spawn_detached_with_log) I was not able to independently verify or refute in this pass — flagging it as still open rather than asserting it either way. None of this changes any finding below; it is reported because the task asks this review to say plainly when a standing objection is unresolved on the current commit.
Blocking
-
settle_exited_servicecan leave a crashed service's manifest readingstoppedforever, even though it correctly reports the death as a crash —apps/rocm/src/service_launch/readiness.rs:116-157(early return at 152-154), interacting withapps/rocm/src/managed_service_status.rs:408-416.
When a concurrent liveness-refresh reconciles a dying--waitlaunch's dead pids tostatus = "stopped"with no marker (which that function does unconditionally by its own doc comment: "dead pids alone do not distinguish a stopped service from a crashed one"),settle_exited_servicereadslatest.status == on_diskand returnsdeliberate = falsewithout ever writing.died_during_waitstill prints the correct crash message, but the persisted manifest is left reading"stopped", never corrected to"failed"— contradicting the PR's own acceptance criterion for this path. Traced directly, not hypothetical; no test constructs this starting state before calling the function.
Confidence 85 · logic · Fix: in theelse if latest.status == on_diskbranch, correct status to"failed"(and merge the engine-state file) when!deliberate, instead of returning early without writing. -
apps/rocmd/src/service.rs:137's unconditional marker move can erase a concurrent stop's already-correct evidence — unlike the guardedif let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }used for the identical operation inapps/rocm/src/service_control.rs:283-284and inmanaged_service_status.rs'ssettle_pending_stop_key_cleanup, this call site does it unconditionally:record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();. The surrounding comment acknowledges the record is re-read because a concurrent liveness refresh "may have persisted something while the stop ran" — concretely, if that refresh already moved the marker (stop_requested_unix_ms → settled_stop_unix_ms) before this re-read, the unconditional.take()here (Noneon an already-Nonefield) overwrites the just-writtensettled_stop_unix_mswithNone, erasing the one piece of durable evidence the whole PR relies on to prove a death was requested — reproducing, through this one path, the exact crash-indistinguishable-from-a-stop state ROCMAI-541 exists to close. This file is not named anywhere in the PR's own entry-point table despite receiving the same fix.
Confidence 90 · logic · Fix: guard it the same way asservice_control.rs:283-284. -
crates/rocm-core/src/managed_child.rs's own type-level doc comment (andfrom_pid's, line 79) is stale and self-contradicts the correct documentation 20 lines below it — the type doc (line 60) andfrom_pid's doc (line 79) say the Windows arm "defers toprocess_is_running"/reads the exit code with no qualification, whileis_running's own doc (lines ~84-93) correctly says the Windows arm usesprobe_processspecifically becauseprocess_is_runningcollapses a transientOpenProcessfailure into a false crash report — which is also what the code actually does. Within one new file, the first thing a reader sees documents the old, unsafe behavior as current. This repo's ownAGENTS.md§3 requires a comment stating how code behaves be checked against that code before pushing; it demonstrably wasn't, within a single new file.
Confidence 90 · mechanical (doc-accuracy) · Fix: update both stale doc comments to matchis_running's.
Non-blocking
-
ManagedServiceRecord::write()(crates/rocm-core/src/service_record.rs:668) takes no lock, unlikemerge_json_state_file/write_json_state_file(crates/rocm-core/src/endpoint_readiness.rs), which acquire aFileLock. A remaining plain.write()caller (e.g.stop_internal_managed_service_confirmed_withinservice_control.rs) is not ordered against a concurrent merge, so a.write()landing in the merge's narrow read-serialize-rename window can still silently overwrite fields the merge just set. Narrower than the pre-PR behavior, not a regression, and consistent with the PR's own disclosed "merge_json_state_fileis unsynchronized against the stop path" limitation. Confidence 70 · architectural · Fix: havewrite()take the same lock, or move every writer onto the merge path. -
The new crash-vs-stop backstop in
apps/rocm/src/engine_ipc.rs(record_engine_serve_http_failure/engine_stop_was_deliberate_in) is unreachable dead code for vLLM —engines/vllm/src/process.rs:162callsstd::process::exit()directly on a non-zero exit instead of returningErr, sorun_builtin_engine_serve_http(engine_ipc.rs:361-408) never gets control back to run the new mechanism for vLLM. Not user-visible-incorrect today (vLLM already writes"failed"itself, just above, independent of the new mechanism), but the new mechanism silently covers only 2 of the 3 dispatched engines. Confidence 75 · architectural · Fix: make vLLM'sserve_httpreturnErrinstead of exiting, or state the gap explicitly. -
apps/rocm/src/engine_ipc.rs:406's new crash-recording call site is exercised by no test, and the adjacent test-module comment's pointer to "the PR body" for justification doesn't hold up — the PR's actual Tests/Known-limitations sections don't address this specific call site. Confidence 75 · logic (partially self-acknowledged gap) · Fix: either add a seam a unit test can reach without starting the background thread, or state this specific gap in the PR text per this repo's own AGENTS.md §3. -
The external-stop control scenario's assertion is negative, not positive (
tests/e2e-cucumber/tests/e2e/serving_steps.rs:4346:assert_ne!(status, "failed")rather thanassert_eq!(status, "stopped")), so it would still pass if a regression left the record in some other wrong status. Confidence 85 · logic (test-quality) · Fix: assert the status equals"stopped"directly.
Decisions for the author
- tradeoff —
apps/rocm/src/service_launch/start.rs:181-199: a mid-wait external stop makesrocm serve --waitreportOkwithstatus: "stopped"and exit 0, i.e. the CLI's own wait "succeeds" by reporting a stop. Consistent with the attach path's wording, but worth the author explicitly confirming this is the intended UX for a foreground--waitthe user is actively watching. - non-blocking-improvement — the "was this death requested?" sequence (check stop evidence →
still_watching_this_supervisor→mark_service_failed→ bail) is independently hand-written in bothattach.rs'sServerExitedarm andreadiness.rs'sfinalize_managed_launch_readiness/start.rs, sharing only the low-level primitives, not the orchestration. No drift found today, but nothing forces a future change to be applied in both places. A shared helper would close this structurally. - answered-by-intent — the PR body states: "docs/architecture.md gained managed_child.rs, but the readiness.rs, process.rs and engine_ipc.rs entries still describe their older contents." This would otherwise be a blocking doc-staleness finding for three files that changed substantially (readiness.rs alone gained ~1127 lines); it's retired by this explicit disclosure. Confirm this is still the plan to fix promptly, not indefinitely.
Positive signals
StartupDeath(spawn.rs) andManagedChildLiveness(managed_child.rs) are clean, minimal enums replacing boolean blindness for "why did this exit," with tests that drive all three probe outcomes on every platform rather than only the one a real Windows CI run could construct.settle_exited_service's redesign to read stop evidence from the same snapshot it stamps (rather than a second independent read) correctly closes a documented double-read race its own doc comment describes.- Exit-code propagation was traced end-to-end (not just "it calls
bail!"): bothattach.rs's andfinalize_managed_launch_readiness's error paths reachmain.rs's exit-code mapping toExitCode::FAILUREun-swallowed. - The guarded "moved, not dropped" pattern in
service_control.rsandmanaged_service_status.rsis correctly implemented and has a dedicated regression test for the exact race it defends against — which is what makes the missing guard inapps/rocmd/src/service.rs(Blocking #2) stand out as an omission rather than an unknown risk. - The e2e-test-hooks scripted-engine mechanism is double-gated (compile-time Cargo feature and a runtime env var), backed by a real
xtaskrelease-workflow check (the_release_workflow_builds_no_feature_gated_binaries) that scans for any enabled feature, not just this one. rocmd's recovery watcher already has a restart-backoff guard, so the newfailed-as-recovery-candidate behavior doesn't create an unbounded crash-respawn loop.
Deployment notes
- Behavior change: a crashed fresh launch now reads
failed(previouslystopped) and becomes arocmdrecovery candidate in cases that previously were not retried at all — worth watching after rollout, independent of the open "should it be" design question above. - Remediation hint text changed from a bare
rocm logs <id>(which never selected a service) torocm logs --service <id>/rocm services logs <id>; both verified as valid, existing subcommands.
What this covered
Full review of the whole change, fanned out into 9 parallel independent workers grouped by coupling: service_launch/{spawn,readiness}.rs; service_launch/{attach,launch_transaction,start}.rs + service_launch.rs; service_control.rs; rocm-core/{managed_child,process,service_record,lib}.rs; rocmd/{service,watchers/server_recover}.rs; cli.rs+serve_cmd.rs; engine_ipc.rs+engine_runtime_select.rs+managed_service_status.rs; the new/changed e2e tests; and the docs (README/architecture/manual-testing/testing). Each worker read its assigned files completely plus directly relevant surrounding code, not just diff hunks. Driver-level passes: full history of the 43 commits unique to this branch (extensive iterative churn specifically on the stop-vs-crash race — consistent with the defects found above), impact-analysis greps for every new/renamed identifier across the whole repo (not just the diff), and direct re-verification of the PR's own table and several "Known limitations" claims rather than trusting them, per this task's explicit instruction.
One worker (cli.rs/serve_cmd.rs — exit-code and hint-text review) did not deliver its final report back to me before this review had to be submitted; I independently spot-checked its core claims myself (exit-code propagation traced via a sibling worker's cross-file trace to ExitCode::FAILURE; both hint-text forms confirmed as valid existing subcommands; no stray bare rocm logs <id> hint found anywhere in the diff) and found nothing, but that file group did not get the same from-scratch independent pass the other eight did — treat that area as lighter-touch than the rest.
Did not run: cargo build/test/clippy (no Rust toolchain in this environment) — relied on the PR's own CI, which is fully green on every required check at the reviewed commit (clippy, build-and-test, Test (affected crates), the mock E2E tests job, Windows build/test, docs build); one self-hosted hardware lane (E2E tests (Strix Halo, Windows, current)) shows cancelled rather than failure, consistent with the self-hosted-runner availability gaps this branch's own commit history separately discusses, and is not treated as a blocking signal. Ticket ROCMAI-541 is not retrievable from this environment (no tracker access granted) — judged against the PR's own description of the acceptance criteria instead. Reconciliation against standing review discussion was done only after blind findings were locked, per the method, and is reported in the narrative above rather than folded into any finding's confidence score.
🤖 Automated by an AI agent
| } | ||
| } else if latest.status == on_disk { | ||
| return; | ||
| return deliberate; |
There was a problem hiding this comment.
Blocking — settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash.
When the on-disk record is already non-live (managed_service_is_live(&latest) is false) and latest.status == on_disk, this returns deliberate without ever writing. refresh_managed_service_runtime_liveness_with (apps/rocm/src/managed_service_status.rs:408-416) reconciles any record with dead tracked pids to status = "stopped" with no marker — including a "starting" record mid---wait — by its own doc comment's admission ("dead pids alone do not distinguish a stopped service from a crashed one"). So if a concurrent rocm services/rocm dash read (or rocmd's own recovery tick) reconciles a dying --wait launch to "stopped" before this function runs, deliberate is false (correctly — no stop markers exist) and died_during_wait prints the correct crash message, but the persisted manifest is left reading "stopped" and is never corrected to "failed". This directly contradicts the PR's own acceptance criterion ("the engine state file is marked failed on any error exit") for this one path.
This matches a standing, currently-unresolved reviewer objection on this exact area (the --wait race) — no commit since that review has touched this function.
Suggested fix: in the else if latest.status == on_disk branch, correct status to "failed" (and merge the engine-state file) when !deliberate, rather than returning early without writing; only the "nothing to add, preserve an already-failed record's own reason" case needs the pure early return.
Confidence: 85 · logic · blocking.
| // crash and was stamped `failed` — the exact symptom ROCMAI-541 exists | ||
| // to remove, reintroduced through the one stop path that is not | ||
| // `rocm services stop`. | ||
| record.settled_stop_unix_ms = record.stop_requested_unix_ms.take(); |
There was a problem hiding this comment.
Blocking — unconditional marker move can erase a concurrent stop's already-correct evidence.
record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();This is unconditional, unlike the equivalent "moved, not dropped" logic everywhere else in this PR (apps/rocm/src/service_control.rs:283-284's if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }, and managed_service_status.rs's settle_pending_stop_key_cleanup, both guarded).
The surrounding comment (lines 95-99) explicitly acknowledges the record is re-read fresh here because another writer — "rocm's liveness refresh" — may have persisted something while the stop ran. Concretely: (1) this stop signals the pids and they die; (2) concurrently, a rocm services list/rocm dash read reconciles the dead pids and — correctly, via the guarded code elsewhere — moves stop_requested_unix_ms into settled_stop_unix_ms, writing None/Some(ts) to disk; (3) this function's re-read at line ~100 now sees stop_requested_unix_ms == None, settled_stop_unix_ms == Some(ts); (4) all_stopped is still true (computed from this stop's own pid-tracking, independent of the record read); (5) the unconditional .take() here assigns None over the already-correct settled_stop_unix_ms, erasing the one piece of durable evidence the whole PR relies on to prove this death was requested.
The in-code comment one line above this ("the same move rocm services stop makes on its confirmed path, and for the same reason") claims parity with the guarded version, but the guard itself is missing here — and this file/path is not named anywhere in the PR's own "shape of the fix" table or "the evidence a stop leaves" narrative, despite receiving the identical fix.
Suggested fix: guard this exactly like service_control.rs:283-284: if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }.
Confidence: 90 · logic · blocking.
| /// its *real* wait status rather than merely probing whether the pid still | ||
| /// exists. | ||
| /// | ||
| /// On Windows there is no equivalent gap: [`process_is_running`] there opens a |
There was a problem hiding this comment.
Blocking — this type's own doc comment (and from_pid's, line 79) is stale and self-contradicts the correct documentation 20 lines below it.
This comment and from_pid's say the Windows arm "defers to process_is_running" / "opens a fresh handle... reads the real exit code" with no further qualification. But ManagedChildLiveness::is_running's own doc comment (lines ~84-93) correctly and explicitly says the opposite: "The Windows arm asks probe_process rather than process_is_running... process_is_running collapses Unqueryable into 'not running'... Unqueryable therefore reads as 'may still be running'" — which is also what the actual code at the bottom of this file does (pid_child_is_running(*pid, probe_process)).
So within this single new file, two doc comments describe the old, less-safe process_is_running-based behavior as current, while the correct one — on the method that actually matters — documents the fixed behavior. A future maintainer reading only the type-level doc (the first thing anyone reads) would believe this type behaves the unsafe way ROCMAI-541 exists to move away from.
This matches a standing, currently-unresolved reviewer objection ("the managed_child.rs type doc is stale") — no commit since that review has touched this file.
Fix: update this comment and from_pid's (line 79) to describe the probe_process/Unqueryable-aware behavior, consistent with is_running's doc.
Confidence: 90 · mechanical (doc-accuracy) · blocking (this repo's own AGENTS.md §3 requires a doc/comment stating how the code behaves be checked against that code before pushing, and it demonstrably was not, within a single new file).
| )), | ||
| }; | ||
| if let Err(error) = &result { | ||
| record_engine_serve_http_failure(&service_id, &state_path, log_path.as_deref(), error); |
There was a problem hiding this comment.
Non-blocking — this new crash-recording call site is exercised by no test, and the test module's own comment pointing at "the PR body" for justification doesn't hold up.
The test module (lines ~624-635) says this arm's wiring is "still uncovered — see the PR body," because reaching it requires spawn_alongside_engine, which starts a background thread a unit test must not do. The e2e scripted-engine hook doesn't reach it either: it takes the early-return branch at lines 332-340, which is structurally parallel to but textually separate from this post-match call site.
I checked the PR's actual description: its "Tests" and "Known limitations" sections don't address this specific call site (they discuss readiness/restart/--wait being unit-tested only, and a different limitation about SIGTERM'd Lemonade/vLLM adapters) — so the comment's pointer to "the PR body" for this particular gap isn't actually substantiated there.
Per this repo's own AGENTS.md §3, an untestable path needs its gap stated in the PR text with a reason; the code comment attempts an argument (background-thread-in-a-unit-test) but the PR body itself doesn't carry it.
Confidence: 75 · logic (test-coverage gap, partially self-acknowledged) · non-blocking.
| ); | ||
| } | ||
|
|
||
| #[then("the service record does not read failed")] |
There was a problem hiding this comment.
Non-blocking — the external-stop control scenario's record-state assertion is negative (assert_ne!(status, "failed")) rather than positive (assert_eq!(status, "stopped")).
assert_ne!(record["status"], "failed", "manifest wrongly marked failed: {record}");
assert_ne!(engine_state["status"], "failed", "engine state wrongly marked failed: {engine_state}");This is the one scenario specifically designed to prove a deliberate stop is not misclassified as the crash serve-scripted-death-after-startup exercises. As written, it would still pass if a regression left the record in some other wrong status (e.g. stuck "starting") as long as it isn't literally "failed" — it doesn't confirm the record actually reads "stopped". This matches a standing, currently-unresolved reviewer objection on this exact scenario; no commit since that review has strengthened this assertion.
Suggested fix: assert record["status"] == "stopped" (and the engine state, if it tracks status) directly, in addition to or instead of the negative check.
Confidence: 85 · logic (test-quality) · non-blocking.
Main landed ROCMAI-703 (#1050), which adds its own serve scenario and its own testing.md section, so both conflicts are with it. tests/e2e-cucumber/features/model_serving.feature: main took `serve-60`. Main's keeps 60; this branch's two become `serve-61` and `serve-62` and move after it, so each number still equals its position. Four prose references follow: the two scenario entries and the repro filter in `docs/testing.md` (now `serve-(61|62)`), the `(serve-62)` note in `service_launch/readiness.rs`, and the feature file's own "exactly like serve-60's crash" line. The other `-n "serve-60"` filters in docs are main's own and are left alone. docs/testing.md: two independent sections appended at the same place, kept whole. Unnumbered headings, so nothing renumbers. This is the third time these two scenarios have moved (57/58 -> 59/60 -> 60/61 -> 61/62); the `@id:` tags are the binding key and have never changed, so no step binding moved with them. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Correction to my earlier note: the colliding number has moved.
As before, each is correct within its own branch (a scenario's number equals its position in its own file); the clash only exists across the two, so whichever merges first forces the other to renumber. Worth noting this is the third time these numbers have moved — #854's pair has gone 57/58 → 59/60 → 60/61 → 61/62 purely from I will renumber whichever branch lands second. |
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (25 files, +4105/-180, base 088faa43…head d5b4d66c). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees "server process exited""), expanded well beyond the ticket's Windows-only scope to the launch/restart/attach/--wait entry points on every platform; the ticket is still "In Progress" and the PR does not claim to close it standalone.
Blocking
-
settle_exited_servicecan leave a crashed service's manifest readingstoppedforever, even though it correctly reports the death as a crash —apps/rocm/src/service_launch/readiness.rs:153(theelse if latest.status == on_disk { return deliberate; }early return), interacting withapps/rocm/src/managed_service_status.rs:408-418's unconditional dead-pid→"stopped"reconciliation (no stop-marker check, by its own comment: "dead pids alone do not distinguish a stopped service from a crashed one"). When that reconciliation runs concurrently (e.g. arocm services/rocm dashlisting) before this function's own read,deliberateis correctlyfalsebut the manifest is never corrected to"failed"— contradicting this PR's own acceptance criterion. Traced directly against the current head.
Confidence 85 · logic · Fix: correct the status to"failed"in this branch when!deliberate, rather than returning early without writing. -
apps/rocmd/src/service.rs:137's unconditional marker move can erase a concurrent stop's already-correct durable evidence —record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();, unlike the guardedif let Some(requested) = ... { Some(requested) }pattern used for the identical operation inservice_control.rsandmanaged_service_status.rs'ssettle_pending_stop_key_cleanup. If a concurrent liveness refresh already moved the marker before this function's own re-read,.take()on the already-Nonefield assignsNoneover the just-readSome(X), erasing the one piece of durable evidence this whole PR relies on — reproducing the exact crash-indistinguishable-from-a-stop defect ROCMAI-541 exists to close, through the one stop path not named in the PR's own "shape of the fix" table.
Confidence 90 · logic · Fix: guard it the same way asservice_control.rs. -
crates/rocm-core/src/managed_child.rs's type-level doc comment (line 63) andfrom_pid's doc (line 79) are stale and self-contradictis_running's own doc 20 lines below — both say the Windows arm "defers toprocess_is_running," whileis_running()actually (and deliberately) usesprobe_processinstead, specifically to avoid the bugprocess_is_runningwould cause. Within one new file, the first doc a reader sees documents the old, unsafe behavior as current.
Confidence 85 · mechanical (doc-accuracy) · Fix: update both to matchis_running's doc.
Non-blocking
tests/e2e-cucumber/features/model_serving.feature:1011's comment misnames its own sibling scenario — says "arm serve-59 exercises," but the sibling defined two lines above in this same diff isserve-61;serve-59is an unrelated, pre-existing scenario. The PR description also calls these two new scenarios "serve-59"/"serve-60" throughout, whiledocs/testing.md(also new) correctly says serve-61/serve-62 — a renumbering this one comment and the description missed.
Confidence 85 · mechanical · Fix: s/serve-59/serve-61/, and fix the description.
Decisions for the author
- answered-by-intent — Whether a
failedrecord should be arocmdrecovery candidate at all is left open by design: "Whether rocmd recovery should respawn afailedrecord is still undecided;a_crash_verdict_record_is_a_recovery_candidatepins what it does today, not what it should do." Would otherwise be worth a Deployment note as an unreviewed behavior change; the PR already discloses it as a known limitation, so just confirm it's still the plan to resolve it in a follow-up.
Positive signals
- The "moved, not dropped"
settled_stop_unix_mspattern inservice_control.rsandmanaged_service_status.rsis correctly guarded and has dedicated regression tests for the exact concurrent-listing race it defends against (a_second_listing_does_not_clear_the_stop_a_first_one_settled) — which is what makes the missing guard inapps/rocmd/src/service.rsstand out as an inconsistency rather than an unknown risk. ManagedChildLiveness'sUnqueryable-vs-Gonedistinction is tested by driving all threeProcessProbeoutcomes on every platform, not only the one a real Windows CI run could construct.- The scripted-engine E2E hook is double-gated (compile-time
e2e-test-hooksfeature and a runtime env var) and backed byxtask'sthe_release_workflow_builds_no_feature_gated_binaries, which scans for any enabled feature rather than just this one. - The
rocm logs <id>→rocm services logs <id>remediation-hint fix is backed by a test (cli.rs) that actually parses both suggested commands and confirms the old buggy form resolves to a positional search query rather than a service selector.
Deployment notes
- Crashed launches/restarts/
--waitnow persiststatus: "failed"(previouslystoppedin some of these paths), making themrocmdrecovery candidates in cases that were not retried before — see "Decisions for the author" above. - A deliberate stop of a real Lemonade/vLLM-backed service can still show
failedon disk (the adapter's own SIGTERM handling writes it, read back viarefresh_from_engine_state), per the PR's own disclosed limitation.
What this covered
Full diff read (git diff origin/main...pr-854, base 088faa43…head d5b4d66c), plus full-source reads of the highest-risk files: managed_child.rs, process.rs, service_record.rs, readiness.rs, spawn.rs, attach.rs, service_control.rs (stop+restart paths), launch_transaction.rs, start.rs, managed_service_status.rs, apps/rocmd/{service,watchers/server_recover}.rs, engine_ipc.rs, and cli.rs's new test; wide-context diffs for the rest (service_launch.rs, docs, the e2e feature/steps files). engine_runtime_select.rs and serve_cmd.rs got a lighter, diff-only pass. Confirmed via git merge-tree that the current head merges cleanly against current origin/main despite several touched files (README.md, cli.rs, rocm-core/lib.rs, docs, tests/e2e-cucumber/tests/e2e.rs) having also changed independently on main since this PR's base.
Degraded fan-out. The method calls for splitting this review across independent workers; the environment's shared subagent-concurrency pool was saturated by concurrent sibling jobs for the entire review (confirmed via repeated "concurrent subagent limit reached" errors, including on retry), so this ran single-context instead. Recommend re-running with fan-out available for a second, independent pass.
CI on the reviewed commit is green on every completed, non-hardware-specific check (clippy, build-and-test, affected-crate tests, the E2E mock lane, Windows build/test, Windows release/lifecycle, docs build, security scans); Skill checks (skillscope) is explicitly skipped/advisory, and the MI300X/Strix Halo Windows hardware lanes are queued/cancelled rather than failing, consistent with self-hosted-runner availability gaps this branch's history discusses elsewhere — not treated as blocking signals.
Did not run: cargo build/test/clippy (no Rust toolchain in this environment) — all findings above are argued from direct reading and tracing of the source and tests, not execution. History/prior-changes passes were not run as dedicated separate passes.
Reconciliation, done after the blind pass: this PR carries a standing CHANGES_REQUESTED review from this same account at an earlier commit (e00fac0f, not the current head). git log confirms only a merge-from-main separates that commit from the current head — no new work responds to it. Three of that review's points (the settle_exited_service finding, the rocmd/service.rs unguarded marker move, and the managed_child.rs stale doc) were independently re-derived and re-verified by me directly against the current head's source before inclusion above, rather than taken on trust.
Ordering deviation, disclosed rather than left unstated: the PR title/description and mergeable-state were fetched together with CI status before the blind code-reading pass began. The findings above were derived and re-verified from the diff and source directly (the scenario-numbering finding, in particular, came from grepping the feature file itself); none rely on the description's framing.
🤖 Automated by an AI agent
| &serde_json::json!({ "status": "failed" }), | ||
| ); | ||
| } | ||
| } else if latest.status == on_disk { |
There was a problem hiding this comment.
Blocking — settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash.
Trace: refresh_managed_service_runtime_liveness_with (managed_service_status.rs:408-418) reconciles any record whose tracked pids are gone to status = "stopped", with no stop-marker check at all (its own comment: "dead pids alone do not distinguish a stopped service from a crashed one"). If that reconciliation runs concurrently — e.g. a rocm services/rocm dash listing — before this function's own read_manifest call, then here latest.status is already "stopped", deliberate is correctly computed as false (no marker was ever set), and on_disk == "stopped". managed_service_is_live(&latest) is false for "stopped" (is_live() only accepts ready/running/starting/recovering), so control falls to this else if latest.status == on_disk branch and returns false without writing anything.
died_during_wait's caller still prints the correct crash message to the console (since it uses the returned deliberate boolean), but the persisted manifest is left reading "stopped" — never corrected to "failed", contradicting this PR's own acceptance criterion for this exact path ("the engine state file is marked failed on any error exit"). rocm services list/rocm dash would show this service as cleanly stopped even though the CLI itself just told the user it crashed.
Confidence 85 · logic · Fix: in this branch, when !deliberate and the status is not already "failed", correct it to "failed" (and merge the engine-state file) instead of returning early without writing — the guard should protect against clobbering a newer, correct write, not against correcting a status this function's own verdict disagrees with.
| // crash and was stamped `failed` — the exact symptom ROCMAI-541 exists | ||
| // to remove, reintroduced through the one stop path that is not | ||
| // `rocm services stop`. | ||
| record.settled_stop_unix_ms = record.stop_requested_unix_ms.take(); |
There was a problem hiding this comment.
Blocking — this unconditional marker move can erase a concurrent stop's already-correct durable evidence.
Unlike the guarded if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); } used for the identical operation in apps/rocm/src/service_control.rs (stop_internal_managed_service_confirmed_with) and in managed_service_status.rs's settle_pending_stop_key_cleanup, this call site does it unconditionally: record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();.
record here is freshly re-read from disk a few lines above specifically because "other writers ... may have persisted something while the stop ran." If a concurrent liveness refresh already moved the marker (stop_requested_unix_ms → settled_stop_unix_ms) before that re-read, the fresh record already has stop_requested_unix_ms: None and settled_stop_unix_ms: Some(X). .take() on the already-None field returns None, and this line assigns that None straight over the Some(X) just read — erasing the one piece of durable evidence the whole PR relies on to prove this death was requested, and reproducing the exact crash-indistinguishable-from-a-stop state ROCMAI-541 exists to close, through the one stop path (rocmd's own) that still has the unguarded version. This file also isn't named in the PR description's "shape of the fix" table, even though it received the identical fix as service_control.rs.
Confidence 90 · logic · Fix: guard it the same way as service_control.rs's if let Some(requested) = ... pattern.
| Self::Unix(child) | ||
| } | ||
|
|
||
| /// Build from a bare pid on Windows, where [`process_is_running`] already |
There was a problem hiding this comment.
Blocking — stale, self-contradicting doc comments within this same new file.
This from_pid doc ("where [process_is_running] already reads the real exit code") and the type-level doc a few lines above (line 63: "...so the bare pid is enough, and this variant just defers to it [process_is_running]") both describe the Windows arm as deferring to process_is_running. But is_running()'s own doc and its actual implementation (Self::Windows(pid) => pid_child_is_running(*pid, probe_process)) explicitly do not use process_is_running — they use probe_process instead, specifically because process_is_running collapses ProcessProbe::Unqueryable into "not running," which would report a transient OpenProcess denial on a healthy engine as a crash (the exact bug this type exists to prevent).
So within one new file, the type-level doc and from_pid's doc describe the old, unsafe behavior as current, while is_running's doc (20-ish lines below) correctly describes what the code actually does. A reader who only reads the type doc first gets the wrong contract.
Confidence 85 · mechanical (doc-accuracy) · Fix: update this doc and the type-level doc (line ~60-63) to say the Windows arm uses probe_process (via pid_child_is_running), matching is_running's doc.
| # externally triggered stop in the same post-startup window must NOT be | ||
| # treated as the crash above. `rocm services stop` run from a second | ||
| # invocation kills the pid the attached `--verbose` session is watching, | ||
| # landing in the identical `AttachOutcome::ServerExited` arm serve-59 |
There was a problem hiding this comment.
Non-blocking — stale scenario-number cross-reference, introduced by this PR.
This comment (for the new serve-62 scenario) says the external-stop scenario lands "in the identical AttachOutcome::ServerExited arm serve-59 exercises." But the sibling scenario defined two lines above in this same diff is titled Scenario: serve-61 - A managed engine that dies after startup fails loudly — serve-59 is a different, pre-existing scenario in this file ("Only the user who started a key-protected server can read its key"). The PR description also calls these two new scenarios "serve-59"/"serve-60" throughout, while docs/testing.md (also added by this PR) correctly refers to them as serve-61/serve-62. Looks like a renumbering (likely from this long-lived branch catching up with scenarios added on main) that this one comment — and the PR description — missed.
Confidence 85 · mechanical · Fix: s/serve-59/serve-61/ here, and fix the PR description's references.
Review finding: `apps/rocmd/src/service.rs` moved the stop marker with a bare `record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();`, while the identical operation in `service_control.rs` and `managed_service_status.rs` is guarded. A concurrent listing's liveness refresh can settle the stop first; `.take()` then yields `None`, and assigning that over the `settled_stop_unix_ms` the refresh had just written correctly leaves both fields unset -- which reads as a crash. That is precisely the defect this change exists to close, still reachable through the one stop path its own table did not name. Also corrects two doc comments in `crates/rocm-core/src/managed_child.rs` that contradicted `is_running`'s doc twenty lines below: both said the Windows arm defers to `process_is_running`, when it deliberately asks `probe_process` instead so a process this one cannot open is not mistaken for one that exited. The first doc a reader met described the behaviour the type was built to avoid. And renumbers a feature-file comment that named `serve-59` as its own sibling; that sibling is `serve-61` after the merge, and `serve-59` is an unrelated pre-existing scenario. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Three blocking findings; two fixed in Blocking 2 — Blocking 3 — Non-blocking — the feature-file comment naming Blocking 1 — Your trace is right. What makes it more than a one-liner is that the early return you point at is itself load-bearing. So the fix you suggest — "correct the status to That is a change to the write discipline of the function this whole PR is built around, and I would rather it be a deliberate decision than something I slipped in alongside two unrelated fixes. Taking it next, as its own change with its own test. CI note: five Decision confirmed: whether a Leaving CHANGES_REQUESTED standing while blocking 1 is open. |
… to stopped `settle_exited_service` reported the death as a crash but could leave the manifest reading `stopped` forever. `managed_service_status`'s liveness reconciliation demotes any record whose recorded PIDs are gone to `stopped`, unconditionally and without knowing why they went, and a `rocm services` or `rocm dash` listing running alongside the wait is enough to do it. When that lands before this function's own read, `on_disk` is already `stopped`, the refresh from the engine state file (which a SIGKILL or OOM kill never got to write) learns nothing, and the `latest.status == on_disk` early return exits without writing — so `rocm services list` shows a clean stop for a death nothing asked for, which is this change set's own acceptance criterion failing. The early return is load-bearing and stays: it is what stops the stale snapshot being written back over whatever a concurrent writer put on disk meanwhile. The correction therefore goes through `mark_service_failed`'s read-modify-write merge, which replaces only the `status` key under the state file lock, rather than `latest.write()`, which would trade this defect for the one the guard exists to prevent. A record already reading `failed` carries the engine's own reason and is left entirely alone. Two tests, each falsifiable on its own: one drives the interleaving and asserts both halves (the manifest and the engine state file end `failed`, and a planted key the record type does not model survives), the other pins the already-`failed` guard through the engine state file a needless correction would invent. Mutating the correction away, dropping the `failed` term, and swapping the merge for a whole-file write each fail exactly one of them. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Two open items, pushed as 1. The five
|
| Mutation | Result |
|---|---|
| correction removed (restores today's behaviour) | a_crash_is_still_recorded_when_a_listing_already_reconciled_it_to_stopped FAILS |
&& latest.status != "failed" dropped |
a_record_that_already_failed_is_left_entirely_alone FAILS |
mark_service_failed replaced with latest.status = "failed"; latest.write() |
a_crash_is_still_recorded_when_a_listing_already_reconciled_it_to_stopped FAILS (planted key clobbered) |
a_wait_death_is_a_crash_unless_a_stop_asked_for_it passes unchanged, including its third arm.
Gates (Linux devbox)
cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo clippy --workspace --all-targets --features e2e-test-hooks -- -D warnings; cargo clippy -p e2e-cucumber --test e2e -- -D warnings; cargo test -p e2e-cucumber --test feature_naming; cargo xtask check-architecture-doc; check-crate-edges; check-module-size — all clean. cargo test -p rocm --bin rocm: 1946 passed, 0 failed. cargo test -p rocmd: 294 passed, 5 failed — the five above, which fail identically on origin/main on this host.
|
Retracting something I said on this PR two comments ago. I reported that five The tree I ran Checked properly, against current The structural evidence agrees, and I verified it independently:
So this is a property of the devbox — all five assertions are about a grandchild being reaped with its parent, which is the Worth saying plainly: "it passes on main" is only evidence if the tree you ran is actually current main. Mine was not, and I did not check before reporting it as a regression. |
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change. Title/description identify this as ROCMAI-541 (ticket content not retrieved, per instructions for this repo); the diff's substance matches the description's stated scope (crash-vs-stop disambiguation at every managed-child exit point, Windows stdio attach, the scripted-engine e2e hook) except for the scenario-number mismatch noted below.
Blocking
- PR description cites the wrong scenario IDs for its own new regression tests —
tests/e2e-cucumber/features/model_serving.feature:1001,1017
The description's "Tests" section, its "verified empirically ... turned serve-60 red" claim, and its mutation-check list all citeserve-59/serve-60for the two scenarios this PR adds. Those two numbers are already taken by pre-existing, unrelated scenarios ("Only the user who started a key-protected server can read its key" / "A launch whose first chat request fails is not presented as ready"). The scenarios this PR actually adds areserve-61(@id:serve-scripted-death-after-startup) andserve-62(@id:serve-scripted-external-stop-while-attached). The work itself is correctly implemented and tested — this is a documentation accuracy issue in the PR's own account of what it verified, not a code defect — but a reader trying to verify the "removing the status term turned serve-60 red" claim would land on the wrong, unrelated scenario.
Confidence 90 · documentation-accuracy · Fix: update the description's scenario-number references to serve-61/serve-62.
Non-blocking
StopWatch's stop-detection narrowing is applied to two of three sibling wait paths —apps/rocm/src/service_launch/readiness.rs:624(StopWatch, new),apps/rocm/src/service_launch/readiness.rs:406(wait_on_existing_service, unchanged),apps/rocm/src/service_control.rs:947(record_restart_readiness)
start.rs's fresh-launch wait andservice_control.rs's restart wait both sample aStopWatchacross the loop to narrow the "stop requested+confirmed within one poll, then raced byreset_for_restart" residual gap the type's own doc comment names.wait_on_existing_service(reached whenrocm serve --waitre-attaches to an already-live/starting service) has no such sampling, relying only on the one-shot durable check — which covers the common case this PR fixes, just without the extra narrowing its siblings get. Separately,record_restart_readiness's ownstop_seen_while_waiting=truebranch has no dedicated test, unlike its siblingfinalize_managed_launch_readiness_trusts_the_latch_after_the_marker_is_cleared. Both are narrow, already-accepted-elsewhere categories of risk (the authors' own comments describe the underlying residual gap as deliberately accepted), not a clear regression.
Confidence 70 · architectural · Fix: addStopWatchtowait_on_existing_servicefor consistency (or document why it's excluded), and add a test driving the restart path's latch to true.- Stale comment in a test, contradicted by its own assertion —
apps/rocm/src/service_launch/spawn.rs:1453(assertion), ~:1337(stale comment, unchanged by this diff)
The comment right above thepersisted.status, "failed"assertion was correctly updated to explain the new outcome, but the test function's leading summary comment, a few lines after#[test], still says "...mark the recordstopped(afailedone would be respawned by rocmd recovery)...". The two comments in the same test now disagree.
Confidence 85 · mechanical · Fix: update the leading comment to match.
Decisions for the author
docs/architecture.mdentries forreadiness.rs,spawn.rs,service_control.rs,engine_ipc.rsandattach.rsstill describe their pre-PR contents — answered-by-intent
PR body, "Known limitations": "docs/architecture.mdgainedmanaged_child.rs, but thereadiness.rs,process.rsandengine_ipc.rsentries still describe their older contents." Would otherwise have been a blocking architectural/doc-accuracy finding (these entries omit the new crash-vs-stop machinery that is now most of what each file does); the PR explicitly discloses the gap for three of the five files in question.service_control.rsandattach.rs's entries are similarly stale and not named in that limitations note — worth folding into the same follow-up.
Positive signals
ManagedChildLiveness(crates/rocm-core/src/managed_child.rs) is a clean, well-reasoned split: Unix zombie-reap via retainedChild/try_wait(), Windows viaprobe_processtreatingUnqueryableas "may still be running" rather than "gone" — with a unit test specifically pinned to the one case (Unqueryable) that discriminates it from the oldprocess_is_runningcheck.- The
settled_stop_unix_ms"move, don't drop" redesign replacing the unsoundstatus == "stopped"evidence is sound and consistently wired acrossservice_record.rs/managed_service_status.rs/service_control.rs; several tests pin the exact mutation that would reopen the original bug. serve-61/serve-62give genuine, non-tautological regression coverage: exit code, user-visible message, log content, manifest status and engine-state status are all asserted together, so no single narrow fix can pass them by accident.- The
e2e-test-hooks/ROCM_E2E_SCRIPTED_ENGINEgating is layered consistently (feature-gated function definition plus feature-gated call site) with no reachable leak into a build without the feature. - Windows gets real, new crash-detection coverage here (
spawn_detached_with_log,settle_restart_startup_death) rather than a paper parity claim — previously-absent paths, not just mirrors of existing Unix ones.
Deployment notes
- Behavior change: a managed
rocm serve/services restart/--waitthat crashes now exits non-zero and leaves the recordfailed, where it previously exited 0 with "server process exited". Any script or integration relying on the old exit-0-on-crash behavior needs to adjust. - Remediation hint text changed from
rocm logs <id>torocm services logs <id>in multiple places; verified no stale form of the old hint survives anywhere in the tree. - Author-disclosed, accepted limitation: a deliberate stop of a real Lemonade/vLLM-backed service can still show
failedbecause those adapters treat SIGTERM as an unsuccessful exit and writefailedthemselves; not covered by any automated test, only by a manual-testing doc section.
What this covered
Full review of the whole change: 25 files, merge-base 088faa43…head 4c70975f (diff id ba25ed19a668). Every changed file read completely, split across independent blind passes grouped by coupling (readiness.rs+spawn.rs; attach.rs+service_control.rs+launch_transaction.rs+start.rs+service_launch.rs; managed_service_status.rs+service_record.rs+process.rs+managed_child.rs+lib.rs; engine_ipc.rs+serve_cmd.rs+cli.rs+engine_runtime_select.rs; rocmd/service.rs+watchers/server_recover.rs; the three e2e test files), plus a dedicated agent-instruction-adherence/code-comment pass and a history/prior-commits pass. All passes ran blind (no PR description or review threads). CI checked via get_check_runs at the reviewed head: every required check (clippy — which folds in module-size/architecture-doc-presence/crate-edges — build-and-test on both platforms, Windows debug/release, E2E consolidated and mock lanes, security scans, docs build) is green; CodeQL reports neutral (no alerts), one Skill checks (skillscope) job is skipped (not applicable).
Did not run: no cargo binary was available to any pass, so module-size/architecture-doc-presence claims were cross-checked by manual line counts and docs/architecture.md inspection rather than by running the actual xtask checks — relying on CI's green clippy job for the automated verdict on those two.
Not reconciled against this PR's many prior review rounds' threads: each candidate finding above was independently re-derived and verified against the current head rather than taken from history, per this review's blind-then-reconcile methodology.
🤖 Automated by an AI agent
| # Before ROCMAI-541 this made a clean, intentional, externally-triggered | ||
| # stop fail exactly like serve-61's crash. | ||
| @id:serve-scripted-external-stop-while-attached @requires-no-gpu | ||
| Scenario: serve-62 - An externally stopped attached serve exits cleanly, not as a crash |
There was a problem hiding this comment.
This scenario is serve-62, not serve-60 as the PR description says in several places (its "Tests" section, the "verified empirically ... turned serve-60 red" claim, and the mutation-check list). serve-59/serve-60 are both pre-existing, unrelated scenarios already in this file ("Only the user who started a key-protected server can read its key" / "A launch whose first chat request fails is not presented as ready") — the new ones this PR adds are serve-61 (line 1001, @id:serve-scripted-death-after-startup) and serve-62 (here, @id:serve-scripted-external-stop-while-attached). Worth fixing the description's scenario numbers so the "verified empirically" claim points at the scenario it actually describes.
| assert_eq!( | ||
| persisted.status, "stopped", | ||
| "a post-spawn failure must mark the record abandoned, not leave it stuck" | ||
| persisted.status, "failed", |
There was a problem hiding this comment.
Non-blocking: the comment right above this assertion was correctly updated to explain the new "failed" outcome, but the leading comment at the top of this same test function (a few lines after the #[test] attribute, ~line 1337: "...mark the record stopped (a failed one would be respawned by rocmd recovery)...") is unchanged by this diff and now contradicts both this assertion and the comment right above it. Worth updating that top-of-function summary too so the two comments in the same test don't disagree.
| /// | ||
| /// `stop_requested_unix_ms` is a stop's *live* evidence (see | ||
| /// [`managed_service_stop_was_requested`] for why the record's status is no | ||
| /// evidence at all), but it is deliberately short-lived: |
There was a problem hiding this comment.
Non-blocking/design question: StopWatch is sampled across the wait loop in the fresh-launch path (start.rs) and the restart path (service_control.rs's record_restart_readiness), narrowing the "stop requested+confirmed inside one poll interval, then raced by reset_for_restart" gap this type's doc comment describes. wait_on_existing_service (above, unchanged by this diff) has no StopWatch of its own — it relies solely on the one-shot died_during_wait/settle_exited_service check at the end, which this PR's fix to the durable settled_stop_unix_ms evidence already covers for the common case, but without the extra narrowing its two siblings get. Was leaving this third wait path without a StopWatch deliberate (e.g. its exposure window is considered negligible), or worth adding for consistency? Separately, record_restart_readiness's stop_seen_while_waiting=true branch doesn't appear to have a dedicated test, unlike finalize_managed_launch_readiness's equivalent (finalize_managed_launch_readiness_trusts_the_latch_after_the_marker_is_cleared).
…it leaves The comment above the assertion was updated when the outcome changed to `failed`; the test's own leading summary a hundred lines earlier still said the record is marked `stopped` because `a failed one would be respawned by rocmd recovery`. It now carries the same account, including that whether a `failed` record should be a recovery candidate is the open question the PR's Known limitations records -- the assertion pins today's behaviour, not the desired one. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Blocking — the description citing the wrong scenario IDs — fixed. You are right, and right that it is the kind of error that wastes a reader's time in the worst way: someone checking the "removing the status term turned serve-60 red" claim would have landed on an unrelated pre-existing scenario and concluded the claim was false. The body now says Non-blocking — the test's two contradicting comments — fixed ( Non-blocking — Decision — the stale So: the blocking item is closed and one non-blocking is fixed. Two remain open by choice, both stated above with reasons rather than left silent. |
ROCMAI-541: a managed
rocm servethat crashes must say soA managed engine that died reported success.
rocm serve --verbose/--foregroundexited 0 with "server process exited", and the engine state file kept itsstarting/0 defaults — sorocm services,rocm dashand the log all agreed nothing had gone wrong. Acceptance criterion: the engine state file is markedfailedon any error exit.Body regenerated from the final diff at @juhovainio's request. The previous version still listed a limitation this PR now fixes, and omitted several user-visible changes.
The shape of the fix
Every exit point where a managed child can die owes the same verdict: was this death asked for? A stop that recorded itself stays a stop; anything else stamps
failedon the manifest and the engine state file, and exits non-zero quoting the engine's own last words.finalize_managed_launch_readinessrecord_restart_readiness/update_restarting_recordServerExitedarm (attach.rs)try_wait()branch (spawn.rs)ensure_spawned_child_survives_startup's call sitesettle_restart_startup_death(service_control.rs)rocm serve --waitsettle_exited_service(readiness.rs)The last three were added in review.
The evidence a stop leaves — previously listed as a known limitation, now fixed
managed_service_stop_was_requestedused to acceptstatus == "stopped"as proof a death was deliberate. It never was:refresh_managed_service_runtime_livenessreconciles any record whose tracked pids are gone to that status, saying so in its own comment. A crash reconciled by a concurrentrocm services/rocm dash/rocmdread was therefore indistinguishable from a stop — exit 0, "server process exited", the original symptom inside its own fix.Verified empirically before changing anything: removing the status term turned serve-62 (
@id:serve-scripted-external-stop-while-attached) red, which also proved serve-62 was passing on the unsound term and could not tell the two apart.Fixed by making the stop's own evidence durable.
stop_internal_managed_service_confirmed_withnow moves its marker intosettled_stop_unix_msinstead of dropping it. That field already existed for the sibling case, is written only by a stop, and is cleared byreset_for_restart, so a stop settled before a relaunch cannot mask a crash after it. The record's status is no longer consulted. serve-62 now passes with the status term gone, and goes red if the confirmed stop drops its evidence.User-visible changes
rocm serve --verbose/--foregroundexits non-zero on a crash, where it exited 0 with "server process exited".--waitdeath is now recordedfailed(it wasstopped), and a stop during the wait gets its own message — "was stopped before it became ready" — instead of being reported as the launch failing to come up.rocm services restartalso fail loudly on a crashed engine.failed, and are picked up by rocmd recovery.rocm logs <id>(which never selected a service — the positional argument is a search query) torocm services logs <id>.Also in this diff
spawn_detached_with_logattaches the child's stdout/stderr, sorocm serve --verboseandrocm services logscan show why a Windows engine died — the diagnosis this ticket is about was structurally unavailable on the platform it was filed for. It forces handle inheritance, matchingspawn_hidden_console_with_log; unverified on Windows.ManagedChildLiveness(split intomanaged_child.rsto stay under the module-size gate) andspawn_detached_with_log.ManagedChildLiveness::Windowsasksprobe_processrather thanprocess_is_running, so a transientOpenProcessfailure reads as "may still be running" instead of a crash.engine_ipc.rs) and itsserve_cmd.rsbypass, both#[cfg(feature = "e2e-test-hooks")]/env-gated.xtask'sthe_release_workflow_builds_no_feature_gated_binariesenforces that it cannot reach a shipped build.wait_for_service_http_readyremoved; the_with_progressvariant is the only readiness wait.spawn.rsfield-merges instead ofrecord.write()on the early-exit path, so a concurrent stop'sstop_requested_unix_msis not erased.Tests
serve-61 (
@id:serve-scripted-death-after-startup) and serve-62 (@id:serve-scripted-external-stop-while-attached) cover the attached--verbosepath only — a real managed child driven to a crash, and the external-stop control. Launch readiness, restart,--waitand the new hint text are unit-tested only.Mutation-checked, each turning its test red: restoring
--wait's unconditional"stopped"; droppingmark_service_failedfrom the spawn verdict; deleting the marker check from the restart verdict; unwiring the policy-error stamp; and making a confirmed stop drop its evidence (serve-62).Known limitations
failedrecord — not from this check, but because the Lemonade and vLLM adapters read a SIGTERM'd server as an unsuccessful exit and writefailedthemselves, whichrefresh_from_engine_statereads back. No automated test exercises a real-adapter stop;docs/manual-testing.md4e tells the tester how to tell the two apart.windows-debugis the only check on the Windows paths, which this PR adds to.ManagedChildLiveness— pre-existing, deferred to the reconciliation with fix(serve): watch a Windows engine through its handle and keep a concurrent stop (ROCMAI-675) #437's held-handle design.merge_json_state_fileis unsynchronized against the stop path, so the field-subset write narrows the lost-update window rather than closing it.failedrecord is still undecided;a_crash_verdict_record_is_a_recovery_candidatepins what it does today, not what it should do.docs/architecture.mdgainedmanaged_child.rs, but thereadiness.rs,process.rsandengine_ipc.rsentries still describe their older contents.Verification
Linux devbox (this Mac cannot build
rocm-engine-lemonade): clippy workspace, the same with--features e2e-test-hooks, clippy on the e2e test target,cargo fmt --all --check,cargo test -p rocm --bin rocm(1868),cargo test -p rocm-core --lib(1497),feature_naming,check-crate-edges/check-module-size/check-architecture-doc, andcargo xtask e2e -- -n '(serve-61|serve-62)'— 2 scenarios, 9 steps,0 unexpected failure(s).