Skip to content

fix(serve): log and fail managed-serve crashes instead of exiting 0 (ROCMAI-541) - #854

Open
r0x0r wants to merge 47 commits into
mainfrom
fix/rocmai-541-managed-serve-logging
Open

r0x0r wants to merge 47 commits into
mainfrom
fix/rocmai-541-managed-serve-logging

Conversation

@r0x0r

@r0x0r r0x0r commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

ROCMAI-541: a managed rocm serve that crashes must say so

A managed engine that died reported success. rocm serve --verbose/--foreground exited 0 with "server process exited", and the engine state file kept its starting/0 defaults — so rocm services, rocm dash and the log all agreed nothing had gone wrong. Acceptance criterion: the engine state file is marked failed on any error exit.

Body regenerated from the final diff at @juhovainio's request. The previous version still listed a limitation this PR now fixes, and omitted several user-visible changes.

The shape of the fix

Every exit point where a managed child can die owes the same verdict: was this death asked for? A stop that recorded itself stays a stop; anything else stamps failed on the manifest and the engine state file, and exits non-zero quoting the engine's own last words.

Path Where
Launch readiness wait finalize_managed_launch_readiness
Restart readiness record_restart_readiness / update_restarting_record
Attached session the ServerExited arm (attach.rs)
Unix post-spawn early exit the try_wait() branch (spawn.rs)
Windows post-spawn early exit ensure_spawned_child_survives_startup's call site
Windows restart early exit settle_restart_startup_death (service_control.rs)
rocm serve --wait settle_exited_service (readiness.rs)

The last three were added in review.

The evidence a stop leaves — previously listed as a known limitation, now fixed

managed_service_stop_was_requested used to accept status == "stopped" as proof a death was deliberate. It never was: refresh_managed_service_runtime_liveness reconciles any record whose tracked pids are gone to that status, saying so in its own comment. A crash reconciled by a concurrent rocm services / rocm dash / rocmd read was therefore indistinguishable from a stop — exit 0, "server process exited", the original symptom inside its own fix.

Verified empirically before changing anything: removing the status term turned serve-62 (@id:serve-scripted-external-stop-while-attached) red, which also proved serve-62 was passing on the unsound term and could not tell the two apart.

Fixed by making the stop's own evidence durable. stop_internal_managed_service_confirmed_with now moves its marker into settled_stop_unix_ms instead of dropping it. That field already existed for the sibling case, is written only by a stop, and is cleared by reset_for_restart, so a stop settled before a relaunch cannot mask a crash after it. The record's status is no longer consulted. serve-62 now passes with the status term gone, and goes red if the confirmed stop drops its evidence.

User-visible changes

  • rocm serve --verbose/--foreground exits non-zero on a crash, where it exited 0 with "server process exited".
  • A --wait death is now recorded failed (it was stopped), and a stop during the wait gets its own message — "was stopped before it became ready" — instead of being reported as the launch failing to come up.
  • Background launch and rocm services restart also fail loudly on a crashed engine.
  • Crashed records become failed, and are picked up by rocmd recovery.
  • Remediation hints changed from rocm logs <id> (which never selected a service — the positional argument is a search query) to rocm services logs <id>.

Also in this diff

  • Windows got the service log. spawn_detached_with_log attaches the child's stdout/stderr, so rocm serve --verbose and rocm services logs can show why a Windows engine died — the diagnosis this ticket is about was structurally unavailable on the platform it was filed for. It forces handle inheritance, matching spawn_hidden_console_with_log; unverified on Windows.
  • New rocm-core public API: ManagedChildLiveness (split into managed_child.rs to stay under the module-size gate) and spawn_detached_with_log.
  • ManagedChildLiveness::Windows asks probe_process rather than process_is_running, so a transient OpenProcess failure reads as "may still be running" instead of a crash.
  • Scripted-engine E2E hook (engine_ipc.rs) and its serve_cmd.rs bypass, both #[cfg(feature = "e2e-test-hooks")]/env-gated. xtask's the_release_workflow_builds_no_feature_gated_binaries enforces that it cannot reach a shipped build.
  • wait_for_service_http_ready removed; the _with_progress variant is the only readiness wait.
  • spawn.rs field-merges instead of record.write() on the early-exit path, so a concurrent stop's stop_requested_unix_ms is not erased.

Tests

serve-61 (@id:serve-scripted-death-after-startup) and serve-62 (@id:serve-scripted-external-stop-while-attached) cover the attached --verbose path only — a real managed child driven to a crash, and the external-stop control. Launch readiness, restart, --wait and the new hint text are unit-tested only.

Mutation-checked, each turning its test red: restoring --wait's unconditional "stopped"; dropping mark_service_failed from the spawn verdict; deleting the marker check from the restart verdict; unwiring the policy-error stamp; and making a confirmed stop drop its evidence (serve-62).

Known limitations

  • A deliberate stop can still leave a failed record — not from this check, but because the Lemonade and vLLM adapters read a SIGTERM'd server as an unsuccessful exit and write failed themselves, which refresh_from_engine_state reads back. No automated test exercises a real-adapter stop; docs/manual-testing.md 4e tells the tester how to tell the two apart.
  • Windows is unverified by me. Nothing available can build the MSVC target; CI's windows-debug is the only check on the Windows paths, which this PR adds to.
  • Windows PID reuse on ManagedChildLiveness — pre-existing, deferred to the reconciliation with fix(serve): watch a Windows engine through its handle and keep a concurrent stop (ROCMAI-675) #437's held-handle design.
  • merge_json_state_file is unsynchronized against the stop path, so the field-subset write narrows the lost-update window rather than closing it.
  • No systematic entry-point parity test. Three of the gaps above were the same shape and each was found by review rather than a test.
  • Whether rocmd recovery should respawn a failed record is still undecided; a_crash_verdict_record_is_a_recovery_candidate pins what it does today, not what it should do.
  • docs/architecture.md gained managed_child.rs, but the readiness.rs, process.rs and engine_ipc.rs entries still describe their older contents.

Verification

Linux devbox (this Mac cannot build rocm-engine-lemonade): clippy workspace, the same with --features e2e-test-hooks, clippy on the e2e test target, cargo fmt --all --check, cargo test -p rocm --bin rocm (1868), cargo test -p rocm-core --lib (1497), feature_naming, check-crate-edges / check-module-size / check-architecture-doc, and cargo xtask e2e -- -n '(serve-61|serve-62)' — 2 scenarios, 9 steps, 0 unexpected failure(s).

r0x0r added 7 commits October 15, 2026 11:45
On Windows, spawn_managed_engine_child launched the detached
__engine-serve-http child with spawn_detached_no_inherit, which leaves its
stdout/stderr unattached to anything. Linux attaches them to the service log
via attach_background_stdio, so a crash's reason lands there; on Windows it
reached nothing. A dying engine therefore left the log silent past its last
explicit progress line, and both the managed and --verbose/--foreground serve
paths only ever saw the child's pid disappear, which they treated as a benign
stop: "server process exited" (or an unreadable "starting" status), exit 0,
and the engine's own state file stuck at whatever it last wrote.

Fix the Windows spawn (both the launch and the restart path) to attach the
service log via the existing spawn_hidden_console_with_log primitive, mirroring
Unix. As a platform-generic backstop — covering a gap in the attach, and any
engine code path that returns an error without updating its own state file —
run_builtin_engine_serve_http now logs the full error chain to the service log
and this process's own structured log, and marks the engine state file
`failed`, on any error exit.

Neither platform's rocm serve actually inspected the child's fate after
spawning it, though: a readiness wait that times out is indistinguishable from
one that will never succeed because the child already died, and the attached
log-streaming loop treated every self-exit as a graceful stop. Add
finalize_managed_launch_readiness (background path) and an equivalent check in
run_attached_service's ServerExited arm: both now confirm the child is still
alive before accepting a non-Serving readiness result, and fail the launch
with a log-tail-carrying error (and a failed engine state file) when it is not.
A readiness timeout on a still-alive child keeps reporting its previous status
with exit 0 unchanged, since a large model can legitimately take longer than
the readiness budget.

pid_is_running is injected into finalize_managed_launch_readiness so the
dead-child branch is unit-testable without spawning and killing a real
process, the same seam autostart_spawn_in_flight already uses nearby for an
identical reason. New tests cover both liveness outcomes plus the untouched
Serving-trusts-itself case, and assert the failure message and the resulting
process exit code together, reading back the on-disk manifest and engine
state file to confirm both land on `failed`.

Not covered by a new e2e scenario: exercising this live (rather than via the
injected-liveness unit tests) needs a real __engine-serve-http spawn that
survives past both platforms' early-startup liveness checks before dying,
without GPU hardware. The one close precedent in this codebase (a scripted
engine death at startup) needed its own preflight- and engine-preparation-
bypass plumbing built up over several review rounds; replicating an equivalent
for a later-than-startup death is a separate, larger effort left to a
follow-up rather than folded into this fix.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…(ROCMAI-541)

Round-2 review fixes on top of 5b4fe55:

B1 — the Unix liveness check was inert: the managed engine is a direct
child (setsid only, no double-fork), and process_is_running's kill(pid,
0) probe succeeds for a zombie this process has not reaped yet. A
dead-but-unreaped child was therefore still reported "running" on
Linux, reproducing the original symptom despite the round-1 fix.
Added rocm_core::ManagedChildLiveness, which retains the spawned
std::process::Child on Unix and calls try_wait() (reaping it) instead
of signal-probing; Windows keeps using process_is_running, which
already reads the real exit code via OpenProcess/GetExitCodeProcess
and needs no parent-side reaping. Threaded through
finalize_managed_launch_readiness (launch path), run_attached_service
and stream_attached_logs(_no_tty) (attached path), and the restart
path's early-death check and (new) record_restart_readiness liveness
gate.

B2 — AttachOutcome::Stop only ever fires from this attached session's
own keystroke reader; a stop triggered by `rocm services stop` from
another shell, `rocm dash`, or rocmd's restart-recovery kills the
watched pid directly, landing in ServerExited instead — which the
round-1 fix treated unconditionally as a crash. This made every
piped/CI `serve --verbose` (no keystroke capture at all) fail on an
ordinary stop. ServerExited now checks the manifest's
stop_requested_unix_ms marker (set before any such stop signals
anything) and status=="stopped" before concluding a crash; when either
is present it keeps the old benign-stop behavior (exit 0, no `failed`
stamp). Fixing this also surfaced a related gap: the crash branch only
marked the engine state file `failed`, never the manifest itself: now
does both.

B3 — added e2e coverage per AGENTS.md's Gherkin requirement for exit
code / output changes: serve-25/26 in model_serving.feature exercise a
real managed child dying after startup, and a real external stop in
the same window, via a new scripted-engine test hook
(ROCM_E2E_SCRIPTED_ENGINE[_DEATH_MS], e2e-test-hooks-gated) that stands
in for the real engine dispatch and needs no GPU. Confirmed serve-25
hangs (the exact ROCMAI-541 symptom) when the B1 fix is reverted.

N1 — removed the vacuous `exit_code_for(...) == ExitCode::FAILURE`
assertion from finalize_managed_launch_readiness's dead-child test: it
could not fail regardless of correctness (every non-FixExitCode/
ClapExitCode error maps to FAILURE there), and the test never goes
through the real run()/exit_code_for call chain it implied.

N2 — extended the same liveness fix to the restart path:
record_restart_readiness now takes a liveness check and fails closed
(marking both manifest and engine state `failed`) instead of
recording `starting` forever when a restarted engine dies during its
own readiness wait; the Unix early-death branch now also marks the
engine state file, matching the launch path.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…(ROCMAI-541)

Round-3 review fixes on top of 0df6166:

B1 — managed_service_stop_was_requested was only ever consulted on the
attached (--verbose/--foreground) ServerExited path. The background
launch path's own readiness-wait failure branch
(finalize_managed_launch_readiness) and the restart path's
(record_restart_readiness) both still treated a dead-and-not-Serving
child as a crash unconditionally, so a `rocm services stop`/`rocm
dash` stop landing during either's 45s readiness wait still got
stamped `failed` on both files and a non-zero exit, exactly like a
crash. Both now check the stop marker first and return cleanly
(exit 0, no `failed` stamp) when it is set, mirroring the attach path.
finalize_managed_launch_readiness now also refreshes its record from
disk in that case and reports to its caller (via a new `bool` return)
whether to skip its own follow-up bookkeeping, so that caller's
subsequent write can no longer clobber a stop marker written since
the record was captured at spawn time. Fixed the same class of bug at
its root in the crash branch too: it was writing `record.write()`
(the whole, stale, spawn-time snapshot) instead of merging just the
`failed` field, which could just as easily have erased a marker
concurrently written by the stop itself.

B2 — the Unix branch of spawn_managed_engine_child (an engine that
dies before the readiness wait even starts) never touched the
manifest or the engine state file on its bail-out, leaving both at
their just-created `starting`/pid-0 defaults forever — this is
ROCMAI-541's original, literal symptom, on the one code path round 1
and round 2 had not actually reached despite both rounds' commit
messages implying otherwise (they fixed the restart path's equivalent
branch). Now marks both `failed` via merge, same as every other
failure branch in this file.

N1 — corrected two comments that mischaracterized rocmd's
restart-recovery watcher as killing the engine it supervises directly;
per apps/rocmd/src/watchers.rs, it only ever fires once the service is
already dead/unreachable on its own and reacts by clearing
stop_requested_unix_ms and re-exec'ing a fresh supervisor, never
signalling the old engine.

N2 — guarded run_attached_service's ServerExited `failed` manifest
merge on the manifest still naming the exact engine pid (+ start-time
token) this session spawned and watched, so a reaped child's stale
write can no longer clobber a manifest a successful restart has since
legitimately replaced.

N3 — added a unit test for ManagedChildLiveness's Unix arm: spawns a
real short-lived child, polls until is_running() reads false, and
asserts a second call still reads false (pins the no-double-reap
contract the type exists for).

N4 — dropped finalize_managed_launch_readiness's vestigial child_pid
parameter; its only caller already ignored it in favor of the
liveness closure.

N5 — serve-25 now captures its service_id from the CLI's own printed
output at launch time instead of re-deriving "the newest record" in
its Then step; serve-26's attach-wait loop now calls
assert_scripted_hook_was_reached on timeout, so a binary built without
the e2e-test-hooks feature fails with a clear diagnostic instead of
a generic 30s timeout panic.

Left as-is per this round's explicit scope: the Windows
ManagedChildLiveness PID-reuse race (a known fork against unmerged PR
#437) and the serve-25/26 scenario id collision with that PR's own
serve-25 — both cross-PR sequencing calls for a human, not this round.

Verified on the MI300X box: fmt --check, build and clippy -D warnings
(both with and without --features rocm/e2e-test-hooks), full workspace
test suite, and cargo xtask e2e -n 'serve-2[2-6]' (serve-22/25/26
pass; serve-23/24 fail on pre-existing environment gaps unrelated to
this change — missing libdrm_amdgpu.so.1 / a vLLM import error in this
sandbox).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…, not engine identity (ROCMAI-541)

Round-4 review fix on top of 7ad214c:

Blocking — round 3's N2 guard compared the manifest's engine_pid/
engine_start_ticks against run_attached_service's watched child_pid
before stamping failed on a confirmed crash. refresh_from_engine_state
overwrites engine_pid with whatever the engine itself reports as its
server_pid, which for an engine like Lemonade that spawns its own
llama-server child is *that* child's pid, not the __engine-serve-http
process this loop actually watches — so for the single most important
case this whole ticket is about (a Lemonade-backed server that was
running and then died), the comparison was false and the manifest
never got stamped failed, silently undoing the earlier fix for that
case. It also didn't stop the stale-write problem it was built for:
the engine-state-file write right below it was unconditionally
unguarded, and refresh_from_engine_state would propagate that status
straight back into a newer manifest anyway. serve-25 didn't catch this
because the scripted test engine never writes a server_pid, so
engine_pid == child_pid happened to hold there by coincidence, not by
exercising the real-engine case.

Fixed by guarding on supervisor_pid/supervisor_start_ticks instead —
recorded for this exact child at spawn and never touched by
refresh_from_engine_state (confirmed by reading it end to end: it only
ever adopts server_pid/pid into engine_pid/engine_start_ticks), but
rewritten by every legitimate restart path via
record_supervisor_identity. Applied the same guard to BOTH the
manifest write and the engine-state-file write, where previously only
the manifest had one.

N1 — spawn_managed_engine_child's and start_managed_service's "normal"
(non-dead-child) record.write() calls are now merges of only the
fields each site actually owns (status/supervisor+engine identity at
spawn; status/inference_verified_at_unix_ms at readiness) instead of a
whole-file write of the in-memory spawn-time snapshot, closing the gap
round 3's commit message overstated as "fixed at its root": that round
only fixed the dead-child branches, not these.

N2 — corrected a doc comment citing the old serve-scripted-death/
serve-scripted-stop scenario ids; the real ones are
serve-scripted-death-after-startup and
serve-scripted-external-stop-while-attached.

N3 — serve-26's two remaining steps (another_shell_stops_scripted_service,
assert_scripted_stop_not_marked_failed) now read off the service_id
captured the moment user_attaches_scripted_serve_in_background first
observes the record running, instead of re-scanning the services dir
for "the newest record" afterward. Removed the now-unused
newest-record scan helper.

N4 — checked: server-recover defaults to disabled per scenario (fresh
isolated config.json each run, and WatcherUserConfig::enabled is only
true when a scenario explicitly opts in via `rocm automations enable`),
so serve-25's manifest-failed assertion isn't exposed to that race in
this harness. No change needed.

Verified on the MI300X box: fmt --check, build and clippy -D warnings
(both with and without --features rocm/e2e-test-hooks), full workspace
test suite (one unrelated pre-existing flake in
cli_progress::tests::animated_spinner_keeps_ticking_without_progress_calls
under parallel load, passes in isolation), and cargo xtask e2e -n
'serve-2[2-6]' (serve-22/25/26 pass; serve-23/24 fail on the same
pre-existing environment gap as before — missing libdrm_amdgpu.so.1 —
unrelated to this change).

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
The branch predates the main.rs modularization, so its changes had to be
relocated rather than merged in place:

- `restart_internal_managed_service_with`, `record_restarted_child` and
  `record_restart_readiness` (plus their tests) moved to `services_text.rs`;
  the Windows log attach, the `ManagedChildLiveness` threading and the
  restart-path liveness gate are reapplied there.
- `run_builtin_engine_serve_http` moved to `engine_ipc.rs`; the failure
  capture (`record_engine_serve_http_failure`) and the scripted-engine E2E
  hook (`run_scripted_e2e_engine_serve_http`) are reapplied there, and the
  new `aim` arm added since the branch is covered too.
- `serve_cmd.rs`'s no-GPU refusal was rewritten by #703, so the
  `scripted_engine_death` bypass is expressed against `reuse_existing` /
  `host_has_no_usable_gpu` / `no_usable_gpu_refusal` instead of the old
  inline bail.
- the branch's serve-25/26 scenario numbers now collide with scenarios that
  landed meanwhile, so they are renumbered serve-33/34; their `@id:` tags
  were already unique and are unchanged.

Not yet verified — the workspace does not build on Darwin, so this merge
still needs a build/clippy/test run on the Linux dev box.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…ayout

Fallout from moving the fix off main.rs, caught by the first build on the
Linux box:

- `read_managed_service_manifest` now lives in `services_text.rs` and was
  private, so `service_launch.rs`'s five `crate::`-rooted calls could not
  resolve. Re-pointed at `crate::services_text::` and made it `pub(crate)`.
- dropped a duplicate `crate::apply_app_path_env` import left over from the
  pre-refactor path; `engine_runtime_select::apply_app_path_env` is the one
  that exists.
- gated `engine_ipc.rs`'s `Duration` import on `e2e-test-hooks`, since only
  the scripted-engine hook uses it.

Plus the rustfmt reflow the longer module paths force.

Verified on the MI300X box: build (with and without rocm/e2e-test-hooks),
clippy --workspace --all-targets and clippy -p e2e-cucumber --test e2e both
-D warnings, fmt --check, and the full workspace suite (0 failures). The
nine ROCMAI-541 unit tests run and pass. serve-33/34 pass under
HIP_VISIBLE_DEVICES="" (which puts the harness on its no-GPU mock premise),
and serve-33 fails when the ServerExited crash branch is reverted to the
pre-fix `Ok(())`, so it discriminates rather than passing vacuously.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Review findings on the merged ROCMAI-541 fix.

The Windows spawn reused `spawn_hidden_console_with_log`, which swaps
`DETACHED_PROCESS | CREATE_NO_WINDOW` for `CREATE_NEW_CONSOLE`. That buys
nothing here and costs a `conhost` per engine: a managed engine is a
background service meant to outlive the CLI, and the console was incidental
to the only thing wanted from that helper, the log attach. `STARTF_USESTDHANDLES`
is set independently of the creation flags, so the log can be attached while
staying detached.

Adds `spawn_detached_with_log`: `spawn_detached_no_inherit`'s flags plus the
log. The handle duplication both variants need moves into a shared
`spawn_windows_with_log`, so `spawn_hidden_console_with_log` keeps its exact
previous behaviour — it still has a caller, the lemond daemon in
`engines/lemonade/src/process.rs`, which does want a console.

Blanket `bInheritHandles` is unchanged and still inherent to passing std
handles this way; the new function's doc comment says so, and scoping it
would need `STARTUPINFOEXW` with an explicit handle list.

Also: document serve-33/34 in docs/testing.md, which AGENTS.md requires for
an observable-behaviour change, including how to run them on a GPU host; and
correct a comment still placing `run_scripted_e2e_engine_serve_http` in
main.rs rather than engine_ipc.rs.

The Windows paths here are `cfg(windows)` and cannot be compiled on the Linux
box (the MSVC target's C dependencies need `lib.exe`), so they are checked by
CI's windows-build-and-test job, not locally. Linux re-verified: fmt, clippy
-D warnings, full suite, and serve-33/34 still pass.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from a team as a code owner November 3, 2026 12:06
@r0x0r
r0x0r requested a review from rominf November 3, 2026 12:06

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this. I read the full diff against ROCMAI-541 and have four things I'd want addressed before merge; details are inline or below.

Blocking

  1. CI hasn't run on this head commit (only the docs check shows up) and the PR conflicts with main. The Windows spawn/liveness paths, the changed spawn_hidden_console_with_log path used by the lemond daemon, and the removal of #[cfg(not(windows))] on the read_optional_tail_lines import are all unverified. Needs a rebase and green required checks.
  2. serve-33 pins the wrapper text ("exited unexpectedly") and the non-zero exit, but not the engine's own failure reason. That string is emitted even when the log tail is empty, so the scenario would stay green if the reason never reached the log or the CLI output, which is the core ask of the ticket. Please assert the scripted death message in the output and the service log too.
  3. The Windows log attach (the headline fix) has no test that runs on Windows; the new liveness test is Unix-only and serve-33 would not catch child output still going nowhere. A pure helper or injectable seam that the Windows job can exercise would do it.
  4. Docs aren't synced: rocm serve --verbose now exits non-zero with a new message where it used to print "server process exited" and exit 0. README, the --help text and docs/manual-testing.md still describe the old behavior.

Non-blocking / questions

  • On the non-verbose path a crash is only reported after the full 45s readiness wait, and that path only has a unit test that doesn't assert the exit code.
  • engine_ipc.rs: the wrapper stamps failed on any Err from the engine entry. If an engine returns Err when stopped via rocm services stop, an intentional stop would read failed; I couldn't tell from the diff whether the stop marker is consulted on this path.
  • managed_service_stop_was_requested treats status == "stopped" as a requested stop. Please confirm nothing other than an operator stop (liveness refresh, rocmd reconciliation) can write stopped before that check, or a crash would be reported as a clean exit.
  • The error chain can end up in the service log more than once (explicit append, tracing, and the CLI's own Error: print now that stdio is attached).
  • The "mark failed" merge pair is repeated in five places and the stop-vs-crash decision in three; a helper each would help.
  • Mutation evidence in the PR text covers serve-33 only; serve-34, the restart-path tests and the stop-marker tests should be shown to fail with their target broken too.
  • At about 2100 lines, with stop-marker, restart-path, zombie-reaping and merge-vs-write conversions beyond the ticket, this would be easier to review split up.
  • docs/testing.md cites engine_ipc.rs and serve_cmd.rs as bare backtick paths; AGENTS asks for relative links.

assert_scripted_hook_was_reached(&output);
assert_ne!(rc, 0, "expected a non-zero exit, got 0:\n{output}");
assert!(
output.contains("exited unexpectedly"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This asserts only "exited unexpectedly", which managed_engine_exit_failure_detail emits even when the log tail is empty. Please also assert that the scripted death message ("scripted e2e engine death after startup") appears in the output and the service log; otherwise the ticket's core ask (reason reaches log and CLI) isn't covered and the Windows attach could regress unnoticed.

Comment thread apps/rocm/src/service_launch.rs Outdated
/// ROCMAI-541 acceptance criterion 2: a managed engine that died before
/// ever answering HTTP must fail `rocm serve` with both a non-zero exit
/// code and a message naming it, not just the unreadable "starting"
/// status the readiness timeout alone used to leave behind. `pid_is_running`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale doc: it says pid_is_running is injected as |_| false, but the function takes child_is_alive: impl FnMut() -> bool and the test passes || false.

Comment thread apps/rocm/src/service_launch.rs Outdated
);
if proceed_with_launch_bookkeeping {
if readiness == EndpointReadiness::Serving {
// Latch the verification the wait just performed, so the readiness

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The liveness check only runs after the 45s readiness wait returns, so a crashed engine makes non-verbose rocm serve hang the full 45s before failing, and this path has no scenario (the unit test doesn't assert the exit code). Consider feeding liveness into the wait loop, or covering it with a scenario.

/// [`spawn_hidden_console_with_log`] already uses; scoping it would need
/// `STARTUPINFOEXW` with an explicit handle list.
#[cfg(windows)]
pub fn spawn_detached_with_log(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing std handles makes bInheritHandles true for every managed engine (previously none were inherited). A background rocm serve whose stdout/stderr is a captured pipe may now keep that pipe open for the engine's lifetime. Please verify on Windows with piped output; scoping the inheritance would need a STARTUPINFOEX handle list.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 46b7fe2

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (merge-base c39ee0c6 through head 46b7fe23, all 7 commits). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees server process exited", status In Progress). The description's account of the diff is accurate and unusually self-aware — it already names the main-branch scenario-number collision and the relationship to sibling PR #437 — so there is no description/scope mismatch to report.

Blocking

  • The PR has a real merge conflict with current main, from an independent scenario-number collision — tests/e2e-cucumber/features/model_serving.feature:538,554
    This PR adds Scenario: serve-33/serve-34. main has independently added serve-33 through serve-39 for unrelated work (a native-Windows platform gate and gated-AIM scenarios) using the same numbers. Confirmed by actually dry-running the merge (git merge --no-commit --no-ff origin/main from the PR head) — CONFLICT (content): Merge conflict in tests/e2e-cucumber/features/model_serving.feature — and by GitHub's own mergeable_state: "dirty" on the live PR.
    Confidence 100 · architectural · Fix: rebase onto current main and renumber this PR's scenarios to the next free IDs.

  • No required CI check has executed for this commit. Only a Read-the-Docs build shows up (get_check_runs returns 0 check runs; get_status shows a single success status, the docs build). build-and-test, clippy, prek, and the required windows-build-and-test lane have not run at all, so none of the Windows spawn/liveness changes, the spawn_hidden_console_with_log refactor, or the #[cfg(not(windows))] removal on read_optional_tail_lines's import have been verified by CI at this commit.
    Confidence 100 · architectural · Fix: resolve the conflict above, push, and let the required checks run green before this is re-reviewable with confidence.

  • README.md and docs/manual-testing.md were not updated for the new rocm serve --verbose/--foreground crash behavior, violating the project's own same-change doc-sync rule (AGENTS.md §5). The change makes a crashing managed engine exit non-zero with a new message and a failed manifest/engine-state stamp, where it previously printed server process exited and exited 0. Grepping both files for either string returns nothing, and the one passage in README.md that already documents the --verbose/--foreground attach behavior (README.md:891-893, Ctrl-D/Ctrl-C) says nothing about what happens on a crash. Only docs/testing.md was touched.
    Confidence 85 · mechanical · Fix: add a sentence to README.md and docs/manual-testing.md describing the new exit code/message/failed-stamp behavior on an attached crash.

  • The headline Windows fix (log attachment) has no test coverage reachable from any required CI check — neither unit nor e2e. crates/rocm-core/src/process.rs's new spawn_detached_with_log (line 101), spawn_windows_with_log (line 138) and the ManagedChildLiveness::Windows arm (line 461) have no #[cfg(windows)] test exercising them — the file's existing Windows tests (WindowsFixture-based, further down) are all pre-existing and unrelated; one of this PR's own doc comments admits "this repo's test environment has no Windows runner to exercise it on." Separately, the new e2e scenarios (model_serving.feature:538,554) depend on the ROCM_E2E_SCRIPTED_ENGINE/e2e-test-hooks mechanism and are tagged @requires-no-gpu only (no Windows tag or inclusion list). The one required Windows job that runs e2e at all (.github/workflows/ci.yml:932-954, inside windows-release) is explicitly E2E_ONLY_LIFECYCLE=1 with E2E_ALSO_RUN_IDS naming three unrelated comfyui scenarios, and is "Deliberately NOT built with rocm/e2e-test-hooks" per its own comment — so even if these two scenarios were added to that list, the scripted-engine hook they depend on would not compile in. This also means the PR description's claim that "CI's windows-build-and-test job is the real check on" the Windows paths overstates what that job actually exercises: it will confirm the code compiles and existing tests still pass, not that the new log-attach logic works.
    Confidence 85 · logic · Fix: add an injectable/pure-helper Windows unit test for spawn_detached_with_log/ManagedChildLiveness::Windows, and either get serve-33/34 running on a required Windows lane with e2e-test-hooks, or add a dedicated Windows-only scenario/unit test.

  • managed_service_stop_was_requested's reliance on record.status == "stopped" is not sound proof of an intentional stop, and this PR is the first to build a safety contract on it. apps/rocm/src/service_launch.rs:321-325 treats stop_requested_unix_ms.is_some() || record.status == "stopped" as "this exit was not a crash." But apps/rocm/src/managed_service_status.rs:274-277 (pre-existing, unrelated to this PR) writes status = "stopped" the instant a tracked pid is found dead during an ordinary status read (rocm services list/status, rocm dash, a daemon poll) — its own comment says so explicitly: "dead pids alone do not distinguish a stopped service from a crashed one." No stop_requested_unix_ms is set on that path. If a concurrent status read lands during this PR's new 45s launch/restart readiness windows (or the long-lived attach loop) right as a genuine crash happens, that reconciliation can flip status to "stopped" first, and this PR's new guard would then read the exit as a clean stop — silently swallowing the crash this whole PR exists to surface. The new tests only cover the case where stop_requested_unix_ms is set together with status: "stopped", never status == "stopped" alone.
    Confidence 85 · logic · Fix: only trust stop_requested_unix_ms as proof of an intentional stop; have the liveness-reconciliation path use a distinct status (e.g. "crashed"/"unreachable") for a bare dead-pid discovery instead of reusing "stopped".

  • serve-33's "engine state... reads failed" assertion doesn't test the code path it's meant to cover. tests/e2e-cucumber/tests/e2e/serving_steps.rs:2344-2355 asserts both record["status"] == "failed" (the manifest) and engine_state["status"] == "failed". The engine-state file, however, is already stamped "failed" unconditionally by engine_ipc.rs's record_engine_serve_http_failure, which runs inside the detached child itself the instant the scripted engine errors — independent of service_launch.rs's own ServerExited handling and its still_watching_this_supervisor guard. Breaking or deleting the CLI-side (service_launch.rs) engine-state write would leave this assertion green; only the manifest-status half actually depends on the new parent-side logic.
    Confidence 70 · logic · Fix: assert something that only the parent-side guarded write can produce (e.g. a supervisor-identity-mismatch case), or move the engine-state check to a dedicated unit test of run_attached_service's guard.

Non-blocking

None scoring above the reporting threshold — see Decisions for the author for lower-confidence/judgment items.

Decisions for the author

  • ManagedChildLiveness::Windows reopens the child by bare PID on every liveness check (crates/rocm-core/src/process.rs:461-479), across windows up to 45s (launch/restart readiness) or the whole attached session — a PID-reuse race. This isn't a new hole (Windows's process_is_running has always had this property, and PR #499's own history already accepts the equivalent gap for the stop path), but this PR is the first to key the crash/starting decision on it in three places. This PR's own commit 7ad214c2 explicitly flags it as deferred: "Left as-is per this round's explicit scope: the Windows ManagedChildLiveness PID-reuse race (a known fork against unmerged PR #437)... cross-PR sequencing calls for a human, not this round." PR #437 uses a race-free held-handle design for the same problem. Tradeoff: merge now with this narrow, already-accepted-elsewhere residual gap and track reconciliation with #437 separately, vs. hold this PR until it adopts #437's approach. Since this reasoning lives only in a commit message and not in the PR description, it's worth carrying forward into the description either way so a future reader doesn't have to dig for it.
  • Scope/size — at ~1624 insertions across 10 files, bundling the narrow Windows-log-attach fix with a broader stop-vs-crash redesign touching three call sites (launch, restart, attach) makes this harder to review and rebase (as the conflict above shows) than necessary. Consider whether the log-attach fix and the liveness/stop-vs-crash logic could land as two PRs. (non-blocking-improvement)
  • Duplicated "mark failed" merge pattern across roughly six call sites (engine_ipc.rs, services_text.rs ×2, service_launch.rs ×3) with no shared helper is itself not a defect, but it's the kind of duplication that produced the asymmetry in the blocking status-check finding above — factoring it into one helper that always checks the stop marker first would make that class of bug harder to reintroduce. (non-blocking-improvement)

Positive signals

  • Doc comments throughout are unusually rigorous — nearly every non-obvious decision names the exact race or prior bug it guards against (e.g. why merge_json_state_file is used instead of record.write(), why supervisor_pid/supervisor_start_ticks rather than engine_pid is the right identity to gate on).
  • Mutation-style self-verification was actually performed and documented, not just asserted: the PR text states serve-33 was confirmed to fail when the crash branch is reverted to the pre-fix Ok(()), matching AGENTS.md's own "shown to fail when the code it covers is broken" rule.
  • Commit history is clean and well-labeled (round-by-round B1/B2/B3/N1-N5 commits, each individually signed off), not a trail of "wip"/"fix" noise.
  • The spawn_hidden_console_with_log → shared spawn_windows_with_log refactor is a careful, verified-safe extraction: the Lemonade daemon's existing caller is confirmed unchanged in behavior.

Deployment notes

rocm serve --verbose/--foreground changes observable behavior on a managed-engine crash: it now exits non-zero with a new message and stamps the manifest/engine-state failed, where it previously exited 0 and printed server process exited. Any external automation or script keyed on the old string or the old exit code will break — this should be called out explicitly once the doc-sync finding above is addressed.

What this covered

Read completely: apps/rocm/src/engine_ipc.rs, apps/rocm/src/serve_cmd.rs (diff + surrounding context), apps/rocm/src/service_launch.rs, apps/rocm/src/services_text.rs, crates/rocm-core/src/lib.rs, crates/rocm-core/src/process.rs, docs/testing.md, tests/e2e-cucumber/features/model_serving.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs — via four independent background review passes (engine_ipc/services_text/serve_cmd; e2e tests/docs; AGENTS.md agent-instruction adherence; git history/blame) plus my own direct, complete read of service_launch.rs/process.rs/lib.rs, README.md/docs/manual-testing.md, and .github/workflows/ci.yml's Windows e2e configuration. Also ran a local dry-run merge against origin/main to confirm the conflict, and checked the live PR's CI/check status directly via the GitHub API. The root AGENTS.md was loaded and applied as the agent-instruction file.

Fan-out ran at reduced parallelism for one group: the shared environment's subagent-concurrency ceiling was saturated by other sessions for part of this review, so the "core launch/liveness" file group (service_launch.rs/process.rs/lib.rs) was reviewed directly by this driving context rather than by an independent worker — all other passes ran as independent background workers as intended.

Disclosure on the mandatory pre-review claim-check: resolving the live PR state required fetching existing reviews and the PR body, which necessarily exposed this context to the full text of an existing human review (by a different reviewer, state CHANGES_REQUESTED, same commit) and the PR's own description before all findings above were settled. No finding above was derived from that review's content — each was independently re-derived and verified from the code, CI configuration, and git history as cited inline. Several findings overlap with that existing review's points (CI/merge-conflict status, the Windows test gap, doc sync, the stop-vs-crash status ambiguity, double-logging, the repeated merge pattern, PR size) — that is convergent independent discovery, not copying. The PR description was read in full as part of this same required call; per method it was used only at the end, to state the ticket relationship and check for a scope mismatch, not to shape findings. Not reconciled against that review's thread or any other discussion.

Did not run: cargo build/test/clippy myself (reasoned from code, the PR's own stated verification, and CI configuration rather than executing the suite in this sandbox).


🤖 Generated with Claude Code

main moved 9 commits since the first merge, and the resulting conflict meant
GitHub could not build refs/pull/854/merge — so no pull_request workflow ran
at all, which is why only the docs check reported. Merging unblocks CI.

ROCMAI-823 took serve-33/34 meanwhile, so these are renumbered serve-40/41
once more; their @id: tags are unchanged and remain the stable reference.

Review fixes:

- serve-40 asserted only the wrapper text ("exited unexpectedly") and a
  non-zero exit. `managed_engine_exit_failure_detail` emits that wrapper even
  when the log tail is EMPTY, so the scenario would have stayed green in
  exactly the failure this ticket is about. It now pins the engine's own
  death message in both the service log and the CLI output.
- the Windows log attach — the headline fix — had no test on the platform the
  bug was reported on: the liveness test is Unix-only and the scripted-engine
  scenarios run the Unix branch. Adds a cfg(windows) test spawning a child
  through `spawn_detached_with_log` and asserting BOTH streams reach the log,
  stderr included, since that is where an engine's error chain goes.
- README's --verbose section said nothing about what a crash does; it now
  states the non-zero exit and the `failed` stamp, and that a deliberate stop
  still exits 0.
- extracted `mark_service_failed` for the manifest+engine-state pair that was
  repeated at three sites, with the merge-not-write reasoning stated once.
- docs/testing.md: relative links instead of bare backtick paths.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — this was a genuinely useful read, and point 2 in particular found a real hole. Pushed as 8c81370. Taking the points in order.

Fixed

2. serve-40 didn't pin the failure reason. You were right, and for the reason you gave: managed_engine_exit_failure_detail emits "exited unexpectedly" on both branches, including when the log tail is empty, so the scenario would have stayed green in exactly the failure this ticket is about. It now asserts the engine's own death message in both the service log and the CLI output.

I checked that the new assertion actually discriminates rather than just being longer. Breaking only the error-message tail was not enough to fail it — on Linux --verbose also streams the log to the terminal, so the reason still reaches the user by a second path, which is worth knowing. Disabling both delivery paths (the Unix attach_background_stdio and the record_engine_serve_http_failure backstop) fails it precisely where intended, while the old assertions still pass:

✘  Then serving fails with a non-zero exit and names the crash
   the engine's own failure reason never reached the service log — the bug this scenario exists for

3. No Windows-runnable test for the log attach. Added one in process.rs: it spawns a child through spawn_detached_with_log and asserts both streams land in the log — stderr included, since that is where an engine's error chain goes. It runs on the windows-build-and-test lane.

4. Documented the behavior change. README's --verbose section now states the non-zero exit, the quoted log tail and the failed stamp, and that a deliberate stop still exits 0.

Non-blocking, also done: extracted mark_service_failed for the manifest+engine-state pair that was repeated at three sites, with the merge-not-write reasoning stated once; docs/testing.md now uses relative links.

Mutation evidence, as asked. Each test fails when its own target is broken:

Mutation Fails
failure detail + backstop stop reaching the log serve-40 (new assertion)
attach path ignores the stop marker serve-41
record_restart_readiness liveness gate removed restart_readiness_fails_closed_when_the_restarted_child_died
restart path ignores the stop marker restart_readiness_treats_a_requested_stop_as_clean_not_a_crash
ServerExited crash branch back to Ok(()) serve-40 (original evidence)

Two corrections

1a. CI wasn't a separate problem from the conflict — it was caused by it. With the PR unmergeable GitHub can't build refs/pull/854/merge, so pull_request-triggered workflows never start; only the Read the Docs check (which keys off the branch) reported. Merging main fixed both at once, and CI is running now.

1b. The #[cfg(not(windows))] removal on read_optional_tail_lines is required, not accidental. managed_engine_exit_failure_detail is deliberately platform-generic — that is the point of the fix, since the log now exists on Windows too — and calls it unconditionally:

pub(crate) fn managed_engine_exit_failure_detail(service_id: &str, log_path: &Path) -> String {
    let recent_lines = read_optional_tail_lines(log_path, 80, "service log");

Restoring the gate would fail the Windows build on an unresolved import. The gate was correct before only because the sole Windows-side caller didn't exist yet. (The still-gated managed_engine_startup_failure_detail is the Unix-only one, which is probably where the expectation came from.)

4a. The docs weren't stale. rg "server process exited" matches only the source and my own new docs/testing.md text — nothing in README, --help or docs/manual-testing.md ever described the old exit-0 behavior, so there was nothing to correct. I've added the missing description rather than fixed a stale one.

Open questions, answered

  • Does the engine_ipc.rs wrapper stamp failed on an intentional stop? No. A stop kills the child, so the engine process is gone and that code never runs — the stop is adjudicated in the attach/launch/restart paths, which all consult the marker first. serve-41 covers this end to end.
  • Can anything but an operator stop write stopped before the check? Not in the window that matters. stop_requested_unix_ms is written before any stop signals anything, and it is the first term; status == "stopped" is the second, reached only when a stop already confirmed the child gone. A liveness refresh demotes to failed/stopped only for a record that is already not running.
  • Duplicate error chain in the log. Real, and deliberate belt-and-braces: the attach covers the normal path, the explicit append covers a gap in the attach. Duplication beats silence, which was the bug. Happy to dedupe if you'd rather.
  • Non-verbose path only reports after the 45s wait. Correct and intentional — a still-alive child that is merely slow must not be failed early, so the verdict waits for the readiness budget. Worth a follow-up to report a dead child sooner.

On splitting

Fair, and I'd push back only partly. The stop-vs-crash and zombie-reaping work isn't scope creep: the first version of this fix turned every ordinary piped serve --verbose stop into a failure, and on Linux the liveness check was inert against an unreaped child — so each is a defect the headline fix introduced or exposed. Separating them would land a fix that's known-broken in between. The restart path is the one genuinely separable piece. Happy to split that out if you want it smaller.

…ture

The new `a_detached_child_writes_both_streams_into_the_log` failed on the
Windows lane, in the fixture rather than the code under test. Its temp dir
embedded `std::thread::current().id()`, whose Debug form is `ThreadId(2)`,
and `quote_windows_arg` only quotes an argument containing whitespace or a
quote — so the path reached cmd bare and `(` ended the token early:

    'C:\...\rocm-core-detached-log-664-ThreadId' is not recognized as an
    internal or external command

Dropped the thread id; the pid alone is unique, since only this test uses
the name.

The failure was still informative: cmd's complaint reached the log file via
stderr, which is exactly the attachment the test exists to prove. What it
caught was a malformed fixture command, not a broken attach.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · a849ce1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (base 87a7c7a7 through head a849ce1, 9 commits; the 11-file, 1745/89-line diff matches the PR's own additions/deletions). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees server process exited", status In Progress). All three of the ticket's acceptance criteria (error chain reaches the service log on Windows, rocm serve reports failure with a non-zero exit, the engine state file is stamped failed on any error exit) are addressed by this change; the PR does not claim to close the ticket, and the extra stop-vs-crash/restart-path work is additive to the literal AC text.

Blocking

  • The PR currently conflicts with main, from a fresh scenario-number collision. tests/e2e-cucumber/features/model_serving.feature: this PR's Scenario: serve-40/serve-41 collide with an independently-added serve-40 from main's current tip (commit db7e5408, "ROCMAI-827: start the background helper only while an automation needs it", #838), which uses the same number for unrelated work. Confirmed by directly dry-running git merge --no-commit --no-ff origin/main from the PR head — CONFLICT (content): Merge conflict in tests/e2e-cucumber/features/model_serving.feature — and by the live PR's own mergeable_state: "dirty".
    Confidence 100 · architectural · Fix: rebase onto current main and renumber this PR's two scenarios to the next free IDs.

  • The "child died before its readiness wait even started" branches on both the launch and restart paths stamp failed without first checking whether a stop was requested, contradicting this file's own documented contract. apps/rocm/src/service_launch.rs (spawn_managed_engine_child's immediate post-spawn liveness check) and apps/rocm/src/services_text.rs (restart_internal_managed_service_with's equivalent branch, ~line 1085) both call mark_service_failed/an unconditioned engine-state merge the instant a freshly-spawned child is found dead within its first ~200ms — neither consults managed_service_stop_was_requested first. mark_service_failed's own doc comment in service_launch.rs states the precondition explicitly: "Callers reach this only after managed_service_stop_was_requested has already ruled out an intentional/external stop." Every sibling failure branch this PR adds (finalize_managed_launch_readiness, record_restart_readiness, the attached ServerExited arm) does consult it first; these two do not, so an external rocm services stop landing in that narrow post-spawn window can be reported as a crash. This is corroborated directly from this PR's own git history: commit 7ad214c2's message describes adding the stop-check explicitly to finalize_managed_launch_readiness/record_restart_readiness (its "B1") as a separate, later fix from the one that touched this exact branch ("B2" — which only fixed the manifest/engine-state write mechanism, not the missing stop-check), so the gap was never actually closed for this branch in either round. On the restart side specifically, the bug compounds: update_restarting_record's CAS guard (names_same_processes + status match) would detect a concurrent stop (the status would already differ) and return Ok(None), but that Option return value is discarded — so the engine-state file is still stamped failed unconditionally and the restart command still bail!s with a failure message regardless of what the CAS detected. No test exercises either branch against a concurrent stop (unlike the readiness-wait branches, which have dedicated tests for exactly that race).
    Confidence 85 · logic · Fix: have both branches call managed_service_stop_was_requested before marking failed, the same way their sibling branches do; have the restart branch check update_restarting_record's return value before writing the engine-state file or bailing.

  • ManagedChildLiveness::Windows — the liveness arm that is the actual crash-detection primitive on Windows — has no test reachable by any required CI check. Its own test module says so directly: the only test exercising ManagedChildLiveness (managed_child_liveness_unix_reports_dead_after_exit_and_does_not_double_reap) is #[cfg(not(windows))], and its doc comment states the Windows arm "is not exercised here." Separately verified against .github/workflows/ci.yml: the one required Windows e2e lane (windows-release's "Acceptance install lifecycle" step) runs with E2E_ONLY_LIFECYCLE=1 and a fixed E2E_ALSO_RUN_IDS allowlist of unrelated scenarios, and its own comment states it is "Deliberately NOT built with rocm/e2e-test-hooks" — so the new scripted-engine scenarios this PR adds could not run there even if added to that list; their compiled-in test seam does not exist in that binary. By contrast, the sibling Windows fix in the same file — spawn_detached_with_log, the log-attachment half of this same ticket — does have a dedicated #[cfg(windows)] #[test] (a_detached_child_writes_both_streams_into_the_log), which runs and passed in the "Windows build and test (debug)" CI job on this commit. So a Windows CI runner capable of exercising cfg(windows) unit tests is demonstrably available and used for one half of the fix, but not the other. This is exactly the case AGENTS.md §3 names: "Platform-gated code gets a test that runs on its gate... If a path stays untested, say which and why in the PR text" — the PR body's "CI's windows-build-and-test job is the real check on" the Windows paths overstates what that job verifies for this specific arm (compiles and doesn't regress existing tests, not that the new liveness logic is correct).
    Confidence 85 · architectural · Fix: add a #[cfg(windows)] #[test] for ManagedChildLiveness::Windows (a real detached child, killed and polled for liveness, analogous to the existing Unix test), or at minimum say in the PR text that this specific arm stays unverified and why.

  • docs/manual-testing.md was not updated for the new rocm serve --verbose/--foreground crash behavior, though its own Section 3 walks through exactly that scenario. The change makes an attached managed engine that dies on its own exit non-zero with a crash message and a failed manifest/engine-state stamp, where it previously exited 0 and printed "server process exited." README.md was correctly updated (new paragraph after the existing Ctrl-D/Ctrl-C bullets) and docs/testing.md documents the new e2e coverage, but docs/manual-testing.md's "3. Lemonade GPU Verification" section — which already manually walks a tester through this exact --foreground attached session and its Ctrl+D/Ctrl+C behavior — says nothing about what happens if the engine dies on its own, so a manual QA pass following this checklist would not catch a regression here. AGENTS.md §5 is explicit: "when a command's ... observable behavior changes, update README.md, its --help/doc comment, docs/testing.md, and docs/manual-testing.md in the same change — do not leave user-facing docs for a follow-up." (The --help text for --verbose/--foreground makes no claim about exit behavior either way, so no change is needed there.)
    Confidence 85 · mechanical · Fix: add a step to Section 3 (e.g., kill the Lemonade child while attached) naming the expected non-zero exit, crash message, and failed status.

Non-blocking

  • The new scripted-death e2e fixture uses a fixed 1.5s sleep with no explicit synchronization against the ~200ms early-startup liveness-check window it is meant to clear. tests/e2e-cucumber/tests/e2e/serving_steps.rs's scripted-engine death timer is a bare wall-clock delay, not gated on any observable signal that the liveness check actually completed; on a slow/loaded CI runner this margin (currently ~7.5x) could in principle narrow enough to land the death inside the startup window instead of after it, producing an intermittent failure in a different code path than intended. No actual flakiness was observed or reproduced; this is a structural risk rather than a confirmed one.
    Confidence 70 · logic · Fix: have the scripted engine wait on an explicit signal (a file/env marker the real liveness check writes) rather than a timer, or make the margin generous and document why it's believed sufficient.

Decisions for the author

  • ManagedChildLiveness::Windows re-derives liveness from a bare PID on every check (a PID-reuse race) across windows up to 45s or a whole attached session — tradeoff. This is not a new hole (process_is_running has always had this property), and this PR's own commit 7ad214c2 documents it as a deliberate, already-known deferral: "Left as-is per this round's explicit scope: the Windows ManagedChildLiveness PID-reuse race (a known fork against unmerged PR #437)... cross-PR sequencing calls for a human, not this round." That reasoning currently lives only in a commit message; carrying it into the PR description would save a future reader from digging for it.
  • The "mark failed" merge pattern is duplicated across roughly six call sites with no shared helper — non-blocking-improvement. Not a defect by itself, but it's the kind of duplication that let the blocking stop-check asymmetry above exist; one helper that always checks the stop marker first would make that class of bug harder to reintroduce.
  • PR scope/size — non-blocking-improvement. At ~1745 insertions bundling the narrow Windows-log-attach fix with a broader stop-vs-crash redesign across three call sites, this is harder to review and rebase (as the live conflict above shows) than the two pieces would be split.

Positive signals

  • Doc comments throughout are unusually rigorous about why, not just what — e.g. mark_service_failed's comment states its own precondition and the reason merges are used instead of whole-record writes (to avoid erasing a concurrently-written stop marker), which is exactly the contract the blocking finding above traces a violation of.
  • record_engine_serve_http_failure's tracing call and the explicit service-log append do not actually double-log: apps/rocm/src/logging.rs wires the process's tracing subscriber to a dedicated, file-only, non-blocking appender (~/.rocm/logs/rocm-cli.log.*) that explicitly never touches stdout/stderr, so the two writes land in two different files rather than duplicating in the one rocm logs tails.
  • Mutation-style self-verification was actually performed and documented: the PR text states the crash scenario was confirmed to fail when the fix is reverted to the pre-fix Ok(()).
  • Commit history is clean, round-labeled (B1/B2/B3/N1-N5), and individually signed off rather than a "wip"/"fix" trail — and one of those commit messages is itself the clearest evidence for the blocking stop-check gap above.

Deployment notes

rocm serve --verbose/--foreground changes observable behavior on a managed-engine crash: it now exits non-zero with a new message and stamps the manifest/engine-state failed, where it previously exited 0 and printed "server process exited." Any external automation keyed on the old string or exit code will break — worth calling out explicitly once the doc-sync finding above is addressed.

What this covered

Read completely: apps/rocm/src/engine_ipc.rs, apps/rocm/src/serve_cmd.rs, apps/rocm/src/service_launch.rs, apps/rocm/src/services_text.rs, crates/rocm-core/src/lib.rs, crates/rocm-core/src/process.rs, docs/testing.md, docs/manual-testing.md, README.md, tests/e2e-cucumber/features/model_serving.feature, tests/e2e-cucumber/tests/e2e.rs, tests/e2e-cucumber/tests/e2e/serving_steps.rs — via three independent background review passes grouped by coupling (engine_ipc/serve_cmd/lib.rs; service_launch/process/services_text; e2e-tests/docs/README), each given the full files plus an exact diff patch, followed by my own direct agent-instruction-adherence pass against the root AGENTS.md and a git-history/blame pass (git log, reading the PR's own round-by-round commit messages) over the same files. Verified via direct code tracing rather than assumption: the merge conflict (live git merge --no-commit --no-ff dry-run against current origin/main), the CAS/discard bug in update_restarting_record's caller, the Windows CI job's actual e2e configuration (.github/workflows/ci.yml), and the tracing-subscriber's file-only destination (apps/rocm/src/logging.rs, read for this purpose though outside the PR's diff). Did not run cargo build/test/clippy myself; relied on the GitHub Checks API, which shows all 25 check runs for this commit completed with conclusion success (one skipped, the non-required skill-evals job) — no failing required check to report.

Did not run as separate passes: "prior changes" (no prior-PR review comments were read, by design — only commit_id/login/state metadata was used for the mandatory claim-check); "code comments" was folded into the coupled-group passes above rather than run as a standalone pass.

Disclosure: the mandatory pre-review claim-check (get_reviews) and the PR-metadata fetch needed to read mergeable_state both returned full review/PR bodies as a side effect, which is more than the task intended me to see before findings were settled — this included an existing human reviewer's CHANGES_REQUESTED review and a prior automated review at an older commit. No finding above was taken from either; each was independently re-derived and directly verified against the code, git history, and CI configuration as cited inline (the live merge-conflict re-check in particular shows the conflict is a new, different collision than any earlier round would have seen, since main has moved again). Several findings likely overlap with points raised in those earlier texts — that reflects the same defects being independently reachable from the code, not reliance on having read them. Not reconciled against either review's thread.


🤖 Generated with Claude Code

Review of PR 854 (automated pass) found that the stop-vs-crash guard rests on
a status that cannot carry it, and it is right.

`managed_service_stop_was_requested` treated `status == "stopped"` as proof of
an intentional stop. But `refresh_managed_service_runtime_liveness` reconciles
ANY record whose tracked pids are all dead to `"stopped"`, with no marker —
`dash.rs` calls it "demoting a crashed server's stale `running` record to
`stopped`". So an ordinary concurrent `rocm services`/`rocm dash`/daemon read
landing between a crash and the check could relabel that crash as a clean stop
and have it reported as one: the exact failure this change exists to prevent.

Adds `StopWatch`, sampled on every liveness poll on all three paths (attach,
launch readiness, restart readiness). `stop_requested_unix_ms` is the only
sound evidence of an intentional stop, but it is transient — written before a
stop signals and cleared once it confirms — so a watcher reading it only after
the child is gone can miss it. Latching it while the child is still being
watched catches any stop in flight, including an *unconfirmed* one, which
re-stamps the marker and leaves the status alone.

The restart path switches to `wait_for_service_http_ready_with_progress`
purely for its per-tick hook; nothing is rendered. That left the non-progress
`wait_for_service_http_ready` wrapper with no production caller, so it is
removed and its three tests call the progress form directly.

The latch does NOT close the race, and the status term therefore stays.
Verified empirically: with the term removed, the external-stop scenario goes
red, because a stop can be requested, signal, and confirm — clearing its own
marker — well inside one 200ms poll. A confirmed stop leaves no other trace,
so status is the only evidence it happened. Closing this properly means making
the two distinguishable on disk (a distinct status for a bare dead-pid
reconciliation, or a durable marker for a confirmed stop), which changes
bookkeeping shared with `rocmd` and the dashboard. That is deliberately out of
scope here and is now documented at the guard with the reasoning and the
evidence, rather than left as an unstated assumption.

Also from the same review:

- docs/manual-testing.md gains an attached-crash section (4b) covering the new
  exit code, message and `failed` stamp, with the deliberate-stop control.
- serve-40's two assertions now say which writer each one pins: the manifest
  is parent-side only, while the engine-state file is stamped by the child's
  own backstop and would stay green if the parent write were deleted.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Addressing the automated pass (nowycondro), which reviewed 46b7fe23. Several of its blocking items were already fixed by pushes after that commit; one was a real finding that I had answered wrongly earlier in this thread, and it is now fixed as far as it can be without changing shared bookkeeping. Head is now 94df45aa.

The one that mattered — and where I was wrong

@juhovainio asked whether anything other than an operator stop can write stopped before the guard reads it. I said "not in the window that matters." That was wrong, and the automated pass caught it independently.

refresh_managed_service_runtime_liveness reconciles any record whose tracked pids are all dead to "stopped", writing no marker — dash.rs:379-382 describes it as "demoting a crashed server's stale running record to stopped". So an ordinary concurrent rocm services / rocm dash / daemon read landing between a crash and the check relabels that crash as a clean stop, and it gets reported as one. That is precisely the failure this PR exists to prevent.

What I changed. Added StopWatch, sampled on every liveness poll across all three paths (attach, launch readiness, restart readiness). stop_requested_unix_ms is the only sound evidence of an intentional stop, but it is transient — written before a stop signals, cleared once it confirms — so reading it only after the child is gone can miss it. Latching it while the child is still watched catches any stop in flight, including an unconfirmed one (which re-stamps the marker and leaves the status alone, so the status term never covered it).

What I could not close, and why. The latch narrows the race; it does not close it. I removed the status == "stopped" term to see whether the latch could stand alone, and the external-stop scenario went red:

✘  Then the attached serve exits zero and reports a stop, not a crash
   an externally stopped attach must exit 0, got 1

A stop can be requested, signal, and confirm — clearing its own marker — well inside one 200 ms poll, and a confirmed stop leaves no other trace. So the status term stays. The residual race is now documented at the guard with that evidence and the two ways to close it properly (a distinct status for a bare dead-pid reconciliation, or a durable marker for a confirmed stop) — both of which change bookkeeping shared with rocmd and the dashboard, which is why they are not in this PR. Raising a follow-up.

I'd rather ship this as a documented, narrowed, pre-existing race than either leave it as an unstated assumption or expand this PR into shared status semantics.

Already fixed before this pass was written

The review ran against 46b7fe23; 8c813708 and a849ce14 landed after it.

  • Merge conflict / scenario-number collision — fixed. main had taken serve-33…39, so these are now serve-40/41; @id: tags unchanged.
  • No CI had run — this was caused by the conflict: an unmergeable PR has no refs/pull/854/merge, so pull_request workflows never start. All checks have since run green, Windows included.
  • Windows test gap — fixed, and it earned its keep: the new cfg(windows) test spawning through spawn_detached_with_log failed on first run, catching that my fixture's temp dir embedded ThreadId(2) and quote_windows_arg only quotes on whitespace, so ( ended the token early. The failure was self-demonstrating — cmd's own complaint reached the log via stderr, which is exactly the attachment under test. So the claim that windows-build-and-test "will confirm the code compiles, not that the log-attach works" no longer holds: it now exercises both streams.
  • Doc sync — README done earlier; docs/manual-testing.md now has §4b with the attached-crash procedure, the expected non-zero exit and failed stamp, and the deliberate-stop control.
  • Duplicated mark-failed pattern — extracted into mark_service_failed.

Also addressed now

  • serve-40's engine-state assertion doesn't test the parent-side path — correct. The engine-state file is stamped by record_engine_serve_http_failure inside the child, so that half would stay green with the parent write deleted; only the manifest half is parent-side. Both assertions now carry a comment saying which writer each pins. Kept rather than removed: the in-child backstop is part of the same fix and worth pinning — just not as evidence for the parent path.
  • PID-reuse race on ManagedChildLiveness::Windows — carried into the PR description as you suggested, so it isn't buried in a commit message. Still deferred to the fix(serve): watch a Windows engine through its handle and keep a concurrent stop (ROCMAI-675) #437 reconciliation.
  • Deployment note — the breaking-change warning (external automation keyed on server process exited or exit 0) is in the description.

On splitting

Keeping it as one PR. The stop-vs-crash work isn't adjacent scope — it exists because the headline fix would otherwise turn every ordinary piped serve --verbose stop into a failure, and on Linux the liveness check was inert against an unreaped child. Splitting lands a knowingly broken intermediate state. The restart path remains the one genuinely separable piece if you want it out.

Verification

fmt, clippy --workspace --all-targets -D warnings, clippy -p e2e-cucumber --test e2e -D warnings, and the full workspace suite are green on this head. serve-40/41 passed on the functionally identical tree; the only change after that run was deleting the now-unused wait_for_service_http_ready wrapper and repointing its three tests, which the suite covers. The dev box was being rescheduled repeatedly while I finished, so the final e2e confirmation comes from CI's Linux lane, which runs these @requires-no-gpu scenarios.

main took serve-40/41 (ROCMAI-827 #838 and the OOM-guidance scenario), so
these are renumbered serve-42/43 once more; the @id: tags are unchanged and
remain the stable reference.

Review finding, and a real one: the "child died before its readiness wait even
started" branches on both the launch and restart paths stamped `failed`
without first asking whether a stop had been requested — contradicting
`mark_service_failed`'s own documented precondition, and unlike every sibling
branch (`finalize_managed_launch_readiness`, `record_restart_readiness`, the
attached `ServerExited` arm), which all consult it. A `rocm services stop`
landing in that ~200ms post-spawn window killed the child we had just spawned
and was then reported as a crash. The launch branch's own comment already
anticipated a concurrent stop writing its marker in "this same narrow window";
it simply never read it.

Both branches now check first and report the stop instead.

On the restart side the same branch discarded `update_restarting_record`'s
return value, so a CAS that detected a concurrent writer (returning `None`,
having deliberately written nothing) was followed by an unconditional
engine-state stamp anyway — and since `refresh_from_engine_state` reads that
file straight back into the manifest, the clobber the CAS had just prevented
came back through the other door. The engine-state write is now gated on the
manifest write having actually landed.

Also adds the Windows half of the liveness coverage:
`managed_child_liveness_windows_tracks_a_real_child_through_termination`
spawns a real child, asserts it reads as running, terminates it, and asserts
it stops reading as running and stays that way. `ManagedChildLiveness::Windows`
is the crash-detection primitive on the reported platform and cannot be
constructed off Windows, so the Unix test could not reach it and the
scripted-engine scenarios run the Unix arm — it had nothing exercising it,
while its sibling `spawn_detached_with_log` did.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Second automated pass (against a849ce1) addressed. Head is now 9970658d.

The real find: the post-spawn branches never checked for a stop

This one was correct and I'd missed it twice. The "child died before its readiness wait even started" branches — on both the launch and restart paths — called mark_service_failed without first asking whether a stop had been requested. Every sibling branch (finalize_managed_launch_readiness, record_restart_readiness, the attached ServerExited arm) consults it, and mark_service_failed's own doc comment states the precondition. So a rocm services stop landing in that ~200 ms post-spawn window killed the child we had just spawned and was then reported as a crash.

What makes it a fair catch: the launch branch's own comment already anticipated "a stop_requested_unix_ms a concurrent stop could have written to disk in this same narrow window" — it just never read it. Both branches now check first and report the stop instead.

The compounding bug on the restart side was right too. That branch discarded update_restarting_record's return value, so a CAS that detected a concurrent writer — returning None, having deliberately written nothing — was followed by an unconditional engine-state stamp anyway. Since refresh_from_engine_state reads that file straight back into the manifest, the clobber the CAS had just prevented returned through the other door. The engine-state write is now gated on the manifest write having landed.

Windows liveness coverage

Added managed_child_liveness_windows_tracks_a_real_child_through_termination: spawns a real child, asserts it reads as running, terminates it, asserts it stops reading as running and stays that way. Your argument was the persuasive part — a Windows runner demonstrably can exercise cfg(windows) unit tests, since spawn_detached_with_log's test runs there, so "no Windows runner" was never the real reason this arm was untested.

Already landed before this pass

The review ran against a849ce1; 94df45aa landed after it and covers two of its blocking items:

  • docs/manual-testing.md — added §4b, an attached-crash procedure with the expected non-zero exit, crash message and failed stamp, plus the deliberate-stop control. You suggested §3; I put it next to the existing Local Server Records section since that is where service records and rocm services logs are already exercised. Happy to move it if you'd rather it sat in the Lemonade walkthrough.
  • PID-reuse deferral and the deployment/breaking-change note — both carried into the PR description, out of the commit messages.

Scenario renumbering, again

main took serve-40/41 (ROCMAI-827 #838, and the OOM-guidance scenario), so these are now serve-42/43. Third renumber; the @id: tags have never moved and remain the stable reference. Worth noting this churn is an argument for the split you suggest — it is the feature file that collides every time, not the code.

Not taken

The 1.5 s scripted-death timer. I left it. The margin is ~7.5x over the ~200 ms startup window, and gating the scripted engine on a signal the real liveness check writes would couple the fixture to the very code path under test. If it ever flakes, the fix is a longer margin, not tighter coupling — I'd rather not pay that coupling for a risk neither of us has observed.

Verification

fmt, clippy --workspace --all-targets -D warnings, clippy -p e2e-cucumber --test e2e -D warnings, and the full workspace suite are green. serve-42/43 pass under HIP_VISIBLE_DEVICES="". CI is running now; the merge conflict is cleared, so the required lanes report again.

One process note on this round: the first attempt at this merge truncated its own conflict output and I missed three of the five conflicts, then git add -A staged the markers. Caught by the build before pushing, redone with a proper three-way merge for the file whose conflict region split a function body.

r0x0r added 2 commits November 9, 2026 13:52
…zero

README and docs/manual-testing.md already describe the ROCMAI-541 behaviour;
the `--help` long description still stopped at the Ctrl-C stop path, so the
one place a user looks first never mentioned that an unexpected engine exit
is now a loud non-zero failure rather than "server process exited" / exit 0.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	apps/rocm/src/engine_runtime_select.rs
#	apps/rocm/src/main.rs
#	apps/rocm/src/service_launch.rs
#	apps/rocm/src/services_text.rs
#	crates/rocm-core/src/lib.rs
#	tests/e2e-cucumber/features/model_serving.feature
@r0x0r

r0x0r commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main (66 behind) and cleared the last blocking item. Pushed as 550f9a14.

Your blocking items

1. CI hadn't run / conflicts. Merged origin/main (b0224ecb). Six files conflicted. This was not a blind merge — main had restructured most of what this PR touches:

  • services_text.rs → service_control.rs. main moved the whole restart path (restart_internal_managed_service_with, record_restarted_child, record_restart_readiness, update_restarting_record, unload_lemonade_service_model) into a new service_control.rs. Every one of this PR's hunks there showed an empty main side, which means relocated, not deleted. I took main's services_text.rs verbatim and ported the restart-crash fix plus both of its tests (restart_readiness_fails_closed_when_the_restarted_child_died, restart_readiness_treats_a_requested_stop_as_clean_not_a_crash) into service_control.rs, against main's newer signatures (managed_engine_startup_failure_detail now takes a formatted status; restart_readiness_timeout() replaces the hardcoded 45s).
  • service_launch.rs — the hard one. main rewrote 2,544 lines of it: spawn_managed_engine_child was split, with the spawn moved into a new launch_managed_engine_child_locked, and --wait added (wait_for_record_ready, wait_past_readiness_budget, died_during_wait, record_wait_outcome). I re-threaded this PR's ManagedChildLiveness handle through that new shape: launch_managed_engine_child_locked now returns (u32, ManagedChildLiveness), and the Windows branch keeps main's record_engine_identity + ensure_spawned_child_survives_startup while spawning via spawn_detached_with_log (the headline fix).
    • died_during_wait is checked before finalize_managed_launch_readiness, deliberately: --wait waits far past the readiness budget and owns its own death message and settling, so the ROCMAI-541 check must not preempt it.
    • main's record_wait_outcome re-reads the manifest before writing, which already solves the clobber this PR's hand-rolled merge_json_state_file was guarding against on that path, so I use main's. The spawn-path merge is kept, since nothing on main covers it there.
    • Two non-test, non-Windows callers of process_is_running (in stream_attached_logs/_no_tty) are now served by liveness, so the bare import went away.
  • Scenario renumbering: serve-42/serve-43 → serve-50/serve-51, since main now ends at serve-49. docs/testing.md updated to match, including the -n "serve-5[01]" invocation. Note the serve-42/serve-43 that remain in docs are main's own WSL/--wait scenarios, untouched.

2. serve-33 didn't assert the engine's own failure reason. Already addressed in 8c813708: the step now pins SCRIPTED_ENGINE_DEATH_REASON in both the service log and the CLI output, with a comment saying why the wrapper string alone is not enough (it prints even on an empty tail).

3. No Windows-runnable test for the log attach. Also in 8c813708: a_detached_child_writes_both_streams_into_the_log in rocm-core/src/process.rs is a real #[cfg(windows)] test that drives spawn_detached_with_log end to end through a .bat fixture and asserts both streams reach the log — stderr specifically, since that is where an engine's error chain goes.

4. Docs not synced. README and docs/manual-testing.md were already updated; the --help text was not, which is the half you were right about. Added in 2aa5322d — and note it had moved to cli.rs on main, so it is there, not main.rs.

Verification (Linux MI300X; the Mac cannot build rocm-engine-lemonade)

  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo clippy -p e2e-cucumber --test e2e -- -D warnings — clean (this target is harness = false, so check/test --all-targets give a false green on it)
  • cargo test --workspace — green, 0 failures
  • cargo test -p e2e-cucumber --test feature_naming — 4/4, so the renumbering is consistent
  • cargo fmt --all --check, xtask check-crate-edges, xtask check-architecture-doc — pass

Not verified

Your non-blocking points (the repeated mark-failed pair, the duplicated error chain in the log, engine_ipc.rs's failed stamp on any Err, the status == "stopped" question, and the split-up suggestion) are not addressed in this push — happy to take the split as a follow-up if you'd prefer that over reviewing this as one.

@r0x0r
r0x0r dismissed juhovainio’s stale review November 9, 2026 12:40

Addressed in 550f9a1: merged main (66 behind, service_launch.rs and the restart path both relocated/rewritten on main), and synced the --help text that items 2-4 had left behind. Details in the PR comment.

@r0x0r
r0x0r requested a review from juhovainio November 9, 2026 12:40

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (base e2e5c5bb3d3…head caebb946eb8). Implements the ROCMAI-541 acceptance criterion ("the engine state file is marked failed on any error exit") across every exit point of a managed engine child; ticket ROCMAI-541 is named but not retrievable from this environment (no tracker access here), so this is judged against the PR description alone.

Blocking

  • The Windows post-spawn early-exit check still uses the crash-prone probe this very PR replaces everywhere else — apps/rocm/src/service_launch/spawn.rs:559-577 (the ensure_spawned_child_survives_startup call), contrast apps/rocm/src/service_control.rs:705-711 (ensure_restarted_child_survives_startup).
    The Windows launch path passes rocm_core::process_is_running as the is_running check (spawn.rs:563). process_is_running's own doc comment (crates/rocm-core/src/process.rs:606-611) warns: "a live process this one may not open reads as not running here; a decision that must not take it for gone... asks probe_process instead." ensure_spawned_child_survives_startup returning false here is exactly such a decision: it feeds straight into settle_startup_death, which (absent a recorded stop) calls mark_service_failed and bails — i.e. a transient OpenProcess failure on a perfectly healthy new engine gets reported as a crash. This PR introduces ManagedChildLiveness specifically to fix this class of false negative (Windows::is_running uses probe_process(..).may_be_running(), treating Unqueryable as "may still be running"), and the sibling restart path got the fix — service_control.rs:710 passes || liveness.is_running(), with a comment explicitly contrasting itself against "a bare process_is_running pid probe." The launch path even constructs the needed ManagedChildLiveness::from_pid(child_pid) two lines later (spawn.rs:580) — it is simply not used for the check itself. This is new code in this PR (the PR's own table lists "Windows post-spawn early exit" as one of three paths "added in review"), unverified on Windows per the author's own disclosure, and currently reintroduces ROCMAI-541's symptom on the one platform the ticket is about.
    Confidence 90 · logic · Fix: construct liveness before the check (as the restart path does) and pass || liveness.is_running() instead of the raw function.

  • The PR description names the wrong regression scenarios — description body ("Tests" and "Also in this diff" sections) vs. tests/e2e-cucumber/features/model_serving.feature:926-951 and docs/testing.md's new "Managed-engine crash vs. stop" section.
    The description repeatedly says "serve-55 and serve-56 cover the attached --verbose path" and that mutation-testing "turn[s] serve-56 red". serve-55/serve-56 are pre-existing, unrelated scenarios ("The recommended small assistant serves from its short name alone" and "Serving a Hugging Face repo with no quantization variant is refused", model_serving.feature:884,903). The scenarios this PR actually adds are serve-57/serve-58 (@id:serve-scripted-death-after-startup, @id:serve-scripted-external-stop-while-attached), which the PR's own docs/testing.md addition correctly names. A reviewer or future maintainer following the description to the named scenarios lands on the wrong tests entirely.
    Confidence 95 · mechanical · Fix: correct every "serve-55"/"serve-56" reference in the PR description to "serve-57"/"serve-58".

  • A stop can still lose its own evidence through an untouched helper, reopening the crash-vs-stop confusion this PR closes elsewhere — apps/rocm/src/managed_service_status.rs:264-286 (settle_pending_stop_key_cleanup), consumed by apps/rocm/src/service_launch/readiness.rs:499-503 (managed_service_stop_was_requested) and apps/rocm/src/service_control.rs:270 (stop_internal_managed_service_confirmed_with).
    This PR's new contract (stated directly in managed_service_stop_was_requested's doc) is that stop_requested_unix_ms/settled_stop_unix_ms are the only evidence of a deliberate stop, and that every deliberate stop sets one of them. settle_pending_stop_key_cleanup's non-empty-recorded-process branch computes consumed_marker = record.stop_requested_unix_ms.take() and then discards it — only the empty-process branch a few lines below assigns it to settled_stop_unix_ms. This function is unmodified by this PR but runs ahead of stop_internal_managed_service_confirmed_with's own re-read whenever a concurrent rocm services/rocm dash/rocmd liveness refresh observes the signalled PIDs already dead first — a race stop_internal_managed_service_confirmed_with's own comment explicitly anticipates ("a record that reached 'stopped' by another route still gets its deferred cleanup", "load_managed_service's liveness refresh would act on this stop's own marker before the verdict below has been reached"). When that race lands, the record ends up status: "stopped" with both markers unset, so a later managed_service_stop_was_requested read for the same service (e.g. the attached ServerExited check) sees no evidence at all and would misreport the stop as a crash — the original symptom, via a path this PR didn't touch.
    Confidence 72 · architectural · Fix: have the non-empty-process branch move the marker to settled_stop_unix_ms too, the same way the empty-process branch and stop_internal_managed_service_confirmed_with do.

  • A required-looking check fails at this head commit — E2E consolidated report (self-hosted), FAILURE, run 43446100233.
    The job log shows this is not a test-content failure: the e2e-gpu (MI300X) and e2e-gpu-strix-windows lanes were themselves cancelled, and the consolidator errors because a cancelled lane uploaded no report ("lane e2e-gpu ran (needs result: cancelled) but uploaded no e2e-gpu-report"). Windows build and test (debug) — the one check the author names as covering the Windows paths this PR touches — is green. Still, this is a red check on the current head and needs a rerun (or triage of why the self-hosted lanes were cancelled) before merge.
    Confidence 100 · architectural · Fix: rerun the self-hosted E2E workflow; confirm the consolidated report goes green once the GPU/Windows lanes actually execute.

Non-blocking

  • The doc comment for the PR's central function is attached to the wrong one — apps/rocm/src/service_launch/readiness.rs:579-647. A ~55-line doc block describing a "failure branch" that "marks the engine's own state file failed" and returns true/false is clearly written for finalize_managed_launch_readiness (line 648), but a missing blank line before the next paragraph attaches the whole block to still_watching_this_supervisor (line 634) instead. The single most intricate function in this diff ends up with no rustdoc at all, while the guard function carries doc text describing different behavior.
    Confidence 85 · mechanical · Fix: insert the blank line/doc split so each function keeps its own comment.
  • The new "move, don't drop" invariant isn't directly asserted in service_control.rs's own primary tests — apps/rocm/src/service_control.rs (services_stop_reports_a_confirmed_stop_as_stopped, a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing). Both exercise the real confirmed-stop path whose new line is record.settled_stop_unix_ms = record.stop_requested_unix_ms.take(), but neither asserts settled_stop_unix_ms.is_some() afterward. The mirrored change in apps/rocmd/src/service.rs did get that assertion added to its sibling test (service.rs:662).
    Confidence 70 · mechanical · Fix: add the same settled_stop_unix_ms.is_some() assertion to the service_control.rs tests.

Decisions for the author

  • docs/architecture.md's entries for readiness.rs, process.rs and engine_ipc.rs still describe their pre-PR contents — answered-by-intent. The PR's "Known limitations" section states: "docs/architecture.md gained managed_child.rs, but the readiness.rs, process.rs and engine_ipc.rs entries still describe their older contents." Independently checked: the entries still read as generically true (not contradicted by the new code) but don't mention the new crash/stop-verdict logic or the scripted-engine hook. Would otherwise be a documentation-accuracy finding; confirm this is still the intended follow-up scope.

Positive signals

  • serve-57/serve-58 deliberately pin two independent writers of failed (the parent-side manifest write vs. the in-child record_engine_serve_http_failure backstop) with comments explaining why each assertion would or wouldn't survive the other writer being deleted — unusually resistant to a shallow partial fix passing the test.
  • The cli.rs regression test for the rocm logs→rocm services logs remediation-hint fix positively proves the old text was wrong (rocm logs <id> parses as Logs { service: None, .. }, a search query) rather than only asserting the new text is right.
  • settled_stop_unix_ms's doc comment (crates/rocm-core/src/service_record.rs) adds a "Writers" table naming all four call sites that touch the field; independently verified against the actual code at all four and found accurate.

Deployment notes

  • Exit-code/behavior change for scripts: background launches, rocm serve --wait, and rocm services restart now exit non-zero and leave the record failed on a genuine crash where they previously exited 0 (or, for --wait, left stopped/starting). Anything scripting around the old exit codes needs to adjust.
  • Windows engines are now logged (spawn_detached_with_log), which the author flags as forcing blanket handle inheritance in the child, matching the existing spawn_hidden_console_with_log model rather than a new exposure.
  • New rocm-core public API (ManagedChildLiveness, spawn_detached_with_log) — any out-of-tree consumer of this crate should note the surface addition.

What this covered

Every file in the diff (23 files, +3448/-150) read in full against e2e5c5bb3d34605888fcab61f4511ab24f215c3c…caebb946eb8390f1fc284f84bd94bd8bd9a186af: crates/rocm-core/src/{managed_child.rs,process.rs,lib.rs,service_record.rs}, apps/rocm/src/service_launch/{spawn.rs,attach.rs,start.rs} and service_launch.rs, apps/rocm/src/service_launch/readiness.rs, apps/rocm/src/service_control.rs + apps/rocmd/src/{service.rs,watchers/server_recover.rs}, apps/rocm/src/{cli.rs,engine_ipc.rs,engine_runtime_select.rs,serve_cmd.rs}, the e2e-cucumber test additions, and README.md/docs/architecture.md/docs/manual-testing.md/docs/testing.md — split across independent blind passes grouped by coupling, each reading surrounding/caller code beyond the diff, plus a consolidation pass that re-traced the two highest-severity candidates (the Windows liveness asymmetry and the stop-marker race) directly against source before scoring.

Did not run: a dedicated "prior changes"/version-history pass (git-log/blame context was read incidentally while tracing call sites, not as a separate pass); the workspace test/clippy suite locally (relied on the PR's reported CI results per-check, see Blocking). This PR has several earlier automated-review rounds at prior commits; per method this round was produced blind and was not reconciled against their content — it may re-find things they already raised, or diverge from them, as an independent check against the current head rather than a continuation.


🤖 Automated by an AI agent

… crash

Two paths could still report a healthy or deliberately-stopped managed
engine as crashed, which is ROCMAI-541's own symptom surviving inside its
fix.

The Windows post-spawn early exit was handed `rocm_core::process_is_running`
while the sibling restart path already asked the `ManagedChildLiveness` it
held. That probe's own doc comment warns that a live process this one may
not open reads as not running there, and this check's `false` goes straight
to `settle_startup_death`, which stamps the manifest and the engine state
file `failed` and aborts the launch -- so a transient `OpenProcess` denial
on a perfectly healthy new engine was a reported crash. The launch already
constructed the right value two lines below the check; it is now built
before it and used for it. `ensure_spawned_child_survives_startup` takes the
answer instead of the pid, so passing the raw probe is a type error rather
than a silent one, and `rocm-core` grows an all-platform `pid_child_is_running`
seam -- only its call site is `#[cfg(windows)]` -- so the Linux lane pins the
one answer the two probes disagree about.

`settle_pending_stop_key_cleanup` consumed a stop's marker and discarded it
on its non-empty-process branch, leaving `status: "stopped"` as the only
trace. That is not evidence: the same function writes that status for a
record whose pids are merely gone. It runs ahead of
`stop_internal_managed_service_confirmed_with`'s re-read whenever a
concurrent `rocm services`/`rocm dash`/`rocmd` liveness refresh sees the
signalled pids dead first -- the race that function's comment anticipates --
so the record ended with both markers unset and the next
`managed_service_stop_was_requested` called the operator's own stop a crash.
The marker now moves into `settled_stop_unix_ms` on both branches, as the
confirmed-stop path and `rocmd` already do.

Both fixes are pinned by tests that fail when the fix is reverted.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all three actionable blocking findings in f8f09e2dd. Verified on a Linux devbox (this Mac cannot build the workspace).

1. The Windows post-spawn early exit ran on the raw probe

Correct — and the shape of the bug was the inconsistency itself. The restart path got the fix (service_control.rs:710, || liveness.is_running(), with a comment contrasting itself against "a bare process_is_running pid probe"); the launch path was left on rocm_core::process_is_running while constructing the very value it should have used two lines below the check. So the launch path reported a healthy new engine as crashed on a transient OpenProcess denial — ROCMAI-541's own symptom, on ROCMAI-541's own platform.

Fixed as suggested: liveness is now built before the check and passed as || liveness.is_running(). Two things were added so this cannot silently regress:

  • ensure_spawned_child_survives_startup now takes the answer, not the pid (impl FnOnce() -> bool), mirroring ensure_restarted_child_survives_startup. Passing the raw probe is now a compile error rather than a plausible-looking argument:
error[E0593]: function is expected to take 0 arguments, but it takes 1 argument
   --> apps/rocm/src/service_launch/spawn.rs:744:60
    |
744 |         ensure_spawned_child_survives_startup(&log, || {}, rocm_core::process_is_running)?;
    |         -------------------------------------              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected function that takes 0 arguments
    |         |
    |         required by a bound introduced by this call
  • The liveness policy is extracted into rocm_core::managed_child::pid_child_is_running, compiled on every platform with only the call site #[cfg(windows)] (the pattern ensure_restarted_child_survives_startup already uses). This matters because ManagedChildLiveness::Windows cannot be constructed off Windows, and even the Windows fixture test can only produce Running/Gone from a real child — so Unqueryable, the only answer the two candidate probes disagree about, had nothing exercising it anywhere. Mutating that policy to process_is_running's collapse (probe(pid) == ProcessProbe::Running):
---- managed_child::tests::an_unqueryable_pid_child_is_not_reported_gone stdout ----
thread 'managed_child::tests::an_unqueryable_pid_child_is_not_reported_gone' panicked at
crates/rocm-core/src/managed_child.rs:144:13:
assertion `left == right` failed: Unqueryable must read as running=true
  left: false
 right: true

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 1502 filtered out

spawn.rs also gains a_child_that_could_not_be_queried_does_not_fail_the_launch, asserting the composed launch verdict over all three probe answers.

Stated plainly: the call-site change itself is inside a #[cfg(windows)] block, so no Linux test can observe it directly — the MSVC target cannot even be type-checked here (aws-lc-sys fails its native build when cross-compiling). The type error above and the policy test are what pin it; the call site itself still rests on CI's windows-build-and-test.

2. Wrong scenario numbers in the description

Correct, and this is the second time this PR's renumbering has been missed in prose — a docs/testing.md repro command still read serve-5[56] and was fixed in an earlier round. The description is corrected: every serve-55/serve-56 that meant this PR's scenarios now reads serve-57 (@id:serve-scripted-death-after-startup) / serve-58 (@id:serve-scripted-external-stop-while-attached), including the mutation note and the cargo xtask e2e -n repro command.

Swept for the rest rather than fixing only what was reported: grep -rn "serve-5[0-9]" . --exclude-dir=.git --exclude-dir=target across the whole tree, plus the PR body and every commit message on the branch. The remaining in-tree hits are all correct — the model_serving.feature and docs/testing.md references to serve-50…serve-56 genuinely name the pre-existing Anthropic-endpoint and small-assistant scenarios. Commit messages on the branch do contain older numbers, but only inside the renumbering tables of the commits that performed each renumbering (serve-56 -> serve-57, etc.); those are accurate history for their own commit and were deliberately left alone rather than rewritten under review.

3. settle_pending_stop_key_cleanup dropped the stop's own evidence

Correct, including the reasoning about when it lands. The non-empty-process branch did stop_requested_unix_ms.take() and discarded the value, leaving status: "stopped" as the only trace — and that is not evidence, since the same function writes that status for a record whose pids are merely gone. Via the race stop_internal_managed_service_confirmed_with's comment anticipates, the record ended with both fields unset, and the next managed_service_stop_was_requested reported the operator's own stop as a crash.

The marker now moves into settled_stop_unix_ms on both branches. Worth flagging: the pre-existing test asserted the defect — a_completed_pending_stop_reads_stopped_whatever_live_status_it_had carried assert_eq!(record.settled_stop_unix_ms, None, "a record that names processes reads stopped and needs no stamp"). That assertion is inverted and the test renamed, plus a second test that goes through the real predicate against the persisted manifest. Reverting the one-line fix:

---- managed_service_status::tests::a_completed_pending_stop_reads_stopped_and_keeps_its_evidence stdout ----
panicked at apps/rocm/src/managed_service_status.rs:1247:13:
assertion `left == right` failed: starting: the consumed marker must move to the settled stamp,
or this stop ends with no evidence it was deliberate and the next read calls it a crash
  left: None
 right: Some(1)

---- managed_service_status::tests::a_stop_settled_by_a_concurrent_liveness_refresh_is_still_readable_as_a_stop stdout ----
panicked at apps/rocm/src/managed_service_status.rs:1280:9:
a stop the liveness refresh settled must still read as a stop, not a crash

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 1893 filtered out

Both pass on restore.

4. E2E consolidated report (self-hosted) — cancelled lanes, not a test failure, not re-run

Agreed with your own reading of that log, so no code change. The e2e-gpu (MI300X) and e2e-gpu-strix-windows lanes were cancelled — no hardware available — and the consolidator hard-fails when a lane it expects a report from produced none ("lane e2e-gpu ran (needs result: cancelled) but uploaded no e2e-gpu-report"). That is infrastructure state, not anything this diff can affect. I deliberately did not re-run it: a rerun proves nothing about the change and just re-queues self-hosted capacity. It needs a maintainer with runner access, or a re-run once the lanes are back.

Verification

Linux devbox, each gate run separately: cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; the same with --features e2e-test-hooks; cargo clippy -p e2e-cucumber --test e2e -- -D warnings; cargo test -p e2e-cucumber --test feature_naming (4 passed); cargo xtask check-architecture-doc, check-crate-edges, check-module-size; cargo test -p rocm --bin rocm (1887 passed) and cargo test -p rocm-core --lib (1498 passed). All exit 0.

Two tests flaked once each under parallel load and passed isolated and on two further full runs, both unrelated to this diff: therock::family::tests::resolve_family_uses_managed_runtime_before_host_detection (cross-test ROCM_CLI_THEROCK_FAMILY env leakage) and process::tests::loopback_lease_blocks_a_second_bind_until_released (port contention).

Still open from this and earlier rounds

Not addressed here, and I am not claiming otherwise:

  • abandon_failed_managed_launch still writes the whole record rather than the fields it owns.
  • merge_json_state_file is still unlocked.
  • The engine-dispatch arm of the crash backstop is still untested.
  • The cli.rs parse test proves the parse, not its call sites.
  • The Windows cfg paths in this PR — including finding 1's call site — are exercised only by CI's windows-build-and-test; nothing available here can run them.
  • Your two non-blocking items (the misattached doc block at readiness.rs:579-647, and the missing settled_stop_unix_ms.is_some() assertion in service_control.rs's own stop tests) are untouched in this round.
  • docs/architecture.md's readiness.rs / process.rs / engine_ipc.rs entries still describe their pre-PR contents — confirming your read: that remains intended follow-up scope, as the PR's "Known limitations" says.

main added serve-57 (registered vision GGUF retains the Lemonade router)
and serve-58 (simple registered GGUF served directly by the GPU backend)
to model_serving.feature, colliding with this branch's serve-57/serve-58.

Keep both pairs: main's retain 57/58 and this branch's scripted-engine
regressions move to serve-59 and serve-60 in the same relative order, so
feature_naming's position-equals-index rule still holds with no gaps. The
@id: tags are untouched, since expectations.toml and the step bindings key
on those rather than on the numbers.

Swept the numbered references and repointed them at 59/60: the two
cross-references inside the scenarios' own comments, the two bullets and
the repro command in docs/testing.md, and the doc comment in
service_launch/readiness.rs. The repro filter becomes
`-n "serve-(59|60)"`; left as `serve-5[78]` it would have matched main's
two new @requires-gpu scenarios, which the documented
HIP_VISIBLE_DEVICES="" run skips, so the command would have run nothing
and looked clean.

Merge only: no review feedback is addressed here, and the three review
fixes at f8f09e2 are carried through unchanged.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main — third renumbering of this PR's two scenarios (57/58 → 59/60)

Merge only. No review item is addressed by this push; the three review fixes already on the branch are carried through untouched (see below).

The conflict

One conflict, in tests/e2e-cucumber/features/model_serving.feature. main appended two scenarios at exactly the index this PR's two already occupied:

main this PR (before) this PR (after)
57 serve-lemonade-complex-registered-inference — registered vision GGUF retains the Lemonade router serve-scripted-death-after-startup —
58 serve-lemonade-simple-registered-direct-route — simple registered GGUF served directly by the GPU backend serve-scripted-external-stop-while-attached —
59 — — serve-scripted-death-after-startup
60 — — serve-scripted-external-stop-while-attached

Resolved by keeping both pairs: main's retain 57/58 and sit first, this PR's move to 59/60 in the same relative order. feature_naming's position-equals-index rule holds with no gaps — proven by gate 5 below, not by eye.

@id: tags are unchanged. expectations.toml and the step bindings key on those, not on the numbers, so nothing downstream moved.

Every numbered reference updated

Swept with grep -rn "serve-5\|serve-6" (note: not serve-5[0-9], which silently misses a bracketed filter regex like serve-5[78] — that is how a stale one survived a previous round). Each hit was classified as main's or this PR's before editing; main's new scenarios are referenced only by @id: anywhere else in the tree, so every numbered serve-57/serve-58 hit was this PR's.

  • model_serving.feature — the two scenario titles, plus the two cross-references inside serve-60's own comment block (AttachOutcome::ServerExited arm serve-59, and "like serve-59's crash").
  • docs/testing.md — the two bullets under "Managed-engine crash vs. stop (ROCMAI-541)".
  • docs/testing.md — the repro filter, now HIP_VISIBLE_DEVICES="" cargo xtask e2e -- -n "serve-(59|60)". This one mattered: left as serve-5[78] it would now match main's two new scenarios, which are @requires-gpu and therefore skipped under the documented HIP_VISIBLE_DEVICES="" — the documented command would have run zero scenarios and exited clean. I checked the new regex against every scenario name in every .feature file: it matches exactly serve-59 and serve-60 and nothing else. Alternation is already the house style here (cf. update-(09|10)).
  • apps/rocm/src/service_launch/readiness.rs:484 — doc comment, (serve-58) → (serve-60).
  • The PR description, corrected to 57/58 last round, now says 59/60 (8 occurrences, including its own -n '(serve-59|serve-60)' command line).

The three review fixes survived

main has not touched spawn.rs, service_control.rs or managed_service_status.rs since the merge base, so no hunk competed with them. git diff f8f09e2dd over those three files is empty — the Windows launch-path liveness fix, the settle_pending_stop_key_cleanup marker move and their tests are byte-identical. readiness.rs differs by the one-line comment renumber above and nothing else.

Gates (Linux devbox — this Mac cannot build the workspace)

Run one at a time, each from a clean sync of the merge result:

gate result
cargo fmt --all --check pass
cargo clippy --workspace --all-targets -- -D warnings pass
cargo clippy --workspace --all-targets --features e2e-test-hooks -- -D warnings pass
cargo clippy -p e2e-cucumber --test e2e -- -D warnings pass (run separately; the --all-targets sweep false-greens the e2e step files)
cargo test -p e2e-cucumber --test feature_naming 4 passed — the renumbering proof
cargo xtask check-architecture-doc / check-crate-edges / check-module-size pass
cargo test -p rocm --bin rocm 1894 passed, 0 failed, 8 ignored

No flakes hit.

Why this keeps happening

This is the third collision on the same two scenarios (55/56 → 56/57 → 57/58 → 59/60). The cause is structural, not bad luck: both these scenarios and everyone else's are appended to the end of model_serving.feature, which is a hot file, and the index is positional — so any two branches that append concurrently collide, and the loser has to renumber and chase every prose reference to the old numbers. The first two rounds each left stale prose behind that a reviewer had to catch. Worth considering whether the positional-index rule could key on @id: instead, or whether new scenarios should be inserted by topic rather than appended.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

placeholder

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Correction: the previous review posted by this same automated process on this pull request, consisting only of the word "placeholder", was posted in error due to a tool-usage mistake and should be disregarded. This review below is the intended one.)

Review — needs work

Full review of the whole change (all 24 touched files, read completely at the diffed commits, not just the hunks). Implements ROCMAI-541 (an internal AMD tracker identifier; not retrievable from here).

Blocking

  • A crash in the fresh-launch startup window is overwritten back to stopped by its own cleanup path — apps/rocm/src/service_launch/spawn.rs:87-99 (settle_startup_death) and :422-433 (spawn_managed_engine_child's Err arm), composed with apps/rocm/src/service_launch/launch_transaction.rs:159-211 (abandon_failed_managed_launch_with) and crates/rocm-core/src/service_record.rs:633-636 (mark_launch_abandoned).
    settle_startup_death correctly merge-writes "failed" into both the manifest and the engine-state file and returns Crash, which becomes an Err bubbled up through launch_managed_engine_child_locked. spawn_managed_engine_child's Err arm then unconditionally calls abandon_failed_managed_launch(paths, &record) with the stale in-memory record captured before the crash (status still "starting"). That cleanup clones it, calls mark_launch_abandoned() (sets status "stopped" — by its own doc, deliberately, to keep rocmd's recovery scan from respawning it), and does an unconditional whole-file abandoned.write(), clobbering the "failed" that was merged into the manifest moments earlier. The engine-state file is untouched by this cleanup and keeps "failed", leaving a permanently self-inconsistent record that refresh_from_engine_state can never reconcile (it only re-reads the engine-state file when the manifest status is starting/running/recovering/ready). This is exactly the "Unix post-spawn early exit" row the PR's own description lists as fixed, and the mirror-image sibling path this PR added for restarts (service_control.rs's settle_restart_startup_death/update_restarting_record) explicitly uses a CAS-style guarded write to avoid this same clobber — the fresh-launch path has no equivalent guard. The existing test (a_child_lost_in_the_startup_window_is_marked_failed_unless_a_stop_asked_for_it, spawn.rs:803) calls settle_startup_death directly and never goes through spawn_managed_engine_child's full Err path, so it passes despite the clobber.
    Confidence 90 · architectural · Fix: either make the Crash verdict skip/short-circuit the record-clobbering write (merge rather than mark_launch_abandoned+write()), or give abandon_failed_managed_launch the same guarded-write discipline update_restarting_record uses. Add a test that drives a crashing child through the full spawn_managed_engine_child call and asserts the final on-disk manifest status is "failed".

  • The stop's own durable evidence (settled_stop_unix_ms) can be clobbered back to unset by the explicit stop path itself — apps/rocm/src/service_control.rs:270, inside stop_internal_managed_service_confirmed_with (:188).
    record.settled_stop_unix_ms = record.stop_requested_unix_ms.take(); runs on a record freshly re-read from disk after terminate(). If a concurrent rocm services/rocm dash/rocmd listing's liveness refresh (managed_service_status.rs:286-287) observes the same signalled PIDs already dead first and performs this exact same move itself (stop_requested_unix_ms → None, settled_stop_unix_ms → Some(ts)), then by the time this function's own re-read happens, stop_requested_unix_ms is already None. The line above still runs unconditionally: .take() on an already-None field returns None, and that None is assigned into settled_stop_unix_ms, erasing the Some(ts) the concurrent refresh had just correctly written. The on-disk record ends up with both fields unset — the exact "crash indistinguishable from a stop" bug class this PR exists to close, reachable through the explicit stop path's own confirmation step rather than the liveness-refresh path it was written to guard against.
    Confidence 85 · logic (race condition) · Fix: only move the marker when one is actually present, e.g. if let Some(ts) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(ts); }, leaving an already-settled value alone.

  • No test anywhere in service_control.rs asserts settled_stop_unix_ms's actual value after a confirmed stop — apps/rocm/src/service_control.rs (whole mod tests).
    settled_stop_unix_ms is this PR's central fix for distinguishing a stop from a crash, but every confirmed-stop test in this file (services_stop_reports_a_confirmed_stop_as_stopped, a_confirmed_stop_clears_the_recorded_processes_so_a_second_stop_signals_nothing, stop_final_write_keeps_what_others_wrote_while_it_ran, a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it) only checks stop_requested_unix_ms == None, never settled_stop_unix_ms. Reverting line 270 to the old record.stop_requested_unix_ms = None; would not fail a single test in this file — behaviour this change introduces, untested, and the direct reason the previous finding shipped undetected.
    Confidence 90 · test-quality · Fix: add settled_stop_unix_ms assertions to the confirmed-stop tests, and extend a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it to assert on_disk.settled_stop_unix_ms == Some(_) (which would currently fail given the finding above).

  • settle_exited_service's "was this deliberate" verdict and the record it stamps come from two independent, unsynchronized reads — apps/rocm/src/service_launch/readiness.rs:107-145.
    let deliberate = managed_service_stop_was_requested(paths, service_id); performs its own file read; ManagedServiceRecord::read_manifest(paths, service_id) immediately after performs a second, independent read. A stop marker written in the gap between the two pins deliberate = false from the stale first read while the second read could already show the record live, stamping "failed" for a death that was in fact just requested. New code (the whole --wait crash-vs-stop distinction this PR adds), with no comment acknowledging this particular window, unlike the sibling checks elsewhere in this same file that discuss exactly this class of race at length.
    Confidence 75 · logic · Fix: derive deliberate from the same manifest snapshot already read for latest/on_disk, instead of a separate call to managed_service_stop_was_requested.

  • StopWatch's doc comments describe a "status term" in managed_service_stop_was_requested that this same PR deleted — apps/rocm/src/service_launch/readiness.rs:546 ("That case is caught by the status term in managed_service_stop_was_requested") and :567 ("The status term still backs the one-shot check at the end of a wait"), versus the actual implementation at :499-505: record.stop_requested_unix_ms.is_some() || record.settled_stop_unix_ms.is_some() — no status term anywhere. The PR's own stated purpose is removing exactly that status == "stopped" term as unsound; both the function and the StopWatch doc that still refers to it are new lines in this diff, so this is a self-contradiction introduced within one change, not inherited drift. A maintainer trusting StopWatch's doc would believe a safety net exists that was deleted by the very same PR — precisely the kind of comment that later causes a regression when someone "cleans up" what looks like dead code on its word.
    Confidence 85 · mechanical (doc/comment accuracy in new code) · Fix: update StopWatch's doc to name the actual residual-race closer (settled_stop_unix_ms durability, or whatever the intended remaining gap is), not a "status term" that no longer exists.

  • A test asserting a guard's effect can't actually fail if that guard is deleted — apps/rocm/src/service_launch/readiness.rs:1409-1416 (test a_wait_death_is_a_crash_unless_a_stop_asked_for_it, third arm), exercising the else if latest.status == on_disk { return deliberate; } guard at :140-142. In this arm is_live(&latest) is already false, so neither branch of the preceding if is_live mutates latest.status; whether the guard early-returns or is deleted and falls through to write(), the write is status: "failed" either way, and the test's only assertion (after.status == "failed") passes identically both ways. Per this project's own test-quality bar, mutating a single branch inside the change and staying green means the test does not cover that branch.
    Confidence 80 · logic/test-quality · Fix: assert something the guard actually controls — e.g. that an unrelated field written to disk between the two reads survives the write (demonstrating the guard avoids a stale-snapshot overwrite), rather than a result identical either way.

  • A GPU-refusal waiver derived variable was not updated for the new scripted_engine_death bypass it claims to fully mirror — apps/rocm/src/serve_cmd.rs:906, with the exemption added at :885-889. The no-GPU bail gained a third bypass term (scripted_engine_death), but let gpu_refusal_waived_for_reuse = reuse_existing && !scripted_backend_failure;'s own comment ("the scripted backend failure is the one remaining term") is now false — it omits !scripted_engine_death. Traced the consumer at apps/rocm/src/service_launch/spawn.rs:367: if gpu_refusal_waived_for_reuse { bail!(...) }, fired when "nothing live to reuse" is discovered at spawn time. If reuse_existing and scripted_engine_death were ever both true (not exercised by the current e2e scenarios, which always do a fresh launch), the flag would wrongly read true and hard-bail instead of reaching the scripted-death path, defeating the feature for that combination.
    Confidence 78 · logic · Fix: add && !scripted_engine_death to the expression and correct the comment's "one remaining term" claim.

  • The remediation-command test proves the named commands parse, not that they clear the condition that printed them — apps/rocm/src/cli.rs:2676 (the_log_commands_the_crash_messages_name_actually_parse). This project's own standard for a remediation hint requires the named command to "actually clear the condition that printed it," asserted together with the resulting state in the same test. This test (a real improvement over nothing) only asserts rocm services logs <id> / rocm logs --service <id> parse via clap; the only other related check found (readiness.rs:1498-area) only asserts the printed message contains the command text, not that running it against a failed record prints that record's log. Could not confirm or rule out stronger coverage in tests/e2e-cucumber/features/logs.feature (out of this review's deepest scope).
    Confidence 70 · test-quality · Fix: either point to an existing e2e scenario that closes this gap in a comment near the test, or add one that seeds a failed record with a known log tail and asserts the named command actually prints it.

Non-blocking

None beyond the items below.

Decisions for the author

  • Several of the blocking findings above share a root enabler worth addressing as a class, not just as point-fixes — non-blocking-improvement. The PR's "shape of the fix" is seven independent entry points that each owe the same verdict, and its own "Known limitations" already names the risk ("No systematic entry-point parity test. Three of the gaps above were the same shape and each was found by review rather than a test.") Two of the findings above (the fresh-launch clobber, and the explicit-stop-path clobber) are exactly that pattern recurring a fourth and fifth time, in each case because a call site did a plain/unconditional write where a sibling call site (the restart path's CAS guard, mark_service_failed's merge-write) got the same problem right. Worth considering a single shared "stamp this verdict without clobbering what's already there" primitive used by every entry point, plus a test that drives each real entry point end to end (as the existing a_child_lost_in_the_startup_window... test does not, per the first finding) rather than unit-testing each settle_*/mark_* helper in isolation.
  • engine_ipc.rs's in-child stop-detection (engine_stop_was_deliberate_in, checking the engine's own state file plus stop_requested_unix_ms) is a separate, independently-built mechanism from the rest of the PR's central one (managed_service_stop_was_requested, checking stop_requested_unix_ms/settled_stop_unix_ms) — tradeoff. Investigated for a concrete bug (whether the early, pre-confirmation stop_requested_unix_ms signal it relies on could misclassify a genuine unrelated crash as deliberate during the window between the stop marker being written and the engine actually being signalled) and could not establish one with confidence: the window is plausibly intentional given how the Lemonade/vLLM adapters report a stop-triggered shutdown as an Err. Flagging only because it is a second, parallel implementation of "was this exit requested" with different inputs than the rest of the PR's carefully-documented mechanism; worth the author's own judgment on whether to unify them.

Positive signals

  • crates/rocm-core/src/managed_child.rs's pid_child_is_running test seam lets the Unix-zombie-vs-Windows-Unqueryable distinction — the entire point of ManagedChildLiveness — be exercised on every platform via dependency injection, not just on a real Windows CI run.
  • The scripted-engine e2e scenarios (serve-59/serve-60, tests/e2e-cucumber/tests/e2e/serving_steps.rs) drive a real spawned child through the actual AttachOutcome::ServerExited production path and assert message text, exit code, and both the manifest's and engine-state's status together — the comments in these tests explicitly reason through "would this still fail if the production write were reverted," exactly the standard this review applies.
  • apps/rocm/src/cli.rs:2676's remediation test deliberately also asserts that the old, broken rocm logs <id> spelling still parses (so the original defect was silent), which is a sharper regression guard than simply testing the new spelling in isolation.
  • docs/architecture.md:329's entry for the new managed_child.rs explicitly cites PR #437 by number for the known Windows PID-reuse limitation it knowingly leaves unresolved — genuine prior-art awareness rather than a silently deferred gap.
  • The xtask check (assert_prebuilt_e2e_lanes_enable_test_hooks / the shipped-release-build scan in xtask/src/workflow_contract.rs) that the PR claims keeps the e2e-test-hooks feature out of shipped binaries was read directly and is a real, reasonably robust static check on CI workflow text, not a vague claim.

Deployment notes

  • This PR changes recorded service state semantics (stopped vs failed) that rocmd's recovery scan reads to decide whether to respawn a service; the PR's own text flags that whether rocmd should respawn a failed record is still an open question, pinned by a test rather than resolved. Operators relying on auto-recovery should be aware a crash during the fresh-launch startup window will currently still read stopped (see the first finding above), not failed, so it will not be picked up by that recovery path despite the PR's stated intent.
  • Windows paths (spawn_detached_with_log, the Windows restart early-exit path, ManagedChildLiveness::Windows) are explicitly disclosed by the author as unverified on real Windows hardware; only windows-debug CI exercises them.

What this covered

Every file in the 24-file diff was read completely at the reviewed commits (6f3185d97503554ec387f3a44d2979d457f3a4c9…7c0c281260e79e6f9e40f4c225067b62d75a4dab), plus the surrounding/callee code needed to judge each change in place (launch_transaction.rs, managed_service_status.rs, service_record.rs, engine_runtime_select.rs, and others not themselves in the diff). The review fanned out across independent workers grouped by coupling: core liveness/spawn (managed_child.rs, process.rs, lib.rs, service_record.rs, spawn.rs, start.rs, service_launch.rs), readiness+attach (readiness.rs, attach.rs), service_control+status (service_control.rs, managed_service_status.rs), CLI+rocmd recovery (cli.rs, engine_runtime_select.rs, serve_cmd.rs, rocmd/service.rs, rocmd/watchers/server_recover.rs), a history/prior-art pass (git log/blame plus PRs #437 and #960), and a docs pass (docs/architecture.md, docs/manual-testing.md, docs/testing.md, README.md) — all run as genuinely independent background workers blind to each other's results, each subsequently spot-verified directly against the source by the coordinating reviewer before being included above.

A shared concurrency limit on background workers (outside this review's control) meant two planned passes — the e2e-test-hooks/scripted-engine-hook group (engine_ipc.rs, the Cucumber step definitions and feature file) and the dedicated AGENTS.md-compliance pass — could not be scheduled as independent background workers after repeated attempts; both were instead conducted directly by the coordinating reviewer (same method, not independently blind) and are reflected above with that caveat. Module-size (all changed files comfortably under the 1,500-production-line cap), markdown-link discipline, and crate-layering (rocmd still never imports from rocm) were all checked directly and found clean. CI on the reviewed commit was a mix of completed/in-progress at review time (clippy, build-and-test, Windows build/test, and the Linux/Windows/Strix-Halo E2E lanes visible so far were green; a couple of GPU-hardware E2E lanes were still queued/in-progress) — not re-polled to completion as part of this review. Not reconciled against any discussion on this pull request.


🤖 Automated by an AI agent

…cleans up after

A crash in the fresh-launch startup window was reported to the user as a
clean stop, and the record was left permanently self-contradictory.

`settle_startup_death` merges `failed` into both the manifest and the
engine state file and returns `Crash`, which bubbles up as `Err`.
`spawn_managed_engine_child`'s `Err` arm then handed cleanup the
in-memory record captured *before* the spawn (still `starting`), marked
it abandoned and wrote the whole file — putting `stopped` back over the
`failed` recorded moments earlier, while the engine state file kept it.
`refresh_from_engine_state` will not re-read an engine state file behind
a `stopped` manifest, so the two could never agree again.

Cleanup now applies its verdict to the manifest as it reads on disk and
leaves a record that is already not live alone, so a recorded cause (and
any stop marker a concurrent `services stop` wrote) survives. A merge
rather than the restart path's CAS: declining to write would strand the
record at `starting`, still reserving its port. This also makes the
fresh-launch path agree with the two siblings that already leave a crash
at `failed`, with the recovery consequence noted at the call site.

The explicit stop path's own confirmation step could likewise erase
`settled_stop_unix_ms`: `= stop_requested_unix_ms.take()` assigns `None`
across when a concurrent liveness refresh already moved the marker,
ending the record with both fields unset. Guarded, and asserted by the
confirmed-stop tests, which previously checked only that the marker was
gone.

`settle_exited_service` took its deliberate/crash verdict from a
separate file read to the one it stamped; both now come from one
snapshot through a shared predicate. `StopWatch`'s doc no longer credits
a `status` term this change removes.

Signed-off-by: Roman Sirokov <roman@flowrl.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks — this was an accurate review. Five findings addressed in 433b9025f. CHANGES_REQUESTED should stay standing; several of your other blocking items are untouched and listed at the bottom.

Verified on Linux (the workstation here cannot build the workspace). Gates: fmt, clippy --workspace --all-targets, clippy --workspace --all-targets --features e2e-test-hooks, clippy -p e2e-cucumber --test e2e, test -p e2e-cucumber --test feature_naming, xtask check-architecture-doc / check-crate-edges / check-module-size, test -p rocm --bin rocm — all green (1897 passed).


Finding 1 — fresh-launch crash overwritten back to stopped

Correct, and it reproduced exactly as described. Wrote the test first and watched it fail against the unfixed code:

---- service_launch::spawn::tests::a_startup_window_crash_stays_failed_after_the_launch_cleanup stdout ----
thread '...' panicked at apps/rocm/src/service_launch/spawn.rs:929:9:
assertion `left == right` failed: the cleanup must not overwrite the crash it is cleaning up after:
  managed engine exited immediately with status exit status: 3; inspect .../lemonade-startup-crash-cleanup.log
  left: "stopped"
 right: "failed"
test result: FAILED. 0 passed; 1 failed; 1903 filtered out

Then fixed, and watched it pass:

test service_launch::spawn::tests::a_startup_window_crash_stays_failed_after_the_launch_cleanup ... ok
test result: ok. 1 passed; 0 failed; 1903 filtered out

It drives a real child (exit 3) through the whole spawn_managed_engine_child call via the existing TEST_SPAWN_HOOK seam, so it goes through settle_startup_death → bail! → the Err arm → abandon_failed_managed_launch, and asserts the final manifest status, the engine state file, and that the record no longer reserves its port — the bug was that the first two disagreed, so either alone would have missed it.

Which option, and why. Neither verbatim: abandon_failed_managed_launch now applies its verdict to the manifest as read from disk, and leaves a record that is already not is_live() with the status it has. A full CAS in the style of update_restarting_record is wrong here — that function declines to write at all when the record moved under it, which is right for a restart that can give up its turn, but this is the last write of a failed transaction, and declining would strand the record at starting: still is_live(), still reserving its port against every later launch, with nothing left to clear it. Merging releases the port unconditionally while leaving a recorded cause alone. mark_launch_abandoned()'s stopped still applies to abandonments with no recorded cause, which is what its doc is actually about.

One thing you did not have in view, and a decision I had to make. An existing test, a_failed_spawn_past_the_first_write_terminates_the_child_and_keeps_an_abandoned_record, asserted persisted.status == "stopped" for this exact path, with a comment reasoning that failed would be respawned by rocmd recovery (pinned on the rocmd side by abandoned_launch_record_is_not_a_recovery_candidate). So the stopped write was load-bearing, not only a clobber, and your requested outcome conflicts with it. I went your way and changed that assertion to failed, because the two sibling entry points — readiness::settle_exited_service and service_control::settle_restart_startup_death — already leave a crash at failed, so this makes the entry points agree rather than adding a new exposure, which is the parity point in your "Decisions for the author". The consequence is real and I have not hidden it: rocmd's recovery scan may now respawn a fresh launch that crashed at startup. That is the open question the PR's own "Known limitations" records; flagging it explicitly so you can overrule me if you would rather rocmd learn to skip a failed record that never reached readiness, which would be a separate change.

Does this close the earlier non-blocking note about the whole-record write? Partially, not fully. The write is no longer of a stale record — concurrent fields, including a stop marker written in the window, now survive because the base is a fresh read. But it is still a whole-file write() of a record read a moment earlier, under no lock, so a writer landing between that read and the write still loses. The unlocked merge_json_state_file / read-modify-write pattern remains open across this file.

Finding 2 — settled_stop_unix_ms clobbered by the stop path itself

Correct, and to be direct about provenance: that line was introduced by an earlier round of work on this PR, not inherited. It is service_control.rs:270 in the diff you reviewed, part of the same change that added the field's new meaning.

Guarded as you suggested. Mutation evidence is under finding 3.

I also checked the sibling move in managed_service_status.rs's settle_pending_stop_key_cleanup and found something worth reporting: the guarded form was applied there too, with a comment claiming it closes the mirror-image race — but it cannot fire. That function returns early at stop_requested_unix_ms.is_none(), so the marker is always Some by the time the move runs. I kept the guarded spelling (it states the precondition locally and keeps the line correct if that early return is ever relaxed) but rewrote the comment to say so rather than claim a race it does not close. The test I added there asserts the outcome, not the line, and I have said in its doc that it stays green if either protection is removed and goes red only when both are — because the guard is not independently falsifiable.

Finding 3 — no test asserted settled_stop_unix_ms

Correct. Assertions added to the confirmed-stop tests, including a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it as you asked. Reverted finding 2's guard to the original record.settled_stop_unix_ms = record.stop_requested_unix_ms.take(); to confirm they bite:

test service_control::tests::a_listing_that_completes_the_stop_before_finalization_does_not_unconfirm_it ... FAILED
thread '...' panicked at apps/rocm/src/service_control.rs:1427:13:
ready: finalizing a stop a listing already settled must keep the evidence, not clear it
test result: FAILED. 61 passed; 1 failed; 1842 filtered out

Restored → green. That is your predicted failure exactly: the listing settles first, finalization moves a marker that is no longer there.

The new managed_service_status.rs test, under the composed mutation (early return removed and guard reverted):

assertion `left == right` failed: a stop already settled must survive the next listing,
  or the pair of them ends with the same both-fields-unset record a crash leaves
  left: None
 right: Some(1)

Finding 4 — two unsynchronized reads in settle_exited_service

Correct. deliberate is now read off the same snapshot that gets stamped, through a record_shows_stop_was_requested(&record) predicate that managed_service_stop_was_requested is also built from, so the two cannot be spelled differently. The unreadable-manifest branch still falls back to the standalone read, which is sound there precisely because that branch writes nothing.

Mutation — re-spelling it inline as marker-only, which is the drift the split exists to prevent:

thread '...a_wait_death_is_deliberate_on_either_half_of_the_stop_evidence' panicked at readiness.rs:1502:
a stop is a stop whether or not a listing has already settled its marker (consumed=true)
test result: FAILED. 0 passed; 1 failed; 1904 filtered out

Restored → green. Being plain about what that does and does not prove: it proves the predicate discriminates on both halves of the evidence. It does not prove the interleaving is gone — the fix is that there is now only one read, which is structural, and no read-injection seam exists here to drive the race single-threaded. I did not add one, and the test's doc says this rather than implying coverage.

Finding 5 — StopWatch doc citing a term this PR deletes

Correct, and a self-contradiction within one diff as you say. StopWatch's doc now names the actual residual-race closer: a confirmed stop moves its marker into settled_stop_unix_ms rather than dropping it, so the one-shot check at the end of a wait still finds evidence the latch missed. The remaining gap is now named rather than hand-waved — reset_for_restart deliberately clears both fields, so a record reset for relaunch between the stop and the end of the wait leaves nothing for either check.

I re-read the surrounding comments and found one more reference you did not cite: the inline comment in StopWatch::sample (:567 in your numbering) justified using the marker alone because of the status term. With that term gone the stated reason was vacuous, so I replaced it with the narrower reason that actually still holds — this latches forever, so it may only fire on evidence written during the watch, and settled_stop_unix_ms is durable by design and can outlive the stop that wrote it. The one-shot check does take the durable field and accepts the mirror-image exposure; that trade is now written down at both ends.

This has no test. It is a comment-only change and I am not claiming coverage for it.


Still open — not dismissed

  • The unlocked merge_json_state_file / read-modify-write pattern. Finding 1's fix narrows the window but does not close it; no locking added.
  • Your finding on a_wait_death_is_a_crash_unless_a_stop_asked_for_it's third arm (the guard whose deletion the test cannot detect) — untouched.
  • Your finding on gpu_refusal_waived_for_reuse missing && !scripted_engine_death at serve_cmd.rs:906 — untouched.
  • cli.rs:2676 still proves only that the remediation commands parse, not that they clear the condition that printed them. No e2e scenario pointed at yet.
  • The engine-dispatch arm of the crash backstop remains untested.
  • The Windows cfg paths are still gated only by CI's windows-build-and-test; none of this was run on real Windows hardware.
  • Your "root enabler" point — a single shared "stamp this verdict without clobbering" primitive — is not done. Finding 1 is the fourth instance of the pattern and I fixed it as another point-fix; the class fix is still owed.

One pre-existing flake seen while verifying, unrelated to this change: therock::family::tests::resolve_family_uses_managed_runtime_before_host_detection failed in 1 of 5 full-suite runs under high parallelism and passes isolated (10/10 in its module). It reads the process-global ROCM_CLI_THEROCK_FAMILY while sibling tests in the same binary mutate the environment. Nothing in this change touches therock/.

@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Two review rounds landed here at older heads — caebb946 (nowycondro) and the juhovainio round. The head is now 433b9025f, with two fix commits after those reviews (f8f09e2dd, 433b9025f). Going through every finding against the current code, with what I checked.

nowycondro's round

  1. Windows post-spawn check used the crash-prone raw probe — fixed. spawn.rs:581-585 now passes || liveness.is_running(), and the signature takes the liveness answer rather than a pid, so passing process_is_running back in is a type error rather than a regression someone can reintroduce quietly.
  2. PR description named serve-55/56 — fixed, and it moved again since. You were right that it pointed at unrelated pre-existing scenarios. It has since been corrected, and after merging main the scenarios this PR adds are now serve-59 and serve-60 (not 57/58 — main has since taken 57 and 58 for the Lemonade registered-GGUF pair). The description and docs/testing.md's repro filter both say 59/60.
  3. settle_pending_stop_key_cleanup discarding the marker — fixed. The non-empty-recorded-process branch now moves it: if let Some(consumed_marker) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(consumed_marker); }. Worth noting the comment there is explicit that this guard cannot currently fire — the stop_requested_unix_ms.is_none() early return at the top already guarantees Some by that point — so it documents the precondition and keeps the line correct if that early return is ever relaxed. Pinned by a_second_listing_does_not_clear_the_stop_a_first_one_settled, which asserts the outcome rather than the line.
  4. E2E consolidated report (self-hosted) red — resolved. Your read of the mechanism was right (a cancelled lane uploads no report and the consolidator hard-fails). On the current head every check is green; nothing is failing or queued.

Non-blocking:

  • Doc block attached to the wrong function — fixed. You were exactly right: the ~46-line block describing the failure branch, the failed stamp and the true/false return was attaching to still_watching_this_supervisor, leaving finalize_managed_launch_readiness with no rustdoc at all. The block now sits directly above finalize_managed_launch_readiness, and still_watching_this_supervisor keeps its own.
  • settled_stop_unix_ms not asserted in service_control.rs's own tests — fixed. Those assertions are now in, including on_disk.settled_stop_unix_ms.is_some() in the confirmed-stop and listing-race tests.

juhovainio's round

  1. Fresh-launch crash overwritten back to stopped by abandon_failed_managed_launch — fixed in 433b9025f ("stop a failed launch's cleanup from erasing the crash it cleans up after"). This was the strongest finding in either round; the stale-in-memory-record clobber was real.
  2. settled_stop_unix_ms clobbered by the stop path's own unconditional .take() — fixed. service_control.rs now reads if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }, so an already-settled value is left alone instead of being overwritten with None.
  3. No test asserted settled_stop_unix_ms's value after a confirmed stop — fixed, as above.
  4. settle_exited_service's verdict and record from two unsynchronized reads — fixed. deliberate is now derived from the same latest snapshot via record_shows_stop_was_requested(&latest), not a second independent managed_service_stop_was_requested file read.
  5. StopWatch's doc describing a "status term" this PR deleted — fixed. The doc now describes stop_requested_unix_ms as the stop's live evidence and the move into settled_stop_unix_ms; the phantom status term is gone.
  6. The vacuous third arm of a_wait_death_is_a_crash_unless_a_stop_asked_for_it — still open, and you are right. I want to be plain about this one rather than claim it. The arm now carries a comment that itself concedes the problem ("a failed -> failed rewrite would be invisible"), but the assertion is still only after.status == "failed" on a record whose status was set to "failed" first, so deleting the latest.status == on_disk guard leaves it green. Acknowledging a dead assertion in a comment is not the same as making it discriminate. I will take your suggested shape — write an unrelated field to disk between the two reads and assert it survives, which is something only the guard controls — and will confirm it by deleting the guard and watching the test go red before I claim it fixed.

So: one blocking item genuinely open (juhovainio #6), everything else addressed. The doc-block move is not pushed yet; it goes up with that fix.

One thing worth flagging that neither round could have seen: this PR's serve-59/60 and #678's new serve-59 now both claim 59 on top of the same main. Whichever merges first forces the other to renumber, and feature_naming will catch it rather than letting it through silently.

Review finding: `a_wait_death_is_a_crash_unless_a_stop_asked_for_it`'s third
arm claimed to cover `settle_exited_service`'s `latest.status == on_disk`
early return, but asserted only that a record already reading `failed` still
read `failed` afterwards. Delete the guard and the write that follows stamps
`failed` over `failed`, so the assertion passed either way — the arm could
not fail for the reason it existed.

What the guard controls is not the status but whether the stale snapshot is
written back at all. The arm now plants a key on disk that
`ManagedServiceRecord` does not model: a read drops it and `write()`
re-serializes without it, so it survives exactly when the early return skips
the write. That turns the concurrent overwrite the guard exists to prevent
into something observable without racing for it.

Confirmed by deleting the guard: the new assertion fails with
`left: None, right: Some("concurrent")`, where before the whole arm stayed
green.

Also moves the ~46-line doc block that describes the failure branch, the
`failed` stamp and the true/false return onto
`finalize_managed_launch_readiness`, which it was written for. A missing
blank line had attached it to `still_watching_this_supervisor` — which has
its own doc — leaving the more intricate function with no rustdoc at all.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: the one blocking item I left open is now fixed, pushed as 75f27a79d.

juhovainio #6 — the third arm of a_wait_death_is_a_crash_unless_a_stop_asked_for_it. You were right that it could not fail for its stated reason, and the comment I had added only conceded the problem rather than solving it.

The fix follows your suggestion. What the latest.status == on_disk early return controls is not the status — a failed → failed rewrite is invisible — but whether the stale snapshot is written back at all. So the arm now plants a key on disk that ManagedServiceRecord does not model. The type does not deny unknown fields, so a read drops the key and write() re-serializes without it: the key survives exactly when the early return skips the write, and is gone the moment it does not. That makes the concurrent overwrite the guard exists to prevent observable deterministically, without having to win a race to see it.

Mutation evidence — deleted the guard:

} else if latest.status == on_disk {
    return deliberate;
}

and the arm goes red on precisely the new assertion:

assertion `left == right` failed: the refresh learned nothing, so the stale
snapshot must not be written back over whatever else reached disk meanwhile
  left: None
 right: Some("concurrent")

Restored, and green again. Before this change, that same deletion left the whole arm passing.

The same commit carries the doc-block move from nowycondro's non-blocking item: the ~46 lines describing the failure branch, the failed stamp and the true/false return now sit on finalize_managed_launch_readiness, which they were written for, instead of attaching to still_watching_this_supervisor.

Verified on Linux: cargo test -p rocm --bin rocm (1897), cargo fmt --all --check, clippy --workspace --all-targets, the same with --features e2e-test-hooks, clippy -p e2e-cucumber --test e2e, feature_naming, and check-architecture-doc / check-crate-edges / check-module-size.

That clears every blocking finding from both rounds. The serve-59/60 collision with #678 noted above is still the one thing to watch at merge time.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this fresh at the current head. I couldn't demonstrate a hard blocker from the diff, but a few things are worth settling before merge, so I'm requesting changes. Inline comments cover the specifics.

  • --wait race: when a concurrent liveness reconcile has already written stopped, --wait reports the crash but never stamps failed. That contradicts the README and the "mark failed on any error exit" criterion for that path, and the unit test doesn't cover that arm.
  • Windows handle inheritance: spawn_detached_with_log now inherits handles for every managed engine, which can keep a captured parent pipe open. This was raised in earlier rounds and is still unverified; the Windows CI jobs were still pending when I looked.
  • Recovery loop: a crashed fresh launch now stays failed, which makes it a rocmd recovery candidate, so an engine that crashes on every start can be respawned repeatedly. The PR lists this as an open question, but it ships in this change, so it should be decided here.
  • Docs drift: the managed_child.rs type doc is stale, and the PR body itself says three docs/architecture.md entries (readiness.rs, process.rs, engine_ipc.rs) are out of date; they should be fixed in the same change.
  • Tests/scenarios: serve-60 only asserts "not failed" rather than stopped, and background launch, restart, --wait and the new log-hint text are covered by unit tests only, with no scenario or stated reason.

AC1 and AC2 of ROCMAI-541 look met; AC3 is met except for the --wait gap above. This is a diff-only read; I did not build or run anything.

&serde_json::json!({ "status": "failed" }),
);
}
} else if latest.status == on_disk {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the record already reads non-live (e.g. a concurrent rocm services reconcile wrote stopped with no stop marker), this branch returns deliberate (false) without stamping anything. The --wait caller then reports a crash while the manifest stays stopped and the engine state is untouched, which contradicts the README's "a --wait death leaves the record failed". The attach path stamps via mark_service_failed regardless of status; consider doing the same here (guarded on !deliberate) and adding a test arm for a pre-reconciled stopped record.

program,
args,
env_overrides,
DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW | CREATE_UNICODE_ENVIRONMENT,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Handing std handles to CreateProcessW makes this spawn use bInheritHandles=TRUE for every managed engine. A parent whose own stdout/stderr is a redirected pipe (CI wrapper, script capturing rocm serve) can have that pipe held open by the long-lived engine, so the caller never sees EOF. This was raised in earlier rounds and is still unaddressed or untested; either scope inheritance with an explicit handle list or add a Windows test that a captured-pipe parent returns.

Comment thread crates/rocm-core/src/managed_child.rs Outdated
/// its *real* wait status rather than merely probing whether the pid still
/// exists.
///
/// On Windows there is no equivalent gap: [`process_is_running`] there opens a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale: this says the Windows variant defers to process_is_running, but is_running now goes through probe_process via pid_child_is_running, precisely to avoid that function's collapse of unqueryable into gone. Please update the sentence.

"an externally stopped attach must exit 0, got {rc}:\n{combined}"
);
assert!(
combined.contains("server process exited"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stop outcome is pinned by wording here and by assert_ne!(.., "failed") in the final step. Please assert the positive state (stopped) too, so the scenario cannot pass for any non-failed record.

// `rocmd`'s own restart-recovery watcher needs no such carve-out
// here. Only when neither is present is this an unexplained exit
// — i.e. a crash.
if stop_seen_while_attached || managed_service_stop_was_requested(paths, &service_id) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Background launch, rocm services restart and --wait also changed exit code/message but have no scenario (only serve-59/60 cover attach). Per AGENTS.md section 3, either add scenarios or state in the PR why they cannot run on the no-GPU lane.

Three conflicts.

tests/e2e-cucumber/features/model_serving.feature: main added its own
`serve-59` (the endpoint-key ownership scenario). Main's keeps 59; this
branch's two become `serve-60` and `serve-61` and move after it, so each
number still equals its position. Four prose references followed the
renumber: two scenario lists and the repro filter in `docs/testing.md`
(now `serve-(60|61)`), the `(serve-60)` note in
`service_launch/readiness.rs`, and the feature file's own
"exactly like serve-59's crash" line. The `@id:` tags are the binding key
and are unchanged, so no step binding moved.

docs/architecture.md: this branch adds a `managed_child.rs` row while main
rewrote the neighbouring `managed_runtime.rs` description to mention the
loader shims. Both are kept, and main's newer description is the one that
survives — taking this branch's side wholesale would have silently reverted
that doc update.

tests/e2e-cucumber/tests/e2e/serving_steps.rs: two pairs of independent step
definitions appended at the same place. Both sides of each pair are kept.
In both cases the split left each side's function body unterminated and
sharing one tail, so the tail is reproduced for each rather than closing only
whichever block happened to come last.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main; the PR is mergeable again at e00fac0f0. Three conflicts, and one of them moved this PR's scenario numbers again, so flagging it before it confuses anyone reading the earlier comments.

model_serving.feature. main has since added its own serve-59 (@id:serve-endpoint-key-readable-by-owner-only). Main's keeps 59; this PR's two scenarios are now serve-60 and serve-61 (they were 59/60, and before that 57/58 — the numbers have moved twice through no fault of their own, because the rule is that a scenario's number equals its position in the file).

The @id: tags are the binding key and are unchanged, so no step binding moved. Four prose references followed the renumber: two scenario lists and the repro filter in docs/testing.md (now HIP_VISIBLE_DEVICES="" cargo xtask e2e -- -n "serve-(60|61)"), the (serve-60) note in service_launch/readiness.rs, and the feature file's own "exactly like serve-59's crash" line.

docs/architecture.md. This branch adds a managed_child.rs row; main rewrote the neighbouring managed_runtime.rs description to mention the loader shims. Both kept, with main's newer description surviving — taking this branch's side wholesale would have silently reverted that doc update.

serving_steps.rs. Two pairs of independent step definitions appended at the same place; both sides of each pair kept. In both cases the conflict split left each side's function body unterminated while sharing a single closing tail, so the tail is reproduced for each rather than closing only whichever block came last.

Verified on Linux before pushing: feature_naming (the gate that catches a bad renumber), cargo fmt --all --check, clippy --workspace --all-targets, the same with --features e2e-test-hooks, clippy -p e2e-cucumber --test e2e, the three xtask checks, and cargo test -p rocm --bin rocm (1929 passed).


On the red CI: E2E tests (MI300X, current) and E2E consolidated report (self-hosted) are failing here, and on every other open PR in this repo right now. The job log gives the reason:

serving will fail with Failed to infer device type; this torch is built for a different ROCm version than the installed SDK

That is a prewarmed-runtime image problem, not a code one. The strongest evidence is that it reproduces identically on #1095, whose diff touches only rocm-dash-tui and cannot affect torch or the ROCm SDK. Documenting rather than re-running; it needs an infra fix, not a change here.

@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up on a scenario-number collision between this PR and a sibling, so whoever merges second is not surprised by a red feature_naming.

After main took serve-59 (the endpoint-key ownership scenario), both of these renumbered independently and now claim the same number:

Each is correct within its own branch, because the rule is that a scenario's number equals its position in its file. The collision only exists across them, so whichever merges first forces the other to renumber. That is cheap and safe: cargo test -p e2e-cucumber --test feature_naming fails loudly on a duplicate or out-of-position number, so this cannot slip through silently — it just needs doing, along with the prose references (docs/testing.md's scenario lists and repro filter, and apps/rocm/src/therock/family.rs for #678 / service_launch/readiness.rs for #854).

The @id: tags are the binding key and are unaffected either way, so no step binding moves.

No action needed now; I will renumber whichever branch lands second.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (25 files, +4105/-180 against base 9101286a, head e00fac0f). Implements ROCMAI-541 ("a managed rocm serve that crashes must say so"); the ticket itself is not retrievable from here (no tracker access), so this is judged against the PR's own description of the acceptance criteria. The description's "shape of the fix" table and its claims about what was added in review were checked line-by-line against the diff and held up, with one material omission: apps/rocmd/src/service.rs received the identical "moved, not dropped" stop-evidence fix applied to service_control.rs, but is named nowhere in the table or in "the evidence a stop leaves" narrative — and that undocumented instance is exactly where the fix's guard was dropped (see Blocking #2).

Reconciliation note (done after blind findings were locked): this PR carries a standing CHANGES_REQUESTED review (juhovainio, at commit 75f27a79, the last substantive commit before the current head — the only commit since is a merge from main, not new work responding to it). Three of its five points are independently reconfirmed by this blind review: the --wait race (Blocking #1), the stale managed_child.rs type doc (Blocking #3), and the weak assertion in the external-stop control scenario (Non-blocking, serving_steps.rs:4346). A fourth point (recovery respawn loop) is mitigated by an existing backoff guard in apps/rocmd/src/watchers/server_recover.rs that I confirmed independently, though the underlying design question ("should a failed record be a recovery candidate at all") is still left open by this PR, as the reviewer notes. A fifth point (Windows handle inheritance in spawn_detached_with_log) I was not able to independently verify or refute in this pass — flagging it as still open rather than asserting it either way. None of this changes any finding below; it is reported because the task asks this review to say plainly when a standing objection is unresolved on the current commit.

Blocking

  • settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash — apps/rocm/src/service_launch/readiness.rs:116-157 (early return at 152-154), interacting with apps/rocm/src/managed_service_status.rs:408-416.
    When a concurrent liveness-refresh reconciles a dying --wait launch's dead pids to status = "stopped" with no marker (which that function does unconditionally by its own doc comment: "dead pids alone do not distinguish a stopped service from a crashed one"), settle_exited_service reads latest.status == on_disk and returns deliberate = false without ever writing. died_during_wait still prints the correct crash message, but the persisted manifest is left reading "stopped", never corrected to "failed" — contradicting the PR's own acceptance criterion for this path. Traced directly, not hypothetical; no test constructs this starting state before calling the function.
    Confidence 85 · logic · Fix: in the else if latest.status == on_disk branch, correct status to "failed" (and merge the engine-state file) when !deliberate, instead of returning early without writing.

  • apps/rocmd/src/service.rs:137's unconditional marker move can erase a concurrent stop's already-correct evidence — unlike the guarded if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); } used for the identical operation in apps/rocm/src/service_control.rs:283-284 and in managed_service_status.rs's settle_pending_stop_key_cleanup, this call site does it unconditionally: record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();. The surrounding comment acknowledges the record is re-read because a concurrent liveness refresh "may have persisted something while the stop ran" — concretely, if that refresh already moved the marker (stop_requested_unix_ms → settled_stop_unix_ms) before this re-read, the unconditional .take() here (None on an already-None field) overwrites the just-written settled_stop_unix_ms with None, erasing the one piece of durable evidence the whole PR relies on to prove a death was requested — reproducing, through this one path, the exact crash-indistinguishable-from-a-stop state ROCMAI-541 exists to close. This file is not named anywhere in the PR's own entry-point table despite receiving the same fix.
    Confidence 90 · logic · Fix: guard it the same way as service_control.rs:283-284.

  • crates/rocm-core/src/managed_child.rs's own type-level doc comment (and from_pid's, line 79) is stale and self-contradicts the correct documentation 20 lines below it — the type doc (line 60) and from_pid's doc (line 79) say the Windows arm "defers to process_is_running"/reads the exit code with no qualification, while is_running's own doc (lines ~84-93) correctly says the Windows arm uses probe_process specifically because process_is_running collapses a transient OpenProcess failure into a false crash report — which is also what the code actually does. Within one new file, the first thing a reader sees documents the old, unsafe behavior as current. This repo's own AGENTS.md §3 requires a comment stating how code behaves be checked against that code before pushing; it demonstrably wasn't, within a single new file.
    Confidence 90 · mechanical (doc-accuracy) · Fix: update both stale doc comments to match is_running's.

Non-blocking

  • ManagedServiceRecord::write() (crates/rocm-core/src/service_record.rs:668) takes no lock, unlike merge_json_state_file/write_json_state_file (crates/rocm-core/src/endpoint_readiness.rs), which acquire a FileLock. A remaining plain .write() caller (e.g. stop_internal_managed_service_confirmed_with in service_control.rs) is not ordered against a concurrent merge, so a .write() landing in the merge's narrow read-serialize-rename window can still silently overwrite fields the merge just set. Narrower than the pre-PR behavior, not a regression, and consistent with the PR's own disclosed "merge_json_state_file is unsynchronized against the stop path" limitation. Confidence 70 · architectural · Fix: have write() take the same lock, or move every writer onto the merge path.

  • The new crash-vs-stop backstop in apps/rocm/src/engine_ipc.rs (record_engine_serve_http_failure/engine_stop_was_deliberate_in) is unreachable dead code for vLLM — engines/vllm/src/process.rs:162 calls std::process::exit() directly on a non-zero exit instead of returning Err, so run_builtin_engine_serve_http (engine_ipc.rs:361-408) never gets control back to run the new mechanism for vLLM. Not user-visible-incorrect today (vLLM already writes "failed" itself, just above, independent of the new mechanism), but the new mechanism silently covers only 2 of the 3 dispatched engines. Confidence 75 · architectural · Fix: make vLLM's serve_http return Err instead of exiting, or state the gap explicitly.

  • apps/rocm/src/engine_ipc.rs:406's new crash-recording call site is exercised by no test, and the adjacent test-module comment's pointer to "the PR body" for justification doesn't hold up — the PR's actual Tests/Known-limitations sections don't address this specific call site. Confidence 75 · logic (partially self-acknowledged gap) · Fix: either add a seam a unit test can reach without starting the background thread, or state this specific gap in the PR text per this repo's own AGENTS.md §3.

  • The external-stop control scenario's assertion is negative, not positive (tests/e2e-cucumber/tests/e2e/serving_steps.rs:4346: assert_ne!(status, "failed") rather than assert_eq!(status, "stopped")), so it would still pass if a regression left the record in some other wrong status. Confidence 85 · logic (test-quality) · Fix: assert the status equals "stopped" directly.

Decisions for the author

  • tradeoff — apps/rocm/src/service_launch/start.rs:181-199: a mid-wait external stop makes rocm serve --wait report Ok with status: "stopped" and exit 0, i.e. the CLI's own wait "succeeds" by reporting a stop. Consistent with the attach path's wording, but worth the author explicitly confirming this is the intended UX for a foreground --wait the user is actively watching.
  • non-blocking-improvement — the "was this death requested?" sequence (check stop evidence → still_watching_this_supervisor → mark_service_failed → bail) is independently hand-written in both attach.rs's ServerExited arm and readiness.rs's finalize_managed_launch_readiness/start.rs, sharing only the low-level primitives, not the orchestration. No drift found today, but nothing forces a future change to be applied in both places. A shared helper would close this structurally.
  • answered-by-intent — the PR body states: "docs/architecture.md gained managed_child.rs, but the readiness.rs, process.rs and engine_ipc.rs entries still describe their older contents." This would otherwise be a blocking doc-staleness finding for three files that changed substantially (readiness.rs alone gained ~1127 lines); it's retired by this explicit disclosure. Confirm this is still the plan to fix promptly, not indefinitely.

Positive signals

  • StartupDeath (spawn.rs) and ManagedChildLiveness (managed_child.rs) are clean, minimal enums replacing boolean blindness for "why did this exit," with tests that drive all three probe outcomes on every platform rather than only the one a real Windows CI run could construct.
  • settle_exited_service's redesign to read stop evidence from the same snapshot it stamps (rather than a second independent read) correctly closes a documented double-read race its own doc comment describes.
  • Exit-code propagation was traced end-to-end (not just "it calls bail!"): both attach.rs's and finalize_managed_launch_readiness's error paths reach main.rs's exit-code mapping to ExitCode::FAILURE un-swallowed.
  • The guarded "moved, not dropped" pattern in service_control.rs and managed_service_status.rs is correctly implemented and has a dedicated regression test for the exact race it defends against — which is what makes the missing guard in apps/rocmd/src/service.rs (Blocking #2) stand out as an omission rather than an unknown risk.
  • The e2e-test-hooks scripted-engine mechanism is double-gated (compile-time Cargo feature and a runtime env var), backed by a real xtask release-workflow check (the_release_workflow_builds_no_feature_gated_binaries) that scans for any enabled feature, not just this one.
  • rocmd's recovery watcher already has a restart-backoff guard, so the new failed-as-recovery-candidate behavior doesn't create an unbounded crash-respawn loop.

Deployment notes

  • Behavior change: a crashed fresh launch now reads failed (previously stopped) and becomes a rocmd recovery candidate in cases that previously were not retried at all — worth watching after rollout, independent of the open "should it be" design question above.
  • Remediation hint text changed from a bare rocm logs <id> (which never selected a service) to rocm logs --service <id> / rocm services logs <id>; both verified as valid, existing subcommands.

What this covered

Full review of the whole change, fanned out into 9 parallel independent workers grouped by coupling: service_launch/{spawn,readiness}.rs; service_launch/{attach,launch_transaction,start}.rs + service_launch.rs; service_control.rs; rocm-core/{managed_child,process,service_record,lib}.rs; rocmd/{service,watchers/server_recover}.rs; cli.rs+serve_cmd.rs; engine_ipc.rs+engine_runtime_select.rs+managed_service_status.rs; the new/changed e2e tests; and the docs (README/architecture/manual-testing/testing). Each worker read its assigned files completely plus directly relevant surrounding code, not just diff hunks. Driver-level passes: full history of the 43 commits unique to this branch (extensive iterative churn specifically on the stop-vs-crash race — consistent with the defects found above), impact-analysis greps for every new/renamed identifier across the whole repo (not just the diff), and direct re-verification of the PR's own table and several "Known limitations" claims rather than trusting them, per this task's explicit instruction.

One worker (cli.rs/serve_cmd.rs — exit-code and hint-text review) did not deliver its final report back to me before this review had to be submitted; I independently spot-checked its core claims myself (exit-code propagation traced via a sibling worker's cross-file trace to ExitCode::FAILURE; both hint-text forms confirmed as valid existing subcommands; no stray bare rocm logs <id> hint found anywhere in the diff) and found nothing, but that file group did not get the same from-scratch independent pass the other eight did — treat that area as lighter-touch than the rest.

Did not run: cargo build/test/clippy (no Rust toolchain in this environment) — relied on the PR's own CI, which is fully green on every required check at the reviewed commit (clippy, build-and-test, Test (affected crates), the mock E2E tests job, Windows build/test, docs build); one self-hosted hardware lane (E2E tests (Strix Halo, Windows, current)) shows cancelled rather than failure, consistent with the self-hosted-runner availability gaps this branch's own commit history separately discusses, and is not treated as a blocking signal. Ticket ROCMAI-541 is not retrievable from this environment (no tracker access granted) — judged against the PR's own description of the acceptance criteria instead. Reconciliation against standing review discussion was done only after blind findings were locked, per the method, and is reported in the narrative above rather than folded into any finding's confidence score.


🤖 Automated by an AI agent

}
} else if latest.status == on_disk {
return;
return deliberate;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash.

When the on-disk record is already non-live (managed_service_is_live(&latest) is false) and latest.status == on_disk, this returns deliberate without ever writing. refresh_managed_service_runtime_liveness_with (apps/rocm/src/managed_service_status.rs:408-416) reconciles any record with dead tracked pids to status = "stopped" with no marker — including a "starting" record mid---wait — by its own doc comment's admission ("dead pids alone do not distinguish a stopped service from a crashed one"). So if a concurrent rocm services/rocm dash read (or rocmd's own recovery tick) reconciles a dying --wait launch to "stopped" before this function runs, deliberate is false (correctly — no stop markers exist) and died_during_wait prints the correct crash message, but the persisted manifest is left reading "stopped" and is never corrected to "failed". This directly contradicts the PR's own acceptance criterion ("the engine state file is marked failed on any error exit") for this one path.

This matches a standing, currently-unresolved reviewer objection on this exact area (the --wait race) — no commit since that review has touched this function.

Suggested fix: in the else if latest.status == on_disk branch, correct status to "failed" (and merge the engine-state file) when !deliberate, rather than returning early without writing; only the "nothing to add, preserve an already-failed record's own reason" case needs the pure early return.

Confidence: 85 · logic · blocking.

Comment thread apps/rocmd/src/service.rs Outdated
// crash and was stamped `failed` — the exact symptom ROCMAI-541 exists
// to remove, reintroduced through the one stop path that is not
// `rocm services stop`.
record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — unconditional marker move can erase a concurrent stop's already-correct evidence.

record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();

This is unconditional, unlike the equivalent "moved, not dropped" logic everywhere else in this PR (apps/rocm/src/service_control.rs:283-284's if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }, and managed_service_status.rs's settle_pending_stop_key_cleanup, both guarded).

The surrounding comment (lines 95-99) explicitly acknowledges the record is re-read fresh here because another writer — "rocm's liveness refresh" — may have persisted something while the stop ran. Concretely: (1) this stop signals the pids and they die; (2) concurrently, a rocm services list/rocm dash read reconciles the dead pids and — correctly, via the guarded code elsewhere — moves stop_requested_unix_ms into settled_stop_unix_ms, writing None/Some(ts) to disk; (3) this function's re-read at line ~100 now sees stop_requested_unix_ms == None, settled_stop_unix_ms == Some(ts); (4) all_stopped is still true (computed from this stop's own pid-tracking, independent of the record read); (5) the unconditional .take() here assigns None over the already-correct settled_stop_unix_ms, erasing the one piece of durable evidence the whole PR relies on to prove this death was requested.

The in-code comment one line above this ("the same move rocm services stop makes on its confirmed path, and for the same reason") claims parity with the guarded version, but the guard itself is missing here — and this file/path is not named anywhere in the PR's own "shape of the fix" table or "the evidence a stop leaves" narrative, despite receiving the identical fix.

Suggested fix: guard this exactly like service_control.rs:283-284: if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); }.

Confidence: 90 · logic · blocking.

Comment thread crates/rocm-core/src/managed_child.rs Outdated
/// its *real* wait status rather than merely probing whether the pid still
/// exists.
///
/// On Windows there is no equivalent gap: [`process_is_running`] there opens a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this type's own doc comment (and from_pid's, line 79) is stale and self-contradicts the correct documentation 20 lines below it.

This comment and from_pid's say the Windows arm "defers to process_is_running" / "opens a fresh handle... reads the real exit code" with no further qualification. But ManagedChildLiveness::is_running's own doc comment (lines ~84-93) correctly and explicitly says the opposite: "The Windows arm asks probe_process rather than process_is_running... process_is_running collapses Unqueryable into 'not running'... Unqueryable therefore reads as 'may still be running'" — which is also what the actual code at the bottom of this file does (pid_child_is_running(*pid, probe_process)).

So within this single new file, two doc comments describe the old, less-safe process_is_running-based behavior as current, while the correct one — on the method that actually matters — documents the fixed behavior. A future maintainer reading only the type-level doc (the first thing anyone reads) would believe this type behaves the unsafe way ROCMAI-541 exists to move away from.

This matches a standing, currently-unresolved reviewer objection ("the managed_child.rs type doc is stale") — no commit since that review has touched this file.

Fix: update this comment and from_pid's (line 79) to describe the probe_process/Unqueryable-aware behavior, consistent with is_running's doc.

Confidence: 90 · mechanical (doc-accuracy) · blocking (this repo's own AGENTS.md §3 requires a doc/comment stating how the code behaves be checked against that code before pushing, and it demonstrably was not, within a single new file).

)),
};
if let Err(error) = &result {
record_engine_serve_http_failure(&service_id, &state_path, log_path.as_deref(), error);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — this new crash-recording call site is exercised by no test, and the test module's own comment pointing at "the PR body" for justification doesn't hold up.

The test module (lines ~624-635) says this arm's wiring is "still uncovered — see the PR body," because reaching it requires spawn_alongside_engine, which starts a background thread a unit test must not do. The e2e scripted-engine hook doesn't reach it either: it takes the early-return branch at lines 332-340, which is structurally parallel to but textually separate from this post-match call site.

I checked the PR's actual description: its "Tests" and "Known limitations" sections don't address this specific call site (they discuss readiness/restart/--wait being unit-tested only, and a different limitation about SIGTERM'd Lemonade/vLLM adapters) — so the comment's pointer to "the PR body" for this particular gap isn't actually substantiated there.

Per this repo's own AGENTS.md §3, an untestable path needs its gap stated in the PR text with a reason; the code comment attempts an argument (background-thread-in-a-unit-test) but the PR body itself doesn't carry it.

Confidence: 75 · logic (test-coverage gap, partially self-acknowledged) · non-blocking.

);
}

#[then("the service record does not read failed")]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — the external-stop control scenario's record-state assertion is negative (assert_ne!(status, "failed")) rather than positive (assert_eq!(status, "stopped")).

assert_ne!(record["status"], "failed", "manifest wrongly marked failed: {record}");
assert_ne!(engine_state["status"], "failed", "engine state wrongly marked failed: {engine_state}");

This is the one scenario specifically designed to prove a deliberate stop is not misclassified as the crash serve-scripted-death-after-startup exercises. As written, it would still pass if a regression left the record in some other wrong status (e.g. stuck "starting") as long as it isn't literally "failed" — it doesn't confirm the record actually reads "stopped". This matches a standing, currently-unresolved reviewer objection on this exact scenario; no commit since that review has strengthened this assertion.

Suggested fix: assert record["status"] == "stopped" (and the engine state, if it tracks status) directly, in addition to or instead of the negative check.

Confidence: 85 · logic (test-quality) · non-blocking.

Main landed ROCMAI-703 (#1050), which adds its own serve scenario and its own
testing.md section, so both conflicts are with it.

tests/e2e-cucumber/features/model_serving.feature: main took `serve-60`.
Main's keeps 60; this branch's two become `serve-61` and `serve-62` and move
after it, so each number still equals its position. Four prose references
follow: the two scenario entries and the repro filter in `docs/testing.md`
(now `serve-(61|62)`), the `(serve-62)` note in
`service_launch/readiness.rs`, and the feature file's own
"exactly like serve-60's crash" line. The other `-n "serve-60"` filters in
docs are main's own and are left alone.

docs/testing.md: two independent sections appended at the same place, kept
whole. Unnumbered headings, so nothing renumbers.

This is the third time these two scenarios have moved (57/58 -> 59/60 ->
60/61 -> 61/62); the `@id:` tags are the binding key and have never changed,
so no step binding moved with them.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Correction to my earlier note: the colliding number has moved.

main has since landed ROCMAI-703 (#1050), which took serve-60 for its smoke-test-failure scenario. Both branches renumbered past it independently, so the collision is now at serve-61, not serve-60:

As before, each is correct within its own branch (a scenario's number equals its position in its own file); the clash only exists across the two, so whichever merges first forces the other to renumber. feature_naming fails loudly on a duplicate, so it cannot slip through silently.

Worth noting this is the third time these numbers have moved — #854's pair has gone 57/58 → 59/60 → 60/61 → 61/62 purely from main inserting scenarios ahead of them. The @id: tags are the binding key and have never changed, so no step binding has moved with them, and nothing about the tests' meaning has changed.

I will renumber whichever branch lands second.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (25 files, +4105/-180, base 088faa43…head d5b4d66c). Implements ROCMAI-541 ("Windows: managed-serve failures are never logged; the user only sees "server process exited""), expanded well beyond the ticket's Windows-only scope to the launch/restart/attach/--wait entry points on every platform; the ticket is still "In Progress" and the PR does not claim to close it standalone.

Blocking

  • settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash — apps/rocm/src/service_launch/readiness.rs:153 (the else if latest.status == on_disk { return deliberate; } early return), interacting with apps/rocm/src/managed_service_status.rs:408-418's unconditional dead-pid→"stopped" reconciliation (no stop-marker check, by its own comment: "dead pids alone do not distinguish a stopped service from a crashed one"). When that reconciliation runs concurrently (e.g. a rocm services/rocm dash listing) before this function's own read, deliberate is correctly false but the manifest is never corrected to "failed" — contradicting this PR's own acceptance criterion. Traced directly against the current head.
    Confidence 85 · logic · Fix: correct the status to "failed" in this branch when !deliberate, rather than returning early without writing.

  • apps/rocmd/src/service.rs:137's unconditional marker move can erase a concurrent stop's already-correct durable evidence — record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();, unlike the guarded if let Some(requested) = ... { Some(requested) } pattern used for the identical operation in service_control.rs and managed_service_status.rs's settle_pending_stop_key_cleanup. If a concurrent liveness refresh already moved the marker before this function's own re-read, .take() on the already-None field assigns None over the just-read Some(X), erasing the one piece of durable evidence this whole PR relies on — reproducing the exact crash-indistinguishable-from-a-stop defect ROCMAI-541 exists to close, through the one stop path not named in the PR's own "shape of the fix" table.
    Confidence 90 · logic · Fix: guard it the same way as service_control.rs.

  • crates/rocm-core/src/managed_child.rs's type-level doc comment (line 63) and from_pid's doc (line 79) are stale and self-contradict is_running's own doc 20 lines below — both say the Windows arm "defers to process_is_running," while is_running() actually (and deliberately) uses probe_process instead, specifically to avoid the bug process_is_running would cause. Within one new file, the first doc a reader sees documents the old, unsafe behavior as current.
    Confidence 85 · mechanical (doc-accuracy) · Fix: update both to match is_running's doc.

Non-blocking

  • tests/e2e-cucumber/features/model_serving.feature:1011's comment misnames its own sibling scenario — says "arm serve-59 exercises," but the sibling defined two lines above in this same diff is serve-61; serve-59 is an unrelated, pre-existing scenario. The PR description also calls these two new scenarios "serve-59"/"serve-60" throughout, while docs/testing.md (also new) correctly says serve-61/serve-62 — a renumbering this one comment and the description missed.
    Confidence 85 · mechanical · Fix: s/serve-59/serve-61/, and fix the description.

Decisions for the author

  • answered-by-intent — Whether a failed record should be a rocmd recovery candidate at all is left open by design: "Whether rocmd recovery should respawn a failed record is still undecided; a_crash_verdict_record_is_a_recovery_candidate pins what it does today, not what it should do." Would otherwise be worth a Deployment note as an unreviewed behavior change; the PR already discloses it as a known limitation, so just confirm it's still the plan to resolve it in a follow-up.

Positive signals

  • The "moved, not dropped" settled_stop_unix_ms pattern in service_control.rs and managed_service_status.rs is correctly guarded and has dedicated regression tests for the exact concurrent-listing race it defends against (a_second_listing_does_not_clear_the_stop_a_first_one_settled) — which is what makes the missing guard in apps/rocmd/src/service.rs stand out as an inconsistency rather than an unknown risk.
  • ManagedChildLiveness's Unqueryable-vs-Gone distinction is tested by driving all three ProcessProbe outcomes on every platform, not only the one a real Windows CI run could construct.
  • The scripted-engine E2E hook is double-gated (compile-time e2e-test-hooks feature and a runtime env var) and backed by xtask's the_release_workflow_builds_no_feature_gated_binaries, which scans for any enabled feature rather than just this one.
  • The rocm logs <id> → rocm services logs <id> remediation-hint fix is backed by a test (cli.rs) that actually parses both suggested commands and confirms the old buggy form resolves to a positional search query rather than a service selector.

Deployment notes

  • Crashed launches/restarts/--wait now persist status: "failed" (previously stopped in some of these paths), making them rocmd recovery candidates in cases that were not retried before — see "Decisions for the author" above.
  • A deliberate stop of a real Lemonade/vLLM-backed service can still show failed on disk (the adapter's own SIGTERM handling writes it, read back via refresh_from_engine_state), per the PR's own disclosed limitation.

What this covered

Full diff read (git diff origin/main...pr-854, base 088faa43…head d5b4d66c), plus full-source reads of the highest-risk files: managed_child.rs, process.rs, service_record.rs, readiness.rs, spawn.rs, attach.rs, service_control.rs (stop+restart paths), launch_transaction.rs, start.rs, managed_service_status.rs, apps/rocmd/{service,watchers/server_recover}.rs, engine_ipc.rs, and cli.rs's new test; wide-context diffs for the rest (service_launch.rs, docs, the e2e feature/steps files). engine_runtime_select.rs and serve_cmd.rs got a lighter, diff-only pass. Confirmed via git merge-tree that the current head merges cleanly against current origin/main despite several touched files (README.md, cli.rs, rocm-core/lib.rs, docs, tests/e2e-cucumber/tests/e2e.rs) having also changed independently on main since this PR's base.

Degraded fan-out. The method calls for splitting this review across independent workers; the environment's shared subagent-concurrency pool was saturated by concurrent sibling jobs for the entire review (confirmed via repeated "concurrent subagent limit reached" errors, including on retry), so this ran single-context instead. Recommend re-running with fan-out available for a second, independent pass.

CI on the reviewed commit is green on every completed, non-hardware-specific check (clippy, build-and-test, affected-crate tests, the E2E mock lane, Windows build/test, Windows release/lifecycle, docs build, security scans); Skill checks (skillscope) is explicitly skipped/advisory, and the MI300X/Strix Halo Windows hardware lanes are queued/cancelled rather than failing, consistent with self-hosted-runner availability gaps this branch's history discusses elsewhere — not treated as blocking signals.

Did not run: cargo build/test/clippy (no Rust toolchain in this environment) — all findings above are argued from direct reading and tracing of the source and tests, not execution. History/prior-changes passes were not run as dedicated separate passes.

Reconciliation, done after the blind pass: this PR carries a standing CHANGES_REQUESTED review from this same account at an earlier commit (e00fac0f, not the current head). git log confirms only a merge-from-main separates that commit from the current head — no new work responds to it. Three of that review's points (the settle_exited_service finding, the rocmd/service.rs unguarded marker move, and the managed_child.rs stale doc) were independently re-derived and re-verified by me directly against the current head's source before inclusion above, rather than taken on trust.

Ordering deviation, disclosed rather than left unstated: the PR title/description and mergeable-state were fetched together with CI status before the blind code-reading pass began. The findings above were derived and re-verified from the diff and source directly (the scenario-numbering finding, in particular, came from grepping the feature file itself); none rely on the description's framing.


🤖 Automated by an AI agent

&serde_json::json!({ "status": "failed" }),
);
}
} else if latest.status == on_disk {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — settle_exited_service can leave a crashed service's manifest reading stopped forever, even though it correctly reports the death as a crash.

Trace: refresh_managed_service_runtime_liveness_with (managed_service_status.rs:408-418) reconciles any record whose tracked pids are gone to status = "stopped", with no stop-marker check at all (its own comment: "dead pids alone do not distinguish a stopped service from a crashed one"). If that reconciliation runs concurrently — e.g. a rocm services/rocm dash listing — before this function's own read_manifest call, then here latest.status is already "stopped", deliberate is correctly computed as false (no marker was ever set), and on_disk == "stopped". managed_service_is_live(&latest) is false for "stopped" (is_live() only accepts ready/running/starting/recovering), so control falls to this else if latest.status == on_disk branch and returns false without writing anything.

died_during_wait's caller still prints the correct crash message to the console (since it uses the returned deliberate boolean), but the persisted manifest is left reading "stopped" — never corrected to "failed", contradicting this PR's own acceptance criterion for this exact path ("the engine state file is marked failed on any error exit"). rocm services list/rocm dash would show this service as cleanly stopped even though the CLI itself just told the user it crashed.

Confidence 85 · logic · Fix: in this branch, when !deliberate and the status is not already "failed", correct it to "failed" (and merge the engine-state file) instead of returning early without writing — the guard should protect against clobbering a newer, correct write, not against correcting a status this function's own verdict disagrees with.

Comment thread apps/rocmd/src/service.rs Outdated
// crash and was stamped `failed` — the exact symptom ROCMAI-541 exists
// to remove, reintroduced through the one stop path that is not
// `rocm services stop`.
record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — this unconditional marker move can erase a concurrent stop's already-correct durable evidence.

Unlike the guarded if let Some(requested) = record.stop_requested_unix_ms.take() { record.settled_stop_unix_ms = Some(requested); } used for the identical operation in apps/rocm/src/service_control.rs (stop_internal_managed_service_confirmed_with) and in managed_service_status.rs's settle_pending_stop_key_cleanup, this call site does it unconditionally: record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();.

record here is freshly re-read from disk a few lines above specifically because "other writers ... may have persisted something while the stop ran." If a concurrent liveness refresh already moved the marker (stop_requested_unix_ms → settled_stop_unix_ms) before that re-read, the fresh record already has stop_requested_unix_ms: None and settled_stop_unix_ms: Some(X). .take() on the already-None field returns None, and this line assigns that None straight over the Some(X) just read — erasing the one piece of durable evidence the whole PR relies on to prove this death was requested, and reproducing the exact crash-indistinguishable-from-a-stop state ROCMAI-541 exists to close, through the one stop path (rocmd's own) that still has the unguarded version. This file also isn't named in the PR description's "shape of the fix" table, even though it received the identical fix as service_control.rs.

Confidence 90 · logic · Fix: guard it the same way as service_control.rs's if let Some(requested) = ... pattern.

Comment thread crates/rocm-core/src/managed_child.rs Outdated
Self::Unix(child)
}

/// Build from a bare pid on Windows, where [`process_is_running`] already

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — stale, self-contradicting doc comments within this same new file.

This from_pid doc ("where [process_is_running] already reads the real exit code") and the type-level doc a few lines above (line 63: "...so the bare pid is enough, and this variant just defers to it [process_is_running]") both describe the Windows arm as deferring to process_is_running. But is_running()'s own doc and its actual implementation (Self::Windows(pid) => pid_child_is_running(*pid, probe_process)) explicitly do not use process_is_running — they use probe_process instead, specifically because process_is_running collapses ProcessProbe::Unqueryable into "not running," which would report a transient OpenProcess denial on a healthy engine as a crash (the exact bug this type exists to prevent).

So within one new file, the type-level doc and from_pid's doc describe the old, unsafe behavior as current, while is_running's doc (20-ish lines below) correctly describes what the code actually does. A reader who only reads the type doc first gets the wrong contract.

Confidence 85 · mechanical (doc-accuracy) · Fix: update this doc and the type-level doc (line ~60-63) to say the Windows arm uses probe_process (via pid_child_is_running), matching is_running's doc.

# externally triggered stop in the same post-startup window must NOT be
# treated as the crash above. `rocm services stop` run from a second
# invocation kills the pid the attached `--verbose` session is watching,
# landing in the identical `AttachOutcome::ServerExited` arm serve-59

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking — stale scenario-number cross-reference, introduced by this PR.

This comment (for the new serve-62 scenario) says the external-stop scenario lands "in the identical AttachOutcome::ServerExited arm serve-59 exercises." But the sibling scenario defined two lines above in this same diff is titled Scenario: serve-61 - A managed engine that dies after startup fails loudly — serve-59 is a different, pre-existing scenario in this file ("Only the user who started a key-protected server can read its key"). The PR description also calls these two new scenarios "serve-59"/"serve-60" throughout, while docs/testing.md (also added by this PR) correctly refers to them as serve-61/serve-62. Looks like a renumbering (likely from this long-lived branch catching up with scenarios added on main) that this one comment — and the PR description — missed.

Confidence 85 · mechanical · Fix: s/serve-59/serve-61/ here, and fix the PR description's references.

Review finding: `apps/rocmd/src/service.rs` moved the stop marker with a bare
`record.settled_stop_unix_ms = record.stop_requested_unix_ms.take();`, while
the identical operation in `service_control.rs` and
`managed_service_status.rs` is guarded. A concurrent listing's liveness
refresh can settle the stop first; `.take()` then yields `None`, and assigning
that over the `settled_stop_unix_ms` the refresh had just written correctly
leaves both fields unset -- which reads as a crash. That is precisely the
defect this change exists to close, still reachable through the one stop path
its own table did not name.

Also corrects two doc comments in `crates/rocm-core/src/managed_child.rs`
that contradicted `is_running`'s doc twenty lines below: both said the
Windows arm defers to `process_is_running`, when it deliberately asks
`probe_process` instead so a process this one cannot open is not mistaken for
one that exited. The first doc a reader met described the behaviour the type
was built to avoid.

And renumbers a feature-file comment that named `serve-59` as its own
sibling; that sibling is `serve-61` after the merge, and `serve-59` is an
unrelated pre-existing scenario.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Three blocking findings; two fixed in 4c70975f9, one left open with a reason.

Blocking 2 — apps/rocmd/src/service.rs:137's unguarded marker move — fixed. You were right, and right that the inconsistency is what makes it stand out: the same move is guarded in service_control.rs and in managed_service_status.rs's settle_pending_stop_key_cleanup, and this was the one place still taking it bare. A concurrent listing's liveness refresh settles the stop first, .take() then yields None, and assigning that over the settled_stop_unix_ms the refresh had just written correctly leaves both fields unset — which reads as a crash. Now guarded the same way, with the reasoning recorded next to it.

Blocking 3 — managed_child.rs's stale docs — fixed. Both the type-level doc and from_pid's said the Windows arm defers to process_is_running, twenty lines above is_running's doc explaining it deliberately does not. Corrected so the first doc a reader meets describes probe_process and the Unqueryable distinction, rather than the behaviour the type exists to avoid.

Non-blocking — the feature-file comment naming serve-59 as its own sibling — fixed to serve-61. You are right that the description is stale too; it has been renumbered three times by main inserting scenarios ahead of it, and I will correct it in the next pass.


Blocking 1 — settle_exited_service leaving a crashed record reading stopped — open, and I want to explain why rather than ship a quick fix.

Your trace is right. What makes it more than a one-liner is that the early return you point at is itself load-bearing. else if latest.status == on_disk { return deliberate; } exists to avoid writing latest — a snapshot read earlier in the same function — back over whatever reached disk meanwhile. There is a regression test pinning exactly that: it plants a key the record type does not model and asserts it survives, and deleting the guard drops it.

So the fix you suggest — "correct the status to failed in this branch when !deliberate" — reintroduces the stale-snapshot clobber the guard was added for, because the correction would go out through latest.write(), a whole-file write. Doing both correctly means writing the status through merge_json_state_file (read-modify-write) instead, so the crash verdict lands without taking the rest of the snapshot with it, and only when the on-disk status is not already failed.

That is a change to the write discipline of the function this whole PR is built around, and I would rather it be a deliberate decision than something I slipped in alongside two unrelated fixes. Taking it next, as its own change with its own test.


CI note: five rocmd supervise:: process-tree-kill tests fail on this branch (an_engine_child_dropped_before_its_wait_is_terminated_and_reaped and four siblings). Not mine and not load: they fail identically with --test-threads=1, they fail on the branch head without my change, and my diff touches none of supervise.rs. They pass on a pristine main checkout on the same box, so this is something this branch introduced earlier — worth a look independently of these three findings.

Decision confirmed: whether a failed record should be a rocmd recovery candidate is still intended as a follow-up; a_crash_verdict_record_is_a_recovery_candidate pins today's behaviour, not the desired one.

Leaving CHANGES_REQUESTED standing while blocking 1 is open.

… to stopped

`settle_exited_service` reported the death as a crash but could leave the
manifest reading `stopped` forever. `managed_service_status`'s liveness
reconciliation demotes any record whose recorded PIDs are gone to `stopped`,
unconditionally and without knowing why they went, and a `rocm services` or
`rocm dash` listing running alongside the wait is enough to do it. When that
lands before this function's own read, `on_disk` is already `stopped`, the
refresh from the engine state file (which a SIGKILL or OOM kill never got to
write) learns nothing, and the `latest.status == on_disk` early return exits
without writing — so `rocm services list` shows a clean stop for a death
nothing asked for, which is this change set's own acceptance criterion
failing.

The early return is load-bearing and stays: it is what stops the stale
snapshot being written back over whatever a concurrent writer put on disk
meanwhile. The correction therefore goes through `mark_service_failed`'s
read-modify-write merge, which replaces only the `status` key under the state
file lock, rather than `latest.write()`, which would trade this defect for the
one the guard exists to prevent. A record already reading `failed` carries the
engine's own reason and is left entirely alone.

Two tests, each falsifiable on its own: one drives the interleaving and
asserts both halves (the manifest and the engine state file end `failed`, and
a planted key the record type does not model survives), the other pins the
already-`failed` guard through the engine state file a needless correction
would invent. Mutating the correction away, dropping the `failed` term, and
swapping the merge for a whole-file write each fail exactly one of them.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Two open items, pushed as 086662de4.

1. The five rocmd supervise:: failures are not this branch's

They are not a regression from this PR. On the same Linux devbox, with the same toolchain and the same --test-threads=1 invocation, a pristine origin/main checkout (75fc9c9d2) fails the identical five tests, with identical messages:

supervise::tests::a_failed_wait_leaves_the_engine_guard_armed
supervise::tests::an_engine_child_dropped_before_its_wait_is_terminated_and_reaped
supervise::tests::supervise_terminates_the_engine_when_recording_its_pid_fails
supervise::tests::supervise_terminates_the_engine_when_recording_ready_fails
supervise::tests::terminate_recovery_supervisor_kills_and_reaps_the_child_and_its_tree

main: 54 passed; 5 failed — branch: 54 passed; 5 failed. The only difference in the two runs is the test count (239 vs 240 filtered out), which is this branch's one added rocmd test.

Corroborating evidence that this is environmental rather than a defect on main:

  • apps/rocmd/src/supervise.rs is byte-identical between origin/main and this branch, and so is every file under crates/rocm-core/src/process* and proc_lifecycle.rs that the tree-kill path goes through. This branch's process.rs diff is entirely #[cfg(windows)] (spawn_detached_with_log) plus a Windows-only test; ManagedChildLiveness is a new module nothing in rocmd calls.
  • The CI run for 75fc9c9d2 on main is green.

So the failures are specific to this host — all five assert that a grandchild was killed along with the process above it, i.e. the /proc-walk tree kill in terminate_verified(.., KillScope::Tree, ..), which is exactly the part most likely to behave differently inside a container's PID view. I did not chase it further: it belongs to the devbox, not to this PR, and "fixing" the tests here would have encoded a host artefact as a contract change. Worth a separate look at why the devbox differs from CI.

2. settle_exited_service leaving a crash reading stopped — fixed

Confirmed exactly as described: when managed_service_status's unconditional dead-pid -> "stopped" reconciliation (a rocm services or rocm dash listing) lands before this function's own read, on_disk is already stopped, the refresh from the engine state file learns nothing because a SIGKILL/OOM kill never wrote one, and else if latest.status == on_disk { return deliberate; } exits without writing. The caller reports the crash; the manifest says stopped forever.

The early return stays. It is what stops the stale snapshot being written back over whatever reached disk meanwhile, and a_wait_death_is_a_crash_unless_a_stop_asked_for_it's third arm pins that by planting a key the record type does not model. Correcting the status through latest.write() would have traded this defect for that one. Instead the correction goes through mark_service_failed, which is already the read-modify-write merge used elsewhere here: it replaces only the status key in the manifest and the engine state file, under the state-file lock. A record already reading failed carries the engine's own reason and is left entirely alone.

Two new tests, both in service_launch::readiness:

  • a_crash_is_still_recorded_when_a_listing_already_reconciled_it_to_stopped — drives the interleaving (manifest already stopped, no stop marker on either half of the evidence) and asserts both halves: the manifest and the engine state file end failed, and a planted concurrent key survives.
  • a_record_that_already_failed_is_left_entirely_alone — pins the != "failed" term. A failed -> failed merge is invisible in the status, so this one watches the engine state file instead: a needless correction would invent one the engine never wrote.

Mutation evidence (each run alone, each caught by exactly one test):

Mutation Result
correction removed (restores today's behaviour) a_crash_is_still_recorded_when_a_listing_already_reconciled_it_to_stopped FAILS
&& latest.status != "failed" dropped a_record_that_already_failed_is_left_entirely_alone FAILS
mark_service_failed replaced with latest.status = "failed"; latest.write() a_crash_is_still_recorded_when_a_listing_already_reconciled_it_to_stopped FAILS (planted key clobbered)

a_wait_death_is_a_crash_unless_a_stop_asked_for_it passes unchanged, including its third arm.

Gates (Linux devbox)

cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo clippy --workspace --all-targets --features e2e-test-hooks -- -D warnings; cargo clippy -p e2e-cucumber --test e2e -- -D warnings; cargo test -p e2e-cucumber --test feature_naming; cargo xtask check-architecture-doc; check-crate-edges; check-module-size — all clean. cargo test -p rocm --bin rocm: 1946 passed, 0 failed. cargo test -p rocmd: 294 passed, 5 failed — the five above, which fail identically on origin/main on this host.

@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Retracting something I said on this PR two comments ago.

I reported that five rocmd supervise:: tests were a regression this branch introduced, on the strength of "they pass on a pristine main checkout on the same box". That control was invalid and the conclusion was wrong.

The tree I ran main in was over a day stale — dated 11-16 10:03 — and it does not contain the failing tests at all. grep -c 'fn an_engine_child_dropped_before_its_wait_is_terminated_and_reaped' returns 0 there. It reported "43 passed, 0 failed" because the five tests did not exist to run, and I read that as them passing.

Checked properly, against current main (75fc9c9d2) on the same devbox, same toolchain, same --test-threads=1: main fails the identical five tests with identical messages — 54 passed; 5 failed, the same as this branch. The only delta is one extra filtered test, this branch's own addition.

The structural evidence agrees, and I verified it independently:

  • apps/rocmd/src/supervise.rs and apps/rocmd/src/proc_lifecycle.rs are byte-identical between origin/main and this branch.
  • The only crates/rocm-core/src/process.rs change is a #[cfg(windows)] spawn helper plus doc comments; nothing on the Linux tree-kill path.
  • ManagedChildLiveness is a new module and nothing in rocmd calls it.
  • main's own CI is green on that commit.

So this is a property of the devbox — all five assertions are about a grandchild being reaped with its parent, which is the /proc tree walk in terminate_verified(.., KillScope::Tree, ..), exactly the part whose behaviour differs inside a container's PID namespace. It is not a defect in main or in this branch, and nothing here should be changed for it.

Worth saying plainly: "it passes on main" is only evidence if the tree you ran is actually current main. Mine was not, and I did not check before reporting it as a regression.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change. Title/description identify this as ROCMAI-541 (ticket content not retrieved, per instructions for this repo); the diff's substance matches the description's stated scope (crash-vs-stop disambiguation at every managed-child exit point, Windows stdio attach, the scripted-engine e2e hook) except for the scenario-number mismatch noted below.

Blocking

  • PR description cites the wrong scenario IDs for its own new regression tests — tests/e2e-cucumber/features/model_serving.feature:1001,1017
    The description's "Tests" section, its "verified empirically ... turned serve-60 red" claim, and its mutation-check list all cite serve-59/serve-60 for the two scenarios this PR adds. Those two numbers are already taken by pre-existing, unrelated scenarios ("Only the user who started a key-protected server can read its key" / "A launch whose first chat request fails is not presented as ready"). The scenarios this PR actually adds are serve-61 (@id:serve-scripted-death-after-startup) and serve-62 (@id:serve-scripted-external-stop-while-attached). The work itself is correctly implemented and tested — this is a documentation accuracy issue in the PR's own account of what it verified, not a code defect — but a reader trying to verify the "removing the status term turned serve-60 red" claim would land on the wrong, unrelated scenario.
    Confidence 90 · documentation-accuracy · Fix: update the description's scenario-number references to serve-61/serve-62.

Non-blocking

  • StopWatch's stop-detection narrowing is applied to two of three sibling wait paths — apps/rocm/src/service_launch/readiness.rs:624 (StopWatch, new), apps/rocm/src/service_launch/readiness.rs:406 (wait_on_existing_service, unchanged), apps/rocm/src/service_control.rs:947 (record_restart_readiness)
    start.rs's fresh-launch wait and service_control.rs's restart wait both sample a StopWatch across the loop to narrow the "stop requested+confirmed within one poll, then raced by reset_for_restart" residual gap the type's own doc comment names. wait_on_existing_service (reached when rocm serve --wait re-attaches to an already-live/starting service) has no such sampling, relying only on the one-shot durable check — which covers the common case this PR fixes, just without the extra narrowing its siblings get. Separately, record_restart_readiness's own stop_seen_while_waiting=true branch has no dedicated test, unlike its sibling finalize_managed_launch_readiness_trusts_the_latch_after_the_marker_is_cleared. Both are narrow, already-accepted-elsewhere categories of risk (the authors' own comments describe the underlying residual gap as deliberately accepted), not a clear regression.
    Confidence 70 · architectural · Fix: add StopWatch to wait_on_existing_service for consistency (or document why it's excluded), and add a test driving the restart path's latch to true.
  • Stale comment in a test, contradicted by its own assertion — apps/rocm/src/service_launch/spawn.rs:1453 (assertion), ~:1337 (stale comment, unchanged by this diff)
    The comment right above the persisted.status, "failed" assertion was correctly updated to explain the new outcome, but the test function's leading summary comment, a few lines after #[test], still says "...mark the record stopped (a failed one would be respawned by rocmd recovery)...". The two comments in the same test now disagree.
    Confidence 85 · mechanical · Fix: update the leading comment to match.

Decisions for the author

  • docs/architecture.md entries for readiness.rs, spawn.rs, service_control.rs, engine_ipc.rs and attach.rs still describe their pre-PR contents — answered-by-intent
    PR body, "Known limitations": "docs/architecture.md gained managed_child.rs, but the readiness.rs, process.rs and engine_ipc.rs entries still describe their older contents." Would otherwise have been a blocking architectural/doc-accuracy finding (these entries omit the new crash-vs-stop machinery that is now most of what each file does); the PR explicitly discloses the gap for three of the five files in question. service_control.rs and attach.rs's entries are similarly stale and not named in that limitations note — worth folding into the same follow-up.

Positive signals

  • ManagedChildLiveness (crates/rocm-core/src/managed_child.rs) is a clean, well-reasoned split: Unix zombie-reap via retained Child/try_wait(), Windows via probe_process treating Unqueryable as "may still be running" rather than "gone" — with a unit test specifically pinned to the one case (Unqueryable) that discriminates it from the old process_is_running check.
  • The settled_stop_unix_ms "move, don't drop" redesign replacing the unsound status == "stopped" evidence is sound and consistently wired across service_record.rs/managed_service_status.rs/service_control.rs; several tests pin the exact mutation that would reopen the original bug.
  • serve-61/serve-62 give genuine, non-tautological regression coverage: exit code, user-visible message, log content, manifest status and engine-state status are all asserted together, so no single narrow fix can pass them by accident.
  • The e2e-test-hooks/ROCM_E2E_SCRIPTED_ENGINE gating is layered consistently (feature-gated function definition plus feature-gated call site) with no reachable leak into a build without the feature.
  • Windows gets real, new crash-detection coverage here (spawn_detached_with_log, settle_restart_startup_death) rather than a paper parity claim — previously-absent paths, not just mirrors of existing Unix ones.

Deployment notes

  • Behavior change: a managed rocm serve/services restart/--wait that crashes now exits non-zero and leaves the record failed, where it previously exited 0 with "server process exited". Any script or integration relying on the old exit-0-on-crash behavior needs to adjust.
  • Remediation hint text changed from rocm logs <id> to rocm services logs <id> in multiple places; verified no stale form of the old hint survives anywhere in the tree.
  • Author-disclosed, accepted limitation: a deliberate stop of a real Lemonade/vLLM-backed service can still show failed because those adapters treat SIGTERM as an unsuccessful exit and write failed themselves; not covered by any automated test, only by a manual-testing doc section.

What this covered

Full review of the whole change: 25 files, merge-base 088faa43…head 4c70975f (diff id ba25ed19a668). Every changed file read completely, split across independent blind passes grouped by coupling (readiness.rs+spawn.rs; attach.rs+service_control.rs+launch_transaction.rs+start.rs+service_launch.rs; managed_service_status.rs+service_record.rs+process.rs+managed_child.rs+lib.rs; engine_ipc.rs+serve_cmd.rs+cli.rs+engine_runtime_select.rs; rocmd/service.rs+watchers/server_recover.rs; the three e2e test files), plus a dedicated agent-instruction-adherence/code-comment pass and a history/prior-commits pass. All passes ran blind (no PR description or review threads). CI checked via get_check_runs at the reviewed head: every required check (clippy — which folds in module-size/architecture-doc-presence/crate-edges — build-and-test on both platforms, Windows debug/release, E2E consolidated and mock lanes, security scans, docs build) is green; CodeQL reports neutral (no alerts), one Skill checks (skillscope) job is skipped (not applicable).

Did not run: no cargo binary was available to any pass, so module-size/architecture-doc-presence claims were cross-checked by manual line counts and docs/architecture.md inspection rather than by running the actual xtask checks — relying on CI's green clippy job for the automated verdict on those two.

Not reconciled against this PR's many prior review rounds' threads: each candidate finding above was independently re-derived and verified against the current head rather than taken from history, per this review's blind-then-reconcile methodology.


🤖 Automated by an AI agent

# Before ROCMAI-541 this made a clean, intentional, externally-triggered
# stop fail exactly like serve-61's crash.
@id:serve-scripted-external-stop-while-attached @requires-no-gpu
Scenario: serve-62 - An externally stopped attached serve exits cleanly, not as a crash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scenario is serve-62, not serve-60 as the PR description says in several places (its "Tests" section, the "verified empirically ... turned serve-60 red" claim, and the mutation-check list). serve-59/serve-60 are both pre-existing, unrelated scenarios already in this file ("Only the user who started a key-protected server can read its key" / "A launch whose first chat request fails is not presented as ready") — the new ones this PR adds are serve-61 (line 1001, @id:serve-scripted-death-after-startup) and serve-62 (here, @id:serve-scripted-external-stop-while-attached). Worth fixing the description's scenario numbers so the "verified empirically" claim points at the scenario it actually describes.

assert_eq!(
persisted.status, "stopped",
"a post-spawn failure must mark the record abandoned, not leave it stuck"
persisted.status, "failed",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the comment right above this assertion was correctly updated to explain the new "failed" outcome, but the leading comment at the top of this same test function (a few lines after the #[test] attribute, ~line 1337: "...mark the record stopped (a failed one would be respawned by rocmd recovery)...") is unchanged by this diff and now contradicts both this assertion and the comment right above it. Worth updating that top-of-function summary too so the two comments in the same test don't disagree.

///
/// `stop_requested_unix_ms` is a stop's *live* evidence (see
/// [`managed_service_stop_was_requested`] for why the record's status is no
/// evidence at all), but it is deliberately short-lived:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking/design question: StopWatch is sampled across the wait loop in the fresh-launch path (start.rs) and the restart path (service_control.rs's record_restart_readiness), narrowing the "stop requested+confirmed inside one poll interval, then raced by reset_for_restart" gap this type's doc comment describes. wait_on_existing_service (above, unchanged by this diff) has no StopWatch of its own — it relies solely on the one-shot died_during_wait/settle_exited_service check at the end, which this PR's fix to the durable settled_stop_unix_ms evidence already covers for the common case, but without the extra narrowing its two siblings get. Was leaving this third wait path without a StopWatch deliberate (e.g. its exposure window is considered negligible), or worth adding for consistency? Separately, record_restart_readiness's stop_seen_while_waiting=true branch doesn't appear to have a dedicated test, unlike finalize_managed_launch_readiness's equivalent (finalize_managed_launch_readiness_trusts_the_latch_after_the_marker_is_cleared).

…it leaves

The comment above the assertion was updated when the outcome changed to
`failed`; the test's own leading summary a hundred lines earlier still said
the record is marked `stopped` because `a failed one would be respawned by
rocmd recovery`. It now carries the same account, including that whether a
`failed` record should be a recovery candidate is the open question the PR's
Known limitations records -- the assertion pins today's behaviour, not the
desired one.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Blocking — the description citing the wrong scenario IDs — fixed. You are right, and right that it is the kind of error that wastes a reader's time in the worst way: someone checking the "removing the status term turned serve-60 red" claim would have landed on an unrelated pre-existing scenario and concluded the claim was false.

The body now says serve-61 (@id:serve-scripted-death-after-startup) and serve-62 (@id:serve-scripted-external-stop-while-attached) throughout, including the mutation list and the cargo xtask e2e -n command. I had flagged this myself two rounds ago and said I would fix it, then didn't — the numbers have moved three times (57/58 → 59/60 → 60/61 → 61/62) as main inserted scenarios ahead of them, and the description kept being the thing left behind. The @id: tags never moved, which is why the tags are quoted alongside every number now.

Non-blocking — the test's two contradicting comments — fixed (6a8a7dc0c). The comment above the assertion had been updated when the outcome became failed; the test's leading summary a hundred lines earlier still said the record is marked stopped "(a failed one would be respawned by rocmd recovery)". It now carries the same account, and names the open question explicitly: whether a failed record should be a recovery candidate is undecided, and the assertion pins today's behaviour rather than the desired one.

Non-blocking — StopWatch on wait_on_existing_service — not changed, and I want to give the reasoning rather than silently skip it. You are right that the three sibling wait paths are inconsistent. The asymmetry is defensible, though: start.rs and service_control.rs both wait across a window in which this process initiated the launch or restart, so a stop arriving mid-wait is a live race they have to sample for. wait_on_existing_service re-attaches to a service someone else already started, where the one-shot durable check covers the case this PR exists to fix. Adding the sampling would be harmless and more uniform; leaving it is defensible; what is not defensible is neither doing it nor saying why. I would rather fold it into the same follow-up as the record_restart_readiness latch test than widen this PR again.

Decision — the stale docs/architecture.md entries. You are right that service_control.rs and attach.rs belong in that limitations note alongside readiness.rs, process.rs and engine_ipc.rs; the note named three of the five. Folding all five into the follow-up.

So: the blocking item is closed and one non-blocking is fixed. Two remain open by choice, both stated above with reasons rather than left silent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants