Repository navigation
Description
Symptom
On Windows, rocm serve exits 0 when the managed engine process dies during startup. Nothing is printed to stderr, and the command reports a launched service. On Linux and macOS the same failure is caught and reported with the child's log tailed into the error.
Root cause
spawn_managed_engine_child (apps/rocm/src/main.rs:6646) branches on platform after building the service record:
- non-Windows —
command.spawn(), thenthread::sleep(200ms), thenchild.try_wait(). If the child has already exited itbail!s withmanaged_engine_startup_failure_detail(status, &record.log_path). - Windows —
rocm_core::spawn_detached_no_inherit(..), which returns a bareu32PID. There is noChildhandle, and so no liveness or exit check of any kind. Control falls straight through torecord.status = "running"andOk(ManagedSpawn::Spawned { .. }).
The caller does not rescue it. start_managed_service waits up to 45s for HTTP readiness, then sets record.status = status_for_readiness(readiness) and returns Ok(ManagedLaunchReport { .. }) regardless of the readiness outcome — a timeout is recorded as a status, not raised as an error. So a dead child produces a successful command.
This has been the case since the initial import; it is not a recent regression.
There is no diagnostic either
On non-Windows the child's stdio is attached to the service log via attach_background_stdio(&mut command, Some(&record.log_path)). The Windows path reaches spawn_windows_no_inherit(..) with std_handles: None (crates/rocm-core/src/lib.rs:1167), so the detached child gets no stdout/stderr handles at all. A user therefore sees neither a non-zero exit nor a captured startup error.
A naive fix has a trap
The obvious repair — poll the PID for liveness — carries a PID-reuse race, and the guard that exists for exactly that is inert on Windows. record.supervisor_start_ticks comes from rocm_core::process_start_ticks, which reads /proc/{pid}/stat and returns None on every non-Linux target (crates/rocm-core/src/proc_lifecycle.rs:307 and :314). A correct fix needs a real process handle, or a Win32 process start-time identity, rather than a bare PID.
How it surfaced
PR #351 reworks the Lemonade recovery scenario to inject its failure inside the engine's Serve path — i.e. in the detached child. The seam it replaces injected the failure in the parent's Install RPC, which failed synchronously on every platform and so never exercised this gap. serve-18 consequently reports serve unexpectedly succeeded (rc 0, empty stderr) on the Windows lane while passing on the Linux GPU lanes. The scenario is correct; it is asserting behaviour the platform does not implement.
Observed vs inferred
Every file, line and branch cited above is read from the tree at b0d598d0. That the scenario's injected failure occurs inside the detached child and is invisible to the parent is inferred from those two branches plus the rc 0 / empty-parent-stderr evidence in the job log — I have no Windows host, and the job log does not include the child's service log.
Activity
test(e2e): drive the Lemonade recovery scenario with a planted runtime
test(e2e): drive the Lemonade recovery scenario with a planted runtime
r0x0r commentedon Nov 4, 2026
Windows repro on real hardware, plus updated code pointers
This reproduces outside the e2e harness on a Windows 11 host (Radeon iGPU, Lemonade 11.5.1 with the llama.cpp Vulkan backend), on nightly
6eeec62. I re-read the code paths below onmainat0b8a1d1b, and they are unchanged.Repro (no fault injection needed). Serve a Hugging Face repo without a
:variant:server process exited. Exit code 0.managed service launched/readiness: starting. The record later showsstatus: stopped.%USERPROFILE%\.rocm\services\<service_id>.log) is 0 bytes.The engine process actually fails straight away inside
serve_hf_checkpointwith "`Qwen/Qwen2.5-1.5B-Instruct` needs an explicit quantization variant to serve ...". I only got that message by running the hidden wrapper directly, so its stderr reaches the console:That command is a usable workaround for anyone debugging this on Windows until it's fixed. Adding the suggested
:Q4_K_Mgets a second error from Lemonade,No supported model files found in repository ..., which is hidden in the same way.Updated pointers. The code cited in the description has moved:
spawn_managed_engine_childis now inapps/rocm/src/service_launch.rs. The#[cfg(windows)]branch callsrocm_core::spawn_detached_no_inherit(¤t_exe, &serve_args, &env_refs)and gets back a bare PID. There is still notry_waitequivalent and no stdout/stderr handles. The non-Windows branch still doesattach_background_stdioand checkstry_waitafter 200 ms.rocm_core::spawn_detached_no_inheritis incrates/rocm-core/src/process.rs(formerlylib.rs). The same module already hasspawn_hidden_console_with_log, which might be a starting point for handing the child the service log as stdout/stderr on Windows.A related gap that affects every platform. In attached mode,
AttachOutcome::ServerExitedinservice_launch.rsprintsserver process exitedand returnsOk(()). Even on Linux, an engine that dies after the 200 mstry_waitwindow makesrocm serveexit 0. Returning an error withmanaged_engine_startup_failure_detail(...)(or the log tail) would fix that on every platform. It may be worth a separate issue, since it doesn't involve the Windows spawn.Possible fix directions:
CreateProcessW'shProcess, orOpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION)immediately after spawn), thenWaitForSingleObject(h, 200)andGetExitCodeProcess. This avoids the PID-reuse problem described above, because the handle stays bound to the process object even after the PID is reused.ServerExitedreturn an error.Testing: serve-18 (
serve-lemonade-preparation-recovery) is kept off Windows only because of this issue, so it could be turned on for Windows to cover the fix. The bareowner/repocase above also reproduces this deterministically without fault injection, though it needs the Lemonade runtime installed and the serve GPU checks to pass on the Windows lane. For the "test runs on the gate it covers" rule, a pure helper behind the spawn would let a Windows unit test run a child that writes to stderr and exits 1, then check that the error is returned and the text lands in the log.Separately, the bare-
owner/repocase will be fixed at the source by rejecting it before the engine process starts. That makes this repro go away, but the gap here stays.test(e2e): drive the Lemonade recovery scenario with a planted runtime