Skip to content

serve: on Windows a managed engine that dies at startup is reported as success #467

Open

Description

@rominf

Symptom

On Windows, rocm serve exits 0 when the managed engine process dies during startup. Nothing is printed to stderr, and the command reports a launched service. On Linux and macOS the same failure is caught and reported with the child's log tailed into the error.

Root cause

spawn_managed_engine_child (apps/rocm/src/main.rs:6646) branches on platform after building the service record:

  • non-Windows — command.spawn(), then thread::sleep(200ms), then child.try_wait(). If the child has already exited it bail!s with managed_engine_startup_failure_detail(status, &record.log_path).
  • Windows — rocm_core::spawn_detached_no_inherit(..), which returns a bare u32 PID. There is no Child handle, and so no liveness or exit check of any kind. Control falls straight through to record.status = "running" and Ok(ManagedSpawn::Spawned { .. }).

The caller does not rescue it. start_managed_service waits up to 45s for HTTP readiness, then sets record.status = status_for_readiness(readiness) and returns Ok(ManagedLaunchReport { .. }) regardless of the readiness outcome — a timeout is recorded as a status, not raised as an error. So a dead child produces a successful command.

This has been the case since the initial import; it is not a recent regression.

There is no diagnostic either

On non-Windows the child's stdio is attached to the service log via attach_background_stdio(&mut command, Some(&record.log_path)). The Windows path reaches spawn_windows_no_inherit(..) with std_handles: None (crates/rocm-core/src/lib.rs:1167), so the detached child gets no stdout/stderr handles at all. A user therefore sees neither a non-zero exit nor a captured startup error.

A naive fix has a trap

The obvious repair — poll the PID for liveness — carries a PID-reuse race, and the guard that exists for exactly that is inert on Windows. record.supervisor_start_ticks comes from rocm_core::process_start_ticks, which reads /proc/{pid}/stat and returns None on every non-Linux target (crates/rocm-core/src/proc_lifecycle.rs:307 and :314). A correct fix needs a real process handle, or a Win32 process start-time identity, rather than a bare PID.

How it surfaced

PR #351 reworks the Lemonade recovery scenario to inject its failure inside the engine's Serve path — i.e. in the detached child. The seam it replaces injected the failure in the parent's Install RPC, which failed synchronously on every platform and so never exercised this gap. serve-18 consequently reports serve unexpectedly succeeded (rc 0, empty stderr) on the Windows lane while passing on the Linux GPU lanes. The scenario is correct; it is asserting behaviour the platform does not implement.

Observed vs inferred

Every file, line and branch cited above is read from the tree at b0d598d0. That the scenario's injected failure occurs inside the detached child and is invisible to the parent is inferred from those two branches plus the rc 0 / empty-parent-stderr evidence in the job log — I have no Windows host, and the job log does not include the child's service log.

Activity

  1. r0x0r commented on Nov 4, 2026

    @r0x0r
    Collaborator

    Windows repro on real hardware, plus updated code pointers

    This reproduces outside the e2e harness on a Windows 11 host (Radeon iGPU, Lemonade 11.5.1 with the llama.cpp Vulkan backend), on nightly 6eeec62. I re-read the code paths below on main at 0b8a1d1b, and they are unchanged.

    Repro (no fault injection needed). Serve a Hugging Face repo without a :variant:

    rocm serve Qwen/Qwen2.5-1.5B-Instruct            # attached
    rocm serve Qwen/Qwen2.5-1.5B-Instruct --managed  # managed
    
    • Attached: prints the serve plan, then server process exited. Exit code 0.
    • Managed: reports managed service launched / readiness: starting. The record later shows status: stopped.
    • In both cases the service log (%USERPROFILE%\.rocm\services\<service_id>.log) is 0 bytes.

    The engine process actually fails straight away inside serve_hf_checkpoint with "`Qwen/Qwen2.5-1.5B-Instruct` needs an explicit quantization variant to serve ...". I only got that message by running the hidden wrapper directly, so its stderr reaches the console:

    rocm __engine-serve-http lemonade dbg Qwen/Qwen2.5-1.5B-Instruct --port 11501 --state-path <tmp>.json
    

    That command is a usable workaround for anyone debugging this on Windows until it's fixed. Adding the suggested :Q4_K_M gets a second error from Lemonade, No supported model files found in repository ..., which is hidden in the same way.

    Updated pointers. The code cited in the description has moved:

    • spawn_managed_engine_child is now in apps/rocm/src/service_launch.rs. The #[cfg(windows)] branch calls rocm_core::spawn_detached_no_inherit(&current_exe, &serve_args, &env_refs) and gets back a bare PID. There is still no try_wait equivalent and no stdout/stderr handles. The non-Windows branch still does attach_background_stdio and checks try_wait after 200 ms.
    • rocm_core::spawn_detached_no_inherit is in crates/rocm-core/src/process.rs (formerly lib.rs). The same module already has spawn_hidden_console_with_log, which might be a starting point for handing the child the service log as stdout/stderr on Windows.

    A related gap that affects every platform. In attached mode, AttachOutcome::ServerExited in service_launch.rs prints server process exited and returns Ok(()). Even on Linux, an engine that dies after the 200 ms try_wait window makes rocm serve exit 0. Returning an error with managed_engine_startup_failure_detail(...) (or the log tail) would fix that on every platform. It may be worth a separate issue, since it doesn't involve the Windows spawn.

    Possible fix directions:

    1. Diagnostics: give the Windows child the service log as stdout/stderr, as on Linux, so the error is at least captured.
    2. Exit detection: keep a real process handle from creation (e.g. CreateProcessW's hProcess, or OpenProcess(SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION) immediately after spawn), then WaitForSingleObject(h, 200) and GetExitCodeProcess. This avoids the PID-reuse problem described above, because the handle stays bound to the process object even after the PID is reused.
    3. Attached mode: make ServerExited return an error.

    Testing: serve-18 (serve-lemonade-preparation-recovery) is kept off Windows only because of this issue, so it could be turned on for Windows to cover the fix. The bare owner/repo case above also reproduces this deterministically without fault injection, though it needs the Lemonade runtime installed and the serve GPU checks to pass on the Windows lane. For the "test runs on the gate it covers" rule, a pure helper behind the spawn would let a Windows unit test run a child that writes to stderr and exits 1, then check that the error is returned and the text lands in the log.

    Separately, the bare-owner/repo case will be fixed at the source by rejecting it before the engine process starts. That makes this repro go away, but the gap here stays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions