Skip to content

fix(engines): escalate Lemonade and vLLM teardown to SIGKILL (ROCMAI-452) - #1088

Open
rominf wants to merge 4 commits into
mainfrom
ROCMAI-452-fix-terminate-pid-escalation
Open

rominf wants to merge 4 commits into
mainfrom
ROCMAI-452-fix-terminate-pid-escalation

Conversation

@rominf

@rominf rominf commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm serve with Lemonade, and the Lemonade backend probe and install, could hang indefinitely while tearing down a server. Lemonade's terminate_pid(pid, force) ignored force and only sent SIGTERM (TerminateProcess on Windows), and every caller then did an unbounded child.wait(). A lemond or llama-server that ignores SIGTERM, for example while it is still loading a model, was never stopped, so the wait never returned. vLLM's serve_http readiness-failure teardown had the same pattern (terminate_process_tree, then an unbounded wait), and so did Lemonade's router-serve teardown, stop_server in router_serve.rs, added by #631. (ROCMAI-452)

What changes

  • Lemonade terminate_pid goes through rocm_core::terminate_verified with KillScope::Tree and the engine's 10 s STOP_GRACE: SIGTERM to the process tree, then SIGKILL for anything still running when force is set. Tree scope matters because lemond runs each loaded model in a llama-server child, and killing lemond alone would orphan it with the GPU still held. Every production caller passes force = true, so these teardowns are now bounded: the backend install and backend-selection probes in llamacpp_backend.rs, the model-download server and the direct-server supervisor (supervise_direct_server) in direct_llama.rs, and the router-serve stop_server, which switches from terminate_process_tree to terminate_pid.
  • vLLM readiness failure moves into fail_unready_server, which stops the tree the same way (SIGTERM, then SIGKILL after 10 s), reaps the child and records the failed state. It does not signal a server the readiness poll already reaped, because that PID may belong to an unrelated process by then. After a stop it cannot confirm, it kills the Child it owns before waiting. A failed write of the failed state becomes a stderr warning, so the readiness error stays the error returned.
  • Unconfirmed stops are reported on stderr in both engines, worded by a new TerminationOutcome::unconfirmed_reason(). It is derived from the same outcome as stopped() and returns None exactly when stopped() is true, so the warning cannot disagree with the decision it reports.
  • rocm_core::terminate_process and terminate_process_tree document that they only request termination and neither wait nor escalate, pointing at terminate_verified.
  • The Lemonade STOP_SCOPE comment no longer claims the recorded PID is always the llama-server: on the router path it is lemond, so a Single stop that escalates can leave its backend running. The scope itself is unchanged.
  • The docs/architecture.md rows for Lemonade state.rs and router_serve.rs and vLLM process.rs describe the bounded teardown.

User-visible change: a failed Lemonade serve (router or direct), model download or backend probe/install, or a vLLM readiness failure, now ends within about 2 × 10 s instead of possibly hanging forever. When a stop cannot be confirmed, stderr shows:

  • Lemonade: Warning: process N was not confirmed stopped: <reason>.
  • vLLM: Warning: process N was not confirmed stopped: <reason>. Killing it; processes under it are not reached by that kill.
  • vLLM, when the state file cannot be written: Warning: could not record the vLLM service as failed in <path>: <error>

<reason> is one of: it, or a process under it, was still running when the stop ended; its identity could not be confirmed, so it was not signalled; it is gone, but a process under it may still be running; it exited before the processes under it were stopped, and they may still be running.

Tests

There is no Gherkin scenario (AGENTS.md §3): the hang needs a server that ignores SIGTERM on a teardown failure path, which the e2e harness cannot stage; service_stop.feature documents the same limit for timed-out and unverified stops. The behaviour is covered by unit tests that spawn real processes on Linux, each shown to fail with its target reverted:

  • Lemonade state.rs: a forced stop SIGKILLs a child (forced_terminate_pid_kills_a_child_that_ignores_sigterm) and a grandchild (forced_terminate_pid_kills_a_grandchild_that_ignores_sigterm) that ignore SIGTERM, and the caller's following wait returns (terminate_then_wait_returns_for_a_child_that_ignores_sigterm); an unforced stop leaves such a child running and reports it unconfirmed (unforced_terminate_pid_leaves_a_child_that_ignores_sigterm_running); a cooperative child stops on SIGTERM without escalation (terminate_pid_stops_a_cooperative_child_with_sigterm); terminate_pid itself escalates only after the production 10 s grace (terminate_pid_escalates_after_the_stop_grace_and_the_callers_wait_returns); a re-exec test reads the real stderr warning together with the resulting process state, and checks a confirmed stop does not warn (terminate_pid_reports_an_unconfirmed_stop_on_stderr_and_not_a_confirmed_one).
  • Lemonade router: real_router_stop_kills_a_server_tree_that_ignores_sigterm stages a fake lemond and backend that both ignore SIGTERM; stop_server kills and reaps both within a bound, and with the fix reverted the test fails through its watchdog instead of hanging.
  • vLLM fail_unready_server_*: kills a SIGTERM-ignoring tree and records failed, at a short grace and at the production grace; does not signal a server the readiness poll already reaped; reports an unconfirmed stop on real stderr and kills the server, while a confirmed stop neither warns nor kills.
  • rocm-core: unconfirmed_reason_is_given_exactly_for_an_unconfirmed_stop pins unconfirmed_reason() against stopped() for every TerminationOutcome variant.
  • Windows: forced_terminate_pid_stops_a_long_running_child_and_wait_returns (cfg(windows)), run by the Windows CI job.

Tradeoffs and limits

  • A server that needs more than 10 s to shut down cleanly is now SIGKILLed on these teardown paths instead of being waited on.
  • Lemonade keeps the bare-PID terminate_pid(pid, force) -> bool signature and does not fall back to a kill after an Unverified stop (one whose identity could not be read, so nothing was signalled); the caller's wait can then still block, which is reported on stderr. vLLM owns its Child and kills it. On Linux, Unverified is effectively unreachable for one's own unreaped child.
  • On both engines, a process that survives SIGKILL, such as one in uninterruptible sleep, still blocks the caller's wait.
  • On Windows, terminate_pid terminates lemond itself without a tree walk. The stop counts as confirmed once Windows reports the exit status, and the caller's wait() completes the asynchronous TerminateProcess.
  • Lemonade's detached serve-http relaunch nulls stderr, so the warning is not visible on that path.

Verification

Run locally on Linux on this head (based on 75fc9c9d):

  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings: clean;
  • cargo test -p rocm-engine-lemonade -p rocm-engine-vllm -p rocm-core: all pass (rocm-core 1646 unit tests with 6 ignored, plus 29 property tests; Lemonade 295; vLLM 174);
  • cargo xtask check-architecture-doc, cargo xtask check-module-size and the prek hooks: pass.

Not run locally: the cfg(windows) test, which the Windows CI job runs (a race in it against the asynchronous TerminateProcess is fixed); cargo test --workspace --all-targets and cargo xtask smoke, which CI runs.

Follow-ups (not in this PR)

  • A shared rocm-core helper to stop an owned, unreaped child: Lemonade, vLLM and rocmd each hand-roll it.
  • Lemonade STOP_SCOPE = Tree for router mode, matching the recorded-PID stop paths in rocm services stop and rocmd, which already use Tree.
  • The direct_llama.rs test cleanups call terminate_pid with recorded PIDs, which its doc forbids.
  • A drop guard in the router test, and the duplicated test helpers.
  • vLLM's write_running_state(..)? after the spawn returns without stopping the server it just started.
  • Make the vLLM "Killing it" wording depend on the outcome.
  • The meaning of "stopped" on Windows in terminate_verified.
  • ComfyUI's stop does not escalate.

Checklist

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No row references ROCMAI-452.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration in cli.rs and the dispatch wiring in main.rs/lib.rs are expected, not a violation). (Not applicable.)
  • If this PR adds, renames or removes a production source file (or a .rs file under xtask/src or tests/e2e-cucumber), docs/architecture.md has the matching entry (cargo xtask check-architecture-doc fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (No file added; the Lemonade state.rs and router_serve.rs and vLLM process.rs entries are updated.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.
  • Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3).
  • Any existing scenario this PR moves off the blocking mock e2e job (for example by a tag) is justified as a deletion (AGENTS.md §8). (Not applicable: no scenario moves.)
  • Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11). (No option's meaning changes.)
  • Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5).

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the ROCMAI-452-fix-terminate-pid-escalation branch from e65e738 to 8f1acaf Compare November 16, 2026 13:05
… (ROCMAI-452)

Lemonade's terminate_pid(pid, force) ignored `force` and only sent
SIGTERM, and every caller follows it with an unbounded child.wait(). A
lemond or llama-server that ignores SIGTERM therefore hung the CLI
indefinitely during teardown.

Route terminate_pid through rocm_core::terminate_verified with the Tree
scope and the engine's 10 s stop grace: SIGTERM to the process tree,
then SIGKILL for anything still running when `force` is set, so the
call is bounded and the following wait returns. Tree scope matters
because lemond runs each loaded model in a llama-server child, and
killing lemond alone would orphan that child with the GPU still held.

vLLM's serve_http had the same pattern in its readiness-failure
teardown (terminate_process_tree, then wait); it now goes through a
stop_spawned_server helper with the same bounded escalation.

Document on terminate_process and terminate_process_tree that they only
request termination and neither wait nor escalate, pointing at
terminate_verified, and refresh the architecture entries. Regression
tests cover a child and a grandchild that ignore SIGTERM, the unforced
case, a cooperative child, and the Windows forced path.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The bounded teardown from the previous commit still returned only a
bool, and every caller discards it before waiting on the child. A stop
that could not be confirmed, either because the process outlived the
grace periods or because its identity could not be read and nothing was
signalled, was followed by the same silent wait.

Both engines now write the reason for an unconfirmed stop to stderr.
The text comes from a new TerminationOutcome::unconfirmed_reason(),
which is None exactly when stopped() is true, so the warning cannot
disagree with the decision it reports.

vLLM's readiness-failure teardown moves into fail_unready_server. It
owns the unreaped Child, so after an unconfirmed stop it kills the
server before waiting, which bounds the wait on every path except a
server that survives SIGKILL. A server the readiness poll already
reaped is not signalled, because its PID may belong to an unrelated
process by then. A failed write of the `failed` state is reported on
stderr, so the readiness error stays the one returned.

Lemonade keeps terminate_pid(pid, force) -> bool and does not kill on
an unconfirmed stop, because it is called with bare PIDs; its docs now
say that an Unverified process is left unsignalled and a following
wait can block.

Tests re-run the test binary to read the warning on real stderr
together with the resulting process state, cover the production stop
grace through both entry points, and check that a confirmed stop
neither warns nor kills and is reaped before the teardown returns.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The router serve path, which moved into its own module when registered
GGUF models gained a router route, stops a server that failed to load
or verify with a plain SIGTERM to its process tree and then waits on the
child. A lemond, or a backend under it, that ignores SIGTERM therefore
hung the serve on that failure path, the same hang the previous commits
removed from the other teardown sites.

Route it through terminate_pid like its siblings: SIGTERM to the tree,
SIGKILL after the stop grace, and a stderr warning when the stop cannot
be confirmed. A test stages a fake lemond and a backend child that both
ignore SIGTERM, and checks that the stop returns within a bound, reaps
the server and kills the backend.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-452)

The unconfirmed_reason test pins the reason strings, so a consistent
wrong edit to the production text and the test would stay green. Name
the engine tests that prove the warning is printed exactly when a stop
is unconfirmed, next to the state it describes, as the place the
behaviour is actually checked.

The STOP_SCOPE comment said Lemonade state always records the
llama-server itself. That holds on the direct-serve path only: the
router path records lemond, whose model runs in a child process, so a
Single stop that escalates to SIGKILL can leave that child running.
Say so, without changing the scope.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the ROCMAI-452-fix-terminate-pid-escalation branch from 8f1acaf to 3baa68a Compare November 17, 2026 16:16
@rominf
rominf marked this pull request as ready for review November 17, 2026 17:22
@rominf
rominf requested a review from a team as a code owner November 17, 2026 17:22
@rominf
rominf requested a review from fredespi November 17, 2026 17:22

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed the full diff against the merge-base (75fc9c9d), plus the surrounding code in crates/rocm-core/src/proc_lifecycle.rs, engines/lemonade/src/{lib,state,router_serve}.rs, engines/vllm/src/process.rs, and docs/architecture.md. No blocking findings.

What I verified directly (not just taken from the description):

  • TerminationOutcome::unconfirmed_reason() returns None iff stopped() is true, across every variant — both functions match exhaustively with no wildcard arm, so a future added variant fails to compile until both (and the pinning test) are updated. This backs the PR's claim that the warning can't disagree with the decision it reports.
  • Every production call site of Lemonade's terminate_pid passes force = true (router_serve.rs, direct_llama.rs, llamacpp_backend.rs); no production caller reads a PID from a state file and passes it to this function (only test-cleanup code does, which the function's own doc comment says not to do).
  • vLLM's fail_unready_server correctly avoids re-signalling a PID the readiness poll already reaped: it gates on child.try_wait(), and std::process::Child caches an observed exit status, so a second check never touches a possibly-recycled PID. The kill-then-wait() ordering on the owned Child is correct (no double-wait, no missing wait on any path), and a failed "failed"-state write only warns on stderr without replacing the returned readiness error.
  • Spot-checked that every test name the description cites by name actually exists in the diff (10/10 found, one definition each), and that the new tests assert on OS-observable state (signal, /proc, identity) bounded by a watchdog — not just "the call returned" — so reverting the escalation would plausibly make them fail/hang as claimed, not just pass trivially.
  • docs/architecture.md rows for the touched modules match what the code does.
  • CI: all required checks completed and green (build/test on Linux and Windows, E2E, security scans, coverage); the one "neutral" legacy CodeQL run is superseded by the newer codeql/* jobs, which passed.

Two pre-existing, already-disclosed limitations I confirmed are genuinely unchanged by this PR (so not blocking):

  • stop_service's STOP_SCOPE = Single (engines/lemonade/src/lib.rs) can leave a router-mode llama-server child running (still holding the GPU) if a forced stop escalates on lemond alone. I checked whether this PR newly enables that outcome (vs. just documenting it) — it doesn't: stop_service already called terminate_verified directly before this PR, and that function's SIGTERM→SIGKILL escalation already existed at the merge-base, so this exact risk predates this PR. The new comment just states it honestly, and it's explicitly carried as a named follow-up in the description (widen STOP_SCOPE to Tree for router mode).
  • The Windows LemondChild::Pid path in the new terminate_pid captures process identity at call time but holds no process handle, so if lemond already exited its PID could in principle be recycled before the call. This is strictly better than the prior code (which had zero identity verification on any platform) and is disclosed in the function's own doc comment immediately next to the risk. Windows-only, narrow, and not a regression.

Nitpick (inline): one vLLM test deliberately waits out the real 10s production grace period rather than an injected shorter one, adding real wall-clock time to the suite — looks intentional and is already noted in the test's own doc comment.

Positive signals: the escalation tests spawn real SIGTERM-ignoring process trees and assert on OS-level outcomes rather than mocking anything; the unconfirmed_reason()/stopped() consistency is enforced by the match arms themselves, not just by convention; the doc comments added to terminate_pid and terminate_process/terminate_process_tree are specific about exactly what is and isn't guaranteed on each platform.

Not independently re-run: the Rust test suite itself (relied on CI's green result plus reading test bodies/assertions directly instead).


🤖 Automated by an AI agent

/// service recorded as failed. Takes about [`STOP_GRACE`].
#[cfg(target_os = "linux")]
#[test]
fn fail_unready_server_escalates_after_the_stop_grace() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick, non-blocking: this test waits out the real production STOP_GRACE (10s) plus its own bound, which adds real wall-clock time to the suite. Looks deliberate (pinning the actual constant rather than an injected one), and the doc comment already says so — just flagging the cost for anyone later wondering why this one test is slow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants