Repository navigation
Conversation
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
e65e738 to
8f1acaf
Compare
… (ROCMAI-452) Lemonade's terminate_pid(pid, force) ignored `force` and only sent SIGTERM, and every caller follows it with an unbounded child.wait(). A lemond or llama-server that ignores SIGTERM therefore hung the CLI indefinitely during teardown. Route terminate_pid through rocm_core::terminate_verified with the Tree scope and the engine's 10 s stop grace: SIGTERM to the process tree, then SIGKILL for anything still running when `force` is set, so the call is bounded and the following wait returns. Tree scope matters because lemond runs each loaded model in a llama-server child, and killing lemond alone would orphan that child with the GPU still held. vLLM's serve_http had the same pattern in its readiness-failure teardown (terminate_process_tree, then wait); it now goes through a stop_spawned_server helper with the same bounded escalation. Document on terminate_process and terminate_process_tree that they only request termination and neither wait nor escalate, pointing at terminate_verified, and refresh the architecture entries. Regression tests cover a child and a grandchild that ignore SIGTERM, the unforced case, a cooperative child, and the Windows forced path. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The bounded teardown from the previous commit still returned only a bool, and every caller discards it before waiting on the child. A stop that could not be confirmed, either because the process outlived the grace periods or because its identity could not be read and nothing was signalled, was followed by the same silent wait. Both engines now write the reason for an unconfirmed stop to stderr. The text comes from a new TerminationOutcome::unconfirmed_reason(), which is None exactly when stopped() is true, so the warning cannot disagree with the decision it reports. vLLM's readiness-failure teardown moves into fail_unready_server. It owns the unreaped Child, so after an unconfirmed stop it kills the server before waiting, which bounds the wait on every path except a server that survives SIGKILL. A server the readiness poll already reaped is not signalled, because its PID may belong to an unrelated process by then. A failed write of the `failed` state is reported on stderr, so the readiness error stays the one returned. Lemonade keeps terminate_pid(pid, force) -> bool and does not kill on an unconfirmed stop, because it is called with bare PIDs; its docs now say that an Unverified process is left unsignalled and a following wait can block. Tests re-run the test binary to read the warning on real stderr together with the resulting process state, cover the production stop grace through both entry points, and check that a confirmed stop neither warns nor kills and is reaped before the teardown returns. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The router serve path, which moved into its own module when registered GGUF models gained a router route, stops a server that failed to load or verify with a plain SIGTERM to its process tree and then waits on the child. A lemond, or a backend under it, that ignores SIGTERM therefore hung the serve on that failure path, the same hang the previous commits removed from the other teardown sites. Route it through terminate_pid like its siblings: SIGTERM to the tree, SIGKILL after the stop grace, and a stderr warning when the stop cannot be confirmed. A test stages a fake lemond and a backend child that both ignore SIGTERM, and checks that the stop returns within a bound, reaps the server and kills the backend. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…CMAI-452) The unconfirmed_reason test pins the reason strings, so a consistent wrong edit to the production text and the test would stay green. Name the engine tests that prove the warning is printed exactly when a stop is unconfirmed, next to the state it describes, as the place the behaviour is actually checked. The STOP_SCOPE comment said Lemonade state always records the llama-server itself. That holds on the direct-serve path only: the router path records lemond, whose model runs in a child process, so a Single stop that escalates to SIGKILL can leave that child running. Say so, without changing the scope. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
8f1acaf to
3baa68a
Compare
juhovainio
left a comment
There was a problem hiding this comment.
Summary
Reviewed the full diff against the merge-base (75fc9c9d), plus the surrounding code in crates/rocm-core/src/proc_lifecycle.rs, engines/lemonade/src/{lib,state,router_serve}.rs, engines/vllm/src/process.rs, and docs/architecture.md. No blocking findings.
What I verified directly (not just taken from the description):
TerminationOutcome::unconfirmed_reason()returnsNoneiffstopped()istrue, across every variant — both functions match exhaustively with no wildcard arm, so a future added variant fails to compile until both (and the pinning test) are updated. This backs the PR's claim that the warning can't disagree with the decision it reports.- Every production call site of Lemonade's
terminate_pidpassesforce = true(router_serve.rs,direct_llama.rs,llamacpp_backend.rs); no production caller reads a PID from a state file and passes it to this function (only test-cleanup code does, which the function's own doc comment says not to do). - vLLM's
fail_unready_servercorrectly avoids re-signalling a PID the readiness poll already reaped: it gates onchild.try_wait(), andstd::process::Childcaches an observed exit status, so a second check never touches a possibly-recycled PID. The kill-then-wait()ordering on the ownedChildis correct (no double-wait, no missing wait on any path), and a failed "failed"-state write only warns on stderr without replacing the returned readiness error. - Spot-checked that every test name the description cites by name actually exists in the diff (10/10 found, one definition each), and that the new tests assert on OS-observable state (signal,
/proc, identity) bounded by a watchdog — not just "the call returned" — so reverting the escalation would plausibly make them fail/hang as claimed, not just pass trivially. docs/architecture.mdrows for the touched modules match what the code does.- CI: all required checks completed and green (build/test on Linux and Windows, E2E, security scans, coverage); the one "neutral" legacy CodeQL run is superseded by the newer
codeql/*jobs, which passed.
Two pre-existing, already-disclosed limitations I confirmed are genuinely unchanged by this PR (so not blocking):
stop_service'sSTOP_SCOPE = Single(engines/lemonade/src/lib.rs) can leave a router-modellama-serverchild running (still holding the GPU) if a forced stop escalates onlemondalone. I checked whether this PR newly enables that outcome (vs. just documenting it) — it doesn't:stop_servicealready calledterminate_verifieddirectly before this PR, and that function's SIGTERM→SIGKILL escalation already existed at the merge-base, so this exact risk predates this PR. The new comment just states it honestly, and it's explicitly carried as a named follow-up in the description (widenSTOP_SCOPEtoTreefor router mode).- The Windows
LemondChild::Pidpath in the newterminate_pidcaptures process identity at call time but holds no process handle, so iflemondalready exited its PID could in principle be recycled before the call. This is strictly better than the prior code (which had zero identity verification on any platform) and is disclosed in the function's own doc comment immediately next to the risk. Windows-only, narrow, and not a regression.
Nitpick (inline): one vLLM test deliberately waits out the real 10s production grace period rather than an injected shorter one, adding real wall-clock time to the suite — looks intentional and is already noted in the test's own doc comment.
Positive signals: the escalation tests spawn real SIGTERM-ignoring process trees and assert on OS-level outcomes rather than mocking anything; the unconfirmed_reason()/stopped() consistency is enforced by the match arms themselves, not just by convention; the doc comments added to terminate_pid and terminate_process/terminate_process_tree are specific about exactly what is and isn't guaranteed on each platform.
Not independently re-run: the Rust test suite itself (relied on CI's green result plus reading test bodies/assertions directly instead).
🤖 Automated by an AI agent
| /// service recorded as failed. Takes about [`STOP_GRACE`]. | ||
| #[cfg(target_os = "linux")] | ||
| #[test] | ||
| fn fail_unready_server_escalates_after_the_stop_grace() { |
There was a problem hiding this comment.
Nitpick, non-blocking: this test waits out the real production STOP_GRACE (10s) plus its own bound, which adds real wall-clock time to the suite. Looks deliberate (pinning the actual constant rather than an injected one), and the doc comment already says so — just flagging the cost for anyone later wondering why this one test is slow.
Why
rocm servewith Lemonade, and the Lemonade backend probe and install, could hang indefinitely while tearing down a server. Lemonade'sterminate_pid(pid, force)ignoredforceand only sentSIGTERM(TerminateProcesson Windows), and every caller then did an unboundedchild.wait(). Alemondorllama-serverthat ignoresSIGTERM, for example while it is still loading a model, was never stopped, so the wait never returned. vLLM'sserve_httpreadiness-failure teardown had the same pattern (terminate_process_tree, then an unboundedwait), and so did Lemonade's router-serve teardown,stop_serverinrouter_serve.rs, added by #631. (ROCMAI-452)What changes
terminate_pidgoes throughrocm_core::terminate_verifiedwithKillScope::Treeand the engine's 10 sSTOP_GRACE:SIGTERMto the process tree, thenSIGKILLfor anything still running whenforceis set. Tree scope matters becauselemondruns each loaded model in allama-serverchild, and killinglemondalone would orphan it with the GPU still held. Every production caller passesforce = true, so these teardowns are now bounded: the backend install and backend-selection probes inllamacpp_backend.rs, the model-download server and the direct-server supervisor (supervise_direct_server) indirect_llama.rs, and the router-servestop_server, which switches fromterminate_process_treetoterminate_pid.fail_unready_server, which stops the tree the same way (SIGTERM, thenSIGKILLafter 10 s), reaps the child and records thefailedstate. It does not signal a server the readiness poll already reaped, because that PID may belong to an unrelated process by then. After a stop it cannot confirm, it kills theChildit owns before waiting. A failed write of thefailedstate becomes a stderr warning, so the readiness error stays the error returned.TerminationOutcome::unconfirmed_reason(). It is derived from the same outcome asstopped()and returnsNoneexactly whenstopped()is true, so the warning cannot disagree with the decision it reports.rocm_core::terminate_processandterminate_process_treedocument that they only request termination and neither wait nor escalate, pointing atterminate_verified.STOP_SCOPEcomment no longer claims the recorded PID is always thellama-server: on the router path it islemond, so aSinglestop that escalates can leave its backend running. The scope itself is unchanged.docs/architecture.mdrows for Lemonadestate.rsandrouter_serve.rsand vLLMprocess.rsdescribe the bounded teardown.User-visible change: a failed Lemonade serve (router or direct), model download or backend probe/install, or a vLLM readiness failure, now ends within about 2 × 10 s instead of possibly hanging forever. When a stop cannot be confirmed, stderr shows:
Warning: process N was not confirmed stopped: <reason>.Warning: process N was not confirmed stopped: <reason>. Killing it; processes under it are not reached by that kill.Warning: could not record the vLLM service as failed in <path>: <error><reason>is one of:it, or a process under it, was still running when the stop ended;its identity could not be confirmed, so it was not signalled;it is gone, but a process under it may still be running;it exited before the processes under it were stopped, and they may still be running.Tests
There is no Gherkin scenario (AGENTS.md §3): the hang needs a server that ignores
SIGTERMon a teardown failure path, which the e2e harness cannot stage;service_stop.featuredocuments the same limit for timed-out and unverified stops. The behaviour is covered by unit tests that spawn real processes on Linux, each shown to fail with its target reverted:state.rs: a forced stopSIGKILLs a child (forced_terminate_pid_kills_a_child_that_ignores_sigterm) and a grandchild (forced_terminate_pid_kills_a_grandchild_that_ignores_sigterm) that ignoreSIGTERM, and the caller's followingwaitreturns (terminate_then_wait_returns_for_a_child_that_ignores_sigterm); an unforced stop leaves such a child running and reports it unconfirmed (unforced_terminate_pid_leaves_a_child_that_ignores_sigterm_running); a cooperative child stops onSIGTERMwithout escalation (terminate_pid_stops_a_cooperative_child_with_sigterm);terminate_piditself escalates only after the production 10 s grace (terminate_pid_escalates_after_the_stop_grace_and_the_callers_wait_returns); a re-exec test reads the real stderr warning together with the resulting process state, and checks a confirmed stop does not warn (terminate_pid_reports_an_unconfirmed_stop_on_stderr_and_not_a_confirmed_one).real_router_stop_kills_a_server_tree_that_ignores_sigtermstages a fakelemondand backend that both ignoreSIGTERM;stop_serverkills and reaps both within a bound, and with the fix reverted the test fails through its watchdog instead of hanging.fail_unready_server_*: kills aSIGTERM-ignoring tree and recordsfailed, at a short grace and at the production grace; does not signal a server the readiness poll already reaped; reports an unconfirmed stop on real stderr and kills the server, while a confirmed stop neither warns nor kills.unconfirmed_reason_is_given_exactly_for_an_unconfirmed_stoppinsunconfirmed_reason()againststopped()for everyTerminationOutcomevariant.forced_terminate_pid_stops_a_long_running_child_and_wait_returns(cfg(windows)), run by the Windows CI job.Tradeoffs and limits
SIGKILLed on these teardown paths instead of being waited on.terminate_pid(pid, force) -> boolsignature and does not fall back to a kill after anUnverifiedstop (one whose identity could not be read, so nothing was signalled); the caller's wait can then still block, which is reported on stderr. vLLM owns itsChildand kills it. On Linux,Unverifiedis effectively unreachable for one's own unreaped child.SIGKILL, such as one in uninterruptible sleep, still blocks the caller's wait.terminate_pidterminateslemonditself without a tree walk. The stop counts as confirmed once Windows reports the exit status, and the caller'swait()completes the asynchronousTerminateProcess.serve-httprelaunch nulls stderr, so the warning is not visible on that path.Verification
Run locally on Linux on this head (based on
75fc9c9d):cargo fmt --all --checkandcargo clippy --workspace --all-targets -- -D warnings: clean;cargo test -p rocm-engine-lemonade -p rocm-engine-vllm -p rocm-core: all pass (rocm-core 1646 unit tests with 6 ignored, plus 29 property tests; Lemonade 295; vLLM 174);cargo xtask check-architecture-doc,cargo xtask check-module-sizeand theprekhooks: pass.Not run locally: the
cfg(windows)test, which the Windows CI job runs (a race in it against the asynchronousTerminateProcessis fixed);cargo test --workspace --all-targetsandcargo xtask smoke, which CI runs.Follow-ups (not in this PR)
rocm-corehelper to stop an owned, unreaped child: Lemonade, vLLM androcmdeach hand-roll it.STOP_SCOPE=Treefor router mode, matching the recorded-PID stop paths inrocm services stopandrocmd, which already useTree.direct_llama.rstest cleanups callterminate_pidwith recorded PIDs, which its doc forbids.write_running_state(..)?after the spawn returns without stopping the server it just started.terminate_verified.Checklist
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No row references ROCMAI-452.)docs/architecture.md(the clap declaration incli.rsand the dispatch wiring inmain.rs/lib.rsare expected, not a violation). (Not applicable.).rsfile underxtask/srcortests/e2e-cucumber),docs/architecture.mdhas the matching entry (cargo xtask check-architecture-docfails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. (No file added; the Lemonadestate.rsandrouter_serve.rsand vLLMprocess.rsentries are updated.)e2ejob (for example by a tag) is justified as a deletion (AGENTS.md §8). (Not applicable: no scenario moves.)