Repository navigation
refactor(engine-ipc): share one engine stdio client between rocm and rocmd - #959
Conversation
…nd how they read its answer rocm and rocmd each carry their own client for the JSON-over-stdio request to an engine process, and the two have drifted: they build the child's command line differently, set different environment, treat a failing exit status differently, word their errors differently, and only rocm guards the request write against SIGPIPE. Nothing pinned any of it, so the first step toward one client is to pin what each does today. Both clients are driven against a stand-in engine child, a shell script that records its argv, environment and stdin and then answers (or crashes) as the test asks. The stand-in lives in rocm-engine-protocol behind a test-support feature so the two callers share one. Each caller's out-of-process half is split into a small function that takes the executable to run (engine_stdio_child_request in rocm, engine_request_via in rocmd), so a test can aim it at the stand-in; the bodies are unchanged and the public entry points call them with the same arguments as before. The tests assert the child's argv (rocm: stdio; rocmd: __engine-stdio <engine>), the single newline-terminated request line, the environment each caller sets (rocm: the app path variables, the engine environments root, the progress variable for installs; rocmd: the endpoint key file when it holds a usable key), what each does with a nonzero exit, an unparseable answer and an error envelope, and the SIGPIPE case under the default action in a re-executed child. Mutating rocm's write guard away turns its SIGPIPE test red. No behaviour change. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ared by rocm and rocmd rocm and rocmd each carried their own client for the JSON-over-stdio request to an engine process, and the two had drifted. They now run the child through one implementation in rocm-engine-protocol (stdio_client.rs); each caller still chooses which program to run, and what it asks of the client is a small set of options: the app paths, the engine environments root, an endpoint key file, a progress request, and whether to forward the child's stderr as it arrives. rocm still answers built-in engines in its own process and resolves the engine binary; rocmd still re-executes its own executable as __engine-stdio. Where the two behaved differently, the client takes the behaviour that reports more or fails more safely: - A failing exit status is an error only when the child printed no answer; an answer on stdout is decoded even if the child then exited non-zero (rocm's rule; rocmd discarded the answer for the status). - A parse failure carries the child's stderr, and the decode errors use rocm's wording (rocmd's said "failed to parse engine response", "engine response did not include data" and "failed to deserialize engine response data"). - The write of the request is guarded against SIGPIPE (rocm's rule). rocmd had no guard, so the daemon hosted in rocm, which runs with SIGPIPE at its default action, was killed by a write to an engine that had already exited instead of reporting why. - The child runs on the caller's app paths in both (rocmd now sets the ROCM_CLI_*_DIR variables; it relied on inheriting them), and the spawn error names the program in both. - A request whose write fails kills and reaps the child, so the long-lived daemon does not collect a zombie per failure. - Forwarded stderr no longer panics if this process's stderr is closed. The endpoint key stays an option only rocmd uses, since none of rocm's requests is about a running service's endpoint; rocm still sets no key. Stderr is forwarded for rocm and only kept for rocmd. Neither client has, or gains, a timeout. Also moved to where both can reach them: with_sigpipe_ignored (and its re-exec test, same name) from rocm's main.rs to rocm-core; the three ROCM_CLI_*_DIR variables as AppPaths::env_vars, which rocm's app_path_env_vars and rocmd's capture wrapper now call instead of each listing them; and the byte-identical optional_arg, from both binaries into rocm-engine-protocol. The fake engine written for the tests now plants its script with rocm_core::fake_executable instead of retrying on ETXTBSY. Both callers' tests now run through the shared client and flip what the previous commit pinned (rocmd's app paths, nonzero-with-answer, SIGPIPE under the default action, error wording). Breaking either caller's options, the client's SIGPIPE guard or an environment variable turns a test red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
… the standalone daemon rocmd asked an engine to do something by re-executing its own executable as `__engine-stdio <engine>`. That verb is served by rocm. Embedded as `rocm daemon` the daemon is rocm and it works, but the standalone rocmd binary shipped beside rocm answered every such request with "unrecognized subcommand '__engine-stdio'" and exit status 2. The request was made by the stop tool (`rocmd sandbox-tool stop_server`, `rocmd mcp-server`), by the supervisor's readiness wait and by the server-recover watcher's healthcheck, so from the standalone daemon the engine `Stop` always came back as an error and no healthcheck ever succeeded; for an engine whose stop decides confirmation, the stop could never be confirmed. When the running executable is exactly `rocmd`, the request now runs the `rocm` beside it (rocm_core::sibling_binary_path), and says so if there is none. Anything else, the embedded daemon, a test binary or a renamed copy, keeps using its own executable as before. The `service-stop-02` scenario drives the standalone rocmd and now also asserts that the engine `Stop` request was answered. Run against the previous commit's rocmd it fails with the engine_stop error above; with this change it passes, along with the rest of the service-stop scenarios. The choice of executable is a pure function with its own unit test that also runs on Windows. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`rocm` runs with SIGPIPE at its default action. When it forwards an engine child's progress to a stderr whose reader has gone (a closed pipe), the write delivers the signal and ends the CLI before `write_all` can return, so discarding the result did not protect the request. Both forwarding writes in `collect_stderr` (a line of progress, and the note recorded when reading the child's stderr fails) now go through one helper that runs the write inside `with_sigpipe_ignored`, which turns the signal into the error that is then discarded. `with_sigpipe_ignored` is now safe to overlap. It saved and restored the disposition per call, so two overlapping regions (the stderr forwarder of one request next to the stdin write of another in the daemon) could restore "ignored" over a default action, or the default under a region still open. It remembers the disposition in force before the first region and restores it when the last one ends, and restores on unwind. The re-executed test installs SIG_DFL in a fresh process and forwards into a real pipe whose reader is closed, once per write; the existing `Closed` mock returns the error itself and cannot raise the signal. A second re-executed test covers nested and interleaved regions. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
`FakeEngine` built its script by interpolating the canned response and the scratch directory into single-quoted shell literals. A valid JSON response with an apostrophe (`can't stop`) ended the literal early, so the fake exited with a syntax error instead of printing the envelope, and a `TMPDIR` containing an apostrophe broke the `dir=` line the same way. Both values now go through one `shell_quote` (close the literal, emit an escaped quote, reopen it), and `FakeEngine::new_in` lets a test choose the scratch base without changing the process-wide TMPDIR. Tests cover the quoting helper, the response and the scratch directory against the script itself, and the apostrophe response through the real client. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…cannot be read run_engine_stdio killed and reaped the child when writing the request failed, but a failed read of its stdout returned through `?` first, leaving the child running or as a zombie. The client now runs in the long-lived daemon, so each such failure leaked a process. The read moves into read_response, which kills and waits on the child before returning the error; the stderr collector is not joined on that path because a grandchild holding the pipe could block the join, and the thread ends once the pipe closes. The test reads through a failing reader on a real child and checks that its /proc entry is gone, which a kill without a wait would not achieve. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The interleaved half of overlapping_guards_restore_the_original_disposition_once only made sure both guards were open before the first left, not that the first thread to be spawned entered first. If the other closure entered first, a guard that saved and restored the disposition per call passed both assertions. The second thread now waits on a barrier that the first reaches from inside its guard, so the first region always opens, and closes, before the second opens its own; with a per-call save and restore in place of the shared counter the test fails on every run (five of five). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
Gherkin coverage is missing for sandbox and MCP tool reports and service state when the engine responds before exiting non-zero or emits malformed stdout with stderr.
1 open issue
What changed in this PR
Consolidates rocm and rocmd engine communication into one shared stdio client without changing the wire format, while preserving each caller’s executable selection and options.
Changes:
- Centralizes request framing, environment setup, response decoding, stderr handling, and child cleanup.
- Shares an overlap-safe SIGPIPE guard and fixes standalone
rocmdto invoke its siblingrocm. - Adds shared fake-engine fixtures, regression tests, an end-to-end assertion, and supporting documentation.
Verification used the supplied diff and targeted source inspection; tests were not rerun.
| File | Description |
|---|---|
tests/e2e-cucumber/tests/e2e/service_stop_steps.rs |
Asserts that the daemon receives an engine stop response. |
tests/e2e-cucumber/features/service_stop.feature |
Adds the engine-response assertion to the stop scenario. |
docs/testing.md |
Documents shared-client tests and stop-scenario coverage. |
docs/architecture.md |
Updates module ownership and shared-helper descriptions. |
crates/rocm-engine-protocol/src/stdio_client.rs |
Implements the shared stdio client and regression tests. |
crates/rocm-engine-protocol/src/lib.rs |
Exports the client and shared optional-argument helper. |
crates/rocm-engine-protocol/src/fake_engine.rs |
Adds a reusable shell-based engine test fixture. |
crates/rocm-engine-protocol/Cargo.toml |
Adds client dependencies and test-support configuration. |
crates/rocm-core/src/sigpipe.rs |
Adds the overlap-safe SIGPIPE guard and tests. |
crates/rocm-core/src/paths.rs |
Centralizes child app-path variables and engine-root naming. |
crates/rocm-core/src/lib.rs |
Exports shared path and SIGPIPE helpers. |
Cargo.lock |
Records added dependencies. |
apps/rocmd/src/supervise.rs |
Uses the shared optional-argument helper. |
apps/rocmd/src/engine_ipc.rs |
Adopts the client and fixes standalone executable selection. |
apps/rocmd/src/common.rs |
Reuses app-path variables and removes duplicated argument logic. |
apps/rocmd/Cargo.toml |
Enables shared fixtures and Unix signal-test dependencies. |
apps/rocm/src/main.rs |
Removes the relocated SIGPIPE helper and tests. |
apps/rocm/src/engine_runtime_select.rs |
Delegates app-path variables and removes duplicated argument logic. |
apps/rocm/src/engine_ipc.rs |
Delegates external-engine communication to the shared client. |
apps/rocm/Cargo.toml |
Enables the shared fake-engine fixture for tests. |
Review effort: Balanced
| } | ||
| let key_file = usable_endpoint_key_file(paths, service_id); | ||
| let options = stdio_options(paths, key_file.as_deref()); | ||
| decode_engine_response(run_engine_stdio( |
There was a problem hiding this comment.
Agreed, the new handling needed more than the clean-engine step in service-stop-02. Commit 06371bb adds two scenarios in a new engine_ipc.feature (its own feature key, so the ids cannot collide with service_stop.feature's numbering): engine-ipc-01 and engine-ipc-02. Each copies the real rocmd into a scratch directory next to a stand-in rocm, which is the executable the standalone daemon runs for an engine request, so the daemon's own stop_server tool reaches a child the scenario controls. The stand-in records its argv and the request it was sent, so the scenarios also check that it was run once as __engine-stdio vllm and received one stop request for the planted server.
engine-ipc-01 is an engine that answers the stop and then exits non-zero: the report carries the engine's answer (attempted, stopped, graceful) rather than an error built from the exit status. engine-ipc-02 is an engine that prints something that is not an answer, writes to stderr and exits non-zero: the report's engine error is the parse failure with that stderr appended. Both assert the report together with the state the stop left behind: the recorded process has exited, the record reads stopped and names no process, and the endpoint key file is gone. The planted server is a vllm one, for which the engine's Stop is advisory, so in -02 the stop still completes and the scenario says so; an engine whose stop decides the confirmation is not staged, because the harness teardown would then run that engine's real stop.
I checked that they fail when the code is broken: restoring the old rule (a failing exit status is an error even when the child printed an answer) turns both red, dropping the stderr from the parse failure turns engine-ipc-02 red, and making the standalone daemon run itself instead of the rocm beside it turns both red. They are Linux-only and run in the mock lane with the rest of the e2e suite (no GPU); docs/testing.md describes them and how to run them with -n "engine-ipc".
…non-zero or prints something unreadable The engine client shared by rocm and rocmd changed what the standalone daemon's tools report when an engine child ends badly: an answer printed before a failing exit is now decoded, and a parse failure carries the child's stderr. Only service-stop-02 exercised the daemon against an engine, and only a clean one. Two scenarios in a new engine_ipc.feature copy the real rocmd into a scratch directory next to a stand-in rocm, which is the executable the standalone daemon runs for an engine request. The stand-in records its argv and the request it was sent and then answers and exits non-zero (engine-ipc-01), or prints something that is not an answer, writes to stderr and exits non-zero (engine-ipc-02). Each checks what the stand-in itself recorded, the message in the stop_server report, and the server's process, record and endpoint key together: for the server used here (vllm) the engine's Stop is advisory, so the stop completes in both and the report says why the request did or did not succeed. The steps live in service_stop_steps.rs with the planting helpers they reuse. The scenarios use their own feature key so they cannot collide with the numbering of service_stop.feature. Reverting the exit-status rule, dropping stderr from the parse failure, or making the standalone daemon run itself turns the scenarios red. docs/testing.md describes them. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
juhovainio
left a comment
There was a problem hiding this comment.
Summary
PR #959 replaces the two separate JSON-over-stdio engine clients in rocm and rocmd with one client, run_engine_stdio, in rocm-engine-protocol (new stdio_client.rs). The callers still choose the program and pass an EngineStdioOptions. The PR also:
- moves
with_sigpipe_ignoredintorocm-coreand makes it safe for overlapping regions; - moves
AppPaths::env_vars,ENGINE_ENVS_ROOT_ENVandoptional_argto shared places; - adds a shared
FakeEnginetest double; - fixes a real bug: the standalone
rocmdre-executed itself with__engine-stdio, a verb onlyrocmserves. It now runs therocmbeside it.
I read the new client against the old rocm client and the old rocmd client, and the rocm side is a faithful port. I found no blocking correctness bug. Confidence is fairly high. Remaining findings are diagnostic regressions and robustness issues. CI is green on everything that has finished. The W7900 E2E job was still queued and I did not wait for it.
PR author: jussielo-amd. Commit reviewed: 06371bb (8 commits, head ref split-engine-ipc-client, same repo).
Blocking
None.
Non-blocking
-
crates/rocm-engine-protocol/src/stdio_client.rs:176-183. When the child exits non-zero but its stdout is non-empty and not valid JSON, the error is "failed to parse engine response envelope[; stderr: …]" and drops the exit status. That is the new "decode the answer even on a failing exit" rule.
- Example: an engine that is OOM-killed or crashes partway through printing its answer now reports only a parse failure.
- The old
rocmdreported "exited with status …" for any failure. Forrocmdthe status information is lost. - Suggested fix: when
!status.success(), add the status to the parse-failure context, for example "failed to parse engine response envelope (engine exited with status X); stderr: …".engine-ipc-02asserts the parse-failure-plus-stderr wording, so its expected text would need updating.
-
crates/rocm-engine-protocol/src/stdio_client.rs:143-149. When the request write fails (EPIPE), the child is killed and reaped without collecting its stderr.
- The PR description and the
sigpipe.rsmodule docs say the guard lets the caller report why the engine exited. In practice the error is only "failed to write engine request: Broken pipe" and the engine's own message is discarded. The oldrocmcode behaved the same, but the stated intent is not met. - Suggested fix: if
child.try_wait()shows the child has already exited, read its stderr (EOF is certain) and append it to the error before returning. Otherwise keep the kill and wait. - Test gap: the write-failure kill/reap path has no test of its own.
a_failed_read_of_the_answer_kills_and_reaps_the_childcovers only the read path, and the SIGPIPE test only checks the error text.
- The PR description and the
-
crates/rocm-engine-protocol/src/stdio_client.rs:165-167. On the success path,
reader.join()afterwait()can block forever if the engine left a grandchild holding the stderr pipe.read_to_endon stdout has the same exposure. The PR's own comment onread_responseacknowledges this for the failure path. Now that the client lives in the long-livedrocmdand has no timeout, one hung request would also stall the supervisor's readiness wait and the healthcheck. The old clients shared this, so it is not a regression. Consider a bounded join, or at least a comment stating the accepted limitation. -
crates/rocm-engine-protocol/src/stdio_client.rs:243-245.
forward_best_effortflips the process-wide SIGPIPE disposition once per forwarded stderr line. During a longrocminstall, other threads in the process (for example one writing to a stdout piped intohead) will intermittently get EPIPE instead of the conventional SIGPIPE termination. The window is small and the design is documented. An alternative is to guard once around the whole forwarding loop, or skip the guard unless stderr is actually broken. -
apps/rocmd/src/engine_ipc.rs:153. The standalone check compares
file_stem == "rocmd"againstcurrent_exe().- On Linux,
current_exe()returns…/rocmd (deleted)if the binary was replaced while the daemon is running (an in-place upgrade). The stem then no longer matches, the code falls back to the deleted self-exe, and the spawn fails with an unrelated error. - It is also not clear what happens if a packager ships it under another name, such as
rocmd-<version>. The fallback silently reintroduces the original "unrecognized subcommand" failure. - A cheap hardening is to strip a trailing " (deleted)" before the check.
- On Linux,
Questions for the author
- Is the daemon hang described above on the success path (grandchild holding stderr or stdout) acceptable, or do you want a bounded wait before the respawn-guard follow-up lands?
- Should the parse-failure message carry the exit status when the child exited non-zero? The old
rocmdsurfaced the status in every failing case. - Is the "(deleted)"
current_exe()case worth handling, given that standalonerocmdis the one that now depends on exact-name detection?
What I could not verify
- I did not build or run any tests. The sandbox has no Rust toolchain (
cargois absent), so the unit tests, the e2e scenariosengine-ipc-01,engine-ipc-02andservice-stop-02, and the mutation checks in the PR description are unchecked by me. I relied on the PR's reported results and on the check runs, which are green except the queued W7900 E2E job. The review is by reading the code. - I did not read
docs/testing.mdor the full step implementation of the e2e scenarios, only the feature file head and the diff stats. - I did not read the Jira ticket ROCMAI-875.
- Windows behaviour was not checked beyond noting that the SIGPIPE code is
cfg(unix)-gated and a no-op on Windows.
| } | ||
| bail!("engine stdio process exited with status {status}: {stderr}"); | ||
| } | ||
| serde_json::from_slice(&stdout).with_context(|| { |
There was a problem hiding this comment.
When the child exits non-zero with non-empty but unparseable stdout, this error drops the exit status. A killed or crashed engine that printed a partial answer is reported as a bare parse failure (the old rocmd always reported the status). Please add the status to the context when !status.success(); engine-ipc-02's expected text will need updating.
There was a problem hiding this comment.
Agreed, and this is the answer to question 2 (the exit status in the parse failure). Fixed in 3e292d6. When the child exited with a failing status and left something on stdout that is not an envelope, the error is now failed to parse engine response envelope (engine stdio process exited with status exit status: 1); stderr: ..., worded with the same note the failing-exit branch uses (one exit_note function builds both, so the two cannot drift). A child that exited cleanly with an unparseable answer adds no status, and a failing child that printed nothing is still reported by its status alone, so every failing case now names the status, as the old rocmd did. I swept the other error branches: the only other one that could drop a status is a failed read of stdout, where the child is killed by us and its status is our own SIGKILL, so there is nothing to report; the failed-write path is covered by the thread on that point. Tests: an_unparseable_answer_from_a_failing_child_names_the_exit_status pins the three cases against the fake engine (it fails if the status is dropped), and engine-ipc-02 now expects the new text in the stop_server report while still asserting the server's process, record and endpoint key file in the same scenario. docs/testing.md describes both.
| stdin.write_all(b"\n")?; | ||
| Ok::<(), anyhow::Error>(()) | ||
| }); | ||
| if let Err(error) = written { |
There was a problem hiding this comment.
On a failed request write the child is killed and reaped without reading its stderr, so the engine's own explanation is discarded. This falls short of the stated aim of reporting why the engine exited. If try_wait() shows it already exited, read stderr and append it to the error. This write-failure kill and reap path also has no test of its own; only the read path is covered.
There was a problem hiding this comment.
You are right that the stderr was discarded. Fixed in 87c0d2f: the error is now failed to write engine request: <cause>; stderr: <text>. One deviation from the suggestion: I do not gate the read on try_wait showing an exit. The pipe closes while the process is still exiting, so a try_wait made right after the failed write can still see the child running and would drop exactly the text this adds. The child is always killed (nothing to do if it has exited) and waited for, and then its stderr is read. The read is bounded by a one second grace instead of assuming EOF is certain, because a grandchild that inherited the pipe keeps it open past the exit; what had arrived by then is reported and the error path never blocks on it. Tests: a_request_to_an_engine_that_exited_reports_what_it_said_on_stderr runs the whole client against a fake that exits without reading a 1 MiB request (red if the stderr read is removed), a_request_that_cannot_be_written_kills_and_reaps_a_child_that_is_still_running checks the child's /proc entry and fails instead of hanging if the kill is removed, and the_stderr_of_a_failed_write_is_not_waited_for_past_the_grace uses a grandchild that keeps the pipe open (red if the bound is removed). The last two run the step on a real child with an injected write error, since whether a real write fails before the child closes its stdin is a race.
| .wait() | ||
| .context("failed waiting for engine stdio response")?; | ||
| let stderr = stderr_reader | ||
| .map(|reader| reader.join().unwrap_or_default()) |
There was a problem hiding this comment.
reader.join() after wait() blocks forever if a grandchild inherited the stderr pipe. read_response's docs call this out for the failure path, but the success path has it too, and now in the long-lived rocmd. Consider a bounded wait, or document the accepted limitation.
There was a problem hiding this comment.
You are right, and this is the answer to question 1 (is the success-path hang acceptable). The hang exists: after wait() the join of the stderr collector, and the read_to_end of stdout before it, wait for every holder of the pipes, so a grandchild that outlives the engine blocks the request. It is not new: the old rocm client read stdout to the end and joined its stderr thread, and the old rocmd used wait_with_output, which reads both pipes to EOF. I decided to document it precisely rather than bound it in this change (ace9d9f: module comment, docs/testing.md and the architecture entry). Bounding it exactly means reading both pipes on their own threads with shared buffers, waiting for the child itself without a limit (installs run for minutes) and for the pipes only up to a deadline after it exits, and deciding what to report when the deadline passes (a complete answer already read, a partial one, or an error); it also reorders the kill-on-failed-read handling this PR added. That is a timeout policy for an engine contract that has none, not a small mechanical fix, so I would rather decide it on its own than slip it into a refactor that is otherwise behaviour-preserving. The tradeoff is that a daemon request can still wait on a stray grandchild until it is fixed separately. The one wait this client adds itself, the stderr of a child whose request could not be written, is bounded (see the thread on that path).
| /// write, so a closed pipe comes back as the error that is then discarded | ||
| /// rather than as a signal that kills the process. | ||
| fn forward_best_effort(sink: &mut impl Write, bytes: &[u8]) { | ||
| let _ = with_sigpipe_ignored(|| sink.write_all(bytes)); |
There was a problem hiding this comment.
This flips the process-wide SIGPIPE disposition once per forwarded stderr line. During a long install, other threads writing to a closed stdout pipe get EPIPE instead of SIGPIPE in those windows. Consider guarding the whole forwarding loop once, or only guarding when a write has actually failed.
There was a problem hiding this comment.
I checked both suggestions and do not think either is right, so the code stays and a test now pins why (9d46e16). The guard flips the process-wide disposition twice per forwarded line, which is real but costs two signal calls beside a write. Guarding once around the whole loop would keep SIGPIPE ignored for the entire run, and the loop spends almost all its time blocked in reads (an engine install is quiet for long stretches): every other thread's write to a closed stdout would then return an error for that whole time instead of ending rocm quietly, which is what the default action is for. Guarding only after a failed write cannot work, since under the default action a write to a closed pipe ends the process before it can fail. The guard is counted, so overlapping regions restore the original disposition once. The new test, forwarding_guards_each_write_and_never_the_reads_between_them, runs the collector in a fresh process with the default action over a reader and a sink that record the disposition seen inside every read and every write, and requires ignored inside each write, the default action inside each read, and the default action after the loop; wrapping the whole collector in the guard turns it red.
| ) -> Result<PathBuf> { | ||
| // Exactly `rocmd` (or `rocmd.exe`). A renamed or test binary is not known to | ||
| // lack the verb, and has no `rocm` beside it to prefer. | ||
| if current.file_stem().is_some_and(|stem| stem == "rocmd") { |
There was a problem hiding this comment.
The exact rocmd stem match fails when Linux current_exe() returns …/rocmd (deleted) after an in-place upgrade. The code then falls back to the nonexistent self exe and the engine request fails with an unrelated spawn error. Consider stripping a trailing " (deleted)" before comparing.
There was a problem hiding this comment.
Confirmed, and this is the answer to question 3 (the (deleted) case). On Linux, current_exe() of a process whose executable was replaced reports <path> (deleted), which is the usual state of a long-lived daemon right after an in-place upgrade; the stem then read rocmd (deleted), did not match, and the daemon ran that name as the engine launcher, so every request failed to spawn with an unrelated error. Fixed in b648ab7: a pure without_deleted_marker strips only an exact trailing (deleted) from the file name before the check, so the sibling rocm (looked up in the same directory) is found. The same strip applies to the path an embedded rocm daemon runs itself by, which would otherwise name a file that does not exist. an_executable_replaced_while_running_is_still_told_apart_by_its_name runs on every platform with injected paths, and goes red when the strip is removed. On your second point: a standalone rocmd shipped under another name is not recognised, the check falls back to running its own executable, and that reintroduces the unrecognized subcommand failure (the comment now says so). Fixing that exactly would need a capability check rather than a name check, which is not tiny, so I left it; shipping the binary as rocmd is the supported layout.
…replaced On Linux, current_exe() of a process whose executable file has since been replaced or removed ends in " (deleted)", which is the usual state of a long-lived daemon right after an in-place package upgrade. The standalone check compared the file stem against "rocmd", so "rocmd (deleted)" no longer matched, the daemon ran that name as the engine launcher, and every engine request failed to spawn with an error that had nothing to do with the cause. The marker is now stripped from the file name first, by a pure helper, so the daemon still finds the rocm beside it. The same strip applies to the path an embedded rocm daemon runs itself by, which would otherwise name a file that does not exist. A standalone rocmd shipped under another name is still not recognised: it runs itself, and answers the engine request with "unrecognized subcommand". The comment says so. The test runs on every platform with injected paths and fails when the strip is removed. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…wer cannot be parsed A child that exited with a failing status after printing something that is not a response envelope was reported as "failed to parse engine response envelope" with its stderr, which dropped the status that the old rocmd always reported for a failing child. The status is often why the answer was cut short, so the parse failure now carries it, worded as the failing-exit error already is: "failed to parse engine response envelope (engine stdio process exited with status ...); stderr: ...". A child that exited cleanly adds nothing, and a failing child that printed nothing is still reported by its status alone. The wording of the exit note comes from one function, so the two messages cannot drift. The unit test pins all three cases against the fake engine. The engine-ipc-02 scenario now expects the new text in the stop_server report, still together with what the stop did to the process, record and key, and docs/testing.md describes it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…equest cannot be written A request whose write failed (the engine exited, or closed its stdin, before reading it) killed and reaped the child but dropped its stderr, although the engine's own message is usually the reason it was not listening and the error was meant to let the caller report why. The child is still killed and waited for, and its stderr is now read and appended to the error: "failed to write engine request: <cause>; stderr: <text>". The kill is not made conditional on try_wait showing an exit first, because the pipe closes while the process is still exiting, so a check right after the failed write can see it running and would discard the very text this adds. The read of the stderr is bounded by a short grace: a grandchild that inherited the pipe keeps it open past the exit, and this error path must not block on it; what had arrived by then is reported. Tests: the whole client against a fake that exits without reading (the stderr must be in the error), the step on a real child that is still running (killed and reaped, by its /proc entry; the test fails instead of hanging when the kill is removed), and on a child whose grandchild holds the pipe (returns within the grace, with what had been written). Removing the stderr read, the kill, or the bound each turns one of them red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…t exits is unbounded After the child exits, the stdio client reads its stdout to the end and joins its stderr collector, so an engine that leaves a grandchild holding either pipe makes the request wait for that grandchild. The clients this one replaced had the same exposure (rocm read stdout to the end and then joined its stderr thread; rocmd used wait_with_output), so this is not a regression, but the client now lives in the long-lived daemon. Bounding it exactly would mean reading both pipes on their own threads, waiting for the child itself without a limit and for the pipes only up to a deadline after it exits, and deciding what to report when the deadline passes (a complete answer, a partial one, an error). That is a timeout policy for an engine contract that has none, and it reorders the kill-on-failed-read handling, so it is not folded into this change. The module comment, docs/testing.md and the architecture entry now say precisely what is and is not bounded. The one wait this client adds itself, the stderr of a child whose request could not be written, is bounded. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…nd not the reads between them Forwarding the child's stderr guards each write against SIGPIPE separately, which flips the process-wide disposition twice per forwarded line. Guarding once around the whole forwarding loop would remove that, but the loop spends nearly all its time blocked in reads (an engine install is quiet for long stretches), so the disposition would stay ignored for the whole run: every other thread's write to a closed stdout would then return an error instead of ending rocm quietly, which is what the default action is for. Guarding only after a failed write is not possible, since under the default action a write to a closed pipe ends the process before it can fail. The cost of the per-line guard, two signal calls beside a write, is negligible. So the code stays as it is and a test pins the scope: a fresh process with the default action runs the collector over a reader and a sink that record the disposition seen inside every read and every write, and the test requires ignored inside each write and the default action inside each read and after the loop. Wrapping the whole collector in the guard turns it red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ill on a missing pipe, note a failing exit after an answer
A failed read of the child's stdout dropped the stderr already collected, unlike every other failure path; the error now carries it ("...: pipe went away; stderr: ..."), through the same bounded grace as a failed write. A child whose stdin or stdout pipe is missing is killed and reaped instead of being left running (unreachable while the command pipes all three streams, so the check is a small function a test drives with a missing pipe).
A child that answers and then exits with a failing status is still read as having answered, but its status is no longer dropped without a trace: the daemon asks the client (note_failing_exit_after_answer) to leave a one-line note on its stderr, which is its log. The CLI does not ask, since its stderr is a terminal. The decision is unchanged.
StderrCollector::finish now takes the collected bytes instead of cloning them. The three-line loop that set the app-path variables on a command, repeated in the client and in rocm's apply_app_path_env, is one AppPaths::apply_env in rocm-core, and rocm's copy and its imports are gone.
Each new test was shown red by removing the behaviour it covers; docs/testing.md describes them.
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
r0x0r
left a comment
There was a problem hiding this comment.
I read all 25 files in this PR end to end. This is a good refactor: the two divergent engine stdio clients genuinely collapse into one, each caller's own choices stay pinned by its own tests rather than being quietly homogenised, and every behaviour change is spelled out in the description instead of being left for a reviewer to discover.
The part I liked most is that starting the stderr collector before the request write isn't just tidier ordering — it fixes a real deadlock both old clients had, and an_engine_that_writes_a_lot_of_stderr_before_reading_its_request_does_not_deadlock hangs without it. That's a bug fix smuggled inside a refactor, in the good sense.
xtask/src/engine_stdio_client_contract.rs is also the right call. It guards the existence of a second client, which no behavioural test can observe, and because it's bidirectional the exemption can't quietly outlive the reason it was granted.
The three deliberate limits — the unbounded post-exit pipe wait, the undecoded answer after an undrained large request, and process-wide SIGPIPE during a stalled forward — are stated in the module comment, in docs/testing.md and in the PR body rather than left implicit. I'd much rather review a known limit than find it later.
Deployment notes worth flagging loudly, because they're user-visible: rocmd's engine errors change. An answer on stdout now wins over a failing exit status, where it used to be discarded. Parse failures now name the exit code or signal. And the wording moves from exited with status exit status: N to exited with code N, so anything parsing daemon stop or healthcheck error text needs updating. rocmd also now sets ROCM_CLI_*_DIR on the engine child explicitly instead of relying on inheritance.
One tradeoff I want to confirm rather than block on. A failed request can leave a stderr-drain thread alive indefinitely: stdio_client.rs:343 detaches the collector and it has to keep draining, because closing the pipe would SIGPIPE a surviving grandchild. So a long-lived daemon meeting a repeatedly-misbehaving engine accumulates threads. Memory growth is bounded by the abandon flag, but thread count isn't. This is not a regression — the old rocmd hung the calling thread instead, which is worse — so I'm fine shipping it. I'd just like confirmation it's a conscious choice.
I also couldn't verify the coverage number from outside: cargo xtask coverage for rocm-engine-protocol isn't among this PR's checks, so the reported 99.16% against a 98.37% floor is taken on trust.
On this approval: it covers the code, and only the code. Merge is still independently gated by three red REQUIRED security checks on this head — Security scan / trivy, Security scan / gitleaks, and Security scan / Security scanners. None of them are this PR's doing (details in the inline comment), but they have to go green regardless, and my approving the code does not clear them.
| // | ||
| // SPDX-License-Identifier: MIT | ||
|
|
||
| //! The client side of the engine stdio protocol: run an engine as a child |
There was a problem hiding this comment.
This comment is about CI state, not this line — GitHub needs an anchor inside the diff, so I've put it on the new shared client rather than claiming the problem lives here.
Three REQUIRED security checks are red on this head and gate merge: Security scan / trivy, Security scan / gitleaks, and Security scan / Security scanners.
None of it looks caused by this change. Trivy is failing on Dockerfile misconfigurations (ds-0002, EXPOSE 22, ds-0026) in tests/remote-ssh/Dockerfile, which this diff doesn't touch at all. gitleaks is red on sibling PRs #960 and #961 too, so it isn't specific to this branch either.
Suggested path: rerun, or escalate as infrastructure per AGENTS.md section 12. Worth noting that #1000 ("ci(security): allow model names and runtime keys assigned in Rust sources in gitleaks") has since merged, so rebasing onto current main may well clear the gitleaks half on its own.
| - [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) — mechanical relocation out of [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), no owned types: which runtime an engine installs into and where its environment lives. `engine_manages_own_runtime` says whether an engine brings its own runtime; `env_root_for_engine_install`, `env_root_for_runtime` and `env_root_for_self_managed_engine` resolve the engine environment root; `runtime_manifest_for_selector` resolves a selector to one installed runtime; `runtime_key_for_python` names the runtime whose install root contains an interpreter (the private `runtime_key_owning_python` is the decision behind it). It is the neutral layer that [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs) and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) (with its `torch_alignment.rs` child) both import from, so neither imports the other for these helpers. It imports `recover_setup_runtime_registration` from [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs), `select_runtime_manifest` and `validate_runtime_manifest_for_activation` from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and the runtime-manifest model from [`therock.rs`](../apps/rocm/src/therock.rs); it does not go through the re-exports in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs), which would lead back to it via `runtime_services.rs`. Its functions are `pub(crate)` for those two modules plus [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) and [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs). Their tests moved with them. | ||
| - [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-213): resolving an engine's binary path (`resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `find_engine_plugin_binary`), the stdio request/response envelope used to talk to an external engine binary or the in-process built-in handler (`engine_request`, `engine_request_with_env_root`, `decode_engine_response`, the `ScopedEnvVar`/`with_scoped_builtin_engine_env` env-scoping helpers and the `BUILTIN_ENGINE_ENV_LOCK` mutex they serialize on, `builtin_engine_request`, `run_builtin_engine_stdio`), and the built-in HTTP-serving path (`builtin_engine_available`, `run_builtin_engine_serve_http`, `builtin_engine_serve_http_args`). One item that sits physically inside this span in `main.rs` stayed there instead of moving: `impl From<WatcherModeArg> for WatcherMode` is unrelated clap-arg conversion that merely happened to be interleaved between `find_engine_plugin_binary` and `engine_request` (it has since moved to [`cli.rs`](../apps/rocm/src/cli.rs), next to the enum it converts). `resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `engine_request`, `engine_request_with_env_root`, `builtin_engine_available`, and `builtin_engine_serve_http_args` are `pub(crate)` and read from other modules (for example [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and [`service_launch.rs`](../apps/rocm/src/service_launch.rs)), not from `main.rs`. `run_builtin_engine_stdio` and `run_builtin_engine_serve_http` are `pub(crate)` and reached directly from `main.rs`'s `dispatch()` (the `Command::EngineStdio` and `__engine-serve-http` arms); `find_engine_plugin_binary`, `decode_engine_response`, `with_scoped_builtin_engine_env`, `builtin_engine_request`, and `ScopedEnvVar` have no callers outside this module and stay private. | ||
| - [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-215): deciding which runtime/env a `serve`/`engines install` invocation resolves to (`EngineSelection`, `resolve_engine_selection`, `validate_engine_selection_runtime`, `single_ready_runtime_key`, `resolve_runtime_selector_to_exact_key`), small argv-building helpers used when relaunching a managed engine (`optional_arg`, `engine_recipe_json_arg`, `parse_engine_recipe_json_arg`, `daemon_run_argv`, `managed_service_launcher_path`), the `ROCM_CLI_*` environment variables every managed child inherits (`app_path_env_vars`/`app_path_env_var_values`/`app_path_env_var_refs`/`apply_app_path_env`), and polling a freshly launched service until it can answer inference requests (`wait_for_service_http_ready`, `wait_for_service_http_ready_with_progress`, `service_http_readiness_paths`, `http_get_local_service`, `http_post_local_service_json`, `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`, `model_list_ready_for_model`, `service_model_gpu_ready`, `service_model_reports_rocm_backend`, `service_model_names_match`). `single_ready_runtime_key` does not decide which runtimes count as ready: it asks [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs)'s `select_single_ready_runtime`, which `rocm comfyui install` shares. `const fn status_for_readiness` sits physically interleaved in this span in `main.rs` but stayed there instead of moving — it has callers outside this cluster ([`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `record_restart_readiness`, which `restart_internal_managed_service_with` calls), so it stays a shared crate-root function, reached only from this module's own tests, via `use crate::status_for_readiness;`. Which items are `pub(crate)` and who calls them is documented in the module's own doc comment rather than repeated here. The cluster's regression tests live in the module's own `mod tests`. What remains in `main.rs` immediately after this cluster is `treat_as_natural_language`, dispatch-adjacent natural-language-invocation routing left over from an earlier phase (ROCMAI-87), explicitly out of scope and untouched by this split. | ||
| - [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-213): resolving an engine's binary path (`resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `find_engine_plugin_binary`), the request entry points, which either answer from the in-process built-in handler or run the resolved engine binary with `stdio` (`engine_request`, `engine_request_with_env_root`, and `engine_stdio_child_request`, which hands the child to the stdio client `rocm` shares with `rocmd`, from [`stdio_client.rs`](../crates/rocm-engine-protocol/src/stdio_client.rs), and chooses what `rocm` asks of it: the app paths and engine environments root for the child, progress for an install, and the child's stderr forwarded to the terminal), the `ScopedEnvVar`/`with_scoped_builtin_engine_env` env-scoping helpers and the `BUILTIN_ENGINE_ENV_LOCK` mutex they serialize on, `builtin_engine_request`, `run_builtin_engine_stdio`), and the built-in HTTP-serving path (`builtin_engine_available`, `run_builtin_engine_serve_http`, `builtin_engine_serve_http_args`). One item that sits physically inside this span in `main.rs` stayed there instead of moving: `impl From<WatcherModeArg> for WatcherMode` is unrelated clap-arg conversion that merely happened to be interleaved between `find_engine_plugin_binary` and `engine_request` (it has since moved to [`cli.rs`](../apps/rocm/src/cli.rs), next to the enum it converts). `resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `engine_request`, `engine_request_with_env_root`, `builtin_engine_available`, and `builtin_engine_serve_http_args` are `pub(crate)` and read from other modules (for example [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and [`service_launch.rs`](../apps/rocm/src/service_launch.rs)), not from `main.rs`. `run_builtin_engine_stdio` and `run_builtin_engine_serve_http` are `pub(crate)` and reached directly from `main.rs`'s `dispatch()` (the `Command::EngineStdio` and `__engine-serve-http` arms); `find_engine_plugin_binary`, `with_scoped_builtin_engine_env`, `builtin_engine_request`, and `ScopedEnvVar` have no callers outside this module and stay private. |
There was a problem hiding this comment.
Small docs nit: there's an orphaned close-paren on this line.
The parenthetical that opens at (`engine_request`, `engine_request_with_env_root`... is already closed at "the child's stderr forwarded to the terminal)". That leaves the ) after run_builtin_engine_stdio unmatched — and as a result the ScopedEnvVar/with_scoped_builtin_engine_env and BUILTIN_ENGINE_ENV_LOCK clause reads as though it sits inside a group that has already been closed, when it's actually part of the top-level list.
Fix is just to drop the stray ).
…ropping apply_app_path_env Removing the cfg(not(windows)) import of apply_app_path_env left its attribute attached to the next use in service_control.rs, service_launch/spawn.rs and service_launch/helper_autostart.rs, which made managed_service_launcher_path unresolved on Windows (and app_path_env_var_values / app_path_env_var_refs in service_control.rs). Only a Windows-target build shows it. The attribute now goes with the line it belonged to, and the whole workspace passes clippy with -D warnings for x86_64-pc-windows-msvc. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 5ad422fa
Review — needs work
Full review of the whole change (29 files, +3639/-421 across crates/rocm-engine-protocol, crates/rocm-core, apps/rocm, apps/rocmd, xtask, tests/e2e-cucumber, docs). Implements ROCMAI-875 ("share one engine IPC client between rocm and rocmd") — its acceptance criteria (one shared client, old copies deleted, rocmd never depends on rocm, every dependent engine verified, docs/architecture.md updated, a test that fails on re-divergence) are each met by the diff, with one caveat noted below on the re-divergence test's actual coverage.
Blocking
-
A new, non-test diagnostic write in the shared client is not SIGPIPE-guarded, unlike every other write in this module —
crates/rocm-engine-protocol/src/stdio_client.rs:263-268.
The module guards every write to a sink this process doesn't control (the request write, and each stderr-forward write) withwith_sigpipe_ignored, becauserocmresets SIGPIPE toSIG_DFLat startup and hostsrocmdin the same process. Theeprintln!added fornote_failing_exit_after_answer(a new option this PR introduces, enabled only byrocmd'sstdio_options,apps/rocmd/src/engine_ipc.rs:216) is not wrapped. Ifrocmd's own stderr is a closed pipe — plausible for a long-lived daemon whose stderr is its log and can be redirected to something that closes (log rotation, a collector pipe, a restarted supervisor) — this write either delivers SIGPIPE and kills the process outright (when hosted insiderocm, which runs underSIG_DFL) or panics viaeprintln!'s internal.expect()on theEPIPEit gets back (under the standalonerocmdbinary's defaultSIG_IGN). The one test for this option (a_failing_exit_after_an_answer_is_noted_only_when_asked_for) captures the re-exec'd child's stderr throughCommand::output(), an actively-read pipe, so it cannot exercise the closed-pipe case — this is new behaviour the diff introduces with no test covering its failure mode, on the path specifically meant to make the daemon more robust.
Confidence 85 — traced directly against the module's own documented invariant and the one test that exists for this code path. Fix: wrap theeprintln!inwith_sigpipe_ignored, matching every other write to a process-external sink in this file, and add a closed-pipe test alongside the existingforwarding_stderr_to_a_closed_pipe_is_not_fatal_under_sig_dfltest one function above it. -
The new "one client" contract test's detection is narrower than it claims, in a way the codebase's own style already produces —
xtask/src/engine_stdio_client_contract.rs:118-127(is_a_client/pipes_stdin).
pipes_stdinrequires.stdin(andpiped()to appear on the same source line. Splitting them — aStdiobuilt on one line and passed as a parameter, with.stdin(stdin)called on another — defeats detection entirely; I reproduced the exact check against that shape and confirmed it. This isn't a contrived evasion: the identicalfn f(..., stdin: Stdio) { ... .stdin(stdin) ... }shape already exists twice in this repo (run_shell_command_with_stdininapps/rocm/src/driver_install.rs:407,run_argv_with_stdininapps/rocm/src/system_packages.rs:70), so an ordinary future refactor of a re-diverged client is quite likely to land in this blind spot, not just an adversarial one. The module's own doc comment discloses a different, narrower gap ("a copy that builds its stdin another way... is outside what it can see") but not this one, so a reader is led to trust a check that's weaker than described — per this project's own standard, a check that can't fail on a branch it claims to cover is worse than no check, because it tells the next reviewer the ground is covered when it isn't. It does still fail against the straightforward case (pasting the old client body back), which is what the PR's test plan demonstrates — but the ticket's acceptance criterion ("a test fails if a second copy re-diverges") is only partially met by it.
Confidence 85 — reproduced the detection logic directly, and the two example functions are current code in this repo, not a hypothetical. Fix: tightenpipes_stdinto look for both tokens inside the same function body (e.g. track brace depth, or scan per top-levelfn) rather than per source line, or add arg/text-scan test asserting the check still fires against a deliberately split-line fixture.
Non-blocking
None.
Decisions for the author
- The unbounded wait on a child's pipes after it exits, when a grandchild holds one open — tradeoff. Both old clients had this limit too (
rocmread stdout to the end and joined its stderr thread;rocmdusedwait_with_output), so nothing new is exposed, and the module comment and PR description both already state this plainly as a deliberate, not-yet-solved limit rather than an oversight. Worth the daemon's maintainers explicitly confirming this is acceptable forrocmd's own readiness-wait loop (apps/rocmd/src/supervise.rs'swait_for_service_ready), which polls with an overall timeout but has no inner deadline around a single blocking engine request — a hung grandchild on one poll could outlast the loop's own budget. Not a new regression and not something to fix here; flagging only so it's a confirmed-acceptable choice rather than a quietly-inherited one.
Positive signals
- The stderr-forwarding/SIGPIPE-guard interaction is implemented exactly to its stated spec (guard held only around the write, never the blocking reads between lines) and is pinned by a dedicated test that would fail if the scope were widened or misplaced.
- The stderr collector starts before the request write, not after — the correct fix for the classic both-sides-blocked-on-a-full-pipe deadlock — and is exercised by a dedicated large-stderr-before-read test.
- The SIGPIPE guard's "remembered disposition" design (capture only at the first entrant, restore only at the last) is correct under nested and concurrent interleavings, including panic-driven unwinding, and its test suite uses
thread::scope+Barriers plus re-exec'd child processes to make an otherwise-racy interleaving deterministic — the right technique for testing process-global signal state. AppPaths::env_vars/apply_envcleanly centralizes three previously hand-duplicatedROCM_CLI_*_DIRenv-var names into one round-tripping contract withAppPaths::discover, and every production call site was actually migrated (verified by grep; no stray literal survives outside tests and docs).- New e2e coverage (
engine-ipc-01/-02, theservice-stop-02addition) asserts exact JSON/error-string values and independently-observable server state together, not just a loose substring match — consistent with this project's stated testing standard for claims about the CLI's own behaviour.
Deployment notes
No migration or breaking external-API change. User-visible behaviour differences are intentional and documented by the PR for rocmd specifically (answer-wins-over-exit-status, SIGPIPE-safe write, explicit ROCM_CLI_*_DIR env threading for the spawned rocm __engine-stdio child). CI at this head is green across all 41 checks (build/test on Linux and Windows, clippy, CodeQL, security scanners, coverage, e2e); a GitHub security-scan bot comment flagging 3 "generic-api-key" gitleaks hits is a verified false positive on test-fixture strings (model/version identifiers such as "Qwen3-4B-Instruct-2507-GGUF"), not a real secret, and is on commits not part of the final diff against main.
What this covered
Every file changed in the PR, read completely at the PR head (5ad422fa9c4307933eed28e5b48d9489bf80a5da) against merge-base 0d5a04471bfd0ef668de909ec780597cba98145a on main, plus the surrounding callers/callees needed to judge each change in place (e.g. apps/rocmd/src/service.rs, watchers/server_recover.rs, supervise.rs, engines/vllm/src/process.rs, crates/rocm-core/src/service_record.rs). Split across 8 independent blind fan-out passes grouped by coupling (the shared client + its envelope types; rocm-core's SIGPIPE guard and AppPaths; rocm's engine_ipc/main.rs; rocm's smaller call sites; rocmd's engine_ipc/supervise; the xtask contract test and docs; the e2e Cucumber coverage; a dedicated history/prior-changes pass over the whole set), each independently verified and scored, then consolidated and adversarially re-checked against the source directly by the driver (including reproducing the contract test's detection logic and reading the exact SIGPIPE-guard call sites). Design review (Step 7) surfaced the one tradeoff above; no fundamental approach problem. Ticket ROCMAI-875 was retrieved and read after the code findings were settled; the PR's description was read at the same point and found accurate against what each pass independently verified, with no scope mismatch in either direction. Did not run: a full cargo test/cargo clippy build in this sandbox (no Rust toolchain available here); correctness rests on manual tracing including, in one case, reproducing the flagged check's exact string-matching logic against real code in the repo, and in another, verified against the module's own test suite's actual capture technique. Not reconciled against any prior review thread or discussion on this PR — this was a blind pass from the code and the ticket alone.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
… guard and only for a parsed answer; see split-line stdio clients The note added for a child that answers and then fails was written with eprintln!, which is neither SIGPIPE-guarded nor non-panicking: with stderr a closed pipe it would end a daemon hosted in rocm (SIGPIPE at its default action) or panic one that ignores the signal, instead of returning the answer. It now goes through the same guarded, error-ignoring write as the stderr forwarding. It also said "the answer was used" as soon as stdout was non-empty, which is false when the output is unreadable and the child failed. The note is now made after the envelope has parsed; an unparseable answer is reported by the existing error, with the status. One test asserts the outcome and the stderr together for both cases, and another runs with stderr a pipe without a reader under SIGPIPE's default action and requires the answer back and the process alive. Each was shown red by removing the behaviour (a plain eprint, and noting without the parse check). The one-client contract scan looked for .stdin( and piped() on the same line, so a Stdio built in one place and handed to .stdin(..) in another (a shape two helpers in the tree already use) went unseen. It now looks for the ingredients anywhere in a file, with a fixture for the split shape that fails under the per-line check; the module comment and docs/testing.md state the coarser trade-off and the limits that remain. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because UTF-8 characters split across reads can trigger discarded Windows-console forwarding errors, silently losing valid progress or diagnostic output.
2 resolved since last review
Review effort: Balanced
…r is not lost on a Windows console The stderr collector publishes each chunk it reads as it arrives and forwarded the same chunk. A chunk can end in the middle of a multi-byte character, and Rust's stdio on a Windows console refuses a write that is not valid UTF-8; the best-effort forwarding discards that error, so valid progress or diagnostic output was silently lost. A small pure type, Utf8Carry, now sits between the collector and the sink: it forwards the complete UTF-8 of what has arrived, carries an incomplete trailing sequence to the next chunk, replaces bytes that can never be text with U+FFFD, and flushes what is still carried at the end of the stream lossily. The kept copy used for errors is still the raw bytes, published at once, so an unterminated diagnostic is still reported when the grace period expires. Tests run on every platform: every two-point split of a string with two-, three- and four-byte characters is forwarded whole and valid at each boundary; invalid bytes and a cut-off tail are replaced and flushed; and the collector, fed one byte at a time, forwards the characters whole and keeps the raw bytes. Forwarding raw chunks turns all three red, and dropping the final flush turns the tail test red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ion string literal The output contract fails a non-ASCII literal in production code, and the escape for U+FFFD in the UTF-8 carry counted as one. The replacement is now its three UTF-8 bytes as a named constant. Behaviour is unchanged and the carry tests pin it. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Daemon subprocess cleanup and process-wide signal changes need maintainer verification because child-process tests are Unix-only and Windows validation was limited to cross-compilation rather than runtime testing.
Review effort: Balanced
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 6b9a6f9de6
Review — no blocking findings
Full review of the whole change (0d5a044...6b9a6f9, 29 files, +3940/-421).
Claims to close ROCMAI-875 (an internal AMD tracker — named but not retrieved here, per this repo's own ticket-ID policy). The description's own account of the ticket's ask ("one shared client, and a test that fails if a second copy re-diverges") matches what the diff delivers: a single stdio_client.rs, both callers migrated onto it, and a static contract test (xtask/src/engine_stdio_client_contract.rs) guarding against a second copy.
Blocking
None
Non-blocking
None
Decisions for the author
- The static one-client contract scan is a source-text heuristic, not a call-graph analysis — tradeoff
xtask/src/engine_stdio_client_contract.rsflags any file that both pipes a child's stdin (.stdin(+piped(), anywhere in the file) and names an engine envelope type. This is cheap and already caught the shape of the exact regression it's meant to prevent (a pasted-backrocmdclient), but a hand-builtStdioor an aliased envelope type would slip past it. The module comment anddocs/testing.mdalready state this limit explicitly, so this looks like a deliberate, disclosed choice rather than an oversight — flagging only so the author confirms that's still the intended scope as the codebase grows. - The wait for a child's pipes after it exits is unbounded when a grandchild holds one open — tradeoff
stdio_client.rs's module comment states this plainly: an engine that leaves a grandchild holding stdout or stderr open makes a request wait for that grandchild, with no timeout. This is explicitly the same exposure both replaced clients already had (rocmread stdout to the end and joined its stderr thread;rocmdusedwait_with_output, which waits on both), so it is not a new or widened blast radius for the long-livedrocmddaemon — but it's worth the author confirming that carrying it forward unbounded (rather than scoping a deadline into this change) is still the right call, since fixing it properly needs a timeout policy the engine contract doesn't have today.
Positive signals
- The SIGPIPE guard's reference counting (
crates/rocm-core/src/sigpipe.rs) correctly handles nested and cross-thread overlapping regions, restoring the pre-guard disposition only once the last region closes, and is verified by a two-thread, barrier-synchronized test (overlapping_guards_restore_the_original_disposition_once) rather than just a single-threaded nested case. - Unusually tight test-to-doc-comment traceability: nearly every specific behavioral claim in
stdio_client.rs's module comment (the stderr-before-write deadlock avoidance, the UTF-8 chunk-boundary carrying, the signal-vs-exit-code wording, the note-only-for-a-parsed-answer rule) has a correspondingly named unit test, and the PR's own test plan lists the mutation that was shown red for each one. - The contract test's allowlist is tested bidirectionally — a planted old-style client is caught, and the allowlist's own
stdio_client.rsentry is shown to go stale if it stops matching — avoiding the common failure mode of an exemption list nobody prunes. - The two new
engine_ipc.featurescenarios and the extraservice-stop-02assertion land exactly on the PR's stated user-visiblerocmdbehavior changes (answer-wins over a failing exit status; parse failure reported with exit status and stderr), satisfying this repo's own requirement that user-observable CLI behavior be covered by a Gherkin scenario, not only a unit test.
Deployment notes
rocmd's wire-adjacent error strings for an engine request change shape: a failing exit status with no answer and an unparseable answer are now worded from the exit code or signal (engine stdio process exited with code N / was terminated by signal N) rather than from the previous ad hoc wording, and a failing exit after a successful answer no longer discards that answer (it's decoded and reported, with the failing status only noted on the daemon's own stderr log). Any external tooling that parses rocmd's sandbox-tool/MCP JSON error text for these cases should account for the new wording.
What this covered
Degraded review — fan-out was unavailable. This environment's shared subagent pool was saturated by unrelated workloads at review time: five consecutive attempts to spawn independent per-file-group workers (over roughly five minutes of retries) each failed with "concurrent subagent limit reached," and the pool never freed a slot. Per the review method's own fallback, this was run single-context instead of as independent fanned-out passes, which is expected to find a narrower slice of the real defect surface than a fanned-out review would (measured elsewhere at roughly 12-30% of a fan-out's yield). Re-running with fan-out available is recommended if a second opinion is wanted.
Within that constraint, every file in the diff was read in full at the current head (not excerpted): the new crates/rocm-engine-protocol/src/stdio_client.rs (1970 lines, including its tests), fake_engine.rs, the rocm-engine-protocol and rocm-core lib.rs diffs, rocm-core/src/paths.rs and the new sigpipe.rs (258 lines), both callers' engine_ipc.rs in full (apps/rocm, apps/rocmd), every smaller call-site diff (engine_runtime_select.rs, engines_cmd.rs, main.rs, service_control.rs, service_launch/*.rs, chat/mcp_bridge.rs, apps/rocmd/common.rs, apps/rocmd/supervise.rs), both Cargo.tomls and Cargo.lock, the new xtask/src/engine_stdio_client_contract.rs (301 lines) and its xtask/src/main.rs wiring, the docs/architecture.md and docs/testing.md diffs, the new engine_ipc.feature plus the service_stop.feature/service_stop_steps.rs/feature_naming.rs changes. Checked against the repo's root AGENTS.md in full (module-size limit, architecture-doc-entry requirement, subprocess/FFI checklist, markdown-citation rules, the bare-ticket-ID allowance). Checked CI: all 41 check runs on this commit are green or correctly skipped (no failing or pending check). Checked for merge-conflict markers in the diff (none) and ran a leak scan for internal-reference terms across the diff (clean; the only matches were ordinary identifier names like restart_internal_managed_service_with).
Did not run: an independent fan-out (see above, environment-limited); "history" beyond git log on the touched files' own commits (most of the surface is new code, so there is little prior history to a different file to check, and the PR's own commit history shows three post-review fix-up commits already folded into the head read here); "prior changes" (no access to this PR's own review-thread content was used or needed, by design — the method's blindness rule excludes it, and no separate prior-PR history on these exact files was found to check). Not reconciled against any discussion on this PR.
Signed-off-by: Jussi Elo <jussi.elo@amd.com> # Conflicts: # crates/rocm-core/src/paths.rs
…erge in paths.rs Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The shared transport changes Windows subprocess handling for both CLI and daemon, but verification is limited to cross-compilation and Unix-only child-process tests, leaving Windows spawning, pipe draining, and failure cleanup unvalidated at runtime.
Review effort: Balanced



Closes ROCMAI-875.
Summary
rocmandrocmdeach carried their own client for the JSON-over-stdio request to an engine process, and the two had drifted: they built the child's command line differently, set different environment, treated a failing exit status differently, worded their errors differently, and onlyrocmguarded the write of the request against SIGPIPE. This change replaces both with one client,stdio_client.rsinrocm-engine-protocol. Each caller still chooses which program to run (rocmthe resolved engine binary withstdio, or its built-in handler with no child;rocmdtherocmexecutable with__engine-stdio <engine>) and passes a small set of options: the app paths, the engine environments root, an endpoint key file, a progress request, and whether to forward the child's stderr as it arrives. The wire format is unchanged. A few helpers both binaries had copies of moved to where both can reach them:with_sigpipe_ignored(fromrocm'smain.rstorocm-core), the threeROCM_CLI_*_DIRvariables asAppPaths::env_vars, andoptional_arg(torocm-engine-protocol).Where the two clients differed, the shared one takes the behaviour that reports more or fails more safely, so these are intended user-visible differences for
rocmd(the daemon and its sandbox/MCP tools, the supervisor's readiness wait and the server-recover healthcheck):rocmdused to discard the answer for the status.failed to parse engine response envelope (engine stdio process exited with code N); stderr: ...; a child ended by a signal readswas terminated by signal N), so no failing case drops the status the oldrocmdreported. The status is worded from the exit code or signal rather than fromExitStatus's own text, which already readsexit status: Nand would double up. Decode errors userocm's wording.rocmdhad no guard, so the daemon hosted inrocm(which runs with SIGPIPE at its default action) was killed by a write to an engine that had already exited instead of reporting why. The guard is also safe to overlap now: it remembers the disposition in force before the first guarded region and restores it when the last one ends, so two overlapping regions on different threads cannot restore "ignored" over a default action.print!to a closed pipe on another thread would fail instead of ending the process quietly. The option itself is tested end to end: a test runs the public entry point in a re-executed test binary and reads that process's stderr (the child's line is there withforward_stderrand absent without it), and another passes aVec-backed sink to the private entry point.failed to write engine request: <cause>; stderr: <text>), which is usually why it was not listening. That stderr is read for at most a short grace, so a grandchild holding the pipe cannot hold the error path. The child's stderr is read on a thread of its own from the moment the child starts, published chunk by chunk as it arrives (what is forwarded to the terminal is complete UTF-8, carrying an incomplete trailing character to the next chunk, because a Windows console rejects a write that is not valid UTF-8) (so an unterminated last diagnostic is not lost when a failed write's grace period expires), and once a request has failed the capture is marked abandoned: what it held is released and later output is drained and discarded in fixed-size chunks, so a descendant that keeps writing cannot grow the daemon, so an engine that fills it before draining a request larger than a pipe buffer cannot deadlock the write (both replaced clients started reading it only after the write). A request whose answer cannot be read kills and reaps the child in the same way; its stderr collector is not joined there because a grandchild holding the pipe could block the join. A failed read of the child's stdout reports the stderr already collected like every other failure, and a child whose stdin or stdout pipe is missing is killed and reaped. A child that answers and then exits with a failing status is still read as having answered, but the daemon (not the CLI) asks the client to note the status on its stderr, which is its log, so it is no longer dropped without a trace (the note is written through the same SIGPIPE-guarded, error-ignoring write as the stderr forwarding, and only for an answer that parsed). A failed write does not read the child's stdout, so an answer printed by an engine that exits without draining a request larger than a pipe buffer is not decoded; every engine here reads stdin to the end before answering, so that needs an engine that breaks the protocol, and the module comment says so.rocmdnow sets theROCM_CLI_*_DIRvariables instead of relying on inheriting them), and the spawn error names the program in both.One limit is shared with the clients this replaces and is stated in the module comment,
docs/testing.mdand the architecture entry rather than changed here: after the child exits, the answer is read to the end of its stdout and the stderr collector is joined, so an engine that leaves a grandchild holding either pipe makes the request wait for it (rocmread stdout to the end and joined its stderr thread;rocmdusedwait_with_output). Bounding it exactly means reading both pipes on their own threads and choosing what to report when a deadline passes, which is a timeout policy the engine contract does not have today.The standalone
rocmdbinary cannot serve__engine-stdio(that verb isrocm's alone), which the managed-service recovery change already on main (DaemonHost, an explicit flag set once by the embedded entry point) solves for every daemon spawn. This PR adds no standalone-daemon logic of its own:rocmd's engine requests run therocmbinary thatDaemonHost::rocm_binary()finds (the one besiderocmd, or the embedded daemon's own executable) through the shared client. What this PR adds for that path is scenario coverage, below. One related limit is not addressed here, because it is inDaemonHostand not in this change: after an in-place upgrade Linux reports an embedded daemon's executable as<path> (deleted), which cannot be run.Two small changes ride along (and the three-line loop that set the app-path variables on a command is one
AppPaths::apply_env, replacing rocm'sapply_app_path_env):AppPaths::env_varsandAppPaths::discovershare one named constant perROCM_CLI_*_DIRvariable (CONFIG_DIR_ENV,DATA_DIR_ENV,CACHE_DIR_ENV), and the doc comment of the SIGPIPE guard says its restore goes throughsignal(), which reports only the handler (this process only ever setsSIG_DFLandSIG_IGNfor SIGPIPE).The ticket asks for one shared client and for a test that fails if a second copy re-diverges. Both halves are covered: the behavioural tests of the client and of each caller pin what they choose (and fail against a restored old copy of rocmd's client, five of them), and a static contract test,
xtask/src/engine_stdio_client_contract.rs, fails when any file other than the shared client both pipes a child's stdin and names an engine envelope (looked for anywhere in the file, not on one line), which is what a client is made of. It is a source-text scan, so a copy that makes its pipe by hand or names the types differently is outside what it sees;docs/testing.mdsays so.The test stand-in for an engine child (
FakeEngine, behind atest-supportfeature ofrocm-engine-protocol) is shared by both callers' tests. It now quotes the canned response and its scratch directory for the shell, so a response such ascan't stopor aTMPDIRwith an apostrophe reaches the client intact.Split from #929 at the reviewer's request; the other parts are #961 (log-tail routine), #960 (stop routine) and the respawn guard, which follows once this and #960 have merged.
Test plan
Ran, on Linux, with a private target dir, on the head before the last merge of main unless noted:
cargo test -q --locked -p rocm-engine-protocol(68 passed),-p rocmd(249 passed, 1 ignored),-p rocm-core(1268 passed, 6 ignored),-p xtask(774 passed, includingprepush98 and the new contract test),-p rocm --bin rocm engine_ipc(12 passed), and the unit tests of the three engine crates that depend onrocm-engine-protocol(rocm-engine-lemonade,-vllm,-aim; all passed, run before the last merge of main). The wholecargo test -p rocm --bin rocm --no-fail-fast(1694 passed, 9 ignored, 0 failed) was run before the review fixes and the later merges of main and was not re-run after them (CI carries it).cargo fmt --all -- --check;cargo clippy --locked -p rocm-engine-protocol -p rocm-core -p rocmd -p rocm --all-targets -- -D warnings;cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings; the same clippy forx86_64-pc-windows-msvcover the whole workspace with--all-targets(check only, nothing was linked or run on Windows).cargo xtask manifest --check,check-crate-edges,check-architecture-doc,lychee.rocm-engine-protocol(97.78% against its floor of 98.37%); the floor was not re-blessed. Tests were added for the new branches (the neutral wording of a status with no code or signal, the collector's deadline, test sinks and failure-only assertion arguments).cargo llvm-cov --no-cfg-coverageover the protocol,rocmdandrocmtests measures the crate at 98.85% lines (1637 of 1656), above the floor less its 0.2 point tolerance; the CI run covers the whole workspace.service-stop(5 scenarios) andengine-ipc(2 scenarios) scenarios through the real harness with a debugrocm/rocmdbuilt withrocm/e2e-test-hooks, and thefeature_namingtest.waitafter thekillin the read-failure path turnsa_failed_read_of_the_answer_kills_and_reaps_the_childred; replacing the guard's shared counter with a per-call save and restore turnsoverlapping_guards_restore_the_original_disposition_oncered (3 of 3 runs); dropping the exit status from the parse failure turns its unit test andengine-ipc-02red; removing the stderr read, the kill, or the bound from the failed-write path turns one of its three tests red each; wrapping the whole stderr collector in the SIGPIPE guard turns the guard-scope test red; replacing theforward_stderrforwarding withNoneturns the real-stderr test red; removing the signal branch of the exit note turns the signal test red; removing the lock from the environment test's comparison turns its race test red on the first round; line-buffering the published stderr turns the grace-period test red; removing the drop handling turns the abandoned-collector test red; starting the stderr collector after the write instead of before turns the deadlock test red (it hangs and fails after 20 seconds); putting the old rocmd client body back intoapps/rocmd/src/engine_ipc.rsturns the contract scan and five rocmd pinning tests red; changingservice-stop-02's expected engine answer turns it red.Not run, besides the above: a test of the Windows spawn path on its own gate (the child-process tests are Unix-only; the pure parts run everywhere), the full workspace test suite,
cargo xtask e2e,scripts/smoke_local.pyandcargo xtask coverage(CI carries them), any test on Windows (the SIGPIPE re-exec tests, the fake-engine child tests and the process-entry tests are Unix-only; the pure parts run everywhere), and any GPU-backed lane.Scenarios:
service-stop-02intests/e2e-cucumber/features/service_stop.feature(Linux, mock lane) gains a step asserting the standalonerocmdgot an answer to its engineStoprequest, and asserts the whole answer (stoppedis false there because the planted record names no engine process, which is the real engine's answer, not a gap). Two more scenarios,engine-ipc-01andengine-ipc-02intests/e2e-cucumber/features/engine_ipc.feature(Linux, mock lane), drive the standalone daemon'sstop_servertool against an engine that answers and then exits non-zero, and one that prints something unreadable, writes to stderr and fails (the report names the exit status and carries the stderr), and assert the tool's report together with the server's process, record and endpoint key file. The rest of the behaviour above is internal to the shared client and covered by unit tests that drive each caller's real entry point against the stand-in engine child.The
rocm-side wording of a failing engine (exited with code N) has no scenario of its own: no existing scenario runs a failing engine throughrocm, and the text comes from the shared client, which is pinned by unit tests (a_nonzero_exit_with_no_answer_reports_the_status_and_stderr,an_unparseable_answer_from_a_failing_child_names_the_exit_status,a_child_ended_by_a_signal_is_reported_by_the_signal) and asserted in thestop_serverreport byengine-ipc-02, which runs the same client.