Repository navigation
feat(engine): let a caller withhold an inherited endpoint key file - #1034
Conversation
An engine plugin is a third-party binary. `rocm` spawns one with `endpoint_key_file: None`, which until now meant only "set no key" — so a `ROCM_SERVE_API_KEY_FILE` this process had inherited passed straight through to it, handing a third party the path to a mode-0600 file holding a live endpoint credential. Clearing it unconditionally on the `None` branch would have been wrong: the shared builder also serves `rocmd`, whose `None` means "this loopback service needs no key", not "this child may not see one" — those are its own children, and its `the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key` expects a keyless child to inherit the ambient value. So the withholding is an explicit option the caller asks for. `rocm` sets it for plugin spawns; `rocmd` does not, and nothing about its behaviour changes — its test is untouched and still green. The test asserts CLEARED rather than absent, because `Command` records `env_remove` as an explicit `(key, None)` and a plain "is it present" check would pass with the clearing deleted. Both directions are pinned: a caller that asks gets the removal, one that does not gets no entry at all. Split out of #678 on review: it is hardening that stands on its own and is not part of ROCMAI-550. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
juhovainio
left a comment
There was a problem hiding this comment.
Nothing here is blocking, but I'd like the first point below addressed before merge.
Requested change
- The rocm call site (
withhold_inherited_key_file: trueinapps/rocm/src/engine_ipc.rs) isn't covered by any test. The new test buildsEngineStdioOptionsby hand in the protocol crate, so flipping rocm's value tofalsekeeps everything green and silently re-opens the leak. Please add a test over rocm's own options/command construction that asserts the key-file variable is explicitly cleared, and confirm it goes red when the flag is flipped.
Smaller points
- The PR text names no scenario and doesn't say why none applies for the child-env behavior. One sentence (internal hardening, covered by the unit test plus the call-site test) or an e2e scenario with a fake plugin dumping its env would do.
- Please confirm no other rocm plugin/engine spawn path bypasses
engine_stdio_command(a directCommand::newon a plugin binary would still inheritROCM_SERVE_API_KEY_FILE). I couldn't verify this from the diff alone. - The opt-in bool fails open: a new caller that forgets it leaks by default. A 3-state enum (
Pass(path)/Inherit/Withhold) would make the choice explicit; fine to keep as is if deliberate.
The else if ordering (a real key file wins over withholding) and the explicit (key, None) assertion are good.
| endpoint_key_file: None, | ||
| // An external plugin is a third-party binary: never hand it a path to | ||
| // a live endpoint credential, including one we inherited ourselves. | ||
| withhold_inherited_key_file: true, |
There was a problem hiding this comment.
Nothing tests that rocm actually sets this flag: the new test hand-builds EngineStdioOptions in the protocol crate, so flipping this to false keeps every test green and re-opens the leak. Please add a test over rocm's own stdio options / plugin command asserting the key var is explicitly cleared, or make withholding the default.
| /// because "no key file for this request" and "this child may not have one" | ||
| /// are different statements: `rocmd` passes `None` for a loopback service | ||
| /// that simply needs no key, and its children are its own. | ||
| pub withhold_inherited_key_file: bool, |
There was a problem hiding this comment.
Fail-open default: a new caller that forgets this bool leaks by default. Consider a 3-state enum (Pass(path) / Inherit / Withhold) so the two meanings of None can't be conflated; fine to keep as is if deliberate.
| // a loopback service that needs no key, and clearing there would | ||
| // change what its own children see. See | ||
| // `an_external_plugin_child_is_not_handed_the_endpoint_key_file`. | ||
| command.env_remove(ENDPOINT_API_KEY_FILE_ENV); |
There was a problem hiding this comment.
Please confirm no other rocm plugin spawn path bypasses engine_stdio_command; a direct Command::new on a plugin binary would still inherit ROCM_SERVE_API_KEY_FILE.
…hat it can The protocol crate's test proves the flag causes the clearing, but it hand-builds the options — so flipping `rocm`'s own `withhold_inherited_key_file` to `false` re-opened the leak with every test still green. That is half a chain, and the untested half is the one a future edit would touch. Assert it at the call site. Verified by the mutation the test exists for: flipping the value turns it red. Over `stdio_options` rather than a built `Command`, because `engine_stdio_command` is private to the protocol crate and widening it for a test would trade a real API change for a marginally stronger assertion. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
|
Requested change done. Pushed as The rocm call site wasn't coveredYou're right, and it's the more important half: the protocol test hand-builds the options, so flipping
Asserted on No other rocm spawn path bypasses itConfirmed. The only other caller of The fail-open boolDeliberate, but you've identified the real cost, so let me give the reasoning rather than just assert it. A 3-state enum ( If you'd prefer the enum I'll do it here; otherwise it's a clean follow-up, and the call-site test above is what keeps the current default honest in the meantime. No scenario, and whyAdded to the PR text: this is internal child-environment hardening with no user-visible surface, covered by the protocol-crate test (the clearing happens) plus the call-site test (rocm asks for it). An e2e scenario would need a fake plugin binary that dumps its environment — real coverage, but a new fixture for a behaviour two unit tests already pin at both ends. Verification (Linux MI300X)
|
Call site now covered in c12a8ab: every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file, verified red when rocm's flag is flipped to false. Confirmed no other rocm plugin spawn path bypasses engine_stdio_command. The fail-open bool is deliberate with reasoning given, and the missing-scenario rationale is in the PR text.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head c12a8ab9
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (apps/rocm/src/engine_ipc.rs, apps/rocmd/src/engine_ipc.rs, crates/rocm-engine-protocol/src/stdio_client.rs).
Split out of #678 per that PR's own description (confirmed by reading #678: it names this PR as where the engine-protocol key-file withholding independently lives); explicitly not part of ROCMAI-550, and no other ticket is named.
Blocking
- CI is red on two of the jobs that would actually run this PR's tests, at this exact head —
Test (affected crates)(ubuntu-24.04 and ubuntu-26.04) andE2E tests(both variants).
Not a code defect: both failed inside the sharedInstall native build depsstep withTemporary failure resolving 'azure.archive.ubuntu.com'while runningapt-get install libcap-dev— a transient DNS/network blip on the runner, before any crate compiled or any test executed. The siblingbuild-and-testjob's identical step succeeded a few minutes earlier on this same commit, which is consistent with a flake rather than anything this diff touches. Practically, though, this means there is currently no CI-green run proving the new tests (every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file,an_external_plugin_child_is_not_handed_the_endpoint_key_file) actually pass — only the author's local MI300X run (named in the PR body) attests to that today.
Confidence 100 · mechanical · Fix: re-runTest (affected crates)andE2E testson this head (no code change expected) and confirm green before merge.
Non-blocking
docs/architecture.md's entry forstdio_client.rsenumeratesEngineStdioOptions's fields and is now missing the one this PR adds —docs/architecture.md:609. The entry parenthetically lists "app paths, engine environments root, endpoint key file, progress request" as the fieldsEngineStdioOptionsthreads;withhold_inherited_key_file— the field this PR's two call sites deliberately differ on — isn't added.cargo xtask check-architecture-doconly requires an entry to exist per source file, not that its prose stays accurate (the repo's ownAGENTS.mdsays as much: "the check reads no prose, so keeping an entry's description true is still on you"), so this won't fail CI, but the enumeration is now incomplete by its own standard.
Confidence 85 · mechanical · Fix: add the new field to that parenthetical.
Decisions for the author
withhold_inherited_key_file: boolfails open under#[derive(Default)]— non-blocking-improvement. Today both call sites (rocm:true,rocmd:false) are explicit, so there's no live defect. But the field defaults tofalse(inherit), so a future third-party-plugin spawn path built withEngineStdioOptions { ..Default::default() }would silently re-open exactly the leak this PR closes, with nothing short of a reviewer catching the omission. A 3-state enum (e.g.Pass(path)/InheritAmbient/Withhold) would turn a forgotten choice into a compile error instead of a silent default. Worth weighing against the ceremony cost for what are, today, exactly two call sites.
Positive signals
an_external_plugin_child_is_not_handed_the_endpoint_key_fileasserts the distinction that actually matters —Command::get_envs()reports an explicit(key, None)tombstone fromenv_remove, not merely "absent" — and pins both directions (withhold clears; no-withhold never touches the variable) in one test, so flipping either caller's option value would be caught.- Splitting the proof across two crates (
rocm'sstdio_options()asserts it asks to withhold; the protocol crate's test proves withholding actually clears the var) is a deliberate, well-justified workaround forengine_stdio_commandbeing private to the protocol crate — each test's own doc comment explains why it's scoped that way, rather than leaving a reader to wonder why the assertion isn't on a builtCommand.
Deployment notes
None.
What this covered
Read apps/rocm/src/engine_ipc.rs, apps/rocmd/src/engine_ipc.rs and crates/rocm-engine-protocol/src/stdio_client.rs in full (base f04b785b…head c12a8ab9), plus impact-analysis surface: confirmed by repo-wide grep that no other site in the workspace constructs EngineStdioOptions; checked every other path that could spawn a third-party engine plugin binary (apps/rocm/src/service_launch/spawn.rs, apps/rocmd/src/supervise.rs — both spawn rocm itself via __engine-serve-http, not a plugin binary, so outside this PR's stated threat model; apps/rocm/src/examine_report_text.rs and model_catalog_text.rs only resolve and display the binary path, never spawn it); read docs/architecture.md's entries for the three touched files and for stdio_client.rs; read the repo's AGENTS.md. Single-context pass, not fanned out to subagents — a judgment call under the method's sizing step given 3 files / 113 added lines all tightly coupled around one struct, not a degraded run.
Did not run: the "prior changes" pass (searching historical review comments on earlier PRs touching these three files) — not attempted, judged low-value at this size. No Rust toolchain is available in this environment, so cargo test/cargo clippy were not independently executed here; CI's own results were read instead (see Blocking) rather than assumed. The dedupe check mandated before this review (confirming no prior review from this account at this exact head) necessarily surfaced an existing dismissed human review from juhovainio on an older commit of this PR; it was read only for that check and to confirm the current head already contains the rocm-call-site test it had asked for, and did not otherwise shape this review's findings, which were reached independently from the code and the repo's own AGENTS.md. Not otherwise reconciled against any discussion on this PR.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
|
CI note: the four red checks are runner infrastructure, not this change.
DNS resolution failing on the runner while installing system dependencies. Nothing in this PR can reach that — the diff is three Rust files and touches no CI config, and the same checks were green on the previous head. They should clear on the next run; I'll confirm rather than assume. |
juhovainio
left a comment
There was a problem hiding this comment.
I went through this at the current head and didn't find a defect in the change itself: the flag is opt-in, rocmd behavior is unchanged, and the protocol-side test covers both the cleared and the absent case. I'm requesting changes only for a few things to confirm or state before merge:
- Please confirm every rocm external-plugin spawn goes through
stdio_options; the new rocm test only pins that function's output. - Please confirm no other inherited variable carries the endpoint key itself, since only the key-file path variable is cleared.
- The PR body should say why there is no cucumber scenario, and whether the rocm-side test was shown to fail with the flag flipped to false (only the protocol test's mutation check is claimed today).
- FYI:
EngineStdioOptionsispubwith a new field.Defaultgives false so the default is safe, but out-of-tree struct literals would stop compiling.
On CI: E2E tests and Test (affected crates) fail on both ubuntu runners in "Install native build deps" (apt cannot resolve the mirror, exit 100). That looks like infrastructure, not this change; they need a rerun.
| /// on `stdio_options` rather than on a built `Command` because | ||
| /// `engine_stdio_command` is private to the protocol crate. | ||
| #[test] | ||
| fn every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file() { |
There was a problem hiding this comment.
Please confirm (and note in the PR body) that you broke this by flipping the flag to false and saw this test go red, and that every plugin spawn path goes through stdio_options.
| // a loopback service that needs no key, and clearing there would | ||
| // change what its own children see. See | ||
| // `an_external_plugin_child_is_not_handed_the_endpoint_key_file`. | ||
| command.env_remove(ENDPOINT_API_KEY_FILE_ENV); |
There was a problem hiding this comment.
Only the key-file path variable is cleared. If any other inherited variable carries the endpoint key itself, a third-party plugin would still see it; worth confirming none exists.
|
All four confirmed, and the PR body now carries each answer rather than leaving it in a comment thread. Does every rocm external-plugin spawn go through
|
| test | mutation | result |
|---|---|---|
an_external_plugin_child_is_not_handed_the_endpoint_key_file (protocol) |
delete the env_remove |
FAILED |
every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file (rocm) |
flip rocm's flag to false |
FAILED |
No scenario, and why
In the body now: internal child-environment hardening with no user-visible surface, bracketed by the two unit tests above. An e2e scenario would need a fake plugin that dumps its environment — real coverage, but a new fixture for a behaviour both ends already pin.
The pub struct gaining a field
Noted in the body. Default gives false, so the default behaviour is the safe, unchanged one; an out-of-tree struct literal would need the new field to keep compiling.
CI
Agreed, and thanks for checking it independently — apt cannot resolve the mirrors (Temporary failure resolving 'azure.archive.ubuntu.com', exit 100) before anything is built. It is fleet-wide rather than PR-specific: PR #1040 is failing the same way right now. I'll confirm on the next run rather than assume.
All four confirmed and written into the PR body: engine_stdio_command is the only plugin spawn path (the other two resolve callers only print the path); ROCM_SERVE_API_KEY_FILE is the only variable carrying the endpoint key, and it carries a path — with the honest caveat that unrelated provider keys still inherit; the rocm-side mutation was run and is now stated; and the pub-struct field note is included.
Three leftovers from the split, all of which made this diff claim work it no longer ships. `stdio_client.rs` still carried an older, UNCONDITIONAL version of the endpoint key-file clearing — the one that broke `rocmd`'s contract before #1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now, and `rocmd`'s own `the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key` is green again here. `docs/testing.md` still described four support levels and pointed at examine-50/51/52. This change ships three levels and examine-49; the fourth level and those scenarios are #1036's, and the text now says so. The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead here — nothing can produce that level without the probe — so it goes back to #1036 with the level it tests. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tions The `stdio_client.rs` entry enumerates the fields `EngineStdioOptions` threads to the child, and the field the two call sites deliberately differ on was missing from that list. `check-architecture-doc` reads no prose, so nothing would have caught it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Confirmations and one doc fix —
|
juhovainio
left a comment
There was a problem hiding this comment.
Looks good. The clearing is scoped correctly: rocm's external plugin spawns ask for it, rocmd keeps its current behaviour, and the protocol test checks the explicit env_remove entry rather than mere absence. Earlier points (call-site test, scenario justification, spawn paths) are addressed.
Two small non-blocking things, inline below. Also optional: the new protocol test repeats full EngineStdioOptions literals, ..Default::default() would cut churn when fields are added. I still lean slightly toward an enum over the bool, since a forgotten flag fails open, but the documented default is a reasonable call.
CI: the red jobs all failed at "Install native build deps" on apt DNS errors, so tests never ran there; they need a rerun.
| /// is the other half of the chain, and without it flipping the value here | ||
| /// to `false` would re-open the leak with every test still green. Asserted | ||
| /// on `stdio_options` rather than on a built `Command` because | ||
| /// `engine_stdio_command` is private to the protocol crate. |
There was a problem hiding this comment.
Is engine_stdio_command really private to the protocol crate? The PR text says rocm spawns through it. If it is reachable here, asserting that the built Command's get_envs() has (ROCM_SERVE_API_KEY_FILE, None) would prove the end state rather than only the flag. Otherwise this comment is fine as is.
| } | ||
| if let Some(key_file) = options.endpoint_key_file { | ||
| command.env(ENDPOINT_API_KEY_FILE_ENV, key_file); | ||
| } else if options.withhold_inherited_key_file { |
There was a problem hiding this comment.
This precedence is not covered: endpoint_key_file: Some(p) with withhold_inherited_key_file: true must set the path, not remove it. Swapping the branch order stays green today. A one-line case in the new test would pin it.
Approved and green — enqueuedThanks for the approval. CI is now fully green on For the record, since the red checks were discussed above and nothing about the code changed: the seven failing Linux jobs were the There is also a |
Dequeued without a failure — re-enqueuedThis was dropped from the merge queue at 17:58 after ~20 minutes, and I want to record why, since it was not a defect in this PR. The merge-queue CI run ( Nothing changed on this PR — still Re-enqueued (position 4). If it drops a second time with the same cancelled-Sphinx signature, that is a queue/workflow problem worth fixing rather than something to keep retrying, and I will stop and say so instead of re-enqueuing again. |
Three leftovers from the split, all of which made this diff claim work it no longer ships. `stdio_client.rs` still carried an older, UNCONDITIONAL version of the endpoint key-file clearing — the one that broke `rocmd`'s contract before #1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now, and `rocmd`'s own `the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key` is green again here. `docs/testing.md` still described four support levels and pointed at examine-50/51/52. This change ships three levels and examine-49; the fourth level and those scenarios are #1036's, and the text now says so. The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead here — nothing can produce that level without the probe — so it goes back to #1036 with the level it tests. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Split out of #678 on review — this is hardening that stands on its own and is not part of ROCMAI-550.
Symptom
An engine plugin is a third-party binary.
rocmspawns one withendpoint_key_file: None, which until now meant only "set no key". So aROCM_SERVE_API_KEY_FILEthatrocmhad itself inherited passed straight through to that child — handing a third party the path to a mode-0600 file holding a live endpoint credential.Why not just clear it on the
NonebranchBecause the stdio builder is shared with
rocmd, and the two callers mean different things byNone:endpoint_key_file: Nonemeansrocm(plugin spawn)rocmd(loopback service)rocmd's children are its own, and itsthe_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_keyexpects a keyless child to inherit the ambient value. An unconditional clear would have broken that contract — and would have done so silently, since the test only passes today when the variable happens to be unset.Change
EngineStdioOptionsgainswithhold_inherited_key_file.rocmsets it for plugin spawns;rocmddoes not, and nothing about its behaviour changes.Test
an_external_plugin_child_is_not_handed_the_endpoint_key_fileasserts cleared, not merely absent:Commandrecordsenv_removeas an explicit(key, None)entry, so a plain "is it present" check would pass with the clearing deleted. Both directions are pinned — a caller that asks gets the removal, one that does not gets no entry at all.Verified by deleting the clearing: the test fails.
Verification
Linux MI300X:
cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo test -p rocm-engine-protocol --lib, andcargo test -p rocmd --lib the_endpoint_key_file(rocmd's own test, unmodified, still green).cargo xtask check-architecture-doc/check-module-sizepass.Not verified: nothing
cfg(windows)-specific here, but the Linux box cannot build the MSVC target in any case.Scope of the clearing
ROCM_SERVE_API_KEY_FILEis the only environment variable that carries theendpoint key, and it carries a path —
resolve_endpoint_api_key()reads thekey by opening the file that variable names. There is no variable holding the
key value itself, so clearing this one closes the whole env-borne route for the
endpoint credential.
Worth stating plainly so "confirmed" is not read too broadly: unrelated
provider secrets (
OPENAI_API_KEY,ANTHROPIC_API_KEY,ROCMDASH_CHAT_API_KEY)do still pass through to children by ordinary inheritance. They are a separate
concern from the endpoint key and this change does not touch them.
Every plugin spawn goes through this path
engine_stdio_commandis the only placerocmspawns an engine plugin binary,and
stdio_optionsis its only options source. The other two callers ofresolve_engine_binary_path*—model_catalog_text.rsandexamine_report_text.rs— resolve the path only to print it(
external path=…,adapter_available path=…); neither spawns.Why no cucumber scenario
This is internal child-environment hardening with no user-visible surface. It is
pinned at both ends by unit tests instead: the protocol test proves the flag
causes the clearing, and the rocm test proves
rocmasks for it. An e2escenario would need a fake plugin binary that dumps its environment — real
coverage, but a new fixture for a behaviour two tests already bracket.
Mutation evidence
Both halves of the chain were checked, not just the protocol side:
an_external_plugin_child_is_not_handed_the_endpoint_key_file(protocol)env_removeevery_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file(rocm)falseNote for out-of-tree callers
EngineStdioOptionsispuband gains a field.Defaultyieldsfalse, so thedefault behaviour is the safe, unchanged one — but an out-of-tree struct literal
would stop compiling until it adds the field.