Skip to content

feat(engine): let a caller withhold an inherited endpoint key file - #1034

Merged
r0x0r merged 3 commits into
mainfrom
split/engine-key-file-withholding
Nov 11, 2026
Merged

r0x0r merged 3 commits into
mainfrom
split/engine-key-file-withholding

Conversation

@r0x0r

@r0x0r r0x0r commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Split out of #678 on review — this is hardening that stands on its own and is not part of ROCMAI-550.

Symptom

An engine plugin is a third-party binary. rocm spawns one with endpoint_key_file: None, which until now meant only "set no key". So a ROCM_SERVE_API_KEY_FILE that rocm had itself inherited passed straight through to that child — handing a third party the path to a mode-0600 file holding a live endpoint credential.

Why not just clear it on the None branch

Because the stdio builder is shared with rocmd, and the two callers mean different things by None:

caller endpoint_key_file: None means
rocm (plugin spawn) this child may not see a key
rocmd (loopback service) this service needs no key

rocmd's children are its own, and its the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key expects a keyless child to inherit the ambient value. An unconditional clear would have broken that contract — and would have done so silently, since the test only passes today when the variable happens to be unset.

Change

EngineStdioOptions gains withhold_inherited_key_file. rocm sets it for plugin spawns; rocmd does not, and nothing about its behaviour changes.

Test

an_external_plugin_child_is_not_handed_the_endpoint_key_file asserts cleared, not merely absent: Command records env_remove as an explicit (key, None) entry, so a plain "is it present" check would pass with the clearing deleted. Both directions are pinned — a caller that asks gets the removal, one that does not gets no entry at all.

Verified by deleting the clearing: the test fails.

Verification

Linux MI300X: cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo test -p rocm-engine-protocol --lib, and cargo test -p rocmd --lib the_endpoint_key_file (rocmd's own test, unmodified, still green). cargo xtask check-architecture-doc / check-module-size pass.

Not verified: nothing cfg(windows)-specific here, but the Linux box cannot build the MSVC target in any case.

Scope of the clearing

ROCM_SERVE_API_KEY_FILE is the only environment variable that carries the
endpoint key, and it carries a path — resolve_endpoint_api_key() reads the
key by opening the file that variable names. There is no variable holding the
key value itself, so clearing this one closes the whole env-borne route for the
endpoint credential.

Worth stating plainly so "confirmed" is not read too broadly: unrelated
provider secrets (OPENAI_API_KEY, ANTHROPIC_API_KEY, ROCMDASH_CHAT_API_KEY)
do still pass through to children by ordinary inheritance. They are a separate
concern from the endpoint key and this change does not touch them.

Every plugin spawn goes through this path

engine_stdio_command is the only place rocm spawns an engine plugin binary,
and stdio_options is its only options source. The other two callers of
resolve_engine_binary_path* — model_catalog_text.rs and
examine_report_text.rs — resolve the path only to print it
(external path=…, adapter_available path=…); neither spawns.

Why no cucumber scenario

This is internal child-environment hardening with no user-visible surface. It is
pinned at both ends by unit tests instead: the protocol test proves the flag
causes the clearing, and the rocm test proves rocm asks for it. An e2e
scenario would need a fake plugin binary that dumps its environment — real
coverage, but a new fixture for a behaviour two tests already bracket.

Mutation evidence

Both halves of the chain were checked, not just the protocol side:

test mutation result
an_external_plugin_child_is_not_handed_the_endpoint_key_file (protocol) delete the env_remove FAILED
every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file (rocm) flip rocm's flag to false FAILED

Note for out-of-tree callers

EngineStdioOptions is pub and gains a field. Default yields false, so the
default behaviour is the safe, unchanged one — but an out-of-tree struct literal
would stop compiling until it adds the field.

An engine plugin is a third-party binary. `rocm` spawns one with
`endpoint_key_file: None`, which until now meant only "set no key" — so
a `ROCM_SERVE_API_KEY_FILE` this process had inherited passed straight
through to it, handing a third party the path to a mode-0600 file holding
a live endpoint credential.

Clearing it unconditionally on the `None` branch would have been wrong:
the shared builder also serves `rocmd`, whose `None` means "this loopback
service needs no key", not "this child may not see one" — those are its
own children, and its
`the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key`
expects a keyless child to inherit the ambient value.

So the withholding is an explicit option the caller asks for.
`rocm` sets it for plugin spawns; `rocmd` does not, and nothing about its
behaviour changes — its test is untouched and still green.

The test asserts CLEARED rather than absent, because `Command` records
`env_remove` as an explicit `(key, None)` and a plain "is it present"
check would pass with the clearing deleted. Both directions are pinned:
a caller that asks gets the removal, one that does not gets no entry at
all.

Split out of #678 on review: it is hardening that stands on its own and
is not part of ROCMAI-550.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r
r0x0r requested a review from a team as a code owner November 11, 2026 13:11
@r0x0r
r0x0r requested a review from tomastola November 11, 2026 13:11
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing here is blocking, but I'd like the first point below addressed before merge.

Requested change

  • The rocm call site (withhold_inherited_key_file: true in apps/rocm/src/engine_ipc.rs) isn't covered by any test. The new test builds EngineStdioOptions by hand in the protocol crate, so flipping rocm's value to false keeps everything green and silently re-opens the leak. Please add a test over rocm's own options/command construction that asserts the key-file variable is explicitly cleared, and confirm it goes red when the flag is flipped.

Smaller points

  • The PR text names no scenario and doesn't say why none applies for the child-env behavior. One sentence (internal hardening, covered by the unit test plus the call-site test) or an e2e scenario with a fake plugin dumping its env would do.
  • Please confirm no other rocm plugin/engine spawn path bypasses engine_stdio_command (a direct Command::new on a plugin binary would still inherit ROCM_SERVE_API_KEY_FILE). I couldn't verify this from the diff alone.
  • The opt-in bool fails open: a new caller that forgets it leaks by default. A 3-state enum (Pass(path) / Inherit / Withhold) would make the choice explicit; fine to keep as is if deliberate.

The else if ordering (a real key file wins over withholding) and the explicit (key, None) assertion are good.

endpoint_key_file: None,
// An external plugin is a third-party binary: never hand it a path to
// a live endpoint credential, including one we inherited ourselves.
withhold_inherited_key_file: true,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing tests that rocm actually sets this flag: the new test hand-builds EngineStdioOptions in the protocol crate, so flipping this to false keeps every test green and re-opens the leak. Please add a test over rocm's own stdio options / plugin command asserting the key var is explicitly cleared, or make withholding the default.

/// because "no key file for this request" and "this child may not have one"
/// are different statements: `rocmd` passes `None` for a loopback service
/// that simply needs no key, and its children are its own.
pub withhold_inherited_key_file: bool,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fail-open default: a new caller that forgets this bool leaks by default. Consider a 3-state enum (Pass(path) / Inherit / Withhold) so the two meanings of None can't be conflated; fine to keep as is if deliberate.

// a loopback service that needs no key, and clearing there would
// change what its own children see. See
// `an_external_plugin_child_is_not_handed_the_endpoint_key_file`.
command.env_remove(ENDPOINT_API_KEY_FILE_ENV);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please confirm no other rocm plugin spawn path bypasses engine_stdio_command; a direct Command::new on a plugin binary would still inherit ROCM_SERVE_API_KEY_FILE.

…hat it can

The protocol crate's test proves the flag causes the clearing, but it
hand-builds the options — so flipping `rocm`'s own
`withhold_inherited_key_file` to `false` re-opened the leak with every
test still green. That is half a chain, and the untested half is the one
a future edit would touch.

Assert it at the call site. Verified by the mutation the test exists for:
flipping the value turns it red.

Over `stdio_options` rather than a built `Command`, because
`engine_stdio_command` is private to the protocol crate and widening it
for a test would trade a real API change for a marginally stronger
assertion.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Requested change done. Pushed as c12a8ab9.

The rocm call site wasn't covered

You're right, and it's the more important half: the protocol test hand-builds the options, so flipping rocm's withhold_inherited_key_file to false left every test green and re-opened the leak. I'd proved the mechanism works and not that anything uses it.

every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file asserts it at the call site, across both stream_progress values. Verified with exactly the mutation you described — flipping rocm's value to false:

test engine_ipc::tests::every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file ... FAILED

Asserted on stdio_options rather than a built Command: engine_stdio_command is private to the protocol crate, and exporting it for a test would be a real API widening in exchange for a marginally stronger assertion. Happy to export it if you'd rather have the command-level check.

No other rocm spawn path bypasses it

Confirmed. The only other caller of resolve_engine_binary_path_with_paths is model_catalog_text.rs, which resolves the path to print it (adapter_available path=…) and never spawns. The one Command::new elsewhere in engine_ipc.rs is a test helper re-executing current_exe, not a plugin binary. So engine_stdio_command is the single plugin spawn path.

The fail-open bool

Deliberate, but you've identified the real cost, so let me give the reasoning rather than just assert it. A 3-state enum (Pass(path) / Inherit / Withhold) is the better type — it makes the two meanings of None unspeakable rather than merely documented. I kept the bool because the alternative rewrites every EngineStdioOptions literal in both binaries and the protocol crate's tests, inside a PR whose entire point is to be a small, reviewable extraction from #678.

If you'd prefer the enum I'll do it here; otherwise it's a clean follow-up, and the call-site test above is what keeps the current default honest in the meantime.

No scenario, and why

Added to the PR text: this is internal child-environment hardening with no user-visible surface, covered by the protocol-crate test (the clearing happens) plus the call-site test (rocm asks for it). An e2e scenario would need a fake plugin binary that dumps its environment — real coverage, but a new fixture for a behaviour two unit tests already pin at both ends.

Verification (Linux MI300X)

cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo test -p rocm-engine-protocol --lib, cargo test -p rocmd --lib the_endpoint_key_file (rocmd's own test, still unmodified and green), and the new rocm test.

@r0x0r
r0x0r dismissed juhovainio’s stale review November 11, 2026 13:46

Call site now covered in c12a8ab: every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file, verified red when rocm's flag is flipped to false. Confirmed no other rocm plugin spawn path bypasses engine_stdio_command. The fail-open bool is deliberate with reasoning given, and the missing-scenario rationale is in the PR text.

@r0x0r
r0x0r requested a review from juhovainio November 11, 2026 13:46

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head c12a8ab9

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (apps/rocm/src/engine_ipc.rs, apps/rocmd/src/engine_ipc.rs, crates/rocm-engine-protocol/src/stdio_client.rs).
Split out of #678 per that PR's own description (confirmed by reading #678: it names this PR as where the engine-protocol key-file withholding independently lives); explicitly not part of ROCMAI-550, and no other ticket is named.

Blocking

  • CI is red on two of the jobs that would actually run this PR's tests, at this exact head — Test (affected crates) (ubuntu-24.04 and ubuntu-26.04) and E2E tests (both variants).
    Not a code defect: both failed inside the shared Install native build deps step with Temporary failure resolving 'azure.archive.ubuntu.com' while running apt-get install libcap-dev — a transient DNS/network blip on the runner, before any crate compiled or any test executed. The sibling build-and-test job's identical step succeeded a few minutes earlier on this same commit, which is consistent with a flake rather than anything this diff touches. Practically, though, this means there is currently no CI-green run proving the new tests (every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file, an_external_plugin_child_is_not_handed_the_endpoint_key_file) actually pass — only the author's local MI300X run (named in the PR body) attests to that today.
    Confidence 100 · mechanical · Fix: re-run Test (affected crates) and E2E tests on this head (no code change expected) and confirm green before merge.

Non-blocking

  • docs/architecture.md's entry for stdio_client.rs enumerates EngineStdioOptions's fields and is now missing the one this PR adds — docs/architecture.md:609. The entry parenthetically lists "app paths, engine environments root, endpoint key file, progress request" as the fields EngineStdioOptions threads; withhold_inherited_key_file — the field this PR's two call sites deliberately differ on — isn't added. cargo xtask check-architecture-doc only requires an entry to exist per source file, not that its prose stays accurate (the repo's own AGENTS.md says as much: "the check reads no prose, so keeping an entry's description true is still on you"), so this won't fail CI, but the enumeration is now incomplete by its own standard.
    Confidence 85 · mechanical · Fix: add the new field to that parenthetical.

Decisions for the author

  • withhold_inherited_key_file: bool fails open under #[derive(Default)] — non-blocking-improvement. Today both call sites (rocm: true, rocmd: false) are explicit, so there's no live defect. But the field defaults to false (inherit), so a future third-party-plugin spawn path built with EngineStdioOptions { ..Default::default() } would silently re-open exactly the leak this PR closes, with nothing short of a reviewer catching the omission. A 3-state enum (e.g. Pass(path) / InheritAmbient / Withhold) would turn a forgotten choice into a compile error instead of a silent default. Worth weighing against the ceremony cost for what are, today, exactly two call sites.

Positive signals

  • an_external_plugin_child_is_not_handed_the_endpoint_key_file asserts the distinction that actually matters — Command::get_envs() reports an explicit (key, None) tombstone from env_remove, not merely "absent" — and pins both directions (withhold clears; no-withhold never touches the variable) in one test, so flipping either caller's option value would be caught.
  • Splitting the proof across two crates (rocm's stdio_options() asserts it asks to withhold; the protocol crate's test proves withholding actually clears the var) is a deliberate, well-justified workaround for engine_stdio_command being private to the protocol crate — each test's own doc comment explains why it's scoped that way, rather than leaving a reader to wonder why the assertion isn't on a built Command.

Deployment notes

None.

What this covered

Read apps/rocm/src/engine_ipc.rs, apps/rocmd/src/engine_ipc.rs and crates/rocm-engine-protocol/src/stdio_client.rs in full (base f04b785b…head c12a8ab9), plus impact-analysis surface: confirmed by repo-wide grep that no other site in the workspace constructs EngineStdioOptions; checked every other path that could spawn a third-party engine plugin binary (apps/rocm/src/service_launch/spawn.rs, apps/rocmd/src/supervise.rs — both spawn rocm itself via __engine-serve-http, not a plugin binary, so outside this PR's stated threat model; apps/rocm/src/examine_report_text.rs and model_catalog_text.rs only resolve and display the binary path, never spawn it); read docs/architecture.md's entries for the three touched files and for stdio_client.rs; read the repo's AGENTS.md. Single-context pass, not fanned out to subagents — a judgment call under the method's sizing step given 3 files / 113 added lines all tightly coupled around one struct, not a degraded run.

Did not run: the "prior changes" pass (searching historical review comments on earlier PRs touching these three files) — not attempted, judged low-value at this size. No Rust toolchain is available in this environment, so cargo test/cargo clippy were not independently executed here; CI's own results were read instead (see Blocking) rather than assumed. The dedupe check mandated before this review (confirming no prior review from this account at this exact head) necessarily surfaced an existing dismissed human review from juhovainio on an older commit of this PR; it was read only for that check and to confirm the current head already contains the rocm-call-site test it had asked for, and did not otherwise shape this review's findings, which were reached independently from the code and the repo's own AGENTS.md. Not otherwise reconciled against any discussion on this PR.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: the four red checks are runner infrastructure, not this change.

Test (affected crates) and E2E tests (both Ubuntu variants) failed during the apt step, before anything was built:

W: Failed to fetch .../dists/noble/InRelease  Temporary failure resolving 'azure.archive.ubuntu.com'
E: Failed to fetch .../libcap-dev_2.66-5ubuntu2.4_amd64.deb  Temporary failure resolving 'azure.archive.ubuntu.com'
##[error]Process completed with exit code 100.

DNS resolution failing on the runner while installing system dependencies. Nothing in this PR can reach that — the diff is three Rust files and touches no CI config, and the same checks were green on the previous head.

They should clear on the next run; I'll confirm rather than assume.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this at the current head and didn't find a defect in the change itself: the flag is opt-in, rocmd behavior is unchanged, and the protocol-side test covers both the cleared and the absent case. I'm requesting changes only for a few things to confirm or state before merge:

  • Please confirm every rocm external-plugin spawn goes through stdio_options; the new rocm test only pins that function's output.
  • Please confirm no other inherited variable carries the endpoint key itself, since only the key-file path variable is cleared.
  • The PR body should say why there is no cucumber scenario, and whether the rocm-side test was shown to fail with the flag flipped to false (only the protocol test's mutation check is claimed today).
  • FYI: EngineStdioOptions is pub with a new field. Default gives false so the default is safe, but out-of-tree struct literals would stop compiling.

On CI: E2E tests and Test (affected crates) fail on both ubuntu runners in "Install native build deps" (apt cannot resolve the mirror, exit 100). That looks like infrastructure, not this change; they need a rerun.

/// on `stdio_options` rather than on a built `Command` because
/// `engine_stdio_command` is private to the protocol crate.
#[test]
fn every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please confirm (and note in the PR body) that you broke this by flipping the flag to false and saw this test go red, and that every plugin spawn path goes through stdio_options.

// a loopback service that needs no key, and clearing there would
// change what its own children see. See
// `an_external_plugin_child_is_not_handed_the_endpoint_key_file`.
command.env_remove(ENDPOINT_API_KEY_FILE_ENV);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the key-file path variable is cleared. If any other inherited variable carries the endpoint key itself, a third-party plugin would still see it; worth confirming none exists.

@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

All four confirmed, and the PR body now carries each answer rather than leaving it in a comment thread.

Does every rocm external-plugin spawn go through stdio_options?

Yes. engine_stdio_command is the only place rocm spawns a plugin binary, and stdio_options is its only options source. The other two callers of resolve_engine_binary_path* resolve the path only to print it:

  • model_catalog_text.rs:783 → adapter_available path={}
  • examine_report_text.rs:426 → external path={}

Neither constructs a Command.

Does any other inherited variable carry the endpoint key?

No — for the endpoint key. ROCM_SERVE_API_KEY_FILE is the only variable that carries it, and it carries a path: resolve_endpoint_api_key() reads the key by opening the file that variable names. There is no variable holding the key value, so clearing this one closes the whole env-borne route.

One honest qualification, so "confirmed" isn't read more broadly than it should be: unrelated provider secrets (OPENAI_API_KEY, ANTHROPIC_API_KEY, ROCMDASH_CHAT_API_KEY) still reach children by ordinary inheritance. Different concern, untouched here — but a plugin child is not a secret-free environment, and I'd rather say so than let this read as a general guarantee.

The rocm-side mutation

It was run — I just hadn't stated it. Both halves of the chain, now in the body:

test mutation result
an_external_plugin_child_is_not_handed_the_endpoint_key_file (protocol) delete the env_remove FAILED
every_external_plugin_spawn_asks_to_withhold_the_endpoint_key_file (rocm) flip rocm's flag to false FAILED

No scenario, and why

In the body now: internal child-environment hardening with no user-visible surface, bracketed by the two unit tests above. An e2e scenario would need a fake plugin that dumps its environment — real coverage, but a new fixture for a behaviour both ends already pin.

The pub struct gaining a field

Noted in the body. Default gives false, so the default behaviour is the safe, unchanged one; an out-of-tree struct literal would need the new field to keep compiling.

CI

Agreed, and thanks for checking it independently — apt cannot resolve the mirrors (Temporary failure resolving 'azure.archive.ubuntu.com', exit 100) before anything is built. It is fleet-wide rather than PR-specific: PR #1040 is failing the same way right now. I'll confirm on the next run rather than assume.

@r0x0r
r0x0r dismissed juhovainio’s stale review November 11, 2026 14:40

All four confirmed and written into the PR body: engine_stdio_command is the only plugin spawn path (the other two resolve callers only print the path); ROCM_SERVE_API_KEY_FILE is the only variable carrying the endpoint key, and it carries a path — with the honest caveat that unrelated provider keys still inherit; the rocm-side mutation was run and is now stated; and the pub-struct field note is included.

@r0x0r
r0x0r requested a review from juhovainio November 11, 2026 14:41
r0x0r added a commit that referenced this pull request Nov 11, 2026
Three leftovers from the split, all of which made this diff claim work it
no longer ships.

`stdio_client.rs` still carried an older, UNCONDITIONAL version of the
endpoint key-file clearing — the one that broke `rocmd`'s contract before
#1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now,
and `rocmd`'s own
`the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key`
is green again here.

`docs/testing.md` still described four support levels and pointed at
examine-50/51/52. This change ships three levels and examine-49; the
fourth level and those scenarios are #1036's, and the text now says so.

The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead
here — nothing can produce that level without the probe — so it goes back
to #1036 with the level it tests.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…tions

The `stdio_client.rs` entry enumerates the fields `EngineStdioOptions`
threads to the child, and the field the two call sites deliberately differ
on was missing from that list. `check-architecture-doc` reads no prose, so
nothing would have caught it.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Confirmations and one doc fix — bed02289f

Thanks @juhovainio and the automated pass. Three of the four asks were already in the PR body when the 14:18 review was written (the "why no cucumber scenario" section and the mutation table covering both halves of the chain, including flipping rocm's flag to false); re-confirming the two that were not, and fixing the doc gap.

1. Does every rocm external-plugin spawn go through stdio_options?

Yes, and I re-checked it against the current tree rather than quoting my earlier answer. Workspace-wide, EngineStdioOptions is constructed in exactly two places:

  • apps/rocm/src/engine_ipc.rs:176 (in stdio_options) — withhold_inherited_key_file: true
  • apps/rocmd/src/engine_ipc.rs:210 — false, behavior unchanged

There is no third. And ENDPOINT_API_KEY_FILE_ENV is set on a child in exactly one place, stdio_client.rs:166, inside engine_stdio_command — so the variable this PR withholds cannot reach a child by any route that bypasses these options. The other resolve_engine_binary_path* callers (examine_report_text.rs, model_catalog_text.rs) only resolve and print the path; they never spawn.

2. Does any other inherited variable carry the endpoint key itself?

Yes — and this PR does not clear it. I want to be precise rather than reassuring here, because the short answer "no" would be wrong.

ROCM_SERVE_API_KEY carries the key value, not a path, and rocm serve reads it (serve_cmd.rs:537). It is not cleared by this change, so a plugin child does inherit it when the user has it set.

What makes that a different situation from the one this PR closes:

  • rocm never sets ROCM_SERVE_API_KEY on a child — grep for any .env("ROCM_SERVE_API_KEY", …) or set_var comes back empty. It is purely an input the user puts in their own shell. A plugin inheriting it is the same exposure as the user running that plugin directly from the same shell.
  • ROCM_SERVE_API_KEY_FILE is the one rocm/rocmd set programmatically on engine children (stdio_client.rs:166), which is exactly why an inherited value reaching a third-party plugin is this PR's concern.
  • The engine adapters deliberately read only the _FILE carrier and never the value-bearing variable — lib.rs:37 spells out the reason: "the engine child inherits the launching shell's environment, so reading a variable like ROCM_SERVE_API_KEY would let a stray value in the user's shell silently authenticate an endpoint we intended to leave open".

So the boundary this PR draws is "what rocm itself propagates", not "everything that could carry a secret". Provider keys (OPENAI_API_KEY and friends) still inherit too, by the same reasoning. If you want the stronger boundary — an allow-list environment for plugin children rather than inherit-and-subtract — that is a real design change and I would rather it be its own PR than be smuggled in here.

3. Doc fix

@nowycondro was right that docs/architecture.md:609 enumerates EngineStdioOptions's fields and was missing the one this PR adds — the single field the two call sites differ on. Added in bed02289f. As that review notes, check-architecture-doc reads no prose, so nothing would have caught it.

On the bool fails-open point (both reviewers)

Taking it as a deliberate keep, not an oversight. Both call sites are explicit today and Default gives the unchanged (false) behavior, so there is no live defect; a 3-state enum would turn a forgotten choice into a compile error, which is genuinely better, but it is API churn on a pub type for two call sites. If a third plugin-spawning path ever appears, that is the moment the enum earns its cost — happy to be overruled.

CI

Test (affected crates) and E2E tests on both runners are still the azure.archive.ubuntu.com apt DNS failure in "Install native build deps", before any crate compiles. Same outage is hitting other authors' PRs (#1040, #997, #998). build-and-test passed on the same commit.

Verification (Linux devbox; this Mac cannot build rocm-engine-lemonade)

cargo clippy --workspace --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo fmt --all --check, cargo xtask check-crate-edges / check-module-size (419 files) — all clean; cargo test -p rocm --bin rocm withhold — 4 passed.

One honest note on my own process: the first run of that test came back red, and it was my tooling, not the code — a stale build artifact from the mutation run above, because my file transfer preserved mtimes and shared a CARGO_TARGET_DIR, so cargo reused the mutated binary without recompiling. Re-ran after forcing a rebuild and it is green. Mentioning it because "the test went red once" is the kind of thing worth saying out loud rather than quietly re-running until it passes.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The clearing is scoped correctly: rocm's external plugin spawns ask for it, rocmd keeps its current behaviour, and the protocol test checks the explicit env_remove entry rather than mere absence. Earlier points (call-site test, scenario justification, spawn paths) are addressed.

Two small non-blocking things, inline below. Also optional: the new protocol test repeats full EngineStdioOptions literals, ..Default::default() would cut churn when fields are added. I still lean slightly toward an enum over the bool, since a forgotten flag fails open, but the documented default is a reasonable call.

CI: the red jobs all failed at "Install native build deps" on apt DNS errors, so tests never ran there; they need a rerun.

/// is the other half of the chain, and without it flipping the value here
/// to `false` would re-open the leak with every test still green. Asserted
/// on `stdio_options` rather than on a built `Command` because
/// `engine_stdio_command` is private to the protocol crate.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is engine_stdio_command really private to the protocol crate? The PR text says rocm spawns through it. If it is reachable here, asserting that the built Command's get_envs() has (ROCM_SERVE_API_KEY_FILE, None) would prove the end state rather than only the flag. Otherwise this comment is fine as is.

}
if let Some(key_file) = options.endpoint_key_file {
command.env(ENDPOINT_API_KEY_FILE_ENV, key_file);
} else if options.withhold_inherited_key_file {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This precedence is not covered: endpoint_key_file: Some(p) with withhold_inherited_key_file: true must set the path, not remove it. Swapping the branch order stays green today. A one-line case in the new test would pin it.

@r0x0r
r0x0r added this pull request to the merge queue Nov 11, 2026
@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Approved and green — enqueued

Thanks for the approval. CI is now fully green on bed02289f and this is in the merge queue at position 1.

For the record, since the red checks were discussed above and nothing about the code changed: the seven failing Linux jobs were the azure.archive.ubuntu.com apt-mirror outage, failing in "Install native build deps" before any crate compiled. I re-ran them rather than pushing anything, and they passed — the outage has cleared. build-and-test (both runners), Test (affected crates) (both), E2E tests (both) and Coverage are all green, as are the Windows lanes, which had been passing throughout.

There is also a ci/apt-mirror-fallback branch in this repo adding a public-mirror fallback for exactly this failure mode, so the class of outage that blocked this PR for most of the day is being addressed properly rather than by re-running.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@r0x0r
r0x0r added this pull request to the merge queue Nov 11, 2026
@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Dequeued without a failure — re-enqueued

This was dropped from the merge queue at 17:58 after ~20 minutes, and I want to record why, since it was not a defect in this PR.

The merge-queue CI run (43069895249, branch gh-readonly-queue/main/pr-1034-...) reports cancelled, but no job failed. The only non-success job was Sphinx docs build (-W), itself cancelled; CodeQL passed and every other job completed. The cancellation took the run with it, the queue entry was removed, and #994 took the slot.

Nothing changed on this PR — still bed02289f, APPROVED, MERGEABLE/CLEAN, with the full Linux and Windows suites green on that head after the apt outage cleared.

Re-enqueued (position 4). If it drops a second time with the same cancelled-Sphinx signature, that is a queue/workflow problem worth fixing rather than something to keep retrying, and I will stop and say so instead of re-enqueuing again.

Merged via the queue into main with commit 3636a64 Nov 11, 2026
58 of 65 checks passed
@r0x0r
r0x0r deleted the split/engine-key-file-withholding branch November 11, 2026 19:08
r0x0r added a commit that referenced this pull request Nov 17, 2026
Three leftovers from the split, all of which made this diff claim work it
no longer ships.

`stdio_client.rs` still carried an older, UNCONDITIONAL version of the
endpoint key-file clearing — the one that broke `rocmd`'s contract before
#1034 made it opt-in. Reset to main's. The guard lives in #1034 alone now,
and `rocmd`'s own
`the_endpoint_key_file_reaches_the_child_only_when_it_holds_a_usable_key`
is green again here.

`docs/testing.md` still described four support levels and pointed at
examine-50/51/52. This change ships three levels and examine-49; the
fourth level and those scenarios are #1036's, and the text now says so.

The `NoGpuBackend` arm in `assert_support_level_advice_fits` was dead
here — nothing can produce that level without the probe — so it goes back
to #1036 with the level it tests.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants