Skip to content

test(core): let e2e builds re-root the hardware probes - #510

Merged
rominf merged 12 commits into
mainfrom
test/e2e-host-root
Oct 12, 2026
Merged

rominf merged 12 commits into
mainfrom
test/e2e-host-root

Conversation

@rominf

@rominf rominf commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

First of a short series that lets the cucumber E2E suite describe the GPU machine a scenario needs, instead of depending on the runner's hardware. Today a scenario about eight Instinct GPUs, a WSL2 distribution or a host with no GPU can only run where such a machine happens to be. The CLI reads fixed host paths to learn what it is running on, so nothing else can stand in.

  • Adds rocm_core::host_path(). In a build with the new rocm-core/e2e-test-hooks feature, it re-roots absolute host paths under $ROCM_CLI_TEST_HOST_ROOT. Without the feature it is the identity and never reads the environment, so a release build always probes the real machine. This is the same model as the existing ComfyUI source-archive override.
  • Routes these probes through it:
    • the KFD topology (/sys/class/kfd/kfd/topology/nodes), /dev/kfd, the render nodes under /dev/dri, and /sys/class/drm
    • /sys/module/amdgpu and /sys/module/amdgpu/version
    • /dev/dxg, /proc/version, the /usr/lib/wsl/lib directory check and /usr/lib/wsl/lib/libdxcore.so, and the WSL ROCDXG check (lib/librocdxg.so, share/rocdxg/dids.conf under /opt/rocm and each discovered install)
    • /etc/os-release for the distro name examine reports (detect_distro_name), plus /proc/cpuinfo, /proc/meminfo, /proc/cmdline and the modprobe directories /etc/modprobe.d, /usr/lib/modprobe.d and /run/modprobe.d
    • /proc/modules, read only when lsmod fails
  • The CLI keeps printing the logical path (/dev/kfd, /etc/modprobe.d/x.conf), never the re-rooted one.
  • Container detection (/.dockerenv, /run/.containerenv, /proc/1/cgroup) is routed too.
  • rocm-core gains an e2e-test-hooks feature. The E2E build (cargo build -p rocm -p rocmd --features rocm/e2e-test-hooks, as cargo xtask e2e runs it) turns it on for both binaries, because Cargo builds one shared rocm-core for that invocation and rocm's feature enables it. rocmd and the two engine crates also declare a forwarding feature, but that edge fires only when the feature is named on them, and the standalone rocm-engine-vllm/rocm-engine-lemonade binaries are never built with it. Neither engine nor rocmd calls a re-rooted probe today.
  • Documented beside the ComfyUI override in docs/release-trust.md.

Risk: low. The release build gets a PathBuf where it had a &Path, and nothing else changes; the identity is pinned by a unit test that really sets the variable.

Deliberately left on the real host, and known gaps:

  • ROCm install discovery reads the real host: which installs exist, and their paths and versions. The one re-rooted read under them is the WSL ROCDXG existence check, which looks for its files under /opt/rocm and under each discovered install's path inside the root.
  • Programs the probes execute, such as ldconfig, run from the real host. If a scenario needs to control the linker-cache answer, the follow-up is to inject the cache text, not to re-root the executable. lspci and uname are not re-rooted either.
  • The /usr/lib/wsl/lib loader entry an engine is launched with (the existence check on that directory is routed), /proc/<pid> liveness and /dev/shm sizing stay on the real host.
  • Every package-manager plan reads the real /etc/os-release: the OpenMPI install hints in rocm-core, and the driver, OpenMPI and runtime-library installs the rocm binary runs. Their commands act on the real machine. Only the distro name examine reports follows a simulated root.
  • The dashboard's amd-smi pre-flight follows the simulated root only to hide a GPU: the real /dev/kfd keeps a veto, because the check gates a real process. On WSL, the reachability verdict still bypasses the device check, as before.
  • The host-root unit tests in rocm-core run in CI through a new feature-on nextest step in the affected-crates job, whenever rocm-core is affected. The rocm-binary test that read_os_release ignores the root runs only locally with --features e2e-test-hooks. Under a feature-on threaded cargo test, other tests that read the real host can overlap the window in which the root is set, which is why CI uses nextest.

Follow-ups in this series add an E2E_HARDWARE=simulated|real switch with a @requires-real-gpu tag, then the simulated-machine Given steps and fake host tools. Those move 19 scenarios that only need the CLI to see a GPU (detection, diagnose, WSL driver plans, --gpu refusals) onto every Linux lane, including the GitHub-hosted one. This PR changes no scenario. Follow-ups: #517 (simulated machines and the drift check), then #518 (narrowing the per-PR GPU lanes to a smoke test).

Scenario coverage (AGENTS.md §3)

No user-observable behaviour changes: release builds are unaffected, and the hook only exists under e2e-test-hooks. So there is no new scenario here. The re-rooting is exercised end to end by the follow-up PR's simulated-machine scenarios, which fail if any routed probe ignores the root.

Test plan

  • cargo test -p rocm-core --lib hardware_root passes both with and without --features e2e-test-hooks. The non-feature test sets ROCM_CLI_TEST_HOST_ROOT and asserts the path is unchanged; the feature test asserts re-rooting and that an empty value counts as unset.

  • cargo clippy --workspace --all-targets -- -D warnings is clean both with and without --features rocm/e2e-test-hooks.

  • Manual check on a WSL2 machine with a planted root holding an MI300X KFD node, /dev/kfd and a non-WSL /proc/version:

    • a hooks build of rocm examine reports wsl: false, detected_gfx_target: gfx942, driver_status: amdgpu_available and default_engine: vllm
    • a release build given the same variable still reports the real WSL2 machine
  • The hardware_root.rs module doc, docs/release-trust.md and this description list the same routed reads, checked against every host_path/read_host_text call site.

  • Not run locally: the full cargo test --workspace. Two proc_lifecycle tree tests fail on WSL2 against unmodified main. CI runs the full suite.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (n/a, no bug fix)

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (n/a, no subcommand; the helper is its own module)

  • Every new or changed user-facing message was read against the code path that runs after it. (n/a, no user-facing message changes)

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this one carefully, because a seam that re-points the hardware probes is the kind of change that either leaks into release builds or quietly leaves half the probes reading the real machine.

The release-build half holds up. hardware_root() under #[cfg(not(feature = "e2e-test-hooks"))] is a const fn returning None and genuinely cannot reach the environment, no crate enables the feature by default, nothing in CI uses --all-features, and the release build passes no --features at all. The identity test that pins this does run. I also checked the two string reconstructions in probe_modules and probe_devices and they are byte-for-byte what the old entry.path() produced, so default output is unchanged.

What I would want addressed is the other half: consistency, and one conversion I think is actively wrong.

The ldconfig change routes an executable path through a seam the module's own docs say is for reads only, and I think it makes the function worse in both directions rather than better. Details inline.

Two probes were left outside the seam where I cannot see a reason. Container detection in examine.rs, and the dash collectors' own /dev/kfd check, which the new rocmd feature flag does not reach. Both produce exactly the mixed answer this change exists to prevent: a simulated host that reports the runner's real containerization, or the runner's real GPU presence, while every other fact is simulated.

I would also push back gently on the PR text's claim that the feature is forwarded from rocm, rocmd and both engines so the existing build turns it on everywhere. It does end up on for rocm and rocmd, but through Cargo unifying the single shared rocm-core build in that one combined invocation, not through the forwarding edges, and the standalone engine binaries never get it at all. Harmless today, but the sentence promises a guarantee the build does not give.

Finally, the new tests cover host_path and nothing downstream of it. None of the roughly thirty converted call sites is exercised, and as you note, no CI job enables the feature for unit tests, so the one test proving re-rooting actually happens never runs automatically. The two missed probes above are what that gap looks like in practice.

None of this is a release-safety problem, and the overall design reads well to me. I am asking for changes because the two unrouted probes will silently undermine the follow-up's simulated scenarios, and because the ldconfig conversion looks like it should be reverted rather than refined.

A few concerns I chased that did not hold: no double re-rooting in rocm_relative_file_exists, strip_prefix("/") is component-based so the path algebra is right, GpuProbeSources::host's new lifetime has no missed callers, the logical-versus-physical reporting contract holds everywhere I looked in lib.rs and main.rs, and the e2e lanes all build with the same feature set so there is no mixed-binary run.

Comment thread crates/rocm-core/src/lib.rs Outdated
for program in ["ldconfig", "/sbin/ldconfig", "/usr/sbin/ldconfig"] {
if let Some(text) = capture_optional_command(program, &["-p"]) {
let program = host_path(program);
if let Some(text) = capture_optional_command(&program.to_string_lossy(), &["-p"]) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This routes an exec path through the seam, which the host_path module docs specifically argue against: "anything a real process acts on rather than reads to describe the machine" is the stated exclusion, and capture_optional_command bottoms out in Command::new(program).spawn().

I think it also makes the function worse in both directions.

Where ldconfig is on PATH, the bare relative candidate is tried first and passes through host_path unchanged, so the real host's linker cache answers and the re-rooting never happens at all.

Where it is not on PATH — the exact case the doc comment directly above says these absolute fallbacks exist for, a non-root user on Debian and derivatives — the fallbacks now resolve to <root>/sbin/ldconfig and <root>/usr/sbin/ldconfig. A fixture root populated with sysfs and procfs text will not have a working linker binary there, so the loop falls through to None, ldconfig_lists_librocdxg() degrades to "could not ask", and the ROCDXG verdict changes under simulation on precisely the hosts the fallback was added to rescue.

I would leave these three candidates off the seam entirely. If the e2e suite needs to control this answer, injecting the cache text (or an explicit None) is a far smaller seam than making the probe executable-from-fixture.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, reverted in e22d8726. The three candidates are back off the seam, with a doc line saying why, and executed programs are now in the hardware_root exclusion list. If the suite later needs to control this answer, I'll inject the cache text as you suggest, in a follow-up.


/// [`read_text`] for an absolute host path, read where [`crate::host_path`]
/// says the host is.
fn read_host_text(path: &str) -> String {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anchoring here because the real location is not in the diff: probe_container, at crates/rocm-core/src/examine.rs:2201-2218, still reads /.dockerenv and /run/.containerenv through Path::new(...).exists() and /proc/1/cgroup through read_text, so it never reaches this new helper.

It sits in the same Linux probe sequence as the probes you did convert — probe_env immediately before it, probe_shared_memory immediately after — and e.in_container / e.container_kind feed the diagnose scoring directly. CI runners very often run inside a container themselves, so a scenario describing a bare-metal machine picks up the runner's real containerization alongside otherwise-simulated facts. That is the mixed answer this change exists to remove, and it is the same shape as WSL detection, which you did route.

It also appears in neither list in the hardware_root module docs: not in the routed examples, not in the deliberate exclusions. So it reads as an oversight rather than a decision. If it is deliberate, the exclusion list is the place to say why. Otherwise the two existence checks want crate::host_path and the cgroup read wants read_host_text.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An oversight, thanks. Fixed in 52c56b1d: both marker checks go through crate::host_path and the cgroup read through read_host_text, and they're now listed among the routed probes in the module docs. ffe5386e covers them, planting the container once by marker and once by cgroup alone.

Comment thread apps/rocmd/Cargo.toml
workspace = true

[features]
e2e-test-hooks = ["rocm-core/e2e-test-hooks"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth checking what this flag actually buys rocmd today.

I could not find a call from apps/rocmd/src/ into any probe this PR converted, and the GPU-presence check the daemon's dashboard path depends on lives outside the seam: crates/rocm-dash-collectors/src/amd_smi.rs opens the literal /dev/kfd in device_accessible (KFD_DEVICE at line 23, reached from line 84 via preflight_passes). That crate deliberately has no rocm-core dependency.

The fallback is reached in exactly the case this feature targets. gpu_reachable_for_preflight(is_wsl_host, has_usable_gpu) in apps/rocm/src/dash.rs is false on any non-WSL host by design, so on a simulated bare-metal machine preflight_passes falls straight through to device_accessible(Path::new(KFD_DEVICE)) against the runner's real device. This one is not in the known-gaps list in the PR description.

preflight_passes already takes an injectable &Path, so this should not need a new crate edge — dash.rs already depends on rocm-core, and could resolve the path through host_path and thread it in the same way it already threads the WSL verdict. Failing that, it belongs in the known gaps, because as written rocmd gains a feature flag that does nothing for its own GPU probe.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Routed in 894d5cbc, with one constraint. This pre-flight decides whether a real amd-smi process may start, so re-rooting /dev/kfd alone isn't safe: a simulated host that plants /dev/kfd on a GPU-less runner would launch amd-smi into the D-state hang the check exists to prevent. Instead dash.rs resolves the host-root path through rocm_core::host_path and passes it through RunnerOptions, the same way as the WSL verdict, and amd-smi runs only if both the real node and the host-root one are readable. A simulated GPU-less host now hides the runner's GPU, and a simulated GPU can't start amd-smi on a machine without one. A new four-case test pins this: dropping either half of the rule fails it. Without a root the two paths are equal and the second open is skipped, so release behaviour is unchanged, and no new crate edge is needed. On WSL, the reachability verdict still stands in for the device check, as before.

On the rocmd half: you're right that its feature edge does nothing today. rocmd calls no converted probe and doesn't depend on the dash crates; the dashboard daemon runs inside rocm. The description is corrected.

Comment thread engines/vllm/Cargo.toml
workspace = true

[features]
e2e-test-hooks = ["rocm-core/e2e-test-hooks"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An accuracy point on the PR description rather than on this line itself.

"The feature is forwarded from rocm, rocmd and both engines, so the existing cargo build -p rocm -p rocmd --features rocm/e2e-test-hooks ... turns it on everywhere" is not quite how this resolves.

--features rocm/e2e-test-hooks is scoped to the rocm package and never names rocmd, so rocmd's own edge does not fire. It gets the feature because Cargo unifies the single shared rocm-core build across that one combined invocation. Build rocmd by itself and the edge still will not fire.

These engine edges likewise apply to the engine libraries that rocm links, not to the standalone rocm-engine-vllm and rocm-engine-lemonade binaries. Those are built only in the ci.yml lane that compiles -p rocm -p rocmd -p rocm-engine-lemonade -p rocm-engine-vllm -p xtask, which passes no features at all. The e2e lanes never build them.

Nothing breaks today, since neither engine calls host_path and rocmd calls none of the converted probes. But the sentence asserts a guarantee the build does not provide, and whoever later adds a probe behind one of those entry points gets the real host silently. Worth either narrowing the claim or having the e2e build name the packages explicitly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and the description is narrowed. rocmd gets the feature because Cargo builds one shared rocm-core for the combined invocation, not through its own edge, and the standalone engine binaries never get it. Nothing in those entry points calls a re-rooted probe today.

Comment thread docs/release-trust.md Outdated
the DRM cards under `/sys/class/drm`, `/sys/module/amdgpu`, `/proc/version`,
`/proc/cpuinfo`, `/proc/meminfo`, `/proc/cmdline`, `/proc/modules`,
`/etc/os-release`, the modprobe configuration directories, and the WSL plumbing
under `/usr/lib/wsl` and `/opt/rocm`. The E2E suite can point those reads at a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list and the paragraph a dozen lines below contradict each other on /opt/rocm. Here it is named among the paths that can be pointed at a simulated host; below, "discovered ROCm installs ... are never re-rooted."

Both describe the same literal string, and only the narrow case is true. The hardcoded /opt/rocm existence check in rocm_relative_file_exists is re-rooted, but discover_rocm_installs() — which produces the "ROCm install detected" fact and version that examine reports — uses raw fs::read_dir, fs::canonicalize and fs::read_to_string with no host_path anywhere. This is a trust document, so someone auditing it would reasonably come away believing the reported install version is simulatable. It is not.

I would name the specific check that is re-rooted (the WSL ROCDXG capability probe) rather than the bare path. The /usr/lib/wsl entry has the same shape, but there the closing paragraph's "loader entry an engine is launched with" does disambiguate adequately.

While you are in here: the list does not carry the /etc/os-release carve-out you mention in the PR description, where the install-hint helpers in crates/rocm-core/src/openmpi.rs still read the real file. That is invisible to someone reading only this document.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a200d963. The list now names the WSL ROCDXG file check under /opt/rocm instead of the bare path. It says plainly that install discovery, and the version examine reports, always read the real host, and it carries the /etc/os-release install-hint carve-out. 894d5cbc adds the dash pre-flight as routed, with the real device keeping a veto.

}

#[cfg(test)]
mod tests {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests pin host_path_under's algebra and the env-var honouring, all against the single literal /dev/kfd. That is the plumbing, not what the plumbing was installed for.

Nothing exercises any of the roughly thirty converted call sites. The existing _in-suffixed seams (detect_kfd_gfx_target_in, linux_kfd_gpu_node_count_in) take a directory directly and bypass host_path, so even the thin wrappers that do call it are uncovered. Reverting any single conversion back to a raw literal would compile and pass the entire suite.

Combined with the gap you already flag — no unit-test, nextest or clippy job enables rocm-core/e2e-test-hooks — the one test proving re-rooting actually happens never runs anywhere automated. The feature-off identity test does run, which is the safety-critical direction, so this is not a release risk. It is that the seam's usefulness currently rests entirely on review.

I am not asking for a test per call site. One test that plants a root holding a KFD node and a /dev/kfd, runs the probes against it and asserts the simulated answer would have caught both missed probes I flagged elsewhere in this review, and would keep catching the next one.

Separately and minor: host_path_under does not guard ... An input containing .. after the leading slash walks back out of the root, because join does not normalise. Not reachable through any current call site, since they are all fixed literals, so I would just state the constraint in the doc comment rather than add validation.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in ffe5386e. One test plants a root with a KFD topology, /dev/kfd, a non-WSL /proc/version and container markers, runs the real converted probes with the feature on, and asserts the simulated answers. Reverting the container, cgroup, KFD or /proc/version conversions each make it fail. A /dev/dxg revert only went undetected on our host because it has no real /dev/dxg. host_path_under's doc now states the no-.. precondition. The test runs only with --features e2e-test-hooks, and the description says so.

@rominf
rominf dismissed volen-silo’s stale review October 5, 2026 11:27

Dismissed as stale: this review is of 2daa432. Every finding is addressed, with a reply in each thread: ldconfig back off the seam (e22d872), container detection routed (52c56b1), a planted-root test of the converted probes (ffe5386), release-trust accuracy (a200d96), and the dash /dev/kfd pre-flight routed with the real device keeping a veto (894d5cb). The description is corrected. Please re-review the current head.

@rominf

rominf commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (5f72ac6a). The only conflict was additive: #407 added pub mod model_readiness; in the same spot as hardware_root, and both are kept. The new module reads no host paths directly, so there was nothing new to route. Verified locally: rocm-core lib tests without (476) and with (477) e2e-test-hooks, xtask (255), check-crate-edges, dash collectors (108), dash:: (24), e2e-cucumber lib and feature_naming, and clippy -D warnings on the workspace with and without the feature, plus --test e2e. No code change.

@rominf
rominf force-pushed the test/e2e-host-root branch from 894d5cb to f7ef476 Compare October 5, 2026 11:46

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Five commits against six points, and most of them are properly closed. Going through them in order, because you did enough work here that it should be said explicitly rather than inferred from which comments I did not repeat.

Closed.

  1. ldconfig_cache — reverted. The three candidates are off the seam and "programs the probes execute" is now in the exclusion list, which is the right place for it. Injecting the cache text later, if the suite ever needs to control that answer, is the smaller change it always was.

  2. probe_container — both marker checks and the cgroup read now go through the seam, and planting the container twice (once by marker, once by cgroup alone) is a better test than I asked for. I reverted each of the two conversions by hand and the test caught both.

  3. The dash pre-flight — this is the one I expected to be awkward and the answer is better than the one I suggested. Requiring both the real node and the host-root node, rather than substituting one for the other, keeps the D-state hazard the check exists for while still letting a simulated GPU-less host hide the runner's GPU. I traced it rather than taking it on trust: dash.rs is the only construction site that should re-root and the only one that does, every other RunnerOptions comes from ..Default::default() and gets the real device, and the two test-only entry points pass the real literal deliberately. With no root the two paths compare equal, the second open is short-circuited, and the expression collapses to exactly the old single check. device_accessible is a bare read-only open, so a planted regular file and a real character device behave the same for it. No new crate edge, as you said.

  4. The forwarding claim — narrowed, and the narrowed version matches what the build actually does. The combined -p rocm -p rocmd invocation is the only reason the daemon gets it, and the standalone engine binaries are built in a lane that passes no features at all.

  5. docs/release-trust.md — the /opt/rocm contradiction is gone and the install-version point is now stated plainly enough that nobody auditing it would come away believing the reported version is simulatable. Two smaller accuracy gaps left, in a comment below.

  6. The test — real, and it does the job. I did not take the "reverting X makes it fail" claim at face value; I reverted six conversions one at a time and ran it. Four fail immediately. The two WSL ones do not, and the /proc/version one needs more than the /dev/dxg caveat you already wrote down — details inline.

Still open. Three things, in descending order of how much they matter.

The first is mine as much as yours: /etc/os-release is re-rooted into the three package-manager install plans, which then run real apt/dnf/amdgpu-install against the machine. That was in the first push and I missed it; the new prose asserting the opposite is what made it visible. It is the one case where extending the seam reached something a real process acts on, which is exactly the rule the module docs state.

The second is that the new test, good as it is, still is not run by anything. No unit-test or clippy job passes the feature, so the test is not even compiled in CI. The thing I was worried about last round — that reverting a conversion compiles and passes everything automated — is still true, and now the test itself can rot unnoticed too.

The third is two small doc inaccuracies left over from the rewrite.

I also re-derived the release-build containment from scratch rather than trusting my earlier pass, since extending a seam is how that sort of property usually gets breached. It holds: the non-feature hardware_root() is a const fn returning None with no env read, hardware_root() is private, the feature is in no default list and no crate enables it transitively, nothing in the repo passes --all-features, and the release lane builds with no --features at all. The new unconditional rocm_core::host_path call in dash.rs is a pure identity call without the feature, and there is already a contract test keeping the Windows lifecycle lane off the hooks build. Nothing the five commits added reaches the seam from a shipping binary.

Comment thread apps/rocm/src/main.rs Outdated

fn read_os_release() -> Result<String> {
fs::read_to_string("/etc/os-release").context("failed to read /etc/os-release")
fs::read_to_string(rocm_core::host_path("/etc/os-release"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is at least as much mine as yours — it was in the first push, I did not flag it, and what made it visible is the new prose asserting the opposite.

read_os_release() is re-rooted, and all three of its callers build a package-manager plan that then acts on the real machine: install_driver (main.rs:3419) feeds build_driver_install_plan and executes it when --yes is given and the plan is mutating, and ensure_openmpi_for_vllm (main.rs:9834) and ensure_torch_runtime_dep (main.rs:10009) do the same for the OpenMPI and libatomic/libnuma installs that run before serving. There is no fourth caller that merely describes the host; examine's distro name comes from detect_distro_name in rocm-core, which has its own re-rooted read and is the one that belongs on the seam.

So in a hook build with a root set, rocm install driver --yes takes its distro, codename and package manager from the fixture and then runs apt or dnf or amdgpu-install against the real runner. That is precisely what the module's own rule excludes — "anything a real process acts on rather than reads to describe the machine" — and the self-hosted and nightly lanes do build release binaries with the feature on real hardware, so it is reachable as soon as a scenario sets a root.

It also turns three statements from incomplete into wrong: the release-trust.md bullet "The package-manager install hints, which read the real /etc/os-release", the same sentence at hardware_root.rs:23, and the matching line in the PR description. All three cite openmpi.rs, which is accurate for openmpi.rs's own helpers — it is these three call sites that are not covered by it.

Smallest fix is to put read_os_release back on the real file and leave the seam to detect_distro_name. A simulated distro in examine already works through that one.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and confirmed: all three callers build package-manager plans that run on the real machine. Fixed in 48755993. read_os_release reads the real /etc/os-release again, and the seam stays with detect_distro_name for the distro name examine reports. A feature-on test plants a different os-release under a root and asserts the real file comes back. Routing the read through host_path again fails that test. The three statements that were wrong (release-trust.md, the hardware_root docs and the PR description) are corrected in 68bf43aa, and the description now names all three plans. One related path: under a simulated WSL root, install_driver picks the ROCDXG plan, but its execute phase checks the real /dev/dxg and libdxcore.so before installing anything, so a bare-metal runner stops there.

/// or cgroup answers differently whether the test runs bare, under Docker
/// or under Podman. The GPU is a gfx1101, and `/proc/version` names no WSL
/// kernel.
#[cfg(all(feature = "e2e-test-hooks", target_os = "linux"))]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does what I asked for, and I checked rather than assumed — I reverted each conversion in turn and ran the test.

Caught: the container markers, the cgroup read, the KFD topology directory (both the detect_linux_kfd_gfx_target and linux_kfd_gpu_node_count wrappers), and stat_device's host_path. That is the useful half and it is genuinely useful.

Not caught: both halves of the WSL check. You already wrote down /dev/dxg. /proc/version has the same hole for a different reason, and the reason is the assertion rather than the host. assert!(!wsl) is the answer a bare-metal runner gives anyway, so an unrouted read agrees with a routed one and the assertion passes either way. It failed for you (and for me) only because the machine underneath is WSL2. I confirmed that by pointing the unrouted read at a path that does not exist — which is what a non-WSL runner looks like to this code — and the test passed with the conversion gone. Planting a WSL-naming /proc/version and asserting is_wsl_host() is true fails everywhere except a real WSL host, which is the direction that holds. Worth doing, because "a /dev/dxg that makes a bare-metal runner read as WSL" is the example the module doc leads with.

The other half is that nothing runs this. cargo nextest run and cargo test --workspace --all-targets are the only unit-test invocations in CI and neither passes the feature; cargo clippy --locked --workspace --all-targets does not either. So the test is not run, not linted, and not even compiled anywhere automated. The nightly and self-hosted lanes do compile with --features rocm/e2e-test-hooks, but they build, they do not test. Two consequences: reverting a conversion still compiles and passes everything CI does, which is the gap I raised last round unchanged; and this test can now break or rot without anyone finding out. A single cargo test -p rocm-core --features e2e-test-hooks step closes both.

Minor, while you are in here: fix.rs's current_os_reports_wsl_exactly_when_is_wsl_host_does reaches crate::is_wsl_host() twice without taking HOST_ROOT_TEST_LOCK, so with the feature on it can straddle this test's root and compare two different answers. I ran the suite with the feature three times without reproducing it, so the window is small — but it is the same race the *_TEST_LOCK convention exists for, seen from the reader's side rather than the writer's.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed in 7e00663a.

  • The test now plants a WSL kernel, then a /dev/dxg alone, and asserts that each reads as WSL. Pointing either unrouted read at a nonexistent path, as you did, now fails it.
  • CI: a new step in the affected-crates test job runs rocm-core's lib tests with e2e-test-hooks whenever rocm-core is affected. It uses nextest, like the step before it, so the test that sets the root can't leak it into another test that probes the host.
  • current_os_reports_wsl_exactly_when_is_wsl_host_does now holds HOST_ROOT_TEST_LOCK.

The one feature-on test in the rocm binary, the os-release one above, still runs only locally. Running it in CI would need a feature-on rocm build in that job, and the description says so.

Comment thread docs/release-trust.md Outdated
`/proc/1/cgroup`, the container markers `/.dockerenv` and `/run/.containerenv`,
`/etc/os-release`, the modprobe configuration directories, the WSL plumbing
under `/usr/lib/wsl`, and the WSL ROCDXG capability check, which looks for
`lib/librocdxg.so` and `share/rocdxg/dids.conf` under `/opt/rocm`. The E2E

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things the rewrite left slightly off. Both small, but this is the document people audit.

rocm_relative_file_exists (lib.rs:2628-2636) checks host_path("/opt/rocm").join(relative) and then host_path(&install.path).join(relative) for every discovered install, so the re-rooted part is not only the hardcoded /opt/rocm — a simulated root can answer the ROCDXG question for a versioned install too. As written, this sentence and the "ROCm install discovery ... always the real ones" bullet below describe a cleaner split than the code makes.

That discovered-install branch is also the one your new host_path_under doc comment names as the sole non-literal caller, so the .. precondition and this sentence are describing the same path from two directions. Worth having them say the same thing.

Second, hardware_root.rs's own routed list (lines 9-12) never mentions the ROCDXG check or /opt/rocm at all, and its exclusion paragraph carves out only install discovery. So the module doc still reads as though nothing under /opt is re-rooted — the same contradiction I flagged in the trust document, now living one file over.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 68bf43aa. release-trust.md and the hardware_root module docs now both say that discovery itself reads the real host, and that the ROCDXG check in rocm_relative_file_exists looks under /opt/rocm and under each discovered install's path inside the root. The routed list names that check, and the exclusion paragraph carves out only discovery itself. The host_path_under .. note now names that same caller, and notes that $ROCM_PATH is taken verbatim there.

@rominf
rominf dismissed volen-silo’s stale review October 5, 2026 13:05

Dismissed as stale: this review is of f7ef476. All three remaining points are addressed, with a reply in each thread: package-manager plans read the real /etc/os-release (4875599); the WSL halves of the planted-root test can now fail, the host-root tests run in CI, and the fix.rs test holds HOST_ROOT_TEST_LOCK (7e00663); and the docs match the code (68bf43a). Please re-review the current head.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Third round. Three of the four things I left open are closed, and I checked each by building the failure rather than by reading the fix.

read_os_release is off the seam, and I could not find a sibling that still is. It reads /etc/os-release directly again, and all three callers — the driver install, the OpenMPI install and the runtime-library install — build their plans from the real distro. I re-routed it through host_path locally and the new test failed on the planted ID=planted-distro, so the guard is a real tripwire and not a no-op.

I also chased the path you volunteered, since is_wsl_host() is still routed and build_driver_install_plan branches on it. Your account holds, and I checked it rather than taking it: a simulated WSL root does steer install_driver into the ROCDXG plan, but that plan's first two commands are test -e /dev/dxg and test -e /usr/lib/wsl/lib/libdxcore.so as shell text, which host_path cannot reach, and execute_driver_install_plan walks the command vector in order and propagates the first failure. Nothing mutating is reachable past those two on a machine that is not really WSL. Raising it yourself was the right call.

Both WSL halves are now caught. I rebuilt the reverts rather than trusting the new assertions. This machine is WSL2, so a plain revert fails for the wrong reason; I simulated a bare-metal runner by pointing each unrouted read at a path that does not exist, which is exactly the case the old assert!(!wsl) could not distinguish. Reverting /dev/dxg now fails on "a planted /dev/dxg reads as WSL", and reverting /proc/version fails on "a planted WSL kernel reads as WSL". That is the direction that holds everywhere except a real WSL host.

CI now runs them. I worked the new step's condition against every selection the affected-crates tool can emit — --workspace, a lone -p rocm-core, rocm-core mid-list, and the empty selection — and the trailing-space anchoring on both sides also rules out a false match if a crate named rocm-core-something ever appears. The feature is an empty feature list, so --locked stays valid, and nextest is installed earlier in the same job. Using nextest for process isolation is the right instinct for a test that sets a process-wide root.

The documents now match the code. rocm_relative_file_exists re-rooting both /opt/rocm and each discovered install is stated in the trust document, in the module doc and in the host_path_under precondition, and the trust document's routed list is complete against every host_path/read_host_text call in the tree. The PR description agrees with both.

Containment re-derived from scratch, and it holds. Not carried over from last round: without the feature hardware_root() is a const fn returning None, which is stronger than a convention — env::var_os is not const, so that body structurally cannot grow an environment read. The function is private in both variants, no crate declares a default feature at all, no dependency edge force-enables it, there is no --all-features anywhere in the workflows or xtask, and every job that produces a shipped artifact builds with no --features. With no root the path is returned by to_path_buf() on the catch-all arm regardless of shape, so the default really is byte-for-byte unchanged. The one runtime-derived path into host_path is still the discovered install path, and it is absolute in every branch that reaches it. I had not noticed before that xtask's workflow-contract tests already assert the Windows lifecycle lane does not carry the feature — that is a machine check on the property, not just a comment, and it is a good thing to have.

One thing did not close, and there are two residuals worth a sentence. None of them is serious, and I would not hold the branch for any of them on its own merits — the request-changes state is procedural, since something survives. Details inline.

For what it is worth across three rounds: the seam ended up narrower and better evidenced than it started, and most of that came from you reverting things and reporting what did not fail.

Comment thread crates/rocm-core/src/hardware_root.rs Outdated
//! distro name, the WSL plumbing under `/usr/lib/wsl`, the WSL ROCDXG check
//! (`lib/librocdxg.so` and `share/rocdxg/dids.conf`, looked for under
//! `/opt/rocm` and under each discovered ROCm install), the container markers
//! (`/.dockerenv`, `/run/.containerenv`, `/proc/1/cgroup`) — resolve their path

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one that did not close, and it is a smaller version of what I flagged last round rather than a new problem.

The ROCDXG and /opt/rocm gap is fixed — the list names the check now, and the exclusion paragraph below carves out discovery itself rather than everything under /opt. But the list still reads as the enumeration ("The probes that decide what GPU, driver and platform the CLI is running on — ... — resolve their path through host_path"), and several routed reads are not in it:

  • /proc/meminfo (lib.rs:2506)
  • /proc/cmdline (examine.rs:617)
  • /proc/modules (examine.rs:1601)
  • /sys/module/amdgpu and /sys/module/amdgpu/version (lib.rs:2743, lib.rs:2775, and passive_path_check in main.rs:3650)
  • the modprobe configuration directories (examine.rs:1613-1614)

All five are in release-trust.md's list and in the PR description, both of which I checked against every host_path/read_host_text call site and found complete. So the two documents now describe different sets, with this one being the short one — the same shape of defect I raised about the trust document, just surviving one file over after the rewrite fixed the other.

Not a contradiction this time: nothing here claims those paths are not routed, and the exclusion paragraph does not mention them either. That is why I would not block on it. But this list is the thing a reader reaches for first when they want to know what the seam covers, and it is the same omission as in the original push, so it is worth bringing level with the trust document while you are here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brought level in 9ddd114. The module doc is now an explicit bulleted list, rebuilt from every host_path/read_host_text call site after rebasing onto current main. It now includes /proc/meminfo, /proc/cmdline, /proc/modules (the lsmod fallback), /sys/module/amdgpu and its version file, and the three modprobe directories by name.

Going through the call sites also turned up two reads that none of the three documents named exactly: the /dev/dri render nodes, and /usr/lib/wsl/lib/libdxcore.so next to the /usr/lib/wsl/lib directory check. release-trust.md and the description now name them too, so all three list the same set.

Comment thread crates/rocm-core/src/hardware_root.rs Outdated
/// after the leading `/` would walk back out of the root. This is a documented
/// precondition rather than a check: callers pass fixed literals, apart from
/// the ROCDXG check in `rocm_relative_file_exists`, which re-roots each
/// discovered ROCm install path (a `$ROCM_PATH` is taken verbatim there).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now names the right caller, and the $ROCM_PATH parenthetical is accurate: discovery wraps the variable in a PathBuf unmodified, so it reaches host_path exactly as the user wrote it.

One thing the parenthetical arguably undersells, since it sits inside a note about ... If $ROCM_PATH is relative, the consequence is not a walk out of the root — strip_prefix("/") fails and the match falls to the catch-all arm, so the path is returned unchanged and the ROCDXG check reads the real filesystem relative to the process's working directory, with the simulated root silently not applied. That is consistent with the documented "a relative path is returned unchanged" rule just above, so the code is doing what it says; it is only that .. and "relative" are two different outcomes and the reader is pointed at the first one.

Purely a test-fidelity nuance, not a release one — in a shipping build the root is None and both arms are the same arm. Worth half a sentence here only because this is the one call site where the input is not a literal.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, those are two different outcomes. 9ddd114 adds the half sentence: a relative $ROCM_PATH fails strip_prefix("/") and the catch-all arm returns it unchanged. The simulated root then silently does not apply, and that ROCDXG lookup reads relative to the working directory. That is separate from .. walking out of the root. The code is unchanged, since it follows the documented relative-path rule.

Comment thread .github/workflows/ci.yml
needs.changes.outputs.rust == 'true'
&& (steps.sel.outputs.sel == '--workspace'
|| contains(format('{0} ', steps.sel.outputs.sel), '-p rocm-core '))
run: cargo nextest run --locked -p rocm-core --lib --features e2e-test-hooks

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This closes the main half of what I raised: the rocm-core tests are now compiled and run in CI, so reverting a conversion no longer passes everything automated. I checked the condition against every selection the affected-crates tool can produce and it behaves correctly in all of them, including the empty selection (skipped, consistent with the step above) and a hypothetical rocm-core-prefixed crate name (correctly not matched).

Two residuals, both acknowledged rather than hidden, so I am noting them rather than asking for them:

The rocm-binary test is still never compiled or run anywhere automated. That matters a little more than the count suggests, because it is the guard for the one thing in this PR I thought was genuinely risky — a re-routed read_os_release would still pass every CI job. Your description says so plainly, which is the right handling given the cost is a feature-on rocm build in that job. If that build is ever cheap enough, this is the test to add.

And no clippy invocation enables the feature, so the feature-gated code is compiled by this step but never linted. Smaller than it was — "not run, not compiled" is gone and only "not linted" remains — and the comment above is accurate in saying so.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged; both residuals are as you describe, re-checked against ci.yml after the rebase.

  • The read_os_release guard in the rocm binary stays local-only. Running it in CI needs a second, feature-on build of the whole rocm binary in the affected-crates job, which one test does not justify. The self-hosted E2E lanes build rocm with --features rocm/e2e-test-hooks, so the feature-gated product code is type-checked there, but not the #[cfg(test)] test.
  • Linting: neither CI clippy invocation enables the feature, so that code is compiled but not linted. I ran cargo clippy --workspace --all-targets --features rocm/e2e-test-hooks -- -D warnings locally on the rebased head, and it is clean.

If the feature-on rocm build ever gets cheap, this test is the first to add.

Comment thread apps/rocm/src/main.rs Outdated
.expect("write planted os-release");

let read = {
let mut env = ScopedTestEnv::new();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, and the mirror image of the fix.rs lock you added this round, so worth knowing about even if the answer is to leave it.

ScopedTestEnv::new() takes a process-wide mutex, so this test is serialised against every other test in this binary that touches the environment. It is not serialised against a test that merely reads a routed path while this one holds the root — which is the reader-side exposure you just fixed in fix.rs by taking HOST_ROOT_TEST_LOCK. There is no equivalent move available here: that lock is #[cfg(test)] pub(crate) to rocm-core, so it does not exist in this crate's test binary at all.

The window is one file read, the planted root contains only etc/os-release so any other probe would read it as a machine with nothing present, and this test does not run in CI anyway — so the blast radius is a local feature-on run. I would not restructure anything for it. Flagging it mostly because ScopedTestEnv's own doc comment is explicit that the hazard is a mutation racing a read, and this is the first user of it that plants something the rest of the probe surface reads.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving it as is, for the reasons you give.

  • HOST_ROOT_TEST_LOCK is #[cfg(test)] pub(crate) in rocm-core, so it does not exist in this binary. Exporting it would put a test-only symbol into rocm-core's API to close a one-read window.
  • ScopedTestEnv already serializes this test against every env-touching test here.
  • The planted root holds only etc/os-release, so an overlapping probe would see an empty machine, not a believable wrong one.

If more tests in this binary start planting roots, the right fix is a shared host-root guard exported behind the feature.

@rominf
rominf force-pushed the test/e2e-host-root branch from 68bf43a to 9ddd114 Compare October 6, 2026 09:23
@rominf

rominf commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main at 975dd2c8 (now 9ddd114d). There were two conflicts, both in crates/rocm-core/src/lib.rs:

  • The pub use fix::{…} re-export. Kept main's new catalog-manifest exports next to pub use hardware_root::host_path.
  • ldconfig_cache. Main made it pub(crate) for the new code-object-manager probe in examine.rs. It keeps that, together with this PR's unrouted body and its doc.

I checked main's new probe for host reads: its /opt sibling scan, managed-runtime walk and ldconfig -p are install discovery or executed programs, so under the seam's rules they stay on the real host. There are no scenario id or expectation collisions.

On top of the rebase, 9ddd114d brings the hardware_root module doc's list level with docs/release-trust.md and adds the relative-$ROCM_PATH note.

Re-ran on the rebased head, all clean:

  • rocm-core --lib, with and without e2e-test-hooks
  • rocm --bin rocm, rocm-dash-collectors, xtask and e2e-cucumber
  • all three clippy invocations, including the feature-on one
  • fmt and xtask manifest --check

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fourth pass, on 9ddd114d. Both points from last round are closed, and I re-derived the release-containment argument from scratch on the rebased tree rather than carrying the earlier conclusion forward. Nothing blocking survives. What is left below is three small things plus one fact about the merge state.

The routed-read list is now right. I rebuilt the ground truth by grepping every host_path / host_path_under / read_host_text call site in the workspace, 21 distinct path groups, and checked it against the module doc and the trust document in both directions. No path is routed in code and missing from a list, and no path is named in a list without a matching call site. The two documents and the PR description now describe the same set, including the /dev/dri render nodes and libdxcore.so that none of them named before. The relative-$ROCM_PATH half-sentence is accurate: discovery wraps the variable unmodified, strip_prefix("/") fails, and the catch-all arm returns it as written.

Containment still holds, re-derived rather than remembered. hardware_root() without the feature is a const fn returning None, so it is structurally incapable of reading the environment, and both variants are private. No crate in the workspace declares a default feature at all, nothing force-enables e2e-test-hooks through a dependency edge, and there is no --all-features anywhere in the repo. The release and nightly release builds run cargo build --release -p rocm -p rocmd -p xtask with no --features, so the shipping artifact never carries it. With no root the catch-all arm returns path.to_path_buf(), so the default is byte-for-byte what it was, and every join is guarded by strip_prefix("/"), so an absolute argument cannot replace the root. The one site that joins a caller-supplied path, the ROCDXG check, joins only its two relative literals. The amd-smi pre-flight collapses to the single device_accessible call it always was once the two paths are equal, which the cfg(not(...)) test pins.

The new CI step is real and it ran. On this head it executed cargo nextest run --locked -p rocm-core --lib --features e2e-test-hooks over 552 tests, including converted_probes_read_the_simulated_host_root and the hook-build identity test. I checked the guard condition against every selection the affected-crates tool can emit: the empty selection skips, no workspace crate name collides on the rocm-core prefix, and because any revert of a conversion is itself a change under crates/rocm-core/, the selection always contains the crate, so the step cannot be dodged by the very change it exists to catch.

Your two acknowledged residuals are exactly as you describe and I am not re-raising them: the rocm-binary os-release test is still never compiled anywhere automated (the GPU lanes cargo build the binary, which does not compile cfg(test) code), and no clippy invocation enables the feature.

Separately, as a fact rather than a review point: the branch now conflicts with main in two files, apps/rocm/src/dash.rs and docs/release-trust.md. Both trace to a single commit that landed on main after your rebase base, the one that moved the dash test clock out of a compiled-in seam. In dash.rs it is the field you added beside test_clock_offset_path in runner_options; in the trust document it is a new section landing in the same place as yours. Nothing for me to do about it, just flagging it.

Comment thread .github/workflows/ci.yml
needs.changes.outputs.rust == 'true'
&& (steps.sel.outputs.sel == '--workspace'
|| contains(format('{0} ', steps.sel.outputs.sel), '-p rocm-core '))
run: cargo nextest run --locked -p rocm-core --lib --features e2e-test-hooks

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step closes the gap it was written for, and I confirmed it ran on this head. One thing about its depth, which I would leave as-is but would not want unrecorded.

The test it enables, converted_probes_read_the_simulated_host_root, plants and asserts on five probe entry points: the KFD node count and gfx target, stat_device("/dev/kfd"), is_wsl_host in all three directions, and probe_container twice. That is five of the twenty-one routed path groups. The other sixteen have no planted-root assertion anywhere, so reverting the conversion on /proc/meminfo, /proc/cpuinfo, /proc/cmdline, /proc/modules, /sys/module/amdgpu, /sys/class/drm, the /dev/dri render nodes, the modprobe directories, detect_distro_name's /etc/os-release, the /usr/lib/wsl/lib check or the ROCDXG lookup would still pass every automated job, exactly as it would before this step existed.

That is the same shape as the gap you just closed, one level narrower: "not compiled" has become "compiled, partially exercised". I am not asking for sixteen more assertions here, and the follow-up's simulated-machine scenarios are the natural place for breadth. But the comment above reads as though the step covers the conversions generally, and it covers about a quarter of them, so it is worth a clause saying which.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and thanks for checking it ran. The comment now says the step is a sample, not full coverage.

It names what the planted-root test asserts:

  • the KFD node count and gfx target
  • /dev/kfd, via stat_device
  • /proc/version and /dev/dxg, via is_wsl_host
  • /run/.containerenv and /proc/1/cgroup, via probe_container

It also lists the routed reads with no unit assertion: /dev/dri, /sys/class/drm, /sys/module/amdgpu, the other /proc files, the modprobe directories, /etc/os-release, and the /usr/lib/wsl/lib and ROCDXG checks. Those are covered end to end only by #517's simulated-machine scenarios. Assertions left as they are, as you suggested. Fixed in c766a25.

Comment thread docs/release-trust.md
re-rooted read under them is the ROCDXG file check above, which looks for its
two files under each discovered install's path inside the simulated root.
- The `/usr/lib/wsl/lib` loader entry an engine is launched with, and programs
the probes execute, such as `ldconfig`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small asymmetry, and the mirror image of the one that just got fixed in the other direction.

The module doc's exclusions paragraph names /proc/<pid> liveness and /dev/shm sizing as deliberately unrouted. This list does not: it stops at install discovery, the loader entry and executed programs, and the package-manager plans. I confirmed both are genuinely unrouted (/dev/shm is measured with a raw statvfs on the literal path, so it never reaches host_path at all), so nothing here is wrong. The trust document is simply the shorter of the two again, on the exclusions side this time rather than the routed side.

Not worth a round on its own. Worth two lines while the file is open, given the point of the rewrite was to stop these two drifting apart.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in c766a25: the exclusions list now has a bullet for /proc/<pid> liveness and /dev/shm sizing, matching the hardware_root.rs module doc. I re-checked that both stay unrouted. /proc/{pid}/stat and /proc/{pid}/cgroup are read on the literal path, and /dev/shm goes through SHM_PATH into a raw statvfs. Neither reaches host_path.

Comment thread docs/release-trust.md
environment entirely and unconditionally returns the hardcoded default URL, so
a stray environment variable can never redirect a production install.

## Simulated Host Root

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not about this PR's correctness, but about what the document will say once it merges.

Main has just gained a ## Dashboard Test Clock File section that lands immediately before this one, and it argues the opposite case in so many words: the test clock reads from a state file rather than a compiled-out feature so the suite drives "the same binary that ships, rather than one built with a test-only feature". Yours argues that being compiled out is what makes the seam safe.

Both are right for their own seam, and main's section already carries the principle that separates them: the clock "can skew telemetry but cannot select code or redirect a download", whereas a re-rooted hardware probe decides which package-manager plan runs against the real machine. Since you have to resolve a conflict in this file anyway, one sentence connecting the two would stop the trust document reading as though it endorses both "compiled out is safest" and "the same binary that ships is safest" three paragraphs apart with nothing in between.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. While resolving the conflict I kept both sections and added one sentence here linking this section to the test-clock section (c766a25). It says the clock is read from a state file by the binary that ships, while this seam is compiled out. The reason is the one that section already gives: the clock can skew telemetry but cannot select code or redirect a download. A re-rooted hardware probe decides which plan runs against the real machine, so it must not exist in a shipped build.

@rominf
rominf force-pushed the test/e2e-host-root branch from 9ddd114 to c766a25 Compare October 6, 2026 12:26
@rominf

rominf commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto 4ee61616 (now c766a256), which includes #466. Both conflicts came from #466:

  • apps/rocm/src/dash.rs: kept amd_smi_host_kfd_device beside the clock field, which now takes dash_test_clock_offset_path(paths).
  • docs/release-trust.md: kept main's Dashboard Test Clock File section and this PR's Simulated Host Root section, in that order.

#466 also dropped the dash clock's e2e-test-hooks branch. Nothing here counted on it.

The round-4 items are in c766a256. 9ddd114d is now de54e7cf, with the same content, reworded to drop a co-author trailer that does not belong in this repo's history.

Re-ran on the rebased head, all clean:

  • rocm-core --lib (578 passed; 579 with e2e-test-hooks), xtask (271) and rocm --bin rocm (1015)
  • e2e-cucumber lib, feature_naming and --test e2e check, plus rocm-dash-collectors
  • check-crate-edges, the three clippy invocations, fmt and manifest --check

@rominf
rominf dismissed stale reviews from volen-silo and volen-silo October 7, 2026 06:58

Stale (left on 68bf43a); addressed. The routed-read list is complete and the relative-ROCM_PATH note added: de54e7c. The CI residuals and the test-lock note are answered in their threads.

@rominf
rominf force-pushed the test/e2e-host-root branch from c766a25 to 3388ba7 Compare October 7, 2026 12:47
@rominf

rominf commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto b863f77a to clear the conflict with #528 (ROCMAI-85), which moved the host/GPU detection surface out of crates/rocm-core/src/lib.rs into the new crates/rocm-core/src/host_gpu.rs.

All 18 routed probes this PR adds to lib.rs are now in host_gpu.rs, re-applied by hand. The rebase carried none of them across: git surfaced the move as two large "deleted here / modified there" conflicts, and resolving them toward main — which is the correct post-move lib.rs — drops every routing while leaving the module, the docs and the tests looking intact.

Measured what review would have had to catch: with the routing dropped, 586 of 587 rocm-core tests still pass. Only converted_probes_read_the_simulated_host_root's GPU-node assertion fails. That is the sampling gap the new CI step's comment already describes, now with a number on it.

I verified the re-application mechanically rather than by eye: the routed call-site inventory is unchanged at 34 (18 moved lib.rs → host_gpu.rs; examine.rs 13, main.rs 2, dash.rs 1), and the set of routed path literals is identical before and after. I also swept every routed literal for a production read that is not wrapped in host_path/read_host_text; the only hits are display labels, catalog strings and test assertions, plus passive_path_check, which routes internally.

Three changes beyond pure relocation, all forced by the move:

Without the second and third, cargo test -p rocm-core --lib --features e2e-test-hooks does not compile, while the plain --lib build stays green, since the test is feature-gated. Folded into the commit that owns the test, so no commit in the stack is broken.

hardware_root.rs's routed list and docs/release-trust.md needed no edits: both describe paths and name dash.rs/examine.rs/openmpi.rs, never lib.rs. I re-checked every path in both lists against the post-rebase tree.

Local gates green: fmt; rocm-core --lib 586, and 587 with e2e-test-hooks; xtask 271; e2e-cucumber --lib 136; feature_naming 4; cargo check --test e2e; rocm-dash-collectors 108; rocm --bin rocm 1014; all three clippy invocations; check-crate-edges; manifest --check.

One pre-existing gap noticed while sweeping, not touched here: apps/rocm/src/main.rs reads /sys/class/drm unrouted for dash VRAM telemetry. It is byte-identical before the rebase, so not a regression, but /sys/class/drm is on the routed list. Happy to route it here or in a follow-up, whichever you prefer.

rominf added 9 commits October 9, 2026 06:51
The E2E suite can only exercise GPU behaviour on the self-hosted lanes,
because every hardware probe reads fixed host paths: /dev/kfd, the KFD
topology, /sys/module/amdgpu, /dev/dxg, /proc/version, /etc/os-release
and /usr/lib/wsl. A scenario cannot describe a host it is not running on.

Route those reads through rocm_core::host_path. In a build with the new
rocm-core e2e-test-hooks feature it re-roots absolute paths under
ROCM_CLI_TEST_HOST_ROOT; without the feature it is the identity and never
reads the environment, so a release build always probes the real machine.
The CLI keeps printing the logical path, so the root changes what the
probes find, not what users read.

The feature is forwarded from rocm, rocmd and both engines so the
existing `--features rocm/e2e-test-hooks` build turns it on everywhere.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
ldconfig_cache executes the program it names, so routing its candidates
through host_path pointed the absolute fallbacks at a fixture root that
holds no linker binary. On Debian-style hosts, where ldconfig is not on a
non-root PATH, the probe then fell through to "could not ask" and the
ROCDXG verdict changed under simulation; where ldconfig is on PATH the
bare name was never re-rooted anyway. Executed programs belong with the
paths the hardware_root module already leaves on the real host.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
probe_container sits in the same Linux probe sequence as the probes
already routed through host_path, and its answer feeds the diagnose
scoring directly, yet it still read /.dockerenv, /run/.containerenv and
/proc/1/cgroup from the real machine. A simulated bare-metal host
therefore picked up a containerised runner's real containerisation.

Route all three, and list them in the hardware_root module docs next to
the other routed probes. The exclusion list now also names executed
programs such as ldconfig and the install hints that still read the
real /etc/os-release.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The hardware_root tests pinned host_path's path algebra but nothing
downstream of it, so reverting any converted probe to a raw literal
compiled and passed. Plant a host root holding a KFD topology with one
GPU node, a /dev/kfd, a non-WSL /proc/version and container markers,
run the real probes with the e2e-test-hooks feature on, and assert they
describe the planted machine. The container is planted once by marker
and once by cgroup alone, so the test fails whether the runner is bare,
under Docker or under Podman.

The tests that set the root now share one lock, and host_path_under
documents that its input must not contain "..".

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The simulated-host-root list named /opt/rocm as re-rootable while the
paragraph below said discovered ROCm installs are never re-rooted. Only
the WSL ROCDXG capability check's file lookups are re-rooted; install
discovery, and the version examine reports, always read the real host.
Name that check instead of the bare path, add the container markers, and
list what stays on the real machine: install discovery, the WSL loader
entry and executed programs such as ldconfig, and the install hints that
read the real /etc/os-release.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The rocm dash amd-smi pre-flight opened the literal /dev/kfd, so on a
simulated GPU-less host the dashboard still found the runner's real GPU
while every other probe reported none. Re-rooting that path alone would
be unsafe: the check decides whether a real amd-smi process may start,
and a planted /dev/kfd on a machine without one would launch amd-smi
into the uninterruptible hang the check exists to prevent.

Thread the host-root view of /dev/kfd from apps/rocm (which resolves it
through rocm_core::host_path) into RunnerOptions, the detect plan and the
collector, and run amd-smi only when both it and the real /dev/kfd are
readable. A simulated host can now hide the GPU but never start amd-smi
on a host without one. Without a root the two paths are equal and the
second open is skipped, so the check is unchanged. The WSL reachability
verdict still substitutes for the device check as before. The dash crates
keep no rocm-core dependency.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
read_os_release was routed through host_path, but all three callers --
the driver install, and the OpenMPI and runtime-library installs before
serving -- build apt/dnf/amdgpu-install plans that run against the real
machine. Under a simulated host root, `rocm install driver --yes` would
take its distro from the fixture and then run the resulting commands on
the runner, which is exactly what the hardware_root rule excludes.

Read the real file again. The descriptive distro name examine reports
comes from rocm_core::detect_distro_name, which keeps following the root.
A feature-on test plants a different os-release under a root and checks
read_os_release still returns the real one.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The planted-root test only asserted that a non-WSL root reads as "not
WSL", which a bare-metal runner answers anyway, so an unrouted
/proc/version or /dev/dxg read passed everywhere but a real WSL host.
Plant a WSL kernel, then a /dev/dxg alone, and assert both read as WSL.

No CI job enabled e2e-test-hooks for unit tests, so these tests were not
even compiled. Add a step to the affected-crates test job that runs
rocm-core's lib tests with the feature whenever rocm-core is in the
selection. It uses nextest like the step before it, so the test that
sets the root cannot leak it into another test probing the host.

fix.rs's is_wsl_host comparison now holds the host-root lock, so in a
hook build it cannot compare answers from either side of a root change.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The ROCDXG check in rocm_relative_file_exists re-roots /opt/rocm and
every discovered install path, so "install discovery is always real"
was a cleaner split than the code makes. Say that discovery itself is
real and that this one existence check looks under the discovered paths
inside the root, in both the hardware_root module docs and
release-trust.md, and make the host_path_under ".." note describe the
same caller.

Also correct the /etc/os-release statements: the reported distro name
follows the root, and every package-manager plan -- the OpenMPI install
hints and the rocm binary's driver, OpenMPI and runtime-library installs
-- reads the real file.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit 2068d58 Oct 12, 2026
31 of 32 checks passed
@rominf
rominf deleted the test/e2e-host-root branch October 12, 2026 09:55
r0x0r added a commit that referenced this pull request Oct 12, 2026
…nobs-note

Resolved apps/rocm/Cargo.toml: main (#510) widened `e2e-test-hooks` to pull
in `rocm-core` and `rocm-engine-vllm`; this branch added the separate
`e2e-oom-fault-injection` feature. Kept both.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@rominf

rominf commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

Following up the non-blocking note in the approval: #642 adds to the Device struct and its path field that the path is always the logical host path, never the re-rooted one, so a consumer of rocm examine --json sees it on the serialized type.

rominf added a commit that referenced this pull request Oct 13, 2026
Four places parsed `/etc/os-release` — the driver plan (which also picks the
distro for the package installs it approves in `ensure_openmpi_for_vllm` and
`ensure_torch_runtime_dep`), `rocm examine`, the OpenMPI install hint, and
the `PRETTY_NAME` reader — each with its own `trim_matches`. Against `sh`
sourcing the same file, they disagreed with each other:

  file                      sh       driver plan  rocm examine  OpenMPI hint
  ID='ubuntu'               ubuntu   'ubuntu'     'ubuntu'      ubuntu
  ID=debian, then ID=ubuntu ubuntu   debian       ubuntu        debian

`os-release(5)` allows single quotes, so a spec-valid single-quoted file read
as an unsupported distro. And on a file with a duplicated `ID`,
`rocm examine` reported Ubuntu while the driver plan was built for Debian 12.

All four now read through `rocm_core::os_release`, which keeps one rule: the
value `sh` would assign, or nothing. For a file of blank lines, comments and
spec-valid assignments it returns `sh`'s value — one matching quote pair
removed, `\"`, `\\`, `` \` `` and `\$` decoded inside double quotes, nothing
decoded inside single quotes, `\x` meaning `x` in a bare value, a `#` comment
after a value, the last assignment of a key winning.

Any other line makes the whole file unreadable, not just the key it names.
To `sh` such a line is a command or the start of one — `export ID=…` and
`X=1; ID=…` assign, `unset ID` removes, an unterminated quote swallows the
lines after it — and a command can change any variable, including ones set
before it. Reading past it would mean reporting values `sh` does not assign.
A CRLF file is unreadable too: `sh` assigns `ubuntu\r`, which matches no
distro.

An unreadable file is reported, not hidden. `parse` names the first
offending line, and the driver plan's reason becomes `/etc/os-release line N
is not a plain assignment: "<line>"; no driver commands were planned because
the distro cannot be read.` instead of "this distro is not supported", while
`rocm examine` records the same line in `probe_failures`.

Behaviour changes:
- `rocm examine` reports single-quoted and duplicated keys as `sh` reads
  them, and an unreadable file as a probe failure naming its line.
- The driver plan, and the package installs it approves, read the same
  distro `rocm examine` reports.
- The OpenMPI hint no longer accepts `ID =x`; to `sh` that runs a command
  named `ID`.

Real files: run against `dash`, all 50 container images checked and 396 of
the 397 files in which-distro/os-release (10dfd5bc) read exactly as `dash`
reads them; the one exception is a CRLF file, which reads as unreadable.
No file read a value `dash` does not assign.

Tests:
- Against `/bin/sh` itself (empty environment, a sentinel `HOME`, a `PATH`
  that does not exist): every spec-valid encoding of every value up to three
  characters, and every right-hand side up to three characters over a
  quote-, backslash- and newline-heavy alphabet, checked for the key it sets
  and a key after it. Each must read as `sh` reads it or not at all, a list
  of well-formed cases must read as a value, and no earlier value may come
  back.
- Nine mutations of the parser's rules each turn the suite red.
- End to end through the driver plan: every supported distro rewritten in
  single quotes plans exactly as in double quotes; a duplicated `ID` plans
  from its last assignment; an unreadable file is refused with the message
  above and no commands. `rocm examine`'s probe failure is asserted with its
  state. These fail on the code before this change.

No e2e scenario in this commit: a later commit in this change plants
/etc/os-release under the simulated host root #510 added, and runs
`rocm examine --json` against it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
`rocm examine` reads /etc/os-release through `rocm_core::host_path`
since #510, so a scenario can describe a machine other than the runner.
Three scenarios run `rocm examine --json` against a planted file:

- examine-20: every value single-quoted reads as RHEL 9.4.
- examine-21: a file naming Debian 12 and then Rocky Linux 9.5 reads as
  Rocky, the last assignment, for the distro fields and the name alike.
- examine-22: a file with `export ID=debian` on line 3 reports no
  distro, and names the line both in `probe_failures` and in the
  distribution name a person reads; message and state in one scenario.

The planted distros are ones no runner here has, so a probe that read
the real file would report a different one. The driver plan is not
covered this way: it builds commands for the real machine, so it reads
the real file by design.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
The planted-root test from #510 covers the KFD, WSL and container probes
but not /etc/os-release, so reverting either of `rocm examine`'s two
os-release reads (the distro fields and the distro name) to the real
path left every unit test green; only the e2e scenarios noticed. Plant
a single-quoted file naming a distro no runner has, and assert both
reads report it, which also fails if either read skips the shared
parser.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
Four places parsed `/etc/os-release` — the driver plan (which also picks the
distro for the package installs it approves in `ensure_openmpi_for_vllm` and
`ensure_torch_runtime_dep`), `rocm examine`, the OpenMPI install hint, and
the `PRETTY_NAME` reader — each with its own `trim_matches`. Against `sh`
sourcing the same file, they disagreed with each other:

  file                      sh       driver plan  rocm examine  OpenMPI hint
  ID='ubuntu'               ubuntu   'ubuntu'     'ubuntu'      ubuntu
  ID=debian, then ID=ubuntu ubuntu   debian       ubuntu        debian

`os-release(5)` allows single quotes, so a spec-valid single-quoted file read
as an unsupported distro. And on a file with a duplicated `ID`,
`rocm examine` reported Ubuntu while the driver plan was built for Debian 12.

All four now read through `rocm_core::os_release`, which keeps one rule: the
value `sh` would assign, or nothing. For a file of blank lines, comments and
spec-valid assignments it returns `sh`'s value — one matching quote pair
removed, `\"`, `\\`, `` \` `` and `\$` decoded inside double quotes, nothing
decoded inside single quotes, `\x` meaning `x` in a bare value, a `#` comment
after a value, the last assignment of a key winning.

Any other line makes the whole file unreadable, not just the key it names.
To `sh` such a line is a command or the start of one — `export ID=…` and
`X=1; ID=…` assign, `unset ID` removes, an unterminated quote swallows the
lines after it — and a command can change any variable, including ones set
before it. Reading past it would mean reporting values `sh` does not assign.
A CRLF file is unreadable too: `sh` assigns `ubuntu\r`, which matches no
distro.

An unreadable file is reported, not hidden. `parse` names the first
offending line, and the driver plan's reason becomes `/etc/os-release line N
is not a plain assignment: "<line>"; no driver commands were planned because
the distro cannot be read.` instead of "this distro is not supported", while
`rocm examine` records the same line in `probe_failures`.

Behaviour changes:
- `rocm examine` reports single-quoted and duplicated keys as `sh` reads
  them, and an unreadable file as a probe failure naming its line.
- The driver plan, and the package installs it approves, read the same
  distro `rocm examine` reports.
- The OpenMPI hint no longer accepts `ID =x`; to `sh` that runs a command
  named `ID`.

Real files: run against `dash`, all 50 container images checked and 396 of
the 397 files in which-distro/os-release (10dfd5bc) read exactly as `dash`
reads them; the one exception is a CRLF file, which reads as unreadable.
No file read a value `dash` does not assign.

Tests:
- Against `/bin/sh` itself (empty environment, a sentinel `HOME`, a `PATH`
  that does not exist): every spec-valid encoding of every value up to three
  characters, and every right-hand side up to three characters over a
  quote-, backslash- and newline-heavy alphabet, checked for the key it sets
  and a key after it. Each must read as `sh` reads it or not at all, a list
  of well-formed cases must read as a value, and no earlier value may come
  back.
- Nine mutations of the parser's rules each turn the suite red.
- End to end through the driver plan: every supported distro rewritten in
  single quotes plans exactly as in double quotes; a duplicated `ID` plans
  from its last assignment; an unreadable file is refused with the message
  above and no commands. `rocm examine`'s probe failure is asserted with its
  state. These fail on the code before this change.

No e2e scenario in this commit: a later commit in this change plants
/etc/os-release under the simulated host root #510 added, and runs
`rocm examine --json` against it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
`rocm examine` reads /etc/os-release through `rocm_core::host_path`
since #510, so a scenario can describe a machine other than the runner.
Three scenarios run `rocm examine --json` against a planted file:

- examine-20: every value single-quoted reads as RHEL 9.4.
- examine-21: a file naming Debian 12 and then Rocky Linux 9.5 reads as
  Rocky, the last assignment, for the distro fields and the name alike.
- examine-22: a file with `export ID=debian` on line 3 reports no
  distro, and names the line both in `probe_failures` and in the
  distribution name a person reads; message and state in one scenario.

The planted distros are ones no runner here has, so a probe that read
the real file would report a different one. The driver plan is not
covered this way: it builds commands for the real machine, so it reads
the real file by design.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
The planted-root test from #510 covers the KFD, WSL and container probes
but not /etc/os-release, so reverting either of `rocm examine`'s two
os-release reads (the distro fields and the distro name) to the real
path left every unit test green; only the e2e scenarios noticed. Plant
a single-quoted file naming a distro no runner has, and assert both
reads report it, which also fails if either read skips the shared
parser.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
Four places parsed `/etc/os-release` — the driver plan (which also picks the
distro for the package installs it approves in `ensure_openmpi_for_vllm` and
`ensure_torch_runtime_dep`), `rocm examine`, the OpenMPI install hint, and
the `PRETTY_NAME` reader — each with its own `trim_matches`. Against `sh`
sourcing the same file, they disagreed with each other:

  file                      sh       driver plan  rocm examine  OpenMPI hint
  ID='ubuntu'               ubuntu   'ubuntu'     'ubuntu'      ubuntu
  ID=debian, then ID=ubuntu ubuntu   debian       ubuntu        debian

`os-release(5)` allows single quotes, so a spec-valid single-quoted file read
as an unsupported distro. And on a file with a duplicated `ID`,
`rocm examine` reported Ubuntu while the driver plan was built for Debian 12.

All four now read through `rocm_core::os_release`, which keeps one rule: the
value `sh` would assign, or nothing. For a file of blank lines, comments and
spec-valid assignments it returns `sh`'s value — one matching quote pair
removed, `\"`, `\\`, `` \` `` and `\$` decoded inside double quotes, nothing
decoded inside single quotes, `\x` meaning `x` in a bare value, a `#` comment
after a value, the last assignment of a key winning.

Any other line makes the whole file unreadable, not just the key it names.
To `sh` such a line is a command or the start of one — `export ID=…` and
`X=1; ID=…` assign, `unset ID` removes, an unterminated quote swallows the
lines after it — and a command can change any variable, including ones set
before it. Reading past it would mean reporting values `sh` does not assign.
A CRLF file is unreadable too: `sh` assigns `ubuntu\r`, which matches no
distro.

An unreadable file is reported, not hidden. `parse` names the first
offending line, and the driver plan's reason becomes `/etc/os-release line N
is not a plain assignment: "<line>"; no driver commands were planned because
the distro cannot be read.` instead of "this distro is not supported", while
`rocm examine` records the same line in `probe_failures`.

Behaviour changes:
- `rocm examine` reports single-quoted and duplicated keys as `sh` reads
  them, and an unreadable file as a probe failure naming its line.
- The driver plan, and the package installs it approves, read the same
  distro `rocm examine` reports.
- The OpenMPI hint no longer accepts `ID =x`; to `sh` that runs a command
  named `ID`.

Real files: run against `dash`, all 50 container images checked and 396 of
the 397 files in which-distro/os-release (10dfd5bc) read exactly as `dash`
reads them; the one exception is a CRLF file, which reads as unreadable.
No file read a value `dash` does not assign.

Tests:
- Against `/bin/sh` itself (empty environment, a sentinel `HOME`, a `PATH`
  that does not exist): every spec-valid encoding of every value up to three
  characters, and every right-hand side up to three characters over a
  quote-, backslash- and newline-heavy alphabet, checked for the key it sets
  and a key after it. Each must read as `sh` reads it or not at all, a list
  of well-formed cases must read as a value, and no earlier value may come
  back.
- Nine mutations of the parser's rules each turn the suite red.
- End to end through the driver plan: every supported distro rewritten in
  single quotes plans exactly as in double quotes; a duplicated `ID` plans
  from its last assignment; an unreadable file is refused with the message
  above and no commands. `rocm examine`'s probe failure is asserted with its
  state. These fail on the code before this change.

No e2e scenario in this commit: a later commit in this change plants
/etc/os-release under the simulated host root #510 added, and runs
`rocm examine --json` against it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
`rocm examine` reads /etc/os-release through `rocm_core::host_path`
since #510, so a scenario can describe a machine other than the runner.
Three scenarios run `rocm examine --json` against a planted file:

- examine-20: every value single-quoted reads as RHEL 9.4.
- examine-21: a file naming Debian 12 and then Rocky Linux 9.5 reads as
  Rocky, the last assignment, for the distro fields and the name alike.
- examine-22: a file with `export ID=debian` on line 3 reports no
  distro, and names the line both in `probe_failures` and in the
  distribution name a person reads; message and state in one scenario.

The planted distros are ones no runner here has, so a probe that read
the real file would report a different one. The driver plan is not
covered this way: it builds commands for the real machine, so it reads
the real file by design.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf added a commit that referenced this pull request Oct 13, 2026
The planted-root test from #510 covers the KFD, WSL and container probes
but not /etc/os-release, so reverting either of `rocm examine`'s two
os-release reads (the distro fields and the distro name) to the real
path left every unit test green; only the e2e scenarios noticed. Plant
a single-quoted file naming a distro no runner has, and assert both
reads report it, which also fails if either read skips the shared
parser.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
jussielo-amd pushed a commit to jussielo-amd/rocm-cli that referenced this pull request Oct 14, 2026
* fix: read /etc/os-release with one parser that matches sh

Four places parsed `/etc/os-release` — the driver plan (which also picks the
distro for the package installs it approves in `ensure_openmpi_for_vllm` and
`ensure_torch_runtime_dep`), `rocm examine`, the OpenMPI install hint, and
the `PRETTY_NAME` reader — each with its own `trim_matches`. Against `sh`
sourcing the same file, they disagreed with each other:

  file                      sh       driver plan  rocm examine  OpenMPI hint
  ID='ubuntu'               ubuntu   'ubuntu'     'ubuntu'      ubuntu
  ID=debian, then ID=ubuntu ubuntu   debian       ubuntu        debian

`os-release(5)` allows single quotes, so a spec-valid single-quoted file read
as an unsupported distro. And on a file with a duplicated `ID`,
`rocm examine` reported Ubuntu while the driver plan was built for Debian 12.

All four now read through `rocm_core::os_release`, which keeps one rule: the
value `sh` would assign, or nothing. For a file of blank lines, comments and
spec-valid assignments it returns `sh`'s value — one matching quote pair
removed, `\"`, `\\`, `` \` `` and `\$` decoded inside double quotes, nothing
decoded inside single quotes, `\x` meaning `x` in a bare value, a `#` comment
after a value, the last assignment of a key winning.

Any other line makes the whole file unreadable, not just the key it names.
To `sh` such a line is a command or the start of one — `export ID=…` and
`X=1; ID=…` assign, `unset ID` removes, an unterminated quote swallows the
lines after it — and a command can change any variable, including ones set
before it. Reading past it would mean reporting values `sh` does not assign.
A CRLF file is unreadable too: `sh` assigns `ubuntu\r`, which matches no
distro.

An unreadable file is reported, not hidden. `parse` names the first
offending line, and the driver plan's reason becomes `/etc/os-release line N
is not a plain assignment: "<line>"; no driver commands were planned because
the distro cannot be read.` instead of "this distro is not supported", while
`rocm examine` records the same line in `probe_failures`.

Behaviour changes:
- `rocm examine` reports single-quoted and duplicated keys as `sh` reads
  them, and an unreadable file as a probe failure naming its line.
- The driver plan, and the package installs it approves, read the same
  distro `rocm examine` reports.
- The OpenMPI hint no longer accepts `ID =x`; to `sh` that runs a command
  named `ID`.

Real files: run against `dash`, all 50 container images checked and 396 of
the 397 files in which-distro/os-release (10dfd5bc) read exactly as `dash`
reads them; the one exception is a CRLF file, which reads as unreadable.
No file read a value `dash` does not assign.

Tests:
- Against `/bin/sh` itself (empty environment, a sentinel `HOME`, a `PATH`
  that does not exist): every spec-valid encoding of every value up to three
  characters, and every right-hand side up to three characters over a
  quote-, backslash- and newline-heavy alphabet, checked for the key it sets
  and a key after it. Each must read as `sh` reads it or not at all, a list
  of well-formed cases must read as a value, and no earlier value may come
  back.
- Nine mutations of the parser's rules each turn the suite red.
- End to end through the driver plan: every supported distro rewritten in
  single quotes plans exactly as in double quotes; a duplicated `ID` plans
  from its last assignment; an unreadable file is refused with the message
  above and no commands. `rocm examine`'s probe failure is asserted with its
  state. These fail on the code before this change.

No e2e scenario in this commit: a later commit in this change plants
/etc/os-release under the simulated host root ROCm#510 added, and runs
`rocm examine --json` against it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(os-release): pin the host distro name to the shared parser

The PRETTY_NAME reader moved to host_gpu.rs on main, and reverting it
to its old line scan left every test green: the one fixture it had reads
the same through both parsers. Add a case the two disagree on (a
duplicated key with a trailing comment, and an unreadable file), and
name it in the module's list of readers.

Document in docs/testing.md what an unreadable /etc/os-release does to
the driver plan and to `rocm examine`, and how to run the tests.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(os-release): name the unreadable line wherever the distro is used

Only the driver plan and `rocm examine --json` said which line made
/etc/os-release unreadable. The OpenMPI, libatomic and libnuma installs
`rocm` runs before serving, and the three install hints naming them, fell
back to "Could not map distribution (ID=<unknown>)" or generic advice,
blaming the distribution for a fixable line. The human `rocm examine`
printed `distro: Linux` with no reason.

- `rocm_core::openmpi::plan_for_os_release` builds those plans from the
  file: an unreadable one plans nothing, and the reason names the line
  ahead of the manual-install advice. `ensure_openmpi_for_vllm` and
  `ensure_torch_runtime_dep` use it, and the hints share the same lookup,
  which now appends the line to the manual advice.
- `os_release::id_and_id_like` is the one place `ID` and `ID_LIKE` are
  looked up for those plans, parsing the file once rather than per key.
- The distro name `rocm examine` prints is `Linux (/etc/os-release line N
  is not a plain assignment: "...")` for an unreadable file.
- `UnreadableLine` implements `std::error::Error`.

The plans and hints still read the real /etc/os-release, not the
simulated host root: their commands run on the real machine.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): plant /etc/os-release under the simulated host root

`rocm examine` reads /etc/os-release through `rocm_core::host_path`
since ROCm#510, so a scenario can describe a machine other than the runner.
Three scenarios run `rocm examine --json` against a planted file:

- examine-20: every value single-quoted reads as RHEL 9.4.
- examine-21: a file naming Debian 12 and then Rocky Linux 9.5 reads as
  Rocky, the last assignment, for the distro fields and the name alike.
- examine-22: a file with `export ID=debian` on line 3 reports no
  distro, and names the line both in `probe_failures` and in the
  distribution name a person reads; message and state in one scenario.

The planted distros are ones no runner here has, so a probe that read
the real file would report a different one. The driver plan is not
covered this way: it builds commands for the real machine, so it reads
the real file by design.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(os-release): pin the rules mutants survived; say what the code does

Mutations a review ran each left the suite green. Each now fails a test:

- The parser: `~` after an escaped character or an escaped `:` (`\a~`,
  `a\:~`) must read as itself; `<` and `)` in a bare value, a name that
  starts with a digit, contains `-` or is empty, and DEL and C1 controls
  must make the file unreadable.
- The unreadable driver plan: no `os_id`, `version_id` or `codename`,
  not mutating, no reboot, no preflight checks, and `rocm examine` as the
  one check.
- `rocm examine`: an unreadable file makes the examination `degraded`,
  a readable one leaves it `ok`.

The module doc listed "a space or tab between `=` and the value" as a
refusal, but `K= #c` assigns the empty string; it now says anything
after the value but blanks and a comment is refused. It also claimed
`sh`'s value for every spec-valid file, though `K=a\ ` is refused; that
exception is stated. The caller list in the module doc is gone, since it
goes stale. docs/testing.md now names every reader, the
`rocm diagnose --distro` probe that has the distribution's own `sh`
source the file, what each reader shows for an unreadable file, and the
new scenarios.

The `/bin/sh` oracle tests are Linux-only, and name their scratch
directories with `test_support::unique_suffix`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(examine): read both os-release probes through a planted host root

The planted-root test from ROCm#510 covers the KFD, WSL and container probes
but not /etc/os-release, so reverting either of `rocm examine`'s two
os-release reads (the distro fields and the distro name) to the real
path left every unit test green; only the e2e scenarios noticed. Plant
a single-quoted file naming a distro no runner has, and assert both
reads report it, which also fails if either read skips the shared
parser.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(os-release): offer --yes only when its plan installs; say how to fix the line

vLLM's missing-runtime messages offered `rocm engines install vllm --yes`
even when the plan that command runs had no commands, as it has for an
unreadable /etc/os-release (and for an unknown distribution). The hints
now return whether their plan installs anything, read from the same
plan_for_os_release the install runs, and the offer follows from that.

An unreadable line now comes with what to change on it — LF line
endings, single quotes around the value, or a NAME=value line — in the
driver plan's reason, the package plans' reasons and the hints.

Tests close the gaps mutants survived: ID and ID_LIKE order with a
multi-word ID_LIKE, a derivative's hint equal to its plan's commands,
the three hint wrappers' builder and manual text, an indented comment,
and an indented unreadable line keeping its indentation.
os_release::field had no production caller and is now a test helper.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(examine): plant a PRETTY_NAME the old reader got wrong

The host-root test's doc said a reader bypassing the parser would fail
it, which was false for the distro name: the old PRETTY_NAME reader also
stripped single quotes. A repeated PRETTY_NAME, which the old reader
read as the first and sh as the last, makes the claim true. The CI
step's comment now lists /etc/os-release among the sampled reads.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): an unreadable release file degrades a GPU machine's verdict

examine-23 plants a GPU in the simulated host's kernel topology next to
an unreadable /etc/os-release, and puts an lspci listing an empty bus on
PATH so a runner without lspci adds no failure of its own. The verdict
is then `degraded` with the release-file line as its only failure, on
every Linux lane with or without a GPU. Without the GPU the verdict is
`no-amd-gpu` whatever the file says, which is why examine-22 could not
show it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants