From cab2da09463f828b609b694a0cca8dc53d7a3815 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 12:16:04 +0200 Subject: [PATCH 01/26] test(e2e): pin driver-install-as-root and comfyui runtime contracts (EAI-8071) Two test-only scenarios pinning walkthrough defects, each registered as an xfail in expectations.toml citing its ticket so CI stays green and the row goes stale the day the bug is fixed. EAI-8053: `install driver --dkms` prefixes every plan command with a literal `sudo`, so as root on a host without sudo the first command dies with `sudo: not found`. New driver.feature runs the install with a PATH that omits sudo and asserts the run does not fail merely because sudo is absent. Adds a `@requires-root` tag (skips off root, where the prefix is correct) so the row stays honest on non-root lanes. EAI-8051: `comfyui install` installs ComfyUI's deps into the managed ROCm runtime with no index scoping, so a transitive dep can pull CUDA nvidia-* wheels in and displace the runtime's ROCm torch. New comfyui.feature installs an isolated runtime, installs ComfyUI, and asserts the runtime's torch is still a ROCm build with no nvidia-* distributions. GPU/lifecycle/nightly and isolated, since it may corrupt the runtime it runs against. Signed-off-by: fredespi --- tests/e2e-cucumber/expectations.toml | 26 +++ tests/e2e-cucumber/features/comfyui.feature | 26 +++ tests/e2e-cucumber/features/driver.feature | 19 ++ tests/e2e-cucumber/src/capability.rs | 24 +++ tests/e2e-cucumber/src/expectation.rs | 64 ++++++- tests/e2e-cucumber/tests/e2e.rs | 54 ++++++ tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 163 ++++++++++++++++++ tests/e2e-cucumber/tests/e2e/driver_steps.rs | 54 ++++++ 8 files changed, 428 insertions(+), 2 deletions(-) create mode 100644 tests/e2e-cucumber/features/comfyui.feature create mode 100644 tests/e2e-cucumber/features/driver.feature create mode 100644 tests/e2e-cucumber/tests/e2e/comfyui_steps.rs create mode 100644 tests/e2e-cucumber/tests/e2e/driver_steps.rs diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index c86821cdb3..ea16d701ad 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -178,3 +178,29 @@ flaky = true when = { is_wsl = true, therock_family = "gfx*" } bug = "EAI-7998" reason = "`examine --json` reports has_amd_gpu:false on a WSL2 host whose own summary names a gfx target." + +# --- EAI-8053: `install driver --dkms` prefixes every plan command with a literal +# `sudo `, unconditionally. As root that prefix is unnecessary, and on a root host +# with no `sudo` binary it is fatal: `sh -c "sudo apt-get update"` dies with +# `sudo: not found` before any driver work. The fix is uid-aware (prepend `sudo` +# only when NOT root), so the scenario is @requires-root (it SKIPS off root, where +# the prefix is correct); on a root runner the "no sudo prefix" contract fails +# today. Unconditional there, so `when = {}`. Remove when the plan stops prefixing +# `sudo` while already root. --- +[["driver-install-as-root-does-not-require-sudo"]] +when = {} +bug = "EAI-8053" +reason = "install driver --dkms prefixes sudo even as root, so it fails with `sudo: not found` on a root host without sudo." + +# --- EAI-8051: `comfyui install` installs ComfyUI's dependencies into the managed +# ROCm runtime with no package-index scoping. torch/torchvision/torchaudio are +# filtered out, but a transitive dependency can still pull CUDA `nvidia-*` wheels +# into the runtime and displace its ROCm torch, so installing an optional app +# leaves the machine's base runtime a CUDA build with no AMD GPU support. Runs only +# on the nightly GPU lane (@requires-gpu @lifecycle @nightly) against an isolated, +# throwaway runtime it may corrupt. Unconditional where it runs, so `when = {}`. +# Remove when the install can no longer alter the runtime's torch stack. --- +[["comfyui-install-preserves-the-rocm-runtime"]] +when = {} +bug = "EAI-8051" +reason = "comfyui install can pull CUDA nvidia-* wheels into the managed ROCm runtime, replacing its ROCm torch." diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature new file mode 100644 index 0000000000..e494a4ce1f --- /dev/null +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -0,0 +1,26 @@ +Feature: ComfyUI application management + + # EAI-8051: `rocm comfyui install` installs ComfyUI's dependencies INTO the + # machine's managed ROCm runtime. It filters torch/torchvision/torchaudio out of + # ComfyUI's requirements, but nothing scopes the package index, so a transitive + # dependency can still pull CUDA `nvidia-*` wheels into the runtime and displace + # its ROCm torch. The runtime the whole machine serves models with is then a CUDA + # build with no AMD GPU support — installing an optional app broke the base. + # + # The contract: after installing an optional app, the machine's ROCm runtime must + # still be a ROCm runtime — its torch stays a ROCm (HIP) build and no `nvidia-*` + # CUDA distributions appear in it. We assert the runtime's health, NOT that the + # ComfyUI install exits 0: the real install exits non-zero AND still leaves the + # damage, so an exit-code assertion would miss the defect. + # + # Genuinely destructive and expensive: it needs a real managed runtime (a + # multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind + # @lifecycle and @nightly, against this scenario's own isolated runtime prefix + # (it must never share a runtime tree with other scenarios — it may corrupt it). + @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @lifecycle @nightly + Scenario: 1 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one + Given an isolated machine with a managed ROCm runtime + And the runtime's torch is a ROCm build + When the user installs ComfyUI + Then the runtime's torch is still a ROCm build + And no CUDA nvidia packages were added to the runtime diff --git a/tests/e2e-cucumber/features/driver.feature b/tests/e2e-cucumber/features/driver.feature new file mode 100644 index 0000000000..9fb29f022a --- /dev/null +++ b/tests/e2e-cucumber/features/driver.feature @@ -0,0 +1,19 @@ +Feature: Native driver installation + + # EAI-8053: `rocm install driver --dkms` builds its command plan with a literal + # `sudo ` prefix on every step, unconditionally. When the CLI already runs as + # root the prefix is unnecessary, and on a root host that has no `sudo` binary it + # is actively harmful: the very first command dies with `sudo: not found` before + # any driver work happens. The contract is that being root — the state where the + # commands could otherwise succeed — must not be the thing that breaks the run. + # + # Root-gated: the fix is uid-aware (prepend `sudo` only when NOT root), so the + # "no sudo prefix" contract only has a premise where the runner is actually root. + # Off root the sudo prefix is correct, so @requires-root skips there rather than + # letting the row falsely pass. Linux-only: the plan and its `sh -c` execution + # are the Linux DKMS path. + @id:driver-install-as-root-does-not-require-sudo @requires-os:linux @requires-root + Scenario: 1 - Installing the driver as root does not depend on sudo being present + Given a root machine with no sudo command available + When the user installs the native driver with dkms + Then the install does not fail merely because sudo is missing diff --git a/tests/e2e-cucumber/src/capability.rs b/tests/e2e-cucumber/src/capability.rs index 9ca0d6a0a6..53b4f69127 100644 --- a/tests/e2e-cucumber/src/capability.rs +++ b/tests/e2e-cucumber/src/capability.rs @@ -99,6 +99,10 @@ pub struct HostCapability { pub os_family: String, /// `examine`'s `wsl:` line. pub is_wsl: bool, + /// Whether the test process runs as root (effective UID 0). Always false off + /// Unix. Gates `@requires-root` scenarios whose contract only holds when the + /// runner is root (see [`ScenarioDecl::requires_root`]). + pub is_root: bool, /// First AMD GPU's gfx target from `examine`'s `detected_gfx_target:` line /// (e.g. "gfx942", "gfx1151"), if a real one was reported. pub gfx_target: Option, @@ -348,6 +352,7 @@ fn probe_host_capability() -> HostCapability { HostCapability { os_family, is_wsl, + is_root: process_is_root(), gfx_target, has_amd_gpu, available_engines, @@ -356,6 +361,23 @@ fn probe_host_capability() -> HostCapability { } } +/// Whether the current test process runs as root (effective UID 0). Read from +/// `/proc/self/status` rather than `geteuid` so the check stays inside this +/// crate's `deny(unsafe_code)` policy. The `Uid:` line lists real, effective, +/// saved and filesystem uids; the effective uid (the second field) is the one the +/// driver-install `sudo`-prefix contract turns on. False on any non-Linux host or +/// if the status file cannot be read — off Linux the contract does not apply. +fn process_is_root() -> bool { + let Ok(status) = std::fs::read_to_string("/proc/self/status") else { + return false; + }; + status + .lines() + .find_map(|line| line.strip_prefix("Uid:")) + .and_then(|rest| rest.split_whitespace().nth(1)) + .is_some_and(|euid| euid == "0") +} + /// Run `rocm ` with an isolated config/data/cache root, returning stdout /// (empty string on any failure — the probe must never panic the suite). fn run_probe(root: &std::path::Path, args: &[&str]) -> String { @@ -600,6 +622,7 @@ mod tests { let strix = HostCapability { os_family: "windows".to_owned(), is_wsl: false, + is_root: false, gfx_target: Some("gfx1151".to_owned()), has_amd_gpu: true, available_engines: vec!["lemonade".to_owned(), "vllm".to_owned()], @@ -613,6 +636,7 @@ mod tests { let mi300x = HostCapability { os_family: "linux".to_owned(), is_wsl: false, + is_root: false, gfx_target: Some("gfx942".to_owned()), has_amd_gpu: true, available_engines: vec!["lemonade".to_owned(), "vllm".to_owned()], diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index 99b8c13454..e6cc61e268 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -28,6 +28,7 @@ const REQUIRES_GPU_TAG: &str = "requires-gpu"; const REQUIRES_NO_GPU_TAG: &str = "requires-no-gpu"; const REQUIRES_BARE_METAL_TAG: &str = "requires-bare-metal"; const REQUIRES_WSL_TAG: &str = "requires-wsl"; +const REQUIRES_ROOT_TAG: &str = "requires-root"; const SERVE_TIMEOUT_PREFIX: &str = "serve-timeout:"; const NIGHTLY_TAG: &str = "nightly"; const LIFECYCLE_TAG: &str = "lifecycle"; @@ -82,6 +83,15 @@ pub struct ScenarioDecl { /// host, so it is skipped on native Linux, native Windows and everything /// else. Same reason `@requires-os:linux` cannot stand in for it. pub requires_wsl: bool, + /// `@requires-root`: the scenario's premise is a process running as root + /// (effective UID 0), so it is skipped where the test process is not root. + /// Needed by the driver-install-as-root contract (EAI-8053): the fix is + /// uid-aware (prepend `sudo` only when NOT root), so the "no `sudo` prefix" + /// contract only holds where the runner is actually root. None of the four + /// `expectations.toml` condition keys can express "running as root", so — like + /// `@requires-wsl` — this is a tag, not a row, keeping the xfail row honest on + /// non-root lanes (they SKIP rather than falsely PASS). + pub requires_root: bool, /// Engine the scenario pins via `@requires-engine:` (if any). pub requires_engine: Option, /// OS the scenario requires via `@requires-os:` (e.g. "linux"), if any — @@ -120,6 +130,7 @@ impl ScenarioDecl { let mut requires_no_gpu = false; let mut requires_bare_metal = false; let mut requires_wsl = false; + let mut requires_root = false; let mut requires_engine = None; let mut requires_os = None; let mut serve_timeout_secs = None; @@ -147,6 +158,8 @@ impl ScenarioDecl { requires_bare_metal = true; } else if tag == REQUIRES_WSL_TAG { requires_wsl = true; + } else if tag == REQUIRES_ROOT_TAG { + requires_root = true; } else if tag == NIGHTLY_TAG { nightly = true; } else if tag == LIFECYCLE_TAG { @@ -161,6 +174,7 @@ impl ScenarioDecl { requires_no_gpu, requires_bare_metal, requires_wsl, + requires_root, requires_engine, requires_os, serve_timeout_secs, @@ -363,8 +377,8 @@ pub struct PlatformManifest<'a> { /// 1. Not-applicable → `Skip`: a `@nightly` scenario when nightly isn't included, /// a `@merge-queue` scenario outside the merge queue, a `@requires-gpu` /// scenario on a host with no AMD GPU, a `@requires-bare-metal` scenario on -/// WSL2, a `@requires-os:` scenario on a different OS, or a scenario whose -/// effective engine can't start. +/// WSL2, a `@requires-root` scenario off root, a `@requires-os:` scenario +/// on a different OS, or a scenario whose effective engine can't start. /// 2. First matching `expectations.toml` condition → `ExpectXfail`. /// 3. Otherwise → `ExpectPass`. /// @@ -420,6 +434,11 @@ pub fn resolve( reason: "requires WSL; this host is not running under WSL".to_owned(), }; } + if decl.requires_root && !cap.is_root { + return Expectation::Skip { + reason: "requires the runner to be root; this process is not root".to_owned(), + }; + } if let Some(os) = &decl.requires_os && !os.eq_ignore_ascii_case(&cap.os_family) { @@ -500,6 +519,7 @@ mod tests { "mi300x" => HostCapability { os_family: "linux".into(), is_wsl: false, + is_root: false, gfx_target: Some("gfx942".into()), has_amd_gpu: true, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -509,6 +529,7 @@ mod tests { "strix-ubuntu" => HostCapability { os_family: "linux".into(), is_wsl: false, + is_root: false, gfx_target: Some("gfx1151".into()), has_amd_gpu: true, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -518,6 +539,7 @@ mod tests { "strix-windows" => HostCapability { os_family: "windows".into(), is_wsl: false, + is_root: false, gfx_target: Some("gfx1151".into()), has_amd_gpu: true, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -531,6 +553,7 @@ mod tests { "wsl2" => HostCapability { os_family: "linux".into(), is_wsl: true, + is_root: false, gfx_target: Some("gfx1151".into()), has_amd_gpu: true, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -543,6 +566,7 @@ mod tests { "wsl" => HostCapability { os_family: "linux".into(), is_wsl: true, + is_root: false, gfx_target: None, has_amd_gpu: false, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -554,15 +578,29 @@ mod tests { "wsl-no-passthrough" => HostCapability { os_family: "linux".into(), is_wsl: true, + is_root: false, gfx_target: Some("gfx1151".into()), has_amd_gpu: false, available_engines: vec!["lemonade".into(), "vllm".into()], effective_serve_engine: "lemonade".into(), platform_slug: "strix-halo-wsl".into(), }, + // A no-GPU host running as root — the mock CI lane's shape, where the + // driver-install sudo-prefix contract (EAI-8053) has a premise. + "mock-root" => HostCapability { + os_family: "linux".into(), + is_wsl: false, + is_root: true, + gfx_target: None, + has_amd_gpu: false, + available_engines: vec!["lemonade".into(), "vllm".into()], + effective_serve_engine: "lemonade".into(), + platform_slug: "mock".into(), + }, _ => HostCapability { os_family: "other".into(), is_wsl: false, + is_root: false, gfx_target: None, has_amd_gpu: false, available_engines: vec!["lemonade".into(), "vllm".into()], @@ -622,6 +660,28 @@ serve_timeout_secs = 90 assert!(!decl(&["id:x", "requires-gpu"]).requires_bare_metal); } + #[test] + fn root_tag_parses_and_gates_on_root() { + // Parses in both shapes; absent by default so no existing scenario changes. + assert!(decl(&["id:x", "requires-root"]).requires_root); + assert!(decl(&["@id:x", "@requires-root"]).requires_root); + assert!(!decl(&["id:x", "requires-gpu"]).requires_root); + + let m = Expectations::default(); + let d = decl(&["id:driver-root", "requires-os:linux", "requires-root"]); + // Root host: the premise holds, so it resolves (here, expected-pass with + // an empty matrix — an xfail row is layered on separately). + assert_eq!( + resolve(&d, &cap("mock-root"), &m, false, false, false), + Expectation::ExpectPass + ); + // Non-root host: skipped, so a `when = {}` xfail row can't XPASS there. + assert!(matches!( + resolve(&d, &cap("mock"), &m, false, false, false), + Expectation::Skip { .. } + )); + } + #[test] fn serve_timeout_tag_parses_seconds() { let d = decl(&["id:serve-large-model-inference", "serve-timeout:2400"]); diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index ea7ea3d1a3..ebfa333827 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -20,8 +20,10 @@ mod e2e { pub mod artifact_steps; pub mod bench_steps; pub mod chat_steps; + pub mod comfyui_steps; pub mod dash_steps; pub mod diagnose_steps; + pub mod driver_steps; pub mod engines_steps; pub mod examine_steps; pub mod lifecycle_steps; @@ -548,6 +550,58 @@ pub fn run_rocm_with_env( ) } +/// Run `rocm` with `PATH` pointed at a temp directory that exposes only the given +/// host tools (by name), so a scenario can prove how the CLI behaves when a tool +/// it shells out to is absent. Returns `(stdout, stderr, rc)` plus the `TempDir`, +/// which the caller must keep alive for the duration of the run. +/// +/// Used by the root-without-`sudo` driver-install scenario (EAI-8053): the driver +/// plan shells its commands through `sh -c`, so `sh` must stay reachable while +/// `sudo` must not. Each requested tool is symlinked from wherever it lives on the +/// current `PATH`; a tool that cannot be found is skipped (its absence is exactly +/// what some scenarios want to arrange). +#[cfg(unix)] +pub fn run_rocm_with_only_tools( + world: &E2eWorld, + args: &[&str], + tools: &[&str], +) -> (String, String, i32, TempDir) { + let bin = TempDir::with_prefix("rocm-e2e-path-").expect("failed to create temp PATH dir"); + for tool in tools { + if let Some(real) = which_on_path(tool) { + let _ = std::os::unix::fs::symlink(&real, bin.path().join(tool)); + } + } + let binary = rocm_binary(); + let mut cmd = std::process::Command::new(&binary); + cmd.args(args); + world.isolate_cmd(&mut cmd); + cmd.env("PATH", bin.path()); + let output = cmd + .output() + .unwrap_or_else(|e| panic!("failed to run {binary}: {e}")); + let rc = output.status.code().unwrap_or(-1); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + record_command(world.current_scenario.as_deref(), args, rc, &stdout); + ( + stdout, + String::from_utf8_lossy(&output.stderr).to_string(), + rc, + bin, + ) +} + +/// Resolve a bare tool name to its absolute path by scanning the current `PATH`, +/// following the same first-match rule a shell would. Returns `None` if the tool +/// is not on `PATH`. +#[cfg(unix)] +fn which_on_path(tool: &str) -> Option { + let path = std::env::var_os("PATH")?; + std::env::split_paths(&path) + .map(|dir| dir.join(tool)) + .find(|candidate| candidate.is_file()) +} + /// Append one `rocm` invocation to `results/commands.jsonl` so the consolidated /// report can build a command × platform coverage table tied to real results. /// Best-effort: a recording failure must never fail a scenario. diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs new file mode 100644 index 0000000000..b617929594 --- /dev/null +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -0,0 +1,163 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +use std::path::{Path, PathBuf}; + +use cucumber::{given, then, when}; + +use crate::E2eWorld; + +/// A one-shot Python probe run against the managed runtime's OWN interpreter (not +/// the ambient one `rocm examine` uses). Emits JSON `{hip, version}` when torch +/// imports — `hip` is non-null for a ROCm/HIP build and null for a CUDA build — or +/// `{error}` when torch cannot be imported at all. +const TORCH_HIP_PROBE: &str = "import json,sys\n\ + out={}\n\ + try:\n\ + \x20 import torch\n\ + \x20 out['hip']=getattr(torch.version,'hip',None)\n\ + \x20 out['version']=torch.__version__\n\ + except Exception as ex:\n\ + \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ + sys.stdout.write(json.dumps(out))\n"; + +/// Locate the active managed runtime's venv interpreter. `rocm examine` prints +/// `Folder: ` for the active runtime; the interpreter lives under a +/// `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. The +/// exact env sub-layout is an internal detail, so search for the interpreter +/// rather than reconstruct the path — black-box, and tolerant of layout changes. +fn active_runtime_python(world: &E2eWorld) -> PathBuf { + let (examine, _, _) = crate::run_rocm(world, &["examine"]); + let folder = examine + .lines() + .find_map(|l| l.trim().strip_prefix("Folder:")) + .map(str::trim) + .unwrap_or_else(|| panic!("no active-runtime 'Folder:' line in examine:\n{examine}")); + find_venv_python(Path::new(folder)).unwrap_or_else(|| { + panic!("could not locate a venv python under the runtime folder {folder}") + }) +} + +/// Depth-limited search for a `bin/python` (Unix) or `Scripts/python.exe` +/// (Windows) under `root`. The managed runtime keeps its interpreter a few levels +/// down; cap the walk so a pathological tree can't hang the scenario. +fn find_venv_python(root: &Path) -> Option { + #[cfg(windows)] + let (bin, exe) = ("Scripts", "python.exe"); + #[cfg(not(windows))] + let (bin, exe) = ("bin", "python"); + let mut frontier = vec![(root.to_path_buf(), 0usize)]; + while let Some((dir, depth)) = frontier.pop() { + let candidate = dir.join(bin).join(exe); + if candidate.is_file() { + return Some(candidate); + } + if depth >= 6 { + continue; + } + let Ok(entries) = std::fs::read_dir(&dir) else { + continue; + }; + for entry in entries.flatten() { + if entry.file_type().is_ok_and(|t| t.is_dir()) { + frontier.push((entry.path(), depth + 1)); + } + } + } + None +} + +/// Whether the interpreter's torch is a ROCm/HIP build. +fn torch_is_rocm(python: &Path) -> bool { + let output = std::process::Command::new(python) + .args(["-c", TORCH_HIP_PROBE]) + .output() + .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); + let stdout = String::from_utf8_lossy(&output.stdout); + let data: serde_json::Value = serde_json::from_str(stdout.trim()) + .unwrap_or_else(|_| panic!("torch probe returned non-JSON:\n{stdout}")); + data.get("hip") + .is_some_and(|hip| !hip.is_null() && hip.as_str().is_some_and(|s| !s.is_empty())) +} + +/// The `nvidia-*` CUDA distributions installed in the interpreter's environment, +/// as reported by `pip list`. A ROCm runtime should have none; ComfyUI's install +/// dragging any in is the EAI-8051 defect. +fn nvidia_distributions(python: &Path) -> Vec { + let output = std::process::Command::new(python) + .args(["-m", "pip", "list", "--format=freeze"]) + .output() + .unwrap_or_else(|e| panic!("failed to run pip list on {}: {e}", python.display())); + let stdout = String::from_utf8_lossy(&output.stdout); + stdout + .lines() + .filter_map(|line| line.split("==").next()) + .map(str::trim) + .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) + .map(str::to_owned) + .collect() +} + +#[given("an isolated machine with a managed ROCm runtime")] +async fn setup_isolated_runtime(world: &mut E2eWorld) { + // DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may + // corrupt the runtime (that is the bug it pins), so it must own a private, + // throwaway runtime prefix. Each World already has isolated ROCM_CLI_* dirs, + // so a plain `install sdk` here lands in this scenario's own tree. + let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); + if stdout.contains("installed: none") { + crate::run_rocm_ok(world, &["install", "sdk"]); + } + let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); + assert!( + !stdout.contains("installed: none"), + "no managed runtime is active after install:\n{stdout}" + ); +} + +#[given("the runtime's torch is a ROCm build")] +async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { + let python = active_runtime_python(world); + assert!( + torch_is_rocm(&python), + "baseline runtime torch is not a ROCm build; scenario premise absent ({})", + python.display() + ); + assert!( + nvidia_distributions(&python).is_empty(), + "runtime already has nvidia-* distributions before ComfyUI install; premise absent" + ); +} + +#[when("the user installs ComfyUI")] +async fn user_installs_comfyui(world: &mut E2eWorld) { + // Do NOT use run_rocm_ok: the real install exits non-zero while still leaving + // the runtime damaged, so the exit code is not the contract (see the feature + // comment). Capture the outcome for diagnostics only. + let (stdout, stderr, rc) = crate::run_rocm(world, &["comfyui", "install"]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the runtime's torch is still a ROCm build")] +async fn assert_torch_still_rocm(world: &mut E2eWorld) { + let python = active_runtime_python(world); + assert!( + torch_is_rocm(&python), + "ComfyUI install replaced the runtime's ROCm torch with a non-ROCm build ({})", + python.display() + ); +} + +#[then("no CUDA nvidia packages were added to the runtime")] +async fn assert_no_nvidia_packages(world: &mut E2eWorld) { + let python = active_runtime_python(world); + let nvidia = nvidia_distributions(&python); + assert!( + nvidia.is_empty(), + "ComfyUI install added CUDA nvidia-* distributions to the ROCm runtime: {}", + nvidia.join(", ") + ); +} diff --git a/tests/e2e-cucumber/tests/e2e/driver_steps.rs b/tests/e2e-cucumber/tests/e2e/driver_steps.rs new file mode 100644 index 0000000000..458a11aa9d --- /dev/null +++ b/tests/e2e-cucumber/tests/e2e/driver_steps.rs @@ -0,0 +1,54 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +use cucumber::{given, then, when}; + +use crate::E2eWorld; + +/// The tools the driver plan legitimately shells out to on the DKMS path, minus +/// `sudo`. The scenario runs `rocm` with a `PATH` exposing only these, so the +/// plan can still execute its commands via `sh -c` while `sudo` is genuinely +/// absent — the exact state EAI-8053 reproduces. `sudo` is deliberately excluded; +/// `apt-get`/`curl` etc. are omitted too, so a fixed (uid-aware) CLI fails at a +/// real package step rather than at `sudo`, which is precisely the distinction the +/// contract asserts. +const PLAN_TOOLS_WITHOUT_SUDO: &[&str] = &["sh", "env"]; + +#[given("a root machine with no sudo command available")] +async fn setup_root_no_sudo(_world: &mut E2eWorld) { + // Applicability is enforced by the scenario's @requires-root tag (the runner + // must be root) and by the sanitized PATH the When step builds (no `sudo`); + // nothing to arrange on the World here. +} + +#[when("the user installs the native driver with dkms")] +async fn user_installs_driver(world: &mut E2eWorld) { + let (stdout, stderr, rc, _path) = crate::run_rocm_with_only_tools( + world, + &["install", "driver", "--dkms", "--yes"], + PLAN_TOOLS_WITHOUT_SUDO, + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the install does not fail merely because sudo is missing")] +async fn assert_not_broken_by_missing_sudo(world: &mut E2eWorld) { + let stdout = world.cli_output.as_deref().unwrap_or(""); + let stderr = world.cli_stderr.as_deref().unwrap_or(""); + let combined = format!("{stdout}\n{stderr}"); + // The bug's signature: the plan prefixed `sudo` even though the process is + // root, so `sh -c "sudo …"` dies with `sudo: not found` and the CLI reports + // the sudo-prefixed command as the failure. A root-aware plan would run the + // command without `sudo` and get to the actual package work. We assert the + // ABSENCE of the sudo-not-found failure, not a successful install (impossible + // in a container), so the row goes stale the day the prefix becomes uid-aware. + assert!( + !combined.contains("sudo: not found") + && !combined.contains("sudo: command not found") + && !combined.contains("driver command failed: sudo "), + "driver install as root failed because it invoked a missing `sudo`:\n{combined}" + ); +} From ffa0d2977c53305f0ac86beddcae1d0d308dddcf Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 12:41:09 +0200 Subject: [PATCH 02/26] test(e2e): fix EAI-8051 comfyui scenario gating and package probe Address pre-PR review of the ComfyUI runtime-preservation scenario: - Drop @lifecycle from the scenario. No lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so @requires-gpu @lifecycle @nightly together made the scenario unreachable on every lane. @requires-gpu @nightly matches runtime-install-sdk-active, the other scenario that does a real install sdk; this scenario mutates only its own isolated runtime prefix, not the OS. - Enumerate installed distributions via importlib.metadata instead of `python -m pip list`. uv-created managed runtimes have no pip module, so the pip probe exited non-zero with empty stdout, which the old code read as "no nvidia packages" and passed the check on a runtime it never inspected. The probe now fails loudly on any error rather than treating it as a clean result. Signed-off-by: fredespi --- tests/e2e-cucumber/expectations.toml | 4 +- tests/e2e-cucumber/features/comfyui.feature | 11 ++-- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 50 +++++++++++++++---- 3 files changed, 51 insertions(+), 14 deletions(-) diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index e5c29514a5..f709da802b 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -226,8 +226,8 @@ reason = "install driver --dkms prefixes sudo even as root, so it fails with `su # filtered out, but a transitive dependency can still pull CUDA `nvidia-*` wheels # into the runtime and displace its ROCm torch, so installing an optional app # leaves the machine's base runtime a CUDA build with no AMD GPU support. Runs only -# on the nightly GPU lane (@requires-gpu @lifecycle @nightly) against an isolated, -# throwaway runtime it may corrupt. Unconditional where it runs, so `when = {}`. +# on the nightly GPU lane (@requires-gpu @nightly) against an isolated, throwaway +# runtime it may corrupt. Unconditional where it runs, so `when = {}`. # Remove when the install can no longer alter the runtime's torch stack. --- [["comfyui-install-preserves-the-rocm-runtime"]] when = {} diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index e494a4ce1f..a97f5fea42 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -15,9 +15,14 @@ Feature: ComfyUI application management # # Genuinely destructive and expensive: it needs a real managed runtime (a # multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind - # @lifecycle and @nightly, against this scenario's own isolated runtime prefix - # (it must never share a runtime tree with other scenarios — it may corrupt it). - @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @lifecycle @nightly + # @nightly, against this scenario's own isolated runtime prefix (it must never + # share a runtime tree with other scenarios — it may corrupt it). Gated + # @requires-gpu @nightly, matching runtime-install-sdk-active, the other scenario + # that does a real `install sdk`. NOT @lifecycle: that tag is for OS-mutating + # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so + # combining it with @nightly would make this scenario unreachable on every lane; + # this mutates only its own isolated runtime prefix, not the OS. + @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly Scenario: 1 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given an isolated machine with a managed ROCm runtime And the runtime's torch is a ROCm build diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index b617929594..232d36c890 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -81,19 +81,51 @@ fn torch_is_rocm(python: &Path) -> bool { .is_some_and(|hip| !hip.is_null() && hip.as_str().is_some_and(|s| !s.is_empty())) } -/// The `nvidia-*` CUDA distributions installed in the interpreter's environment, -/// as reported by `pip list`. A ROCm runtime should have none; ComfyUI's install -/// dragging any in is the EAI-8051 defect. +/// Enumerate installed distributions via `importlib.metadata` and emit their names +/// as a JSON array. Used instead of `pip list` because uv-created managed runtimes +/// have no `pip` module — `python -m pip` there exits non-zero with empty stdout, +/// which a naive reader would misread as "no packages installed" and pass the +/// nvidia check while the runtime is actually corrupted. `importlib.metadata` is in +/// the stdlib, so it is always present; the probe emits `{names}` on success or +/// `{error}` on failure so the caller can fail loudly rather than treat a broken +/// probe as a clean result. +const DISTRIBUTIONS_PROBE: &str = "import json,sys\n\ + out={}\n\ + try:\n\ + \x20 from importlib import metadata\n\ + \x20 out['names']=sorted({(d.metadata['Name'] or '') for d in metadata.distributions()})\n\ + except Exception as ex:\n\ + \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ + sys.stdout.write(json.dumps(out))\n"; + +/// The `nvidia-*` CUDA distributions installed in the interpreter's environment. +/// A ROCm runtime should have none; ComfyUI's install dragging any in is the +/// EAI-8051 defect. Panics if the interpreter cannot be run or the probe reports +/// an error — a probe that cannot enumerate packages must NOT read as "no nvidia +/// packages", which would pass the contract on a runtime it never actually checked. fn nvidia_distributions(python: &Path) -> Vec { let output = std::process::Command::new(python) - .args(["-m", "pip", "list", "--format=freeze"]) + .args(["-c", DISTRIBUTIONS_PROBE]) .output() - .unwrap_or_else(|e| panic!("failed to run pip list on {}: {e}", python.display())); + .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); let stdout = String::from_utf8_lossy(&output.stdout); - stdout - .lines() - .filter_map(|line| line.split("==").next()) - .map(str::trim) + let stderr = String::from_utf8_lossy(&output.stderr); + let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { + panic!("distributions probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") + }); + if let Some(error) = data.get("error").and_then(serde_json::Value::as_str) { + panic!( + "could not enumerate installed distributions on {}: {error}", + python.display() + ); + } + let names = data + .get("names") + .and_then(serde_json::Value::as_array) + .unwrap_or_else(|| panic!("distributions probe returned no 'names' array:\n{stdout}")); + names + .iter() + .filter_map(serde_json::Value::as_str) .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) .map(str::to_owned) .collect() From 6ad735e062e48e5d509406547cc386bf1db59734 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 13:36:10 +0200 Subject: [PATCH 03/26] test(e2e): fix clippy on the e2e test targets CI's clippy lane checks the e2e harness test target with -D warnings, which surfaced two lints the mock-lane gate did not: - comfyui_steps.rs: map(str::trim).unwrap_or_else(panic) -> map_or_else. - e2e.rs: split run_rocm_with_only_tools' overlong first doc paragraph. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e.rs | 9 +++++---- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 6 ++++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index ebfa333827..af21724a3c 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -550,10 +550,11 @@ pub fn run_rocm_with_env( ) } -/// Run `rocm` with `PATH` pointed at a temp directory that exposes only the given -/// host tools (by name), so a scenario can prove how the CLI behaves when a tool -/// it shells out to is absent. Returns `(stdout, stderr, rc)` plus the `TempDir`, -/// which the caller must keep alive for the duration of the run. +/// Run `rocm` with `PATH` pointed at a temp dir exposing only the given host tools. +/// +/// This lets a scenario prove how the CLI behaves when a tool it shells out to is +/// absent. Returns `(stdout, stderr, rc)` plus the `TempDir`, which the caller must +/// keep alive for the duration of the run. /// /// Used by the root-without-`sudo` driver-install scenario (EAI-8053): the driver /// plan shells its commands through `sh -c`, so `sh` must stay reachable while diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 232d36c890..b082ab9f50 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -32,8 +32,10 @@ fn active_runtime_python(world: &E2eWorld) -> PathBuf { let folder = examine .lines() .find_map(|l| l.trim().strip_prefix("Folder:")) - .map(str::trim) - .unwrap_or_else(|| panic!("no active-runtime 'Folder:' line in examine:\n{examine}")); + .map_or_else( + || panic!("no active-runtime 'Folder:' line in examine:\n{examine}"), + str::trim, + ); find_venv_python(Path::new(folder)).unwrap_or_else(|| { panic!("could not locate a venv python under the runtime folder {folder}") }) From f74c53e3abf23f6e5905d51081426e4cd1728d8d Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 14:50:00 +0200 Subject: [PATCH 04/26] test(e2e): make run_rocm_with_only_tools compile on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The helper and its which_on_path were #[cfg(unix)], but driver_steps.rs calls them unconditionally — cucumber step functions are registered on every platform, so the Windows build failed with E0425 (cannot find function). The driver scenario is @requires-os:linux and never runs on Windows, but the step code must still compile there. Drop the cfg gates and gate only the symlink line (Unix symlink; a plain copy on other platforms), so the call resolves everywhere. Verified: Linux clippy --all-targets -D warnings and a Windows cross-check (x86_64-pc-windows-gnu) both pass. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index af21724a3c..e49d3c8eaf 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -561,7 +561,11 @@ pub fn run_rocm_with_env( /// `sudo` must not. Each requested tool is symlinked from wherever it lives on the /// current `PATH`; a tool that cannot be found is skipped (its absence is exactly /// what some scenarios want to arrange). -#[cfg(unix)] +/// +/// Compiles on every platform (cucumber step functions are registered regardless +/// of host), but the only scenario that uses it is `@requires-os:linux`, so its +/// Unix-only symlink path is the only one that runs; on Windows the temp dir is +/// created empty and the scenario is skipped before reaching this call. pub fn run_rocm_with_only_tools( world: &E2eWorld, args: &[&str], @@ -570,7 +574,10 @@ pub fn run_rocm_with_only_tools( let bin = TempDir::with_prefix("rocm-e2e-path-").expect("failed to create temp PATH dir"); for tool in tools { if let Some(real) = which_on_path(tool) { + #[cfg(unix)] let _ = std::os::unix::fs::symlink(&real, bin.path().join(tool)); + #[cfg(not(unix))] + let _ = std::fs::copy(&real, bin.path().join(tool)); } } let binary = rocm_binary(); @@ -595,7 +602,6 @@ pub fn run_rocm_with_only_tools( /// Resolve a bare tool name to its absolute path by scanning the current `PATH`, /// following the same first-match rule a shell would. Returns `None` if the tool /// is not on `PATH`. -#[cfg(unix)] fn which_on_path(tool: &str) -> Option { let path = std::env::var_os("PATH")?; std::env::split_paths(&path) From 1bababf0c23405cb8156b375da52c6f502f9acdc Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 14:59:01 +0200 Subject: [PATCH 05/26] test(e2e): find the runtime via runtimes list, not examine Folder: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The EAI-8051 scenario located the managed runtime's venv by parsing `Folder:` from `rocm examine`, but examine only prints that line for the ACTIVE runtime and takes a different branch otherwise. On the MI300X GPU lane the baseline step panicked with "no active-runtime Folder: line" — a miswired step, reported as an unexpected failure rather than the intended xfail. The isolated `install sdk` itself succeeded. Read `install_root:` from `rocm runtimes list`, which is printed unconditionally for every installed runtime, so discovery no longer depends on which runtime examine considers active. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index b082ab9f50..8016db5984 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -22,22 +22,28 @@ const TORCH_HIP_PROBE: &str = "import json,sys\n\ \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ sys.stdout.write(json.dumps(out))\n"; -/// Locate the active managed runtime's venv interpreter. `rocm examine` prints -/// `Folder: ` for the active runtime; the interpreter lives under a -/// `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. The -/// exact env sub-layout is an internal detail, so search for the interpreter +/// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an +/// `install_root: ` line for each installed runtime; the interpreter lives +/// under a `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. +/// The exact env sub-layout is an internal detail, so search for the interpreter /// rather than reconstruct the path — black-box, and tolerant of layout changes. +/// +/// Reads `runtimes list` rather than `examine`: examine only prints a `Folder:` +/// line for the *active* runtime and takes a different branch when none is marked +/// active, so it is not a reliable source for the install root (this cost a GPU +/// dispatch — the scenario panicked on a missing `Folder:` there). `runtimes list` +/// prints `install_root:` for every installed runtime unconditionally. fn active_runtime_python(world: &E2eWorld) -> PathBuf { - let (examine, _, _) = crate::run_rocm(world, &["examine"]); - let folder = examine + let (listing, _, _) = crate::run_rocm(world, &["runtimes", "list"]); + let root = listing .lines() - .find_map(|l| l.trim().strip_prefix("Folder:")) + .find_map(|l| l.trim().strip_prefix("install_root:")) .map_or_else( - || panic!("no active-runtime 'Folder:' line in examine:\n{examine}"), + || panic!("no 'install_root:' line in `runtimes list`:\n{listing}"), str::trim, ); - find_venv_python(Path::new(folder)).unwrap_or_else(|| { - panic!("could not locate a venv python under the runtime folder {folder}") + find_venv_python(Path::new(root)).unwrap_or_else(|| { + panic!("could not locate a venv python under the runtime install_root {root}") }) } From b51c002bca53731f47c91d7678c693f955bc5ae3 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 17:19:39 +0200 Subject: [PATCH 06/26] test(e2e): judge torch build from dist metadata, not a live import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The EAI-8051 baseline step imported torch in the runtime venv to read torch.version.hip, but importing torch loads the ROCm shared libraries, which need the runtime's LD_LIBRARY_PATH/ROCM_PATH — the product sets that env when it probes torch, our bare interpreter does not. On the MI300X lane the import failed and a runtime with a perfectly good ROCm torch read as "not a ROCm build". Judge the build from the torch distribution's version label via importlib.metadata (`2.x+rocm...` vs `2.x+cu...`) instead — no import, no native load, so a bare interpreter suffices. The baseline and post-install assertions now also surface the actual version string for self-diagnosis on failure. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 60 +++++++++++++------ 1 file changed, 41 insertions(+), 19 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 8016db5984..e6b9c52dd6 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -8,16 +8,23 @@ use cucumber::{given, then, when}; use crate::E2eWorld; -/// A one-shot Python probe run against the managed runtime's OWN interpreter (not -/// the ambient one `rocm examine` uses). Emits JSON `{hip, version}` when torch -/// imports — `hip` is non-null for a ROCm/HIP build and null for a CUDA build — or -/// `{error}` when torch cannot be imported at all. -const TORCH_HIP_PROBE: &str = "import json,sys\n\ +/// Report the installed torch distribution's version string via `importlib.metadata` +/// — WITHOUT importing torch. Emits JSON `{version}` (e.g. `2.7.0+rocm6.4` for a +/// ROCm build, `2.7.0+cu128` for a CUDA build) or `{error}` when no torch +/// distribution is installed. +/// +/// Deliberately does not `import torch`: importing it loads the ROCm/CUDA shared +/// libraries, which need the runtime's `LD_LIBRARY_PATH`/`ROCM_PATH` set up (the +/// product runs its own torch probe *with* that env, ours runs the interpreter +/// bare). The `+rocm` / `+cu` local-version label in the dist metadata is the +/// definitive ROCm-vs-CUDA discriminator and is readable with no native load — so +/// a bare interpreter suffices, and a runtime whose torch is present but whose +/// native libs aren't on our env no longer reads as "not a ROCm build". +const TORCH_DIST_PROBE: &str = "import json,sys\n\ + from importlib import metadata\n\ out={}\n\ try:\n\ - \x20 import torch\n\ - \x20 out['hip']=getattr(torch.version,'hip',None)\n\ - \x20 out['version']=torch.__version__\n\ + \x20 out['version']=metadata.version('torch')\n\ except Exception as ex:\n\ \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ sys.stdout.write(json.dumps(out))\n"; @@ -76,17 +83,24 @@ fn find_venv_python(root: &Path) -> Option { None } -/// Whether the interpreter's torch is a ROCm/HIP build. -fn torch_is_rocm(python: &Path) -> bool { +/// The installed torch distribution's version string, or `None` if no torch +/// distribution is installed. Reads dist metadata without importing torch (see +/// [`TORCH_DIST_PROBE`]), so it works against a bare interpreter. A ROCm wheel +/// carries a `+rocm...` local version, a CUDA wheel `+cu...`; callers judge the +/// build from that label. +fn torch_version(python: &Path) -> Option { let output = std::process::Command::new(python) - .args(["-c", TORCH_HIP_PROBE]) + .args(["-c", TORCH_DIST_PROBE]) .output() .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); let stdout = String::from_utf8_lossy(&output.stdout); - let data: serde_json::Value = serde_json::from_str(stdout.trim()) - .unwrap_or_else(|_| panic!("torch probe returned non-JSON:\n{stdout}")); - data.get("hip") - .is_some_and(|hip| !hip.is_null() && hip.as_str().is_some_and(|s| !s.is_empty())) + let stderr = String::from_utf8_lossy(&output.stderr); + let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { + panic!("torch version probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") + }); + data.get("version") + .and_then(serde_json::Value::as_str) + .map(str::to_owned) } /// Enumerate installed distributions via `importlib.metadata` and emit their names @@ -159,9 +173,13 @@ async fn setup_isolated_runtime(world: &mut E2eWorld) { #[given("the runtime's torch is a ROCm build")] async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { let python = active_runtime_python(world); + let version = torch_version(&python); assert!( - torch_is_rocm(&python), - "baseline runtime torch is not a ROCm build; scenario premise absent ({})", + version + .as_deref() + .is_some_and(|v| v.to_ascii_lowercase().contains("rocm")), + "baseline runtime torch is not a ROCm build; scenario premise absent \ + (torch version: {version:?}, python: {})", python.display() ); assert!( @@ -184,9 +202,13 @@ async fn user_installs_comfyui(world: &mut E2eWorld) { #[then("the runtime's torch is still a ROCm build")] async fn assert_torch_still_rocm(world: &mut E2eWorld) { let python = active_runtime_python(world); + let version = torch_version(&python); assert!( - torch_is_rocm(&python), - "ComfyUI install replaced the runtime's ROCm torch with a non-ROCm build ({})", + version + .as_deref() + .is_some_and(|v| v.to_ascii_lowercase().contains("rocm")), + "ComfyUI install replaced the runtime's ROCm torch with a non-ROCm build \ + (torch version now: {version:?}, python: {})", python.display() ); } From 87654fbfd0550a280dddc2627f9ee8d564490184 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 20 Aug 2026 17:35:12 +0200 Subject: [PATCH 07/26] test(e2e): judge ROCm torch as "not a CUDA build", grounded on the runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The GPU lane showed TheRock's managed torch labels its local version with a git hash (2.11.0+gitd0c8b1f), not +rocm, so requiring "rocm" in the label wrongly failed the baseline on a healthy ROCm runtime. A CUDA wheel, by contrast, always carries +cu. The EAI-8051 corruption replaces the ROCm/TheRock torch with a CUDA one, so judge "still ROCm" as "torch present and NOT a +cu build" — which is exactly the flip the defect causes and needs no positive ROCm label. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index e6b9c52dd6..3e7a80f994 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -85,9 +85,14 @@ fn find_venv_python(root: &Path) -> Option { /// The installed torch distribution's version string, or `None` if no torch /// distribution is installed. Reads dist metadata without importing torch (see -/// [`TORCH_DIST_PROBE`]), so it works against a bare interpreter. A ROCm wheel -/// carries a `+rocm...` local version, a CUDA wheel `+cu...`; callers judge the -/// build from that label. +/// [`TORCH_DIST_PROBE`]), so it works against a bare interpreter. +/// +/// The local-version label identifies the build. A CUDA wheel is unmistakable: +/// `+cu` (e.g. `2.7.0+cu128`). A ROCm build is NOT reliably `+rocm`, though — +/// TheRock's managed torch labels the local version with a git hash +/// (`2.11.0+gitd0c8b1f`, measured on the MI300X lane), so callers judge "ROCm" as +/// "torch is present and is NOT a CUDA build", which is exactly the flip the +/// EAI-8051 corruption would cause. fn torch_version(python: &Path) -> Option { let output = std::process::Command::new(python) .args(["-c", TORCH_DIST_PROBE]) @@ -103,6 +108,15 @@ fn torch_version(python: &Path) -> Option { .map(str::to_owned) } +/// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). The +/// EAI-8051 defect replaces the runtime's ROCm/TheRock torch with a CUDA one, so +/// this turning true is the corruption's signature. A ROCm/TheRock build is +/// deliberately NOT identified positively (its label is a git hash, not `+rocm`): +/// "still ROCm" is "torch present and not a CUDA build". +fn is_cuda_torch(version: &str) -> bool { + version.to_ascii_lowercase().contains("+cu") +} + /// Enumerate installed distributions via `importlib.metadata` and emit their names /// as a JSON array. Used instead of `pip list` because uv-created managed runtimes /// have no `pip` module — `python -m pip` there exits non-zero with empty stdout, @@ -175,10 +189,8 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { let python = active_runtime_python(world); let version = torch_version(&python); assert!( - version - .as_deref() - .is_some_and(|v| v.to_ascii_lowercase().contains("rocm")), - "baseline runtime torch is not a ROCm build; scenario premise absent \ + version.as_deref().is_some_and(|v| !is_cuda_torch(v)), + "baseline runtime torch is absent or already a CUDA build; scenario premise absent \ (torch version: {version:?}, python: {})", python.display() ); @@ -204,10 +216,8 @@ async fn assert_torch_still_rocm(world: &mut E2eWorld) { let python = active_runtime_python(world); let version = torch_version(&python); assert!( - version - .as_deref() - .is_some_and(|v| v.to_ascii_lowercase().contains("rocm")), - "ComfyUI install replaced the runtime's ROCm torch with a non-ROCm build \ + version.as_deref().is_some_and(|v| !is_cuda_torch(v)), + "ComfyUI install replaced the runtime's ROCm torch with a CUDA build \ (torch version now: {version:?}, python: {})", python.display() ); From 2faf11028d5ef1dc2cac34a5fadf37fa96a77e61 Mon Sep 17 00:00:00 2001 From: fredespi Date: Fri, 21 Aug 2026 17:31:36 +0200 Subject: [PATCH 08/26] test(e2e): require the runtime's torch version to be unchanged (EAI-8051) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback (tomastola): "torch is still a ROCm build" only checked the post-install version was not a `+cu` CUDA build, which the MI300X run showed is too weak — ComfyUI replaced torch==2.11.0+gitd0c8b1f with plain torch==2.13.0 and that step PASSED; only the nvidia-* assertion caught the corruption. A CPU-only or non-`+cu` CUDA wheel would satisfy both torch assertions. Capture the exact baseline torch version before the install and require the post-install version to be byte-for-byte identical. That directly tests the invariant that ComfyUI must not replace the managed runtime's torch; the nvidia-* assertion stays as an additional diagnostic. The baseline premise check (is_cuda_torch) is unchanged. Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e.rs | 7 +++++ tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 27 ++++++++++++------- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index e49d3c8eaf..43d07fe56e 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -82,6 +82,12 @@ pub struct E2eWorld { /// launch step knows to pass `--chat-mock` (deterministic offline agent, no /// endpoint detection) instead of driving the real detection/consent path. pub chat_use_mock: bool, + /// The managed runtime's torch distribution version captured BEFORE a ComfyUI + /// install (EAI-8051), so the post-install step can require it to be unchanged. + /// A ComfyUI install must not replace the runtime's torch at all; comparing the + /// exact version catches a swap to any other build (CUDA `+cu`, a plain + /// `2.13.0` CPU wheel, etc.), which a "not a CUDA build" check alone would miss. + pub comfyui_baseline_torch: Option, /// Per-scenario release-lifecycle state (packaging dirs, signing keys, install /// dir, captured logs). `Some` only for `@lifecycle` scenarios; all its paths /// are rooted in `isolated_root` so teardown removes them with the temp dir. @@ -191,6 +197,7 @@ impl Default for E2eWorld { expect_xfail: false, tui: None, chat_use_mock: false, + comfyui_baseline_torch: None, lifecycle: None, } } diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 3e7a80f994..1c8ed6dbd7 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -108,11 +108,12 @@ fn torch_version(python: &Path) -> Option { .map(str::to_owned) } -/// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). The -/// EAI-8051 defect replaces the runtime's ROCm/TheRock torch with a CUDA one, so -/// this turning true is the corruption's signature. A ROCm/TheRock build is -/// deliberately NOT identified positively (its label is a git hash, not `+rocm`): -/// "still ROCm" is "torch present and not a CUDA build". +/// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). Used +/// only for the baseline PREMISE check — that the runtime didn't start out on a +/// CUDA torch. The post-install invariant is stronger: the torch version must be +/// byte-for-byte unchanged (a ComfyUI install must not replace the runtime's torch +/// at all), which also catches a swap to a plain non-`+cu` wheel (e.g. the observed +/// `2.11.0+gitd0c8b1f` → `2.13.0`) that this label check alone would miss. fn is_cuda_torch(version: &str) -> bool { version.to_ascii_lowercase().contains("+cu") } @@ -198,6 +199,9 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { nvidia_distributions(&python).is_empty(), "runtime already has nvidia-* distributions before ComfyUI install; premise absent" ); + // Record the exact baseline version so the post-install step can require it to + // be unchanged (see `assert_torch_still_rocm`). + world.comfyui_baseline_torch = version; } #[when("the user installs ComfyUI")] @@ -215,10 +219,15 @@ async fn user_installs_comfyui(world: &mut E2eWorld) { async fn assert_torch_still_rocm(world: &mut E2eWorld) { let python = active_runtime_python(world); let version = torch_version(&python); - assert!( - version.as_deref().is_some_and(|v| !is_cuda_torch(v)), - "ComfyUI install replaced the runtime's ROCm torch with a CUDA build \ - (torch version now: {version:?}, python: {})", + let baseline = world + .comfyui_baseline_torch + .as_deref() + .expect("no baseline torch version was captured"); + assert_eq!( + version.as_deref(), + Some(baseline), + "ComfyUI install replaced the managed runtime's torch \ + (before: {baseline}, after: {version:?}, python: {})", python.display() ); } From 1fe56c3eb1fead0ebf44b1cb0d290f9a023b9d6a Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 25 Aug 2026 23:47:43 +0200 Subject: [PATCH 09/26] test(e2e): graduate the EAI-8051 scenario to expected-pass The ComfyUI fix landed in 3e59435a (#298), pinning the managed runtime's torch stack through uv constraints. The scenario now asserts behavior main implements; remove its xfail row so a pass is not a fatal stale XPASS. Keep the scenario as the regression guard. Signed-off-by: fredespi --- tests/e2e-cucumber/expectations.toml | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index f709da802b..beaa49aa48 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -221,15 +221,9 @@ when = {} bug = "EAI-8053" reason = "install driver --dkms prefixes sudo even as root, so it fails with `sudo: not found` on a root host without sudo." -# --- EAI-8051: `comfyui install` installs ComfyUI's dependencies into the managed -# ROCm runtime with no package-index scoping. torch/torchvision/torchaudio are -# filtered out, but a transitive dependency can still pull CUDA `nvidia-*` wheels -# into the runtime and displace its ROCm torch, so installing an optional app -# leaves the machine's base runtime a CUDA build with no AMD GPU support. Runs only -# on the nightly GPU lane (@requires-gpu @nightly) against an isolated, throwaway -# runtime it may corrupt. Unconditional where it runs, so `when = {}`. -# Remove when the install can no longer alter the runtime's torch stack. --- -[["comfyui-install-preserves-the-rocm-runtime"]] -when = {} -bug = "EAI-8051" -reason = "comfyui install can pull CUDA nvidia-* wheels into the managed ROCm runtime, replacing its ROCm torch." +# EAI-8051 had a row here — `comfyui install` pulling CUDA `nvidia-*` wheels into +# the managed ROCm runtime and displacing its ROCm torch. The fix landed in +# 3e59435a (#298): the install now pins the runtime's torch stack with a `uv +# --constraint` file, so a transitive dependency can no longer resolve torch from +# PyPI. `comfyui-install-preserves-the-rocm-runtime` therefore EXPECT-PASSES now; +# keeping the row would make it a stale XPASS and fail the lane. From 263224fec26e93bc5224593b2705aab76f9daa87 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 10 Sep 2026 09:28:17 +0200 Subject: [PATCH 10/26] test(e2e): require the ComfyUI install to actually install something A zero exit code alone still permits a vacuous pass: `comfyui::install` guards the whole `uv` step with `if !packages.is_empty()`, so an empty filtered requirement list skips the dependency install and still exits 0, leaving the runtime untouched and every runtime-preservation assertion passing having exercised nothing. Capture the runtime's distribution set alongside the torch version in the baseline step, and require it to have grown after the install. Asserting growth rather than naming an expected ComfyUI dependency keeps the check from rotting as ComfyUI's requirements drift upstream, while still establishing the premise the invariants need. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 8 ++- tests/e2e-cucumber/tests/e2e.rs | 7 +++ tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 60 ++++++++++++++++--- 3 files changed, 64 insertions(+), 11 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 909294b982..b0a72fccf7 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -12,8 +12,11 @@ Feature: ComfyUI application management # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, # so the scenario also requires that: a bail-out (no runtime, download failure) # would otherwise leave the runtime trivially unchanged and report green having - # installed nothing. The runtime-health checks are the ADDITIONAL contract on top - # of a successful install, not a substitute for it. + # installed nothing. It further requires the runtime's package set to have GROWN, + # because a zero exit code alone still permits an empty filtered requirement list, + # which skips the dependency install outright. Together those two make the + # runtime-health checks the ADDITIONAL contract on top of an install that really + # happened, not a substitute for it. # # Genuinely destructive and expensive: it needs a real managed runtime (a # multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind @@ -30,5 +33,6 @@ Feature: ComfyUI application management And the runtime's torch is a ROCm build When the user installs ComfyUI Then the install succeeds + And ComfyUI's dependencies were installed into the runtime And the runtime's torch is still a ROCm build And no CUDA nvidia packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 375f367d5c..d8b04f1961 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -97,6 +97,12 @@ pub struct E2eWorld { /// exact version catches a swap to any other build (CUDA `+cu`, a plain /// `2.13.0` CPU wheel, etc.), which a "not a CUDA build" check alone would miss. pub comfyui_baseline_torch: Option, + /// The managed runtime's installed distribution names captured BEFORE a ComfyUI + /// install (EAI-8051), so the post-install step can require the set to have + /// grown. Without it a run that installs nothing — `comfyui::install` skips the + /// whole `uv` step on an empty requirement list and still exits 0 — would pass + /// every runtime-preservation assertion having exercised nothing. + pub comfyui_baseline_distributions: Option>, /// Per-scenario release-lifecycle state (packaging dirs, signing keys, install /// dir, captured logs). `Some` only for `@lifecycle` scenarios; all its paths /// are rooted in `isolated_root` so teardown removes them with the temp dir. @@ -228,6 +234,7 @@ impl Default for E2eWorld { tui: None, chat_use_mock: false, comfyui_baseline_torch: None, + comfyui_baseline_distributions: None, lifecycle: None, } } diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 4e19600f93..d8c0448c0f 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -140,12 +140,11 @@ const DISTRIBUTIONS_PROBE: &str = "import json,sys\n\ \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ sys.stdout.write(json.dumps(out))\n"; -/// The `nvidia-*` CUDA distributions installed in the interpreter's environment. -/// A ROCm runtime should have none; ComfyUI's install dragging any in is the -/// EAI-8051 defect. Panics if the interpreter cannot be run or the probe reports -/// an error — a probe that cannot enumerate packages must NOT read as "no nvidia -/// packages", which would pass the contract on a runtime it never actually checked. -fn nvidia_distributions(python: &Path) -> Vec { +/// Every distribution name installed in the interpreter's environment, sorted. +/// Panics if the interpreter cannot be run or the probe reports an error — a probe +/// that cannot enumerate packages must NOT read as an empty environment, which +/// would pass the nvidia check on a runtime it never actually inspected. +fn installed_distributions(python: &Path) -> Vec { let output = std::process::Command::new(python) .args(["-c", DISTRIBUTIONS_PROBE]) .output() @@ -168,11 +167,20 @@ fn nvidia_distributions(python: &Path) -> Vec { names .iter() .filter_map(serde_json::Value::as_str) - .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) .map(str::to_owned) .collect() } +/// The `nvidia-*` CUDA distributions installed in the interpreter's environment. +/// A ROCm runtime should have none; ComfyUI's install dragging any in is the +/// EAI-8051 defect. +fn nvidia_distributions(python: &Path) -> Vec { + installed_distributions(python) + .into_iter() + .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) + .collect() +} + #[given("an isolated machine with a managed ROCm runtime")] async fn setup_isolated_runtime(world: &mut E2eWorld) { // DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may @@ -200,13 +208,19 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { (torch version: {version:?}, python: {})", python.display() ); + let distributions = installed_distributions(&python); assert!( - nvidia_distributions(&python).is_empty(), + !distributions + .iter() + .any(|name| name.to_ascii_lowercase().starts_with("nvidia-")), "runtime already has nvidia-* distributions before ComfyUI install; premise absent" ); // Record the exact baseline version so the post-install step can require it to - // be unchanged (see `assert_torch_still_rocm`). + // be unchanged (see `assert_torch_still_rocm`), and the baseline package set so + // it can require the install to have actually added something (see + // `assert_dependencies_installed`). world.comfyui_baseline_torch = version; + world.comfyui_baseline_distributions = Some(distributions); } #[when("the user installs ComfyUI")] @@ -242,6 +256,34 @@ async fn assert_install_succeeded(world: &mut E2eWorld) { ); } +#[then("ComfyUI's dependencies were installed into the runtime")] +async fn assert_dependencies_installed(world: &mut E2eWorld) { + // Closes the last vacuity path a zero exit code leaves open. `comfyui::install` + // guards the whole `uv` install with `if !packages.is_empty()` + // (`apps/rocm/src/comfyui.rs`), so an empty filtered requirement list skips it + // and still exits 0 — leaving the runtime untouched and every invariant below + // passing having installed nothing. + // + // Deliberately asserts the package set GREW rather than naming an expected + // dependency: ComfyUI's requirements drift upstream independently of this + // contract, so a named package would rot, while "the install put something in + // the runtime" is exactly the premise the invariants need and cannot go stale. + let python = active_runtime_python(world); + let baseline = world + .comfyui_baseline_distributions + .as_ref() + .expect("no baseline distribution set was captured"); + let after = installed_distributions(&python); + assert!( + after.iter().any(|name| !baseline.contains(name)), + "ComfyUI install added no distributions to the runtime, so it installed \ + nothing and the runtime-preservation checks would pass vacuously \ + ({} distributions before and after, python: {})", + after.len(), + python.display() + ); +} + #[then("the runtime's torch is still a ROCm build")] async fn assert_torch_still_rocm(world: &mut E2eWorld) { let python = active_runtime_python(world); From 52cc71c726a3b02c5c200226ef51856ee764b7d8 Mon Sep 17 00:00:00 2001 From: "fredespi-agent-app[bot]" <266462846+fredespi-agent-app[bot]@users.noreply.github.com> Date: Wed, 7 Oct 2026 11:12:32 +0200 Subject: [PATCH 11/26] test(e2e): address review of the comfyui-05 scenario Past-tense the #298 comment, document that the rc check catches a #298 revert, rename the torch step to 'unchanged', surface probe errors, assert a single runtime root, and fix the walk doc. Signed-off-by: fredespi-agent-app[bot] <266462846+fredespi-agent-app[bot]@users.noreply.github.com> Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 16 +++--- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 52 +++++++++++++------ 2 files changed, 44 insertions(+), 24 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 137824ea90..7b34241576 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -88,16 +88,18 @@ Feature: ComfyUI install reports progress and makes failures actionable And the final terminal screen shows no ComfyUI download spinner line # EAI-8051: `rocm comfyui install` installs ComfyUI's dependencies INTO the - # machine's managed ROCm runtime. It filters torch/torchvision/torchaudio out of - # ComfyUI's requirements, but nothing scopes the package index, so a transitive - # dependency can still pull CUDA `nvidia-*` wheels into the runtime and displace - # its ROCm torch. The runtime the whole machine serves models with is then a CUDA - # build with no AMD GPU support — installing an optional app broke the base. + # machine's managed ROCm runtime. Before #298 nothing scoped the package index, + # so a transitive dependency could pull CUDA `nvidia-*` wheels into the runtime + # and displace its ROCm torch, leaving a CUDA build with no AMD GPU support -- + # installing an optional app broke the base. #298 now pins the torch stack with + # a `uv --constraint` file; this scenario guards against that regressing. # # The contract: after installing an optional app, the machine's ROCm runtime must # still be a ROCm runtime — its torch is byte-for-byte the build it was before and # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, - # so the scenario also requires that: a bail-out (no runtime, download failure) + # so the scenario also requires that. It is also the step that catches a revert + # of #298 (the post-install GPU probe then bails and the install exits non-zero), + # so do not relax it as a mere premise: a bail-out (no runtime, download failure) # would otherwise leave the runtime trivially unchanged and report green having # installed nothing. It further requires the runtime's package set to have GROWN, # because a zero exit code alone still permits an empty filtered requirement list, @@ -121,5 +123,5 @@ Feature: ComfyUI install reports progress and makes failures actionable When the user installs ComfyUI into the isolated runtime Then the install succeeds And ComfyUI's dependencies were installed into the runtime - And the runtime's torch is still a ROCm build + And the runtime's torch is unchanged And no CUDA nvidia packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index cb44e9ea94..7393c0a5b9 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -46,6 +46,12 @@ //! entirely — this scenario is about the download spinner, not the //! dependency install already covered above. //! +//! `comfyui-05` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real, +//! isolated managed runtime and asserts the runtime's torch is unchanged, no +//! `nvidia-*` distributions appeared, and the install really added packages. It +//! uses its own When phrase because the planted-runtime step above hard-codes +//! `--runtime-id`. +//! //! Black-box throughout: the planted registry manifests are plain JSON matching //! the CLI's on-disk schema, not typed imports from the product crates. @@ -600,13 +606,18 @@ const TORCH_DIST_PROBE: &str = "import json,sys\n\ /// prints `install_root:` for every installed runtime unconditionally. fn active_runtime_python(world: &E2eWorld) -> PathBuf { let (listing, _, _) = crate::run_rocm(world, &["runtimes", "list"]); - let root = listing + let roots: Vec<&str> = listing .lines() - .find_map(|l| l.trim().strip_prefix("install_root:")) - .map_or_else( - || panic!("no 'install_root:' line in `runtimes list`:\n{listing}"), - str::trim, - ); + .filter_map(|l| l.trim().strip_prefix("install_root:")) + .map(str::trim) + .collect(); + assert_eq!( + roots.len(), + 1, + "expected exactly one installed runtime in the isolated world, found {}:\n{listing}", + roots.len() + ); + let root = roots[0]; find_venv_python(Path::new(root)).unwrap_or_else(|| { panic!("could not locate a venv python under the runtime install_root {root}") }) @@ -617,7 +628,7 @@ fn active_runtime_python(world: &E2eWorld) -> PathBuf { /// The documented layout is probed FIRST: `root` itself is the initial frontier /// entry, so a managed runtime — whose interpreter is exactly `/bin/ /// python` — is found on the first iteration with no directory traversal at all. -/// The breadth walk below is only a fallback for a tree that does not match, and +/// The depth-first walk below is only a fallback for a tree that does not match, and /// is depth-capped so a pathological one cannot hang the scenario rather than /// being a cost the normal path pays. fn find_venv_python(root: &Path) -> Option { @@ -656,7 +667,7 @@ fn find_venv_python(root: &Path) -> Option { /// (`2.11.0+gitd0c8b1f`, measured on the MI300X lane), so callers judge "ROCm" as /// "torch is present and is NOT a CUDA build", which is exactly the flip the /// EAI-8051 corruption would cause. -fn torch_version(python: &Path) -> Option { +fn torch_version(python: &Path) -> Result { let output = std::process::Command::new(python) .args(["-c", TORCH_DIST_PROBE]) .output() @@ -666,9 +677,14 @@ fn torch_version(python: &Path) -> Option { let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { panic!("torch version probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") }); - data.get("version") - .and_then(serde_json::Value::as_str) - .map(str::to_owned) + match data.get("version").and_then(serde_json::Value::as_str) { + Some(v) => Ok(v.to_owned()), + None => Err(data + .get("error") + .and_then(serde_json::Value::as_str) + .unwrap_or("probe reported no version and no error") + .to_owned()), + } } /// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). Used @@ -761,7 +777,7 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { let python = active_runtime_python(world); let version = torch_version(&python); assert!( - version.as_deref().is_some_and(|v| !is_cuda_torch(v)), + version.as_ref().is_ok_and(|v| !is_cuda_torch(v)), "baseline runtime torch is absent or already a CUDA build; scenario premise absent \ (torch version: {version:?}, python: {})", python.display() @@ -777,7 +793,7 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { // be unchanged (see `assert_torch_still_rocm`), and the baseline package set so // it can require the install to have actually added something (see // `assert_dependencies_installed`). - world.comfyui_baseline_torch = version; + world.comfyui_baseline_torch = version.ok(); world.comfyui_baseline_distributions = Some(distributions); } @@ -798,8 +814,10 @@ async fn assert_install_succeeded(world: &mut E2eWorld) { // several paths (no managed runtime, a runtime that isn't `ready`, a non-wheel // format, a failed source download or `uv` acquisition); on any of those the // runtime is TRIVIALLY unchanged and the torch/nvidia assertions would pass - // having exercised nothing. Since #298 the install exits 0 on the measured - // MI300X lane, so requiring that is what makes the rest evidence of anything. + // having exercised nothing. It is also the check that catches a revert of + // #298: without the torch constraint, the post-install GPU probe in + // `comfyui::install` bails, so the install exits non-zero and the torch and + // nvidia steps never run. Do not relax it as "just a premise". let rc = world.cli_rc.expect("no ComfyUI install was run"); assert_eq!( rc, @@ -842,7 +860,7 @@ async fn assert_dependencies_installed(world: &mut E2eWorld) { ); } -#[then("the runtime's torch is still a ROCm build")] +#[then("the runtime's torch is unchanged")] async fn assert_torch_still_rocm(world: &mut E2eWorld) { let python = active_runtime_python(world); let version = torch_version(&python); @@ -852,7 +870,7 @@ async fn assert_torch_still_rocm(world: &mut E2eWorld) { .expect("no baseline torch version was captured"); assert_eq!( version.as_deref(), - Some(baseline), + Ok(baseline), "ComfyUI install replaced the managed runtime's torch \ (before: {baseline}, after: {version:?}, python: {})", python.display() From f256f7ed780ac7bb953ec60b729d2a6a67662431 Mon Sep 17 00:00:00 2001 From: fredespi Date: Wed, 7 Oct 2026 13:08:14 +0200 Subject: [PATCH 12/26] test(e2e): tighten comfyui-05 names, docs and install flags per review Reuse the argument-less install When step, pass --yes to install sdk, rename the premise/identity steps to what they check, fix two doc comments. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 6 +-- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 49 +++++++------------ 2 files changed, 22 insertions(+), 33 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 7b34241576..dc89d9f311 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -111,15 +111,15 @@ Feature: ComfyUI install reports progress and makes failures actionable # multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind # @nightly, against this scenario's own isolated runtime prefix (it must never # share a runtime tree with other scenarios — it may corrupt it). Gated - # @requires-gpu @nightly, matching runtime-install-sdk-active, the other scenario - # that does a real `install sdk`. NOT @lifecycle: that tag is for OS-mutating + # @requires-gpu @nightly, like runtime-install-sdk-active, which also does a + # real `install sdk`. NOT @lifecycle: that tag is for OS-mutating # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so # combining it with @nightly would make this scenario unreachable on every lane; # this mutates only its own isolated runtime prefix, not the OS. @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly Scenario: comfyui-05 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given an isolated machine with a managed ROCm runtime - And the runtime's torch is a ROCm build + And the runtime's torch is not a CUDA build When the user installs ComfyUI into the isolated runtime Then the install succeeds And ComfyUI's dependencies were installed into the runtime diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 7393c0a5b9..12b4a796ad 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -49,8 +49,8 @@ //! `comfyui-05` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real, //! isolated managed runtime and asserts the runtime's torch is unchanged, no //! `nvidia-*` distributions appeared, and the install really added packages. It -//! uses its own When phrase because the planted-runtime step above hard-codes -//! `--runtime-id`. +//! reuses the planted-runtime scenarios' argument-less `comfyui install` When step +//! under a second phrase. //! //! Black-box throughout: the planted registry manifests are plain JSON matching //! the CLI's on-disk schema, not typed imports from the product crates. @@ -381,6 +381,7 @@ async fn plant_two_ready_runtimes(world: &mut E2eWorld) { } #[when("the user installs ComfyUI without choosing a runtime")] +#[when("the user installs ComfyUI into the isolated runtime")] async fn install_comfyui_without_runtime(world: &mut E2eWorld) { let (stdout, stderr, rc) = crate::run_rocm(world, &["comfyui", "install"]); world.cli_output = Some(stdout); @@ -573,17 +574,16 @@ async fn assert_comfyui_spinner_line_cleared(world: &mut E2eWorld) { // --- comfyui-05: installing ComfyUI must not damage the managed ROCm runtime --- /// Report the installed torch distribution's version string via `importlib.metadata` -/// — WITHOUT importing torch. Emits JSON `{version}` (e.g. `2.7.0+rocm6.4` for a -/// ROCm build, `2.7.0+cu128` for a CUDA build) or `{error}` when no torch +/// — WITHOUT importing torch. Emits JSON `{version}` (e.g. `2.11.0+gitd0c8b1f` for +/// TheRock's ROCm build, `2.7.0+cu128` for a CUDA build) or `{error}` when no torch /// distribution is installed. /// /// Deliberately does not `import torch`: importing it loads the ROCm/CUDA shared /// libraries, which need the runtime's `LD_LIBRARY_PATH`/`ROCM_PATH` set up (the /// product runs its own torch probe *with* that env, ours runs the interpreter -/// bare). The `+rocm` / `+cu` local-version label in the dist metadata is the -/// definitive ROCm-vs-CUDA discriminator and is readable with no native load — so -/// a bare interpreter suffices, and a runtime whose torch is present but whose -/// native libs aren't on our env no longer reads as "not a ROCm build". +/// bare). The version string in the dist metadata is readable with no native +/// load, so a bare interpreter suffices. Only a `+cu` label reliably marks a CUDA +/// build; TheRock's ROCm torch is labelled with a git hash, not `+rocm`. const TORCH_DIST_PROBE: &str = "import json,sys\n\ from importlib import metadata\n\ out={}\n\ @@ -604,7 +604,7 @@ const TORCH_DIST_PROBE: &str = "import json,sys\n\ /// active, so it is not a reliable source for the install root (this cost a GPU /// dispatch — the scenario panicked on a missing `Folder:` there). `runtimes list` /// prints `install_root:` for every installed runtime unconditionally. -fn active_runtime_python(world: &E2eWorld) -> PathBuf { +fn sole_runtime_python(world: &E2eWorld) -> PathBuf { let (listing, _, _) = crate::run_rocm(world, &["runtimes", "list"]); let roots: Vec<&str> = listing .lines() @@ -763,18 +763,18 @@ async fn setup_isolated_runtime(world: &mut E2eWorld) { // so a plain `install sdk` here lands in this scenario's own tree. let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); if stdout.contains("installed: none") { - crate::run_rocm_ok(world, &["install", "sdk"]); + crate::run_rocm_ok(world, &["install", "sdk", "--yes"]); } let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); assert!( !stdout.contains("installed: none"), - "no managed runtime is active after install:\n{stdout}" + "no managed runtime is installed after `install sdk`:\n{stdout}" ); } -#[given("the runtime's torch is a ROCm build")] -async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { - let python = active_runtime_python(world); +#[given("the runtime's torch is not a CUDA build")] +async fn assert_baseline_torch_not_cuda(world: &mut E2eWorld) { + let python = sole_runtime_python(world); let version = torch_version(&python); assert!( version.as_ref().is_ok_and(|v| !is_cuda_torch(v)), @@ -790,24 +790,13 @@ async fn assert_baseline_rocm_torch(world: &mut E2eWorld) { "runtime already has nvidia-* distributions before ComfyUI install; premise absent" ); // Record the exact baseline version so the post-install step can require it to - // be unchanged (see `assert_torch_still_rocm`), and the baseline package set so + // be unchanged (see `assert_torch_unchanged`), and the baseline package set so // it can require the install to have actually added something (see // `assert_dependencies_installed`). world.comfyui_baseline_torch = version.ok(); world.comfyui_baseline_distributions = Some(distributions); } -#[when("the user installs ComfyUI into the isolated runtime")] -async fn user_installs_comfyui(world: &mut E2eWorld) { - // Capture the outcome rather than asserting here: the exit code is checked by - // its own Then step, so a failure is reported as that step failing rather than - // as a mid-scenario panic in the action. - let (stdout, stderr, rc) = crate::run_rocm(world, &["comfyui", "install"]); - world.cli_output = Some(stdout); - world.cli_stderr = Some(stderr); - world.cli_rc = Some(rc); -} - #[then("the install succeeds")] async fn assert_install_succeeded(world: &mut E2eWorld) { // The premise for every invariant below. `comfyui::install` bails early on @@ -844,7 +833,7 @@ async fn assert_dependencies_installed(world: &mut E2eWorld) { // dependency: ComfyUI's requirements drift upstream independently of this // contract, so a named package would rot, while "the install put something in // the runtime" is exactly the premise the invariants need and cannot go stale. - let python = active_runtime_python(world); + let python = sole_runtime_python(world); let baseline = world .comfyui_baseline_distributions .as_ref() @@ -861,8 +850,8 @@ async fn assert_dependencies_installed(world: &mut E2eWorld) { } #[then("the runtime's torch is unchanged")] -async fn assert_torch_still_rocm(world: &mut E2eWorld) { - let python = active_runtime_python(world); +async fn assert_torch_unchanged(world: &mut E2eWorld) { + let python = sole_runtime_python(world); let version = torch_version(&python); let baseline = world .comfyui_baseline_torch @@ -879,7 +868,7 @@ async fn assert_torch_still_rocm(world: &mut E2eWorld) { #[then("no CUDA nvidia packages were added to the runtime")] async fn assert_no_nvidia_packages(world: &mut E2eWorld) { - let python = active_runtime_python(world); + let python = sole_runtime_python(world); let nvidia = nvidia_distributions(&python); assert!( nvidia.is_empty(), From 1b23d34857ac45ca56967baa4030b1372c3bb53c Mon Sep 17 00:00:00 2001 From: fredespi Date: Wed, 7 Oct 2026 15:24:17 +0200 Subject: [PATCH 13/26] test(e2e): compare the whole torch stack and use user-facing step text Snapshot torch, torchvision and torchaudio before and after the install, drop harness wording from the comfyui-05 steps, fix the layout doc. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 6 +- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 69 +++++++++++++------ 2 files changed, 52 insertions(+), 23 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index dc89d9f311..f1c817a1cd 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -118,10 +118,10 @@ Feature: ComfyUI install reports progress and makes failures actionable # this mutates only its own isolated runtime prefix, not the OS. @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly Scenario: comfyui-05 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one - Given an isolated machine with a managed ROCm runtime + Given a machine with a managed ROCm runtime And the runtime's torch is not a CUDA build - When the user installs ComfyUI into the isolated runtime + When the user installs ComfyUI without choosing a runtime Then the install succeeds And ComfyUI's dependencies were installed into the runtime - And the runtime's torch is unchanged + And the runtime's torch stack is unchanged And no CUDA nvidia packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 12b4a796ad..6dcb24a2d2 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -49,8 +49,7 @@ //! `comfyui-05` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real, //! isolated managed runtime and asserts the runtime's torch is unchanged, no //! `nvidia-*` distributions appeared, and the install really added packages. It -//! reuses the planted-runtime scenarios' argument-less `comfyui install` When step -//! under a second phrase. +//! reuses `comfyui-03`'s argument-less `comfyui install` When step. //! //! Black-box throughout: the planted registry manifests are plain JSON matching //! the CLI's on-disk schema, not typed imports from the product crates. @@ -381,7 +380,6 @@ async fn plant_two_ready_runtimes(world: &mut E2eWorld) { } #[when("the user installs ComfyUI without choosing a runtime")] -#[when("the user installs ComfyUI into the isolated runtime")] async fn install_comfyui_without_runtime(world: &mut E2eWorld) { let (stdout, stderr, rc) = crate::run_rocm(world, &["comfyui", "install"]); world.cli_output = Some(stdout); @@ -596,8 +594,8 @@ const TORCH_DIST_PROBE: &str = "import json,sys\n\ /// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an /// `install_root: ` line for each installed runtime; the interpreter lives /// under a `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. -/// The exact env sub-layout is an internal detail, so search for the interpreter -/// rather than reconstruct the path — black-box, and tolerant of layout changes. +/// The interpreter is expected at `/bin/python`; see +/// [`find_venv_python`]. /// /// Reads `runtimes list` rather than `examine`: examine only prints a `Folder:` /// line for the *active* runtime and takes a different branch when none is marked @@ -625,12 +623,10 @@ fn sole_runtime_python(world: &E2eWorld) -> PathBuf { /// Locate a `bin/python` (Unix) or `Scripts/python.exe` (Windows) under `root`. /// -/// The documented layout is probed FIRST: `root` itself is the initial frontier -/// entry, so a managed runtime — whose interpreter is exactly `/bin/ -/// python` — is found on the first iteration with no directory traversal at all. -/// The depth-first walk below is only a fallback for a tree that does not match, and -/// is depth-capped so a pathological one cannot hang the scenario rather than -/// being a cost the normal path pays. +/// The interpreter is expected exactly at `/bin/python` (the product +/// creates the venv at `install_root`). `root` is the initial frontier entry, so +/// that layout is found on the first iteration with no traversal. The depth-first +/// walk below is only a depth-capped tolerance fallback for a tree that differs. fn find_venv_python(root: &Path) -> Option { #[cfg(windows)] let (bin, exe) = ("Scripts", "python.exe"); @@ -687,6 +683,39 @@ fn torch_version(python: &Path) -> Result { } } +const TORCH_STACK_PROBE: &str = "import json,sys\n\ + from importlib import metadata\n\ + out={}\n\ + for n in ('torch','torchvision','torchaudio'):\n\ + \x20 try:\n\ + \x20 out[n]=metadata.version(n)\n\ + \x20 except metadata.PackageNotFoundError:\n\ + \x20 out[n]=None\n\ + sys.stdout.write(json.dumps(out))\n"; + +/// Exact versions of the whole torch stack (`torch`, `torchvision`, `torchaudio`), +/// formatted `name=version` (or `name=absent`), one per line. The product pins all +/// three, so the scenario compares all three. +fn torch_stack_versions(python: &Path) -> String { + let output = std::process::Command::new(python) + .args(["-c", TORCH_STACK_PROBE]) + .output() + .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { + panic!("torch stack probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") + }); + ["torch", "torchvision", "torchaudio"] + .iter() + .map(|n| { + let v = data.get(*n).and_then(serde_json::Value::as_str); + format!("{n}={}", v.unwrap_or("absent")) + }) + .collect::>() + .join("\n") +} + /// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). Used /// only for the baseline PREMISE check — that the runtime didn't start out on a /// CUDA torch. The post-install invariant is stronger: the torch version must be @@ -755,7 +784,7 @@ fn nvidia_distributions(python: &Path) -> Vec { .collect() } -#[given("an isolated machine with a managed ROCm runtime")] +#[given("a machine with a managed ROCm runtime")] async fn setup_isolated_runtime(world: &mut E2eWorld) { // DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may // corrupt the runtime (that is the bug it pins), so it must own a private, @@ -793,7 +822,7 @@ async fn assert_baseline_torch_not_cuda(world: &mut E2eWorld) { // be unchanged (see `assert_torch_unchanged`), and the baseline package set so // it can require the install to have actually added something (see // `assert_dependencies_installed`). - world.comfyui_baseline_torch = version.ok(); + world.comfyui_baseline_torch = Some(torch_stack_versions(&python)); world.comfyui_baseline_distributions = Some(distributions); } @@ -849,19 +878,19 @@ async fn assert_dependencies_installed(world: &mut E2eWorld) { ); } -#[then("the runtime's torch is unchanged")] +#[then("the runtime's torch stack is unchanged")] async fn assert_torch_unchanged(world: &mut E2eWorld) { let python = sole_runtime_python(world); - let version = torch_version(&python); + let after = torch_stack_versions(&python); let baseline = world .comfyui_baseline_torch .as_deref() - .expect("no baseline torch version was captured"); + .expect("no baseline torch stack was captured"); assert_eq!( - version.as_deref(), - Ok(baseline), - "ComfyUI install replaced the managed runtime's torch \ - (before: {baseline}, after: {version:?}, python: {})", + after, + baseline, + "ComfyUI install replaced part of the managed runtime's torch stack \ + (before: [{baseline}], after: [{after}], python: {})", python.display() ); } From 15e7c5a10849834f0b868bdfe2950073cf0ffb34 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 8 Oct 2026 10:03:39 +0200 Subject: [PATCH 14/26] test(e2e): simplify comfyui-05 probes and correct its docs One torch-stack probe for premise and baseline, unconditional sdk install, accurate growth message, ComfyUI-specific success step, docs matching the stack comparison. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 8 +- tests/e2e-cucumber/tests/e2e.rs | 13 ++- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 95 ++++--------------- 3 files changed, 26 insertions(+), 90 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index f1c817a1cd..b3cbe6cb4d 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -88,14 +88,14 @@ Feature: ComfyUI install reports progress and makes failures actionable And the final terminal screen shows no ComfyUI download spinner line # EAI-8051: `rocm comfyui install` installs ComfyUI's dependencies INTO the - # machine's managed ROCm runtime. Before #298 nothing scoped the package index, + # machine's managed ROCm runtime. Before #298 nothing constrained the torch stack, # so a transitive dependency could pull CUDA `nvidia-*` wheels into the runtime # and displace its ROCm torch, leaving a CUDA build with no AMD GPU support -- # installing an optional app broke the base. #298 now pins the torch stack with # a `uv --constraint` file; this scenario guards against that regressing. # # The contract: after installing an optional app, the machine's ROCm runtime must - # still be a ROCm runtime — its torch is byte-for-byte the build it was before and + # still be a ROCm runtime — its torch, torchvision and torchaudio versions are unchanged and # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, # so the scenario also requires that. It is also the step that catches a revert # of #298 (the post-install GPU probe then bails and the install exits non-zero), @@ -119,9 +119,9 @@ Feature: ComfyUI install reports progress and makes failures actionable @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly Scenario: comfyui-05 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given a machine with a managed ROCm runtime - And the runtime's torch is not a CUDA build + And the runtime has torch and no CUDA packages When the user installs ComfyUI without choosing a runtime - Then the install succeeds + Then the ComfyUI install succeeds And ComfyUI's dependencies were installed into the runtime And the runtime's torch stack is unchanged And no CUDA nvidia packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 9347c4fa4a..ae6a2d565e 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -136,12 +136,11 @@ pub struct E2eWorld { /// launch step knows to pass `--chat-mock` (deterministic offline agent, no /// endpoint detection) instead of driving the real detection/consent path. pub chat_use_mock: bool, - /// The managed runtime's torch distribution version captured BEFORE a ComfyUI - /// install (EAI-8051), so the post-install step can require it to be unchanged. - /// A ComfyUI install must not replace the runtime's torch at all; comparing the - /// exact version catches a swap to any other build (CUDA `+cu`, a plain - /// `2.13.0` CPU wheel, etc.), which a "not a CUDA build" check alone would miss. - pub comfyui_baseline_torch: Option, + /// The managed runtime's torch/torchvision/torchaudio versions captured BEFORE a + /// ComfyUI install (EAI-8051), so the post-install step can require them to be + /// unchanged. Comparing exact versions catches a swap to any other build, labelled + /// or not (e.g. stock PyPI `2.13.0`, which is CUDA yet carries no `+cu` label). + pub comfyui_baseline_torch_stack: Option, /// The managed runtime's installed distribution names captured BEFORE a ComfyUI /// install (EAI-8051), so the post-install step can require the set to have /// grown. Without it a run that installs nothing — `comfyui::install` skips the @@ -296,7 +295,7 @@ impl Default for E2eWorld { command_env: Vec::new(), tui: None, chat_use_mock: false, - comfyui_baseline_torch: None, + comfyui_baseline_torch_stack: None, comfyui_baseline_distributions: None, lifecycle: None, skill_reference: None, diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 6dcb24a2d2..e92aea2281 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -571,26 +571,6 @@ async fn assert_comfyui_spinner_line_cleared(world: &mut E2eWorld) { // --- comfyui-05: installing ComfyUI must not damage the managed ROCm runtime --- -/// Report the installed torch distribution's version string via `importlib.metadata` -/// — WITHOUT importing torch. Emits JSON `{version}` (e.g. `2.11.0+gitd0c8b1f` for -/// TheRock's ROCm build, `2.7.0+cu128` for a CUDA build) or `{error}` when no torch -/// distribution is installed. -/// -/// Deliberately does not `import torch`: importing it loads the ROCm/CUDA shared -/// libraries, which need the runtime's `LD_LIBRARY_PATH`/`ROCM_PATH` set up (the -/// product runs its own torch probe *with* that env, ours runs the interpreter -/// bare). The version string in the dist metadata is readable with no native -/// load, so a bare interpreter suffices. Only a `+cu` label reliably marks a CUDA -/// build; TheRock's ROCm torch is labelled with a git hash, not `+rocm`. -const TORCH_DIST_PROBE: &str = "import json,sys\n\ - from importlib import metadata\n\ - out={}\n\ - try:\n\ - \x20 out['version']=metadata.version('torch')\n\ - except Exception as ex:\n\ - \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ - sys.stdout.write(json.dumps(out))\n"; - /// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an /// `install_root: ` line for each installed runtime; the interpreter lives /// under a `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. @@ -653,36 +633,6 @@ fn find_venv_python(root: &Path) -> Option { None } -/// The installed torch distribution's version string, or `None` if no torch -/// distribution is installed. Reads dist metadata without importing torch (see -/// [`TORCH_DIST_PROBE`]), so it works against a bare interpreter. -/// -/// The local-version label identifies the build. A CUDA wheel is unmistakable: -/// `+cu` (e.g. `2.7.0+cu128`). A ROCm build is NOT reliably `+rocm`, though — -/// TheRock's managed torch labels the local version with a git hash -/// (`2.11.0+gitd0c8b1f`, measured on the MI300X lane), so callers judge "ROCm" as -/// "torch is present and is NOT a CUDA build", which is exactly the flip the -/// EAI-8051 corruption would cause. -fn torch_version(python: &Path) -> Result { - let output = std::process::Command::new(python) - .args(["-c", TORCH_DIST_PROBE]) - .output() - .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); - let stdout = String::from_utf8_lossy(&output.stdout); - let stderr = String::from_utf8_lossy(&output.stderr); - let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { - panic!("torch version probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") - }); - match data.get("version").and_then(serde_json::Value::as_str) { - Some(v) => Ok(v.to_owned()), - None => Err(data - .get("error") - .and_then(serde_json::Value::as_str) - .unwrap_or("probe reported no version and no error") - .to_owned()), - } -} - const TORCH_STACK_PROBE: &str = "import json,sys\n\ from importlib import metadata\n\ out={}\n\ @@ -693,6 +643,10 @@ const TORCH_STACK_PROBE: &str = "import json,sys\n\ \x20 out[n]=None\n\ sys.stdout.write(json.dumps(out))\n"; +/// Reads versions through `importlib.metadata` WITHOUT importing torch: importing +/// loads the ROCm/CUDA shared libraries, which need the runtime's own +/// `LD_LIBRARY_PATH`/`ROCM_PATH`, while this runs the interpreter bare. +/// /// Exact versions of the whole torch stack (`torch`, `torchvision`, `torchaudio`), /// formatted `name=version` (or `name=absent`), one per line. The product pins all /// three, so the scenario compares all three. @@ -716,16 +670,6 @@ fn torch_stack_versions(python: &Path) -> String { .join("\n") } -/// Whether `version` is a CUDA torch build (carries a `+cuNNN` local label). Used -/// only for the baseline PREMISE check — that the runtime didn't start out on a -/// CUDA torch. The post-install invariant is stronger: the torch version must be -/// byte-for-byte unchanged (a ComfyUI install must not replace the runtime's torch -/// at all), which also catches a swap to a plain non-`+cu` wheel (e.g. the observed -/// `2.11.0+gitd0c8b1f` → `2.13.0`) that this label check alone would miss. -fn is_cuda_torch(version: &str) -> bool { - version.to_ascii_lowercase().contains("+cu") -} - /// Enumerate installed distributions via `importlib.metadata` and emit their names /// as a JSON array. Used instead of `pip list` because uv-created managed runtimes /// have no `pip` module — `python -m pip` there exits non-zero with empty stdout, @@ -790,25 +734,17 @@ async fn setup_isolated_runtime(world: &mut E2eWorld) { // corrupt the runtime (that is the bug it pins), so it must own a private, // throwaway runtime prefix. Each World already has isolated ROCM_CLI_* dirs, // so a plain `install sdk` here lands in this scenario's own tree. - let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); - if stdout.contains("installed: none") { - crate::run_rocm_ok(world, &["install", "sdk", "--yes"]); - } - let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); - assert!( - !stdout.contains("installed: none"), - "no managed runtime is installed after `install sdk`:\n{stdout}" - ); + crate::run_rocm_ok(world, &["install", "sdk", "--yes"]); } -#[given("the runtime's torch is not a CUDA build")] -async fn assert_baseline_torch_not_cuda(world: &mut E2eWorld) { +#[given("the runtime has torch and no CUDA packages")] +async fn assert_baseline_torch_present(world: &mut E2eWorld) { let python = sole_runtime_python(world); - let version = torch_version(&python); + let stack = torch_stack_versions(&python); assert!( - version.as_ref().is_ok_and(|v| !is_cuda_torch(v)), - "baseline runtime torch is absent or already a CUDA build; scenario premise absent \ - (torch version: {version:?}, python: {})", + !stack.contains("torch=absent"), + "baseline runtime has no torch distribution; scenario premise absent \ + ({stack}, python: {})", python.display() ); let distributions = installed_distributions(&python); @@ -822,11 +758,11 @@ async fn assert_baseline_torch_not_cuda(world: &mut E2eWorld) { // be unchanged (see `assert_torch_unchanged`), and the baseline package set so // it can require the install to have actually added something (see // `assert_dependencies_installed`). - world.comfyui_baseline_torch = Some(torch_stack_versions(&python)); + world.comfyui_baseline_torch_stack = Some(stack); world.comfyui_baseline_distributions = Some(distributions); } -#[then("the install succeeds")] +#[then("the ComfyUI install succeeds")] async fn assert_install_succeeded(world: &mut E2eWorld) { // The premise for every invariant below. `comfyui::install` bails early on // several paths (no managed runtime, a runtime that isn't `ready`, a non-wheel @@ -872,7 +808,8 @@ async fn assert_dependencies_installed(world: &mut E2eWorld) { after.iter().any(|name| !baseline.contains(name)), "ComfyUI install added no distributions to the runtime, so it installed \ nothing and the runtime-preservation checks would pass vacuously \ - ({} distributions before and after, python: {})", + ({} distributions before, {} after, python: {})", + baseline.len(), after.len(), python.display() ); @@ -883,7 +820,7 @@ async fn assert_torch_unchanged(world: &mut E2eWorld) { let python = sole_runtime_python(world); let after = torch_stack_versions(&python); let baseline = world - .comfyui_baseline_torch + .comfyui_baseline_torch_stack .as_deref() .expect("no baseline torch stack was captured"); assert_eq!( From 1955d1fe69d5c2229be96d5b324562a7b0382e8b Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 8 Oct 2026 12:45:39 +0200 Subject: [PATCH 15/26] test(e2e): look only at install_root for the comfyui-05 interpreter Drop the fallback walk that could baseline an unrelated interpreter, and note the nvidia check only has teeth on Linux. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 4 ++- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 34 ++++--------------- 2 files changed, 10 insertions(+), 28 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index b3cbe6cb4d..75ca40f043 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -96,7 +96,9 @@ Feature: ComfyUI install reports progress and makes failures actionable # # The contract: after installing an optional app, the machine's ROCm runtime must # still be a ROCm runtime — its torch, torchvision and torchaudio versions are unchanged and - # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, + # no `nvidia-*` CUDA distributions appear in it (that last check only has teeth on + # Linux: PyPI's Windows torch wheels are CPU builds with no `nvidia-*` dependencies, + # so on Windows the exit code and the torch-stack comparison are the guards). Since #298 the install exits 0, # so the scenario also requires that. It is also the step that catches a revert # of #298 (the post-install GPU probe then bails and the install exits non-zero), # so do not relax it as a mere premise: a bail-out (no runtime, download failure) diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index e92aea2281..10f42a8ef9 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -601,36 +601,16 @@ fn sole_runtime_python(world: &E2eWorld) -> PathBuf { }) } -/// Locate a `bin/python` (Unix) or `Scripts/python.exe` (Windows) under `root`. -/// -/// The interpreter is expected exactly at `/bin/python` (the product -/// creates the venv at `install_root`). `root` is the initial frontier entry, so -/// that layout is found on the first iteration with no traversal. The depth-first -/// walk below is only a depth-capped tolerance fallback for a tree that differs. +/// The runtime's interpreter, expected exactly at `/bin/python` +/// (`Scripts/python.exe` on Windows): the product creates the venv at `install_root`. +/// No fallback search, so a missing interpreter is reported rather than replaced by +/// an unrelated one. fn find_venv_python(root: &Path) -> Option { #[cfg(windows)] - let (bin, exe) = ("Scripts", "python.exe"); + let candidate = root.join("Scripts").join("python.exe"); #[cfg(not(windows))] - let (bin, exe) = ("bin", "python"); - let mut frontier = vec![(root.to_path_buf(), 0usize)]; - while let Some((dir, depth)) = frontier.pop() { - let candidate = dir.join(bin).join(exe); - if candidate.is_file() { - return Some(candidate); - } - if depth >= 6 { - continue; - } - let Ok(entries) = std::fs::read_dir(&dir) else { - continue; - }; - for entry in entries.flatten() { - if entry.file_type().is_ok_and(|t| t.is_dir()) { - frontier.push((entry.path(), depth + 1)); - } - } - } - None + let candidate = root.join("bin").join("python"); + candidate.is_file().then_some(candidate) } const TORCH_STACK_PROBE: &str = "import json,sys\n\ From d2a6b52666814906efe0657c30c7e6b05fdba977 Mon Sep 17 00:00:00 2001 From: fredespi Date: Thu, 8 Oct 2026 14:40:38 +0200 Subject: [PATCH 16/26] test(e2e): tidy comfyui-05 docs, listing failure and nvidia predicate Say 'torch stack' in the docs, surface a failing runtimes list, share the nvidia filter and name offenders in the baseline message, widen the feature title. Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 2 +- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 27 ++++++++++--------- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 75ca40f043..1d974c0211 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -1,4 +1,4 @@ -Feature: ComfyUI install reports progress and makes failures actionable +Feature: ComfyUI install reports progress, makes failures actionable and preserves the runtime # `rocm comfyui install` shells out to `uv` to resolve ComfyUI's Python # dependencies. A failed resolve (a version conflict, a yanked release, a diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 10f42a8ef9..fa6616454c 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -47,7 +47,7 @@ //! dependency install already covered above. //! //! `comfyui-05` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real, -//! isolated managed runtime and asserts the runtime's torch is unchanged, no +//! isolated managed runtime and asserts the runtime's torch stack is unchanged, no //! `nvidia-*` distributions appeared, and the install really added packages. It //! reuses `comfyui-03`'s argument-less `comfyui install` When step. //! @@ -572,10 +572,8 @@ async fn assert_comfyui_spinner_line_cleared(world: &mut E2eWorld) { // --- comfyui-05: installing ComfyUI must not damage the managed ROCm runtime --- /// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an -/// `install_root: ` line for each installed runtime; the interpreter lives -/// under a `bin/python` (Unix) / `Scripts/python.exe` (Windows) inside that tree. -/// The interpreter is expected at `/bin/python`; see -/// [`find_venv_python`]. +/// `install_root: ` line for each installed runtime; the interpreter is +/// resolved from it by [`find_venv_python`]. /// /// Reads `runtimes list` rather than `examine`: examine only prints a `Folder:` /// line for the *active* runtime and takes a different branch when none is marked @@ -583,7 +581,7 @@ async fn assert_comfyui_spinner_line_cleared(world: &mut E2eWorld) { /// dispatch — the scenario panicked on a missing `Folder:` there). `runtimes list` /// prints `install_root:` for every installed runtime unconditionally. fn sole_runtime_python(world: &E2eWorld) -> PathBuf { - let (listing, _, _) = crate::run_rocm(world, &["runtimes", "list"]); + let listing = crate::run_rocm_ok(world, &["runtimes", "list"]); let roots: Vec<&str> = listing .lines() .filter_map(|l| l.trim().strip_prefix("install_root:")) @@ -702,7 +700,11 @@ fn installed_distributions(python: &Path) -> Vec { /// A ROCm runtime should have none; ComfyUI's install dragging any in is the /// EAI-8051 defect. fn nvidia_distributions(python: &Path) -> Vec { - installed_distributions(python) + nvidia_only(installed_distributions(python)) +} + +fn nvidia_only(distributions: Vec) -> Vec { + distributions .into_iter() .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) .collect() @@ -728,13 +730,14 @@ async fn assert_baseline_torch_present(world: &mut E2eWorld) { python.display() ); let distributions = installed_distributions(&python); + let nvidia = nvidia_only(distributions.clone()); assert!( - !distributions - .iter() - .any(|name| name.to_ascii_lowercase().starts_with("nvidia-")), - "runtime already has nvidia-* distributions before ComfyUI install; premise absent" + nvidia.is_empty(), + "runtime already has nvidia-* distributions before ComfyUI install; premise \ + absent: {}", + nvidia.join(", ") ); - // Record the exact baseline version so the post-install step can require it to + // Record the exact baseline torch stack so the post-install step can require it to // be unchanged (see `assert_torch_unchanged`), and the baseline package set so // it can require the install to have actually added something (see // `assert_dependencies_installed`). From 0b27d2f25e6c35951bb1fd71f87f247670475bfe Mon Sep 17 00:00:00 2001 From: fredespi Date: Fri, 9 Oct 2026 09:30:48 +0200 Subject: [PATCH 17/26] test(e2e): correct the comfyui-05 runtimes-list rationale Signed-off-by: fredespi --- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index fa6616454c..0a186a5533 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -575,11 +575,10 @@ async fn assert_comfyui_spinner_line_cleared(world: &mut E2eWorld) { /// `install_root: ` line for each installed runtime; the interpreter is /// resolved from it by [`find_venv_python`]. /// -/// Reads `runtimes list` rather than `examine`: examine only prints a `Folder:` -/// line for the *active* runtime and takes a different branch when none is marked -/// active, so it is not a reliable source for the install root (this cost a GPU -/// dispatch — the scenario panicked on a missing `Folder:` there). `runtimes list` -/// prints `install_root:` for every installed runtime unconditionally. +/// Reads `runtimes list` rather than `examine`: without an active runtime, +/// examine's `Folder:` line can be the saved setup root instead of an +/// `install_root`, and which it prints depends on the branch examine takes. +/// `runtimes list` prints `install_root:` for every installed runtime. fn sole_runtime_python(world: &E2eWorld) -> PathBuf { let listing = crate::run_rocm_ok(world, &["runtimes", "list"]); let roots: Vec<&str> = listing From e352bed8c86058832e7f0393f28c2e3265c18126 Mon Sep 17 00:00:00 2001 From: fredespi Date: Fri, 9 Oct 2026 11:19:34 +0200 Subject: [PATCH 18/26] test(e2e): gate comfyui-05 to Linux, normalise nvidia names, fix isolation wording Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 6 ++++-- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 6 +++++- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 1d974c0211..216bdf6b6b 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -117,8 +117,10 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # real `install sdk`. NOT @lifecycle: that tag is for OS-mutating # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so # combining it with @nightly would make this scenario unreachable on every lane; - # this mutates only its own isolated runtime prefix, not the OS. - @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly + # it touches the OS only as much as runtime-01's `install sdk --yes` does (installing + # missing torch system libraries), and otherwise only its own isolated runtime + # prefix. Linux-only until a Windows/WSL2 run of `comfyui install` has been seen. + @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux Scenario: comfyui-05 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given a machine with a managed ROCm runtime And the runtime has torch and no CUDA packages diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 0a186a5533..d4001cabc9 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -705,7 +705,11 @@ fn nvidia_distributions(python: &Path) -> Vec { fn nvidia_only(distributions: Vec) -> Vec { distributions .into_iter() - .filter(|name| name.to_ascii_lowercase().starts_with("nvidia-")) + .filter(|name| { + name.to_ascii_lowercase() + .replace(['_', '.'], "-") + .starts_with("nvidia-") + }) .collect() } From 723015764c1e84632af2e6b822ed721d3cf91631 Mon Sep 17 00:00:00 2001 From: fredespi Date: Fri, 9 Oct 2026 15:30:25 +0200 Subject: [PATCH 19/26] test(e2e): exclude WSL2 from comfyui-05 and fix isolation wording Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 11 ++++++----- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 5 +++-- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 216bdf6b6b..5a0ad42891 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -96,9 +96,7 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # # The contract: after installing an optional app, the machine's ROCm runtime must # still be a ROCm runtime — its torch, torchvision and torchaudio versions are unchanged and - # no `nvidia-*` CUDA distributions appear in it (that last check only has teeth on - # Linux: PyPI's Windows torch wheels are CPU builds with no `nvidia-*` dependencies, - # so on Windows the exit code and the torch-stack comparison are the guards). Since #298 the install exits 0, + # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, # so the scenario also requires that. It is also the step that catches a revert # of #298 (the post-install GPU probe then bails and the install exits non-zero), # so do not relax it as a mere premise: a bail-out (no runtime, download failure) @@ -119,8 +117,11 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # combining it with @nightly would make this scenario unreachable on every lane; # it touches the OS only as much as runtime-01's `install sdk --yes` does (installing # missing torch system libraries), and otherwise only its own isolated runtime - # prefix. Linux-only until a Windows/WSL2 run of `comfyui install` has been seen. - @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux + # prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports + # linux, so the first tag alone would not exclude it) until a Windows/WSL2 run of + # `comfyui install` has been seen. If that gate is lifted, note the `nvidia-*` check has + # no teeth on Windows (PyPI's Windows torch wheels are CPU builds). + @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux @requires-bare-metal Scenario: comfyui-05 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given a machine with a managed ROCm runtime And the runtime has torch and no CUDA packages diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index d4001cabc9..d45e184de5 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -717,8 +717,9 @@ fn nvidia_only(distributions: Vec) -> Vec { async fn setup_isolated_runtime(world: &mut E2eWorld) { // DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may // corrupt the runtime (that is the bug it pins), so it must own a private, - // throwaway runtime prefix. Each World already has isolated ROCM_CLI_* dirs, - // so a plain `install sdk` here lands in this scenario's own tree. + // throwaway runtime prefix. The runtime lands under the World's per-scenario + // data dir (only the download caches are shared), so a plain `install sdk` here + // lands in this scenario's own tree. crate::run_rocm_ok(world, &["install", "sdk", "--yes"]); } From 13c23a85cef06d4a82c32d345161682796583ffa Mon Sep 17 00:00:00 2001 From: fredespi Date: Mon, 12 Oct 2026 12:40:27 +0200 Subject: [PATCH 20/26] test(e2e): drop unreachable Windows arm, name the nvidia check for what it matches Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 2 +- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 7 ++----- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 5a0ad42891..43669f6fb0 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -129,4 +129,4 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv Then the ComfyUI install succeeds And ComfyUI's dependencies were installed into the runtime And the runtime's torch stack is unchanged - And no CUDA nvidia packages were added to the runtime + And no nvidia-* packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index d45e184de5..d77df05517 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -599,13 +599,10 @@ fn sole_runtime_python(world: &E2eWorld) -> PathBuf { } /// The runtime's interpreter, expected exactly at `/bin/python` -/// (`Scripts/python.exe` on Windows): the product creates the venv at `install_root`. +/// (the scenario is native-Linux only): the product creates the venv at `install_root`. /// No fallback search, so a missing interpreter is reported rather than replaced by /// an unrelated one. fn find_venv_python(root: &Path) -> Option { - #[cfg(windows)] - let candidate = root.join("Scripts").join("python.exe"); - #[cfg(not(windows))] let candidate = root.join("bin").join("python"); candidate.is_file().then_some(candidate) } @@ -819,7 +816,7 @@ async fn assert_torch_unchanged(world: &mut E2eWorld) { ); } -#[then("no CUDA nvidia packages were added to the runtime")] +#[then("no nvidia-* packages were added to the runtime")] async fn assert_no_nvidia_packages(world: &mut E2eWorld) { let python = sole_runtime_python(world); let nvidia = nvidia_distributions(&python); From 77498aeae5dcd0d1014e6171b462e2f696b211df Mon Sep 17 00:00:00 2001 From: fredespi Date: Mon, 12 Oct 2026 14:50:25 +0200 Subject: [PATCH 21/26] test(e2e): name the OpenMPI and engine install in comfyui-05's host-impact comment Co-Authored-By: Claude Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 43669f6fb0..bf579c7a51 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -115,9 +115,11 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # real `install sdk`. NOT @lifecycle: that tag is for OS-mutating # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so # combining it with @nightly would make this scenario unreachable on every lane; - # it touches the OS only as much as runtime-01's `install sdk --yes` does (installing - # missing torch system libraries), and otherwise only its own isolated runtime - # prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports + # its Given step's `install sdk --yes` grants system-package consent, so beyond + # runtime-01's torch system libraries it also runs the preferred-engine auto install + # (for vLLM families: a sudo OpenMPI package install when missing, then a full vLLM + # engine install into the runtime), which adds nightly time; otherwise it touches only + # its own isolated runtime prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports # linux, so the first tag alone would not exclude it) until a Windows/WSL2 run of # `comfyui install` has been seen. If that gate is lifted, note the `nvidia-*` check has # no teeth on Windows (PyPI's Windows torch wheels are CPU builds). From d0388f8fa46105a562cd54690ebfbc008655f514 Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 13 Oct 2026 11:54:49 +0200 Subject: [PATCH 22/26] test(e2e): say comfyui-05 does the same host-level work as runtime-01 Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index bf579c7a51..660b4cda99 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -115,11 +115,11 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # real `install sdk`. NOT @lifecycle: that tag is for OS-mutating # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so # combining it with @nightly would make this scenario unreachable on every lane; - # its Given step's `install sdk --yes` grants system-package consent, so beyond - # runtime-01's torch system libraries it also runs the preferred-engine auto install - # (for vLLM families: a sudo OpenMPI package install when missing, then a full vLLM - # engine install into the runtime), which adds nightly time; otherwise it touches only - # its own isolated runtime prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports + # its Given step runs the same `install sdk --yes` as runtime-01, so it does the same + # host-level work: system-package consent, the torch system libraries, and the + # preferred-engine auto install (for vLLM families a sudo OpenMPI package install when + # missing, then a full vLLM engine install into the runtime), which adds nightly time; + # otherwise it touches only its own isolated runtime prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports # linux, so the first tag alone would not exclude it) until a Windows/WSL2 run of # `comfyui install` has been seen. If that gate is lifted, note the `nvidia-*` check has # no teeth on Windows (PyPI's Windows torch wheels are CPU builds). From f3f48683972835933d2957c273a1b7b6583d1880 Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 13 Oct 2026 19:01:14 +0200 Subject: [PATCH 23/26] test(e2e): correct the torch-pin doc comment and the WSL2 gate rationale Signed-off-by: fredespi --- tests/e2e-cucumber/features/comfyui.feature | 5 +++-- tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 608e6a10f5..c1e1a181ec 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -293,8 +293,9 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # preferred-engine auto install (for vLLM families a sudo OpenMPI package install when # missing, then a full vLLM engine install into the runtime), which adds nightly time; # otherwise it touches only its own isolated runtime prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports - # linux, so the first tag alone would not exclude it) until a Windows/WSL2 run of - # `comfyui install` has been seen. If that gate is lifted, note the `nvidia-*` check has + # linux, so the first tag alone would not exclude it). WSL2 is excluded because no + # run of `comfyui install` there has been seen, not because the amdgpu premise is + # known to be absent; no ticket tracks lifting the gate. If that gate is lifted, note the `nvidia-*` check has # no teeth on Windows (PyPI's Windows torch wheels are CPU builds). @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux @requires-bare-metal Scenario: comfyui-14 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 1aaa6d252d..7e2702c9a9 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -1597,8 +1597,9 @@ const TORCH_STACK_PROBE: &str = "import json,sys\n\ /// `LD_LIBRARY_PATH`/`ROCM_PATH`, while this runs the interpreter bare. /// /// Exact versions of the whole torch stack (`torch`, `torchvision`, `torchaudio`), -/// formatted `name=version` (or `name=absent`), one per line. The product pins all -/// three, so the scenario compares all three. +/// formatted `name=version` (or `name=absent`), one per line. The product pins whichever +/// of the three are installed, so the scenario compares all three: a member that +/// appears later changes the string. fn torch_stack_versions(python: &Path) -> String { let output = std::process::Command::new(python) .args(["-c", TORCH_STACK_PROBE]) From d44d78416155e7abcf627c83517d5384a4d3c6a5 Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 13 Oct 2026 19:50:48 +0200 Subject: [PATCH 24/26] docs(e2e): document comfyui-14 and untangle its Windows/WSL2 gate comment Signed-off-by: fredespi --- docs/testing.md | 10 ++++++++++ tests/e2e-cucumber/features/comfyui.feature | 14 +++++++++----- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index d6ed1c3b04..a338d646ab 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -558,6 +558,16 @@ download that is not a ComfyUI release, which is neither installed nor kept: cargo xtask e2e -- -n "comfyui-(0[5-9]|1[0-3])" ``` +`comfyui-14` is different: it installs ComfyUI into a real, isolated managed +runtime and checks the ROCm torch stack survives with no `nvidia-*` packages +added. It runs only with `E2E_INCLUDE_NIGHTLY=1` on a bare-metal Linux GPU host +(`@requires-gpu @nightly @requires-os:linux @requires-bare-metal`) and does a +real `install sdk --yes` first: + +```bash +E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e -- -n comfyui-14 +``` + ## Runtime Selection And Activation List registered managed or read-only runtimes and their exact side-by-side keys: diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index c1e1a181ec..9753c7aeac 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -292,11 +292,15 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv # host-level work: system-package consent, the torch system libraries, and the # preferred-engine auto install (for vLLM families a sudo OpenMPI package install when # missing, then a full vLLM engine install into the runtime), which adds nightly time; - # otherwise it touches only its own isolated runtime prefix. Native Linux only (@requires-os:linux @requires-bare-metal; WSL2 reports - # linux, so the first tag alone would not exclude it). WSL2 is excluded because no - # run of `comfyui install` there has been seen, not because the amdgpu premise is - # known to be absent; no ticket tracks lifting the gate. If that gate is lifted, note the `nvidia-*` check has - # no teeth on Windows (PyPI's Windows torch wheels are CPU builds). + # otherwise it touches only its own isolated runtime prefix. + # + # Native Linux only: @requires-os:linux keeps Windows out, and + # @requires-bare-metal keeps WSL2 out (it reports linux, so the first tag alone + # would not). WSL2 is excluded because no run of `comfyui install` there has + # been seen, not because the amdgpu premise is known to be absent; no ticket + # tracks lifting that gate. Separately, if Windows is ever admitted, note the + # `nvidia-*` check has no teeth there (PyPI's Windows torch wheels are CPU + # builds). @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux @requires-bare-metal Scenario: comfyui-14 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one Given a machine with a managed ROCm runtime From c965e4e3ce492977bbd6a4caa44bb4405cc4c505 Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 13 Oct 2026 20:29:02 +0200 Subject: [PATCH 25/26] docs(e2e): stop recommending -n for comfyui-14; note -n skips the OS gate Signed-off-by: fredespi --- docs/testing.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index a338d646ab..4612f0655e 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -558,16 +558,24 @@ download that is not a ComfyUI release, which is neither installed nor kept: cargo xtask e2e -- -n "comfyui-(0[5-9]|1[0-3])" ``` +`-n` bypasses the suite's applicability resolver, so the `@requires-os:linux` +gate is not applied: run it only on Linux. + `comfyui-14` is different: it installs ComfyUI into a real, isolated managed runtime and checks the ROCm torch stack survives with no `nvidia-*` packages -added. It runs only with `E2E_INCLUDE_NIGHTLY=1` on a bare-metal Linux GPU host -(`@requires-gpu @nightly @requires-os:linux @requires-bare-metal`) and does a -real `install sdk --yes` first: +added. It carries `@requires-gpu @nightly @requires-os:linux @requires-bare-metal` +and does a real `install sdk --yes` first (system-package consent, and a sudo +OpenMPI install on vLLM families). Those tags gate it only in a +harness-selected run, which needs the nightly opt-in and a bare-metal Linux AMD +GPU host: ```bash -E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e -- -n comfyui-14 +E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e ``` +Do not select it with `-n`: that bypasses every gate above, so on WSL2, +Windows or a machine with no GPU it would still run the real install. + ## Runtime Selection And Activation List registered managed or read-only runtimes and their exact side-by-side keys: From 2218e5b855001ff0a71abd5abc92e47bf3d0e48b Mon Sep 17 00:00:00 2001 From: fredespi Date: Tue, 13 Oct 2026 21:31:16 +0200 Subject: [PATCH 26/26] docs(e2e): warn that name_filter bypasses comfyui-14's gates too Signed-off-by: fredespi --- docs/testing.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index 4612f0655e..2e37dbf27c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -573,8 +573,10 @@ GPU host: E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e ``` -Do not select it with `-n`: that bypasses every gate above, so on WSL2, -Windows or a machine with no GPU it would still run the real install. +Do not select it with `-n`/`--name`, or with the `name_filter` input of the +self-hosted CI dispatch (which forwards `--name`): each bypasses every gate +above, so on WSL2, Windows or a machine with no GPU it would still run the real +install. ## Runtime Selection And Activation