diff --git a/docs/testing.md b/docs/testing.md index d6ed1c3b04..2e37dbf27c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -558,6 +558,26 @@ download that is not a ComfyUI release, which is neither installed nor kept: cargo xtask e2e -- -n "comfyui-(0[5-9]|1[0-3])" ``` +`-n` bypasses the suite's applicability resolver, so the `@requires-os:linux` +gate is not applied: run it only on Linux. + +`comfyui-14` is different: it installs ComfyUI into a real, isolated managed +runtime and checks the ROCm torch stack survives with no `nvidia-*` packages +added. It carries `@requires-gpu @nightly @requires-os:linux @requires-bare-metal` +and does a real `install sdk --yes` first (system-package consent, and a sudo +OpenMPI install on vLLM families). Those tags gate it only in a +harness-selected run, which needs the nightly opt-in and a bare-metal Linux AMD +GPU host: + +```bash +E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e +``` + +Do not select it with `-n`/`--name`, or with the `name_filter` input of the +self-hosted CI dispatch (which forwards `--name`): each bypasses every gate +above, so on WSL2, Windows or a machine with no GPU it would still run the real +install. + ## Runtime Selection And Activation List registered managed or read-only runtimes and their exact side-by-side keys: diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index 51cf586634..9753c7aeac 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -1,4 +1,4 @@ -Feature: ComfyUI install reports progress and makes failures actionable +Feature: ComfyUI install reports progress, makes failures actionable and preserves the runtime # `rocm comfyui install` shells out to `uv` to resolve ComfyUI's Python # dependencies. A failed resolve (a version conflict, a yanked release, a @@ -259,3 +259,54 @@ Feature: ComfyUI install reports progress and makes failures actionable When the user reinstalls ComfyUI Then the reinstall fails saying the download is not a ComfyUI release, and does not keep it And the existing ComfyUI code and the user's files are untouched + + # EAI-8051: `rocm comfyui install` installs ComfyUI's dependencies INTO the + # machine's managed ROCm runtime. Before #298 nothing constrained the torch stack, + # so a transitive dependency could pull CUDA `nvidia-*` wheels into the runtime + # and displace its ROCm torch, leaving a CUDA build with no AMD GPU support -- + # installing an optional app broke the base. #298 now pins the torch stack with + # a `uv --constraint` file; this scenario guards against that regressing. + # + # The contract: after installing an optional app, the machine's ROCm runtime must + # still be a ROCm runtime — its torch, torchvision and torchaudio versions are unchanged and + # no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0, + # so the scenario also requires that. It is also the step that catches a revert + # of #298 (the post-install GPU probe then bails and the install exits non-zero), + # so do not relax it as a mere premise: a bail-out (no runtime, download failure) + # would otherwise leave the runtime trivially unchanged and report green having + # installed nothing. It further requires the runtime's package set to have GROWN, + # because a zero exit code alone still permits an empty filtered requirement list, + # which skips the dependency install outright. Together those two make the + # runtime-health checks the ADDITIONAL contract on top of an install that really + # happened, not a substitute for it. + # + # Genuinely destructive and expensive: it needs a real managed runtime (a + # multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind + # @nightly, against this scenario's own isolated runtime prefix (it must never + # share a runtime tree with other scenarios — it may corrupt it). Gated + # @requires-gpu @nightly, like runtime-install-sdk-active, which also does a + # real `install sdk`. NOT @lifecycle: that tag is for OS-mutating + # release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so + # combining it with @nightly would make this scenario unreachable on every lane; + # its Given step runs the same `install sdk --yes` as runtime-01, so it does the same + # host-level work: system-package consent, the torch system libraries, and the + # preferred-engine auto install (for vLLM families a sudo OpenMPI package install when + # missing, then a full vLLM engine install into the runtime), which adds nightly time; + # otherwise it touches only its own isolated runtime prefix. + # + # Native Linux only: @requires-os:linux keeps Windows out, and + # @requires-bare-metal keeps WSL2 out (it reports linux, so the first tag alone + # would not). WSL2 is excluded because no run of `comfyui install` there has + # been seen, not because the amdgpu premise is known to be absent; no ticket + # tracks lifting that gate. Separately, if Windows is ever admitted, note the + # `nvidia-*` check has no teeth there (PyPI's Windows torch wheels are CPU + # builds). + @id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux @requires-bare-metal + Scenario: comfyui-14 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one + Given a machine with a managed ROCm runtime + And the runtime has torch and no CUDA packages + When the user installs ComfyUI without choosing a runtime + Then the ComfyUI install succeeds + And ComfyUI's dependencies were installed into the runtime + And the runtime's torch stack is unchanged + And no nvidia-* packages were added to the runtime diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 4750957032..9bbd9d3088 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -142,6 +142,17 @@ pub struct E2eWorld { /// launch step knows to pass `--chat-mock` (deterministic offline agent, no /// endpoint detection) instead of driving the real detection/consent path. pub chat_use_mock: bool, + /// The managed runtime's torch/torchvision/torchaudio versions captured BEFORE a + /// ComfyUI install (EAI-8051), so the post-install step can require them to be + /// unchanged. Comparing exact versions catches a swap to any other build, labelled + /// or not (e.g. stock PyPI `2.13.0`, which is CUDA yet carries no `+cu` label). + pub comfyui_baseline_torch_stack: Option, + /// The managed runtime's installed distribution names captured BEFORE a ComfyUI + /// install (EAI-8051), so the post-install step can require the set to have + /// grown. Without it a run that installs nothing — `comfyui::install` skips the + /// whole `uv` step on an empty requirement list and still exits 0 — would pass + /// every runtime-preservation assertion having exercised nothing. + pub comfyui_baseline_distributions: Option>, /// Per-scenario release-lifecycle state (packaging dirs, signing keys, install /// dir, captured logs). `Some` only for `@lifecycle` scenarios; all its paths /// are rooted in `isolated_root` so teardown removes them with the temp dir. @@ -291,6 +302,8 @@ impl Default for E2eWorld { command_env: Vec::new(), tui: None, chat_use_mock: false, + comfyui_baseline_torch_stack: None, + comfyui_baseline_distributions: None, lifecycle: None, skill_reference: None, skill_symptom: None, diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 4cfb89f99a..7e2702c9a9 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -69,6 +69,11 @@ //! what the installed release shipped (`.rocm-cli-release-entries`), which an //! install made by this CLI would have. `source_swap`'s unit tests interrupt //! the swap for real at every step. +//! +//! `comfyui-14` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real, +//! isolated managed runtime and asserts the runtime's torch stack is unchanged, no +//! `nvidia-*` distributions appeared, and the install really added packages. It +//! reuses `comfyui-03`'s argument-less `comfyui install` When step. use std::path::{Path, PathBuf}; use std::time::Duration; @@ -1538,3 +1543,262 @@ async fn comfyui_installed_and_ready(world: &mut E2eWorld) { assert!(stdout.contains(" installed: yes\n"), "{stdout}"); assert!(stdout.contains(" AMD GPU check: ready\n"), "{stdout}"); } + +// --- comfyui-14: installing ComfyUI must not damage the managed ROCm runtime --- + +/// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an +/// `install_root: ` line for each installed runtime; the interpreter is +/// resolved from it by [`find_venv_python`]. +/// +/// Reads `runtimes list` rather than `examine`: without an active runtime, +/// examine's `Folder:` line can be the saved setup root instead of an +/// `install_root`, and which it prints depends on the branch examine takes. +/// `runtimes list` prints `install_root:` for every installed runtime. +fn sole_runtime_python(world: &E2eWorld) -> PathBuf { + let listing = crate::run_rocm_ok(world, &["runtimes", "list"]); + let roots: Vec<&str> = listing + .lines() + .filter_map(|l| l.trim().strip_prefix("install_root:")) + .map(str::trim) + .collect(); + assert_eq!( + roots.len(), + 1, + "expected exactly one installed runtime in the isolated world, found {}:\n{listing}", + roots.len() + ); + let root = roots[0]; + find_venv_python(Path::new(root)).unwrap_or_else(|| { + panic!("could not locate a venv python under the runtime install_root {root}") + }) +} + +/// The runtime's interpreter, expected exactly at `/bin/python` +/// (the scenario is native-Linux only): the product creates the venv at `install_root`. +/// No fallback search, so a missing interpreter is reported rather than replaced by +/// an unrelated one. +fn find_venv_python(root: &Path) -> Option { + let candidate = root.join("bin").join("python"); + candidate.is_file().then_some(candidate) +} + +const TORCH_STACK_PROBE: &str = "import json,sys\n\ + from importlib import metadata\n\ + out={}\n\ + for n in ('torch','torchvision','torchaudio'):\n\ + \x20 try:\n\ + \x20 out[n]=metadata.version(n)\n\ + \x20 except metadata.PackageNotFoundError:\n\ + \x20 out[n]=None\n\ + sys.stdout.write(json.dumps(out))\n"; + +/// Reads versions through `importlib.metadata` WITHOUT importing torch: importing +/// loads the ROCm/CUDA shared libraries, which need the runtime's own +/// `LD_LIBRARY_PATH`/`ROCM_PATH`, while this runs the interpreter bare. +/// +/// Exact versions of the whole torch stack (`torch`, `torchvision`, `torchaudio`), +/// formatted `name=version` (or `name=absent`), one per line. The product pins whichever +/// of the three are installed, so the scenario compares all three: a member that +/// appears later changes the string. +fn torch_stack_versions(python: &Path) -> String { + let output = std::process::Command::new(python) + .args(["-c", TORCH_STACK_PROBE]) + .output() + .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { + panic!("torch stack probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") + }); + ["torch", "torchvision", "torchaudio"] + .iter() + .map(|n| { + let v = data.get(*n).and_then(serde_json::Value::as_str); + format!("{n}={}", v.unwrap_or("absent")) + }) + .collect::>() + .join("\n") +} + +/// Enumerate installed distributions via `importlib.metadata` and emit their names +/// as a JSON array. Used instead of `pip list` because uv-created managed runtimes +/// have no `pip` module — `python -m pip` there exits non-zero with empty stdout, +/// which a naive reader would misread as "no packages installed" and pass the +/// nvidia check while the runtime is actually corrupted. `importlib.metadata` is in +/// the stdlib, so it is always present; the probe emits `{names}` on success or +/// `{error}` on failure so the caller can fail loudly rather than treat a broken +/// probe as a clean result. +const DISTRIBUTIONS_PROBE: &str = "import json,sys\n\ + out={}\n\ + try:\n\ + \x20 from importlib import metadata\n\ + \x20 out['names']=sorted({(d.metadata['Name'] or '') for d in metadata.distributions()})\n\ + except Exception as ex:\n\ + \x20 out['error']=type(ex).__name__+': '+str(ex)\n\ + sys.stdout.write(json.dumps(out))\n"; + +/// Every distribution name installed in the interpreter's environment, sorted. +/// Panics if the interpreter cannot be run or the probe reports an error — a probe +/// that cannot enumerate packages must NOT read as an empty environment, which +/// would pass the nvidia check on a runtime it never actually inspected. +fn installed_distributions(python: &Path) -> Vec { + let output = std::process::Command::new(python) + .args(["-c", DISTRIBUTIONS_PROBE]) + .output() + .unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display())); + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| { + panic!("distributions probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}") + }); + if let Some(error) = data.get("error").and_then(serde_json::Value::as_str) { + panic!( + "could not enumerate installed distributions on {}: {error}", + python.display() + ); + } + let names = data + .get("names") + .and_then(serde_json::Value::as_array) + .unwrap_or_else(|| panic!("distributions probe returned no 'names' array:\n{stdout}")); + names + .iter() + .filter_map(serde_json::Value::as_str) + .map(str::to_owned) + .collect() +} + +/// The `nvidia-*` CUDA distributions installed in the interpreter's environment. +/// A ROCm runtime should have none; ComfyUI's install dragging any in is the +/// EAI-8051 defect. +fn nvidia_distributions(python: &Path) -> Vec { + nvidia_only(installed_distributions(python)) +} + +fn nvidia_only(distributions: Vec) -> Vec { + distributions + .into_iter() + .filter(|name| { + name.to_ascii_lowercase() + .replace(['_', '.'], "-") + .starts_with("nvidia-") + }) + .collect() +} + +#[given("a machine with a managed ROCm runtime")] +async fn setup_isolated_runtime(world: &mut E2eWorld) { + // DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may + // corrupt the runtime (that is the bug it pins), so it must own a private, + // throwaway runtime prefix. The runtime lands under the World's per-scenario + // data dir (only the download caches are shared), so a plain `install sdk` here + // lands in this scenario's own tree. + crate::run_rocm_ok(world, &["install", "sdk", "--yes"]); +} + +#[given("the runtime has torch and no CUDA packages")] +async fn assert_baseline_torch_present(world: &mut E2eWorld) { + let python = sole_runtime_python(world); + let stack = torch_stack_versions(&python); + assert!( + !stack.contains("torch=absent"), + "baseline runtime has no torch distribution; scenario premise absent \ + ({stack}, python: {})", + python.display() + ); + let distributions = installed_distributions(&python); + let nvidia = nvidia_only(distributions.clone()); + assert!( + nvidia.is_empty(), + "runtime already has nvidia-* distributions before ComfyUI install; premise \ + absent: {}", + nvidia.join(", ") + ); + // Record the exact baseline torch stack so the post-install step can require it to + // be unchanged (see `assert_torch_unchanged`), and the baseline package set so + // it can require the install to have actually added something (see + // `assert_dependencies_installed`). + world.comfyui_baseline_torch_stack = Some(stack); + world.comfyui_baseline_distributions = Some(distributions); +} + +#[then("the ComfyUI install succeeds")] +async fn assert_install_succeeded(world: &mut E2eWorld) { + // The premise for every invariant below. `comfyui::install` bails early on + // several paths (no managed runtime, a runtime that isn't `ready`, a non-wheel + // format, a failed source download or `uv` acquisition); on any of those the + // runtime is TRIVIALLY unchanged and the torch/nvidia assertions would pass + // having exercised nothing. It is also the check that catches a revert of + // #298: without the torch constraint, the post-install GPU probe in + // `comfyui::install` bails, so the install exits non-zero and the torch and + // nvidia steps never run. Do not relax it as "just a premise". + let rc = world.cli_rc.expect("no ComfyUI install was run"); + assert_eq!( + rc, + 0, + "{}", + e2e_cucumber::cli_failure_report( + &["comfyui", "install"], + rc, + world.cli_output.as_deref().unwrap_or(""), + world.cli_stderr.as_deref().unwrap_or(""), + ) + ); +} + +#[then("ComfyUI's dependencies were installed into the runtime")] +async fn assert_dependencies_installed(world: &mut E2eWorld) { + // Closes the last vacuity path a zero exit code leaves open. `comfyui::install` + // guards the whole `uv` install with `if !packages.is_empty()` + // (`apps/rocm/src/comfyui.rs`), so an empty filtered requirement list skips it + // and still exits 0 — leaving the runtime untouched and every invariant below + // passing having installed nothing. + // + // Deliberately asserts the package set GREW rather than naming an expected + // dependency: ComfyUI's requirements drift upstream independently of this + // contract, so a named package would rot, while "the install put something in + // the runtime" is exactly the premise the invariants need and cannot go stale. + let python = sole_runtime_python(world); + let baseline = world + .comfyui_baseline_distributions + .as_ref() + .expect("no baseline distribution set was captured"); + let after = installed_distributions(&python); + assert!( + after.iter().any(|name| !baseline.contains(name)), + "ComfyUI install added no distributions to the runtime, so it installed \ + nothing and the runtime-preservation checks would pass vacuously \ + ({} distributions before, {} after, python: {})", + baseline.len(), + after.len(), + python.display() + ); +} + +#[then("the runtime's torch stack is unchanged")] +async fn assert_torch_unchanged(world: &mut E2eWorld) { + let python = sole_runtime_python(world); + let after = torch_stack_versions(&python); + let baseline = world + .comfyui_baseline_torch_stack + .as_deref() + .expect("no baseline torch stack was captured"); + assert_eq!( + after, + baseline, + "ComfyUI install replaced part of the managed runtime's torch stack \ + (before: [{baseline}], after: [{after}], python: {})", + python.display() + ); +} + +#[then("no nvidia-* packages were added to the runtime")] +async fn assert_no_nvidia_packages(world: &mut E2eWorld) { + let python = sole_runtime_python(world); + let nvidia = nvidia_distributions(&python); + assert!( + nvidia.is_empty(), + "ComfyUI install added CUDA nvidia-* distributions to the ROCm runtime: {}", + nvidia.join(", ") + ); +}