Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
cab2da0
test(e2e): pin driver-install-as-root and comfyui runtime contracts (…
fredespi Aug 20, 2026
03dea77
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 20, 2026
ffa0d29
test(e2e): fix EAI-8051 comfyui scenario gating and package probe
fredespi Aug 20, 2026
6ad735e
test(e2e): fix clippy on the e2e test targets
fredespi Aug 20, 2026
f74c53e
test(e2e): make run_rocm_with_only_tools compile on Windows
fredespi Aug 20, 2026
1bababf
test(e2e): find the runtime via runtimes list, not examine Folder:
fredespi Aug 20, 2026
b51c002
test(e2e): judge torch build from dist metadata, not a live import
fredespi Aug 20, 2026
87654fb
test(e2e): judge ROCm torch as "not a CUDA build", grounded on the ru…
fredespi Aug 20, 2026
2faf110
test(e2e): require the runtime's torch version to be unchanged (EAI-8…
fredespi Aug 21, 2026
b769da9
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 24, 2026
459d87f
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 25, 2026
1fe56c3
test(e2e): graduate the EAI-8051 scenario to expected-pass
fredespi Aug 25, 2026
ef9ecad
Merge origin/main; drop the superseded EAI-8053 driver half
fredespi Sep 10, 2026
263224f
test(e2e): require the ComfyUI install to actually install something
fredespi Sep 10, 2026
d38d5e1
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi-agent-app[bot] Oct 7, 2026
52cc71c
test(e2e): address review of the comfyui-05 scenario
fredespi-agent-app[bot] Oct 7, 2026
f256f7e
test(e2e): tighten comfyui-05 names, docs and install flags per review
fredespi Oct 7, 2026
1b23d34
test(e2e): compare the whole torch stack and use user-facing step text
fredespi Oct 7, 2026
15e7c5a
test(e2e): simplify comfyui-05 probes and correct its docs
fredespi Oct 8, 2026
1955d1f
test(e2e): look only at install_root for the comfyui-05 interpreter
fredespi Oct 8, 2026
d2a6b52
test(e2e): tidy comfyui-05 docs, listing failure and nvidia predicate
fredespi Oct 8, 2026
0b27d2f
test(e2e): correct the comfyui-05 runtimes-list rationale
fredespi Oct 9, 2026
e352bed
test(e2e): gate comfyui-05 to Linux, normalise nvidia names, fix isol…
fredespi Oct 9, 2026
7230157
test(e2e): exclude WSL2 from comfyui-05 and fix isolation wording
fredespi Oct 9, 2026
13c23a8
test(e2e): drop unreachable Windows arm, name the nvidia check for wh…
fredespi Oct 12, 2026
77498ae
test(e2e): name the OpenMPI and engine install in comfyui-05's host-i…
fredespi Oct 12, 2026
d0388f8
test(e2e): say comfyui-05 does the same host-level work as runtime-01
fredespi Oct 13, 2026
390b7cb
Merge origin/main into e2e-pin-the-contracts-for-eai-8051-and-eai-8053
fredespi Oct 13, 2026
f3f4868
test(e2e): correct the torch-pin doc comment and the WSL2 gate rationale
fredespi Oct 13, 2026
d44d784
docs(e2e): document comfyui-14 and untangle its Windows/WSL2 gate com…
fredespi Oct 13, 2026
c965e4e
docs(e2e): stop recommending -n for comfyui-14; note -n skips the OS …
fredespi Oct 13, 2026
2218e5b
docs(e2e): warn that name_filter bypasses comfyui-14's gates too
fredespi Oct 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,26 @@ download that is not a ComfyUI release, which is neither installed nor kept:
cargo xtask e2e -- -n "comfyui-(0[5-9]|1[0-3])"
```

`-n` bypasses the suite's applicability resolver, so the `@requires-os:linux`
gate is not applied: run it only on Linux.

`comfyui-14` is different: it installs ComfyUI into a real, isolated managed
runtime and checks the ROCm torch stack survives with no `nvidia-*` packages
added. It carries `@requires-gpu @nightly @requires-os:linux @requires-bare-metal`
and does a real `install sdk --yes` first (system-package consent, and a sudo
OpenMPI install on vLLM families). Those tags gate it only in a
harness-selected run, which needs the nightly opt-in and a bare-metal Linux AMD
GPU host:

```bash
E2E_INCLUDE_NIGHTLY=1 cargo xtask e2e
```

Do not select it with `-n`/`--name`, or with the `name_filter` input of the
self-hosted CI dispatch (which forwards `--name`): each bypasses every gate
above, so on WSL2, Windows or a machine with no GPU it would still run the real
install.

## Runtime Selection And Activation

List registered managed or read-only runtimes and their exact side-by-side keys:
Expand Down
53 changes: 52 additions & 1 deletion tests/e2e-cucumber/features/comfyui.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Feature: ComfyUI install reports progress and makes failures actionable
Feature: ComfyUI install reports progress, makes failures actionable and preserves the runtime

# `rocm comfyui install` shells out to `uv` to resolve ComfyUI's Python
# dependencies. A failed resolve (a version conflict, a yanked release, a
Expand Down Expand Up @@ -259,3 +259,54 @@ Feature: ComfyUI install reports progress and makes failures actionable
When the user reinstalls ComfyUI
Then the reinstall fails saying the download is not a ComfyUI release, and does not keep it
And the existing ComfyUI code and the user's files are untouched

# EAI-8051: `rocm comfyui install` installs ComfyUI's dependencies INTO the
# machine's managed ROCm runtime. Before #298 nothing constrained the torch stack,
# so a transitive dependency could pull CUDA `nvidia-*` wheels into the runtime
# and displace its ROCm torch, leaving a CUDA build with no AMD GPU support --
# installing an optional app broke the base. #298 now pins the torch stack with
# a `uv --constraint` file; this scenario guards against that regressing.
#
# The contract: after installing an optional app, the machine's ROCm runtime must
# still be a ROCm runtime — its torch, torchvision and torchaudio versions are unchanged and
# no `nvidia-*` CUDA distributions appear in it. Since #298 the install exits 0,
# so the scenario also requires that. It is also the step that catches a revert
# of #298 (the post-install GPU probe then bails and the install exits non-zero),
# so do not relax it as a mere premise: a bail-out (no runtime, download failure)
# would otherwise leave the runtime trivially unchanged and report green having
# installed nothing. It further requires the runtime's package set to have GROWN,
# because a zero exit code alone still permits an empty filtered requirement list,
# which skips the dependency install outright. Together those two make the
# runtime-health checks the ADDITIONAL contract on top of an install that really
# happened, not a substitute for it.
#
# Genuinely destructive and expensive: it needs a real managed runtime (a
# multi-GiB SDK install) and mutates it, so it runs ONLY on a GPU host, behind
# @nightly, against this scenario's own isolated runtime prefix (it must never
# share a runtime tree with other scenarios — it may corrupt it). Gated
# @requires-gpu @nightly, like runtime-install-sdk-active, which also does a
# real `install sdk`. NOT @lifecycle: that tag is for OS-mutating
# release scenarios and no lane sets E2E_INCLUDE_LIFECYCLE on a GPU host, so
# combining it with @nightly would make this scenario unreachable on every lane;
# its Given step runs the same `install sdk --yes` as runtime-01, so it does the same
# host-level work: system-package consent, the torch system libraries, and the
# preferred-engine auto install (for vLLM families a sudo OpenMPI package install when
# missing, then a full vLLM engine install into the runtime), which adds nightly time;
# otherwise it touches only its own isolated runtime prefix.
#
# Native Linux only: @requires-os:linux keeps Windows out, and
# @requires-bare-metal keeps WSL2 out (it reports linux, so the first tag alone
# would not). WSL2 is excluded because no run of `comfyui install` there has
# been seen, not because the amdgpu premise is known to be absent; no ticket
# tracks lifting that gate. Separately, if Windows is ever admitted, note the
# `nvidia-*` check has no teeth there (PyPI's Windows torch wheels are CPU
# builds).
@id:comfyui-install-preserves-the-rocm-runtime @requires-gpu @nightly @requires-os:linux @requires-bare-metal
Scenario: comfyui-14 - Installing ComfyUI does not replace the ROCm runtime with a CUDA one
Given a machine with a managed ROCm runtime
And the runtime has torch and no CUDA packages
When the user installs ComfyUI without choosing a runtime
Then the ComfyUI install succeeds
And ComfyUI's dependencies were installed into the runtime
And the runtime's torch stack is unchanged
And no nvidia-* packages were added to the runtime
13 changes: 13 additions & 0 deletions tests/e2e-cucumber/tests/e2e.rs
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,17 @@ pub struct E2eWorld {
/// launch step knows to pass `--chat-mock` (deterministic offline agent, no
/// endpoint detection) instead of driving the real detection/consent path.
pub chat_use_mock: bool,
/// The managed runtime's torch/torchvision/torchaudio versions captured BEFORE a
/// ComfyUI install (EAI-8051), so the post-install step can require them to be
/// unchanged. Comparing exact versions catches a swap to any other build, labelled
/// or not (e.g. stock PyPI `2.13.0`, which is CUDA yet carries no `+cu` label).
pub comfyui_baseline_torch_stack: Option<String>,
/// The managed runtime's installed distribution names captured BEFORE a ComfyUI
/// install (EAI-8051), so the post-install step can require the set to have
/// grown. Without it a run that installs nothing — `comfyui::install` skips the
/// whole `uv` step on an empty requirement list and still exits 0 — would pass
/// every runtime-preservation assertion having exercised nothing.
pub comfyui_baseline_distributions: Option<Vec<String>>,
/// Per-scenario release-lifecycle state (packaging dirs, signing keys, install
/// dir, captured logs). `Some` only for `@lifecycle` scenarios; all its paths
/// are rooted in `isolated_root` so teardown removes them with the temp dir.
Expand Down Expand Up @@ -291,6 +302,8 @@ impl Default for E2eWorld {
command_env: Vec::new(),
tui: None,
chat_use_mock: false,
comfyui_baseline_torch_stack: None,
comfyui_baseline_distributions: None,
lifecycle: None,
skill_reference: None,
skill_symptom: None,
Expand Down
264 changes: 264 additions & 0 deletions tests/e2e-cucumber/tests/e2e/comfyui_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,11 @@
//! what the installed release shipped (`.rocm-cli-release-entries`), which an
//! install made by this CLI would have. `source_swap`'s unit tests interrupt
//! the swap for real at every step.
//!
//! `comfyui-14` is the GPU-only EAI-8051 guard: it installs ComfyUI into a real,
//! isolated managed runtime and asserts the runtime's torch stack is unchanged, no
//! `nvidia-*` distributions appeared, and the install really added packages. It
//! reuses `comfyui-03`'s argument-less `comfyui install` When step.

use std::path::{Path, PathBuf};
use std::time::Duration;
Expand Down Expand Up @@ -1538,3 +1543,262 @@ async fn comfyui_installed_and_ready(world: &mut E2eWorld) {
assert!(stdout.contains(" installed: yes\n"), "{stdout}");
assert!(stdout.contains(" AMD GPU check: ready\n"), "{stdout}");
}

// --- comfyui-14: installing ComfyUI must not damage the managed ROCm runtime ---

/// Locate the managed runtime's venv interpreter. `rocm runtimes list` prints an
/// `install_root: <path>` line for each installed runtime; the interpreter is
/// resolved from it by [`find_venv_python`].
///
/// Reads `runtimes list` rather than `examine`: without an active runtime,
/// examine's `Folder:` line can be the saved setup root instead of an
/// `install_root`, and which it prints depends on the branch examine takes.
/// `runtimes list` prints `install_root:` for every installed runtime.
fn sole_runtime_python(world: &E2eWorld) -> PathBuf {
let listing = crate::run_rocm_ok(world, &["runtimes", "list"]);
let roots: Vec<&str> = listing
.lines()
.filter_map(|l| l.trim().strip_prefix("install_root:"))
.map(str::trim)
.collect();
assert_eq!(
roots.len(),
1,
"expected exactly one installed runtime in the isolated world, found {}:\n{listing}",
roots.len()
);
let root = roots[0];
find_venv_python(Path::new(root)).unwrap_or_else(|| {
panic!("could not locate a venv python under the runtime install_root {root}")
})
}

/// The runtime's interpreter, expected exactly at `<install_root>/bin/python`
/// (the scenario is native-Linux only): the product creates the venv at `install_root`.
/// No fallback search, so a missing interpreter is reported rather than replaced by
/// an unrelated one.
fn find_venv_python(root: &Path) -> Option<PathBuf> {
let candidate = root.join("bin").join("python");
candidate.is_file().then_some(candidate)
}

const TORCH_STACK_PROBE: &str = "import json,sys\n\
from importlib import metadata\n\
out={}\n\
for n in ('torch','torchvision','torchaudio'):\n\
\x20 try:\n\
\x20 out[n]=metadata.version(n)\n\
\x20 except metadata.PackageNotFoundError:\n\
\x20 out[n]=None\n\
sys.stdout.write(json.dumps(out))\n";

/// Reads versions through `importlib.metadata` WITHOUT importing torch: importing
/// loads the ROCm/CUDA shared libraries, which need the runtime's own
/// `LD_LIBRARY_PATH`/`ROCM_PATH`, while this runs the interpreter bare.
///
/// Exact versions of the whole torch stack (`torch`, `torchvision`, `torchaudio`),
/// formatted `name=version` (or `name=absent`), one per line. The product pins whichever
/// of the three are installed, so the scenario compares all three: a member that
/// appears later changes the string.
fn torch_stack_versions(python: &Path) -> String {
let output = std::process::Command::new(python)
.args(["-c", TORCH_STACK_PROBE])
.output()
.unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display()));
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| {
panic!("torch stack probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}")
});
["torch", "torchvision", "torchaudio"]
.iter()
.map(|n| {
let v = data.get(*n).and_then(serde_json::Value::as_str);
format!("{n}={}", v.unwrap_or("absent"))
})
.collect::<Vec<_>>()
.join("\n")
}

/// Enumerate installed distributions via `importlib.metadata` and emit their names
/// as a JSON array. Used instead of `pip list` because uv-created managed runtimes
/// have no `pip` module — `python -m pip` there exits non-zero with empty stdout,
/// which a naive reader would misread as "no packages installed" and pass the
/// nvidia check while the runtime is actually corrupted. `importlib.metadata` is in
/// the stdlib, so it is always present; the probe emits `{names}` on success or
/// `{error}` on failure so the caller can fail loudly rather than treat a broken
/// probe as a clean result.
const DISTRIBUTIONS_PROBE: &str = "import json,sys\n\
out={}\n\
try:\n\
\x20 from importlib import metadata\n\
\x20 out['names']=sorted({(d.metadata['Name'] or '') for d in metadata.distributions()})\n\
except Exception as ex:\n\
\x20 out['error']=type(ex).__name__+': '+str(ex)\n\
sys.stdout.write(json.dumps(out))\n";

/// Every distribution name installed in the interpreter's environment, sorted.
/// Panics if the interpreter cannot be run or the probe reports an error — a probe
/// that cannot enumerate packages must NOT read as an empty environment, which
/// would pass the nvidia check on a runtime it never actually inspected.
fn installed_distributions(python: &Path) -> Vec<String> {
let output = std::process::Command::new(python)
.args(["-c", DISTRIBUTIONS_PROBE])
.output()
.unwrap_or_else(|e| panic!("failed to run runtime python {}: {e}", python.display()));
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
let data: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|_| {
panic!("distributions probe returned non-JSON:\nstdout: {stdout}\nstderr: {stderr}")
});
if let Some(error) = data.get("error").and_then(serde_json::Value::as_str) {
panic!(
"could not enumerate installed distributions on {}: {error}",
python.display()
);
}
let names = data
.get("names")
.and_then(serde_json::Value::as_array)
.unwrap_or_else(|| panic!("distributions probe returned no 'names' array:\n{stdout}"));
names
.iter()
.filter_map(serde_json::Value::as_str)
.map(str::to_owned)
.collect()
}

/// The `nvidia-*` CUDA distributions installed in the interpreter's environment.
/// A ROCm runtime should have none; ComfyUI's install dragging any in is the
/// EAI-8051 defect.
fn nvidia_distributions(python: &Path) -> Vec<String> {
nvidia_only(installed_distributions(python))
}

fn nvidia_only(distributions: Vec<String>) -> Vec<String> {
distributions
.into_iter()
.filter(|name| {
name.to_ascii_lowercase()
.replace(['_', '.'], "-")
.starts_with("nvidia-")
})
.collect()
}

#[given("a machine with a managed ROCm runtime")]
async fn setup_isolated_runtime(world: &mut E2eWorld) {
// DELIBERATELY do NOT call `world.use_shared_runtimes()`: this scenario may
// corrupt the runtime (that is the bug it pins), so it must own a private,
// throwaway runtime prefix. The runtime lands under the World's per-scenario
// data dir (only the download caches are shared), so a plain `install sdk` here
// lands in this scenario's own tree.
crate::run_rocm_ok(world, &["install", "sdk", "--yes"]);
}

#[given("the runtime has torch and no CUDA packages")]
async fn assert_baseline_torch_present(world: &mut E2eWorld) {
let python = sole_runtime_python(world);
let stack = torch_stack_versions(&python);
assert!(
!stack.contains("torch=absent"),
"baseline runtime has no torch distribution; scenario premise absent \
({stack}, python: {})",
python.display()
);
let distributions = installed_distributions(&python);
let nvidia = nvidia_only(distributions.clone());
assert!(
nvidia.is_empty(),
"runtime already has nvidia-* distributions before ComfyUI install; premise \
absent: {}",
nvidia.join(", ")
);
// Record the exact baseline torch stack so the post-install step can require it to
// be unchanged (see `assert_torch_unchanged`), and the baseline package set so
// it can require the install to have actually added something (see
// `assert_dependencies_installed`).
world.comfyui_baseline_torch_stack = Some(stack);
world.comfyui_baseline_distributions = Some(distributions);
}

#[then("the ComfyUI install succeeds")]
async fn assert_install_succeeded(world: &mut E2eWorld) {
// The premise for every invariant below. `comfyui::install` bails early on
// several paths (no managed runtime, a runtime that isn't `ready`, a non-wheel
// format, a failed source download or `uv` acquisition); on any of those the
// runtime is TRIVIALLY unchanged and the torch/nvidia assertions would pass
// having exercised nothing. It is also the check that catches a revert of
// #298: without the torch constraint, the post-install GPU probe in
// `comfyui::install` bails, so the install exits non-zero and the torch and
// nvidia steps never run. Do not relax it as "just a premise".
let rc = world.cli_rc.expect("no ComfyUI install was run");
assert_eq!(
rc,
0,
"{}",
e2e_cucumber::cli_failure_report(
&["comfyui", "install"],
rc,
world.cli_output.as_deref().unwrap_or(""),
world.cli_stderr.as_deref().unwrap_or(""),
)
);
}

#[then("ComfyUI's dependencies were installed into the runtime")]
async fn assert_dependencies_installed(world: &mut E2eWorld) {
// Closes the last vacuity path a zero exit code leaves open. `comfyui::install`
// guards the whole `uv` install with `if !packages.is_empty()`
// (`apps/rocm/src/comfyui.rs`), so an empty filtered requirement list skips it
// and still exits 0 — leaving the runtime untouched and every invariant below
// passing having installed nothing.
//
// Deliberately asserts the package set GREW rather than naming an expected
// dependency: ComfyUI's requirements drift upstream independently of this
// contract, so a named package would rot, while "the install put something in
// the runtime" is exactly the premise the invariants need and cannot go stale.
let python = sole_runtime_python(world);
let baseline = world
.comfyui_baseline_distributions
.as_ref()
.expect("no baseline distribution set was captured");
let after = installed_distributions(&python);
assert!(
after.iter().any(|name| !baseline.contains(name)),
"ComfyUI install added no distributions to the runtime, so it installed \
nothing and the runtime-preservation checks would pass vacuously \
({} distributions before, {} after, python: {})",
baseline.len(),
after.len(),
python.display()
);
}

#[then("the runtime's torch stack is unchanged")]
async fn assert_torch_unchanged(world: &mut E2eWorld) {
let python = sole_runtime_python(world);
let after = torch_stack_versions(&python);
let baseline = world
.comfyui_baseline_torch_stack
.as_deref()
.expect("no baseline torch stack was captured");
assert_eq!(
after,
baseline,
"ComfyUI install replaced part of the managed runtime's torch stack \
(before: [{baseline}], after: [{after}], python: {})",
python.display()
);
}

#[then("no nvidia-* packages were added to the runtime")]
async fn assert_no_nvidia_packages(world: &mut E2eWorld) {
let python = sole_runtime_python(world);
let nvidia = nvidia_distributions(&python);
assert!(
nvidia.is_empty(),
"ComfyUI install added CUDA nvidia-* distributions to the ROCm runtime: {}",
nvidia.join(", ")
);
}
Loading