Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
8113718
test(e2e): pin the second README-walkthrough batch (EAI-8046..8049)
fredespi Aug 20, 2026
45bc337
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 20, 2026
9edd21a
test(e2e): assert the durable contract, not each bug's current shape
fredespi Aug 20, 2026
dd733fe
test(e2e): make the alternate-screen latch race-free
fredespi Aug 20, 2026
43f1fb3
test(e2e): satisfy the e2e-harness clippy lints
fredespi Aug 20, 2026
68bebd3
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 22, 2026
378cf78
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Aug 25, 2026
1452ee1
test(e2e): drop the duplicate run_rocm_with_stdin after absorbing main
fredespi Aug 25, 2026
832e106
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 10, 2026
3490bf6
test(e2e): harden the pinned assertions against valid fixes (PR review)
fredespi Sep 10, 2026
2c4b600
test(e2e): tighten the pinned assertions against wrong-reason passes …
fredespi Sep 10, 2026
0f6be36
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 11, 2026
a9cc712
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Sep 25, 2026
0591f0c
Merge origin/main; correct the EAI-8049 premise in engine-shell-02
fredespi Oct 23, 2026
57c52a8
test(e2e): drop speculative output shapes from the pinned assertions
fredespi Oct 23, 2026
98b6b03
Merge origin/main into e2e-pin-the-contracts-for-the-second
fredespi Oct 26, 2026
485d697
test(e2e): make engine-shell-02 flip only when the shell runs the eng…
fredespi Oct 26, 2026
20fcc4a
test(e2e): make the engine-shell interpreter step able to fail
fredespi Oct 27, 2026
9f4bc82
test(e2e): stop the interpreter step hanging the run; check VIRTUAL_ENV
fredespi Oct 27, 2026
abfc769
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Oct 29, 2026
0d286fd
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Oct 30, 2026
5139167
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Nov 2, 2026
0e4d264
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Nov 4, 2026
530f779
Merge remote-tracking branch 'origin/main' into e2e-pin-the-contracts…
fredespi Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,49 @@ bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model."
serve_timeout_secs = 90
flaky = true

# -- README-walkthrough defects (EAI-8047, EAI-8049) --

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this branch is not currently mergeable into main. GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY for this exact head. Independently reproduced with git merge-tree $(git merge-base origin/main pr/292) origin/main pr/292: the only conflict is a textual clash right here, where two independent appends landed near the same spot in this file — main has added further xfail rows in the time since this branch's last sync with it (the branch's merge-base with the current main tip is 962ac0c0, from 2026-11-04; main is ~168 commits ahead of that point as of this review). Every other changed file merges cleanly.

Not a code defect — the fix is mechanical: merge/rebase onto the current main tip, keep both appended blocks, and push.

# Defects found walking the CLI on an MI300X host, each pinned by a
# scenario asserting the CORRECT behaviour. Both are in code paths with no
# hardware, engine or network dependency, and both were re-confirmed in a plain
# Linux container — so each row is scoped `when = {}` and is expected to go stale
# (XPASS) on every lane the day its bug is fixed. Neither is `flaky`: each
# reproduced deterministically.

# --- EAI-8047: listed automation checks cannot be turned on from the listing ---
# `automations list` shows each check by display name only ("Server recovery"),
# while `automations enable` accepts the internal id ("server-recover") and
# rejects anything else with rc=1. The names do not mechanically reduce to the
# ids, so nothing the listing prints is enough to act on. Remove this row once
# the listing exposes an identifier a user can pass back in.
#
# Either direction closes it, and the scenario accepts both: show the id in the
# listing, or make `enable` accept the label exactly as listed. Note that
# `render_automations_text`'s unit tests currently assert the listing does NOT
# contain `command id:` or the raw ids (apps/rocm/src/automations.rs), so hiding them was
# a deliberate choice; a fix that shows the ids has to update those assertions.
[["automations-listed-checks-can-be-enabled"]]
when = {}
bug = "EAI-8047"
reason = "`automations list` publishes no usable identifier, so the checks it lists cannot be enabled from what it shows."

# --- EAI-8049: the engine shell's activation hint names a path that is not there ---
# `engines shell` builds the hint by appending the interpreter directory to the
# resolved env_path. Users reach it through the auto-install branch of
# `resolve_engine_env`, which uses the install response's env_path live, and the
# vLLM install reports that as the directory CONTAINING the command
# (`command.parent()`, engines/vllm/src/install.rs). The hint therefore doubles
# the segment and points at a file nobody can source. Container-measured: with a bin-shaped env_path the hint named
# `<env>/bin/bin/activate` (absent) while a root-shaped one named
# `<env>/bin/activate` (present).
#
# The scenario plants a bin-shaped manifest as a stand-in for that response: no
# install writes such a manifest to disk, it is only the cheapest way to hand the
# shell the value the vLLM install reports. A fix in engines/vllm that reports the
# environment root therefore closes EAI-8049 but does NOT flip this row, since the
# planted manifest is unchanged; delete the row and rewrite or delete the
# scenario together with that fix. Only a fix to the hint composition flips it. ---
[["engine-shell-activation-hint-is-usable"]]
when = {}
bug = "EAI-8049"
reason = "`engines shell` prints an activation hint whose path does not exist, because the env_path it is given (as reported by the vLLM install response) already includes the interpreter directory."
13 changes: 13 additions & 0 deletions tests/e2e-cucumber/features/automations.feature
Original file line number Diff line number Diff line change
Expand Up @@ -197,3 +197,16 @@ Feature: Automation watchers
And a GPU thermal alert for the server reaches the daemon again
Then the daemon records that a stop request is already waiting for the server
And only one stop request for the server is waiting for review

# The scenarios above enable a watcher by an id the test already knows. This one pins
# the complementary contract: the listing is the only place a user learns which
# background checks exist, so whatever it shows has to be enough to act on.
# Deriving each check's identifier from the listing the way a reader would IS
# the contract under test — the identifiers are deliberately not written into
# the scenario or the steps, because hard-coding them would pass against a
# listing that exposes nothing. No fixtures, no GPU, no network.
@id:automations-listed-checks-can-be-enabled
Scenario: automations-13 - Every background check that is listed can be turned on
Given a machine with no background checks turned on
When the user lists the background checks
Then every listed check can be turned on from what the listing shows
32 changes: 32 additions & 0 deletions tests/e2e-cucumber/features/engine_shell.feature
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,35 @@ Feature: Engine shell
And the engine environment's interpreter is the one that runs
When the user leaves the engine shell
Then the engine shell exits successfully

# The env_path is planted bin-shaped, the way the vLLM install RESPONSE reports it
# (`command.parent()` in engines/vllm/src/install.rs; users reach it through the
# auto-install branch of resolve_engine_env). No install writes such a manifest
# to disk — planting one is only the cheapest way to hand the shell that value
# without a multi-GiB install. The shell composes its activation hint by
# appending the interpreter directory to whatever env_path it is given, so a
# value that already points inside it yields a path nobody can source.
#
# Consequence for whoever fixes this: a fix in engines/vllm (report the env root)
# closes EAI-8049 but leaves this fixture unchanged, so the scenario must then be
# rewritten or deleted along with its expectation row. Only a fix to the hint
# composition itself makes this row go stale.
#
# The interpreter step is here because the same doubled path also sets a wrong
# VIRTUAL_ENV and puts a nonexistent bin/bin on PATH: a fix to the hint text
# alone would otherwise flip this row while the shell still runs the wrong
# python. The step runs the interpreter and also checks that VIRTUAL_ENV names
# an environment holding bin/activate, so the row flips only once the hint, PATH
# and VIRTUAL_ENV are all right.
#
# Phrased as "the file it names is there", not as "the path is not doubled": the
# first survives whichever way the mismatch is repaired, the second would go
# stale the moment it is.
@id:engine-shell-activation-hint-is-usable @requires-os:linux
Scenario: engine-shell-02 - The activation hint an engine shell prints refers to a real file
Given a machine with an engine environment installed the way the engine records it
When the user opens a shell for that engine
Then the printed activation hint names a file that exists
And the engine environment's interpreter is the one that runs
When the user leaves the engine shell
Then the engine shell exits successfully
184 changes: 184 additions & 0 deletions tests/e2e-cucumber/tests/e2e/automations_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@
//! proposal history (ROCMAI-729). One scenario (automations-12) starts a real
//! `rocmd run` and feeds it GPU thermal alerts through its loopback webhook; the
//! rest of the watcher/daemon trigger behaviour is covered separately.
//!
//! Two slices live here. The enable/disable/mode steps act on a watcher id the
//! test already knows. The listing steps (scenario 13) assert the complementary
//! discoverability contract: everything the listing shows a user must also be
//! able to act on, so they derive each check's identifier FROM the listing rather
//! than knowing it in advance — hard-coding the real ids would keep passing
//! against a listing that publishes none of them, which is the defect pinned.

use std::io::{Read, Write};
use std::process::Stdio;
Expand Down Expand Up @@ -65,6 +72,112 @@ pub(crate) fn suppress_daemon_spawn(world: &E2eWorld) {
.expect("failed to write automation runtime state");
}

/// Slugify a display name: lowercase, words joined by dashes.
///
/// This is the guess a user is forced into when the listing exposes no real
/// identifier — the current-bug fallback, not the contract. It deliberately does
/// NOT recover the true ids ("Server recovery" → `server-recovery`, not the real
/// `server-recover`), which is exactly why the listing has to publish them.
fn slug_of(display_name: &str) -> String {
display_name
.split_whitespace()
.map(str::to_lowercase)
.collect::<Vec<_>>()
.join("-")
}

/// The identifiers a check block plausibly EXPOSES, most specific first.
///
/// The contract is "enable-able from what the listing shows", so the moment the
/// product publishes an identifier this must pick it up, or a correct fix would
/// stay xfailed forever (the row would never go stale). These are CANDIDATES, not
/// a verdict: the step tries each and counts the check reachable if any works, so
/// a wrong guess here costs nothing and a missing one is the only risk.
/// - a detail line keyed `id:` / `identifier:` / `slug:`, or any key ending in
/// ` id` (`command id: server-recover`)
/// - bracketed on the header — `Server recovery [server-recover]`
/// - a single parenthesised token beside the label — `Server recovery (server-recover)`
///
/// Empty only when the block names no identifier at all, which is today's defect.
fn exposed_identifiers(block: &[&str]) -> Vec<String> {
let mut found = Vec::new();
let mut add = |id: &str| {
let id = id.trim().trim_matches(|c| c == '"' || c == '`');
if !id.is_empty() && !id.contains(char::is_whitespace) && !found.iter().any(|f| f == id) {
found.push(id.to_owned());
}
};
for line in block {
if let Some((key, rest)) = line.trim().split_once(':') {
let key = key.trim().to_ascii_lowercase();
if matches!(key.as_str(), "id" | "identifier" | "slug") || key.ends_with(" id") {
add(rest);
}
}
}
if let Some(header) = block.first().map(|h| h.trim()) {
for (open, close) in [('[', ']'), ('(', ')')] {
if let (Some(o), Some(c)) = (header.rfind(open), header.rfind(close))
&& o < c
{
add(&header[o + 1..c]);
}
}
}
found
}

/// The checks the listing advertises, as `(displayed name, identifier to try)`.
///
/// A check is a two-space-indented header line carrying a parenthesised on/off
/// state — ` Server recovery (off)`. Its own detail lines (`setting:`, `does:`,
/// and potentially an `id:`) are indented further; the report's other sections
/// list events rather than checks. Each header plus the deeper-indented lines
/// under it forms one block, so an identifier the fix exposes on a detail line
/// is seen. The identifier to invoke is the one the block explicitly exposes,
/// falling back to the display-name slug only when it exposes none (today's bug).
fn listed_checks(listing: &str) -> Vec<(String, Vec<String>)> {
let lines: Vec<&str> = listing.lines().collect();
let is_header = |line: &str| line.starts_with(" ") && !line.starts_with(" ");
let mut checks = Vec::new();
for (i, line) in lines.iter().enumerate() {
if !is_header(line) {
continue;
}
let Some(name) = line
.trim()
.strip_suffix("(on)")
.or_else(|| line.trim().strip_suffix("(off)"))
.map(str::trim)
.filter(|name| !name.is_empty())
else {
continue;
};
// The block is this header plus every following more-indented line, up
// to the next header or a less-indented line.
let mut block = vec![name];
for detail in &lines[i + 1..] {
if detail.starts_with(" ") {
block.push(detail);
} else {
break;
}
}
// Exposed identifiers first, then the label exactly as listed (a fix may make
// `enable` accept it), then its slug — today's fallback.
let mut candidates = exposed_identifiers(&block);
for fallback in [name.to_owned(), slug_of(name)] {
if !candidates.contains(&fallback) {
candidates.push(fallback);
}
}
checks.push((name.to_owned(), candidates));
}
checks
}

// ── Given ──────────────────────────────────────────────────────────

/// Plant an autostart *claim* — the harness's own (live) pid and a fresh spawn
/// time — WITHOUT planting the daemon runtime-state. Unlike `suppress_daemon_spawn`
/// (which marks the daemon already *running*, so the CLI's pre-check short-circuits
Expand Down Expand Up @@ -427,6 +540,17 @@ async fn pending_before_review_history(world: &mut E2eWorld) {
}
}

#[given("a machine with no background checks turned on")]
async fn no_background_checks(world: &mut E2eWorld) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this step duplicates an existing one. no_background_checks calls only suppress_daemon_spawn(world) — exactly the same body as fresh_config in config_steps.rs (#[given("a fresh CLI configuration")]). Not wrong (a scenario-appropriate Given phrasing is reasonable even when two step names share an implementation), but it's a literal duplicate rather than a thin wrapper reusing the existing step text — worth a one-line comment noting the deliberate duplication, or reusing the existing phrasing, so the next reader doesn't wonder whether the two are meant to diverge later.

// The scenario's isolated config starts empty, so every check is already off
// and nothing needs planting for the precondition itself. What DOES need
// planting is the running-daemon marker, for the same reason the sibling
// scenarios plant it: `automations enable` would otherwise launch a detached
// `rocm daemon` that outlives the scenario and accumulates on a persistent
// runner.
suppress_daemon_spawn(world);
}

// ── When ───────────────────────────────────────────────────────────

#[when("the user enables an automation watcher in observe mode")]
Expand Down Expand Up @@ -502,6 +626,16 @@ async fn show_old_pending(world: &mut E2eWorld) {
proposal_action(world, OLD_PENDING_PROPOSAL, "show");
}

#[when("the user lists the background checks")]
async fn user_lists_checks(world: &mut E2eWorld) {
let (stdout, stderr, rc) = crate::run_rocm(world, &["automations", "list"]);
assert_eq!(
rc, 0,
"listing the background checks failed (rc={rc}):\n{stdout}\n{stderr}"
);
world.cli_output = Some(stdout);
}

// ── Then ───────────────────────────────────────────────────────────

#[then("the CLI still lists the record before the torn tail")]
Expand Down Expand Up @@ -1170,6 +1304,56 @@ async fn still_one_stop_request_waiting(world: &mut E2eWorld) {
);
}

#[then("every listed check can be turned on from what the listing shows")]
async fn assert_listed_checks_enableable(world: &mut E2eWorld) {
let listing = world
.cli_output
.clone()
.expect("the background checks were never listed");
let checks = listed_checks(&listing);
// Guard the guard: a listing this step failed to parse would otherwise
// "prove" the contract by checking nothing at all.
assert!(
!checks.is_empty(),
"no background checks were found in the listing:\n{listing}"
);

let mut unreachable = Vec::new();
for (name, candidates) in &checks {
let mut last = String::new();
let mut reached = false;
for candidate in candidates {
let (stdout, stderr, rc) =
crate::run_rocm(world, &["automations", "enable", candidate]);
if rc == 0 {
reached = true;
break;
}
last = format!(
"tried {candidate:?}: rc={rc} {}",
stderr
.trim()
.lines()
.next()
.unwrap_or_else(|| stdout.trim())
);
}
if !reached {
unreachable.push(format!(
"{name:?} → none of {} candidate(s) worked; last {last}",
candidates.len()
));
}
}
assert!(
unreachable.is_empty(),
"the listing names {} background check(s) that cannot be turned on from what it \
shows:\n {}\nfull listing:\n{listing}",
unreachable.len(),
unreachable.join("\n "),
);
}

// ── Helpers ────────────────────────────────────────────────────────

fn record(world: &mut E2eWorld, stdout: String, stderr: String, rc: i32) {
Expand Down
Loading
Loading