Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -137,13 +137,6 @@ reason = "Lemonade managed serve reaches ready then shuts down immediately, so t
serve_timeout_secs = 90
flaky = true

# --- EAI-7384: re-provisioning nests the managed runtime path recursively
# (runtimes/wheel/.../runtimes/wheel/). Surfaces wherever a real install runs. ---
[["runtime-path-not-nested"]]
when = {}
bug = "EAI-7384"
reason = "Managed runtime path nests recursively (runtimes/wheel/...) on re-provisioning."

# The GPU-backed bench scenario benchmarks a really served model, so on a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this row only holds if every lane prints a flat active_runtime_root. With it gone, a nested root on any lane turns runtime-02 red instead of being an expected failure. Please confirm the W7900, MI300X and Strix Ubuntu lanes pass with the row removed before merge.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR text records the lane evidence for this head: runtime-02 passed on the MI300X lane and on both Strix Halo Ubuntu lanes in the first attempt of the E2E run, and the Windows lane passed too. The W7900 lane has no completed run on this head yet (it was cancelled in the first attempt and the rerun is still queued), so that is the one gap. If any lane did print a nested root, runtime-02 now fails red there where before it would have been an expected failure, which is the signal the scenario exists to give.

# lemonade LINUX host it inherits the same managed-serve bug as the sibling
# serve/chat rows below `serve-default-engine-working-endpoint`: the serve
Expand Down
1 change: 1 addition & 0 deletions tests/e2e-cucumber/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ pub mod polled_deadline;
pub mod powershell_text_rule;
pub mod quit_and_wait;
pub mod reader_failure;
pub mod runtime_root;
pub mod send_until;
pub mod serve_log;
pub mod shared_runtime;
Expand Down
159 changes: 159 additions & 0 deletions tests/e2e-cucumber/src/runtime_root.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
// Copyright © Advanced Micro Devices, Inc., or its affiliates.
//
// SPDX-License-Identifier: MIT

/// The field `rocm examine` prints for the managed runtime's install folder.
const ACTIVE_RUNTIME_ROOT_FIELD: &str = "active_runtime_root";

/// How many times the `runtimes/wheel` pair of path segments appears in `root`.
///
/// Compared segment by segment, not as a substring, so a folder that merely ends
/// in `runtimes` or starts with `wheel` is not counted. Both separators are
/// accepted because the root is printed in the host's own form.
fn runtimes_wheel_count(root: &str) -> usize {
let segments: Vec<&str> = root
.split(['/', '\\'])
.filter(|segment| !segment.is_empty())
.collect();
segments
.windows(2)
.filter(|pair| pair[0] == "runtimes" && pair[1] == "wheel")
.count()
}

/// The active runtime's folder as `rocm examine` states it, if it states one.
///
/// Only a line whose name is exactly `active_runtime_root` counts. `examine`
/// prints it only while a managed runtime is active, so `None` means there is
/// nothing to judge. A sibling such as `active_runtime_pip_cache_dir` shares the
/// prefix but is another fact and is not read here.
fn active_runtime_root(examine_output: &str) -> Option<&str> {
examine_output.lines().find_map(|line| {
let (name, value) = line.trim().split_once(':')?;
(name == ACTIVE_RUNTIME_ROOT_FIELD).then(|| value.trim())
})
}

/// Judge the active managed runtime's folder from `rocm examine` output.
///
/// Re-provisioning inside an existing runtime once produced a folder holding
/// `runtimes/wheel` twice (`runtimes/wheel/<key>/runtimes/wheel/<key>`). A healthy
/// folder holds it at most once. Returns the folder when it is fine, and the
/// reason when it is not, including when the output states no folder at all,
/// which a scenario that made a runtime active must never see.
///
/// # Errors
///
/// When no `active_runtime_root` line is present, its value is empty, or the
/// folder is nested.
pub fn check_runtime_root_not_nested(examine_output: &str) -> Result<&str, String> {
let Some(root) = active_runtime_root(examine_output) else {
return Err(format!(
"no '{ACTIVE_RUNTIME_ROOT_FIELD}:' line in the examine output, so there is no \
managed runtime folder to judge"
));
};
if root.is_empty() {
return Err(format!(
"the examine output's '{ACTIVE_RUNTIME_ROOT_FIELD}:' line names no folder"
));
}
let count = runtimes_wheel_count(root);
if count > 1 {
return Err(format!(
"the managed runtime folder is recursively nested: `runtimes/wheel` appears \
{count} times in {root}"
));
}
Ok(root)
}

#[cfg(test)]
mod tests {
use super::check_runtime_root_not_nested;

// The shape of an `active_runtime_root` a healthy managed runtime printed on a
// Linux and a Windows GPU lane: `<data dir>/runtimes/wheel/<runtime key>`.
const LINUX_ROOT: &str = "/home/runner/_work/rocm-cli/e2e-prewarm-multi-arch-v2/data/runtimes/wheel/release-wheel-multi-arch-7-14-1-0b5cf9d493993efb";
const WINDOWS_ROOT: &str = r"C:\runner\_work\rocm-cli\e2e-prewarm-multi-arch-v2\data\runtimes\wheel\release-wheel-multi-arch-7-14-1-f1ec7c6b4e815e9f";

fn examine_output(root_line: &str) -> String {
format!(
"ROCm setup check\n AMD GPU detected\n\nDetails\nrocm examine\n os: linux\n \
managed_runtimes: 1\n active_runtime_status: ready\n{root_line}\n \
active_runtime_version: 7.14.1\n"
)
}

#[test]
fn a_flat_linux_root_passes_and_is_returned() {
let output = examine_output(&format!(" active_runtime_root: {LINUX_ROOT}"));
assert_eq!(check_runtime_root_not_nested(&output), Ok(LINUX_ROOT));
}

#[test]
fn a_flat_windows_root_passes_with_its_drive_colon_intact() {
let output = examine_output(&format!(" active_runtime_root: {WINDOWS_ROOT}"));
assert_eq!(check_runtime_root_not_nested(&output), Ok(WINDOWS_ROOT));
}

#[test]
fn a_root_holding_runtimes_wheel_twice_fails_naming_it() {
let nested = format!("{LINUX_ROOT}/runtimes/wheel/release-wheel-other");
let output = examine_output(&format!(" active_runtime_root: {nested}"));
let error = check_runtime_root_not_nested(&output).unwrap_err();
assert!(error.contains("recursively nested"), "{error}");
assert!(error.contains("2 times"), "{error}");
assert!(error.contains(&nested), "{error}");
}

#[test]
fn a_nested_windows_root_fails_in_backslash_form() {
let nested = format!(r"{WINDOWS_ROOT}\runtimes\wheel\release-wheel-other");
let output = examine_output(&format!(" active_runtime_root: {nested}"));
assert!(check_runtime_root_not_nested(&output).is_err());
}

#[test]
fn a_nested_root_fails_when_the_separators_are_mixed() {
let nested = r"C:\data\runtimes\wheel\key/runtimes/wheel/key2";
let output = examine_output(&format!(" active_runtime_root: {nested}"));
assert!(check_runtime_root_not_nested(&output).is_err());
}

#[test]
fn folders_that_only_resemble_the_pair_are_not_counted() {
let root = "/data/old-runtimes/wheel-cache/runtimes/wheel/key";
let output = examine_output(&format!(" active_runtime_root: {root}"));
assert_eq!(check_runtime_root_not_nested(&output), Ok(root));
}

#[test]
fn a_missing_field_fails_loudly() {
let output = "ROCm setup check\n AMD GPU detected\n managed_runtimes: 0\n";
let error = check_runtime_root_not_nested(output).unwrap_err();
assert!(error.contains("active_runtime_root"), "{error}");
}

#[test]
fn an_empty_value_fails_loudly() {
let output = examine_output(" active_runtime_root:");
let error = check_runtime_root_not_nested(&output).unwrap_err();
assert!(error.contains("names no folder"), "{error}");
}

#[test]
fn only_the_exact_field_is_read() {
// A nested path in a sibling field, or in text that mentions the old label,
// is neither read as the root nor allowed to stand in for a missing one.
let nested = format!("{LINUX_ROOT}/runtimes/wheel/x");
let output = format!(
" active_runtime_pip_cache_dir: {nested}\n Folder: {nested}\n active_runtime_root: {LINUX_ROOT}\n"
);
assert_eq!(check_runtime_root_not_nested(&output), Ok(LINUX_ROOT));

let only_siblings =
format!(" active_runtime_pip_cache_dir: {LINUX_ROOT}\n Folder: {LINUX_ROOT}\n");
assert!(check_runtime_root_not_nested(&only_siblings).is_err());
}
}
22 changes: 5 additions & 17 deletions tests/e2e-cucumber/tests/e2e/runtime_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1061,24 +1061,12 @@ async fn assert_examine_reports_no_toolchain(world: &mut E2eWorld) {

#[then("the managed runtime folder path is not recursively nested")]
async fn assert_runtime_path_not_nested(world: &mut E2eWorld) {
// `rocm examine` prints `Folder: <install_root>` for the active runtime.
// `rocm examine` prints `active_runtime_root: <install_root>` for the active

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scenario only inspects the runtime that is already active, so it will not reproduce nesting created by re-provisioning. It is a weak regression guard for the original bug; worth saying so in the PR text. Also worth recording a mutation check for the new unit tests (e.g. change the count > 1 bound to count > 2 and show them go red).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on both points, and both are in the PR text now. It says that runtime-02 inspects only the runtime that is active when it runs, so it is a narrower guard than the original bug and would not reproduce nesting created by re-provisioning. I also ran the mutation you suggested, changing the count > 1 bound to count > 2: the three nested-root unit tests go red and the other six stay green, and the file was restored afterwards.

// managed runtime; the judgement lives in the library so unit tests run it.
let output = world.cli_output.as_ref().expect("no examine output");
let Some(folder) = output
.lines()
.find_map(|l| l.trim().strip_prefix("Folder:"))
.map(str::trim)
else {
panic!("no 'Folder:' line in examine output:\n{output}");
};
// A healthy path contains `runtimes/wheel` at most once. Re-provisioning
// inside an existing runtime produces `runtimes/wheel/.../runtimes/wheel/`
// (dogfooding #17). Count occurrences of the marker segment.
let nested = folder.matches("runtimes/wheel").count() > 1
|| folder.matches("runtimes\\wheel").count() > 1;
assert!(
!nested,
"managed runtime folder path is recursively nested (dogfooding #17):\n{folder}"
);
if let Err(reason) = e2e_cucumber::runtime_root::check_runtime_root_not_nested(output) {
panic!("{reason}\nexamine output:\n{output}");
}
}

#[when("the user checks for runtime updates")]
Expand Down
Loading