Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -379,9 +379,9 @@ jobs:
# superset of this one. `xtask/src/workflow_contract.rs` asserts the
# floor for that reason.
#
# The floor is what matters here. The suite's deterministic failure
# seams (e.g. the scripted Lemonade backend-install failure) are
# compiled out without it, so a pre-built binary that omits the feature
# The floor is what matters here. The suite's test seams (e.g. the
# simulated host root, ROCM_CLI_TEST_HOST_ROOT) are compiled out
# without it, so a pre-built binary that omits the feature
# leaves those scenarios unable to reach their premise — they then fail
# as regressions on whichever lane happens to select them. Every lane
# that pre-builds and exports ROCM_CLI_BINARY must pass it.
Expand Down
4 changes: 3 additions & 1 deletion apps/rocm/src/engines_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ pub(crate) fn env_root_for_service(
/// literal and had drifted several minor versions behind what is installed.
fn managed_engine_runtime_id(engine: &str) -> String {
match engine {
"lemonade" => format!("lemonade-embeddable-{}", rocm_deps::LEMONADE_VERSION),
"lemonade" => rocm_deps::lemonade_env_id(rocm_deps::LEMONADE_VERSION),
_ => "managed-engine-runtime".to_owned(),
}
}
Expand Down Expand Up @@ -619,6 +619,8 @@ mod tests {
.unwrap_err()
.to_string();
assert!(error.contains("no active ROCm runtime is configured"));
// Spelled out rather than via `rocm_deps::lemonade_env_id`, so a change
// to that helper's shape turns this red.
assert_eq!(
resolve_engine_install_runtime_id(&paths, &RocmCliConfig::default(), "lemonade", None)?,
format!("lemonade-embeddable-{}", rocm_deps::LEMONADE_VERSION),
Expand Down
42 changes: 19 additions & 23 deletions apps/rocm/src/serve_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -420,13 +420,17 @@ fn aim_board_note(
/// policy, on a host the probe *authoritatively* reports as having no usable
/// AMD GPU.
///
/// Not "exactly when the bail fires", and the difference is only in test-hook
/// builds: the bail also requires `!scripted_backend_failure` (see the `if`
/// below it), which is `cfg!(feature = "e2e-test-hooks")`/env-gated. Under that
/// scenario the gate can open while the bail cannot fire. That is deliberate —
/// the hook stands in for the GPU the lane does not have, so the reuse path it
/// exercises must still behave as it would on a real GPU-less host — and it
/// costs nothing in a shipped binary, where the term can never be true.
/// Not "exactly when the bail fires". In every build the bail is also skipped
/// when this gate's own answer is a live service to reuse (`reuse_existing`).
/// E2E builds add two more skip terms (see the `if` below it), each env-gated
/// and compiled in only with its feature: `!e2e_simulate_oom_launch()`
/// (`e2e-oom-fault-injection`, either mode) stands in for the GPU the
/// fabricated-launch scenarios' lane does not have, and `!scripted_free_space` (`e2e-test-hooks`) lets the
/// Lemonade free-space refusal, which `rocm serve` only reaches past the bail,
/// be exercised on a GPU-less lane. Under either hook the gate can open while
/// the bail cannot fire. That is deliberate — the pre-gate must still behave as
/// it would on a real GPU-less host — and it costs nothing in a shipped binary,
/// where neither hook term can be true.
///
/// `host_has_no_usable_gpu` is therefore the bail's own predicate, computed once
/// by the caller and handed to both, so the two cannot drift apart. They did:
Expand Down Expand Up @@ -861,24 +865,16 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
//
// Skipped for cpu_only and when reusing an already-running service (nothing is
// launched); permissive when availability cannot be probed on this platform
// (probe returns `None`). The E2E-only
// backend-failure scenario bypasses this host precondition so the black-box
// test reaches Lemonade's backend boundary without real GPU hardware, and the
// E2E-only launch fault injection (either mode) likewise stands in for the
// GPU it does not have.
// (probe returns `None`). The E2E-only launch fault injection (either mode)
// stands in for the GPU its lane does not have.
//
// The scripted free-space override opens the same waiver: it exists to refuse a
// The scripted free-space override waives it too: it exists to refuse a
// Lemonade install on a drive that cannot hold it, which `rocm serve` only
// reaches past this bail.
let scripted_backend_failure = cfg!(feature = "e2e-test-hooks")
&& [
"ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE",
"ROCM_E2E_LEMONADE_FREE_SPACE_BYTES",
]
.iter()
.any(|variable| std::env::var_os(variable).is_some());
let scripted_free_space = cfg!(feature = "e2e-test-hooks")
&& std::env::var_os("ROCM_E2E_LEMONADE_FREE_SPACE_BYTES").is_some();
if !cpu_only
&& !scripted_backend_failure
&& !scripted_free_space
&& !e2e_simulate_oom_launch()
&& !reuse_existing
&& host_has_no_usable_gpu
Expand All @@ -892,10 +888,10 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> {
// Whether the refusal above was waived *only* because a live service was
// expected to be reused. `reuse_existing` already implies `!cpu_only` and
// `host_has_no_usable_gpu` (the pre-gate requires both), so the scripted
// backend failure is the one remaining term. Handed to the spawn, which
// free-space override is the one remaining term. Handed to the spawn, which
// re-checks reuse authoritatively under the launch lock and raises the
// refusal itself if the service is gone by then.
let gpu_refusal_waived_for_reuse = reuse_existing && !scripted_backend_failure;
let gpu_refusal_waived_for_reuse = reuse_existing && !scripted_free_space;
// `--gpu` selects by the amd-smi `gpu` ordinal but is exported via
// `HIP_VISIBLE_DEVICES`; those orderings can diverge when
// `ROCR_VISIBLE_DEVICES`/partitioning is in play, so warn at serve time.
Expand Down
25 changes: 25 additions & 0 deletions crates/rocm-deps/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,22 @@ pub fn lemonade_download_url(version: &str, archive_name: &str) -> String {
)
}

/// The `env_id` a Lemonade embeddable runtime of `version` is recorded under.
///
/// `version` is the bare release number, as in `LEMONADE_VERSION`, not a `v`
/// tag: the id is compared by exact string, so it is not normalised here.
///
/// For example `lemonade-embeddable-11.5.1`. The engine writes it into the install
/// manifest it creates, and `rocm` compares it with what the engine's `Detect`
/// reports to decide whether its own install is already done, so the two must
/// agree. The e2e suite repeats the shape when it plants a runtime, because it
/// imports nothing from this workspace; its serve-18 scenario fails if the
/// planted id stops matching.
#[must_use]
pub fn lemonade_env_id(version: &str) -> String {
format!("lemonade-embeddable-{version}")
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down Expand Up @@ -86,4 +102,13 @@ mod tests {
"https://github.com/lemonade-sdk/lemonade/releases/download/v11.6.0/archive.tar.gz"
);
}

#[test]
fn the_lemonade_env_id_names_the_embeddable_and_its_version() {
assert_eq!(lemonade_env_id("11.5.1"), "lemonade-embeddable-11.5.1");
assert_eq!(
lemonade_env_id(LEMONADE_VERSION),
format!("lemonade-embeddable-{LEMONADE_VERSION}")
);
}
}
6 changes: 4 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -623,7 +623,7 @@ Pinned versions of the third-party runtimes rocm-cli manages, read from the work

| File | Responsibility | Class |
| --- | --- | --- |
| [`lib.rs`](../crates/rocm-deps/src/lib.rs) | Includes the generated constants and derives Lemonade release names and download URLs from them. | flat |
| [`lib.rs`](../crates/rocm-deps/src/lib.rs) | Includes the generated constants and derives Lemonade release names, download URLs and the install `env_id` from them. | flat |

### `crates/e2e-report`

Expand Down Expand Up @@ -811,9 +811,10 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr
| [`src/powershell_text_rule.rs`](../tests/e2e-cucumber/src/powershell_text_rule.rs) | The text rule for the Windows lines `rocm fix` and `rocm diagnose` print, a copy of the rule in `rocm-core` checked against a shared fixture table. | flat |
| [`src/quit_and_wait.rs`](../tests/e2e-cucumber/src/quit_and_wait.rs) | Retry loop for quitting a TUI that may open a quit-confirm prompt first. | flat |
| [`src/reader_failure.rs`](../tests/e2e-cucumber/src/reader_failure.rs) | Failure state shared between a background terminal reader and its waiters. | flat |
| [`src/runtime_pins.rs`](../tests/e2e-cucumber/src/runtime_pins.rs) | Reads a pinned runtime version out of the workspace-root [`runtime-deps.toml`](../runtime-deps.toml) as plain data, so a planted runtime keyed by that pin (the Lemonade `env_id`) follows a version bump without importing `rocm-deps`. | flat |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this row sorts before src/runtime_root.rs in the surrounding alphabetical table.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6898ef9: the runtime_pins.rs row now sorts before runtime_root.rs.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this row is out of alphabetical order — it's placed after runtime_root.rs, but "runtime_pins" sorts before "runtime_root". Confidence 90.

Fix: move this row above the runtime_root.rs row.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6898ef9 (same fix as your earlier comment on this row).

| [`src/runtime_root.rs`](../tests/e2e-cucumber/src/runtime_root.rs) | `check_runtime_root_not_nested`: judges the active runtime folder `rocm examine` prints and rejects a doubly nested `runtimes/wheel` path. | flat |
| [`src/send_until.rs`](../tests/e2e-cucumber/src/send_until.rs) | Retry loop for sending an idempotent terminal input until its effect appears. | flat |
| [`src/serve_log.rs`](../tests/e2e-cucumber/src/serve_log.rs) | Reads back a managed serve's service log and builds the failure report for a serve step, quoting the log tail. | flat |
| [`src/serve_log.rs`](../tests/e2e-cucumber/src/serve_log.rs) | Reads back a managed serve's service log and builds the failure report for a serve step, quoting the log tail; and tells apart the two ways a failed `rocm serve --wait` reports a dead engine (`engine_death_report`). | flat |
| [`src/serve_plan.rs`](../tests/e2e-cucumber/src/serve_plan.rs) | Explains why a `rocm serve` run printed no `resolved model:` line by keeping stderr's refusal alongside stdout. | flat |
| [`src/shared_runtime.rs`](../tests/e2e-cucumber/src/shared_runtime.rs) | Picks which runtime a scenario activates out of the shared pre-warmed runtime tree, which may hold more than one. | flat |
| [`src/stand_in_uname.rs`](../tests/e2e-cucumber/src/stand_in_uname.rs) | The shell script a scenario plants as a stand-in `uname`: `-v` answers a fixed string, `-r` runs a fragment the scenario chooses, and every call is logged so a scenario can count how often the kernel release was asked for. | flat |
Expand All @@ -827,6 +828,7 @@ Gherkin `.feature` files under `features/` backed by Rust step functions that dr
| File | Responsibility | Class |
| --- | --- | --- |
| [`src/bin/fake-docker.rs`](../tests/e2e-cucumber/src/bin/fake-docker.rs) | A `docker` stand-in that always fails, optionally logging its arguments, for the AIM Docker-unavailable scenarios. | flat |
| [`src/bin/fake-lemonade.rs`](../tests/e2e-cucumber/src/bin/fake-lemonade.rs) | The `lemonade` CLI and `lemond` server of a planted Lemonade runtime, chosen by `argv[0]`: `lemond` stays alive for a bounded time, and `lemonade backends install` fails, for `@id:serve-lemonade-preparation-recovery`. | flat |
| [`src/bin/fake-python.rs`](../tests/e2e-cucumber/src/bin/fake-python.rs) | A stand-in for the runtime's Python that answers the ComfyUI install's torch-stack probe and post-install GPU check. | flat |
| [`src/bin/fake-tailscale.rs`](../tests/e2e-cucumber/src/bin/fake-tailscale.rs) | A Tailscale CLI stand-in that serves the status document named by `FAKE_TAILSCALE_STATUS`, for the `rocm remote` scenarios. | flat |
| [`src/bin/fake-uv.rs`](../tests/e2e-cucumber/src/bin/fake-uv.rs) | A `uv` stand-in whose output, exit code, argument log and stdin handling are set by `FAKE_UV_*` variables, for the ComfyUI scenarios. | flat |
Expand Down
7 changes: 4 additions & 3 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -992,8 +992,9 @@ pushed (`PRE_COMMIT_FROM_REF..PRE_COMMIT_TO_REF`, the commits the remote does no
The directory-based selection cannot see a crate's tests reading a file from outside the crate,
so the hook widens it: any Rust file also selects `xtask` (its contract tests scan every source
file), `README.md`, `docs/` and `skills/` also select `rocm` and `xtask` (the advised-commands
test reads them and `rocm` compiles in the assistant skill), and the e2e PowerShell fixture also
selects `rocm-core` (its tests `include_str!` it). The table is `CROSS_CRATE_READS` in
test reads them and `rocm` compiles in the assistant skill), and the e2e `fixtures/` directory also
selects `rocm-core` (its tests `include_str!` the PowerShell cases), and the fake Lemonade runtime's
backends table there also selects `rocm-engine-lemonade` (its tests `include_str!` it). The table is `CROSS_CRATE_READS` in
[prepush.rs](../xtask/src/prepush.rs), and a unit test scans the workspace for `include_str!`,
`include_bytes!` and manifest-relative reads that leave their crate and fails when one is not
covered, so a new cross-crate read cannot be added without a decision. The scan has a limit: a read
Expand Down Expand Up @@ -2049,7 +2050,7 @@ Lemonade copies ROCm CLI keeps (ROCMAI-825), all on the mock lane:
`ROCM_E2E_LEMONADE_FREE_SPACE_BYTES`, compiled only into the
`rocm/e2e-test-hooks` build `cargo xtask e2e` makes, and check against the
disk that nothing was downloaded. That variable also waives serve's no-GPU
refusal, as serve-18's failure hook does.
refusal, so the scenario reaches the install on a host with no GPU.
- `lemonade-installs-03` plants a data-folder copy and a copy inside a runtime
and checks `rocm storage report` lists both and flags the unused one.
- `lemonade-installs-04` uninstalls that runtime and checks the install record
Expand Down
2 changes: 1 addition & 1 deletion engines/lemonade/src/install.rs
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ where
);
}
Ok(LemonadeInstallManifest {
env_id: format!("lemonade-embeddable-{version}"),
env_id: rocm_deps::lemonade_env_id(&version),
version,
runtime_dir,
lemond,
Expand Down
12 changes: 0 additions & 12 deletions engines/lemonade/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,6 @@ pub(crate) const DEFAULT_HOST: &str = "127.0.0.1";
pub(crate) const DEFAULT_MODEL: &str = "Qwen3-4B-Instruct-2507-GGUF";
pub(crate) const LLAMACPP_RECIPE: &str = "llamacpp";
pub(crate) const ROCM_BACKEND_NAME: &str = "rocm";
#[cfg(feature = "e2e-test-hooks")]
const BACKEND_INSTALL_FAILURE_TEST_ENV: &str = "ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE";
const DEFAULT_LOG_TAIL_LINES: usize = 200;
/// How long a stop waits for the server to actually exit after each signal
/// before reporting a timeout (or, under `force`, escalating to `SIGKILL`).
Expand Down Expand Up @@ -398,16 +396,6 @@ fn install_and_record(
fn install_response(request: InstallRequest) -> Result<InstallResponse> {
let paths = AppPaths::discover()?;
paths.ensure()?;
// Debug builds expose a deterministic failure seam for the black-box CLI
// scenario that pins retry count and terminal recovery guidance. Keep it at
// the backend phase: the real defect happens after the embeddable is ready,
// and exercising it must not download or alter a runtime on the test host.
#[cfg(feature = "e2e-test-hooks")]
if std::env::var_os(BACKEND_INSTALL_FAILURE_TEST_ENV).is_some() {
crate::llamacpp_backend::install_llamacpp_backend_with_retry(|| {
anyhow::bail!("scripted Lemonade backend install failure")
})?;
}
let env_root = request
.env_root
.as_deref()
Expand Down
30 changes: 30 additions & 0 deletions engines/lemonade/src/llamacpp_backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,36 @@ mod tests {
}
}

/// The E2E fake runtime's `lemonade backends` table must drive this engine
/// to attempt a `llamacpp:rocm` install: that attempt is the whole premise
/// of `@id:serve-lemonade-preparation-recovery`, which then fails it.
///
/// The fixture prints the table from a data file, and this test reads the
/// same file, so editing it there is checked here, on every lane that runs
/// unit tests. A parser change that stopped accepting it would otherwise
/// make the scenario stop reaching the failure it asserts, with no unit
/// test noticing.
#[test]
fn the_e2e_fake_runtime_table_drives_a_rocm_backend_install() {
const TABLE: &str =
include_str!("../../../tests/e2e-cucumber/fixtures/lemonade_backends_table.txt");
let backends = parse_llamacpp_backend_statuses(TABLE);
assert_eq!(
backends,
vec![
("rocm".to_owned(), "installable".to_owned()),
("vulkan".to_owned(), "unsupported".to_owned()),
]
);
// `false` = not already installed, which is what makes the engine run
// the install the fixture then refuses. `installed` would skip it, and
// the scenario would never see a failure.
assert_eq!(
select_best_llamacpp_backend(&backends),
Some(("rocm".to_owned(), false))
);
}

#[test]
fn parses_llamacpp_backends_from_table() {
let output = "\
Expand Down
7 changes: 7 additions & 0 deletions tests/e2e-cucumber/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,13 @@ path = "src/bin/fake-python.rs"
name = "fake-docker"
path = "src/bin/fake-docker.rs"

# Stand-in for a Lemonade runtime's `lemonade` and `lemond` executables, planted
# by the serve-preparation-recovery scenario so Lemonade's backend install fails
# at a real child process rather than at a seam compiled into the engine.
[[bin]]
name = "fake-lemonade"
path = "src/bin/fake-lemonade.rs"

[dependencies]
axum.workspace = true
cucumber = { version = "0.23", features = ["output-json", "output-junit"] }
Expand Down
Loading
Loading