From e9ae13df360358b564995b026de2e8748e4b66bd Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Fri, 6 Nov 2026 15:05:51 +0000 Subject: [PATCH 001/102] feat(xtask): require every source file to have an entry in docs/architecture.md (#949) * feat(xtask): add check-architecture-doc, requiring every source file to be in the module map docs/architecture.md says it is updated in the same PR as the code it documents, but nothing enforced it. The lychee link check only confirms that links already in the doc resolve, so a new module nobody wrote an entry for passed silently. `cargo xtask check-architecture-doc` starts from the tree instead: every tracked .rs file under apps/*/src, crates/*/src and engines/*/src must be linked from the doc, listed in EXEMPT, or be a test-only helper matched by file stem. EXEMPT holds the 78 files uncited today and, like ALLOWLIST in check-crate-edges, is checked both ways, so it can only shrink. The architecture-doc prek hook runs it at pre-commit and pre-push, over the whole tree on every run, so a commit made with --no-verify is still caught at push. Signed-off-by: Jussi Elo * ci: run check-architecture-doc in the clippy job and pin the wiring with a contract test The step is gated on the rust path filter like the other xtask checks in that job. The filter now also lists docs/architecture.md, because deleting or rewording an entry changes the answer though no .rs file moved. architecture_doc_check_runs_in_hooks_and_ci reads .pre-commit-config.yaml and ci.yml and fails if the hook loses a stage, gains a file filter or stops running on every commit, or if the CI step or the path-filter entry goes missing. Signed-off-by: Jussi Elo * docs: describe the module-map check in architecture, contributing, testing and agent guides architecture.md and CONTRIBUTING.md previously said only that file links in the doc are checked for resolution. They now say the reverse is checked too: a production source file with no entry fails check-architecture-doc, and what the check does not cover (whether an entry's prose is still true). AGENTS.md section 5 and 6 and the PR template ask for the entry in the same change as the file, and say not to add a file to EXEMPT to make the check pass. Signed-off-by: Jussi Elo * fix(xtask): ignore links in HTML comments and code when checking the module map The citation scan looked for a raw `](` in docs/architecture.md, so a link inside an HTML comment, a fenced block or an inline code span still counted. Commenting out the only entry for a file kept the check green although the rendered module map no longer documented it. The doc is now reduced to its rendered link text first: HTML comments, fenced code blocks and code spans are removed in one left-to-right pass, so a backtick inside a comment and a comment marker inside a code span keep their usual precedence. Where it cannot tell, it hides a link, which can only make the check fail. Indented code blocks are not recognised. New tests cover each construct, the precedence cases, CRLF input, and the commented-out-entry scenario at the check level. Signed-off-by: Jussi Elo * chore(xtask): drop dash-daemon registry.rs from EXEMPT now that the module map cites it The module map gained an entry for crates/rocm-dash-daemon/src/registry.rs on main, so the stale-entry check rightly rejected its EXEMPT line after the merge. Signed-off-by: Jussi Elo * fix(xtask): parse complete inline links when checking the module map The citation scan took any `](` as a link, which both over- and under-counted. A destination in angle brackets, `[cli](<../apps/rocm/src/cli.rs>)`, was rejected because the `<` and `>` stayed in the path, so reformatting an existing entry failed the hook and CI. A link missing its `)` or its `[`, or with an escaped bracket, was counted as a citation although the rendered page shows plain text, so breaking the only entry for a file kept the gate green. Links are now parsed as `[text](destination "title")`: an unescaped `[` on the same line, a balanced-parenthesis or `<...>` destination, an optional title and the closing `)`. A link that is not complete is not a citation. A link whose text spans lines is not recognised, which can only make the check fail. New tests cover angle-bracket destinations, each missing delimiter, escaped brackets and parentheses, a stray `[` on an earlier line, nested brackets, parentheses in a path, and several links on one line. Signed-off-by: Jussi Elo * fix(xtask): ratchet the EXEMPT list, drop the EXEMPT hint, and exclude images and nested links Review feedback on the module-map check. The failure message for an uncited file told authors they could add it to EXEMPT, which AGENTS.md and CONTRIBUTING.md say not to do; it now only asks for an entry. "EXEMPT can only shrink" was not enforced, so EXEMPT_CEILING now caps the list's size in the command and in a test: growing the list means raising that constant too, a visible reviewed edit. The docs say "meant to shrink" instead of promising what the code did not guarantee. The link scanner also treated three shapes as citations or non-citations wrongly. An image, ![x](path.rs), renders as an image and no longer counts, while a link whose text contains an image still does. The outer brackets of a link that contains another link no longer count, since links do not nest. A backslash-escaped backtick no longer opens a code span, which had swallowed the `[` of a real link and reported its file as uncited. Signed-off-by: Jussi Elo * docs(xtask): say what the module-map check reads and skips, and document rocm-engine-protocol Review notes on the check. The check reads the index and the working tree. At pre-commit prek stashes unstaged changes first, so it matches what is being committed; at pre-push it reads the working tree, not the commits being pushed, so it only agrees with CI on a clean tree. The module comment, the hook comment and the contract test's doc no longer say pre-push always catches a commit made with --no-verify, and the module comment says CI is the gate. The docs now name what is out of scope (build.rs, tests/, examples/) and which link forms are not recognised (reference-style, a ?query suffix, root-relative paths). The failure message says so too, and a test holds that claim to what the parser does. crates/rocm-engine-protocol/src/lib.rs, a contract surface, now has an entry in docs/architecture.md and comes off EXEMPT, and EXEMPT_CEILING drops to 76. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 0b6b4a6cdbbb373316533f0b4410ed16d984370c) Signed-off-by: juhovainio --- .github/pull_request_template.md | 1 + .github/workflows/ci.yml | 8 + .pre-commit-config.yaml | 20 + AGENTS.md | 5 + CONTRIBUTING.md | 6 +- docs/architecture.md | 4 + docs/testing.md | 16 + xtask/src/architecture_doc.rs | 988 +++++++++++++++++++++++++++++++ xtask/src/main.rs | 6 + xtask/src/workflow_contract.rs | 69 +++ 10 files changed, 1121 insertions(+), 2 deletions(-) create mode 100644 xtask/src/architecture_doc.rs diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index ad8c456508..9a14c25e79 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,4 +6,5 @@ SPDX-License-Identifier: MIT - [ ] If this PR fixes a bug, searched `tests/e2e-cucumber/expectations.toml` for the fixed ticket ID and removed/narrowed any now-stale xfail rows. - [ ] If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per `docs/architecture.md` (the clap declaration in `cli.rs` and the dispatch wiring in `main.rs`/`lib.rs` are expected, not a violation). +- [ ] If this PR adds, renames or removes a production source file, `docs/architecture.md` has the matching entry (`cargo xtask check-architecture-doc` fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify. - [ ] Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will *not* happen. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4bf61e7b7e..77ad26c9d2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -165,6 +165,10 @@ jobs: # `heavy`-gated) doesn't silently skip the test that reads it. - 'docs/ci-hardware-testing.md' - 'tests/e2e-cucumber/README.md' + # `cargo xtask check-architecture-doc` (clippy job) compares this + # module map with the source tree; deleting or rewording an entry + # can break it though no *.rs file moved. + - 'docs/architecture.md' # license_headers_hook_has_no_file_type_filter (xtask) reads this # file's license-headers hook config; an edit here that reopens the # file-type-filter gap must still run that test on the PR. @@ -1420,6 +1424,10 @@ jobs: if: needs.changes.outputs.rust == 'true' run: cargo xtask check-crate-edges + - name: Every production source file has an entry in docs/architecture.md + if: needs.changes.outputs.rust == 'true' + run: cargo xtask check-architecture-doc + - name: Published Doctor catalog is current if: needs.changes.outputs.rust == 'true' run: cargo xtask catalog --check diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fa8a4a489b..ba014b31c7 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -119,6 +119,26 @@ repos: stages: [pre-commit] groups: [local-tools] # runs in the dedicated `docs-links` CI job + # Module map: every production source file needs an entry in + # docs/architecture.md. `lychee` above only checks that links already in the + # doc resolve; this is the other direction, so a new file nobody wrote an + # entry for fails. Pre-commit for fast feedback, pre-push for a commit made + # with --no-verify: it reads the working tree, not the pushed commits, so it + # only agrees with CI when the tree is clean, and CI is the gate. It reads the + # whole tree, so it is not narrowed to changed files: a rename or deletion + # anywhere can change the answer, and a delete-only commit has no changed + # files to match a filter. + - repo: local + hooks: + - id: architecture-doc + name: every source file is in docs/architecture.md + entry: cargo xtask check-architecture-doc + language: system + always_run: true + pass_filenames: false + stages: [pre-commit, pre-push] + groups: [local-tools] # runs in the `clippy` CI job (compiles xtask) + # Rust + PowerShell: mirror the CI checks using the local toolchain. - repo: local hooks: diff --git a/AGENTS.md b/AGENTS.md index 03a03d330c..e9db92e2f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -227,6 +227,10 @@ Keep docs and behavior claims in sync while editing: file outside it), and a link leaving a skill folder in `skills/` (published or embedded elsewhere, §7). This covers citations you add or edit, not converting existing bare paths in an unrelated change +- when you add, rename or remove a production source file, update its entry in + `docs/architecture.md` in the same change; `cargo xtask check-architecture-doc` + fails on a file with no entry (§6), but not on an entry whose description + went stale - when a statement such as "X is not supported" or "there is no Y" becomes false, grep the same surfaces for the old claim before pushing - write the PR body from the final diff, not the first commit, and re-read it after @@ -274,6 +278,7 @@ Guardrails: - `crates/rocm-engine-protocol` is a contract surface; verify all impacted engines after protocol changes - first-party crate-layering invariants (e.g. `rocmd` must never depend on `rocm`) are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`); a new first-party dependency edge failing that check means the edge needs review, not a bypass - source file size is checked by `cargo xtask check-module-size` ([`module_size.rs`](xtask/src/module_size.rs)): no source file may exceed 1,500 production lines (test-gated items excluded), with one limit for every file and no exemption mechanism. A PR that would push a file past it splits the file in the same change; do not raise the limit or opt a file out. Merely editing a file that is already over the limit does not by itself require a split: those files are split under their own tickets, and the check cannot become blocking until none is left. Today the CI step and pre-commit hook only report; see `docs/architecture.md` +- every production source file (a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`) has an entry in `docs/architecture.md`, enforced by `cargo xtask check-architecture-doc` ([architecture_doc.rs](xtask/src/architecture_doc.rs)) in the `clippy` CI job and in the `architecture-doc` prek hook at commit and push. Add the entry in the same change as the file. Do not add the file to the check's `EXEMPT` list to make it pass: that list holds only what was uncited when the check was added and is meant to shrink: its size is capped by `EXEMPT_CEILING`, so growing it is a visible, reviewed exception. Remove an entry from it when you document that file. The check reads no prose, so keeping an entry's description true is still on you - preserve strict GPU-required behavior; do not introduce silent CPU fallback - respect platform gates (for example, native Windows handling for vLLM) - pin third-party GitHub Actions to a full commit SHA with a trailing `# vX.Y.Z` comment, never a moving tag (`@v2`, `@main`); a retagged or compromised action otherwise enters CI silently. Bump the SHA and comment together when upgrading diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2da2536192..efd8be82fa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ prek install # fast checks on every commit prek install -t pre-push # heavier checks on push (clippy + tests of the crates the push affects) ``` -`prek` runs the same checks locally that CI enforces: `cargo fmt`, `clippy`, `cargo test`, `ruff` (Python), `shellcheck` (shell), PowerShell syntax, license headers (`hawkeye`), markdown links (`lychee`), and the generated files (`MANIFEST.md`, `THIRD_PARTY_NOTICES.txt`, and the published Doctor catalog, `doctor-catalog.json`). +`prek` runs the same checks locally that CI enforces: `cargo fmt`, `clippy`, `cargo test`, `ruff` (Python), `shellcheck` (shell), PowerShell syntax, license headers (`hawkeye`), markdown links (`lychee`), the module map (`cargo xtask check-architecture-doc`), and the generated files (`MANIFEST.md`, `THIRD_PARTY_NOTICES.txt`, and the published Doctor catalog, `doctor-catalog.json`). The pre-push hooks (`cargo clippy`, the e2e harness clippy, and `cargo test`) check only the crates your push affects: the crates whose files changed plus every crate that depends on them, selected the way CI's `cargo xtask affected` does. Each step prints the scope it chose and why. @@ -104,10 +104,12 @@ The workspace contains these top-level components: New subcommands and subsystems default to their own file from day one. Don't let them grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. See [`docs/architecture.md`](https://github.com/ROCm/rocm-cli/blob/main/docs/architecture.md) for the two extraction patterns in use, the current module map, and the module-organization convention in full. -The `docs-links` job below checks the file links in that document, including every concrete file citation in the per-crate inventories. It checks that they resolve, not that the prose is accurate. Its section headings and directory names stay bare and unchecked. +The `docs-links` job below checks the file links in that document, including every concrete file citation in the per-crate inventories. It checks that they resolve, not that the prose is accurate. Every production source file must also appear in it (see below). Its section headings and directory names stay bare and unchecked. Crate-layering invariants, such as `rocmd` never depending on `rocm`, are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`). File size is checked by `cargo xtask check-module-size` (`xtask/src/module_size.rs`): no source file may exceed 1,500 production lines, test code excluded, and a change that would cross the limit splits the file instead. Editing a file that is already over the limit does not by itself require a split; those files are split under their own tickets. For now the CI step and the `cargo-module-size` pre-commit hook report offenders without failing. +Every production source file, meaning a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`, must have an entry in `docs/architecture.md`: a relative markdown link to it, which has to render as a link (a path inside an HTML comment, a code block or a code span does not count, and neither does a malformed link such as one missing its `)`, or an image). Use an inline `[text](path)` link with its text on one line; reference-style links, a `?query` suffix and root-relative paths are not recognised. `build.rs` files and the `tests/` and `examples/` trees are out of scope. `cargo xtask check-architecture-doc` (`xtask/src/architecture_doc.rs`) enforces that in the `clippy` CI job and in the `architecture-doc` prek hook, which runs on every commit and every push, over the whole tree. A new file with no entry fails it. This is the opposite direction to the `lychee` check below, which only confirms that links already in the doc still resolve. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. The files that had none when the check was added are listed in `EXEMPT` in `xtask/src/architecture_doc.rs`; that list is meant to shrink, and its size is capped by `EXEMPT_CEILING` in the same file, so growing it is a visible, reviewed exception rather than a way to make the check pass. When you document one of them, remove it from `EXEMPT` in the same change, or the check fails on the stale entry. The check does not read the prose of an entry, so keeping a description true when a file's role changes is still up to you. + Every local (relative-path) Markdown link and `#anchor` fragment in the repository (outside `docs/rocm-docs/`, which `docs-build` covers instead, and hidden directories such as `.github/`, which neither check walks) is checked by the `docs-links` CI job and by the `lychee` prek hook above, both configured by `lychee.toml` (which lists the current exclusions). Both run offline only, so they don't catch broken external `https://` links. Because only links are checked, cite a specific repository file as a link relative to the citing file, `[fix.rs](../crates/rocm-core/src/fix.rs)` from `docs/`, instead of a bare backtick path, which nothing checks. The full rule, including the link forms that fail the check and the places it doesn't apply (`.github/`, links leaving a `skills/` folder, and `docs/rocm-docs/` along with the files it includes, which are this file and README.md), is in [AGENTS.md](https://github.com/ROCm/rocm-cli/blob/main/AGENTS.md) section 5 ("Investigate rocm-cli Before Editing"); it applies to citations you add or edit. diff --git a/docs/architecture.md b/docs/architecture.md index 57c4af8a39..6beec99813 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,6 +10,8 @@ This is the living module map for rocm-cli. It's a contributor-facing reference > Before relying on any entry below, verify current file and function boundaries directly (e.g. `grep`) rather than trusting this doc's wording. Module boundaries shift as the codebase grows, and a stale-but-plausible-looking note is worse than an explicit prompt to check. Every concrete file citation in the Module map's per-crate inventories below — except "the old agent.rs" in the `rocm-dash-tui` entry (historical prose about a deleted file) and `report.json` in the `crates/e2e-report` entry (cucumber's untracked generated output) — is a relative markdown link rather than a bare filename, checked offline by [lychee](https://github.com/lycheeverse/lychee) in CI (the `docs-links` job, config in `lychee.toml`) — a moved or deleted file fails CI immediately, naming the broken link. That check doesn't check that the surrounding prose (e.g. which extraction pattern a module follows) is still accurate; the caution above still applies to everything but link resolution. +A second check runs the other way: `cargo xtask check-architecture-doc` ([`architecture_doc.rs`](../xtask/src/architecture_doc.rs)) fails when a production source file — a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` — has no link in this document, so a new module cannot land without an entry. Only a link that renders as one counts: a path inside an HTML comment, a code block or a code span does not, and neither does a malformed link (one missing its `[` or `)`, or with an escaped bracket), an image, or the outer brackets of a link that contains another link. Write the entry as an inline `[text](path)` link with its text on one line; `` delimiters around the destination are accepted, but reference-style links, a `?query` suffix and root-relative paths are not. `build.rs` files and the `tests/` and `examples/` trees are out of scope. It runs in the `clippy` CI job and in the `architecture-doc` prek hook, at commit and at push. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. The files that were uncited when the check was added, mostly in the dashboard crates, are listed in its `EXEMPT` constant; documenting one means removing it from that list in the same change. + ## Module organization convention New subcommands and subsystems default to their own file from day one — they should not grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. Two extraction patterns already exist in the codebase; use whichever fits: @@ -290,6 +292,8 @@ The other three dashboard crates are described at crate level here, not module b A contract surface: verify all impacted engines after any protocol change here (see [`AGENTS.md`](../AGENTS.md)). +- [`lib.rs`](../crates/rocm-engine-protocol/src/lib.rs) — the whole crate: the protocol and recipe-contract version constants, the request/response envelopes and the per-method request and response types for the `EngineMethod` calls (`detect`, `install`, `capabilities`, `resolve_model`, `launch`, `healthcheck`, `endpoint`, `stop`, `logs`), the `DevicePolicy` and `GpuSelection` types with the GPU-visibility helpers engines apply to a child process, engine-plugin binary naming and discovery, and the endpoint API key file carrier (`ENDPOINT_API_KEY_FILE_ENV`, `resolve_endpoint_api_key`, `is_public_bind_host`) that `rocm` and `rocmd` share. + ### `crates/rocm-deps` Pinned versions of the third-party runtimes rocm-cli manages (from workspace-root [`runtime-deps.toml`](../runtime-deps.toml), turned into constants by [`build.rs`](../crates/rocm-deps/build.rs)). Small and already single-purpose; not part of the modularization effort's target list. diff --git a/docs/testing.md b/docs/testing.md index 431ff82724..984b1e1568 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -51,6 +51,22 @@ The counting rules are unit-tested in `xtask/src/module_size.rs`: cargo test -p xtask module_size ``` +### Module map check + +Every production source file must have an entry in [architecture.md](architecture.md). The +check runs in the `clippy` CI job and in the `architecture-doc` prek hook, and you can run it +on its own: + +```bash +cargo xtask check-architecture-doc +``` + +It fails on a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` that the doc +does not link, and on a stale entry in the `EXEMPT` list in +[architecture_doc.rs](../xtask/src/architecture_doc.rs). Test-only helpers need no entry. The +`lychee` check is complementary, not redundant: it confirms that links already in the doc +resolve, and says nothing about a file the doc never mentions. + ### Tests that touch the environment `std::env::set_var` / `remove_var` change state shared by every thread in the diff --git a/xtask/src/architecture_doc.rs b/xtask/src/architecture_doc.rs new file mode 100644 index 0000000000..27b96e15f8 --- /dev/null +++ b/xtask/src/architecture_doc.rs @@ -0,0 +1,988 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Require every production source file to appear in `docs/architecture.md`. +//! +//! `docs/architecture.md` is the contributor-facing module map and is meant to +//! be "updated in the same PR as the code it documents". Nothing enforced that +//! until now. The `lychee` link check (the `docs-links` CI job and the `lychee` +//! prek hook) runs the other way round: it confirms that every link already in +//! the doc still resolves, so a moved or deleted file is caught, but a file the +//! doc never mentions has no link to check and passes silently. +//! +//! This check starts from the source tree instead. Every tracked `.rs` file +//! under `apps/*/src`, `crates/*/src` and `engines/*/src` must be cited by a +//! relative markdown link in `docs/architecture.md` (a complete inline link, +//! `[text](path)`, with its text on one line, that renders as a link: a path +//! inside an HTML comment, a code block or a code span does not count, and +//! neither does a link missing its `[` or `)`, one with an escaped bracket, an +//! image, or the outer brackets of a link that contains another link), +//! or be listed in [`EXEMPT`], or be a test-only helper (see +//! [`is_test_helper`]). Out of scope: `build.rs` files at a crate's root and the +//! `tests/` and `examples/` trees, which sit outside `/src`. +//! +//! It reads the git index and the working tree, never a git diff, so the answer +//! does not depend on a base ref. That is the same answer in CI and in the +//! pre-commit hook, where prek stashes unstaged changes first. The pre-push hook +//! runs it too, but it reads the working tree, not the commits being pushed: +//! with a clean tree that is the same thing, and with edits that are staged or +//! unstaged it can pass where CI fails. CI is the gate; the hook is feedback. +//! +//! The doc must use inline `[text](path)` links. Reference-style links, a +//! `?query` suffix and a root-relative path render as working links but are not +//! recognised, so the failure message asks for the inline form. +//! +//! [`EXEMPT`] is the files that existed, uncited, when the check was added. It +//! follows the pattern of `ALLOWLIST` in [`crate::crate_edges`] and is checked +//! in both directions: a source file in neither the doc nor the list fails, +//! and so does a list entry that is now cited or no longer exists. The list is +//! meant to shrink as files are documented. Nothing can stop a determined edit +//! from adding to it, so [`EXEMPT_CEILING`] pins its size in a test: growing +//! the list means raising that constant too, a second, visible line in the diff +//! that a reviewer has to accept. +//! +//! What this does not check: whether an entry's prose still describes the file, +//! and functions that move between two files that are both already cited. + +use std::collections::BTreeSet; +use std::path::Path; +use std::process::Command; + +use anyhow::{Context, Result, bail}; + +use crate::paths; + +/// The module map, relative to the workspace root. +const DOC: &str = "docs/architecture.md"; + +/// The directory `DOC` lives in. Its links are relative to this. +const DOC_DIR: &str = "docs"; + +/// Source roots whose `/src` trees are in scope. +const SOURCE_ROOTS: [&str; 3] = ["apps", "crates", "engines"]; + +/// Production files that have no entry in `docs/architecture.md` yet: everything +/// that was uncited when this check was added, mostly the dashboard crates. +/// Do not add to this list to make the check pass: document the file in the doc +/// instead. Growing it is a reviewed exception and also needs +/// [`EXEMPT_CEILING`] raised. Remove an entry when the file gets one. Keep it +/// sorted. +const EXEMPT: &[&str] = &[ + "apps/rocmd/src/main.rs", + "crates/rocm-dash-collectors/src/amd_smi.rs", + "crates/rocm-dash-collectors/src/bench_load.rs", + "crates/rocm-dash-collectors/src/bench_tail.rs", + "crates/rocm-dash-collectors/src/cgroup.rs", + "crates/rocm-dash-collectors/src/docker.rs", + "crates/rocm-dash-collectors/src/engine_registry.rs", + "crates/rocm-dash-collectors/src/host.rs", + "crates/rocm-dash-collectors/src/lemonade.rs", + "crates/rocm-dash-collectors/src/parallel.rs", + "crates/rocm-dash-collectors/src/proc_scan.rs", + "crates/rocm-dash-collectors/src/sysfs.rs", + "crates/rocm-dash-collectors/src/vllm_log.rs", + "crates/rocm-dash-collectors/src/vllm_prom.rs", + "crates/rocm-dash-core/src/bench_rollup.rs", + "crates/rocm-dash-core/src/bench_schema.rs", + "crates/rocm-dash-core/src/config.rs", + "crates/rocm-dash-core/src/efficiency.rs", + "crates/rocm-dash-core/src/metrics.rs", + "crates/rocm-dash-core/src/observation.rs", + "crates/rocm-dash-core/src/partition.rs", + "crates/rocm-dash-core/src/persist.rs", + "crates/rocm-dash-core/src/protocol.rs", + "crates/rocm-dash-core/src/state.rs", + "crates/rocm-dash-core/src/traits.rs", + "crates/rocm-dash-core/src/vram.rs", + "crates/rocm-dash-daemon/src/bench_ring.rs", + "crates/rocm-dash-daemon/src/demo.rs", + "crates/rocm-dash-daemon/src/persist.rs", + "crates/rocm-dash-daemon/src/snapshot_ring.rs", + "crates/rocm-dash-daemon/src/transport.rs", + "crates/rocm-dash-tui/src/client.rs", + "crates/rocm-dash-tui/src/jobs.rs", + "crates/rocm-dash-tui/src/lib.rs", + "crates/rocm-dash-tui/src/llm.rs", + "crates/rocm-dash-tui/src/reconnect.rs", + "crates/rocm-dash-tui/src/registry_probe.rs", + "crates/rocm-dash-tui/src/replay.rs", + "crates/rocm-dash-tui/src/tool_exec.rs", + "crates/rocm-dash-tui/src/transport.rs", + "crates/rocm-dash-tui/src/ui/automations_manager.rs", + "crates/rocm-dash-tui/src/ui/bench.rs", + "crates/rocm-dash-tui/src/ui/bench_run.rs", + "crates/rocm-dash-tui/src/ui/command_screen.rs", + "crates/rocm-dash-tui/src/ui/config_manager.rs", + "crates/rocm-dash-tui/src/ui/core_bars.rs", + "crates/rocm-dash-tui/src/ui/dock.rs", + "crates/rocm-dash-tui/src/ui/engine_manager.rs", + "crates/rocm-dash-tui/src/ui/examine_manager.rs", + "crates/rocm-dash-tui/src/ui/folder_browser.rs", + "crates/rocm-dash-tui/src/ui/format.rs", + "crates/rocm-dash-tui/src/ui/gradient.rs", + "crates/rocm-dash-tui/src/ui/install_manager.rs", + "crates/rocm-dash-tui/src/ui/instance_list.rs", + "crates/rocm-dash-tui/src/ui/launcher.rs", + "crates/rocm-dash-tui/src/ui/logs_view.rs", + "crates/rocm-dash-tui/src/ui/model_picker.rs", + "crates/rocm-dash-tui/src/ui/onboarding.rs", + "crates/rocm-dash-tui/src/ui/runtime_manager.rs", + "crates/rocm-dash-tui/src/ui/serve_wizard.rs", + "crates/rocm-dash-tui/src/ui/services_manager.rs", + "crates/rocm-dash-tui/src/ui/sparkline.rs", + "crates/rocm-dash-tui/src/ui/tabs/bench.rs", + "crates/rocm-dash-tui/src/ui/tabs/chat.rs", + "crates/rocm-dash-tui/src/ui/tabs/hardware.rs", + "crates/rocm-dash-tui/src/ui/tabs/home.rs", + "crates/rocm-dash-tui/src/ui/tabs/mod.rs", + "crates/rocm-dash-tui/src/ui/tabs/observe.rs", + "crates/rocm-dash-tui/src/ui/tabs/pane.rs", + "crates/rocm-dash-tui/src/ui/tabs/rocm.rs", + "crates/rocm-dash-tui/src/ui/tabs/serving.rs", + "crates/rocm-dash-tui/src/ui/update_manager.rs", + "crates/rocm-deps/src/lib.rs", + "engines/aim/src/main.rs", + "engines/lemonade/src/main.rs", + "engines/vllm/src/main.rs", +]; + +/// The most entries [`EXEMPT`] may hold. Lower it when entries are removed; do +/// not raise it to make the check pass, since a file with no entry in the module +/// map should get one. Raising it is the visible, reviewed way to grow the list. +const EXEMPT_CEILING: usize = 76; + +/// Whether `path` is a tracked-file path this check covers: a `.rs` file under +/// `//src/` for one of the [`SOURCE_ROOTS`]. +#[allow(clippy::case_sensitive_file_extension_comparisons)] // Rust sources are always lowercase `.rs` +fn in_scope(path: &str) -> bool { + let parts: Vec<&str> = path.split('/').collect(); + parts.len() >= 4 + && SOURCE_ROOTS.contains(&parts[0]) + && parts[2] == "src" + && path.ends_with(".rs") +} + +/// Whether `path` is a test-only helper that does not need an architecture +/// entry: shared test support, or a property-test module. Matched on the file +/// stem so a new helper of the same kind never needs an [`EXEMPT`] entry. +fn is_test_helper(path: &str) -> bool { + let name = path.rsplit('/').next().unwrap_or(path); + let stem = name.strip_suffix(".rs").unwrap_or(name); + matches!(stem, "test_support" | "test_env" | "proptests") + || stem.ends_with("_proptests") + || stem.ends_with("_props") + || stem.ends_with("_properties") +} + +/// Resolve one markdown link target found in [`DOC`] to a repo-relative `.rs` +/// path, or `None` if it is not a link to a source file inside the repo. +#[allow(clippy::case_sensitive_file_extension_comparisons)] // Rust sources are always lowercase `.rs` +fn resolve_link(target: &str) -> Option { + let target = target.split('#').next()?; + if target.contains("://") || !target.ends_with(".rs") { + return None; + } + let mut parts: Vec<&str> = vec![DOC_DIR]; + for segment in target.split('/') { + match segment { + "" | "." => {} + ".." => { + // Climbing out of the repo root names nothing in the tree. + parts.pop()?; + } + other => parts.push(other), + } + } + Some(parts.join("/")) +} + +/// The first line of `text` without its terminator, and the byte length of that +/// line including the terminator. +fn next_line(text: &str) -> (&str, usize) { + match text.find('\n') { + Some(end) => (&text[..end], end + 1), + None => (text, text.len()), + } +} + +/// A code-fence marker at the start of `line`: its character and run length. +fn fence_marker(line: &str) -> Option<(u8, usize)> { + let line = line.trim_start_matches([' ', '\t']); + let first = *line.as_bytes().first()?; + if first != b'`' && first != b'~' { + return None; + } + let run = line.bytes().take_while(|&byte| byte == first).count(); + // A backtick fence's info string cannot contain a backtick: that line is an + // inline code span, not a fence. + if run < 3 || (first == b'`' && line[run..].contains('`')) { + return None; + } + Some((first, run)) +} + +/// Whether `line` closes a fence opened with `open`. +fn closes_fence(line: &str, open: (u8, usize)) -> bool { + fence_marker(line).is_some_and(|(character, run)| { + character == open.0 && run >= open.1 && line.trim().bytes().all(|byte| byte == open.0) + }) +} + +/// The byte length of the code span that opens at the start of `text` (which +/// starts with a backtick run), or `None` if no run of the same length closes +/// it. Like a CommonMark code span it cannot cross a blank line, so a stray +/// backtick in one paragraph does not pair with one in a later paragraph. +fn code_span_len(text: &str) -> Option { + let window = &text[..text.find("\n\n").unwrap_or(text.len())]; + let open = window.bytes().take_while(|&byte| byte == b'`').count(); + let mut at = open; + while let Some(offset) = window[at..].find('`') { + let start = at + offset; + let run = window[start..] + .bytes() + .take_while(|&byte| byte == b'`') + .count(); + if run == open { + return Some(start + run); + } + at = start + run; + } + None +} + +/// `doc` with everything that does not render as a link removed: HTML comments, +/// fenced code blocks and inline code spans. A link inside any of them is not +/// part of the rendered module map, so it must not count as a citation. Each +/// removed span leaves a space or newline behind so link text and line +/// structure stay in place. +/// +/// It scans once, left to right, so the usual precedence holds: a backtick +/// inside a comment is part of the comment, and `").map_or(rest.len(), |end| end + 3); + out.push(' '); + continue; + } + if rest.starts_with('`') { + if let Some(length) = code_span_len(rest) { + at += length; + out.push(' '); + } else { + let run = rest.bytes().take_while(|&byte| byte == b'`').count(); + out.push_str(&rest[..run]); + at += run; + } + continue; + } + let Some(character) = rest.chars().next() else { + break; + }; + out.push(character); + at += character.len_utf8(); + } + out +} + +/// Whether the byte at `at` is a backslash that escapes the ASCII punctuation +/// character after it. +fn escapes_next(bytes: &[u8], at: usize) -> bool { + bytes[at] == b'\\' && bytes.get(at + 1).is_some_and(u8::is_ascii_punctuation) +} + +/// `text` with the backslash removed from every escaped punctuation character. +fn unescape(text: &str) -> String { + let bytes = text.as_bytes(); + let mut out = String::with_capacity(text.len()); + let mut at = 0; + while at < bytes.len() { + if escapes_next(bytes, at) { + at += 1; + } + // Every byte skipped above or copied here is ASCII, so `at` stays on a + // character boundary. + let Some(character) = text[at..].chars().next() else { + break; + }; + out.push(character); + at += character.len_utf8(); + } + out +} + +/// Advance `at` past spaces, tabs and newlines. +fn skip_space(bytes: &[u8], mut at: usize) -> usize { + while matches!(bytes.get(at), Some(b' ' | b'\t' | b'\n')) { + at += 1; + } + at +} + +/// Parse the rest of an inline link, `text` being everything after its `(`: +/// `destination`, an optional title, and the closing `)`. Returns the +/// destination (its `<...>` delimiters and escapes removed) and the byte length +/// consumed through the `)`, or `None` if it is not a complete inline link. +fn inline_destination(text: &str) -> Option<(String, usize)> { + let bytes = text.as_bytes(); + let mut at = skip_space(bytes, 0); + let destination; + if bytes.get(at) == Some(&b'<') { + let start = at + 1; + at = start; + loop { + match *bytes.get(at)? { + b'>' => break, + b'<' | b'\n' => return None, + _ if escapes_next(bytes, at) => at += 2, + _ => at += 1, + } + } + destination = unescape(&text[start..at]); + at += 1; + } else { + let start = at; + let mut depth = 0usize; + loop { + match bytes.get(at) { + None => return None, + Some(b' ' | b'\t' | b'\n') => break, + Some(b'(') => { + depth += 1; + at += 1; + } + Some(b')') if depth == 0 => break, + Some(b')') => { + depth -= 1; + at += 1; + } + Some(_) if escapes_next(bytes, at) => at += 2, + Some(_) => at += 1, + } + } + destination = unescape(&text[start..at]); + } + at = skip_space(bytes, at); + if let Some(&opener @ (b'"' | b'\'' | b'(')) = bytes.get(at) { + let closer = if opener == b'(' { b')' } else { opener }; + at += 1; + loop { + match *bytes.get(at)? { + byte if byte == closer => break, + _ if escapes_next(bytes, at) => at += 2, + _ => at += 1, + } + } + at = skip_space(bytes, at + 1); + } + (bytes.get(at) == Some(&b')')).then_some((destination, at + 1)) +} + +/// An unmatched `[` while scanning for links. +struct Opener { + /// Preceded by an unescaped `!`: an image, whose destination is not a link. + image: bool, + /// Cleared once a link has formed inside this one: links do not nest, so the + /// outer brackets then render as plain text. + active: bool, +} + +/// The destination of every complete inline link, `[text](destination)`, in +/// `text`. A bracket or parenthesis escaped with a backslash does not count, +/// the `[` must be on the same line as its `]`, and a link missing its `[`, its +/// `)` or its destination is not a link: the rendered page shows it as plain +/// text, so it must not count as a citation either. The same goes for an image, +/// `![alt](destination)`, and for the outer brackets of a link that contains +/// another link; an image inside a link's text is fine. +fn link_destinations(text: &str) -> Vec { + let bytes = text.as_bytes(); + let mut found = Vec::new(); + let mut openers: Vec = Vec::new(); + let mut last_bang = None; + let mut at = 0; + while at < bytes.len() { + if escapes_next(bytes, at) { + at += 2; + continue; + } + match bytes[at] { + b'\n' => openers.clear(), + b'!' => last_bang = Some(at), + b'[' => openers.push(Opener { + image: at > 0 && last_bang == Some(at - 1), + active: true, + }), + b']' => { + if let Some(opener) = openers.pop() + && opener.active + && bytes.get(at + 1) == Some(&b'(') + && let Some((destination, length)) = inline_destination(&text[at + 2..]) + { + if !opener.image { + found.push(destination); + for outer in openers.iter_mut().filter(|outer| !outer.image) { + outer.active = false; + } + } + at += 2 + length; + continue; + } + } + _ => {} + } + at += 1; + } + found +} + +/// Every `.rs` file [`DOC`] links to, as repo-relative paths. Only a complete +/// inline link counts, and not one inside an HTML comment, a code block or a +/// code span. +fn cited_paths(doc: &str) -> BTreeSet { + link_destinations(&link_source_text(doc)) + .iter() + .filter_map(|destination| resolve_link(destination)) + .collect() +} + +/// Check the tracked files against the doc's citations and `exempt`, returning +/// every violation found. +fn check(tracked: &[String], cited: &BTreeSet, exempt: &[&str]) -> Vec { + let in_tree: BTreeSet<&str> = tracked + .iter() + .map(String::as_str) + .filter(|path| in_scope(path)) + .collect(); + + let mut violations = Vec::new(); + + for path in &in_tree { + if !cited.contains(*path) && !exempt.contains(path) && !is_test_helper(path) { + violations.push(format!( + "`{path}` is not cited in {DOC}: add an entry for it to the module map, as an \ + inline `[text](path)` markdown link relative to docs/ (reference-style links, \ + `?query` suffixes and root-relative paths are not recognised)" + )); + } + } + + for entry in exempt { + if !in_tree.contains(entry) { + violations.push(format!( + "EXEMPT entry `{entry}` is stale: no tracked source file has that path any more; \ + remove it from EXEMPT in xtask/src/architecture_doc.rs" + )); + } else if cited.contains(*entry) { + violations.push(format!( + "EXEMPT entry `{entry}` is stale: {DOC} now cites it; remove it from EXEMPT in \ + xtask/src/architecture_doc.rs so a later removal from the doc is caught" + )); + } else if is_test_helper(entry) { + violations.push(format!( + "EXEMPT entry `{entry}` is redundant: test-only helpers are exempt by name; \ + remove it from EXEMPT in xtask/src/architecture_doc.rs" + )); + } + } + + violations +} + +/// Tracked files under the [`SOURCE_ROOTS`], as the index lists them. +fn tracked_files(root: &Path) -> Result> { + let output = Command::new("git") + .arg("-C") + .arg(root) + .args(["ls-files", "-z", "--"]) + .args(SOURCE_ROOTS) + .output() + .context("failed to run `git ls-files`")?; + if !output.status.success() { + bail!( + "`git ls-files` failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + ); + } + Ok(String::from_utf8_lossy(&output.stdout) + .split('\0') + .filter(|path| !path.is_empty()) + .map(str::to_owned) + .collect()) +} + +/// The violation, if any, of `entries` exempt files exceeding `ceiling`. +fn ceiling_violation(entries: usize, ceiling: usize) -> Option { + (entries > ceiling).then(|| { + format!( + "EXEMPT holds {entries} entries but EXEMPT_CEILING is {ceiling}: a file with no entry \ + in the module map should get one, not an EXEMPT line. Growing the list means raising \ + EXEMPT_CEILING in xtask/src/architecture_doc.rs too, which is a reviewed exception" + ) + }) +} + +/// Run the check against the real workspace. +fn violations_in(root: &Path) -> Result> { + let doc = + std::fs::read_to_string(root.join(DOC)).with_context(|| format!("failed to read {DOC}"))?; + let mut violations = check(&tracked_files(root)?, &cited_paths(&doc), EXEMPT); + violations.extend(ceiling_violation(EXEMPT.len(), EXEMPT_CEILING)); + Ok(violations) +} + +/// Fail if a production source file has no entry in `docs/architecture.md`, or +/// [`EXEMPT`] has gone stale. +pub fn run() -> Result<()> { + let root = paths::workspace_root()?; + let violations = violations_in(&root)?; + if !violations.is_empty() { + bail!( + "{DOC} is out of step with the source tree ({} problem(s)):\n{}", + violations.len(), + violations.join("\n") + ); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn files(paths: &[&str]) -> Vec { + paths.iter().map(|path| (*path).to_owned()).collect() + } + + fn cites(paths: &[&str]) -> BTreeSet { + paths.iter().map(|path| (*path).to_owned()).collect() + } + + #[test] + fn cited_file_passes() { + let tracked = files(&["apps/rocm/src/cli.rs"]); + assert!(check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]).is_empty()); + } + + #[test] + fn new_uncited_file_fails_and_is_named() { + let tracked = files(&["apps/rocm/src/cli.rs", "apps/rocm/src/brand_new.rs"]); + let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("apps/rocm/src/brand_new.rs")); + assert!(violations[0].contains("is not cited")); + } + + #[test] + fn a_citation_does_not_cover_the_same_name_in_another_crate() { + let tracked = files(&["apps/rocm/src/cli.rs", "apps/rocmd/src/cli.rs"]); + let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("apps/rocmd/src/cli.rs")); + } + + #[test] + fn exempt_file_passes() { + let tracked = files(&["crates/rocm-dash-core/src/state.rs"]); + let exempt = ["crates/rocm-dash-core/src/state.rs"]; + assert!(check(&tracked, &cites(&[]), &exempt).is_empty()); + } + + #[test] + fn exempt_entry_for_a_missing_file_is_stale() { + let exempt = ["crates/rocm-dash-core/src/gone.rs"]; + let violations = check(&files(&[]), &cites(&[]), &exempt); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("crates/rocm-dash-core/src/gone.rs")); + assert!(violations[0].contains("no tracked source file")); + } + + #[test] + fn exempt_entry_that_is_now_cited_is_stale() { + let tracked = files(&["crates/rocm-dash-core/src/state.rs"]); + let exempt = ["crates/rocm-dash-core/src/state.rs"]; + let cited = cites(&["crates/rocm-dash-core/src/state.rs"]); + let violations = check(&tracked, &cited, &exempt); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("now cites it")); + } + + #[test] + fn exempt_entry_for_a_test_helper_is_redundant() { + let tracked = files(&["crates/rocm-core/src/test_support.rs"]); + let exempt = ["crates/rocm-core/src/test_support.rs"]; + let violations = check(&tracked, &cites(&[]), &exempt); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("redundant")); + } + + #[test] + fn files_outside_the_source_trees_are_ignored() { + let tracked = files(&[ + "apps/rocm/tests/broken_pipe.rs", + "apps/rocm/build.rs", + "crates/rocm-dash-tui/examples/gen_cast.rs", + "xtask/src/anything.rs", + "apps/rocm/src/data.json", + "tests/e2e-cucumber/src/lib.rs", + ]); + assert!(check(&tracked, &cites(&[]), &[]).is_empty()); + } + + #[test] + fn nested_source_files_are_in_scope() { + let tracked = files(&["crates/rocm-dash-tui/src/ui/tabs/home.rs"]); + let violations = check(&tracked, &cites(&[]), &[]); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("ui/tabs/home.rs")); + } + + #[test] + fn test_helpers_need_no_entry() { + let tracked = files(&[ + "apps/rocm/src/test_support.rs", + "apps/rocm/src/therock/test_support.rs", + "crates/rocm-core/src/test_env.rs", + "crates/rocm-core/src/archive_props.rs", + "crates/rocm-core/src/process_tree_properties.rs", + "crates/rocm-core/src/examine_driver_proptests.rs", + "apps/rocm/src/providers/local_http/proptests.rs", + ]); + assert!(check(&tracked, &cites(&[]), &[]).is_empty()); + } + + #[test] + fn helper_matching_is_by_whole_file_stem() { + // Names that merely contain a helper word are production files. + for path in [ + "apps/rocm/src/test_supporter.rs", + "apps/rocm/src/properties.rs", + "apps/rocm/src/gpu_props_view.rs", + "apps/rocm/src/proptests_runner.rs", + ] { + assert!( + !is_test_helper(path), + "{path} must not be treated as a helper" + ); + } + } + + #[test] + fn links_resolve_relative_to_the_doc() { + let doc = "[cli](../apps/rocm/src/cli.rs) and [x](./../crates/a/src/b.rs#L1)"; + assert_eq!( + cited_paths(doc), + cites(&["apps/rocm/src/cli.rs", "crates/a/src/b.rs"]) + ); + } + + #[test] + fn only_in_repo_rust_links_count_as_citations() { + let doc = "[md](../README.md) [url](https://example.com/a/src/x.rs) \ + [out](../../outside/src/x.rs) [bare](apps/rocm/src/cli.rs)"; + assert_eq!(cited_paths(doc), cites(&["docs/apps/rocm/src/cli.rs"])); + } + + #[test] + fn a_link_with_a_title_is_still_a_citation() { + let doc = "[cli](../apps/rocm/src/cli.rs \"the command tree\")"; + assert_eq!(cited_paths(doc), cites(&["apps/rocm/src/cli.rs"])); + } + + const CLI: &str = "[cli](../apps/rocm/src/cli.rs)"; + + fn cited_cli(doc: &str) -> bool { + cited_paths(doc).contains("apps/rocm/src/cli.rs") + } + + #[test] + fn commenting_out_the_only_entry_leaves_the_file_uncited() { + let doc = format!(""); + let tracked = files(&["apps/rocm/src/cli.rs"]); + let violations = check(&tracked, &cited_paths(&doc), &[]); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("apps/rocm/src/cli.rs")); + } + + #[test] + fn a_link_in_an_html_comment_is_not_a_citation() { + assert!(!cited_cli(&format!(""))); + assert!(!cited_cli(&format!( + "before\n\nafter" + ))); + // An unterminated comment hides everything after it, as a renderer does. + assert!(!cited_cli(&format!("\n{CLI}"))); + } + + #[test] + fn a_link_in_a_fenced_code_block_is_not_a_citation() { + assert!(!cited_cli(&format!("```\n{CLI}\n```"))); + assert!(!cited_cli(&format!("```md\n{CLI}\n```"))); + assert!(!cited_cli(&format!("~~~\n{CLI}\n~~~"))); + assert!(!cited_cli(&format!(" ```\n {CLI}\n ```"))); + // A longer closing fence closes it; a shorter or different one does not. + assert!(!cited_cli(&format!("````\n```\n{CLI}\n```\n````"))); + assert!(!cited_cli(&format!("~~~\n```\n{CLI}\n```\n~~~"))); + // An unclosed fence runs to the end of the document. + assert!(!cited_cli(&format!("```\n{CLI}"))); + assert!(cited_cli(&format!("```\ncode\n```\n{CLI}"))); + } + + #[test] + fn a_link_in_an_inline_code_span_is_not_a_citation() { + assert!(!cited_cli(&format!("see `{CLI}` for more"))); + assert!(!cited_cli(&format!("see ``a ` b {CLI}`` for more"))); + assert!(cited_cli(&format!("see `code` and {CLI}"))); + } + + #[test] + fn a_code_span_inside_link_text_still_cites() { + assert!(cited_cli("[`cli.rs`](../apps/rocm/src/cli.rs)")); + assert!(cited_cli( + "- [`cli.rs`](../apps/rocm/src/cli.rs) - the command tree" + )); + } + + #[test] + fn a_stray_backtick_does_not_hide_a_later_paragraph() { + assert!(cited_cli(&format!("a ` stray\n\n{CLI}"))); + assert!(cited_cli(&format!("a ` stray {CLI}"))); + } + + #[test] + fn comment_and_code_markers_take_the_usual_precedence() { + // ``"))); + // A fence marker inside a comment does not open a fence. + assert!(cited_cli(&format!("\n{CLI}"))); + // A backtick inside a comment does not open a code span. + assert!(cited_cli(&format!("\n{CLI}\n"))); + } + + #[test] + fn crlf_documents_are_read_the_same() { + assert!(!cited_cli(&format!("```\r\n{CLI}\r\n```\r\n"))); + assert!(cited_cli(&format!("a ` stray\r\n\r\n{CLI}"))); + } + + #[test] + fn an_angle_bracketed_destination_is_a_citation() { + assert!(cited_cli("[cli](<../apps/rocm/src/cli.rs>)")); + assert!(cited_cli("[cli](<../apps/rocm/src/cli.rs#L10>)")); + assert!(cited_cli("[cli]( <../apps/rocm/src/cli.rs> )")); + assert!(cited_cli( + "[cli](<../apps/rocm/src/cli.rs> \"the command tree\")" + )); + // The delimiters are not part of the path: a destination with a space is + // read whole, and an unterminated one is not a link. + assert_eq!( + cited_paths("[x](<../crates/a b/src/c.rs>)"), + cites(&["crates/a b/src/c.rs"]) + ); + assert!(!cited_cli("[cli](<../apps/rocm/src/cli.rs)")); + assert!(!cited_cli("[cli](<../apps/rocm/src/cli.rs\n>)")); + } + + #[test] + fn a_link_missing_its_closing_parenthesis_is_not_a_citation() { + assert!(!cited_cli("[cli](../apps/rocm/src/cli.rs")); + assert!(!cited_cli("[cli](../apps/rocm/src/cli.rs and more text)")); + assert!(!cited_cli( + "[cli](../apps/rocm/src/cli.rs \"unterminated title)" + )); + assert!(!cited_cli( + "[cli](../apps/rocm/src/cli.rs\n\nnext paragraph)" + )); + } + + #[test] + fn a_link_missing_its_opening_bracket_is_not_a_citation() { + assert!(!cited_cli("cli](../apps/rocm/src/cli.rs)")); + // A stray `[` on an earlier line is not this link's text. + assert!(!cited_cli("a [ stray\ncli](../apps/rocm/src/cli.rs)")); + } + + #[test] + fn an_escaped_bracket_or_parenthesis_is_not_a_citation() { + assert!(!cited_cli("\\[cli](../apps/rocm/src/cli.rs)")); + assert!(!cited_cli("[cli\\](../apps/rocm/src/cli.rs)")); + assert!(!cited_cli("[cli]\\(../apps/rocm/src/cli.rs)")); + // An escaped backslash does not escape the bracket after it. + assert!(cited_cli("[cli\\\\](../apps/rocm/src/cli.rs)")); + } + + #[test] + fn breaking_the_only_entry_leaves_the_file_uncited() { + let tracked = files(&["apps/rocm/src/cli.rs"]); + for broken in [ + "[cli](../apps/rocm/src/cli.rs", + "cli](../apps/rocm/src/cli.rs)", + "\\[cli](../apps/rocm/src/cli.rs)", + ] { + let violations = check(&tracked, &cited_paths(broken), &[]); + assert_eq!(violations.len(), 1, "{broken:?} must not cite the file"); + } + } + + #[test] + fn nested_brackets_and_parentheses_do_not_break_a_link() { + assert!(cited_cli("[a [b] c](../apps/rocm/src/cli.rs)")); + assert_eq!( + cited_paths("[x](../crates/a/src/b(1).rs)"), + cites(&["crates/a/src/b(1).rs"]) + ); + } + + #[test] + fn an_image_is_not_a_citation() { + assert!(!cited_cli("![cli](../apps/rocm/src/cli.rs)")); + assert!(!cited_cli("see ![cli](../apps/rocm/src/cli.rs \"title\")")); + let tracked = files(&["apps/rocm/src/cli.rs"]); + let violations = check( + &tracked, + &cited_paths("![cli](../apps/rocm/src/cli.rs)"), + &[], + ); + assert_eq!(violations.len(), 1); + } + + #[test] + fn an_escaped_exclamation_mark_does_not_make_an_image() { + assert!(cited_cli("\\![cli](../apps/rocm/src/cli.rs)")); + assert!(cited_cli("wow! [cli](../apps/rocm/src/cli.rs)")); + } + + #[test] + fn an_image_inside_a_links_text_leaves_the_link_valid() { + let doc = "[![icon](../apps/rocm/src/icon.rs)](../apps/rocm/src/cli.rs)"; + assert_eq!(cited_paths(doc), cites(&["apps/rocm/src/cli.rs"])); + } + + #[test] + fn a_link_containing_a_link_cites_only_the_inner_one() { + // Links cannot nest: the page renders only the inner link, so the outer + // destination is plain text. + let doc = "[cli [d](../apps/rocm/src/d.rs)](../apps/rocm/src/cli.rs)"; + assert_eq!(cited_paths(doc), cites(&["apps/rocm/src/d.rs"])); + assert!(!cited_cli( + "[cli [details](../README.md)](../apps/rocm/src/cli.rs)" + )); + // Plain brackets in the text do not form a link, so the outer one stays. + assert!(cited_cli("[cli [note]](../apps/rocm/src/cli.rs)")); + assert!(cited_cli("[cli [note] x](../apps/rocm/src/cli.rs)")); + // Later links on the line are unaffected. + let doc = "[a [b](../apps/rocm/src/b.rs)](../x.rs) and [c](../apps/rocm/src/c.rs)"; + assert_eq!( + cited_paths(doc), + cites(&["apps/rocm/src/b.rs", "apps/rocm/src/c.rs"]) + ); + } + + #[test] + fn an_escaped_backtick_does_not_open_a_code_span() { + assert!(cited_cli( + "Use \\` for a backtick; [`cli.rs`](../apps/rocm/src/cli.rs)" + )); + // An escaped backslash does not escape the backtick after it. + assert!(!cited_cli("\\\\`[cli](../apps/rocm/src/cli.rs)`")); + // An escaped `<` does not open a comment. + assert!(cited_cli("\\")); + } + + #[test] + fn the_exempt_list_stays_within_its_ceiling() { + assert_eq!(ceiling_violation(EXEMPT.len(), EXEMPT_CEILING), None); + assert_eq!(ceiling_violation(77, 77), None); + assert_eq!(ceiling_violation(70, 77), None); + let violation = ceiling_violation(78, 77).expect("one over the ceiling fails"); + assert!(violation.contains("78 entries")); + assert!(violation.contains("EXEMPT_CEILING is 77")); + } + + #[test] + fn the_link_forms_the_failure_message_rules_out_are_not_citations() { + // The message tells authors these forms are not recognised; this holds it + // to what the parser does. + assert!(!cited_cli("[cli][c]\n\n[c]: ../apps/rocm/src/cli.rs")); + assert!(!cited_cli("[cli](../apps/rocm/src/cli.rs?plain=1)")); + assert!(!cited_cli("[cli](/apps/rocm/src/cli.rs)")); + let message = &check(&files(&["apps/rocm/src/new.rs"]), &cites(&[]), &[])[0]; + assert!(message.contains("inline `[text](path)`")); + assert!(message.contains("reference-style")); + assert!(message.contains("`?query`")); + assert!(message.contains("root-relative")); + } + + #[test] + fn the_failure_message_for_an_uncited_file_only_asks_for_an_entry() { + let violations = check(&files(&["apps/rocm/src/new.rs"]), &cites(&[]), &[]); + assert_eq!(violations.len(), 1); + assert!(violations[0].contains("add an entry for it to the module map")); + assert!(!violations[0].contains("EXEMPT")); + } + + #[test] + fn several_links_on_a_line_are_all_found() { + let doc = "[a](../apps/rocm/src/a.rs), [b](../apps/rocm/src/b.rs) and [c](<../apps/rocm/src/c.rs>)"; + assert_eq!( + cited_paths(doc), + cites(&[ + "apps/rocm/src/a.rs", + "apps/rocm/src/b.rs", + "apps/rocm/src/c.rs" + ]) + ); + } + + #[test] + fn exempt_list_is_sorted_and_unique() { + let mut sorted = EXEMPT.to_vec(); + sorted.sort_unstable(); + sorted.dedup(); + assert_eq!(sorted, EXEMPT, "keep EXEMPT sorted, without duplicates"); + } + + #[test] + fn current_tree_matches_the_architecture_doc() { + let root = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("xtask crate has a parent directory"); + let violations = violations_in(root).expect("git and the doc are readable in a checkout"); + assert!( + violations.is_empty(), + "docs/architecture.md is out of step with the source tree:\n{}", + violations.join("\n") + ); + } +} diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 403afef349..b38fa44d75 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -12,6 +12,7 @@ mod affected; mod amd_smi_launch_contract; +mod architecture_doc; mod ascii_output_contract; mod catalog; mod coverage; @@ -147,6 +148,10 @@ pub(crate) enum Command { /// non-zero before this code runs. The local pre-commit hook runs this until /// the limit is enforced. CheckModuleSizeAdvisory, + /// Fail if a production source file (a `.rs` file under `apps/*/src`, + /// `crates/*/src` or `engines/*/src`) has no entry in `docs/architecture.md`, + /// or the exemption list in `xtask/src/architecture_doc.rs` has gone stale. + CheckArchitectureDoc, /// Regenerate the published Doctor catalog manifest from the compiled catalog. Catalog { /// Verify the published manifest is current without writing; exit @@ -410,6 +415,7 @@ fn run() -> Result<()> { Command::CheckCrateEdges => crate_edges::run()?, Command::CheckModuleSize => module_size::run()?, Command::CheckModuleSizeAdvisory => module_size::run_advisory()?, + Command::CheckArchitectureDoc => architecture_doc::run()?, Command::Catalog { check } => catalog::run(check)?, Command::EvalFixtures { check } => eval_fixtures::run(check)?, Command::Coverage { bless } => coverage::run(bless)?, diff --git a/xtask/src/workflow_contract.rs b/xtask/src/workflow_contract.rs index 2730c5a13d..484846c346 100644 --- a/xtask/src/workflow_contract.rs +++ b/xtask/src/workflow_contract.rs @@ -6801,6 +6801,75 @@ fi } } + /// `cargo xtask check-architecture-doc` is only a gate if all three places + /// that can fail a change actually run it: the pre-commit hook, the pre-push + /// hook (which reads the working tree, so it catches a commit made with + /// `--no-verify` only when the tree is clean) and a `rust`-gated + /// step in the `clippy` CI job. The hook must also see every commit: the + /// answer depends on the whole tree, so a `files:` filter would let a + /// delete-only or doc-only commit skip it. CI's `rust` path filter must + /// include the doc itself, because deleting an entry changes the answer + /// though no `*.rs` file moved. + #[test] + fn architecture_doc_check_runs_in_hooks_and_ci() { + let config = std::fs::read_to_string(repo_root().join(".pre-commit-config.yaml")) + .expect("reading .pre-commit-config.yaml") + .replace("\r\n", "\n"); + let block = hook_block(&config, "architecture-doc"); + assert!( + hook_block_sets(block, "entry: cargo xtask check-architecture-doc"), + "architecture-doc hook must run `cargo xtask check-architecture-doc`:\n{block}" + ); + assert!( + hook_block_sets(block, "stages: [pre-commit, pre-push]"), + "architecture-doc hook must run at both pre-commit and pre-push:\n{block}" + ); + assert!( + hook_block_sets(block, "always_run: true"), + "architecture-doc hook must set `always_run: true`:\n{block}" + ); + assert!( + hook_block_sets(block, "pass_filenames: false"), + "architecture-doc hook must set `pass_filenames: false`:\n{block}" + ); + for filter_key in ["types:", "types_or:", "files:", "exclude:"] { + assert!( + !block.contains(filter_key), + "architecture-doc hook must not have a file filter (`{filter_key}`): the check \ + reads the whole tree, so narrowing it would let a commit skip it:\n{block}" + ); + } + + let ci = read_workflow("ci.yml"); + let command = "run: cargo xtask check-architecture-doc"; + let at = ci + .find(command) + .unwrap_or_else(|| panic!("ci.yml must run `{command}`")); + let gate = ci[..at] + .lines() + .rev() + .take(2) + .any(|line| line.trim() == "if: needs.changes.outputs.rust == 'true'"); + assert!( + gate, + "the `{command}` step must be gated on `needs.changes.outputs.rust == 'true'`, \ + like the other xtask checks in the clippy job" + ); + + let start = ci + .find(" rust:\n") + .expect("ci.yml defines the `rust` path filter"); + let end = start + + ci[start..] + .find(" heavy:\n") + .expect("ci.yml defines the `heavy` path filter after `rust`"); + assert!( + ci[start..end].contains("- 'docs/architecture.md'"), + "the `rust` path filter must include docs/architecture.md, so a doc-only edit \ + re-runs `{command}`" + ); + } + // Extractor guards: prove the helpers above actually parse the shapes they // claim to — multiline hook blocks, comment-stripping, and single-line pin // tokens alike — so the contract tests above can't silently false-pass on a From cb1d6d0766ad3222ba9718222f20b9a01f7a31cd Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Fri, 6 Nov 2026 16:08:09 +0000 Subject: [PATCH 002/102] refactor: one runtime manifest writer, one Python-environment-root rule, one single-ready-runtime rule and one install-root payload rule (#934) * test: pin how each caller finds a Python environment root from its interpreter Three private helpers turn an interpreter path into the environment folder that contains it, and they do not agree. ComfyUI's runtime environment returns a root only when the interpreter's folder is bin or Scripts (any case) and otherwise leaves VIRTUAL_ENV unset. The adopt command and the vLLM launch both take the grandparent whatever the folder is called: the bin/Scripts check in the adopt helper returns the same value on both of its branches, and vLLM never had one. So an interpreter in an unrelated folder is guessed at by two callers and refused by the third. Add tests that pass on the unmodified code and pin each caller's behaviour for bin, Scripts, SCRIPTS, an unrelated folder, a bare file name and a root-level interpreter, including those differences, so the change that makes the callers agree shows exactly what moves. Signed-off-by: Jussi Elo * refactor: find a Python environment root from its interpreter by one rule The adopt command, the ComfyUI runtime environment and the vLLM launch each kept a private copy of "which environment folder holds this interpreter", and the copies disagreed. ComfyUI gave a root only for an interpreter in a bin or Scripts folder; adopt and vLLM took the grandparent whatever the folder was called (adopt's bin/Scripts check returned the same value on both branches). An interpreter in an unrelated folder was refused by one caller and guessed at by the other two. Add runtime_python_env_root_for_executable to rocm-core, beside the helpers it inverts, and use it from all three. An interpreter belongs to an environment only when it sits directly in a bin or Scripts folder, in any case; everything else is None, because a guessed folder is not an environment. The engine crate cannot reach apps/rocm, so rocm-core is the shared place. Valid venv layouts resolve exactly as before. Two things change for an interpreter outside that layout: `rocm runtimes adopt --python` without --root now stops and asks for --root instead of adopting the interpreter's grandparent, and the vLLM launch fails with its existing "not inside a venv" error instead of searching the wrong folder. Each caller's test goes through the shared function, scenario runtime-23 covers the adopt refusal end to end, and the --root help text, README and docs/testing.md now say how the folder is found. Signed-off-by: Jussi Elo * test(runtime): pin what each install-root payload check answers Two private helpers each decide whether an install root holds a payload: has_nontrivial_directory_contents in therock.rs, which refuses to extract a tarball into a non-empty target and decides whether a repair's replacement is already installed, and runtime_install_root_has_payload in runtime_registry.rs, which activation validation applies to a tarball runtime. Both skip dot-entries (the runtime marker is one) and count any other entry, a dangling symlink included. They differ only for a path that does not exist, or that is a dangling symlink: the first answers empty, the second returns an error. Add one table per helper over the same input kinds (missing path, empty directory, dotfile-only, dot-directory-only, regular file, subdirectory, a file as the root, dangling symlink entry, symlinked and dangling roots, unreadable directory) so the next commit, which collapses them into one, can be seen to change nothing but those two rows. Tests only. Signed-off-by: Jussi Elo * test(runtimes): pin how the ComfyUI and engine paths pick the single ready runtime With no runtime chosen, `rocm comfyui install` and the engine paths (`serve`, `engines install`) each pick "the one ready runtime" with their own copy of the rule. Both count a runtime as ready by `validate_runtime_manifest_for_activation`, but they differ in what happens around it: the engine path loads config and runs `recover_setup_runtime_registration` first and answers no selection on zero or many (its callers print their own advice), while the ComfyUI path never recovers and refuses on zero or many with its own text. Pin both as they are today, for zero, one and several ready runtimes and for a setup runtime the registry has lost. Each test that checks a refusal also checks the state it describes: that nothing was written, and that the remedy the message names (`rocm runtimes activate`, `--runtime-id`, repairing the runtime) clears it. No production code changes. Signed-off-by: Jussi Elo * fix(comfyui): pick the single ready runtime by the same rule as the engine paths With no runtime chosen, `rocm comfyui install` and the engine paths (`rocm serve`, `rocm engines install`) each kept their own copy of "use the one ready runtime". They agreed on what ready means but not on the steps around it: the engine path recovered the runtime `rocm setup` created before counting, ComfyUI never did, so a runtime whose registry entry had been lost made the install refuse with "Install ROCm first" (or silently pick a different runtime registered beside it) while the same machine served fine. Move the rule to `runtime_adopt::select_single_ready_runtime`, beside `recover_setup_runtime_registration`: recover setup, load the registry, count the runtimes `validate_runtime_manifest_for_activation` accepts, and return one, none (with everything registered) or several (the ready ones). Both callers use it and delete their copy. Each keeps its own words: the engine path still treats none and several as "no selection" and leaves the advice to its callers, and ComfyUI still refuses with its own `/runtimes`, `rocm runtimes activate` and `--runtime-id` text. What changes for a ComfyUI user, only when no `--runtime-id`, active runtime or configured default says which runtime to use: the runtime setup created is found even if its registry entry was lost, and finding it writes the entry back (also under `--dry-run`); with another ready runtime registered beside it the install now refuses as ambiguous instead of picking the other one; and an unusable setup manifest now fails the install with the recovery error instead of being skipped. The engine paths behave as before. The ComfyUI characterisation test for recovery flips to the new behaviour, and a test now runs both callers against the same registry states and fails if they disagree. The e2e scenario comfyui-15 covers the recovery through the dry run. Signed-off-by: Jussi Elo * test(comfyui): pin that an unusable setup runtime fails the default selection The previous commit makes the ComfyUI install recover the runtime `rocm setup` created before counting ready runtimes, as the engine paths do. A recovery that fails (the setup folder's runtime no longer passes the activation check) is an error there too; pin it for ComfyUI, with the registry left untouched. Signed-off-by: Jussi Elo * refactor(runtime): one definition of a non-empty install root has_nontrivial_directory_contents (therock.rs) and runtime_install_root_has_payload (runtime_registry.rs) were two copies of the same rule: an install root has a payload when it holds an entry whose name does not start with a dot, so the runtime's own .rocm-cli-runtime.json marker does not count. They answered every input identically except a path that does not exist (or a dangling symlink root): the first said "empty", the second failed with "failed to read". No caller can reach that row. Activation validation rejects a root that is not a directory ("install root is missing") before it asks, tarball extraction asks right after create_dir_all, and the update check maps any failure to "not installed" (unwrap_or(false)) anyway. Keep the first one's answer (a root that is not there holds nothing, so asking never needs an error path for it) and move it to rocm-core as install_root_has_payload next to the other install-root helpers; a root that exists but cannot be listed is still an error. Tarball extraction, the update path's check that a repair's replacement is installed, and activation validation of a tarball runtime now all ask it. The extraction refusal moves into ensure_tarball_install_target_is_clean so the refusal can be tested with its message and the files it leaves alone. The two per-copy tables from the previous commit are replaced by one table over the shared function, plus a test per caller that goes through it: a marker-only root has no payload for validation (message and unusable status asserted together), for the extraction guard, and for the replacement check. Re-introducing a private copy in a caller, or changing the shared predicate, turns these red. No user-visible change. Signed-off-by: Jussi Elo * test(system_packages): pin that the install hints read the real host's os-release The OpenMPI, libatomic and libnuma install hints name package-manager commands for the machine the process runs on, so a simulated host root must not choose their distro. rocm-core's private real_os_release (openmpi.rs) and the apps/rocm one (system_packages.rs) are the same one-line call, read_os_release with the identity resolver; the apps copy already has a test for it, the rocm-core copy had none. Add one that plants a root whose os-release names another distro and checks all three hints are unchanged by it. It lives with the apps copy's test, in the rocm binary's test process, so the environment variable it sets cannot overlap the rocm-core tests that read host paths. Swapping the identity resolver in openmpi.rs for host_path_for_str turns it red. Tests only. Signed-off-by: Jussi Elo * refactor(hardware_root): one reader for the real host's os-release The OpenMPI, libatomic and libnuma install hints (rocm-core/openmpi.rs), the system-package plans and the driver plan (apps/rocm via system_packages.rs) each need the os-release of the machine the process runs on, not a simulated host root. openmpi.rs and system_packages.rs each carried their own real_os_release, identical line for line: read_os_release with the identity resolver. The e2e driver step spelled the same call a third time. They read the same two files in the same order with the same fallback and error rules, so nothing a user can see differs; the risk was one copy being edited and the others not. Move the reader to rocm-core as hardware_root::real_os_release, delete both private copies, and make the identity resolver private so a caller outside hardware_root.rs has no shared way to build a second reader. The apps copy's test (real_os_release_ignores_a_simulated_host_root) stays where it was and now calls the shared function; the hint test from the previous commit holds the three hints to the same rule, and a scan test in the rocm binary fails if any of its source files names read_os_release or the resolver. The shared function is no longer cfg(linux)-gated as openmpi's was; off Linux it reads the same paths and finds nothing, as the apps copy always did. Doc comments, docs/architecture.md and docs/testing.md that named the two copies are updated. Signed-off-by: Jussi Elo * test: pin which manifest files each way of registering a runtime writes A runtime has up to two manifest files: its registry entry and the .rocm-cli-runtime.json that ROCm CLI keeps in the install root. Installing writes both and always overwrites. Importing and adopting write the registry entry only and refuse to replace an existing entry unless asked, and recovering the runtime that setup created restores a missing registry entry from the manifest already in its folder. Nothing asserted that file set per path. These tests assert, for install, import, adopt and setup recovery, which of the two files exist afterwards, and for import and adopt that a duplicate is refused (message and untouched entry together) until replacement is requested. They pass on the current code, so they pin its behaviour before the writers are merged. The import scenario gains an assertion that the registry holds the entry and that the imported folder gains no manifest file, after the first import and after a replacing one. The adopt test's environment fixture moves into a helper so the new adopt test reuses it. Signed-off-by: Jussi Elo * refactor: register every runtime through one manifest writer Two functions wrote a runtime's manifests. The SDK install path wrote the registry entry and the .rocm-cli-runtime.json in the install root and always overwrote; the import, adopt and setup-recovery path wrote only the registry entry and refused to overwrite unless --replace was given. Which files a runtime got depended on which function its caller happened to use, and nothing tied the choice to the thing it stands for. write_runtime_manifest in runtime_registry.rs is now the only writer. It takes the overwrite policy (ManifestOverwrite::Replace or ::Refuse) as its one parameter and decides the install-root file from InstalledRuntimeManifest::owns_install_root: a runtime ROCm CLI installed gets it, an imported or adopted one, which is someone else's folder that uninstall and prune never delete, does not. The private writer in runtime_adopt.rs is gone. The file is left alone when it already holds what would be written, so recovering a runtime from its own manifest does not rewrite it. No user-visible file set changes: install still writes both files and replaces, import and adopt still write the registry entry only and refuse a duplicate without --replace, recovery still only fills a missing entry. The test that pins this per path stays green across the change, and new tests tie the writer's ownership rule to uninstall's, check each policy and the skipped rewrite, and fail if production code outside runtime_registry.rs opens a registry entry for writing itself. Neither file is written atomically; that is left to the shared atomic writer work. Signed-off-by: Jussi Elo * fix(comfyui): keep a --dry-run install preview from restoring the registry entry With no runtime chosen, `rocm comfyui install` now recovers the runtime `rocm setup` created before counting the ready ones, and recovering writes a lost registry entry back. That ran before the install looked at `--dry-run`, so a preview changed a file even though the flag is documented as "Show what would happen without changing files". The recovery is split into finding the setup runtime (read-only) and writing its entry. `select_single_ready_runtime` takes a `SetupRecovery`: `Persist` writes the entry as before, and `PreviewOnly` counts the setup runtime from its own manifest, normalised and ordered as the registry would return it once restored, and writes nothing, so the preview names the runtime the real run will use. The engine paths keep `Persist`; the install derives the mode from `--dry-run`. The flag help stays true, so no wording changes. comfyui-15 now asserts that the preview selects the runtime and the registry still has no entry, and comfyui-16 runs the install against an existing checkout and asserts the entry is written back. Unit tests pin both modes, that preview and real run list the same ready runtimes in the same order, that an unusable setup runtime is reported alike, and the dry-run mapping. Signed-off-by: Jussi Elo * fix(runtimes): judge an install root manifest as current by what it decodes to The shared manifest writer skips rewriting the `.rocm-cli-runtime.json` in an install root when it already holds what would be written, so recovering the runtime `rocm setup` created does not fail on a folder the user cannot write to. It compared bytes. A manifest an older binary wrote omits keys that now read back as defaults (`wheel_composition` and other optional fields) and serialize as explicit values, so the bytes differ although the manifest is the same. Recovery then wrote the missing registry entry and failed rewriting the otherwise usable, read-only file. The writer now also treats a file as current when it decodes to the manifest that would be written once normalised for storage (`install_root_manifest_is_current`). A different manifest, or a file that does not decode, is still rewritten. The regression test recovers a setup runtime whose read-only install root manifest is in the older shape and asserts the registry entry is restored and the file's bytes are untouched; a pure test covers the equal, same-manifest, different-manifest and not-a-manifest cases on every platform. Signed-off-by: Jussi Elo * fix(comfyui): let an explicit runtime choice find the setup runtime a preview named `rocm comfyui install --dry-run` finds the runtime `rocm setup` created even when its registry entry was lost, and prints `rocm comfyui install --runtime-id ` for it without writing the entry. That printed command then failed from the state the preview left: an explicit choice only looked in the registry, so it refused with "Install ROCm first" or "runtime not found". A choice that names a runtime (`--runtime-id`, the active runtime, the configured default) is still answered by the registry first. When the registry alone cannot answer, it is looked up once more among the registry plus the setup runtime, the way the engine paths resolve an explicit `--runtime-id`. `SetupRecovery` decides the write: a preview adds the runtime to the list and writes nothing, a real run writes the entry back. If the setup runtime cannot be used, the registry's own refusal stands rather than an error about a runtime the user did not name. The list the single-ready rule counts from is now `registered_runtimes_with_setup`, shared with the new lookup. Covered by unit tests that follow the printed command from the unchanged registry (preview again, then a real run), a matrix of choices by recovery mode, and the choices that must not change; and by scenario comfyui-17, which runs the printed command as the CLI prints it. Signed-off-by: Jussi Elo * fix(comfyui): report a registry that cannot take the setup runtime back instead of refusing A choice that names a runtime (`--runtime-id`, the active runtime, the configured default) finds the runtime `rocm setup` created when its registry entry was lost. Looking it up discarded every error from that second look, the failure to write the entry back included. With the registry readable but not writable, `rocm comfyui install --dry-run` succeeded, and running the command it printed answered "Install ROCm first" (or "runtime not found") about a runtime that was there, instead of saying the entry could not be restored. The setup runtime is now found and picked without writing, whichever recovery was asked for, and a real run writes the entry back only for a pick it is going to use, propagating that error as it is. Not finding a usable setup runtime still leaves the registry's own refusal, so a choice naming something else and an unusable setup runtime answer as before. The engine paths already propagated the same failure (`rocm serve` and `rocm engines install` through `resolve_runtime_selector_to_exact_key` and the one-ready-runtime rule); nothing pinned it, so a unit test now covers the explicit and the default selection there. Covered by unit tests for each way of naming the runtime, with the preview succeeding and the real run reporting the error and leaving the registry as it was, and for a read-only registry directory holding other runtimes; and by scenario comfyui-18, which runs the printed command against a registry that cannot be written. The fixture is a plain file where the registry directory should be, because a read-only directory does not stop root. Signed-off-by: Jussi Elo * fix(comfyui): set aside an unusable setup runtime while a registered runtime is ready With no runtime chosen, `rocm comfyui install` failed with the setup recovery error ("setup runtime ... is not usable") whenever the folder `rocm setup` named held a runtime that no longer passed the readiness check, even when exactly one other runtime was registered and ready. A choice that names a runtime already keeps the registry's answer in that case, so the two ways of asking disagreed, and the engine commands reached the same rule through the same function. A setup runtime that cannot be used is not a ready runtime and cannot change which runtimes are ready, so `select_single_ready_runtime` now counts the registry alone when the setup runtime is unusable: one ready runtime is picked, several are refused as ambiguous, and nothing is written. When the registry holds no ready runtime the setup runtime was the only candidate and its error is still the answer. A usable setup runtime whose registry entry cannot be written back is unchanged and still reported as that failure, because it is one of the runtimes being counted. The rule is shared, so `rocm serve` and `rocm engines install` pick the registered runtime in this corner too; their other recovery call sites are untouched. The state matrix is a test of its own: the choice (none, a registered runtime, the setup runtime) against 0, 1 or 2 ready registered runtimes, a setup folder that is absent, usable or unusable, a registry that takes writes or not, and preview or real run, each row asserting the message and the registry entries together. Scenario comfyui-19 covers the install preview with an unusable setup runtime beside one ready registered runtime. Signed-off-by: Jussi Elo * fix(runtimes): write the registry entry after the install-root manifest `write_runtime_manifest` wrote the registry entry first and the install-root `.rocm-cli-runtime.json` second. When the second write failed, the caller's error (for setup recovery, "failed to restore setup runtime ... into ") described a runtime that was in fact registered, and the next run succeeded without saying anything. The registry entry is what registering a runtime means, so it is now written last: a failed install-root write leaves the registry exactly as it was, and a failed registry write leaves no new entry, so an error that says a runtime could not be registered or restored matches the registry. A failed registry write can leave the install-root file behind, which holds what a retry writes. Reaching the install-root write from setup recovery is not possible in practice: the manifest recovery restores is the one decoded from that very file, so the file is always what would be written and is skipped. A test pins that with a hand-edited file, asserted on the bytes so it holds for an administrator as well. A second test pins the error and the registry together for each failing write, under both overwrite policies. Signed-off-by: Jussi Elo * test(runtimes): make the single-registry-writer guard read whole files, not a window `only_the_shared_routine_writes_a_registry_entry` looked 200 characters past each `fs::write(` for three names and exempted any file called `test_support.rs`. A writer that built its path another way (the `runtimes` and `registry` segments joined into a variable, one literal, a path held away from the write) passed it; a storage-module function doing exactly that stayed green. The guard now treats a production file as a writer when it both calls a write primitive and names the registry directory by any of its spellings (the two path helpers, the two path segments, or one literal holding both), wherever the two sit in the file. Comments are removed and test modules cut before the scan. It reads the `rocm` app, `rocmd`, `rocm-core` and the engines. The files that call the two path helpers are an explicit reviewed list that is compared both ways, and the two fixture modules are exempt because their declarations are `#[cfg(test)]`, which the test checks, instead of by file name. `registry_writer_detector_sees_every_way_of_building_the_path` pins the detector on nine cases, including the blind spots above. What the guard still does not catch is written in its doc comment: a writer handed the registry path by a function in another file, a write primitive outside its list, and code outside the scanned roots. Signed-off-by: Jussi Elo * test(comfyui): fix clippy findings in the runtime selection state matrix The new state-matrix test tripped three pedantic lints that the workspace denies: constants declared after statements, a two-armed match that is an `if let`/`let else`, and a missing semicolon after an assertion. No change to what the test checks. Signed-off-by: Jussi Elo * fix(runtimes): count a file builder as a write only when a write option is chained on it The single-registry-writer guard treated every `OpenOptions::new(` as a write primitive. `amd_smi.rs` builds the registry directory path only to read it and opens device files with `OpenOptions::new().read(true).open(..)`, so the guard flagged it as a second writer and failed on the merged tree. The detector now reads the method chain after `OpenOptions::new(` or `File::options(` across lines and whitespace: the builder is a read-only probe when the chain reaches `.open(` and calls none of `write`, `append`, `create`, `create_new` or `truncate`. A builder kept in a variable, or a chain that never reaches `.open(`, cannot be judged and still counts as a writer. Every other write primitive is unchanged. `registry_writer_detector_sees_every_way_of_building_the_path` pins eight more cases: the amd_smi shape, read-only probes on one line, over several lines and through `File::options`, and the writing forms (`.write(true)`, a write option on a later line, `.append(true)`, a stored builder, an unreadable chain). The only file under the scanned roots that names the registry directory and uses a file builder is `amd_smi.rs`, so no other file changes verdict. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 2a817e276c377c9d4cb768da7e312015563b8abc) Signed-off-by: juhovainio --- README.md | 4 +- apps/rocm/src/cli.rs | 2 +- apps/rocm/src/comfyui/install.rs | 174 ++- apps/rocm/src/comfyui/runtime_env.rs | 1146 ++++++++++++++++- apps/rocm/src/driver_install.rs | 3 +- apps/rocm/src/driver_plan.rs | 6 +- apps/rocm/src/engine_runtime_select.rs | 163 ++- apps/rocm/src/engines_cmd.rs | 68 + apps/rocm/src/runtime_adopt.rs | 656 +++++++++- apps/rocm/src/runtime_registry.rs | 888 ++++++++++++- apps/rocm/src/runtimes_cmd.rs | 386 +++++- apps/rocm/src/system_packages.rs | 132 +- apps/rocm/src/test_support.rs | 52 + apps/rocm/src/therock.rs | 76 +- apps/rocm/src/therock/update.rs | 60 +- crates/rocm-core/src/hardware_root.rs | 34 +- crates/rocm-core/src/lib.rs | 23 +- crates/rocm-core/src/openmpi.rs | 24 +- crates/rocm-core/src/runtime.rs | 237 ++++ docs/architecture.md | 20 +- docs/testing.md | 79 +- engines/vllm/src/process.rs | 41 +- tests/e2e-cucumber/features/comfyui.feature | 83 ++ tests/e2e-cucumber/features/examine.feature | 6 +- .../features/runtime_lifecycle.feature | 2 + .../features/runtime_setup.feature | 19 + tests/e2e-cucumber/tests/e2e/comfyui_steps.rs | 316 ++++- tests/e2e-cucumber/tests/e2e/driver_steps.rs | 7 +- .../tests/e2e/runtime_lifecycle_steps.rs | 33 +- tests/e2e-cucumber/tests/e2e/runtime_steps.rs | 63 + 30 files changed, 4467 insertions(+), 336 deletions(-) diff --git a/README.md b/README.md index d3c45ec460..545d4347f7 100644 --- a/README.md +++ b/README.md @@ -833,7 +833,9 @@ exits without prompting or making changes. `adopt` registers an existing TheRock-based Python environment as a managed runtime. It does not work with standard ROCm package installs (for example, -`/opt/rocm`); use `rocm install sdk` instead. +`/opt/rocm`); use `rocm install sdk` instead. Without `--root`, the environment +folder is the one that holds the `--python` interpreter's `bin` (or `Scripts`) +folder; an interpreter anywhere else is refused until you pass `--root`. ### Disk space diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 548fc7fb09..222a5ef726 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -912,7 +912,7 @@ left, so it cannot undo more than one activation." /// Python executable inside the existing ROCm environment. #[arg(long)] python: PathBuf, - /// Optional SDK root path. + /// Python environment folder. Taken from --python when that sits in a bin or Scripts folder; required otherwise. #[arg(long)] root: Option, /// Runtime id to assign. diff --git a/apps/rocm/src/comfyui/install.rs b/apps/rocm/src/comfyui/install.rs index 0ac2a164de..c446ddc3b6 100644 --- a/apps/rocm/src/comfyui/install.rs +++ b/apps/rocm/src/comfyui/install.rs @@ -27,6 +27,7 @@ use super::{ }; use crate::cli_progress::AnimatedSpinner; use crate::freeform_planner::format_structured_tool_call; +use crate::runtime_adopt::SetupRecovery; use crate::therock; use anyhow::{Context, Result, bail}; use flate2::read::GzDecoder; @@ -48,7 +49,14 @@ pub(crate) fn install( options: ComfyUiInstallOptions, ) -> Result { paths.ensure()?; - let runtime = select_runtime(paths, config, options.runtime_id.as_deref())?; + // A preview names the runtime the real run will use but does not write the + // registry entry that run would restore. + let runtime = select_runtime( + paths, + config, + options.runtime_id.as_deref(), + SetupRecovery::for_dry_run(options.dry_run), + )?; let app_root = runtime_app_root(&runtime.manifest); let source_path = source_path_from_app_root(&app_root); let log_path = install_log_path_from_app_root(&app_root); @@ -1908,6 +1916,170 @@ mod tests { remove_test_root(&paths); Ok(()) } + + /// `--dry-run` promises not to change files. With nothing chosen, the install + /// finds the runtime `rocm setup` created even though its registry entry was + /// lost, but a preview must not write that entry back: it names the same + /// runtime the real run will use and leaves the registry as it found it. + #[test] + fn dry_run_previews_the_setup_runtime_without_restoring_its_registry_entry() -> Result<()> { + let paths = test_paths("comfyui-dry-run-setup-runtime"); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let config = RocmCliConfig::load(&paths)?; + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + Vec::::new() + ); + + let rendered = install( + &paths, + &config, + ComfyUiInstallOptions { + runtime_id: None, + reinstall: false, + dry_run: true, + }, + )?; + + assert!(rendered.contains(" mode: dry-run\n"), "{rendered}"); + assert!( + rendered.contains(&format!( + " ROCm runtime: {}\n", + therock::runtime_version_display(&manifest.version) + )), + "{rendered}" + ); + assert!( + rendered + .lines() + .any(|line| line.contains("install command:") + && line.contains(&format!("--runtime-id {}", manifest.runtime_key))), + "{rendered}" + ); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + Vec::::new(), + "a preview must not restore the registry entry" + ); + remove_test_root(&paths); + Ok(()) + } + + /// A runtime that setup created and the registry forgot, with a cached + /// ComfyUI archive and a fake interpreter, so a real install into it needs + /// no network and no GPU. + #[cfg(unix)] + fn setup_runtime_the_registry_forgot( + name: &str, + ) -> Result<(AppPaths, RocmCliConfig, therock::InstalledRuntimeManifest)> { + let paths = test_paths(name); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let config = RocmCliConfig::load(&paths)?; + let app_root = runtime_app_root(&manifest); + plant_cached_source_archive( + &app_root, + &[("main.py", "new main"), ("requirements.txt", "torch\n")], + )?; + let python = PathBuf::from( + manifest + .python_executable + .as_deref() + .context("runtime has no Python")?, + ); + rocm_core::fake_executable::write_fake_executable( + &python, + "#!/bin/sh\nprintf '%s' '{\"torch_version\": \"2.4.0\", \"torch_cuda_available\": true, \"device_count\": 1, \"devices\": [\"Fake GPU\"]}' > \"$2\"\n", + )?; + Ok((paths, config, manifest)) + } + + /// The preview prints the install command it would run, and that command + /// has to work from the state the preview left: the registry still without + /// the entry. Its `--runtime-id` is an explicit choice, which used to look + /// only in the registry and fail with "Install ROCm first". Following the + /// printed command previews again without a write, and runs for real with + /// the one write the real run makes. + #[cfg(unix)] + #[test] + fn the_printed_install_command_works_from_the_state_the_preview_left() -> Result<()> { + let (paths, config, manifest) = + setup_runtime_the_registry_forgot("comfyui-printed-command-setup-runtime")?; + let options = |runtime_id: Option, dry_run| ComfyUiInstallOptions { + runtime_id, + reinstall: false, + dry_run, + }; + let printed_runtime_id = |rendered: &str| -> Result { + let line = rendered + .lines() + .find(|line| line.contains("install command:")) + .with_context(|| format!("the preview prints its command:\n{rendered}"))?; + let words: Vec<&str> = line.split_whitespace().collect(); + let at = words + .iter() + .position(|word| *word == "--runtime-id") + .context("the printed command names the runtime")?; + Ok(words[at + 1].to_owned()) + }; + + let preview = install(&paths, &config, options(None, true))?; + let printed = printed_runtime_id(&preview)?; + assert_eq!(printed, manifest.runtime_key, "{preview}"); + assert!( + crate::test_support::runtime_registry_entry_names(&paths).is_empty(), + "the premise: the preview leaves the registry without the entry" + ); + + let again = install(&paths, &config, options(Some(printed.clone()), true)) + .context("the printed command, previewed again")?; + assert_eq!(printed_runtime_id(&again)?, printed, "{again}"); + assert!( + crate::test_support::runtime_registry_entry_names(&paths).is_empty(), + "previewing the printed command must not write the entry either" + ); + + install(&paths, &config, options(Some(printed), false)) + .context("the printed command, run for real")?; + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + vec![format!("{}.json", manifest.runtime_key)], + "the real run restores the entry the printed command's runtime needs" + ); + remove_test_root(&paths); + Ok(()) + } + + /// The other half: the real run does restore the entry, so the runtime the + /// preview named is one `rocm runtimes list` shows once the install has run. + #[cfg(unix)] + #[test] + fn install_restores_the_setup_runtime_registry_entry_the_preview_left_alone() -> Result<()> { + let (paths, config, manifest) = + setup_runtime_the_registry_forgot("comfyui-install-setup-runtime")?; + + install( + &paths, + &config, + ComfyUiInstallOptions { + runtime_id: None, + reinstall: false, + dry_run: false, + }, + )?; + + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + vec![format!("{}.json", manifest.runtime_key)] + ); + remove_test_root(&paths); + Ok(()) + } } /// Property tests for the ComfyUI source unpack, judged on the filesystem by diff --git a/apps/rocm/src/comfyui/runtime_env.rs b/apps/rocm/src/comfyui/runtime_env.rs index 3e496900be..70956108bb 100644 --- a/apps/rocm/src/comfyui/runtime_env.rs +++ b/apps/rocm/src/comfyui/runtime_env.rs @@ -7,7 +7,10 @@ //! //! [`select_runtime`] turns an explicit `--runtime-id` or the configured //! default into the one ready managed Python runtime an install runs against, -//! and explains how to pick one when none or several qualify. The +//! and explains how to pick one when none or several qualify. A choice that +//! names the runtime `rocm setup` created finds it even when its registry entry +//! was lost (see `pick_from_registry`), so the `--runtime-id` an install preview +//! prints works from the state the preview left. The //! `runtime_environment_*` functions build the `VIRTUAL_ENV`, `ROCM_PATH`, //! `PATH` and library-path settings a ComfyUI child process runs with, and //! [`apply_runtime_environment`] puts them on a command. @@ -17,12 +20,16 @@ //! ComfyUI types. use super::{ComfyUiManifest, ComfyUiRuntimeEnvironment, SelectedRuntime}; +use crate::runtime_adopt::{ + SetupRecovery, SingleReadyRuntime, recover_setup_runtime_registration, + registered_runtimes_with_setup, select_single_ready_runtime, +}; use crate::runtimes_cmd::runtime_usability_status; use crate::therock; use anyhow::{Context, Result, bail}; use rocm_core::{ AppPaths, RocmCliConfig, runtime_is_linux, runtime_is_windows, runtime_path_list_join, - runtime_path_list_split, runtime_paths_equivalent, + runtime_path_list_split, runtime_paths_equivalent, runtime_python_env_root_for_executable, }; use std::ffi::OsString; use std::path::{Path, PathBuf}; @@ -32,16 +39,13 @@ pub(super) fn select_runtime( paths: &AppPaths, config: &RocmCliConfig, selector: Option<&str>, + recovery: SetupRecovery, ) -> Result { - let manifests = therock::load_runtime_manifests(paths)?; - if manifests.is_empty() { - bail!( - "Install ROCm first: run `rocm` and pick \"Set up this system\", then install ComfyUI." - ); - } let manifest = match selector.map(str::trim).filter(|value| !value.is_empty()) { - Some(selector) => select_runtime_by_selector(&manifests, selector)?.clone(), - None => select_default_runtime(config, &manifests)?.clone(), + Some(selector) => pick_from_registry(paths, config, recovery, |manifests| { + select_runtime_by_selector(manifests, selector) + })?, + None => select_default_runtime(paths, config, recovery)?, }; if runtime_usability_status(&manifest) != "ready" { bail!( @@ -119,16 +123,85 @@ fn format_runtime_statuses<'a>( .join(", ") } -fn select_default_runtime<'a>( +/// Apply `pick` (a choice that names a runtime: `--runtime-id`, the active +/// runtime or the configured default) to the registry. A runtime that `rocm +/// setup` created and whose registry entry was lost is not a refusal either: +/// when the registry alone has no answer, `pick` runs once more over the +/// registry plus that runtime, the way the engine paths resolve an explicit +/// `--runtime-id`. Which write that implies is `recovery`'s: a preview adds the +/// runtime to the list without writing its entry, so the `--runtime-id` an +/// install preview prints is a command that works from the state the preview +/// left. If setup's runtime cannot be used, the registry's own answer stands. +/// +/// The two failures are not alike. Not finding a usable setup runtime leaves the +/// registry's refusal, because the user named something else or setup left +/// nothing. Failing to write the entry of a setup runtime that was found and +/// picked is not a refusal about a missing runtime, and swallowing it would +/// answer "Install ROCm first" about a runtime that is there, so that error is +/// returned as it is. The runtime is therefore found and picked without writing, +/// and written back only for a pick that is going to be used. +fn pick_from_registry( + paths: &AppPaths, config: &RocmCliConfig, - manifests: &'a [therock::InstalledRuntimeManifest], -) -> Result<&'a therock::InstalledRuntimeManifest> { + recovery: SetupRecovery, + pick: impl for<'a> Fn( + &'a [therock::InstalledRuntimeManifest], + ) -> Result<&'a therock::InstalledRuntimeManifest>, +) -> Result { + let registered = therock::load_runtime_manifests(paths)?; + let refusal = if registered.is_empty() { + install_rocm_first() + } else { + match pick(®istered) { + Ok(manifest) => return Ok(manifest.clone()), + Err(error) => error, + } + }; + // Found without writing, whichever recovery was asked for: the write comes + // after the pick, so its failure is reported as what it is. + let with_setup = match registered_runtimes_with_setup(paths, config, SetupRecovery::PreviewOnly) + { + Ok(with_setup) if with_setup.len() > registered.len() => with_setup, + _ => return Err(refusal), + }; + let picked = pick(&with_setup)?.clone(); + if recovery == SetupRecovery::Persist { + recover_setup_runtime_registration(paths, config)?; + } + Ok(picked) +} + +fn install_rocm_first() -> anyhow::Error { + anyhow::anyhow!( + "Install ROCm first: run `rocm` and pick \"Set up this system\", then install ComfyUI." + ) +} + +fn select_default_runtime( + paths: &AppPaths, + config: &RocmCliConfig, + recovery: SetupRecovery, +) -> Result { if let Some(active_key) = config.active_runtime_key.as_deref() { - return select_runtime_by_selector(manifests, active_key); + return pick_from_registry(paths, config, recovery, |manifests| { + select_runtime_by_selector(manifests, active_key) + }); } - let Some(default_runtime_id) = config.default_runtime_id.as_deref() else { - return select_single_ready_runtime(manifests); - }; + match config.default_runtime_id.as_deref() { + Some(default_runtime_id) => pick_from_registry(paths, config, recovery, |manifests| { + select_runtime_by_default_id(manifests, default_runtime_id) + }), + // Nothing says which runtime to use: the rule the engine paths share. + None => { + single_ready_runtime_or_refusal(select_single_ready_runtime(paths, config, recovery)?) + } + } +} + +fn select_runtime_by_default_id<'a>( + manifests: &'a [therock::InstalledRuntimeManifest], + default_runtime_id: &str, +) -> Result<&'a therock::InstalledRuntimeManifest> { let matches = manifests .iter() .filter(|manifest| manifest.runtime_id.eq_ignore_ascii_case(default_runtime_id)) @@ -156,23 +229,24 @@ fn select_default_runtime<'a>( } } -fn select_single_ready_runtime( - manifests: &[therock::InstalledRuntimeManifest], -) -> Result<&therock::InstalledRuntimeManifest> { - let ready = manifests - .iter() - .filter(|manifest| runtime_usability_status(manifest) == "ready") - .collect::>(); - match ready.as_slice() { - [manifest] => Ok(*manifest), - [] => { - let statuses = format_runtime_statuses(manifests); +/// ComfyUI's words for what [`select_single_ready_runtime`] found. The rule +/// itself, including which runtimes count as ready, is not decided here. +fn single_ready_runtime_or_refusal( + found: SingleReadyRuntime, +) -> Result { + match found { + SingleReadyRuntime::One(manifest) => Ok(*manifest), + SingleReadyRuntime::Zero { registered } if registered.is_empty() => { + Err(install_rocm_first()) + } + SingleReadyRuntime::Zero { registered } => { + let statuses = format_runtime_statuses(®istered); bail!( "No ROCm runtime is ready. Fix one or install another: run `rocm` and pick \"Set up this system\" (or run `rocm install sdk`), then install ComfyUI. Runtimes found: {statuses}. See `rocm runtimes list`." ) } - _ => { - let available = format_available_runtime_keys(ready.iter().copied()); + SingleReadyRuntime::Many { ready } => { + let available = format_available_runtime_keys(&ready); bail!( "Multiple ROCm runtimes are ready. Pick one in `/runtimes`, set a default with `rocm runtimes activate `, or pass `--runtime-id `. Available: {available}. See `rocm runtimes list`." ) @@ -237,7 +311,7 @@ pub(super) fn runtime_environment_for_manifest( } let mut env = ComfyUiRuntimeEnvironment { - venv_root: infer_python_env_root(&manifest.python_executable), + venv_root: runtime_python_env_root_for_executable(&manifest.python_executable), rocm_root: Some(manifest.runtime_root.clone()), path_entries: Vec::new(), library_entries: Vec::new(), @@ -254,7 +328,7 @@ pub(super) fn runtime_environment_from_runtime( python: &Path, ) -> ComfyUiRuntimeEnvironment { let mut env = ComfyUiRuntimeEnvironment { - venv_root: infer_python_env_root(python), + venv_root: runtime_python_env_root_for_executable(python), rocm_root: manifest .rocm_sdk .as_ref() @@ -307,15 +381,6 @@ fn collect_rocm_runtime_paths(root: &Path, env: &mut ComfyUiRuntimeEnvironment) } } -fn infer_python_env_root(python: &Path) -> Option { - let bin_dir = python.parent()?; - let name = bin_dir.file_name()?.to_string_lossy().to_ascii_lowercase(); - if matches!(name.as_str(), "scripts" | "bin") { - return bin_dir.parent().map(Path::to_path_buf); - } - None -} - pub(super) fn apply_runtime_environment( command: &mut Command, env: &ComfyUiRuntimeEnvironment, @@ -397,9 +462,22 @@ mod tests { use super::*; use crate::comfyui::render_status; use crate::comfyui::tests::{ready_runtime_manifest, test_paths, write_runtime_manifest}; + use crate::runtime_adopt::classify_single_ready_runtime; + use crate::test_support::{runtime_registry_entry_names, write_test_pip_runtime}; use rocm_core::{runtime_python_executable_in_env, runtime_rocm_library_filename}; use std::fs; + /// `select_runtime` as a real install runs it: a runtime that only setup + /// recovery can make visible has its registry entry written back. The + /// preview mode has its own tests. + fn persisting_select_runtime( + paths: &AppPaths, + config: &RocmCliConfig, + selector: Option<&str>, + ) -> Result { + select_runtime(paths, config, selector, SetupRecovery::Persist) + } + #[test] fn default_runtime_selection_uses_single_ready_runtime() -> Result<()> { let paths = test_paths("comfyui-single-ready-runtime"); @@ -461,7 +539,8 @@ mod tests { }; let manifests = [manifest]; - let selected = select_default_runtime(&RocmCliConfig::default(), &manifests)?; + let selected = + single_ready_runtime_or_refusal(classify_single_ready_runtime(manifests.to_vec()))?; assert_eq!( selected.runtime_key, "release-pip-gfx120x-all-7-13-0a20260511" @@ -510,8 +589,9 @@ mod tests { let nightly = ready_runtime_manifest(&paths, "nightly-wheel-gfx94x-dcgpu-7-14-0")?; let manifests = [release, nightly]; - let error = select_default_runtime(&RocmCliConfig::default(), &manifests) - .expect_err("two ready runtimes should be ambiguous"); + let error = + single_ready_runtime_or_refusal(classify_single_ready_runtime(manifests.to_vec())) + .expect_err("two ready runtimes should be ambiguous"); let message = error.to_string(); assert_actionable(&message); @@ -545,8 +625,9 @@ mod tests { let unusable = unusable_runtime_manifest(&paths, "release-wheel-gfx94x-dcgpu-7-13-0")?; let manifests = [unusable]; - let error = select_default_runtime(&RocmCliConfig::default(), &manifests) - .expect_err("an unusable runtime is not ready"); + let error = + single_ready_runtime_or_refusal(classify_single_ready_runtime(manifests.to_vec())) + .expect_err("an unusable runtime is not ready"); let message = error.to_string(); assert!( @@ -628,7 +709,7 @@ mod tests { ); // `select_runtime`'s bail when nothing is registered at all. - let error = select_runtime(&paths, &RocmCliConfig::default(), None) + let error = persisting_select_runtime(&paths, &RocmCliConfig::default(), None) .expect_err("no registered runtime means no selection"); let message = error.to_string(); assert!( @@ -653,11 +734,7 @@ mod tests { let unusable = unusable_runtime_manifest(&paths, "nightly-wheel-gfx94x-dcgpu-7-14-0")?; let manifests = [ready, unusable]; - let config = RocmCliConfig { - default_runtime_id: Some("no-such-runtime-id".to_owned()), - ..Default::default() - }; - let error = select_default_runtime(&config, &manifests) + let error = select_runtime_by_default_id(&manifests, "no-such-runtime-id") .expect_err("configured default id matches nothing"); let message = error.to_string(); @@ -689,11 +766,7 @@ mod tests { let nightly = ready_runtime_manifest(&paths, "nightly-wheel-gfx94x-dcgpu-7-14-0")?; let manifests = [release, nightly]; - let config = RocmCliConfig { - default_runtime_id: Some("therock-release:gfx120X-all".to_owned()), - ..Default::default() - }; - let error = select_default_runtime(&config, &manifests) + let error = select_runtime_by_default_id(&manifests, "therock-release:gfx120X-all") .expect_err("two manifests share the configured default id"); let message = error.to_string(); @@ -721,11 +794,7 @@ mod tests { ); let manifests = [ready, unusable]; - let config = RocmCliConfig { - default_runtime_id: Some("therock-release:gfx120X-all".to_owned()), - ..Default::default() - }; - let error = select_default_runtime(&config, &manifests) + let error = select_runtime_by_default_id(&manifests, "therock-release:gfx120X-all") .expect_err("two manifests share the configured default id"); let message = error.to_string(); @@ -847,7 +916,7 @@ mod tests { let unusable = unusable_runtime_manifest(&paths, "release-wheel-gfx94x-dcgpu-7-13-0")?; write_runtime_manifest(&paths, &unusable)?; - let error = select_runtime( + let error = persisting_select_runtime( &paths, &RocmCliConfig::default(), Some("release-wheel-gfx94x-dcgpu-7-13-0"), @@ -958,10 +1027,961 @@ mod tests { Ok(()) } + /// Zero runtimes registered: the refusal points at the launcher's setup row, + /// and selecting wrote nothing. + #[test] + fn comfyui_selection_with_no_registered_runtime_says_to_install_rocm_and_writes_nothing() + -> Result<()> { + let paths = test_paths("comfyui-selection-none-registered"); + + let message = persisting_select_runtime(&paths, &RocmCliConfig::default(), None) + .expect_err("no registered runtime means no selection") + .to_string(); + + assert!( + message.contains("Install ROCm first") + && message.contains("pick \"Set up this system\""), + "unexpected refusal: {message}" + ); + assert_eq!(runtime_registry_entry_names(&paths), Vec::::new()); + Ok(()) + } + + /// One ready runtime and nothing chosen: it is the selection. + #[test] + fn comfyui_selection_with_one_ready_runtime_selects_it() -> Result<()> { + let paths = test_paths("comfyui-selection-one-ready"); + let manifest = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-14-0", + "therock-release:gfx120X-all", + "7.14.0", + 20, + )?; + + let selected = persisting_select_runtime(&paths, &RocmCliConfig::default(), None)?; + + assert_eq!(selected.manifest.runtime_key, manifest.runtime_key); + assert_eq!( + Some(selected.python.display().to_string()), + manifest.python_executable + ); + Ok(()) + } + + /// Registered but none ready: the refusal lists the runtime with the reason + /// it is unusable, and fixing that reason (here: putting the interpreter + /// back) is what clears it. + #[test] + fn comfyui_selection_with_only_an_unusable_runtime_clears_once_it_is_fixed() -> Result<()> { + let paths = test_paths("comfyui-selection-none-ready"); + let manifest = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-14-0", + "therock-release:gfx120X-all", + "7.14.0", + 20, + )?; + let python = PathBuf::from( + manifest + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + ); + fs::remove_file(&python)?; + + let message = persisting_select_runtime(&paths, &RocmCliConfig::default(), None) + .expect_err("an unusable runtime is not ready") + .to_string(); + + assert!( + message.contains("No ROCm runtime is ready") + && message.contains("rocm runtimes list") + && message.contains(&format!("`{}` (unusable", manifest.runtime_key)), + "unexpected refusal: {message}" + ); + assert_eq!( + runtime_registry_entry_names(&paths), + vec![format!("{}.json", manifest.runtime_key)] + ); + + fs::write(&python, "python")?; + let selected = persisting_select_runtime(&paths, &RocmCliConfig::default(), None)?; + assert_eq!(selected.manifest.runtime_key, manifest.runtime_key); + Ok(()) + } + + /// Several ready runtimes and none chosen: the refusal names the remedies, + /// and each one clears it. `rocm runtimes activate ` is run as + /// `activate_runtime`, its implementation, and `--runtime-id` as the selector. + #[test] + fn comfyui_selection_with_several_ready_runtimes_names_remedies_that_clear_it() -> Result<()> { + let paths = test_paths("comfyui-selection-several-ready"); + let first = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 10, + )?; + let second = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-14-0", + "therock-release:gfx120X-all", + "7.14.0", + 20, + )?; + let mut config = RocmCliConfig::default(); + + let message = persisting_select_runtime(&paths, &config, None) + .expect_err("several ready runtimes and no choice must not select") + .to_string(); + + assert!( + message.contains("Multiple ROCm runtimes are ready") + && message.contains("rocm runtimes activate ") + && message.contains("--runtime-id ") + && message.contains(&first.runtime_key) + && message.contains(&second.runtime_key), + "unexpected refusal: {message}" + ); + + let selected = persisting_select_runtime(&paths, &config, Some(&first.runtime_key))?; + assert_eq!(selected.manifest.runtime_key, first.runtime_key); + + crate::runtimes_cmd::activate_runtime(&paths, &mut config, &second.runtime_key)?; + let selected = persisting_select_runtime(&paths, &config, None)?; + assert_eq!(selected.manifest.runtime_key, second.runtime_key); + Ok(()) + } + + /// The setup-recovery half of the rule: a ready runtime that only + /// `recover_setup_runtime_registration` can make visible is found by the + /// ComfyUI path, as it is by the engine paths, and finding it re-files its + /// registry entry, so the selection names a key `rocm runtimes list` shows. + #[test] + fn comfyui_selection_recovers_the_setup_runtime_the_registry_lost() -> Result<()> { + let paths = test_paths("comfyui-selection-setup-recovery"); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let config = RocmCliConfig::load(&paths)?; + assert!( + config.setup.therock_venv.is_some(), + "setup must name the folder for this test to mean anything" + ); + assert_eq!(runtime_registry_entry_names(&paths), Vec::::new()); + + let selected = persisting_select_runtime(&paths, &config, None)?; + + assert_eq!(selected.manifest.runtime_key, manifest.runtime_key); + assert_eq!( + runtime_registry_entry_names(&paths), + vec![format!("{}.json", manifest.runtime_key)] + ); + Ok(()) + } + + /// Recovering the setup runtime can fail, and the ComfyUI path now reports + /// that failure as the engine paths do, rather than carrying on as if setup + /// had named nothing: a setup folder whose runtime is no longer usable is + /// named, with the reason, and nothing is written to the registry. + #[test] + fn comfyui_selection_reports_a_setup_runtime_that_is_no_longer_usable() -> Result<()> { + let paths = test_paths("comfyui-selection-setup-unusable"); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + fs::remove_file( + manifest + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + let config = RocmCliConfig::load(&paths)?; + + let message = persisting_select_runtime(&paths, &config, None) + .expect_err("an unusable setup runtime is an error") + .to_string(); + + assert!( + message.contains("is not usable") && message.contains(&manifest.runtime_key), + "got: {message}" + ); + assert_eq!(runtime_registry_entry_names(&paths), Vec::::new()); + Ok(()) + } + + /// A choice that names the runtime setup created, by `--runtime-id` (key or + /// id), the active runtime or the configured default, finds it even though + /// its registry entry was lost. That is what makes the `--runtime-id` an + /// install preview prints a command that works from the state the preview + /// left: a preview adds the runtime to what it searches and writes nothing, + /// a real run writes the entry back. Each row asserts the pick and the + /// registry together. + #[test] + fn a_choice_naming_the_setup_runtime_finds_it_after_its_registry_entry_was_lost() -> Result<()> + { + type Choice = fn(&therock::InstalledRuntimeManifest, &mut RocmCliConfig) -> Option; + let choices: [(&str, Choice); 4] = [ + ("explicit key", |manifest, _| { + Some(manifest.runtime_key.clone()) + }), + ("explicit id", |manifest, _| { + Some(manifest.runtime_id.clone()) + }), + ("active runtime", |manifest, config| { + config.active_runtime_key = Some(manifest.runtime_key.clone()); + None + }), + ("configured default", |manifest, config| { + config.default_runtime_id = Some(manifest.runtime_id.clone()); + None + }), + ]; + for (label, choice) in choices { + for recovery in [SetupRecovery::PreviewOnly, SetupRecovery::Persist] { + let paths = test_paths(&format!( + "comfyui-choice-setup-{}-{recovery:?}", + label.replace(' ', "-") + )); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let mut config = RocmCliConfig::load(&paths)?; + let selector = choice(&manifest, &mut config); + + let selected = select_runtime(&paths, &config, selector.as_deref(), recovery) + .with_context(|| format!("{label}, {recovery:?}"))?; + + assert_eq!( + selected.manifest.runtime_key, manifest.runtime_key, + "{label}, {recovery:?}" + ); + let expected_entries = match recovery { + SetupRecovery::PreviewOnly => Vec::new(), + SetupRecovery::Persist => vec![format!("{}.json", manifest.runtime_key)], + }; + assert_eq!( + runtime_registry_entry_names(&paths), + expected_entries, + "{label}, {recovery:?}" + ); + } + } + Ok(()) + } + + /// Finding the setup runtime for a named choice must not hide that its + /// registry entry cannot be written back. The preview of such an install + /// succeeds, because it writes nothing; the printed command is the real run, + /// and it has to say why it cannot restore the entry rather than answer as if + /// there were no runtime ("Install ROCm first"). Each row asserts the error + /// and the registry together: the failing run leaves the registry as it + /// found it, and the runtime was still found (the preview, and the + /// unwritable registry's own refusal for a name that is not setup's). + #[test] + fn a_choice_naming_the_setup_runtime_reports_a_registry_that_cannot_take_it_back() -> Result<()> + { + type Choice = fn(&therock::InstalledRuntimeManifest, &mut RocmCliConfig) -> Option; + let choices: [(&str, Choice); 5] = [ + ("explicit key", |manifest, _| { + Some(manifest.runtime_key.clone()) + }), + ("explicit id", |manifest, _| { + Some(manifest.runtime_id.clone()) + }), + ("active runtime", |manifest, config| { + config.active_runtime_key = Some(manifest.runtime_key.clone()); + None + }), + ("configured default", |manifest, config| { + config.default_runtime_id = Some(manifest.runtime_id.clone()); + None + }), + // Not a named choice, but the same persistence, reported the same way. + ("no choice", |_, _| None), + ]; + for (label, choice) in choices { + let paths = test_paths(&format!( + "comfyui-choice-unwritable-{}", + label.replace(' ', "-") + )); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let mut config = RocmCliConfig::load(&paths)?; + let selector = choice(&manifest, &mut config); + let registry = crate::test_support::make_runtime_registry_unwritable(&paths)?; + + let previewed = select_runtime( + &paths, + &config, + selector.as_deref(), + SetupRecovery::PreviewOnly, + ) + .with_context(|| format!("{label}: the preview writes nothing"))?; + assert_eq!( + previewed.manifest.runtime_key, manifest.runtime_key, + "{label}" + ); + + let error = + select_runtime(&paths, &config, selector.as_deref(), SetupRecovery::Persist) + .expect_err("a registry that cannot be written cannot take the entry back"); + let message = format!("{error:#}"); + assert!( + message.contains(&format!( + "failed to restore setup runtime `{}`", + manifest.runtime_key + )), + "{label}: got: {message}" + ); + assert!( + !message.contains("Install ROCm first") && !message.contains("not found"), + "{label}: the persistence error must not be replaced by a refusal about a missing runtime: {message}" + ); + assert_eq!( + fs::read_to_string(®istry)?, + "not a directory", + "{label}: a failed restore leaves the registry as it was" + ); + } + Ok(()) + } + + /// The same, with the registry readable and holding another runtime, which is + /// the state a read-only registry directory leaves. The refusal that stands + /// without the fix is then the registry's "not found", so the error is + /// compared with that. Where the directory stays writable regardless (an + /// administrator), the write succeeds and the test checks the restore + /// instead, so it cannot pass by writing nothing. + #[cfg(unix)] + #[test] + fn a_read_only_registry_directory_is_reported_not_hidden_behind_a_not_found() -> Result<()> { + use crate::runtimes_cmd::runtime_registry_dir; + use std::os::unix::fs::PermissionsExt; + + let paths = test_paths("comfyui-choice-read-only-registry"); + let setup = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 10, + )?; + let config = RocmCliConfig::load(&paths)?; + let registry = runtime_registry_dir(&paths); + let original = fs::metadata(®istry)?.permissions(); + fs::set_permissions(®istry, fs::Permissions::from_mode(0o555))?; + + let selected = select_runtime( + &paths, + &config, + Some(&setup.runtime_key), + SetupRecovery::Persist, + ); + + fs::set_permissions(®istry, original)?; + match selected { + Err(error) => { + let message = format!("{error:#}"); + assert!( + message.contains(&format!( + "failed to restore setup runtime `{}`", + setup.runtime_key + )) && !message.contains("not found"), + "got: {message}" + ); + assert!( + !runtime_registry_entry_names(&paths) + .contains(&format!("{}.json", setup.runtime_key)), + "a failed restore writes no entry" + ); + } + Ok(selected) => { + assert_eq!(selected.manifest.runtime_key, setup.runtime_key); + assert!( + runtime_registry_entry_names(&paths) + .contains(&format!("{}.json", setup.runtime_key)), + "the directory stayed writable, so the entry is restored" + ); + } + } + Ok(()) + } + + /// What the recovery must not turn into: a choice that names some other + /// runtime is answered by the registry whether or not setup's runtime is + /// usable, a name that matches nothing is still refused (with the setup + /// runtime among those found when it is usable), and an unusable setup + /// runtime leaves the registry's own refusal standing instead of replacing + /// it with an error about a runtime the user did not ask for. + #[test] + fn recovering_the_setup_runtime_does_not_change_the_answer_for_other_choices() -> Result<()> { + let paths = test_paths("comfyui-choice-other-than-setup"); + let setup = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let other = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 10, + )?; + let config = RocmCliConfig::load(&paths)?; + let registry = runtime_registry_entry_names(&paths); + assert_eq!(registry, [format!("{}.json", other.runtime_key)]); + + let picked = select_runtime( + &paths, + &config, + Some(&other.runtime_key), + SetupRecovery::PreviewOnly, + )?; + assert_eq!(picked.manifest.runtime_key, other.runtime_key); + + let miss = select_runtime( + &paths, + &config, + Some("no-such-runtime"), + SetupRecovery::PreviewOnly, + ) + .expect_err("nothing is called that") + .to_string(); + assert!( + miss.contains("ROCm runtime not found: `no-such-runtime`") + && miss.contains(&setup.runtime_key), + "{miss}" + ); + assert_eq!(runtime_registry_entry_names(&paths), registry); + + fs::remove_file( + setup + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + for recovery in [SetupRecovery::PreviewOnly, SetupRecovery::Persist] { + let picked = select_runtime(&paths, &config, Some(&other.runtime_key), recovery) + .with_context(|| format!("an unusable setup runtime, {recovery:?}"))?; + assert_eq!(picked.manifest.runtime_key, other.runtime_key); + let miss = select_runtime(&paths, &config, Some("no-such-runtime"), recovery) + .expect_err("nothing is called that") + .to_string(); + assert!( + miss.contains("ROCm runtime not found: `no-such-runtime`") + && !miss.contains("is not usable"), + "{recovery:?}: {miss}" + ); + assert_eq!( + runtime_registry_entry_names(&paths), + registry, + "{recovery:?}" + ); + } + Ok(()) + } + + /// A registry directory that stops writing, as far as this process can be + /// stopped: a read-only directory, which does not stop `root`. `None` where it + /// would not bite (the process can write anyway, or the host has no such + /// modes), so a caller skips the row instead of passing for the wrong reason. + /// Restores the mode on drop. + #[cfg(unix)] + struct ReadOnlyRegistry { + path: PathBuf, + original: fs::Permissions, + } + + #[cfg(unix)] + impl Drop for ReadOnlyRegistry { + fn drop(&mut self) { + let _ = fs::set_permissions(&self.path, self.original.clone()); + } + } + + #[cfg(unix)] + fn make_registry_read_only(paths: &AppPaths) -> Option { + use std::os::unix::fs::PermissionsExt; + + let path = crate::runtimes_cmd::runtime_registry_dir(paths); + let original = fs::metadata(&path).ok()?.permissions(); + fs::set_permissions(&path, fs::Permissions::from_mode(0o555)).ok()?; + let guard = ReadOnlyRegistry { path, original }; + let probe = guard.path.join(".write-probe"); + if fs::write(&probe, "").is_ok() { + let _ = fs::remove_file(probe); + return None; + } + Some(guard) + } + + #[cfg(not(unix))] + struct ReadOnlyRegistry; + + #[cfg(not(unix))] + fn make_registry_read_only(_paths: &AppPaths) -> Option { + None + } + + /// Every state `rocm comfyui install` can pick a runtime from, one row each: + /// what the user chose (nothing, a registered runtime by name, the setup + /// runtime by name) against how many ready runtimes the registry holds (0, 1, + /// 2), what the setup folder holds (nothing, a usable runtime whose registry + /// entry was lost, a runtime that is no longer usable), whether the registry + /// takes a new entry, and whether the install is a preview or a real run. + /// + /// The rules the rows pin: a choice that names a registered runtime is + /// answered by the registry whatever setup holds. With nothing chosen the + /// ready runtimes are counted, and a setup runtime that cannot be used is not + /// one of them: it does not stop a registered ready runtime from being picked + /// (one) or refused as ambiguous (two), and is reported only when it was the + /// last hope (none registered). A setup runtime that can be used is counted, + /// and a failure to write its entry back is reported as that, whatever else + /// the count would have said. Each row asserts the message and the registry + /// together. + #[test] + fn comfyui_runtime_selection_state_matrix() -> Result<()> { + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum Chosen { + Nothing, + Registered, + SetupRuntime, + } + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum SetupFolder { + Absent, + Usable, + Unusable, + } + struct Expected { + selected: Option<&'static str>, + error_has: Vec, + error_lacks: &'static [&'static str], + setup_entry_written: bool, + } + + const SETUP_KEY: &str = "runtime-s"; + const MULTIPLE: &str = "Multiple ROCm runtimes are ready"; + let registered_keys = ["runtime-a", "runtime-b"]; + let restore_failed = format!("failed to restore setup runtime `{SETUP_KEY}`"); + let mut rows_run = 0; + let mut rows_skipped = 0; + + for chosen in [Chosen::Nothing, Chosen::Registered, Chosen::SetupRuntime] { + for ready in 0..=2usize { + for setup in [ + SetupFolder::Absent, + SetupFolder::Usable, + SetupFolder::Unusable, + ] { + for registry_writable in [true, false] { + for recovery in [SetupRecovery::Persist, SetupRecovery::PreviewOnly] { + if chosen == Chosen::Registered && ready == 0 { + continue; // nothing registered to name + } + let row = format!( + "{chosen:?}, {ready} ready, setup {setup:?}, writable {registry_writable}, {recovery:?}" + ); + let paths = test_paths(&format!( + "comfyui-selection-matrix-{}", + rows_run + rows_skipped + )); + for (index, key) in registered_keys.iter().take(ready).enumerate() { + write_test_pip_runtime( + &paths, + key, + "therock-release:gfx120X-all", + "7.14.0", + 10 + index as u128, + )?; + } + if setup != SetupFolder::Absent { + let manifest = + crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, SETUP_KEY, + )?; + if setup == SetupFolder::Unusable { + fs::remove_file( + manifest + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + } + } + let config = RocmCliConfig::load(&paths)?; + // A registry that cannot take an entry: a plain file for + // an empty one (stops `root` too), a read-only directory + // where it holds runtimes. + let registry_path = crate::runtimes_cmd::runtime_registry_dir(&paths); + let _read_only; + if !registry_writable { + if ready == 0 { + crate::test_support::make_runtime_registry_unwritable(&paths)?; + } else { + let Some(guard) = make_registry_read_only(&paths) else { + rows_skipped += 1; + continue; + }; + _read_only = guard; + } + } + rows_run += 1; + + let persists = recovery == SetupRecovery::Persist; + let setup_usable = setup == SetupFolder::Usable; + let write_fails = persists && !registry_writable && setup_usable; + let writes = persists && registry_writable && setup_usable; + let ready_keys = registered_keys[..ready].to_vec(); + let expected = match chosen { + Chosen::Nothing => match (setup, ready) { + (SetupFolder::Usable, _) if write_fails => Expected { + selected: None, + error_has: vec![restore_failed.clone()], + error_lacks: &["Install ROCm first", MULTIPLE], + setup_entry_written: false, + }, + (SetupFolder::Usable, 0) => Expected { + selected: Some(SETUP_KEY), + error_has: vec![], + error_lacks: &[], + setup_entry_written: writes, + }, + (SetupFolder::Usable, _) => Expected { + selected: None, + error_has: std::iter::once(MULTIPLE.to_owned()) + .chain(ready_keys.iter().map(|key| (*key).to_owned())) + .chain([SETUP_KEY.to_owned()]) + .collect(), + error_lacks: &["Install ROCm first"], + setup_entry_written: writes, + }, + (SetupFolder::Unusable, 0) => Expected { + selected: None, + error_has: vec![ + format!("setup runtime `{SETUP_KEY}`"), + "is not usable".to_owned(), + ], + error_lacks: &["Install ROCm first"], + setup_entry_written: false, + }, + (SetupFolder::Absent, 0) => Expected { + selected: None, + error_has: vec!["Install ROCm first".to_owned()], + error_lacks: &["is not usable"], + setup_entry_written: false, + }, + (_, 1) => Expected { + selected: Some("runtime-a"), + error_has: vec![], + error_lacks: &[], + setup_entry_written: false, + }, + (_, _) => Expected { + selected: None, + error_has: std::iter::once(MULTIPLE.to_owned()) + .chain(ready_keys.iter().map(|key| (*key).to_owned())) + .collect(), + error_lacks: &["is not usable", SETUP_KEY], + setup_entry_written: false, + }, + }, + Chosen::Registered => Expected { + selected: Some("runtime-a"), + error_has: vec![], + error_lacks: &[], + setup_entry_written: false, + }, + Chosen::SetupRuntime => match setup { + SetupFolder::Usable if write_fails => Expected { + selected: None, + error_has: vec![restore_failed.clone()], + error_lacks: &["Install ROCm first", "not found"], + setup_entry_written: false, + }, + SetupFolder::Usable => Expected { + selected: Some(SETUP_KEY), + error_has: vec![], + error_lacks: &[], + setup_entry_written: writes, + }, + SetupFolder::Absent | SetupFolder::Unusable => Expected { + selected: None, + error_has: vec![if ready == 0 { + "Install ROCm first".to_owned() + } else { + format!("ROCm runtime not found: `{SETUP_KEY}`") + }], + error_lacks: &["is not usable"], + setup_entry_written: false, + }, + }, + }; + + let selector = match chosen { + Chosen::Nothing => None, + Chosen::Registered => Some("runtime-a"), + Chosen::SetupRuntime => Some(SETUP_KEY), + }; + let outcome = select_runtime(&paths, &config, selector, recovery); + + match (outcome, expected.selected) { + (Ok(selected), Some(key)) => { + assert_eq!(selected.manifest.runtime_key, key, "{row}"); + } + (Ok(selected), None) => panic!( + "{row}: expected a refusal, picked {}", + selected.manifest.runtime_key + ), + (Err(error), Some(key)) => { + panic!("{row}: expected {key}, got: {error:#}") + } + (Err(error), None) => { + let message = format!("{error:#}"); + for needle in &expected.error_has { + assert!( + message.contains(needle.as_str()), + "{row}: `{needle}` missing from: {message}" + ); + } + for needle in expected.error_lacks { + assert!( + !message.contains(needle), + "{row}: `{needle}` must not appear in: {message}" + ); + } + } + } + + let mut entries: Vec = + ready_keys.iter().map(|key| format!("{key}.json")).collect(); + if expected.setup_entry_written { + entries.push(format!("{SETUP_KEY}.json")); + } + entries.sort(); + assert_eq!(runtime_registry_entry_names(&paths), entries, "{row}"); + if !registry_writable && ready == 0 { + assert_eq!( + fs::read_to_string(®istry_path)?, + "not a directory", + "{row}: a failed restore leaves the registry as it was" + ); + } + } + } + } + } + } + // The rows that need a registry holding runtimes that refuses a write skip + // themselves where the process can write anyway; the rest must have run. + assert_eq!(rows_run + rows_skipped, 96, "the matrix has 96 rows"); + assert!(rows_run >= 60, "ran {rows_run}, skipped {rows_skipped}"); + Ok(()) + } + + /// The two paths that pick "the one ready runtime" must not drift apart: + /// for each registry state below, ComfyUI selects a runtime exactly when + /// the engine paths do, and the same one. Both reach the rule through + /// `select_single_ready_runtime`, so a copy of it re-introduced in either + /// caller (a different readiness test, a missing setup recovery) shows up as + /// a disagreement on one of these states. + #[test] + fn comfyui_and_engine_paths_agree_on_the_single_ready_runtime() -> Result<()> { + use crate::engine_runtime_select::{ + resolve_engine_selection, validate_engine_selection_runtime, + }; + + #[derive(Clone, Copy)] + enum State { + Empty, + OneUnusable, + OneReady, + TwoReady, + ReadyAndUnusable, + SetupRuntimeLost, + SetupRuntimeLostBesideAReadyOne, + UnusableSetupRuntimeBesideAReadyOne, + } + for (label, state, expected) in [ + ("empty", State::Empty, None), + ("one-unusable", State::OneUnusable, None), + ("one-ready", State::OneReady, Some("runtime-a")), + ("two-ready", State::TwoReady, None), + ( + "ready-and-unusable", + State::ReadyAndUnusable, + Some("runtime-a"), + ), + ( + "setup-runtime-lost", + State::SetupRuntimeLost, + Some("runtime-s"), + ), + ( + "setup-runtime-lost-beside-a-ready-one", + State::SetupRuntimeLostBesideAReadyOne, + None, + ), + ( + "unusable-setup-runtime-beside-a-ready-one", + State::UnusableSetupRuntimeBesideAReadyOne, + Some("runtime-a"), + ), + ] { + // Each path gets its own copy of the state: a recovery run by one + // writes the registry, which would otherwise hand the other path a + // state it could not have found for itself. + let plant_state = |paths: &AppPaths| -> Result<()> { + let plant = |key: &str| { + write_test_pip_runtime(paths, key, "therock-release:gfx120X-all", "7.14.0", 20) + }; + let break_runtime = |manifest: &therock::InstalledRuntimeManifest| -> Result<()> { + fs::remove_file( + manifest + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + Ok(()) + }; + match state { + State::Empty => {} + State::OneUnusable => break_runtime(&plant("runtime-a")?)?, + State::OneReady => { + plant("runtime-a")?; + } + State::TwoReady => { + plant("runtime-a")?; + plant("runtime-b")?; + } + State::ReadyAndUnusable => { + plant("runtime-a")?; + break_runtime(&plant("runtime-b")?)?; + } + State::SetupRuntimeLost => { + crate::test_support::write_test_setup_runtime_missing_from_registry( + paths, + "runtime-s", + )?; + } + State::SetupRuntimeLostBesideAReadyOne => { + plant("runtime-a")?; + crate::test_support::write_test_setup_runtime_missing_from_registry( + paths, + "runtime-s", + )?; + } + State::UnusableSetupRuntimeBesideAReadyOne => { + plant("runtime-a")?; + break_runtime( + &crate::test_support::write_test_setup_runtime_missing_from_registry( + paths, + "runtime-s", + )?, + )?; + } + } + Ok(()) + }; + let comfyui_paths = test_paths(&format!("comfyui-engine-agree-{label}-comfyui")); + plant_state(&comfyui_paths)?; + let comfyui = persisting_select_runtime( + &comfyui_paths, + &RocmCliConfig::load(&comfyui_paths)?, + None, + ) + .ok() + .map(|selected| selected.manifest.runtime_key); + let engine_paths = test_paths(&format!("comfyui-engine-agree-{label}-engine")); + plant_state(&engine_paths)?; + let engine = validate_engine_selection_runtime( + &engine_paths, + resolve_engine_selection(&RocmCliConfig::load(&engine_paths)?, "vllm", None, None), + )? + .runtime_id; + + assert_eq!(comfyui, engine, "{label}: the two paths disagree"); + assert_eq!(comfyui.as_deref(), expected, "{label}"); + } + Ok(()) + } + fn command_env_value(command: &Command, key: &str) -> Option { command .get_envs() .find(|(name, _)| name.to_string_lossy() == key) .and_then(|(_, value)| value.map(OsString::from)) } + + /// The environment folder ComfyUI exports as `VIRTUAL_ENV`, by the folder + /// the interpreter sits in. Today only a `bin` or `Scripts` folder + /// (any case) identifies a venv; anything else leaves `VIRTUAL_ENV` unset + /// rather than pointing it at a guessed folder. + #[test] + fn runtime_environment_exports_virtual_env_only_for_a_bin_or_scripts_folder() -> Result<()> { + let cases: [(&str, Option<&str>); 7] = [ + ("/opt/venv/bin/python", Some("/opt/venv")), + ("/opt/venv/Scripts/python.exe", Some("/opt/venv")), + ("/opt/venv/SCRIPTS/python.exe", Some("/opt/venv")), + ("/opt/venv/BIN/python", Some("/opt/venv")), + ("/opt/venv/tools/python", None), + ("python", None), + ("/python", None), + ]; + for (python, expected) in cases { + let python = PathBuf::from(python); + let manifest = minimal_runtime_manifest(&python); + + let env = runtime_environment_from_runtime(&manifest, &python); + let mut command = Command::new(&python); + apply_runtime_environment(&mut command, &env)?; + + assert_eq!( + command_env_value(&command, "VIRTUAL_ENV").as_deref(), + expected.map(std::ffi::OsStr::new), + "{}", + python.display() + ); + } + Ok(()) + } + + fn minimal_runtime_manifest(python: &Path) -> therock::InstalledRuntimeManifest { + therock::InstalledRuntimeManifest { + runtime_key: "therock-release-gfx120x-all".to_owned(), + runtime_id: "therock-release:gfx120X-all".to_owned(), + channel: "release".to_owned(), + format: "wheel".to_owned(), + family: "gfx120X-all".to_owned(), + family_source: "test".to_owned(), + version: "7.13.0a20260511".to_owned(), + install_root: PathBuf::from("/nonexistent/runtime"), + selected_artifact_url: "https://example.invalid/simple".to_owned(), + source_layout_generation: None, + index_url: None, + tarball_file_name: None, + python_launcher: None, + python_executable: Some(python.display().to_string()), + pip_cache_dir: None, + rocm_sdk: None, + sdk_torch: None, + wheel_composition: None, + read_only: false, + imported_from: None, + devel: true, + installed_at_unix_ms: 100, + } + } } diff --git a/apps/rocm/src/driver_install.rs b/apps/rocm/src/driver_install.rs index 3c2a73f942..59d6cb7bef 100644 --- a/apps/rocm/src/driver_install.rs +++ b/apps/rocm/src/driver_install.rs @@ -37,7 +37,6 @@ use crate::driver_plan::{ DriverCommandPhase, DriverInstallPlan, PrivilegeEscalation, build_driver_install_plan_read, render_driver_install_plan, }; -use crate::system_packages::real_os_release; /// Whether `rocm install driver` goes on to run the plan's commands. Pure so /// the gate that keeps native Windows from installing anything (its plan is @@ -62,7 +61,7 @@ pub(crate) fn install_driver( ) -> std::result::Result { let examine = ExamineSummary::gather().map_err(|source| DriverInstallError::new(source, false))?; - let os_release = real_os_release(); + let os_release = rocm_core::hardware_root::real_os_release(); // The only place the real privilege level is read; every builder below takes // it as a parameter so both branches stay testable on any host. let plan = diff --git a/apps/rocm/src/driver_plan.rs b/apps/rocm/src/driver_plan.rs index 0c71110f07..44cd8810e5 100644 --- a/apps/rocm/src/driver_plan.rs +++ b/apps/rocm/src/driver_plan.rs @@ -1445,9 +1445,9 @@ pub(crate) mod tests { } /// `build_driver_install_plan_read` is the one seam `install_driver` hands - /// its `real_os_release()` result to. Both are now one-line delegations - /// to `hardware_root::read_os_release`, so reverting either back to a raw - /// `/etc/os-release` read is a type error here, not a silent regression. + /// `hardware_root::real_os_release()`'s result to, so reverting that back to + /// a raw `/etc/os-release` read is a type error here, not a silent + /// regression. /// This still pins the behaviour at the seam itself: an `OsRelease` that /// says the fallback path was read plans exactly what a primary-path read /// with the same content would. diff --git a/apps/rocm/src/engine_runtime_select.rs b/apps/rocm/src/engine_runtime_select.rs index e0c744891f..73589e7967 100644 --- a/apps/rocm/src/engine_runtime_select.rs +++ b/apps/rocm/src/engine_runtime_select.rs @@ -57,7 +57,10 @@ //! `resolve_runtime_selector_to_exact_key` is also `pub(crate)`: `engines_cmd.rs` //! calls it in production, and `runtimes_cmd.rs`'s own test module calls it //! directly too. `single_ready_runtime_key` is called only from -//! `validate_engine_selection_runtime` in this same module and stays private. +//! `validate_engine_selection_runtime` in this same module and stays private; +//! it does not decide which runtime counts as "the one ready runtime" itself, +//! it asks `runtime_adopt::select_single_ready_runtime`, the rule +//! `rocm comfyui install` shares. //! `optional_arg` and //! `engine_recipe_json_arg` are `pub(crate)`, read from `engine_ipc.rs`'s //! `engine_request`; `parse_engine_recipe_json_arg` is `pub(crate)`, read @@ -103,10 +106,8 @@ use rocm_core::{ }; use rocm_engine_protocol::EngineRecipeHint; -use crate::runtimes_cmd::{ - recover_setup_runtime_registration, select_runtime_manifest, - validate_runtime_manifest_for_activation, -}; +use crate::runtime_adopt::{SetupRecovery, SingleReadyRuntime, select_single_ready_runtime}; +use crate::runtimes_cmd::{recover_setup_runtime_registration, select_runtime_manifest}; use crate::therock; #[derive(Debug, Clone)] @@ -211,16 +212,14 @@ pub(crate) fn validate_engine_selection_runtime( fn single_ready_runtime_key(paths: &AppPaths) -> Result> { let config = RocmCliConfig::load(paths).unwrap_or_default(); - recover_setup_runtime_registration(paths, &config)?; - let manifests = therock::load_runtime_manifests(paths)?; - let ready = manifests - .iter() - .filter(|manifest| validate_runtime_manifest_for_activation(manifest).is_ok()) - .collect::>(); - Ok(match ready.as_slice() { - [manifest] => Some(manifest.runtime_key.clone()), - _ => None, - }) + // Zero or several ready runtimes leave the selection open: the caller that + // needs a runtime says so in its own words. + Ok( + match select_single_ready_runtime(paths, &config, SetupRecovery::Persist)? { + SingleReadyRuntime::One(manifest) => Some(manifest.runtime_key), + SingleReadyRuntime::Zero { .. } | SingleReadyRuntime::Many { .. } => None, + }, + ) } pub(crate) fn resolve_runtime_selector_to_exact_key( @@ -611,6 +610,7 @@ mod tests { use serde_json::json; use super::*; + use crate::runtime_registry::validate_runtime_manifest_for_activation; use crate::status_for_readiness; use crate::test_support::write_test_pip_runtime; use crate::tests::{test_app_paths, test_paths}; @@ -1048,4 +1048,137 @@ mod tests { let _ = std::fs::remove_dir_all(root); Ok(()) } + + /// Zero ready runtimes, the registry empty: the engine path answers with no + /// selection rather than an error, and writes nothing. The caller reports + /// the missing runtime in its own words (`engines_cmd.rs` and `serve_cmd.rs`). + #[test] + fn engine_selection_with_no_registered_runtime_stays_unselected_and_writes_nothing() + -> Result<()> { + let (root, paths) = test_paths("no-registered-runtime-selection"); + let selection = validate_engine_selection_runtime( + &paths, + resolve_engine_selection(&RocmCliConfig::default(), "vllm", None, None), + )?; + + assert!(selection.runtime_id.is_none()); + assert!(selection.env_id.is_none()); + assert!(selection.source.is_none()); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + Vec::::new() + ); + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// Zero ready runtimes, one registered but unusable: still no selection. + /// Readiness is `validate_runtime_manifest_for_activation`, so a runtime + /// whose interpreter is gone does not count, and it is not picked as "the + /// only one" either. + #[test] + fn engine_selection_does_not_count_an_unusable_runtime_as_ready() -> Result<()> { + let (root, paths) = test_paths("unusable-runtime-selection"); + let manifest = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-14-0", + "therock-release:gfx120X-all", + "7.14.0", + 20, + )?; + let python = manifest + .python_executable + .as_deref() + .expect("the fixture records an interpreter"); + std::fs::remove_file(python)?; + assert!(validate_runtime_manifest_for_activation(&manifest).is_err()); + + let selection = validate_engine_selection_runtime( + &paths, + resolve_engine_selection(&RocmCliConfig::default(), "vllm", None, None), + )?; + + assert!(selection.runtime_id.is_none()); + assert!(selection.source.is_none()); + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// The setup-recovery half of the rule: a ready runtime that only + /// `recover_setup_runtime_registration` can make visible is found by the + /// engine path, and finding it re-files its registry entry, so the message + /// and the state are one fact (the selection names a key that + /// `rocm runtimes list` now shows). + #[test] + fn engine_selection_recovers_the_setup_runtime_the_registry_lost() -> Result<()> { + let (root, paths) = test_paths("setup-recovery-selection"); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + Vec::::new(), + "the registry must start without the runtime for this test to mean anything" + ); + + let selection = validate_engine_selection_runtime( + &paths, + resolve_engine_selection(&RocmCliConfig::default(), "vllm", None, None), + )?; + + assert_eq!( + selection.runtime_id.as_deref(), + Some(manifest.runtime_key.as_str()) + ); + assert_eq!(selection.source.as_deref(), Some("single_ready_runtime")); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + vec![format!("{}.json", manifest.runtime_key)] + ); + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// Recovering the setup runtime writes its registry entry, and a registry + /// that cannot take it is reported as that on every path that recovers: + /// an explicit `--runtime-id` and the one-ready-runtime default (which + /// `serve` and `engines install` both reach). Neither may answer as if there + /// were no runtime, or carry on without one: the runtime is there, and the + /// failure is what stops the command. The error and the registry are + /// asserted together. + #[test] + fn engine_selection_reports_a_registry_that_cannot_take_the_setup_runtime_back() -> Result<()> { + for explicit in [true, false] { + let (root, paths) = test_paths("setup-recovery-unwritable-registry"); + let manifest = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "release-pip-gfx120x-all-7-14-0", + )?; + let registry = crate::test_support::make_runtime_registry_unwritable(&paths)?; + let runtime_id = explicit.then_some(manifest.runtime_key.as_str()); + + let error = validate_engine_selection_runtime( + &paths, + resolve_engine_selection(&RocmCliConfig::load(&paths)?, "vllm", runtime_id, None), + ) + .expect_err("a registry that cannot be written cannot take the entry back"); + + let message = format!("{error:#}"); + assert!( + message.contains(&format!( + "failed to restore setup runtime `{}`", + manifest.runtime_key + )), + "explicit: {explicit}, got: {message}" + ); + assert_eq!( + std::fs::read_to_string(®istry)?, + "not a directory", + "explicit: {explicit}: a failed restore leaves the registry as it was" + ); + let _ = std::fs::remove_dir_all(root); + } + Ok(()) + } } diff --git a/apps/rocm/src/engines_cmd.rs b/apps/rocm/src/engines_cmd.rs index 84301ee13f..eaee862b45 100644 --- a/apps/rocm/src/engines_cmd.rs +++ b/apps/rocm/src/engines_cmd.rs @@ -835,4 +835,72 @@ mod tests { ); } } + + /// Several ready runtimes and none chosen: the engine path does not guess. + /// `resolve_engine_env` reports the gap in its own words, and each remedy + /// those words name must clear it: `rocm runtimes activate ` + /// (run here as `activate_runtime`, its implementation) and `--runtime-id`. + #[test] + fn engine_install_with_several_ready_runtimes_names_remedies_that_clear_it() -> Result<()> { + let (root, paths) = test_paths("engine-install-several-ready-runtimes"); + let first = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 10, + )?; + write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-14-0", + "therock-release:gfx120X-all", + "7.14.0", + 20, + )?; + let mut config = RocmCliConfig::default(); + + let message = resolve_engine_env(&paths, &config, "vllm", None, None) + .expect_err("several ready runtimes and no choice must not resolve") + .to_string(); + assert!( + message.contains("no active ROCm runtime is configured") + && message.contains("rocm runtimes list") + && message.contains("rocm runtimes activate ") + && message.contains("--runtime-id"), + "unexpected refusal: {message}" + ); + + // `rocm runtimes activate `: the choice is now made. + crate::runtimes_cmd::activate_runtime(&paths, &mut config, &first.runtime_key)?; + let selection = validate_engine_selection_runtime( + &paths, + resolve_engine_selection(&config, "vllm", None, None), + )?; + assert_eq!( + selection.runtime_id.as_deref(), + Some(first.runtime_key.as_str()) + ); + assert_eq!( + selection.source.as_deref(), + Some("config_active_runtime_key") + ); + + // `--runtime-id `: the same choice, for one invocation. + let selection = validate_engine_selection_runtime( + &paths, + resolve_engine_selection( + &RocmCliConfig::default(), + "vllm", + Some(&first.runtime_key), + None, + ), + )?; + assert_eq!( + selection.runtime_id.as_deref(), + Some(first.runtime_key.as_str()) + ); + assert_eq!(selection.source.as_deref(), Some("cli_runtime_id")); + let _ = fs::remove_dir_all(root); + Ok(()) + } } diff --git a/apps/rocm/src/runtime_adopt.rs b/apps/rocm/src/runtime_adopt.rs index 549753818a..18e594f44b 100644 --- a/apps/rocm/src/runtime_adopt.rs +++ b/apps/rocm/src/runtime_adopt.rs @@ -5,9 +5,13 @@ //! Registering a runtime that was not installed by `rocm install sdk`: //! importing a manifest written elsewhere, adopting an existing Python //! environment by probing it, and recovering the registration of the runtime -//! `rocm setup` created. +//! `rocm setup` created. It also holds the one rule for picking "the one ready +//! runtime" when nothing else says which to use +//! ([`select_single_ready_runtime`]), which sits here because it recovers the +//! setup registration first; `rocm comfyui install` and the engine paths both +//! ask it and each words the zero and several cases itself. //! -//! Relocated from `runtimes_cmd.rs` with no behavior change. `runtimes_cmd.rs` +//! The rest was relocated from `runtimes_cmd.rs` with no behavior change. `runtimes_cmd.rs` //! calls into this module for `rocm runtimes import` and `rocm runtimes adopt`, //! and re-exports `newest_installed_runtime_manifest`, //! `recover_setup_runtime_registration` and `setup_runtime_paths` for the @@ -19,11 +23,15 @@ use std::fs; use std::path::{Path, PathBuf}; use anyhow::{Context, Result, bail}; -use rocm_core::{AppPaths, RocmCliConfig, managed_pip_cache_dir, runtime_python_executable_in_env}; +use rocm_core::{ + AppPaths, RocmCliConfig, managed_pip_cache_dir, runtime_python_env_root_for_executable, + runtime_python_executable_in_env, +}; use crate::runtime_registry::{ - paths_equivalent, runtime_manifest_path, runtime_registry_dir, - validate_runtime_manifest_for_activation, + ManifestOverwrite, paths_equivalent, runtime_manifest_path, runtime_registry_dir, + sort_manifests_newest_install_first, validate_runtime_manifest_for_activation, + write_runtime_manifest, }; use crate::therock; @@ -58,7 +66,11 @@ pub(crate) fn import_runtime_manifest( validate_runtime_manifest_for_activation(&manifest) .with_context(|| format!("imported runtime `{}` is not usable", manifest.runtime_key))?; - write_runtime_registry_manifest(paths, &manifest, replace)?; + write_runtime_manifest( + paths, + &manifest, + ManifestOverwrite::for_replace_flag(replace), + )?; Ok(manifest) } @@ -230,7 +242,9 @@ fn resolve_adopt_python_input(input: &Path) -> Result<(PathBuf, Option) return Ok((python, Some(env_root))); } if absolute.is_file() { - let inferred_root = infer_python_env_root(&absolute); + // `None` for an interpreter outside a `bin`/`Scripts` folder: the + // request then asks for `--root` rather than adopting a guessed folder. + let inferred_root = runtime_python_env_root_for_executable(&absolute); return Ok((absolute, inferred_root)); } bail!( @@ -239,15 +253,6 @@ fn resolve_adopt_python_input(input: &Path) -> Result<(PathBuf, Option) ); } -fn infer_python_env_root(python_executable: &Path) -> Option { - let bin_dir = python_executable.parent()?; - let bin_name = bin_dir.file_name()?.to_string_lossy(); - if bin_name.eq_ignore_ascii_case("Scripts") || bin_name == "bin" { - return bin_dir.parent().map(Path::to_path_buf); - } - bin_dir.parent().map(Path::to_path_buf) -} - pub(crate) fn adopt_runtime_from_probe( paths: &AppPaths, request: AdoptRuntimeRequest, @@ -314,7 +319,11 @@ pub(crate) fn adopt_runtime_from_probe( }; validate_runtime_manifest_for_activation(&manifest) .with_context(|| format!("adopted runtime `{}` is not usable", manifest.runtime_key))?; - write_runtime_registry_manifest(paths, &manifest, request.replace)?; + write_runtime_manifest( + paths, + &manifest, + ManifestOverwrite::for_replace_flag(request.replace), + )?; Ok(manifest) } @@ -350,37 +359,17 @@ fn parse_therock_runtime_id(runtime_id: &str) -> Result<(String, String)> { Ok((channel.to_owned(), family.to_owned())) } -fn write_runtime_registry_manifest( - paths: &AppPaths, - manifest: &therock::InstalledRuntimeManifest, - replace: bool, -) -> Result<()> { - let manifest = manifest.clone().normalize_storage_paths(); - let registry_path = runtime_manifest_path(paths, &manifest.runtime_key); - if registry_path.exists() && !replace { - bail!( - "runtime registry entry already exists: {}; pass --replace to overwrite it", - registry_path.display() - ); - } - fs::create_dir_all( - registry_path - .parent() - .context("runtime registry path has no parent directory")?, - )?; - fs::write( - ®istry_path, - serde_json::to_vec_pretty(&manifest) - .context("failed to serialize runtime registry manifest")?, - ) - .with_context(|| format!("failed to write {}", registry_path.display()))?; - Ok(()) +/// The runtime `rocm setup` created, as the manifest in its install tree +/// describes it, and whether the runtime registry already holds its entry. +struct SetupRuntime { + manifest: therock::InstalledRuntimeManifest, + registered: bool, } -pub(crate) fn recover_setup_runtime_registration( - paths: &AppPaths, - config: &RocmCliConfig, -) -> Result> { +/// Find the runtime `rocm setup` created and check that it is usable, without +/// changing anything. `None` when setup named no folder or the folder holds no +/// manifest; an error when it does and the runtime cannot be used. +fn find_setup_runtime(paths: &AppPaths, config: &RocmCliConfig) -> Result> { let Some(setup_root) = config .setup .therock_venv @@ -424,16 +413,190 @@ pub(crate) fn recover_setup_runtime_registration( ) })?; - if !runtime_manifest_path(paths, &manifest.runtime_key).is_file() { - write_runtime_registry_manifest(paths, &manifest, false).with_context(|| { - format!( - "failed to restore setup runtime `{}` into {}", - manifest.runtime_key, - runtime_registry_dir(paths).display() - ) - })?; + let registered = runtime_manifest_path(paths, &manifest.runtime_key).is_file(); + Ok(Some(SetupRuntime { + manifest, + registered, + })) +} + +/// Write the registry entry of a setup runtime that the registry has lost, and +/// nothing when it is there. The one place that write and its error message live. +fn restore_setup_registration(paths: &AppPaths, setup: &SetupRuntime) -> Result<()> { + if !setup.registered { + write_runtime_manifest(paths, &setup.manifest, ManifestOverwrite::Refuse).with_context( + || { + format!( + "failed to restore setup runtime `{}` into {}", + setup.manifest.runtime_key, + runtime_registry_dir(paths).display() + ) + }, + )?; + } + Ok(()) +} + +pub(crate) fn recover_setup_runtime_registration( + paths: &AppPaths, + config: &RocmCliConfig, +) -> Result> { + let Some(setup) = find_setup_runtime(paths, config)? else { + return Ok(None); + }; + restore_setup_registration(paths, &setup)?; + Ok(Some(setup.manifest.runtime_key)) +} + +/// Whether finding the runtime `rocm setup` created may write its lost registry +/// entry back while a runtime is being picked. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum SetupRecovery { + /// Write the entry back, as `rocm serve` and `rocm engines install` do. + Persist, + /// Count the setup runtime as if its entry were there, and write nothing: + /// a `--dry-run` preview must name the runtime the real run will use + /// without changing any file. + PreviewOnly, +} + +impl SetupRecovery { + /// The recovery a command with a `--dry-run` flag uses. + pub(crate) const fn for_dry_run(dry_run: bool) -> Self { + if dry_run { + Self::PreviewOnly + } else { + Self::Persist + } + } +} + +/// What picking "the one ready runtime" found. The caller decides how to report +/// the two cases that are not a pick, because the advice differs by command +/// (`rocm comfyui install` and `rocm serve` name different remedies). +#[derive(Debug)] +pub(crate) enum SingleReadyRuntime { + /// Exactly one registered runtime is ready. + One(Box), + /// No registered runtime is ready. `registered` is everything the registry + /// holds, unusable entries included and possibly none, so a caller can say + /// why each one does not count. + Zero { + registered: Vec, + }, + /// More than one is ready, so there is nothing to pick without a choice. + Many { + ready: Vec, + }, +} + +/// Pick the one ready runtime when nothing else says which to use: no +/// `--runtime-id`, no active runtime, no configured default. This is the only +/// place that rule lives; `rocm comfyui install` and the engine paths +/// (`rocm serve`, `rocm engines install`) both call it. +/// +/// Ready means `validate_runtime_manifest_for_activation` accepts the runtime, +/// the same check `rocm runtimes activate` applies. Setup is recovered first +/// (`recover_setup_runtime_registration`), so the runtime `rocm setup` created +/// counts even if its registry entry was lost. With [`SetupRecovery::Persist`] +/// that writes the entry back; with [`SetupRecovery::PreviewOnly`] nothing is +/// written, and the setup runtime is counted from its own manifest as the +/// registry would hold it once restored, so the answer is the one `Persist` +/// gives. +/// +/// A setup runtime that cannot be used (its manifest is unreadable or points +/// elsewhere, or it fails the readiness check) is not a ready runtime and +/// cannot change the count, so it does not stop a registered ready runtime from +/// being picked, or from being refused as ambiguous with another. That is the +/// answer a choice that names a runtime gets from the registry too. When the +/// registry holds no ready runtime the setup runtime was the only candidate and +/// its error is the answer. A setup runtime that can be used but whose registry +/// entry cannot be written is different: it is one of the runtimes being +/// counted, so the failed write is reported as it is. +pub(crate) fn select_single_ready_runtime( + paths: &AppPaths, + config: &RocmCliConfig, + recovery: SetupRecovery, +) -> Result { + let setup = match find_setup_runtime(paths, config) { + Ok(setup) => setup, + Err(unusable) => { + return match classify_single_ready_runtime(therock::load_runtime_manifests(paths)?) { + SingleReadyRuntime::Zero { .. } => Err(unusable), + answer => Ok(answer), + }; + } + }; + Ok(classify_single_ready_runtime(registered_runtimes_with( + paths, setup, recovery, + )?)) +} + +/// Every runtime the registry holds, with the one `rocm setup` created counted +/// even if its registry entry was lost. With [`SetupRecovery::Persist`] the +/// entry is written back first (`recover_setup_runtime_registration`); with +/// [`SetupRecovery::PreviewOnly`] nothing is written, and the setup runtime is +/// added from its own manifest as the registry would hold it once restored, so +/// both return the same list. An error when the setup runtime exists but +/// cannot be used. +pub(crate) fn registered_runtimes_with_setup( + paths: &AppPaths, + config: &RocmCliConfig, + recovery: SetupRecovery, +) -> Result> { + registered_runtimes_with(paths, find_setup_runtime(paths, config)?, recovery) +} + +/// [`registered_runtimes_with_setup`] for a setup runtime already found. +fn registered_runtimes_with( + paths: &AppPaths, + setup: Option, + recovery: SetupRecovery, +) -> Result> { + match recovery { + SetupRecovery::Persist => { + if let Some(setup) = &setup { + restore_setup_registration(paths, setup)?; + } + therock::load_runtime_manifests(paths) + } + SetupRecovery::PreviewOnly => { + let mut registered = therock::load_runtime_manifests(paths)?; + if let Some(setup) = setup + && !setup.registered + { + // What `load_runtime_manifests` would return for the entry that + // `Persist` would have written. + registered.push( + setup + .manifest + .normalize_storage_paths() + .normalize_host_paths(), + ); + sort_manifests_newest_install_first(&mut registered); + } + Ok(registered) + } + } +} + +/// The pure half of [`select_single_ready_runtime`]: which of `registered` are +/// ready, and whether that is none, one or several. +pub(crate) fn classify_single_ready_runtime( + registered: Vec, +) -> SingleReadyRuntime { + let ready: Vec<_> = registered + .iter() + .filter(|manifest| validate_runtime_manifest_for_activation(manifest).is_ok()) + .cloned() + .collect(); + if ready.is_empty() { + return SingleReadyRuntime::Zero { registered }; + } + match <[therock::InstalledRuntimeManifest; 1]>::try_from(ready) { + Ok([manifest]) => SingleReadyRuntime::One(Box::new(manifest)), + Err(ready) => SingleReadyRuntime::Many { ready }, } - Ok(Some(manifest.runtime_key)) } /// The folder setup was configured for and its pip cache — the pair both @@ -819,4 +982,387 @@ mod tests { let _ = fs::remove_dir_all(root); Ok(()) } + + /// `rocm runtimes adopt --python ` without `--root` takes the + /// environment folder from the shared rule (see + /// `rocm_core::runtime_python_env_root_for_executable`): only an + /// interpreter in a `bin` or `Scripts` folder, in any case, names one. An + /// interpreter in an unrelated folder yields no root, and the request is + /// then refused with `--root` guidance rather than adopting a guessed + /// folder. + #[test] + fn adopt_python_file_infers_a_root_only_from_a_bin_or_scripts_folder() -> Result<()> { + let (root, _) = test_paths("runtime-adopt-infer-root"); + for (folder, expected_root) in [ + ("bin", true), + ("Scripts", true), + ("SCRIPTS", true), + ("tools", false), + ] { + let env_root = root.join(format!("env-{folder}")); + let python = env_root.join(folder).join("python"); + fs::create_dir_all(python.parent().unwrap())?; + fs::write(&python, "python")?; + + let (resolved, inferred) = resolve_adopt_python_input(&python)?; + + assert_eq!(resolved, python, "{folder}"); + assert_eq!( + inferred.as_deref(), + expected_root.then_some(env_root.as_path()), + "{folder}" + ); + } + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + fn pip_runtime(paths: &AppPaths, key: &str) -> Result { + write_test_pip_runtime(paths, key, "therock-release:gfx120X-all", "7.14.0", 20) + } + + /// Readiness is the activation check and nothing looser: a runtime whose + /// interpreter is gone is registered but does not count, and neither is it + /// "the only one". + #[test] + fn single_ready_runtime_counts_only_what_activation_accepts() -> Result<()> { + let (root, paths) = test_paths("single-ready-classify"); + let ready = pip_runtime(&paths, "runtime-a")?; + let broken = pip_runtime(&paths, "runtime-b")?; + fs::remove_file( + broken + .python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + assert!(validate_runtime_manifest_for_activation(&ready).is_ok()); + assert!(validate_runtime_manifest_for_activation(&broken).is_err()); + + match classify_single_ready_runtime(vec![ready.clone(), broken.clone()]) { + SingleReadyRuntime::One(manifest) => assert_eq!(manifest.runtime_key, "runtime-a"), + other => panic!("expected the one ready runtime, got {other:?}"), + } + match classify_single_ready_runtime(vec![broken.clone()]) { + SingleReadyRuntime::Zero { registered } => { + assert_eq!(registered.len(), 1, "unusable entries stay listed"); + } + other => panic!("expected no ready runtime, got {other:?}"), + } + match classify_single_ready_runtime(Vec::new()) { + SingleReadyRuntime::Zero { registered } => assert!(registered.is_empty()), + other => panic!("expected no ready runtime, got {other:?}"), + } + let second = pip_runtime(&paths, "runtime-c")?; + match classify_single_ready_runtime(vec![ready, broken, second]) { + SingleReadyRuntime::Many { ready } => { + let keys: Vec<_> = ready.iter().map(|m| m.runtime_key.as_str()).collect(); + assert_eq!(keys, ["runtime-a", "runtime-c"], "only ready ones listed"); + } + other => panic!("expected several ready runtimes, got {other:?}"), + } + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// What no inferred root costs the user: the request is refused and asks + /// for `--root`, and an explicit `--root` still adopts the same + /// interpreter. + #[test] + fn adopt_request_without_an_inferred_root_asks_for_root() { + let python = PathBuf::from("python"); + let request = |install_root: Option| { + infer_adopt_runtime_request( + python.clone(), + install_root, + None, + None, + None, + false, + &therock::RocmSdkPythonProbe { + rocm_sdk_version: Some("7.13.0".to_owned()), + resolved_target_family: Some("gfx120X-all".to_owned()), + ..therock::RocmSdkPythonProbe::default() + }, + ) + }; + + let error = request(None).unwrap_err().to_string(); + assert!(error.contains("pass --root"), "{error}"); + + let explicit = request(Some(PathBuf::from("/opt/rocm-env"))).unwrap(); + assert_eq!(explicit.install_root, PathBuf::from("/opt/rocm-env")); + } + + /// The selection recovers the setup runtime first, so a runtime the registry + /// lost is the one found, and finding it is what writes its entry back. + #[test] + fn select_single_ready_runtime_recovers_setup_before_counting() -> Result<()> { + let (root, paths) = test_paths("single-ready-recovers-setup"); + let lost = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "runtime-s", + )?; + let config = RocmCliConfig::load(&paths)?; + assert!(crate::test_support::runtime_registry_entry_names(&paths).is_empty()); + + match select_single_ready_runtime(&paths, &config, SetupRecovery::Persist)? { + SingleReadyRuntime::One(manifest) => assert_eq!(manifest.runtime_key, "runtime-s"), + other => panic!("expected the recovered setup runtime, got {other:?}"), + } + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + vec![format!("{}.json", lost.runtime_key)] + ); + + // Without setup naming a folder there is nothing to recover and the + // registry is not touched. + let (other_root, other_paths) = test_paths("single-ready-no-setup"); + match select_single_ready_runtime( + &other_paths, + &RocmCliConfig::default(), + SetupRecovery::Persist, + )? { + SingleReadyRuntime::Zero { registered } => assert!(registered.is_empty()), + other => panic!("expected no ready runtime, got {other:?}"), + } + assert!(crate::test_support::runtime_registry_entry_names(&other_paths).is_empty()); + let _ = fs::remove_dir_all(root); + let _ = fs::remove_dir_all(other_root); + Ok(()) + } + + /// The keys `select_single_ready_runtime` found ready, in the order it lists + /// them, or the message of its refusal to recover. + fn single_ready_outcome( + paths: &AppPaths, + config: &RocmCliConfig, + recovery: SetupRecovery, + ) -> Result, String> { + match select_single_ready_runtime(paths, config, recovery) { + Ok(SingleReadyRuntime::One(manifest)) => Ok(vec![manifest.runtime_key]), + Ok(SingleReadyRuntime::Many { ready }) => { + Ok(ready.into_iter().map(|m| m.runtime_key).collect()) + } + Ok(SingleReadyRuntime::Zero { registered }) => { + Err(format!("none ready of {}", registered.len())) + } + Err(error) => Err(format!("{error:#}")), + } + } + + /// A preview counts the runtime setup created from its own manifest and + /// writes nothing; the real run then finds the same runtime and writes the + /// entry. Both orders of install time are covered: the ready runtimes are + /// listed newest first, and a preview must list them the way the registry + /// will once the entry is back. + #[test] + fn preview_counts_the_setup_runtime_without_writing_and_agrees_with_persist() -> Result<()> { + for (label, other_installed_at) in [("other-newer", 30), ("other-older", 10)] { + let (root, paths) = test_paths(&format!("single-ready-preview-{label}")); + let lost = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "runtime-setup", + )?; + write_test_pip_runtime( + &paths, + "runtime-other", + "therock-release:gfx120X-all", + "7.14.0", + other_installed_at, + )?; + let config = RocmCliConfig::load(&paths)?; + let registry_before = crate::test_support::runtime_registry_entry_names(&paths); + assert_eq!(registry_before, ["runtime-other.json"], "{label}"); + + let preview = single_ready_outcome(&paths, &config, SetupRecovery::PreviewOnly) + .expect("the setup runtime and the other one are both ready"); + + assert_eq!(preview.len(), 2, "{label}: {preview:?}"); + assert!(preview.contains(&lost.runtime_key), "{label}: {preview:?}"); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + registry_before, + "{label}: a preview must not write the lost entry" + ); + let persisted = single_ready_outcome(&paths, &config, SetupRecovery::Persist) + .expect("recovery finds the same two"); + assert_eq!(preview, persisted, "{label}: preview and real run disagree"); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + [ + "runtime-other.json".to_owned(), + "runtime-setup.json".to_owned() + ], + "{label}: the real run writes the entry back" + ); + let _ = fs::remove_dir_all(root); + } + Ok(()) + } + + /// The setup runtime alone is "the one ready runtime" in a preview, and a + /// registry that already holds it is left as it is. + #[test] + fn preview_picks_a_lost_setup_runtime_and_ignores_a_registered_one() -> Result<()> { + let (root, paths) = test_paths("single-ready-preview-one"); + crate::test_support::write_test_setup_runtime_missing_from_registry(&paths, "runtime-s")?; + let config = RocmCliConfig::load(&paths)?; + + assert_eq!( + single_ready_outcome(&paths, &config, SetupRecovery::PreviewOnly), + Ok(vec!["runtime-s".to_owned()]) + ); + assert!(crate::test_support::runtime_registry_entry_names(&paths).is_empty()); + + // Once the entry exists a preview and a real run see one runtime, and + // neither adds a second copy of it. + assert_eq!( + single_ready_outcome(&paths, &config, SetupRecovery::Persist), + Ok(vec!["runtime-s".to_owned()]) + ); + assert_eq!( + single_ready_outcome(&paths, &config, SetupRecovery::PreviewOnly), + Ok(vec!["runtime-s".to_owned()]) + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// A setup runtime that is no longer usable is reported by a preview as the + /// real run reports it, and neither writes anything. + #[test] + fn preview_reports_an_unusable_setup_runtime_as_persist_does() -> Result<()> { + let (root, paths) = test_paths("single-ready-preview-unusable"); + let lost = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "runtime-s", + )?; + fs::remove_file( + lost.python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + let config = RocmCliConfig::load(&paths)?; + + let preview = single_ready_outcome(&paths, &config, SetupRecovery::PreviewOnly) + .expect_err("an unusable setup runtime is an error"); + let persist = single_ready_outcome(&paths, &config, SetupRecovery::Persist) + .expect_err("an unusable setup runtime is an error"); + + assert!(preview.contains("is not usable"), "{preview}"); + assert_eq!(preview, persist); + assert!(crate::test_support::runtime_registry_entry_names(&paths).is_empty()); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Recovering the setup runtime restores the registry entry and writes no + /// install-root file, whatever shape that file has: the manifest it restores + /// is the one read from that very file, so it is always what would be + /// written. A hand-edited file (compact, an extra key, no final newline) is + /// therefore left byte for byte as it was, and recovery cannot fail on a + /// setup folder that is read-only. Asserted on the bytes, so it holds when + /// the process can write anyway. The restored entry is what the file says. + #[test] + fn setup_recovery_restores_the_entry_and_leaves_a_hand_edited_install_root_file_alone() + -> Result<()> { + let (root, paths) = test_paths("setup-recovery-hand-edited-file"); + let lost = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "runtime-s", + )?; + let file = lost.install_root.join(".rocm-cli-runtime.json"); + let mut document = serde_json::to_value(&lost)?; + document["a_key_a_future_release_adds"] = serde_json::json!(true); + let hand_edited = serde_json::to_vec(&document)?; + fs::write(&file, &hand_edited)?; + let config = RocmCliConfig::load(&paths)?; + + let recovered = recover_setup_runtime_registration(&paths, &config)?; + + assert_eq!(recovered.as_deref(), Some("runtime-s")); + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + vec!["runtime-s.json".to_owned()] + ); + assert_eq!( + fs::read(&file)?, + hand_edited, + "recovery must not rewrite the install-root file it restored from" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// A setup runtime that is no longer usable is not a ready runtime, so it + /// does not change what the registry alone answers: one registered ready + /// runtime is picked, two are refused as ambiguous with exactly those two + /// named, and the registry is not written. Only when the registry holds no + /// ready runtime is the setup runtime's own error the answer. Both recoveries + /// agree, and the answer and the registry are asserted together. + #[test] + fn an_unusable_setup_runtime_does_not_hide_the_registrys_ready_runtimes() -> Result<()> { + for recovery in [SetupRecovery::Persist, SetupRecovery::PreviewOnly] { + for (registered, expected) in [ + (vec![], Err("is not usable")), + (vec!["runtime-a"], Ok(vec!["runtime-a"])), + ( + vec!["runtime-a", "runtime-b"], + Ok(vec!["runtime-b", "runtime-a"]), + ), + ] { + let (root, paths) = test_paths("single-ready-unusable-setup"); + for (index, key) in registered.iter().enumerate() { + write_test_pip_runtime( + &paths, + key, + "therock-release:gfx120X-all", + "7.14.0", + 10 + index as u128, + )?; + } + let lost = crate::test_support::write_test_setup_runtime_missing_from_registry( + &paths, + "runtime-s", + )?; + fs::remove_file( + lost.python_executable + .as_deref() + .context("the fixture records an interpreter")?, + )?; + let config = RocmCliConfig::load(&paths)?; + let entries = crate::test_support::runtime_registry_entry_names(&paths); + assert_eq!(entries.len(), registered.len()); + + let outcome = single_ready_outcome(&paths, &config, recovery); + + let label = format!("{recovery:?}, {} registered", registered.len()); + match expected { + Ok(keys) => assert_eq!( + outcome, + Ok(keys.into_iter().map(str::to_owned).collect()), + "{label}" + ), + Err(text) => { + let message = outcome.expect_err("nothing else is ready"); + assert!(message.contains(text), "{label}: {message}"); + } + } + assert_eq!( + crate::test_support::runtime_registry_entry_names(&paths), + entries, + "{label}: the unusable setup runtime is not registered" + ); + let _ = fs::remove_dir_all(root); + } + } + Ok(()) + } + + #[test] + fn a_dry_run_previews_and_a_real_run_persists() { + assert_eq!(SetupRecovery::for_dry_run(true), SetupRecovery::PreviewOnly); + assert_eq!(SetupRecovery::for_dry_run(false), SetupRecovery::Persist); + } } diff --git a/apps/rocm/src/runtime_registry.rs b/apps/rocm/src/runtime_registry.rs index 9f9fa17ce7..41eabb776e 100644 --- a/apps/rocm/src/runtime_registry.rs +++ b/apps/rocm/src/runtime_registry.rs @@ -121,7 +121,16 @@ impl InstalledRuntimeManifest { wheel_composition_includes_devel(self.wheel_composition.as_ref()).unwrap_or(self.devel) } - fn normalize_host_paths(mut self) -> Self { + /// Whether ROCm CLI owns the folder this runtime lives in: it installed it, + /// rather than importing or adopting somebody else's. Ownership is what + /// decides whether the install root carries a `.rocm-cli-runtime.json` (see + /// [`write_runtime_manifest`]) and whether uninstall and prune may delete + /// the folder, so those cannot disagree about which runtimes are whose. + pub(crate) const fn owns_install_root(&self) -> bool { + !self.read_only && self.imported_from.is_none() + } + + pub(crate) fn normalize_host_paths(mut self) -> Self { self.install_root = normalize_manifest_path(self.install_root); self.python_launcher = self .python_launcher @@ -336,33 +345,112 @@ pub(crate) fn runtime_manifest_path(paths: &AppPaths, runtime_key: &str) -> Path runtime_registry_dir(paths).join(format!("{runtime_key}.json")) } -pub(crate) fn save_runtime_manifest( +/// What [`write_runtime_manifest`] does when the registry already holds an entry +/// under the manifest's key. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ManifestOverwrite { + /// Write over it. An install of a runtime ROCm CLI already installed is the + /// same runtime again. + Replace, + /// Refuse, and name `--replace`. Importing or adopting a runtime must not + /// displace a registration the user made earlier unless they ask to. + Refuse, +} + +impl ManifestOverwrite { + /// The policy for a command with a `--replace` flag. + pub(crate) const fn for_replace_flag(replace: bool) -> Self { + if replace { Self::Replace } else { Self::Refuse } + } +} + +/// Write the manifest files of a runtime: the registry entry, and, for a +/// runtime ROCm CLI owns ([`InstalledRuntimeManifest::owns_install_root`]), the +/// `.rocm-cli-runtime.json` in its install root. +/// +/// Every way a runtime gets registered goes through this one routine: an SDK +/// install, `rocm runtimes import`, `rocm runtimes adopt`, and the recovery of +/// the runtime `rocm setup` created. `overwrite` is the only thing a caller +/// chooses. +/// +/// The file in the install root is ROCm CLI's proof that it owns the folder: +/// uninstall and prune delete a folder only when it carries one that matches +/// the registry, and activation requires one of a runtime that is not +/// read-only. An imported or adopted runtime is somebody else's folder, which +/// uninstall and prune never delete and activation accepts without one, so it +/// gets none: a root the user may not be able to write to must not make +/// registering it fail, and an import's source manifest may be that very file +/// in another install. +/// +/// The registry entry is written last, after the install-root file. A failed +/// install-root write therefore leaves the registry exactly as it was, and a +/// failed registry write leaves no new entry, so an error that says a runtime +/// could not be registered or restored never describes one whose entry exists. +/// The install-root file may be left behind by a failed registry write; it holds +/// what a retry writes. +/// +/// The file is not rewritten when it already holds what would be written, so +/// recovering a runtime from its own manifest leaves it alone. "Holds" is judged +/// on the decoded manifest, not the bytes ([`install_root_manifest_is_current`]): +/// a file an older binary wrote omits keys that now read back as defaults and +/// are written out explicitly, and a root the user cannot write to must not turn +/// that into a failure. +/// +/// Neither file is written atomically; that is a separate piece of work shared +/// with the other writers in the workspace. +pub(crate) fn write_runtime_manifest( paths: &AppPaths, manifest: &InstalledRuntimeManifest, + overwrite: ManifestOverwrite, ) -> Result<()> { let manifest = manifest.clone().normalize_storage_paths(); let registry_path = runtime_manifest_path(paths, &manifest.runtime_key); + if overwrite == ManifestOverwrite::Refuse && registry_path.exists() { + bail!( + "runtime registry entry already exists: {}; pass --replace to overwrite it", + registry_path.display() + ); + } fs::create_dir_all( registry_path .parent() .context("runtime manifest registry path has no parent directory")?, )?; - fs::write( - ®istry_path, - serde_json::to_vec_pretty(&manifest).context("failed to serialize runtime manifest")?, - ) - .with_context(|| format!("failed to write {}", registry_path.display()))?; - - let local_manifest_path = manifest.install_root.join(".rocm-cli-runtime.json"); - fs::write( - &local_manifest_path, - serde_json::to_vec_pretty(&manifest) - .context("failed to serialize local runtime manifest")?, - ) - .with_context(|| format!("failed to write {}", local_manifest_path.display()))?; + let encoded = + serde_json::to_vec_pretty(&manifest).context("failed to serialize runtime manifest")?; + + if manifest.owns_install_root() { + let local_manifest_path = manifest.install_root.join(".rocm-cli-runtime.json"); + let already_current = fs::read(&local_manifest_path) + .is_ok_and(|existing| install_root_manifest_is_current(&existing, &encoded)); + if !already_current { + fs::write(&local_manifest_path, &encoded) + .with_context(|| format!("failed to write {}", local_manifest_path.display()))?; + } + } + // Last, because the registry entry is what registering a runtime means; see + // the doc comment. + fs::write(®istry_path, &encoded) + .with_context(|| format!("failed to write {}", registry_path.display()))?; Ok(()) } +/// Whether an install root manifest already holds the manifest that `encoded` +/// is the storage form of. Equal bytes qualify, and so does a file that decodes +/// to the same manifest once normalised for storage: a manifest written before +/// a field existed has no key for it, reads back as the field's default, and +/// serializes with an explicit value, so a byte comparison reads it as changed. +/// A file that does not decode is not current. +fn install_root_manifest_is_current(existing: &[u8], encoded: &[u8]) -> bool { + if existing == encoded { + return true; + } + serde_json::from_slice::(existing) + .ok() + .and_then(|decoded| serde_json::to_vec_pretty(&decoded.normalize_storage_paths()).ok()) + .is_some_and(|normalised| normalised == encoded) +} + pub(crate) fn load_runtime_manifests(paths: &AppPaths) -> Result> { Ok(load_runtime_manifests_reporting_unparsed(paths)?.0) } @@ -628,7 +716,7 @@ fn validate_wheel_runtime_manifest(manifest: &InstalledRuntimeManifest) -> Resul } fn validate_tarball_runtime_manifest(manifest: &InstalledRuntimeManifest) -> Result<()> { - if runtime_install_root_has_payload(&manifest.install_root)? { + if rocm_core::install_root_has_payload(&manifest.install_root)? { Ok(()) } else { bail!( @@ -638,16 +726,6 @@ fn validate_tarball_runtime_manifest(manifest: &InstalledRuntimeManifest) -> Res } } -fn runtime_install_root_has_payload(path: &Path) -> Result { - for entry in fs::read_dir(path).with_context(|| format!("failed to read {}", path.display()))? { - let entry = entry?; - if !entry.file_name().to_string_lossy().starts_with('.') { - return Ok(true); - } - } - Ok(false) -} - pub(crate) fn paths_equivalent(left: &Path, right: &Path) -> bool { let left = normalize_path_for_compare(left); let right = normalize_path_for_compare(right); @@ -791,13 +869,13 @@ mod tests { /// that neither `rocm runtimes list` nor `rocm examine` surfaces `devel`, /// so the state is invisible. /// - /// LIMIT: this covers `save_runtime_manifest` only. It does NOT reach + /// LIMIT: this covers `write_runtime_manifest` only. It does NOT reach /// `install_sdk`, which is what actually threads `include_devel` into the /// manifest — that call needs `uv`, a live index, and a real `rocm_sdk` /// probe, so it has no unit coverage here. Hardcoding `devel: true` at that /// write site passes this test and the rest of the suite. #[test] - fn save_runtime_manifest_replaces_devel_rather_than_merging() -> Result<()> { + fn write_runtime_manifest_replaces_devel_rather_than_merging() -> Result<()> { let (root, paths) = test_paths("manifest-replaces-devel"); let install_root = root.join("install-root"); fs::create_dir_all(&install_root)?; @@ -813,14 +891,14 @@ mod tests { with_devel.devel, "test fixture should start with devel: true" ); - save_runtime_manifest(&paths, &with_devel)?; + write_runtime_manifest(&paths, &with_devel, ManifestOverwrite::Replace)?; let without_devel = InstalledRuntimeManifest { devel: false, installed_at_unix_ms: 2, ..with_devel.clone() }; - save_runtime_manifest(&paths, &without_devel)?; + write_runtime_manifest(&paths, &without_devel, ManifestOverwrite::Replace)?; let manifests = load_runtime_manifests(&paths)?; let reloaded = manifests @@ -836,6 +914,703 @@ mod tests { Ok(()) } + /// A manifest for `runtime_key` whose install root is a folder under `root`. + fn manifest_in( + root: &Path, + runtime_key: &str, + read_only: bool, + imported_from: Option, + ) -> Result { + let install_root = root.join(runtime_key); + fs::create_dir_all(&install_root)?; + Ok(InstalledRuntimeManifest { + install_root, + read_only, + imported_from, + ..test_runtime_manifest(runtime_key, "therock-release:gfx120X-all", 1) + }) + } + + /// The policy is the only thing a caller chooses, and it decides one thing: + /// whether an existing registry entry is replaced or left alone. The refusal + /// is asserted as message and state together, and with the entry still the + /// old bytes, because a refusal that wrote first would print the same text. + #[test] + fn write_runtime_manifest_refuses_an_existing_entry_only_under_the_refuse_policy() -> Result<()> + { + let (root, paths) = test_paths("manifest-overwrite-policy"); + let manifest = manifest_in(&root, "release-wheel-gfx120X-all-7.13.0", false, None)?; + let entry = runtime_manifest_path(&paths, &manifest.runtime_key); + + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Refuse)?; + assert!( + entry.is_file(), + "nothing registered yet, so nothing to refuse" + ); + + fs::write(&entry, "stale entry")?; + let error = write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Refuse) + .unwrap_err() + .to_string(); + assert!( + error.contains("already exists") && error.contains("--replace"), + "{error}" + ); + assert_eq!(fs::read_to_string(&entry)?, "stale entry"); + + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace)?; + let replaced: InstalledRuntimeManifest = serde_json::from_slice(&fs::read(&entry)?)?; + assert_eq!(replaced.runtime_key, manifest.runtime_key); + + assert_eq!( + ManifestOverwrite::for_replace_flag(true), + ManifestOverwrite::Replace + ); + assert_eq!( + ManifestOverwrite::for_replace_flag(false), + ManifestOverwrite::Refuse + ); + let _ = fs::remove_dir_all(&root); + Ok(()) + } + + /// Which manifests get a `.rocm-cli-runtime.json` in the install root: + /// exactly the ones ROCm CLI owns. A read-only runtime and an imported one + /// (the two ways a runtime is not ROCm CLI's) get a registry entry and leave + /// their folder alone, whichever policy registered them. + #[test] + fn write_runtime_manifest_writes_the_install_root_manifest_only_for_an_owned_runtime() + -> Result<()> { + let (root, paths) = test_paths("manifest-install-root-file"); + let cases = [ + ("owned", false, None, true), + ("read-only", true, None, false), + ( + "imported", + false, + Some(PathBuf::from("elsewhere/manifest.json")), + false, + ), + ]; + for overwrite in [ManifestOverwrite::Replace, ManifestOverwrite::Refuse] { + for (label, read_only, imported_from, gets_the_file) in &cases { + let key = format!("{label}-{overwrite:?}"); + let manifest = manifest_in(&root, &key, *read_only, imported_from.clone())?; + + write_runtime_manifest(&paths, &manifest, overwrite)?; + + assert_eq!( + manifest.owns_install_root(), + *gets_the_file, + "{label} / {overwrite:?}" + ); + assert!( + runtime_manifest_path(&paths, &key).is_file(), + "{label} / {overwrite:?}: registry entry" + ); + assert_eq!( + manifest + .install_root + .join(".rocm-cli-runtime.json") + .is_file(), + *gets_the_file, + "{label} / {overwrite:?}: install root manifest" + ); + } + } + let _ = fs::remove_dir_all(&root); + Ok(()) + } + + /// The registry entry is what registers a runtime, so it is written last and + /// each failing write is asserted as the error and the state together. When + /// the install-root file cannot be written (a directory stands where the file + /// should be, which stops an administrator as well) the error names that file + /// and the registry is exactly as it was: no entry, so a caller's "could not + /// register" or "could not restore" is true, under either policy. When the + /// registry entry cannot be written (a directory stands where it should be) + /// the error names the entry and there is still no entry that reads as a + /// runtime. + #[test] + fn a_failed_manifest_write_never_leaves_a_registered_runtime_behind() -> Result<()> { + for overwrite in [ManifestOverwrite::Replace, ManifestOverwrite::Refuse] { + let (root, paths) = test_paths("manifest-write-fails"); + let manifest = manifest_in(&root, "release-wheel-gfx120X-all-7.13.0", false, None)?; + let entry = runtime_manifest_path(&paths, &manifest.runtime_key); + let local = manifest.install_root.join(".rocm-cli-runtime.json"); + + fs::create_dir_all(&local)?; + let error = format!( + "{:#}", + write_runtime_manifest(&paths, &manifest, overwrite) + .expect_err("the install-root manifest cannot be written") + ); + assert!( + error.contains("failed to write") && error.contains(".rocm-cli-runtime.json"), + "{overwrite:?}: {error}" + ); + assert!( + !entry.exists() && load_runtime_manifests(&paths)?.is_empty(), + "{overwrite:?}: a failed install-root write leaves no registry entry" + ); + + fs::remove_dir(&local)?; + fs::create_dir_all(&entry)?; + let error = format!( + "{:#}", + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace) + .expect_err("the registry entry cannot be written") + ); + assert!( + error.contains("failed to write") && error.contains(&entry.display().to_string()), + "{overwrite:?}: {error}" + ); + assert!( + !entry.is_file(), + "{overwrite:?}: a failed registry write leaves no entry" + ); + let _ = fs::remove_dir_all(&root); + } + Ok(()) + } + + /// Recovering a runtime from its own install-root manifest must not touch + /// that file when it already holds what would be written, so a setup folder + /// the user cannot write to does not turn a recovery into an error. The file + /// is made read-only: a write would fail where a skip succeeds. (Not + /// asserted when the file stays writable regardless, as for an + /// administrator.) + #[test] + #[allow(clippy::permissions_set_readonly_false)] + fn write_runtime_manifest_leaves_a_current_install_root_manifest_alone() -> Result<()> { + let (root, paths) = test_paths("manifest-current-install-root-file"); + let manifest = manifest_in(&root, "release-wheel-gfx120X-all-7.13.0", false, None)?; + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace)?; + let local = manifest.install_root.join(".rocm-cli-runtime.json"); + let mut permissions = fs::metadata(&local)?.permissions(); + permissions.set_readonly(true); + fs::set_permissions(&local, permissions.clone())?; + let still_writable = fs::OpenOptions::new().append(true).open(&local).is_ok(); + + let unchanged = write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace); + let changed = InstalledRuntimeManifest { + version: "7.14.0".to_owned(), + ..manifest + }; + let rewritten = write_runtime_manifest(&paths, &changed, ManifestOverwrite::Replace); + + permissions.set_readonly(false); + fs::set_permissions(&local, permissions)?; + unchanged.context("an install root manifest that is already current is not rewritten")?; + if !still_writable { + assert!( + rewritten.is_err(), + "a different manifest is written, which the read-only file refuses" + ); + } + let _ = fs::remove_dir_all(&root); + Ok(()) + } + + /// The state matrix for "is the install root manifest already current": + /// the same bytes, the same manifest in an older shape (keys that hold no + /// value omitted, so a byte comparison reads it as changed), a different + /// manifest in that shape, and a file that is not a manifest at all. Pure, so + /// it runs on every platform, unlike the read-only-file tests that show the + /// effect on the filesystem. + #[test] + fn install_root_manifest_is_current_compares_the_decoded_manifest() -> Result<()> { + let (root, _paths) = test_paths("manifest-current-decoded"); + let manifest = manifest_in(&root, "release-wheel-gfx120X-all-7.13.0", false, None)?; + let stored = manifest.clone().normalize_storage_paths(); + let encoded = serde_json::to_vec_pretty(&stored)?; + let legacy_shaped = |manifest: &InstalledRuntimeManifest| -> Result> { + let mut document = serde_json::to_value(manifest.clone().normalize_storage_paths())?; + let object = document.as_object_mut().context("a JSON object")?; + let empty: Vec = object + .iter() + .filter(|(_, value)| value.is_null()) + .map(|(key, _)| key.clone()) + .collect(); + assert!(!empty.is_empty(), "the fixture needs a null-valued key"); + for key in empty { + object.remove(&key); + } + Ok(serde_json::to_vec_pretty(&document)?) + }; + + assert!(install_root_manifest_is_current(&encoded, &encoded)); + let legacy = legacy_shaped(&manifest)?; + assert_ne!(legacy, encoded, "the legacy shape must differ in bytes"); + assert!( + install_root_manifest_is_current(&legacy, &encoded), + "a legacy-shaped file that decodes to the same manifest is current" + ); + let newer = InstalledRuntimeManifest { + version: "7.14.0".to_owned(), + ..manifest + }; + assert!( + !install_root_manifest_is_current(&legacy_shaped(&newer)?, &encoded), + "a legacy-shaped file that decodes to a different manifest is not current" + ); + assert!(!install_root_manifest_is_current(b"not json", &encoded)); + assert!(!install_root_manifest_is_current(b"", &encoded)); + let _ = fs::remove_dir_all(&root); + Ok(()) + } + + /// Calls that put bytes in a file, which is what a second writer of a registry + /// entry would use. Matched as text, so `tokio::fs::write(` counts too. The + /// file builders (`OpenOptions`, `File::options`) are not here: they write + /// only when a write option is chained on them, see [`open_options_write`]. + const WRITE_PRIMITIVES: [&str; 7] = [ + "fs::write(", + "File::create(", + "File::create_new(", + "fs::copy(", + "fs::rename(", + "fs::hard_link(", + "write_all(", + ]; + + /// The file builders, and the options that make one write. `OpenOptions::new()` + /// followed by `.read(true).open(..)` is a read-only probe and not a writer. + const FILE_BUILDERS: [&str; 2] = ["OpenOptions::new(", "File::options("]; + const WRITE_OPTIONS: [&str; 5] = ["write", "append", "create", "create_new", "truncate"]; + + /// The index just past the `)` that closes the `(` at `open`, skipping string + /// literals, or the end of `text` when it never closes. + fn skip_parens(text: &[u8], open: usize) -> usize { + let mut depth = 0usize; + let mut at = open; + let mut in_string = false; + while at < text.len() { + let byte = text[at]; + if in_string { + match byte { + b'\\' => at += 1, + b'"' => in_string = false, + _ => {} + } + } else { + match byte { + b'"' => in_string = true, + b'(' => depth += 1, + b')' => { + depth -= 1; + if depth == 0 { + return at + 1; + } + } + _ => {} + } + } + at += 1; + } + text.len() + } + + /// Whether `production` builds a file with a builder that can write. The + /// method chain after the builder is read across lines and whitespace: it is + /// a read-only probe only when the chain reaches `.open(` and none of + /// [`WRITE_OPTIONS`] is called on it. A builder that is stored (`let mut o = + /// OpenOptions::new();`) or whose chain does not reach `.open(` cannot be + /// judged and counts as a writer. + fn open_options_write(production: &str) -> bool { + let text = production.as_bytes(); + for builder in FILE_BUILDERS { + for (at, _) in production.match_indices(builder) { + let mut next = skip_parens(text, at + builder.len() - 1); + let mut methods = Vec::new(); + loop { + while text.get(next).is_some_and(u8::is_ascii_whitespace) { + next += 1; + } + if text.get(next) != Some(&b'.') { + break; + } + next += 1; + while text.get(next).is_some_and(u8::is_ascii_whitespace) { + next += 1; + } + let start = next; + while text + .get(next) + .is_some_and(|byte| byte.is_ascii_alphanumeric() || *byte == b'_') + { + next += 1; + } + methods.push(&production[start..next]); + while text.get(next).is_some_and(u8::is_ascii_whitespace) { + next += 1; + } + if text.get(next) == Some(&b'(') { + next = skip_parens(text, next); + } + } + let writes = methods.iter().any(|method| WRITE_OPTIONS.contains(method)); + if writes || !methods.contains(&"open") { + return true; + } + } + } + false + } + + /// Every way `production` writes a file, by name. + fn write_calls(production: &str) -> Vec<&'static str> { + let mut calls: Vec<&'static str> = WRITE_PRIMITIVES + .iter() + .copied() + .filter(|primitive| production.contains(primitive)) + .collect(); + if open_options_write(production) { + calls.push("OpenOptions with a write option"); + } + calls + } + + /// `source` as the compiler sees production code, for a text scan: everything + /// from the file's `#[cfg(test)]` test module on is cut, and comments are + /// removed, so neither a fixture nor a sentence in a comment can be mistaken + /// for a writer. String literals stay, because the path segments of the + /// registry directory are string literals. The scan is not a lexer: a raw + /// string that holds a double quote would confuse it. + fn production_code(source: &str) -> String { + let end = [ + "#[cfg(test)]\nmod tests", + "#[cfg(test)]\npub(crate) mod tests", + ] + .iter() + .filter_map(|marker| source.find(marker)) + .min() + .unwrap_or(source.len()); + let chars: Vec = source[..end].chars().collect(); + let mut code = String::with_capacity(end); + let mut at = 0; + let mut in_string = false; + while at < chars.len() { + let c = chars[at]; + if in_string { + code.push(c); + if c == '\\' && at + 1 < chars.len() { + code.push(chars[at + 1]); + at += 2; + continue; + } + in_string = c != '"'; + at += 1; + continue; + } + match (c, chars.get(at + 1)) { + ('/', Some('/')) => { + while at < chars.len() && chars[at] != '\n' { + at += 1; + } + } + ('/', Some('*')) => { + at += 2; + while at + 1 < chars.len() && !(chars[at] == '*' && chars[at + 1] == '/') { + at += 1; + } + at += 2; + } + // The character literal `'"'` must not open a string. + ('\'', Some('"')) if chars.get(at + 2) == Some(&'\'') => { + code.push_str("'\"'"); + at += 3; + } + ('"', _) => { + in_string = true; + code.push(c); + at += 1; + } + _ => { + code.push(c); + at += 1; + } + } + } + code + } + + /// The ways production code can name the registry directory: through the + /// two helpers in this file, or by spelling the path itself (the `runtimes` + /// and `registry` segments, or one literal holding both). + fn registry_location_spellings(production: &str) -> Vec<&'static str> { + let mut spellings = Vec::new(); + for helper in ["runtime_manifest_path(", "runtime_registry_dir("] { + if production.contains(helper) { + spellings.push(helper); + } + } + if production.contains("\"runtimes\"") && production.contains("\"registry\"") { + spellings.push("the \"runtimes\" and \"registry\" path segments"); + } + if production.contains("runtimes/registry") || production.contains("runtimes\\\\registry") { + spellings.push("a literal path through runtimes/registry"); + } + spellings + } + + /// Why `production` writes a registry entry on its own, one line per way it + /// can reach the registry, or nothing. A file that both writes (any of + /// [`WRITE_PRIMITIVES`], or a file builder with a write option, see + /// [`open_options_write`]) and names the registry directory (by any of + /// [`registry_location_spellings`]) is a writer, however far apart the two + /// are and whatever the path variable is called. A file that only reads or + /// removes entries has no write call and is not one. + fn registry_writer_findings(production: &str) -> Vec { + let writes = write_calls(production); + if writes.is_empty() { + return Vec::new(); + } + registry_location_spellings(production) + .into_iter() + .map(|spelling| format!("{spelling} in a file that calls {}", writes.join(", "))) + .collect() + } + + /// The detector sees a writer however it builds its path, and nothing that + /// only reads. The cases are the ways the previous scan (a 200-character + /// window after `fs::write(` looking for three names) went blind: a path built + /// from literals, a path held in a variable away from the write, one literal, + /// a `//` inside a string literal before the write. A file builder + /// (`OpenOptions`, `File::options`) counts as a write only when a write option + /// is chained on it or the chain cannot be read; a read-only probe in a file + /// that builds the registry path, as `amd_smi.rs` does, is not a writer. + #[test] + fn registry_writer_detector_sees_every_way_of_building_the_path() { + let filler = "// padding\n".repeat(40); + let far_apart = format!( + "fn dir(p: &AppPaths) -> PathBuf {{ p.data_dir.join(\"runtimes\").join(\"registry\") }}\n{filler}fn save(d: &Path) {{ fs::write(d.join(\"x.json\"), b\"\").ok(); }}" + ); + let registry = "fn dir(p: &AppPaths) -> PathBuf { p.data_dir.join(\"runtimes\").join(\"registry\") }\n"; + let cases: [(&str, String, bool); 17] = [ + ( + "amd_smi shape: the registry path is built, and only read-only probes open files", + format!( + "{registry}fn probe(path: &Path) -> bool {{ fs::OpenOptions::new().read(true).open(path).is_ok() }}\nfn device(path: &Path) -> bool {{ std::fs::OpenOptions::new().read(true).open(path).is_ok() }}" + ), + false, + ), + ( + "a read-only probe whose chain spans lines", + format!( + "{registry}fn probe(path: &Path) -> bool {{\n OpenOptions::new()\n .read(true)\n .open(path)\n .is_ok()\n}}" + ), + false, + ), + ( + "a read-only probe through File::options", + format!("{registry}fn probe(p: &Path) -> bool {{ File::options().read(true).open(p).is_ok() }}"), + false, + ), + ( + "OpenOptions with .write(true)", + format!( + "{registry}fn save(d: &Path) {{ OpenOptions::new().write(true).create(true).open(d.join(\"k.json\")).ok(); }}" + ), + true, + ), + ( + "OpenOptions with a write option on a later line", + format!( + "{registry}fn save(d: &Path) {{\n fs::OpenOptions::new()\n .read(true)\n .truncate(true)\n .open(d.join(\"k.json\"))\n .ok();\n}}" + ), + true, + ), + ( + "File::options with .append(true)", + format!("{registry}fn save(d: &Path) {{ File::options().append(true).open(d).ok(); }}"), + true, + ), + ( + "a builder kept in a variable cannot be judged", + format!( + "{registry}fn save(d: &Path) {{ let mut options = OpenOptions::new(); options.write(true); options.open(d).ok(); }}" + ), + true, + ), + ( + "a chain that never reaches .open( cannot be judged", + format!("{registry}fn save() {{ let options = OpenOptions::new().read(true).clone(); drop(options); }}"), + true, + ), + ( + "through the helper", + "fn f(p: &AppPaths) { fs::write(runtime_manifest_path(p, \"k\"), b\"\").ok(); }" + .to_owned(), + true, + ), + ("segments, far from the write", far_apart, true), + ( + "one literal", + "fn f(base: &Path) { let d = base.join(\"runtimes/registry\"); File::create(d.join(\"k.json\")).ok(); }" + .to_owned(), + true, + ), + ( + "a // inside a string before the write", + "fn f(p: &AppPaths) { let note = \"see http://x\"; fs::write(runtime_registry_dir(p).join(note), b\"\").ok(); }" + .to_owned(), + true, + ), + ( + "reads only", + "fn f(p: &AppPaths) { let _ = fs::read_dir(p.data_dir.join(\"runtimes\").join(\"registry\")); }" + .to_owned(), + false, + ), + ( + "writes something else", + "fn f(p: &Path) { fs::write(p.join(\"config.json\"), b\"\").ok(); }".to_owned(), + false, + ), + ( + "only a comment names the writer", + "// fs::write(runtime_manifest_path(p, k), b\"\")\nfn f() {}".to_owned(), + false, + ), + ( + "the writer is in the test module", + "fn f() {}\n#[cfg(test)]\nmod tests {\n fn t(p: &AppPaths) { fs::write(runtime_manifest_path(p, \"k\"), b\"\").ok(); }\n}" + .to_owned(), + false, + ), + ( + "a block comment holds the path", + "fn f(p: &Path) { /* runtime_registry_dir( */ fs::write(p, b\"\").ok(); }" + .to_owned(), + false, + ), + ]; + for (label, source, is_a_writer) in cases { + assert_eq!( + !registry_writer_findings(&production_code(&source)).is_empty(), + is_a_writer, + "{label}" + ); + } + } + + /// Every way of registering a runtime goes through + /// [`write_runtime_manifest`]. A second writer would be free to drift again + /// (a different overwrite rule, a missing install-root manifest), so no + /// production code outside this file may both name the registry directory and + /// write a file. This reads the sources of the `rocm` app, `rocmd`, + /// `rocm-core` and the engines: the writers differ only in how they are + /// called, which no behavioural test of one caller can see in another. + /// + /// Two further rules keep that from being walked around. The files that call + /// the two path helpers are an explicit list, each with the reason it needs + /// the path and none of them writing, so a new caller is reviewed rather than + /// passed, and a stale entry fails too. The two fixture modules are exempt + /// because the modules are `#[cfg(test)]` where they are declared, which is + /// checked, not because of what they are called. + /// + /// What this does not catch, by design: a writer that is handed the registry + /// path by a function in another file and never spells it, a write primitive + /// that is not in [`WRITE_PRIMITIVES`] (a new helper with its own name), a + /// file builder whose write option is set through something other than the + /// methods in [`WRITE_OPTIONS`] (the chain is read as text; a chain that + /// reaches `.open(` with only read options is a read), and code outside the + /// source roots it reads. Review catches those; this test + /// catches the ordinary ways of growing a second writer. + #[test] + fn only_the_shared_routine_writes_a_registry_entry() { + fn rust_files(dir: &Path, found: &mut Vec) { + for entry in fs::read_dir(dir).expect("source folder readable").flatten() { + let path = entry.path(); + if path.is_dir() { + rust_files(&path, found); + } else if path.extension().is_some_and(|ext| ext == "rs") { + found.push(path); + } + } + } + let workspace = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .and_then(Path::parent) + .expect("the app sits two levels below the workspace root"); + let mut roots = vec![ + workspace.join("apps/rocm/src"), + workspace.join("apps/rocmd/src"), + workspace.join("crates/rocm-core/src"), + ]; + for engine in fs::read_dir(workspace.join("engines")) + .expect("engines folder readable") + .flatten() + { + roots.push(engine.path().join("src")); + } + let mut sources = Vec::new(); + for root in &roots { + rust_files(root, &mut sources); + } + assert!(sources.len() > 100, "the scan found the source tree"); + + // This file holds the routine. + let routine = "apps/rocm/src/runtime_registry.rs"; + // Fixture modules, each with the file that declares it. + let fixtures = [ + ("apps/rocm/src/test_support.rs", "apps/rocm/src/main.rs"), + ( + "apps/rocm/src/therock/test_support.rs", + "apps/rocm/src/therock.rs", + ), + ]; + for (fixture, declared_in) in fixtures { + let declaration = + fs::read_to_string(workspace.join(declared_in)).expect("declaring file readable"); + assert!( + declaration.contains("#[cfg(test)]\nmod test_support;"), + "{fixture} is exempt only while {declared_in} declares it under #[cfg(test)]" + ); + } + // Production files that name an entry's path without writing it, and why + // (in sorted order, which is how they are compared). + let path_helper_callers = [ + // Whether setup's runtime is registered, and the error that names the registry. + "apps/rocm/src/runtime_adopt.rs", + // The entry uninstall removes. + "apps/rocm/src/runtime_uninstall.rs", + // The path an activation, import or adopt report prints. + "apps/rocm/src/runtimes_cmd.rs", + // The path an install plan prints. + "apps/rocm/src/therock.rs", + ]; + + let mut helper_callers = Vec::new(); + for path in sources { + let relative = path + .strip_prefix(workspace) + .expect("source under the workspace") + .to_string_lossy() + .replace('\\', "/"); + if relative == routine || fixtures.iter().any(|(fixture, _)| *fixture == relative) { + continue; + } + let production = production_code(&fs::read_to_string(&path).expect("source readable")); + let findings = registry_writer_findings(&production); + assert!( + findings.is_empty(), + "{relative} writes a registry entry itself instead of calling write_runtime_manifest: {findings:?}" + ); + if production.contains("runtime_manifest_path(") + || production.contains("runtime_registry_dir(") + { + helper_callers.push(relative); + } + } + helper_callers.sort(); + assert_eq!( + helper_callers, path_helper_callers, + "the files that call runtime_manifest_path or runtime_registry_dir changed: a new caller must be reviewed (and listed with its reason) and a file that stopped calling them removed" + ); + } + /// A manifest written before `devel` became opt-in has no such field, and /// those installs all had the toolchain. Reading one back must not claim /// otherwise, or the next update would silently strip it. @@ -946,4 +1721,57 @@ mod tests { let _ = fs::remove_dir_all(root); Ok(()) } + + /// Activation validation of a tarball runtime follows + /// `rocm_core::install_root_has_payload`: the root's own `.rocm-cli-runtime.json` + /// marker is a dot-entry and does not make a payload, so a root holding only + /// that is reported unusable (by the message the runtime listing prints), and + /// one more entry makes it ready. A root that is gone is reported by the + /// earlier "install root is missing" check, never reaching the payload rule. + #[test] + fn tarball_validation_follows_the_shared_payload_rule() { + let (root, _) = test_paths("tarball-shared-payload-rule"); + let install_root = root.join("tarball-runtime"); + fs::create_dir_all(&install_root).unwrap(); + fs::write(install_root.join(".rocm-cli-runtime.json"), b"{}").unwrap(); + let manifest = InstalledRuntimeManifest { + format: "tarball".to_owned(), + install_root: install_root.clone(), + ..test_runtime_manifest( + "release-tarball-gfx120X-all-7-13-0", + "therock-release:gfx120X-all", + 1, + ) + }; + + let error = validate_runtime_manifest_for_activation(&manifest) + .expect_err("a root with only the dot-marker has no payload") + .to_string(); + assert_eq!( + error, + format!( + "tarball runtime install root has no payload files: {}", + install_root.display() + ) + ); + assert_eq!( + runtime_usability_status(&manifest), + format!("unusable ({error})") + ); + + fs::write(install_root.join("rocm-install-file"), b"x").unwrap(); + validate_runtime_manifest_for_activation(&manifest) + .expect("one entry beside the marker is a payload"); + assert_eq!(runtime_usability_status(&manifest), "ready"); + + fs::remove_dir_all(&install_root).unwrap(); + let error = validate_runtime_manifest_for_activation(&manifest) + .expect_err("a missing root is unusable") + .to_string(); + assert_eq!( + error, + format!("install root is missing: {}", install_root.display()) + ); + fs::remove_dir_all(root).ok(); + } } diff --git a/apps/rocm/src/runtimes_cmd.rs b/apps/rocm/src/runtimes_cmd.rs index bd7033b402..885e0bade4 100644 --- a/apps/rocm/src/runtimes_cmd.rs +++ b/apps/rocm/src/runtimes_cmd.rs @@ -869,7 +869,7 @@ fn write_active_runtime_marker(paths: &AppPaths, marker: ActiveRuntimeMarker) -> #[cfg(test)] pub(crate) mod tests { - use std::path::PathBuf; + use std::path::{Path, PathBuf}; use super::*; use crate::append_examine_runtime_state; @@ -878,6 +878,7 @@ pub(crate) mod tests { use crate::runtime_install_finalize::{ finalize_successful_sdk_install, render_sdk_install_success, }; + use crate::runtime_registry::{ManifestOverwrite, write_runtime_manifest}; use crate::test_support::write_test_pip_runtime; use crate::tests::{ NEW_RUNTIME_KEY, OLD_RUNTIME_KEY, plant_service_record_on_runtime, test_paths, @@ -2311,9 +2312,14 @@ pub(crate) mod tests { Ok(()) } - #[test] - fn runtime_adopt_records_read_only_manifest_from_probe() -> Result<()> { - let (root, paths) = test_paths("runtime-adopt"); + /// A Python environment outside ROCm CLI's folders that `rocm runtimes adopt` + /// accepts: an interpreter, the SDK libraries the probe names, and the probe + /// that reports them. Returns the environment folder, the request that adopts + /// it, and the probe. + fn adoptable_environment( + root: &Path, + replace: bool, + ) -> Result<(PathBuf, AdoptRuntimeRequest, therock::RocmSdkPythonProbe)> { let external_root = root.join("external-therock-venv"); let scripts_dir = external_root.join(if cfg!(windows) { "Scripts" } else { "bin" }); let python_executable = scripts_dir.join(if cfg!(windows) { @@ -2342,37 +2348,43 @@ pub(crate) mod tests { fs::write(&amdhip, "amdhip")?; fs::write(&hipblas, "hipblas")?; - let adopted = adopt_runtime_from_probe( - &paths, - AdoptRuntimeRequest { - python_executable, - install_root: external_root.clone(), - runtime_id: "therock-release:gfx120X-all".to_owned(), - runtime_key: "adopted-release-pip-gfx120x-all-7-13-0".to_owned(), - replace: false, - }, - therock::RocmSdkPythonProbe { - import_ok: true, - rocm_sdk_version: Some("7.13.0".to_owned()), - root_path: Some(sdk_root.clone()), - bin_path: Some(sdk_bin.clone()), - runtime_roots: vec![sdk_root], - bin_paths: vec![sdk_bin.clone()], - library_paths: vec![sdk_bin], - resolved_libraries: vec![ - therock::RocmSdkLibraryProbe { - shortname: "amdhip64".to_owned(), - paths: vec![amdhip], - }, - therock::RocmSdkLibraryProbe { - shortname: "hipblas".to_owned(), - paths: vec![hipblas], - }, - ], - resolved_target_family: Some("gfx120X-all".to_owned()), - ..therock::RocmSdkPythonProbe::default() - }, - )?; + let request = AdoptRuntimeRequest { + python_executable, + install_root: external_root.clone(), + runtime_id: "therock-release:gfx120X-all".to_owned(), + runtime_key: "adopted-release-pip-gfx120x-all-7-13-0".to_owned(), + replace, + }; + let probe = therock::RocmSdkPythonProbe { + import_ok: true, + rocm_sdk_version: Some("7.13.0".to_owned()), + root_path: Some(sdk_root.clone()), + bin_path: Some(sdk_bin.clone()), + runtime_roots: vec![sdk_root], + bin_paths: vec![sdk_bin.clone()], + library_paths: vec![sdk_bin], + resolved_libraries: vec![ + therock::RocmSdkLibraryProbe { + shortname: "amdhip64".to_owned(), + paths: vec![amdhip], + }, + therock::RocmSdkLibraryProbe { + shortname: "hipblas".to_owned(), + paths: vec![hipblas], + }, + ], + resolved_target_family: Some("gfx120X-all".to_owned()), + ..therock::RocmSdkPythonProbe::default() + }; + Ok((external_root, request, probe)) + } + + #[test] + fn runtime_adopt_records_read_only_manifest_from_probe() -> Result<()> { + let (root, paths) = test_paths("runtime-adopt"); + let (external_root, request, probe) = adoptable_environment(&root, false)?; + + let adopted = adopt_runtime_from_probe(&paths, request, probe)?; assert!(adopted.read_only); assert_eq!(adopted.channel, "release"); @@ -2407,6 +2419,310 @@ pub(crate) mod tests { Ok(()) } + /// Which of the two files a registered runtime can have exist: its registry + /// entry, and the `.rocm-cli-runtime.json` ROCm CLI keeps in the install + /// root as proof that it owns the folder. + fn registered_files(paths: &AppPaths, install_root: &Path, runtime_key: &str) -> (bool, bool) { + ( + runtime_manifest_path(paths, runtime_key).is_file(), + install_root.join(".rocm-cli-runtime.json").is_file(), + ) + } + + /// A runtime ROCm CLI installed itself, with both of its manifest files + /// deleted so a test sees only what the write under test creates. + fn installed_runtime_without_manifest_files( + paths: &AppPaths, + ) -> Result { + let manifest = write_test_pip_runtime( + paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 20, + )?; + fs::remove_file(runtime_manifest_path(paths, &manifest.runtime_key))?; + fs::remove_file(manifest.install_root.join(".rocm-cli-runtime.json"))?; + assert_eq!( + registered_files(paths, &manifest.install_root, &manifest.runtime_key), + (false, false) + ); + Ok(manifest) + } + + /// Registering a runtime whose registry entry exists is refused until the + /// caller asks to replace it. `register(replace)` makes the attempt; the + /// message and the entry on disk are asserted together, and the same call + /// with `replace` set is what clears the refusal. + fn assert_duplicate_registration_needs_replace( + paths: &AppPaths, + runtime_key: &str, + register: impl Fn(bool) -> Result<()>, + ) -> Result<()> { + let entry = runtime_manifest_path(paths, runtime_key); + fs::write(&entry, "stale entry")?; + + let error = register(false).unwrap_err().to_string(); + + assert!( + error.contains("already exists") && error.contains("--replace"), + "{error}" + ); + assert_eq!( + fs::read_to_string(&entry)?, + "stale entry", + "a refused registration must leave the existing entry alone" + ); + register(true)?; + let replaced: therock::InstalledRuntimeManifest = + serde_json::from_slice(&fs::read(&entry)?)?; + assert_eq!(replaced.runtime_key, runtime_key); + Ok(()) + } + + /// Installing writes both files, and a second install of the same key + /// overwrites both without being asked: it is the same runtime again. + #[test] + fn runtime_install_writes_the_registry_entry_and_the_install_root_manifest() -> Result<()> { + let (root, paths) = test_paths("runtime-files-install"); + let manifest = installed_runtime_without_manifest_files(&paths)?; + + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace)?; + + assert_eq!( + registered_files(&paths, &manifest.install_root, &manifest.runtime_key), + (true, true) + ); + let reinstalled = therock::InstalledRuntimeManifest { + version: "7.14.0".to_owned(), + ..manifest.clone() + }; + write_runtime_manifest(&paths, &reinstalled, ManifestOverwrite::Replace)?; + for path in [ + runtime_manifest_path(&paths, &manifest.runtime_key), + manifest.install_root.join(".rocm-cli-runtime.json"), + ] { + let written: therock::InstalledRuntimeManifest = + serde_json::from_slice(&fs::read(&path)?)?; + assert_eq!(written.version, "7.14.0", "{}", path.display()); + } + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Importing registers a folder somebody else owns, so it writes the + /// registry entry and nothing into that folder, and refuses to replace an + /// existing entry unless asked. + #[test] + fn runtime_import_writes_the_registry_entry_only_and_refuses_a_duplicate() -> Result<()> { + let (root, paths) = test_paths("runtime-files-import"); + let manifest = installed_runtime_without_manifest_files(&paths)?; + let exported_manifest = root.join("external-runtime.json"); + fs::write(&exported_manifest, serde_json::to_vec_pretty(&manifest)?)?; + + import_runtime_manifest(&paths, &exported_manifest, false)?; + + assert_eq!( + registered_files(&paths, &manifest.install_root, &manifest.runtime_key), + (true, false) + ); + assert_duplicate_registration_needs_replace(&paths, &manifest.runtime_key, |replace| { + import_runtime_manifest(&paths, &exported_manifest, replace).map(drop) + })?; + assert_eq!( + registered_files(&paths, &manifest.install_root, &manifest.runtime_key), + (true, false), + "replacing an import must not start writing into the folder either" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Adopting has the same contract as importing: registry entry only, and a + /// duplicate needs `--replace`. + #[test] + fn runtime_adopt_writes_the_registry_entry_only_and_refuses_a_duplicate() -> Result<()> { + let (root, paths) = test_paths("runtime-files-adopt"); + let (external_root, request, probe) = adoptable_environment(&root, false)?; + let runtime_key = request.runtime_key.clone(); + + adopt_runtime_from_probe(&paths, request.clone(), probe.clone())?; + + assert_eq!( + registered_files(&paths, &external_root, &runtime_key), + (true, false) + ); + assert_duplicate_registration_needs_replace(&paths, &runtime_key, |replace| { + let request = AdoptRuntimeRequest { + replace, + ..request.clone() + }; + adopt_runtime_from_probe(&paths, request, probe.clone()).map(drop) + })?; + assert_eq!( + registered_files(&paths, &external_root, &runtime_key), + (true, false), + "replacing an adoption must not start writing into the environment either" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Recovering the runtime `rocm setup` created restores its registry entry + /// from the manifest in its folder, leaves that manifest in place, and does + /// not replace an entry that is already there. + #[test] + fn setup_recovery_restores_the_registry_entry_and_keeps_the_install_root_manifest() -> Result<()> + { + let (root, paths) = test_paths("runtime-files-recovery"); + let manifest = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 20, + )?; + let mut config = RocmCliConfig::default(); + config.setup.therock_venv = Some(manifest.install_root.clone()); + let entry = runtime_manifest_path(&paths, &manifest.runtime_key); + fs::remove_file(&entry)?; + assert_eq!( + registered_files(&paths, &manifest.install_root, &manifest.runtime_key), + (false, true) + ); + + let recovered = recover_setup_runtime_registration(&paths, &config)?; + + assert_eq!(recovered.as_deref(), Some(manifest.runtime_key.as_str())); + assert_eq!( + registered_files(&paths, &manifest.install_root, &manifest.runtime_key), + (true, true) + ); + fs::write(&entry, "entry written since")?; + recover_setup_runtime_registration(&paths, &config)?; + assert_eq!( + fs::read_to_string(&entry)?, + "entry written since", + "recovery only fills a missing entry" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Rewrite a manifest file as an older rocm-cli wrote it: without the keys + /// that hold no value, which a newer binary reads back as their defaults and + /// writes out as explicit `null`s. Returns the bytes now on disk. + fn rewrite_as_legacy_shaped(path: &Path) -> Result> { + let mut document: serde_json::Value = serde_json::from_slice(&fs::read(path)?)?; + let object = document + .as_object_mut() + .context("a manifest is a JSON object")?; + let empty: Vec = object + .iter() + .filter(|(_, value)| value.is_null()) + .map(|(key, _)| key.clone()) + .collect(); + assert!( + !empty.is_empty(), + "the fixture must hold a null-valued key for this to mean anything" + ); + for key in empty { + object.remove(&key); + } + let legacy = serde_json::to_vec_pretty(&document)?; + fs::write(path, &legacy)?; + Ok(legacy) + } + + /// Recovering the setup runtime from a manifest an older binary wrote, in a + /// folder the user cannot write to: the file decodes to the very manifest the + /// recovery would write, so it is already current and is left alone. Judging + /// it by bytes would read the missing keys as a change, write the registry + /// entry, and then fail on the read-only file. The registry entry and the + /// file's bytes are asserted together. (Where the file stays writable + /// regardless, as for an administrator, the unchanged bytes still prove the + /// file was not rewritten.) + #[test] + #[allow(clippy::permissions_set_readonly_false)] + fn setup_recovery_leaves_a_read_only_legacy_shaped_install_root_manifest_alone() -> Result<()> { + let (root, paths) = test_paths("runtime-files-recovery-legacy"); + let manifest = write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all-7-13-0", + "therock-release:gfx120X-all", + "7.13.0", + 20, + )?; + let mut config = RocmCliConfig::default(); + config.setup.therock_venv = Some(manifest.install_root.clone()); + let entry = runtime_manifest_path(&paths, &manifest.runtime_key); + let local = manifest.install_root.join(".rocm-cli-runtime.json"); + fs::remove_file(&entry)?; + let legacy = rewrite_as_legacy_shaped(&local)?; + let mut permissions = fs::metadata(&local)?.permissions(); + permissions.set_readonly(true); + fs::set_permissions(&local, permissions.clone())?; + + let recovered = recover_setup_runtime_registration(&paths, &config); + + permissions.set_readonly(false); + fs::set_permissions(&local, permissions)?; + let recovered = recovered + .context("a usable manifest in a folder that cannot be written is recovered")?; + assert_eq!(recovered.as_deref(), Some(manifest.runtime_key.as_str())); + assert!(entry.is_file(), "the registry entry is restored"); + assert_eq!( + fs::read(&local)?, + legacy, + "the install root manifest is already current and must not be rewritten" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// The writer gives a runtime a manifest in its folder exactly when + /// uninstall, which prune asks the same question of, may delete that folder. + /// The two are written in different modules; this is what ties them. + #[test] + fn a_runtime_gets_an_install_root_manifest_exactly_when_uninstall_may_remove_its_folder() + -> Result<()> { + let (root, paths) = test_paths("runtime-files-ownership"); + for (label, read_only, imported) in [ + ("owned", false, false), + ("read-only", true, false), + ("imported", false, true), + ] { + let mut manifest = write_test_pip_runtime( + &paths, + &format!("release-pip-{label}"), + "therock-release:gfx120X-all", + "7.13.0", + 20, + )?; + manifest.read_only = read_only; + manifest.imported_from = imported.then(|| PathBuf::from("elsewhere/manifest.json")); + fs::remove_file(manifest.install_root.join(".rocm-cli-runtime.json"))?; + + write_runtime_manifest(&paths, &manifest, ManifestOverwrite::Replace)?; + + let has_manifest = manifest + .install_root + .join(".rocm-cli-runtime.json") + .is_file(); + let may_remove = + should_remove_runtime_install_root(&manifest)? == InstallRootDecision::Remove; + assert_eq!(has_manifest, may_remove, "{label}"); + assert_eq!(has_manifest, manifest.owns_install_root(), "{label}"); + } + + let _ = fs::remove_dir_all(root); + Ok(()) + } + #[test] fn runtime_uninstall_removes_managed_root_and_clears_active_state() -> Result<()> { let (root, paths) = test_paths("runtime-uninstall-managed"); diff --git a/apps/rocm/src/system_packages.rs b/apps/rocm/src/system_packages.rs index 4e3b5b78b6..22d7dcde32 100644 --- a/apps/rocm/src/system_packages.rs +++ b/apps/rocm/src/system_packages.rs @@ -13,8 +13,9 @@ //! `rocm install sdk` (`SdkInstallConsents`) stays in `runtime_install_finalize.rs`, //! which passes the plain `system_packages` bool in. That module re-exports //! `ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch` and -//! `ensure_libnuma_for_torch` for `install_cmd.rs` and `engines_cmd.rs`, and -//! `driver_install.rs` reads `real_os_release` from here directly. +//! `ensure_libnuma_for_torch` for `install_cmd.rs` and `engines_cmd.rs`. Every plan +//! here reads the real host's os-release through +//! `rocm_core::hardware_root::real_os_release`. use std::process::{Command as ProcessCommand, Stdio}; @@ -62,18 +63,6 @@ const fn system_package_install_action( } } -/// The real host's `/etc/os-release`, falling back to `/usr/lib/os-release` -/// when the former is absent (per `os-release(5)`). -/// -/// Deliberately not routed through `rocm_core::host_path`: every caller builds -/// a package-manager plan (`apt`, `dnf`, `zypper`) that then runs against the -/// real machine, so a simulated host root must not choose its distro. The descriptive distro name `examine` reports comes from -/// `ExamineSummary::gather`'s `distro` field, which does follow a simulated -/// root. -pub(crate) fn real_os_release() -> rocm_core::hardware_root::OsRelease { - rocm_core::hardware_root::read_os_release(rocm_core::hardware_root::identity_path_for_str) -} - /// Run a command given as an argv vector directly, without going through a shell. /// /// Used for [`run_system_package_install_plan`], whose commands are modeled as @@ -117,7 +106,7 @@ pub(crate) fn ensure_openmpi_for_vllm(approved: bool) -> Result<()> { } let plan = rocm_core::openmpi::plan_for_os_release_read( - &real_os_release(), + &rocm_core::hardware_root::real_os_release(), rocm_core::openmpi::build_openmpi_install_plan, ); @@ -291,7 +280,10 @@ fn ensure_torch_runtime_dep(approved: bool, dep: &TorchRuntimeDep) { return; } - let plan = rocm_core::openmpi::plan_for_os_release_read(&real_os_release(), dep.build_plan); + let plan = rocm_core::openmpi::plan_for_os_release_read( + &rocm_core::hardware_root::real_os_release(), + dep.build_plan, + ); println!("{} setup", dep.name); println!(" reason: {}", dep.reason); @@ -440,7 +432,6 @@ fn run_system_package_install_plan( #[cfg(test)] mod tests { - #[cfg(all(feature = "e2e-test-hooks", target_os = "linux"))] use std::fs; use super::*; @@ -533,8 +524,10 @@ mod tests { /// The package-manager plans run against the real machine, so a simulated /// host root must not change the distro they are built for: under a root - /// whose `/etc/os-release` names a different distro, `real_os_release` - /// still returns the real file. + /// whose `/etc/os-release` names a different distro, + /// `rocm_core::hardware_root::real_os_release` still returns the real file. + /// Every caller (the driver, OpenMPI, libatomic and libnuma plans and hints) + /// reads through that one function. #[cfg(all(feature = "e2e-test-hooks", target_os = "linux"))] #[test] fn real_os_release_ignores_a_simulated_host_root() { @@ -558,7 +551,7 @@ mod tests { rocm_core::hardware_root::TEST_HOST_ROOT_ENV, &root.to_string_lossy(), ); - real_os_release() + rocm_core::hardware_root::real_os_release() }; fs::remove_dir_all(&root).ok(); @@ -569,8 +562,8 @@ mod tests { ); // Independent of `hardware_root::read_os_release`: rather than call // the exact same helper with the same identity resolver `real_os_release` - // wraps one line away (which would pass even if its `.context(...)` - // call or resolver closure broke), read both candidate paths directly + // wraps (which would pass even if the resolver it passes broke), read + // both candidate paths directly // and apply the same existence-only fallback rule inline. This mirrors // which variant is correct for each case, rather than accepting // `Absent` or `Unreadable` interchangeably — see `ExpectedOsRelease`'s @@ -605,4 +598,99 @@ mod tests { ), } } + + /// Every package-manager plan this module and `driver_install.rs` build is + /// written for the real machine, so they all read its os-release through + /// `rocm_core::hardware_root::real_os_release`, whose own test + /// (`real_os_release_ignores_a_simulated_host_root`) proves it ignores a + /// simulated host root. A caller that read the file with `read_os_release` + /// and its own resolver would be a second reader that test does not cover, so + /// no source file of this binary may name `read_os_release` or the identity + /// resolver: the one place that does is `hardware_root.rs` in `rocm-core`. + #[test] + fn no_caller_in_this_binary_reads_the_real_os_release_by_hand() { + fn rust_files(dir: &std::path::Path, out: &mut Vec) { + for entry in fs::read_dir(dir).expect("a readable source directory") { + let path = entry.expect("a readable entry").path(); + if path.is_dir() { + rust_files(&path, out); + } else if path.extension().is_some_and(|ext| ext == "rs") { + out.push(path); + } + } + } + + // Spelled in two halves so this file does not match its own scan. + let needles = [ + ["read_os_", "release("].concat(), + ["identity_path_", "for_str"].concat(), + ]; + let mut files = Vec::new(); + rust_files( + &std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src"), + &mut files, + ); + assert!(!files.is_empty(), "the scan found no source files"); + let offenders: Vec = files + .iter() + .filter(|path| { + let text = fs::read_to_string(path).expect("a readable source file"); + needles.iter().any(|needle| text.contains(needle.as_str())) + }) + .map(|path| path.display().to_string()) + .collect(); + assert!( + offenders.is_empty(), + "these files read os-release by hand instead of through \ + rocm_core::hardware_root::real_os_release: {offenders:?}" + ); + } + + /// The OpenMPI, libatomic and libnuma install hints (`rocm_core::openmpi`) + /// name package-manager commands for the machine the process runs on, like + /// the plans in this module, so a simulated host root must not choose their + /// distro either: with a root whose `/etc/os-release` names another + /// distribution, each hint is still the one the real host's file produces. + /// A hint that read through `host_path` would be the planted distro's (the + /// manual advice, as `planted-distro` maps to no package manager). + #[cfg(all(feature = "e2e-test-hooks", target_os = "linux"))] + #[test] + fn the_install_hints_read_the_real_host_and_ignore_a_simulated_root() { + let root = std::env::temp_dir().join(format!( + "rocm-os-release-hint-root-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_nanos()) + )); + fs::create_dir_all(root.join("etc")).expect("create planted /etc"); + fs::write(root.join("etc/os-release"), "ID=planted-distro\n") + .expect("write planted os-release"); + + let hints = || { + [ + rocm_core::openmpi::install_hint(), + rocm_core::openmpi::libatomic_install_hint(), + rocm_core::openmpi::libnuma_install_hint(), + ] + }; + let (real, simulated) = { + let mut env = ScopedTestEnv::new(); + let real = hints(); + env.set( + rocm_core::hardware_root::TEST_HOST_ROOT_ENV, + &root.to_string_lossy(), + ); + (real, hints()) + }; + fs::remove_dir_all(&root).ok(); + + assert_eq!(simulated, real); + if !real[0].plan_installs { + // A host that maps to no package manager reads the same as the + // planted file, so the assertion above held without telling which + // one was read. + eprintln!("host os-release names no known distribution; nothing proven"); + } + } } diff --git a/apps/rocm/src/test_support.rs b/apps/rocm/src/test_support.rs index c724ad973f..e4f0d406cb 100644 --- a/apps/rocm/src/test_support.rs +++ b/apps/rocm/src/test_support.rs @@ -229,6 +229,58 @@ pub(crate) fn write_test_pip_runtime( Ok(manifest) } +/// Leave `paths` as `rocm setup` followed by a lost registry entry would: a ready +/// wheel runtime whose install tree holds its own `.rocm-cli-runtime.json` and +/// that `config.json` names as the setup folder, but that the runtime registry +/// has no entry for. Only `recover_setup_runtime_registration` can make it +/// visible again, so a selection that finds it has run that recovery. +pub(crate) fn write_test_setup_runtime_missing_from_registry( + paths: &AppPaths, + runtime_key: &str, +) -> Result { + let manifest = write_test_pip_runtime( + paths, + runtime_key, + "therock-release:gfx120X-all", + "7.13.0", + 20, + )?; + fs::remove_file(runtime_manifest_path(paths, runtime_key))?; + let mut config = rocm_core::RocmCliConfig::default(); + config.setup.therock_venv = Some(manifest.install_root.clone()); + config.save(paths)?; + Ok(manifest) +} + +/// Make the runtime registry impossible to write for any user, an administrator +/// included: a plain file stands where its directory should be, so listing the +/// registry finds nothing and creating it fails. A read-only directory would +/// not do, because it does not stop `root`, which is who the e2e suite can run as. +/// Returns the path of that file, so a test can show it was left as it was. +pub(crate) fn make_runtime_registry_unwritable(paths: &AppPaths) -> Result { + let registry = runtime_registry_dir(paths); + let _ = fs::remove_dir_all(®istry); + if let Some(parent) = registry.parent() { + fs::create_dir_all(parent)?; + } + fs::write(®istry, "not a directory")?; + Ok(registry) +} + +/// File names in the runtime registry directory, sorted; empty when the +/// directory does not exist. Lets a test assert that a selection did, or did +/// not, write a registry entry. +pub(crate) fn runtime_registry_entry_names(paths: &AppPaths) -> Vec { + let mut names: Vec = fs::read_dir(runtime_registry_dir(paths)) + .into_iter() + .flatten() + .flatten() + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .collect(); + names.sort(); + names +} + mod tests { use super::*; diff --git a/apps/rocm/src/therock.rs b/apps/rocm/src/therock.rs index e31b255aac..1cfe879e86 100644 --- a/apps/rocm/src/therock.rs +++ b/apps/rocm/src/therock.rs @@ -71,7 +71,8 @@ pub(crate) use crate::runtime_registry::{ validate_rocm_sdk_runtime_probe, }; use crate::runtime_registry::{ - runtime_manifest_path, save_runtime_manifest, wheel_composition_includes_devel, + ManifestOverwrite, runtime_manifest_path, wheel_composition_includes_devel, + write_runtime_manifest, }; use crate::shell_quote::shell_quote; use anyhow::{Context, Result, bail}; @@ -730,7 +731,7 @@ fn install_wheel_runtime( devel: include_devel, installed_at_unix_ms: unix_time_millis(), }; - save_runtime_manifest(paths, &manifest)?; + write_runtime_manifest(paths, &manifest, ManifestOverwrite::Replace)?; let _ = writeln!( output, @@ -955,12 +956,7 @@ fn install_tarball_runtime( fs::create_dir_all(paths.cache_dir.join("therock"))?; fs::create_dir_all(&install_root)?; - if has_nontrivial_directory_contents(&install_root)? { - bail!( - "tarball install target {} is not empty; choose a clean prefix or remove the old extraction first", - install_root.display() - ); - } + ensure_tarball_install_target_is_clean(&install_root)?; if let Some(warning) = preflight_tarball_space(&artifact.url, &cache_path, &install_root)? { let _ = writeln!(output, " {warning}"); @@ -1007,7 +1003,7 @@ fn install_tarball_runtime( devel: true, installed_at_unix_ms: unix_time_millis(), }; - save_runtime_manifest(paths, &manifest)?; + write_runtime_manifest(paths, &manifest, ManifestOverwrite::Replace)?; let _ = writeln!(output, " extracted: {}", install_root.display()); let _ = writeln!(output, " manifest: {}", manifest_path.display()); @@ -1109,21 +1105,18 @@ fn resolved_install_root( rocm_core::resolve_path_through_symlinks(&requested) } -fn has_nontrivial_directory_contents(path: &Path) -> Result { - if !path.exists() { - return Ok(false); - } - let entries = - fs::read_dir(path).with_context(|| format!("failed to read {}", path.display()))?; - for entry in entries { - let entry = entry?; - let name = entry.file_name(); - if name.to_string_lossy().starts_with('.') { - continue; - } - return Ok(true); +/// Refuses to extract a tarball into a root that already holds a payload. +/// +/// "Holds a payload" is [`rocm_core::install_root_has_payload`]: dot-entries (the +/// runtime marker) do not count, so a root that has only been prepared passes. +fn ensure_tarball_install_target_is_clean(install_root: &Path) -> Result<()> { + if rocm_core::install_root_has_payload(install_root)? { + bail!( + "tarball install target {} is not empty; choose a clean prefix or remove the old extraction first", + install_root.display() + ); } - Ok(false) + Ok(()) } fn venv_python_path(install_root: &Path) -> PathBuf { @@ -1703,4 +1696,41 @@ mod tests { "ROCm 7.14.0 is the newest version in this repository but has no installable PyTorch wheels for this Python and platform; installing ROCm 7.13.0 instead" ); } + + /// The extraction guard refuses a root that holds a payload and says so, and + /// it follows `rocm_core::install_root_has_payload` for what that means: a + /// root that does not exist yet (the case `install_tarball_runtime` hits when + /// `--prefix` names a new folder) and one holding only dot-entries pass. + /// Refusing must leave the existing files where they were. + #[test] + fn the_tarball_install_guard_follows_the_shared_payload_rule() { + let base = tempfile::tempdir().expect("scratch dir"); + + ensure_tarball_install_target_is_clean(&base.path().join("not-created-yet")) + .expect("a missing root is clean"); + + let marker_only = base.path().join("marker-only"); + fs::create_dir_all(&marker_only).unwrap(); + fs::write(marker_only.join(".rocm-cli-runtime.json"), b"{}").unwrap(); + ensure_tarball_install_target_is_clean(&marker_only) + .expect("a root with only the dot-marker is clean"); + + let populated = base.path().join("populated"); + fs::create_dir_all(populated.join("bin")).unwrap(); + fs::write(populated.join("bin").join("tool"), b"x").unwrap(); + let error = ensure_tarball_install_target_is_clean(&populated) + .expect_err("a root with a payload is refused") + .to_string(); + assert_eq!( + error, + format!( + "tarball install target {} is not empty; choose a clean prefix or remove the old extraction first", + populated.display() + ) + ); + assert!( + populated.join("bin").join("tool").is_file(), + "the refusal must not touch what is already there" + ); + } } diff --git a/apps/rocm/src/therock/update.rs b/apps/rocm/src/therock/update.rs index 3b203c3e2d..5d49245313 100644 --- a/apps/rocm/src/therock/update.rs +++ b/apps/rocm/src/therock/update.rs @@ -17,9 +17,10 @@ //! //! Builds on `family`, `io`, `process`, `python`, `resolve`, `source` and //! `version`, and on `runtime_registry` for the manifests; `runtime_key`, -//! `wheel_runtime_key` and `has_nontrivial_directory_contents`, which name and -//! inspect the install a plan targets, stay in `therock.rs` with the other keys -//! and roots helpers. +//! `wheel_runtime_key`, which name the install a plan targets, stay in +//! `therock.rs` with the other keys and roots helpers. Whether an install root +//! holds anything is `rocm_core::install_root_has_payload`, the one rule the +//! install and activation paths share. use super::family::family_override_or_recovered_arch; use super::io::write_file_atomically; @@ -34,7 +35,7 @@ use super::resolve::{ }; use super::source::{SourceLayout, TheRockChannel}; use super::version::{runtime_version_display, version_relation}; -use super::{has_nontrivial_directory_contents, runtime_key, wheel_runtime_key}; +use super::{runtime_key, wheel_runtime_key}; use crate::runtime_registry::{ InstalledRuntimeManifest, WheelRuntimeComposition, load_runtime_manifests, wheel_composition_device_target, @@ -295,7 +296,7 @@ fn replacement_runtime_is_installed( && candidate.format == source.format && candidate.family == source.family && candidate.wheel_composition.as_ref() == Some(required_composition) - && has_nontrivial_directory_contents(&candidate.install_root).unwrap_or(false) + && rocm_core::install_root_has_payload(&candidate.install_root).unwrap_or(false) }) } @@ -1072,6 +1073,55 @@ mod tests { fs::remove_dir_all(root).ok(); } + /// A replacement counts as installed by `rocm_core::install_root_has_payload`: + /// a root holding only the runtime's own dot-marker was prepared but never + /// filled, so the repair is still on offer; one entry beside the marker is a + /// real install, so it is not. The runtime directory vanishing is covered by + /// `an_installed_replacement_sibling_suppresses_repeat_repair`. + #[test] + fn a_replacement_counts_as_installed_by_the_shared_payload_rule() { + let required = test_wheel_composition("gfx942"); + let target_runtime_key = wheel_runtime_key(TheRockChannel::Release, "7.14.0", &required); + let mut source = test_runtime_manifest( + "release-wheel-multi-arch-7-14-0", + "therock-release:gfx94X-dcgpu", + 1, + ); + source.version = "7.14.0".to_owned(); + let mut replacement = + test_runtime_manifest(&target_runtime_key, "therock-release:gfx94X-dcgpu", 2); + replacement.version = "7.14.0".to_owned(); + replacement.wheel_composition = Some(required.clone()); + let (root, _) = test_paths("replacement-shared-payload-rule"); + replacement.install_root = root.join("replacement"); + fs::create_dir_all(&replacement.install_root).unwrap(); + fs::write( + replacement.install_root.join(".rocm-cli-runtime.json"), + b"{}", + ) + .unwrap(); + let manifests = vec![source.clone(), replacement.clone()]; + let freshness = |manifests: &[InstalledRuntimeManifest]| { + runtime_freshness_with_manifests( + manifests, + &source, + "7.14.0", + Some(&required), + &target_runtime_key, + ) + }; + + assert_eq!( + freshness(&manifests), + RuntimeFreshness::RepairAvailable, + "a root with only the dot-marker has no payload, so it is not a replacement" + ); + + fs::write(replacement.install_root.join("installed.marker"), b"ok").unwrap(); + assert_eq!(freshness(&manifests), RuntimeFreshness::UpToDate); + fs::remove_dir_all(root).ok(); + } + #[test] fn update_json_reports_no_managed_runtimes_as_empty_list() -> Result<()> { let (root, paths) = test_paths("update-json-empty"); diff --git a/crates/rocm-core/src/hardware_root.rs b/crates/rocm-core/src/hardware_root.rs index 14b04f19bc..afde608e4e 100644 --- a/crates/rocm-core/src/hardware_root.rs +++ b/crates/rocm-core/src/hardware_root.rs @@ -103,20 +103,34 @@ pub fn host_path_for_str(path: &str) -> PathBuf { host_path(path) } -/// [`read_os_release`]'s `resolve` parameter for a caller that must always -/// read the real machine. +/// [`read_os_release`]'s `resolve` parameter for the real machine: the path +/// as written. A bare `PathBuf::from` closure is rejected the same way a bare +/// `host_path` is (see [`host_path_for_str`]), hence a named function. /// -/// Never routed through [`host_path`], because it builds a package-manager -/// plan or install hint that runs against the real host, so a simulated -/// host root must not choose its distro (see each caller's own doc comment -/// for why). A bare `PathBuf::from` is rejected the same way a bare -/// `host_path` is (see [`host_path_for_str`]), so every such caller passes -/// this instead of `|p| PathBuf::from(p)`. -#[must_use] -pub fn identity_path_for_str(path: &str) -> PathBuf { +/// Private: the only reader that must always see the real machine is +/// [`real_os_release`], and having one place that names this resolver is what +/// keeps a second, diverging copy of that reader from appearing elsewhere. +fn identity_path_for_str(path: &str) -> PathBuf { PathBuf::from(path) } +/// The real host's `/etc/os-release`, falling back to `/usr/lib/os-release` +/// when the former is absent (per `os-release(5)`). +/// +/// The one reader for every caller that builds a package-manager plan or an +/// install hint (`apt`, `dnf`, `zypper`, ...): those commands run against the +/// real machine, so a simulated host root (see [`host_path`]) must not choose +/// the distro they are written for. Deliberately not routed through +/// [`host_path`] for that reason. The descriptive distro name and id that +/// `examine` reports come from [`read_os_release`] with [`host_path_for_str`] +/// instead, and do follow a simulated root. +/// +/// Reads the same files on every platform; off Linux they are simply absent. +#[must_use] +pub fn real_os_release() -> OsRelease { + read_os_release(identity_path_for_str) +} + #[cfg(feature = "e2e-test-hooks")] fn hardware_root() -> Option { std::env::var_os(TEST_HOST_ROOT_ENV) diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 707c71332c..c142c3332f 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -197,19 +197,20 @@ use runtime::env_path_override; pub use runtime::{ RUNTIME_LIBRARY_PATH_ENV, RuntimeHost, RuntimePlatform, current_executable_path, default_cache_dir, default_config_dir, default_data_dir, default_interactive_shell_program, - managed_logs_dir, managed_pip_cache_dir, managed_runtime_cache_dir, managed_runtime_data_root, - managed_tools_dir, managed_uv_cache_dir, normalize_runtime_path_for_host, - normalize_runtime_path_for_storage, normalize_runtime_path_text_for_host, - normalize_runtime_path_text_for_platform, normalize_runtime_path_text_for_storage, - platform_binary_name, prepend_runtime_path, resolve_path_through_symlinks, - runtime_directory_label, runtime_drive_root_for_key, runtime_drive_roots, runtime_exe_suffix, - runtime_home_dir, runtime_install_root_is_protected, runtime_is_linux, runtime_is_windows, - runtime_os_name, runtime_path_for_child, runtime_path_for_windows_child, - runtime_path_is_same_or_inside, runtime_path_list_join, runtime_path_list_split, - runtime_path_sort_key, runtime_path_text_is_absolute_for_host, + install_root_has_payload, managed_logs_dir, managed_pip_cache_dir, managed_runtime_cache_dir, + managed_runtime_data_root, managed_tools_dir, managed_uv_cache_dir, + normalize_runtime_path_for_host, normalize_runtime_path_for_storage, + normalize_runtime_path_text_for_host, normalize_runtime_path_text_for_platform, + normalize_runtime_path_text_for_storage, platform_binary_name, prepend_runtime_path, + resolve_path_through_symlinks, runtime_directory_label, runtime_drive_root_for_key, + runtime_drive_roots, runtime_exe_suffix, runtime_home_dir, runtime_install_root_is_protected, + runtime_is_linux, runtime_is_windows, runtime_os_name, runtime_path_for_child, + runtime_path_for_windows_child, runtime_path_is_same_or_inside, runtime_path_list_join, + runtime_path_list_split, runtime_path_sort_key, runtime_path_text_is_absolute_for_host, runtime_path_text_is_absolute_for_platform, runtime_paths_equivalent, runtime_python_activation_hint, runtime_python_activation_script, runtime_python_bin_dir_name, - runtime_python_env_bin_dir, runtime_python_executable_in_env, runtime_python_executable_name, + runtime_python_env_bin_dir, runtime_python_env_root_for_executable, + runtime_python_executable_in_env, runtime_python_executable_name, runtime_rocm_library_filename, shell_command_for_host, user_runtime_dir, }; pub use service_identity::{ServiceId, generate_service_id}; diff --git a/crates/rocm-core/src/openmpi.rs b/crates/rocm-core/src/openmpi.rs index 44b6a8843f..2f45df2da2 100644 --- a/crates/rocm-core/src/openmpi.rs +++ b/crates/rocm-core/src/openmpi.rs @@ -707,17 +707,6 @@ pub fn build_libnuma_install_plan(os_id: &str, id_like: &str) -> SystemPackageIn } } -/// The real host's `/etc/os-release`, falling back to `/usr/lib/os-release` -/// when the former is absent (per `os-release(5)`). -/// -/// Not routed through [`crate::host_path`]: the hints name package-manager -/// commands for the real machine, so a simulated host root must not choose -/// their distribution. -#[cfg(target_os = "linux")] -fn real_os_release() -> crate::hardware_root::OsRelease { - crate::hardware_root::read_os_release(crate::hardware_root::identity_path_for_str) -} - const OPENMPI_MANUAL_HINT: &str = "install your distribution's OpenMPI runtime package (providing libmpi.so / libmpi_cxx.so and mpirun)"; const LIBATOMIC_MANUAL_HINT: &str = "install your distribution's libatomic runtime package (providing libatomic.so.1)"; @@ -738,7 +727,7 @@ pub fn install_hint() -> InstallHint { #[cfg(target_os = "linux")] { install_hint_for_read( - &real_os_release(), + &crate::hardware_root::real_os_release(), build_openmpi_install_plan, OPENMPI_MANUAL_HINT, ) @@ -765,7 +754,7 @@ pub fn libatomic_install_hint() -> InstallHint { #[cfg(target_os = "linux")] { install_hint_for_read( - &real_os_release(), + &crate::hardware_root::real_os_release(), build_libatomic_install_plan, LIBATOMIC_MANUAL_HINT, ) @@ -792,7 +781,7 @@ pub fn libnuma_install_hint() -> InstallHint { #[cfg(target_os = "linux")] { install_hint_for_read( - &real_os_release(), + &crate::hardware_root::real_os_release(), build_libnuma_install_plan, LIBNUMA_MANUAL_HINT, ) @@ -1012,10 +1001,9 @@ mod tests { } /// `install_hint_for_read`/`plan_for_os_release_read` are the one seam - /// `openmpi::real_os_release()` and `system_packages::real_os_release()` - /// hand their result to — both now one-line delegations to - /// `hardware_root::read_os_release`, so reverting either back to a raw - /// `/etc/os-release` read is a type error here, not a silent regression. + /// `hardware_root::real_os_release()` hands its result to, from here and + /// from every caller in the `rocm` binary, so reverting any of them back to + /// a raw `/etc/os-release` read is a type error, not a silent regression. /// This still pins the behaviour at the seam itself: an `OsRelease` that /// says the fallback path was read produces the same plan and hint a /// primary-path read with the same content would. diff --git a/crates/rocm-core/src/runtime.rs b/crates/rocm-core/src/runtime.rs index 3535035b07..6d3e8fbb39 100644 --- a/crates/rocm-core/src/runtime.rs +++ b/crates/rocm-core/src/runtime.rs @@ -119,6 +119,31 @@ pub fn runtime_python_executable_in_env(env_root: &Path) -> PathBuf { runtime_python_env_bin_dir(env_root).join(runtime_python_executable_name()) } +/// The environment folder an interpreter belongs to, found from the +/// interpreter's own path: the inverse of [`runtime_python_executable_in_env`] +/// for a caller that holds only the interpreter. +/// +/// An interpreter belongs to an environment only when it sits directly in that +/// environment's `bin` or `Scripts` folder, and the answer is that folder's +/// parent. Both layouts are accepted, in any letter case, whatever the host: a +/// recorded interpreter path is not always from this host's layout, and a +/// Windows file system does not tell `Scripts` from `SCRIPTS`. +/// +/// Anything else gives `None`: an interpreter in some other folder, a bare file +/// name, or one at the file system root. A guess would name a folder that is +/// not an environment, and the caller would then export it as `VIRTUAL_ENV`, +/// look for `lib/python3.*` under it, or register it as a runtime. `None` lets +/// each caller refuse in its own words instead. +pub fn runtime_python_env_root_for_executable(python: &Path) -> Option { + let bin_dir = python.parent()?; + let name = bin_dir.file_name()?.to_str()?; + if name.eq_ignore_ascii_case("bin") || name.eq_ignore_ascii_case("Scripts") { + bin_dir.parent().map(Path::to_path_buf) + } else { + None + } +} + pub fn runtime_python_activation_script(env_root: &Path) -> PathBuf { let script = if runtime_is_windows() { "activate.bat" @@ -695,6 +720,35 @@ pub fn runtime_install_root_is_protected(path: &Path) -> bool { .any(|root| runtime_path_is_same_or_inside(&path, root)) } +/// Whether an install root holds anything the install put there. +/// +/// Dot-entries do not count: the runtime's own `.rocm-cli-runtime.json` marker is +/// one, and a root that holds only that has no payload. Any other entry does, +/// whatever it is: a file, a directory, or a symlink that points nowhere. A path +/// that does not exist holds nothing, so it answers `false` rather than failing; +/// that is what lets a fresh install ask about a root it has not created yet. A +/// root that exists but cannot be listed (not a directory, no permission) is an +/// error, never a quiet `false`. +/// +/// This is the one rule for "is this install root empty" in the workspace: the +/// tarball extraction refuses a target that has a payload, a repair counts a +/// sibling as installed only when it has one, and activation validation rejects a +/// tarball runtime that has none. +pub fn install_root_has_payload(path: &Path) -> Result { + if !path.exists() { + return Ok(false); + } + let entries = + std::fs::read_dir(path).with_context(|| format!("failed to read {}", path.display()))?; + for entry in entries { + let entry = entry?; + if !entry.file_name().to_string_lossy().starts_with('.') { + return Ok(true); + } + } + Ok(false) +} + #[cfg_attr(not(test), allow(dead_code))] #[derive(Debug, Clone, Copy, Eq, PartialEq)] enum RuntimePathSeparator { @@ -1026,6 +1080,57 @@ mod tests { use super::*; use std::fs; + /// The one rule every caller shares. Paths are built with `join`, not + /// written with a separator, so the table means the same on every host. + #[test] + fn env_root_for_executable_needs_a_bin_or_scripts_folder() { + let venv = Path::new("opt").join("venv"); + for (folder, python) in [ + ("bin", "python"), + ("BIN", "python"), + ("Scripts", "python.exe"), + ("SCRIPTS", "python.exe"), + ("scripts", "python.exe"), + ] { + assert_eq!( + runtime_python_env_root_for_executable(&venv.join(folder).join(python)), + Some(venv.clone()), + "{folder}" + ); + } + for folder in ["tools", "bin2", "my-bin", "venv"] { + assert_eq!( + runtime_python_env_root_for_executable(&venv.join(folder).join("python")), + None, + "{folder}" + ); + } + // A bare file name has an empty parent, and a root-level file's parent + // is the file system root: neither is inside a `bin` or `Scripts`. + assert_eq!( + runtime_python_env_root_for_executable(Path::new("python")), + None + ); + assert_eq!( + runtime_python_env_root_for_executable( + &Path::new(std::path::MAIN_SEPARATOR_STR).join("python") + ), + None + ); + } + + /// The inverse really is the inverse: the interpreter this host's layout + /// puts in an environment leads back to that environment. + #[test] + fn env_root_for_executable_inverts_the_hosts_layout() { + let env_root = Path::new("opt").join("venv"); + let python = runtime_python_executable_in_env(&env_root); + assert_eq!( + runtime_python_env_root_for_executable(&python).as_deref(), + Some(normalize_runtime_path_for_host(&env_root).as_path()) + ); + } + fn scratch_dir(name: &str) -> PathBuf { let root = std::env::temp_dir().join(format!( "rocm-core-resolve-{name}-{}-{:?}", @@ -1819,3 +1924,135 @@ mod tests { } } } + +#[cfg(test)] +mod install_root_payload_tests { + use super::*; + use std::fs; + + /// One path per kind of install root. An unreadable directory is only built + /// when this process cannot read it anyway (a root user can, and then the case + /// says nothing). + fn cases(base: &Path) -> Vec<(&'static str, PathBuf)> { + let mut cases = vec![("missing path", base.join("missing"))]; + let dir = |name: &str| { + let path = base.join(name); + fs::create_dir_all(&path).expect("create case dir"); + path + }; + cases.push(("empty dir", dir("empty"))); + let dotfile = dir("dotfile"); + fs::write(dotfile.join(".rocm-cli-runtime.json"), b"{}").expect("write marker"); + cases.push(("dotfile only", dotfile)); + let dotdir = dir("dotdir"); + fs::create_dir_all(dotdir.join(".hidden")).expect("create dot dir"); + cases.push(("dot directory only", dotdir)); + let file = dir("file"); + fs::write(file.join("payload.bin"), b"x").expect("write payload"); + cases.push(("regular file", file)); + let sub = dir("sub"); + fs::create_dir_all(sub.join("bin")).expect("create subdir"); + cases.push(("subdirectory", sub)); + let marker_and_file = dir("marker-and-file"); + fs::write(marker_and_file.join(".rocm-cli-runtime.json"), b"{}").expect("write marker"); + fs::write(marker_and_file.join("payload.bin"), b"x").expect("write payload"); + cases.push(("marker plus a regular file", marker_and_file)); + let not_a_dir = base.join("not-a-dir"); + fs::write(¬_a_dir, b"x").expect("write file root"); + cases.push(("root is a regular file", not_a_dir)); + #[cfg(unix)] + { + use std::os::unix::fs::{PermissionsExt, symlink}; + let dangling = dir("dangling-entry"); + symlink(base.join("nowhere"), dangling.join("link")).expect("dangling entry"); + cases.push(("dangling symlink entry", dangling)); + let link_root = base.join("link-root"); + symlink(base.join("sub"), &link_root).expect("symlink root"); + cases.push(("root is a symlink to a populated dir", link_root)); + let dangling_root = base.join("dangling-root"); + symlink(base.join("nowhere"), &dangling_root).expect("dangling root"); + cases.push(("root is a dangling symlink", dangling_root)); + let locked = dir("locked"); + fs::write(locked.join("payload.bin"), b"x").expect("write payload"); + fs::set_permissions(&locked, fs::Permissions::from_mode(0o000)).expect("lock dir"); + if fs::read_dir(&locked).is_err() { + cases.push(("unreadable dir", locked)); + } else { + fs::set_permissions(&locked, fs::Permissions::from_mode(0o755)).ok(); + } + } + cases + } + + /// The single definition of a non-empty install root, over every input kind. + /// Each caller in `apps/rocm` goes through this function (see + /// `tarball_validation_follows_the_shared_payload_rule`, + /// `the_tarball_install_guard_follows_the_shared_payload_rule` and + /// `a_replacement_counts_as_installed_by_the_shared_payload_rule` there), so a + /// row changing here changes all three together. + #[test] + fn install_root_has_payload_answers_each_input_kind() { + let base = tempfile::tempdir().expect("scratch dir"); + let mut seen: Vec<(&str, String)> = cases(base.path()) + .into_iter() + .map(|(label, path)| { + let outcome = match install_root_has_payload(&path) { + Ok(true) => "payload", + Ok(false) => "empty", + Err(_) => "error", + }; + (label, outcome.to_owned()) + }) + .collect(); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + fs::set_permissions( + base.path().join("locked"), + fs::Permissions::from_mode(0o755), + ) + .ok(); + } + let mut expected: Vec<(&str, String)> = vec![ + ("missing path", "empty"), + ("empty dir", "empty"), + ("dotfile only", "empty"), + ("dot directory only", "empty"), + ("regular file", "payload"), + ("subdirectory", "payload"), + ("marker plus a regular file", "payload"), + ("root is a regular file", "error"), + ] + .into_iter() + .map(|(label, outcome)| (label, outcome.to_owned())) + .collect(); + #[cfg(unix)] + { + expected.extend([ + ("dangling symlink entry", "payload".to_owned()), + ("root is a symlink to a populated dir", "payload".to_owned()), + ("root is a dangling symlink", "empty".to_owned()), + ]); + // Absent from `seen` when running as root, which reads a mode-000 + // directory anyway; then the case is neither built nor asserted. + if seen.iter().any(|(label, _)| *label == "unreadable dir") { + expected.push(("unreadable dir", "error".to_owned())); + } + } + seen.sort(); + expected.sort(); + assert_eq!(seen, expected); + } + + #[test] + fn a_root_that_cannot_be_listed_names_the_path_in_its_error() { + let base = tempfile::tempdir().expect("scratch dir"); + let file = base.path().join("not-a-dir"); + fs::write(&file, b"x").expect("write file root"); + let error = install_root_has_payload(&file).expect_err("a file cannot be listed"); + assert!( + format!("{error:#}").contains(&format!("failed to read {}", file.display())), + "{error:#}" + ); + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 6beec99813..07209fb95a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -46,13 +46,13 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`dash.rs`](../apps/rocm/src/dash.rs) — full domain extraction. - [`dash_seam.rs`](../apps/rocm/src/dash_seam.rs) — full domain extraction. - [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: the command handlers for `rocm diagnose` and `rocm fix` (`diagnose()`, `fix()`) and the report step behind `rocm diagnose --report` (`show_prepared_report`, `perform_delivery`, `established_entry`). `diagnose()` and `fix()` are `pub(crate)` and re-imported in [`main.rs`](../apps/rocm/src/main.rs) for `dispatch()`; the other three functions are private. `diagnose()` gets its `--model` verdict from [`model_fit.rs`](../apps/rocm/src/model_fit.rs), and `fix()` returns a failing exit code through `FixExitCode`, which stays in `main.rs` with the other exit-code plumbing. The report and delivery regression tests live in this module's own `mod tests`; the exit-code tests stay in `main.rs`. -- [`driver_install.rs`](../apps/rocm/src/driver_install.rs) — mechanical relocation, owns types (part of [`main.rs`](../apps/rocm/src/main.rs)'s Phase 6a split, ROCMAI-82; see the Phase 6a variant note above): `install_driver()`/`reconcile_driver_install()`, running an approved plan's shell commands, and the install state persisted to `state.json` (`DriverInstallState`, `DriverReconciliationState` and the passive-check types). The plan it runs lives in [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs), which this module imports from and which does not import from it; the os-release the plan is built from comes from `real_os_release` in [`system_packages.rs`](../apps/rocm/src/system_packages.rs). Its `pub(crate)` surface is `install_driver` and `reconcile_driver_install`, `DriverInstallResult` and `DriverInstallError` (read from `install_cmd.rs`'s `install()`), `driver_install_state_path`, and a re-export of `host_install_driver_only_reports` for [`chat.rs`](../apps/rocm/src/chat.rs) and [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs). +- [`driver_install.rs`](../apps/rocm/src/driver_install.rs) — mechanical relocation, owns types (part of [`main.rs`](../apps/rocm/src/main.rs)'s Phase 6a split, ROCMAI-82; see the Phase 6a variant note above): `install_driver()`/`reconcile_driver_install()`, running an approved plan's shell commands, and the install state persisted to `state.json` (`DriverInstallState`, `DriverReconciliationState` and the passive-check types). The plan it runs lives in [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs), which this module imports from and which does not import from it; the os-release the plan is built from comes from `real_os_release` in [`hardware_root.rs`](../crates/rocm-core/src/hardware_root.rs), the one reader of the real host's os-release. Its `pub(crate)` surface is `install_driver` and `reconcile_driver_install`, `DriverInstallResult` and `DriverInstallError` (read from `install_cmd.rs`'s `install()`), `driver_install_state_path`, and a re-export of `host_install_driver_only_reports` for [`chat.rs`](../apps/rocm/src/chat.rs) and [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs). - [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs) — full domain extraction, crate-level (split out of [`driver_install.rs`](../apps/rocm/src/driver_install.rs)): the plan `rocm install driver` builds and prints. It owns the model (`DriverInstallPlan` with `execution_commands`, `DriverPlanCommand`, `DriverCommandPhase`, and `PrivilegeEscalation`, which decides whether a generated shell line gets a `sudo ` prefix), `render_driver_install_plan` with its approval label, the helpers the builders share (`driver_command`, `driver_root_preflight_checks`, `resolve_shell_default_template`, `is_safe_shell_token`), and the builders: `build_driver_install_plan_read`, which `install_driver` calls with the real host's os-release, and `build_driver_install_plan_at` beneath it, with the apt, dnf and SLES plans, the `ID_LIKE` fallback, the repository URL helpers, `resolve_repo_version_expr` (the `ROCM_CLI_AMDGPU_DRIVER_VERSION` override and the retired version variables it refuses), and the refusal plans for an unreadable os-release or an invalid override. It also owns `install_driver_only_reports_on`/`host_install_driver_only_reports` (true on native Windows, where the plan only reports); `driver_install.rs` re-exports the latter. It declares the child module [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs) and re-exports its `wsl_rocdxg_driver_plan`. Production code in this file uses only `std` and `rocm_core`, and nothing here depends on `driver_install.rs`. The model types and their fields, `execution_commands`, `PrivilegeEscalation::detect`, `driver_command`, `render_driver_install_plan`, `build_driver_install_plan_read`, `install_driver_only_reports_on` and `host_install_driver_only_reports` are `pub(crate)`, as is `invalid_driver_version_override_plan`, which a test that stays in `driver_install.rs` builds a plan with; the rest is private. Its test module is `pub(crate)` so `driver_install.rs`'s tests can reuse its `test_examine` and `os_release_found` helpers. It holds the plan-builder tests, including the one that keeps the supported-distro matrix in step with `dkms_driver_plan_is_supported` in `rocm-core`, and the tests of the template resolver and the shell-token filter. - [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs) — full domain extraction, a child of `driver_plan.rs` that depends on the plan model, `driver_command`, `driver_root_preflight_checks` and `resolve_shell_default_template` in its parent, and on `truthy_env` in [`therock.rs`](../apps/rocm/src/therock.rs) (split out of [`driver_install.rs`](../apps/rocm/src/driver_install.rs)): the WSL2 rocDXG plan. `wsl_rocdxg_driver_plan` builds the plan that installs the `rocdxg-roct` package, `resolve_rocdxg_verification` decides how that download is authenticated (a supplied digest, the digest pinned for the version in `ROCDXG_PINNED_DIGESTS`, or the explicit opt-out), and `wsl_rocdxg_refusal_plan` is the plan that installs nothing and says why when it cannot be authenticated or the version is not a well-formed package version. `ROCDXG_PINNED_DIGESTS` is the trust anchor an unattended root install is checked against, so a change to it is a security-relevant diff; it moved byte for byte. `wsl_rocdxg_driver_plan` is `pub(crate)` and re-exported by `driver_plan.rs` (its builders call it, and tests in `driver_install.rs` use it as a plan fixture); everything else in production code is private. Its tests are the `wsl_rocdxg_*` plan tests, including the one that runs the generated digest step against a real file on Unix. The test-only `scoped_rocdxg_env` helper that clears the rocDXG environment lives here too, `pub(super)` for the plan-builder tests in `driver_plan.rs`. - [`endpoint_keys.rs`](../apps/rocm/src/endpoint_keys.rs) — mechanical relocation, no owned types, reached by qualified path. - [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) — mechanical relocation out of [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), no owned types: which runtime an engine installs into and where its environment lives. `engine_manages_own_runtime` says whether an engine brings its own runtime; `env_root_for_engine_install`, `env_root_for_runtime` and `env_root_for_self_managed_engine` resolve the engine environment root; `runtime_manifest_for_selector` resolves a selector to one installed runtime; `runtime_key_for_python` names the runtime whose install root contains an interpreter (the private `runtime_key_owning_python` is the decision behind it). It is the neutral layer that [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs) and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) (with its `torch_alignment.rs` child) both import from, so neither imports the other for these helpers. It imports `recover_setup_runtime_registration` from [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs), `select_runtime_manifest` and `validate_runtime_manifest_for_activation` from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and the runtime-manifest model from [`therock.rs`](../apps/rocm/src/therock.rs); it does not go through the re-exports in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs), which would lead back to it via `runtime_services.rs`. Its functions are `pub(crate)` for those two modules plus [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) and [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs). Their tests moved with them. - [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-213): resolving an engine's binary path (`resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `find_engine_plugin_binary`), the stdio request/response envelope used to talk to an external engine binary or the in-process built-in handler (`engine_request`, `engine_request_with_env_root`, `decode_engine_response`, the `ScopedEnvVar`/`with_scoped_builtin_engine_env` env-scoping helpers and the `BUILTIN_ENGINE_ENV_LOCK` mutex they serialize on, `builtin_engine_request`, `run_builtin_engine_stdio`), and the built-in HTTP-serving path (`builtin_engine_available`, `run_builtin_engine_serve_http`, `builtin_engine_serve_http_args`). One item that sits physically inside this span in `main.rs` stayed there instead of moving: `impl From for WatcherMode` is unrelated clap-arg conversion that merely happened to be interleaved between `find_engine_plugin_binary` and `engine_request` (it has since moved to [`cli.rs`](../apps/rocm/src/cli.rs), next to the enum it converts). `resolve_engine_binary_path`, `resolve_engine_binary_path_with_paths`, `engine_request`, `engine_request_with_env_root`, `builtin_engine_available`, and `builtin_engine_serve_http_args` are `pub(crate)` and read from other modules (for example [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and [`service_launch.rs`](../apps/rocm/src/service_launch.rs)), not from `main.rs`. `run_builtin_engine_stdio` and `run_builtin_engine_serve_http` are `pub(crate)` and reached directly from `main.rs`'s `dispatch()` (the `Command::EngineStdio` and `__engine-serve-http` arms); `find_engine_plugin_binary`, `decode_engine_response`, `with_scoped_builtin_engine_env`, `builtin_engine_request`, and `ScopedEnvVar` have no callers outside this module and stay private. -- [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-215): deciding which runtime/env a `serve`/`engines install` invocation resolves to (`EngineSelection`, `resolve_engine_selection`, `validate_engine_selection_runtime`, `single_ready_runtime_key`, `resolve_runtime_selector_to_exact_key`), small argv-building helpers used when relaunching a managed engine (`optional_arg`, `engine_recipe_json_arg`, `parse_engine_recipe_json_arg`, `daemon_run_argv`, `managed_service_launcher_path`), the `ROCM_CLI_*` environment variables every managed child inherits (`app_path_env_vars`/`app_path_env_var_values`/`app_path_env_var_refs`/`apply_app_path_env`), and polling a freshly launched service until it can answer inference requests (`wait_for_service_http_ready`, `wait_for_service_http_ready_with_progress`, `service_http_readiness_paths`, `http_get_local_service`, `http_post_local_service_json`, `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`, `model_list_ready_for_model`, `service_model_gpu_ready`, `service_model_reports_rocm_backend`, `service_model_names_match`). `const fn status_for_readiness` sits physically interleaved in this span in `main.rs` but stayed there instead of moving — it has callers outside this cluster ([`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `record_restart_readiness`, which `restart_internal_managed_service_with` calls), so it stays a shared crate-root function, reached only from this module's own tests, via `use crate::status_for_readiness;`. Which items are `pub(crate)` and who calls them is documented in the module's own doc comment rather than repeated here. The cluster's regression tests live in the module's own `mod tests`. What remains in `main.rs` immediately after this cluster is `treat_as_natural_language`, dispatch-adjacent natural-language-invocation routing left over from an earlier phase (ROCMAI-87), explicitly out of scope and untouched by this split. +- [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-215): deciding which runtime/env a `serve`/`engines install` invocation resolves to (`EngineSelection`, `resolve_engine_selection`, `validate_engine_selection_runtime`, `single_ready_runtime_key`, `resolve_runtime_selector_to_exact_key`), small argv-building helpers used when relaunching a managed engine (`optional_arg`, `engine_recipe_json_arg`, `parse_engine_recipe_json_arg`, `daemon_run_argv`, `managed_service_launcher_path`), the `ROCM_CLI_*` environment variables every managed child inherits (`app_path_env_vars`/`app_path_env_var_values`/`app_path_env_var_refs`/`apply_app_path_env`), and polling a freshly launched service until it can answer inference requests (`wait_for_service_http_ready`, `wait_for_service_http_ready_with_progress`, `service_http_readiness_paths`, `http_get_local_service`, `http_post_local_service_json`, `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`, `model_list_ready_for_model`, `service_model_gpu_ready`, `service_model_reports_rocm_backend`, `service_model_names_match`). `single_ready_runtime_key` does not decide which runtimes count as ready: it asks [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs)'s `select_single_ready_runtime`, which `rocm comfyui install` shares. `const fn status_for_readiness` sits physically interleaved in this span in `main.rs` but stayed there instead of moving — it has callers outside this cluster ([`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `record_restart_readiness`, which `restart_internal_managed_service_with` calls), so it stays a shared crate-root function, reached only from this module's own tests, via `use crate::status_for_readiness;`. Which items are `pub(crate)` and who calls them is documented in the module's own doc comment rather than repeated here. The cluster's regression tests live in the module's own `mod tests`. What remains in `main.rs` immediately after this cluster is `treat_as_natural_language`, dispatch-adjacent natural-language-invocation routing left over from an earlier phase (ROCMAI-87), explicitly out of scope and untouched by this split. - [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs) — mechanical relocation, owns private types (Phase 6a): `engines()` (the `rocm engines` command), `ensure_self_managed_engine_ready` (the readiness check [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) runs for engines that bring their own runtime) and `env_root_for_service` (read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)). Which runtime an engine installs into and where its environment lives is answered in [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs), which this module imports from. This module also calls into [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) for `settle_engine_install` and the host system-package installs; that module does not import this one. - [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) — mechanical relocation, dispatch-adjacent, owns private types: `rocm examine`'s command handler (`examine()`), which prints the human report through [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `examine_human_report`, or with `--json` the document built from the private `ExamineJson`/`ExamineJsonSummary` types (the `Examination` verbatim plus a summary of what the CLI knows but the probe cannot). `examine()` is `pub(crate)` and re-imported in [`main.rs`](../apps/rocm/src/main.rs) for `dispatch()`. No test in the crate calls it or builds the document directly. - [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) — mechanical relocation: the launch-summary/examine text surfaces (`render_launch_summary`, `render_examine_text`, `render_examine_text_with_paths`, `examine_human_report`, `render_examine_plain_header`, `plain_status_label`) and the engine-inventory listing shared by `rocm engines` and `rocm examine` (`write_default_engine_legend`, `render_engine_inventory_text_with_paths`, `append_examine_runtime_state`, `append_examine_engine_inventory`, `append_engine_detect_summary`, `friendly_engine_runtime_kind`, `friendly_engine_detect_notes`, `friendly_engine_detect_note_fallback`, `engine_runtime_status_label`). The `DEFAULT_ENGINE_MARKER` constant is shared by both engine-inventory renderers so the two copies cannot drift apart. The `rocm model` catalog renderer is not here; it lives in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), and nothing in this file calls into it outside its own tests. [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs)'s `examine()`, which the `Command::Examine` arm calls, imports and calls `examine_human_report`; `main.rs`'s own `launch_default` calls `render_launch_summary`, re-imported there. `render_examine_text`, `render_examine_text_with_paths` and `render_engine_inventory_text_with_paths` are `pub(crate)`, read from [`chat.rs`](../apps/rocm/src/chat.rs)'s deterministic tool-output summaries; `render_engine_inventory_text_with_paths` is also read from [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s `render_engine_inventory_text`; `append_examine_runtime_state` is also `pub(crate)`, read only from [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s own test module, so `main.rs`'s re-import of it is itself `#[cfg(test)]` rather than carrying an otherwise-unused import into a non-test build. Every other item in this list has no caller outside this module and stays private. This module also reaches items that live at the crate root because they have other callers: `engine_inventory()` (also read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) and [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` (defined in [`model_fit.rs`](../apps/rocm/src/model_fit.rs) and re-imported in `main.rs`) — all reached via `use crate::{...};`. The regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime`/`runtime_manifest_path` through `crate::test_support::`/`crate::runtimes_cmd::`. One test here, `linux_only_engine_runtime_status_is_explicit_on_native_windows`, also calls `model_registry_adapter_availability_note` from `model_catalog_text.rs`, which is `pub(crate)` for that reason; its primary subject is `engine_runtime_status_label`, so it stays. @@ -72,10 +72,10 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`providers/local_http.rs`](../apps/rocm/src/providers/local_http.rs) — full domain extraction, a child of `providers.rs`: the local provider's hand-rolled HTTP/1.1 transport. `post_json_to_local_endpoint` (with `post_json_to_local_endpoint_body` and `local_bearer_header`) is the blocking JSON POST; `stream_json_from_local_endpoint` is the streamed POST, which reads the response in whichever body framing the engine answers with (`read_sse_body` picks between `read_chunked_sse_body`, `read_fixed_sse_body` and `read_close_delimited_sse_body` from the headers, via `header_value`) and feeds an `OpenAiSseEmitter`; `body_truncated_at_eof` and `LOCAL_STREAM_TRUNCATED` name the failure when an engine closes the connection mid-answer; `parse_http_endpoint` reads the `http://` endpoint URL. `post_json_to_local_endpoint` and `stream_json_from_local_endpoint` are `pub(super)`, called from `LocalProvider`, and `LOCAL_STREAM_TRUNCATED` is also `pub(super)` (imported by `providers.rs` for its test module only); the rest is private. Its edges back into `providers` are `OpenAiSseEmitter` from `providers/wire.rs`, the `SseBytesEmitter` trait the emitter is pushed bytes through (which stays in `providers.rs`), and the shared `ProviderStreamEvent` type; nothing in `wire.rs` reads `local_http.rs`, so the edge is one-way. [`providers/local_http/proptests.rs`](../apps/rocm/src/providers/local_http/proptests.rs) holds the property tests of the SSE body readers and is declared from this module so they sit with the readers they test; the example-based test of `parse_http_endpoint` is in this module's own `mod tests`, and the tests that drive `LocalProvider` end to end against a mock server stay in `providers.rs`'s `mod tests` with their server helpers. - [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) — full domain extraction, a child of `providers.rs`: the OpenAI-compatible and Anthropic wire formats. The request-body builders (`openai_chat_request_body`, `openai_stream_chat_request_body`, `openai_local_chat_request_body`, `openai_local_stream_chat_request_body`, `anthropic_chat_request_body`, `anthropic_stream_chat_request_body`, and the helpers behind them, including `local_openai_compatible_request`, `apply_local_model_request_defaults` and `local_model_should_disable_thinking`), the parsers for non-streaming responses (`parse_openai_chat_message`, `parse_anthropic_chat_message`), and the SSE emitters with their line parsers (`OpenAiSseEmitter` and `AnthropicSseEmitter`, which implement the `SseBytesEmitter` trait that stays in `providers.rs`; `parse_openai_sse_line`, `parse_anthropic_sse_line`). The builders, the two chat-message parsers and the two emitters (with their `new` and `finish`) are `pub(super)`, called from `LocalProvider` and `RemoteProvider`; `OpenAiSseEmitter` is also read by `providers/local_http.rs`. Depends on the shared request/response types in `providers.rs` and on `providers/tool_schema.rs`, and on nothing else. The tests of the builders and parsers live in this module's own `mod tests`, with the tool-catalog assertions that read through the request bodies, and [`providers/wire/sse_emitter_proptests.rs`](../apps/rocm/src/providers/wire/sse_emitter_proptests.rs), the split-invariance and panic-guard properties of the emitters and line parsers, is declared from this module; the stream tests that run an emitter against a mock server stay in `providers.rs`'s `mod tests`. - [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) — the argv helpers that `chat.rs`, `tool_policy.rs` and the freeform planner share, in a leaf module whose only `crate::` dependency is the leaf [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs): `chat_cli_arg_value`, `chat_cli_arg_value_checked`, `chat_cli_has_flag` and `ensure_flag` (from `chat.rs`), and `format_structured_tool_call` (which quotes each argument with `shell_quote`), `serve_args_request_cpu_device` and `device_policy_is_cpu_mode` (from `freeform_planner.rs`). They moved so that `freeform_planner.rs` no longer imports `chat` for them; `chat.rs` re-exports `ensure_flag` (read by `freeform_execution.rs`) and `freeform_planner.rs` re-exports `format_structured_tool_call`, so every other caller keeps its path. -- [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): registering a runtime that `rocm install sdk` did not install. `import_runtime_manifest` records a manifest written elsewhere, `adopt_runtime_from_python_options` (with `AdoptRuntimeOptions`, `AdoptRuntimeRequest`, `infer_adopt_runtime_request` and `adopt_runtime_from_probe`) probes an existing Python environment and registers it, and `recover_setup_runtime_registration` registers the runtime `rocm setup` created, with `newest_installed_runtime_manifest` and `setup_runtime_paths` beside it. It depends on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (manifest paths, activation validation, `paths_equivalent`) and `therock.rs`, not on `runtimes_cmd.rs`, which calls into it and re-exports `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths`. `import_runtime_manifest`, `AdoptRuntimeOptions`, `adopt_runtime_from_python_options` (with the fields of `AdoptRuntimeOptions` and `AdoptRuntimeRequest`) and `adopt_runtime_from_probe` are `pub(crate)` for `runtimes()` and for the `runtimes_cmd.rs` tests that adopt or import a runtime and then list or activate it; `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths` are `pub(crate)` too, for the modules that import them through `runtimes_cmd.rs`'s re-exports; the rest is private. Its tests cover the import and adopt rules that do not involve activation or listing. +- [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): registering a runtime that `rocm install sdk` did not install. `import_runtime_manifest` records a manifest written elsewhere, `adopt_runtime_from_python_options` (with `AdoptRuntimeOptions`, `AdoptRuntimeRequest`, `infer_adopt_runtime_request` and `adopt_runtime_from_probe`) probes an existing Python environment and registers it, and `recover_setup_runtime_registration` registers the runtime `rocm setup` created, with `newest_installed_runtime_manifest` and `setup_runtime_paths` beside it. `select_single_ready_runtime` is the one rule for picking "the one ready runtime" when nothing else says which to use: it recovers the setup runtime first, then returns a `SingleReadyRuntime` (`One`, `Zero` with every registered runtime, or `Many` with the ready ones), where ready is `validate_runtime_manifest_for_activation`; its pure half is `classify_single_ready_runtime`. A setup runtime that cannot be used is not a ready runtime and is not counted: it does not stop a registered ready runtime from being picked (or refused as ambiguous), and its own error is returned only when the registry holds no ready runtime, while a failure to write a usable setup runtime's entry back is always returned. Its `SetupRecovery` argument says whether recovering the setup runtime may write its lost registry entry back (`Persist`, what the engine paths and a real ComfyUI install use) or only count the runtime from its own manifest and write nothing (`PreviewOnly`, for a `--dry-run` preview, which must name the runtime the real run will use without changing a file); `SetupRecovery::for_dry_run` maps the flag. `registered_runtimes_with_setup` is the list that rule counts from (the registry plus the setup runtime, written back or not as `SetupRecovery` says), which `comfyui/runtime_env.rs` also searches when a choice that names a runtime is not answered by the registry alone. Both [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) (which treats `Zero` and `Many` as no selection and leaves the advice to its callers) and [`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs) (which words them as refusals) call it directly rather than through `runtimes_cmd.rs`, and neither keeps its own copy. It depends on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (manifest paths, activation validation, `paths_equivalent`) and `therock.rs`, not on `runtimes_cmd.rs`, which calls into it and re-exports `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths`. `import_runtime_manifest`, `AdoptRuntimeOptions`, `adopt_runtime_from_python_options` (with the fields of `AdoptRuntimeOptions` and `AdoptRuntimeRequest`) and `adopt_runtime_from_probe` are `pub(crate)` for `runtimes()` and for the `runtimes_cmd.rs` tests that adopt or import a runtime and then list or activate it; `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths` are `pub(crate)` too, for the modules that import them through `runtimes_cmd.rs`'s re-exports; the rest is private. Its tests cover the import and adopt rules that do not involve activation or listing. - [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) — mechanical relocation, owns types (part of [`main.rs`](../apps/rocm/src/main.rs)'s Phase 6c split, ROCMAI-89, alongside [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) below): settling an engine onto a runtime after an SDK install: the orchestration (`finish_sdk_install`, `settle_engine_install`, `finalize_successful_sdk_install`), the engine-dependency check, the runtime device check and the post-install verdict built on it, and `SdkInstallConsents`. The torch alignment and retention it runs live in the child module [`runtime_install_finalize/torch_alignment.rs`](../apps/rocm/src/runtime_install_finalize/torch_alignment.rs). `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`; `settle_engine_install` is `pub(crate)` and imported by [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), so the import edge between the two runs one way, from `engines_cmd.rs` to this module. This module (and its torch-alignment child) does not import `engines_cmd.rs`: the engine environment and runtime-selection helpers it needs (`engine_manages_own_runtime`, `env_root_for_engine_install`, `runtime_key_for_python`, and `runtime_manifest_for_selector` from the torch-alignment child) are imported from [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs). The host system-package installs live in [`system_packages.rs`](../apps/rocm/src/system_packages.rs); this module re-exports `ensure_libatomic_for_torch`/`ensure_libnuma_for_torch` (called from both `install_cmd.rs`'s `install()` and `engines_cmd.rs`) and `ensure_openmpi_for_vllm` (called from `engines_cmd.rs` and, for the engine auto-install after an SDK install, from here) so those callers keep their imports. - [`runtime_install_finalize/torch_alignment.rs`](../apps/rocm/src/runtime_install_finalize/torch_alignment.rs) — full domain extraction, a child of `runtime_install_finalize.rs` that depends on its parent's device check (`RuntimeDeviceCheck`, `device_check_verdict`) and on `runtime_manifest_for_selector` in [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): what to do about the torch an engine install finds in a managed runtime. `plan_torch_alignment` decides whether the SDK's torch must be realigned to the engine's pin, `align_runtime_torch` runs the realignment, `classify_retained_torch` decides when a torch that already works is kept as an intended divergence instead, and the `render_*` and `report_*` functions with `TorchAlignment` and `TorchRetention` print and log the outcome. The manifest readers (`sdk_torch_build_for_key`, `sdk_torch_build_from_manifest`, `runtime_python_for_key`, `runtime_library_paths_for_key`, `runtime_index_url_for_key`) live here too. The parent stays the caller (`settle_engine_install` and `maybe_auto_install_sdk_preferred_engine`), so it could not be a crate-level sibling without a cycle with the parent's device check. The items the parent calls are `pub(super)`: `TorchAlignment`, `TorchRetention`, `report_torch_alignment`, `report_torch_retention`, `classify_retained_torch`, `retained_diverged_package`, `deliberately_diverged_package`, `sdk_torch_build_for_key`, `runtime_python_for_key` and `runtime_library_paths_for_key`; the rest is private. Its tests cover planning, rendering, the opt-out and the retention rules, and reach the parent's private engine-dependency check as a descendant of it. -- [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) — full domain extraction, crate-level (split out of `therock.rs`): the installed-runtime manifest model (`InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `RocmSdkPackageProbe`, `RocmSdkLibraryProbe`, whose serde shape is the on-disk contract of the registry files and each install's `.rocm-cli-runtime.json`), where those files live (`runtime_registry_dir`, `runtime_manifest_path`), writing and loading them (`save_runtime_manifest`, `load_runtime_manifests`, `load_runtime_manifests_reporting_unparsed`, `sort_manifests_newest_install_first`), the composition readers (`wheel_composition_includes_devel`, `wheel_composition_device_target`), the probe validator (`validate_rocm_sdk_runtime_probe`), and registry selection and validation (`current_runtime_manifest`, `default_runtime_id_matches`, `runtime_keys_text`, `select_runtime_manifest`, `runtime_usability_status`, `validate_runtime_manifest_for_activation`, and the wheel/tarball validators behind it). Lives at the crate level and depends on neither `therock.rs` nor `runtimes_cmd.rs`, so both build on it. [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) still depends on `therock.rs` (it reaches the manifest types and loaders through `therock.rs`'s re-exports), but `therock.rs` no longer depends on `runtimes_cmd.rs` in production code (one of its tests still calls into it). It also holds `paths_equivalent` and `normalize_path_for_compare`, the path comparison that [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs), [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) and `runtimes_cmd.rs` need, and the on-disk active-runtime marker: `ActiveRuntimeMarker` (user state, so its serde shape is as fixed as the manifest's; its fields are `pub(crate)` because `runtimes_cmd.rs` builds it on activation and rollback), `active_runtime_marker_path` and `active_runtime_marker_matches`, which uninstall uses to decide whether the runtime it removes was the active one. `write_active_runtime_marker` stays in `runtimes_cmd.rs` with the activation transaction. `runtimes_cmd.rs` re-exports the selection and path-helper names that other modules import from it. Its own tests cover the manifest round-trip, the `devel`/composition readers, and runtime selector resolution. +- [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) — full domain extraction, crate-level (split out of `therock.rs`): the installed-runtime manifest model (`InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `RocmSdkPackageProbe`, `RocmSdkLibraryProbe`, whose serde shape is the on-disk contract of the registry files and each install's `.rocm-cli-runtime.json`), where those files live (`runtime_registry_dir`, `runtime_manifest_path`), writing them (`write_runtime_manifest`, the one routine every way of registering a runtime goes through: an SDK install, `rocm runtimes import`, `rocm runtimes adopt` and the recovery of the runtime `rocm setup` created. It writes the registry entry, and for a runtime ROCm CLI owns (`InstalledRuntimeManifest::owns_install_root`, the same rule uninstall and prune use to decide whether they may delete the folder) the `.rocm-cli-runtime.json` in its install root; an imported or adopted runtime is someone else's folder, so it gets the registry entry only. The registry entry is written last, so a failed write of either file never leaves a new entry behind and an error that says a runtime could not be registered or restored is true. Its one parameter is `ManifestOverwrite`: an install replaces its own entry, import and adopt refuse an existing one unless `--replace` is given, and recovery refuses so it only fills a missing entry. Neither file is written atomically) and loading them (`load_runtime_manifests`, `load_runtime_manifests_reporting_unparsed`, `sort_manifests_newest_install_first`), the composition readers (`wheel_composition_includes_devel`, `wheel_composition_device_target`), the probe validator (`validate_rocm_sdk_runtime_probe`), and registry selection and validation (`current_runtime_manifest`, `default_runtime_id_matches`, `runtime_keys_text`, `select_runtime_manifest`, `runtime_usability_status`, `validate_runtime_manifest_for_activation`, and the wheel/tarball validators behind it; the tarball one asks `rocm_core::install_root_has_payload` whether the root holds a payload). Lives at the crate level and depends on neither `therock.rs` nor `runtimes_cmd.rs`, so both build on it. [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) still depends on `therock.rs` (it reaches the manifest types and loaders through `therock.rs`'s re-exports), but `therock.rs` no longer depends on `runtimes_cmd.rs` in production code (one of its tests still calls into it). It also holds `paths_equivalent` and `normalize_path_for_compare`, the path comparison that [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs), [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) and `runtimes_cmd.rs` need, and the on-disk active-runtime marker: `ActiveRuntimeMarker` (user state, so its serde shape is as fixed as the manifest's; its fields are `pub(crate)` because `runtimes_cmd.rs` builds it on activation and rollback), `active_runtime_marker_path` and `active_runtime_marker_matches`, which uninstall uses to decide whether the runtime it removes was the active one. `write_active_runtime_marker` stays in `runtimes_cmd.rs` with the activation transaction. `runtimes_cmd.rs` re-exports the selection and path-helper names that other modules import from it. Its own tests cover the manifest round-trip, the `devel`/composition readers, and runtime selector resolution. - [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) — full domain extraction; reads the live local servers at activation time, classifies each against the runtime being activated, renders the report, and moves stale ones with `--restart-services`. `fn runtimes()` lives in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) and `RuntimesCommand` in [`cli.rs`](../apps/rocm/src/cli.rs); this file owns `ServiceRuntimeState`/`RuntimeServiceEntry`/`FailedServiceRestart`/`UnconfirmedServiceRestart`/`ServiceRestartOutcome`/`RuntimeServiceReconciliation`/`ServiceRuntimePin`, reached from `runtimes_cmd.rs` via `crate::runtime_services`. - [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): `rocm runtimes uninstall` and the storage pruning that removes a registered runtime. `plan_runtime_uninstall` builds the plan (`RuntimeUninstallPlan`, with the `InstallRootDecision` for its folder), `print_runtime_uninstall_plan` shows it, `confirm_and_revalidate_runtime_uninstall` asks and then checks that nothing changed while the prompt was open (`revalidate_runtime_uninstall_plan`), and `apply_runtime_uninstall` unregisters the runtime, clears the active and default state that named it and removes its install root when `should_remove_runtime_install_root` says ROCm CLI owns it (`local_runtime_manifest_matches`, `ensure_runtime_install_root_is_safe_to_remove`). `uninstall_runtime` is the non-interactive entry point. It reads the registry, the active-runtime marker and the path comparison from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and does not depend on `runtimes_cmd.rs`, which calls into it from `runtimes()` and re-exports `uninstall_runtime`, `should_remove_runtime_install_root` and `InstallRootDecision`. The planning and apply entry points, `RuntimeUninstallPlan`, `RuntimeUninstallConfirmation` and `RuntimeUninstallResult` (with its fields) are `pub(crate)` for `runtimes()`; `revalidate_runtime_uninstall_plan` and the folder-safety helpers are private. Its tests cover planning, revalidation, the folder-removal rules and uninstalling a read-only imported runtime; the uninstall tests that first activate a runtime stay with the activation tests in `runtimes_cmd.rs`. - [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) — mechanical relocation, owns types (Phase 6c, ROCMAI-89): `runtimes()` and its activate/rollback handlers (the activation transaction, including `write_active_runtime_marker`), plus the runtime list rendering; the import and adopt handlers it dispatches to live in [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) and the uninstall plan, confirmation and apply in [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs); registry selection and manifest validation moved to [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), which this module re-exports for the callers that import them from here. `RuntimesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) (part of the `Command`/`Cli` clap definition, the same way `EnginesCommand` does). `RuntimeActivationResult` is `pub(crate)`, read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary`; `uninstall_runtime`, `should_remove_runtime_install_root` and `InstallRootDecision` (from `runtime_uninstall.rs`) and `ActiveRuntimeMarker` and `active_runtime_marker_path` (from `runtime_registry.rs`) are re-exported for [`storage.rs`](../apps/rocm/src/storage.rs) and [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs); `recover_setup_runtime_registration`, `newest_installed_runtime_manifest` and `setup_runtime_paths` (from `runtime_adopt.rs`) and `normalize_path_for_compare` (from `runtime_registry.rs`) are re-exported for the modules that import them from here, such as [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs), [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) and [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs). [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) is deliberately not one of them: it imports from `runtime_adopt.rs` and `runtime_registry.rs` directly, because going through this module would lead back to it via `runtime_services.rs`. The dependency on the sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) above runs the other way: that module calls into this one (`activate_runtime`, `newest_installed_runtime_manifest`, `recover_setup_runtime_registration`), while this module reaches it only from its test module. @@ -92,8 +92,8 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`setup_cmd.rs`](../apps/rocm/src/setup_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm setup`'s command handler (`setup()`, which runs `setup status` by default and `setup reset`), the text `rocm setup status` prints (`render_setup_status_text`) and the reset that clears the saved first-time prompt state (`reset_setup_prompt_state`). `SetupCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `setup()` and `render_setup_status_text` are `pub(crate)` and re-imported in `main.rs`: `dispatch()` calls the first, and [`chat.rs`](../apps/rocm/src/chat.rs)'s read-only handling of `rocm setup` and `rocm setup status` reaches the second as `crate::render_setup_status_text`; `reset_setup_prompt_state` is private. The three regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime` through `crate::test_support::`. - [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) — shared CLI-output component (`shell_quote`): the CLI's name for `rocm-core`'s POSIX quoter, [`posix_shell_quote`](../crates/rocm-core/src/quoting.rs). `rocm-dash-tui` keeps a copy as `ui::exec::quote_display_arg` in [`exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs), since the crate-edge check forbids it `rocm-core`; a per-character walk and a property test here hold the two to the same output. It covers the remote `ssh` command line, the chat approval card and the plan for a free-form request, the install dry-run `command:` line, the [`service_hints.rs`](../apps/rocm/src/service_hints.rs) hints, the runtime-activation report, the `rocm services logs ` lines `rocm services` lists and the `rocm comfyui install --runtime-id` advice a failed ComfyUI reinstall prints — not every `rocm …` hint the CLI prints; older ones such as `rocm logs ` still interpolate their value bare. Also holds `has_terminal_control_character`, the predicate the chat tool-call guards refuse on. - [`storage.rs`](../apps/rocm/src/storage.rs) — full domain extraction. -- [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and `real_os_release`, the real host's os-release that every one of these plans is built from. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions and `real_os_release` are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), and [`driver_install.rs`](../apps/rocm/src/driver_install.rs) imports `real_os_release` from here directly. `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `real_os_release` ignores a simulated host root. -- [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token`, `slugify` and `has_nontrivial_directory_contents`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read six of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key`, `wheel_runtime_key` and `has_nontrivial_directory_contents` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). +- [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and reads the os-release every one of these plans is built from through `rocm_core::hardware_root::real_os_release`, which has no copy here. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs). `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `rocm_core::hardware_root::real_os_release` and the three OpenMPI/libatomic/libnuma install hints ignore a simulated host root, and a scan that no source file of the binary reads os-release by hand. +- [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token` and `slugify`, plus `ensure_tarball_install_target_is_clean`, the refusal to extract a tarball into a root that already holds a payload. What "holds a payload" means is not defined here: it is `rocm_core::install_root_has_payload`, the one rule shared with `runtime_registry.rs` and `therock/update.rs`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read five of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key` and `wheel_runtime_key` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). - [`therock/version.rs`](../apps/rocm/src/therock/version.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): the version grammar behind runtime selection and update checks. `parse_version` is the strict gate an install is routed on; `parse_version_for_ordering`, `compare_version_strings` and `version_relation` implement the PEP 440 ordering and the "is this a different version" question; `parse_host_version` reads a host-reported ROCm version through the same parser; `runtime_version_display`, `runtime_version_build_date`, `normalize_requested_build_date` and the calendar helpers behind them handle build dates; `is_stable_runtime_version` and the `ParsedVersion`/`VersionOrderKey`/`VersionStage` types back them. The items `therock.rs` and its property-test module use are `pub(super)` (including the `ParsedVersion` and `VersionOrderKey` fields they read); the parsing and calendar helpers behind them are private; and `runtime_version_display` is `pub(crate)`, re-exported by `therock.rs` for the other modules that render versions. Depends on nothing else in `therock`. `therock_version_proptests.rs` stays declared from `therock.rs` because it also reaches update, wheel and registry code. - [`therock/io.rs`](../apps/rocm/src/therock/io.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): unique scratch directories (`linux_temp_dir`, `windows_temp_dir`) and the stage-then-publish atomic file writes (`write_file_atomically`, `stage_file_for_atomic_publish`, `temp_sibling_path`, and the platform `publish_temp_file`/`replace_file_windows` pair) for the metadata cache entries and the staged metadata public key, the startup update-check record and the active-runtime marker; runtime registry manifests do not go through them. `write_file_atomically` is `pub(crate)` and re-exported by `therock.rs` for [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs), which writes the active-runtime marker with it; the rest is `pub(super)` or private. The `cfg(windows)` publish and replace paths compile only on the Windows lane. - [`therock/process.rs`](../apps/rocm/src/therock/process.rs) — full domain extraction, a child of `therock.rs` that depends on `therock/io.rs` for scratch directories (split out of `therock.rs`): running and capturing child commands (`run_command*`, `capture_command_output*`, `capture_python_stdout*`, `run_uv_progress_command`, and the Windows temp-file variants with `windows_child_path`), the bounded wait (`wait_with_output_bounded`), the progress-line output with its `--json` suppression guard (`SuppressProgressOutput`, `progress_line`), and unpacking the SDK tarball (`extract_tarball`, `extract_tarball_and_discard_archive`, which is why it sits here and not in `io.rs`: it spawns `tar` through `run_command`). Its `archive_properties` property tests run on Unix only. The process-wide environment lock these tests take is the single `PROCESS_ENV_TEST_LOCK` in [`test_support.rs`](../apps/rocm/src/test_support.rs), shared with the tests left in `therock.rs`. @@ -106,7 +106,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/fetch.rs`, `therock/family.rs`, `therock/source.rs` and `therock/version.rs` (split out of `therock.rs`): resolving the SDK tarball a request installs. `resolve_tarball_artifact` (and `resolve_tarball_artifact_with_timeout`, which takes the metadata download timeout) resolves the family and source layout, downloads the catalog through the cached metadata fetch, and returns the `TarballArtifact` it chose; `parse_tarball_index_html` reads the catalog's embedded file list into `TarballIndexFile`s and `validate_tarball_file_name` refuses an entry that is not a plain file name; `select_tarball_candidate` picks the newest archive the request can install, with `tarball_family_token` spelling the family the way the next layout's catalog does. `platform_tarball_token`, which names the archives' platform, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. Only `resolve_tarball_artifact*` and `TarballArtifact` are visible beyond this file. - [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/family.rs`, `therock/fetch.rs`, `therock/process.rs`, `therock/python.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for `WheelRuntimeComposition` (split out of `therock.rs`): resolving the wheel stack a request installs. `resolve_pip_runtime` (and `resolve_pip_runtime_with_timeout`, which takes the metadata download timeout and an optional source layout) resolves the family and the layout, reads the aggregate root, decides the device payload and returns a `PipRuntimeResolution`; on the canonical layout `select_matching_pip_package_versions` then takes the newest `rocm` whose `torch`, `torchvision` and `torchaudio` listings agree, and on the next layout `torchaudio_pin` pins a torchaudio first (the hardware-verified release when it is published, read from the vLLM engine's table) and `resolve_published_pip_package_versions` has `uv pip compile` choose the rest. `wheel_runtime_composition` turns a resolution into the exact package specs (`therock_pip_package_specs`) an install gives `uv` and records in the manifest, `published_pip_requirements` is the matching requirement set for `uv pip compile`, and `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` listing so the interpreter can be picked before resolution. Beneath that, `parse_aggregate_package_links` and `parse_aggregate_device_targets` read the package links of the canonical aggregate root and the exact device targets it publishes, and `validate_aggregate_index_layout` refuses a root that lacks the expected packages or any device payload; `AggregateDeviceTarget::resolve` picks the one exact target a host gets, or records why it could not (`Undetermined`); `load_simple_index_versions` fetches a package's simple-index listing through the cached metadata download and `parse_simple_index_versions` reads the versions out of it, dropping wheels the interpreter's tags do not accept when given them (`wheel_stem_matches_compatibility` and the `wheel_*_tag_matches` helpers); `package_release_key` orders the PEP 440 versions found there (`PackageReleaseKey`). `slugify`, which keys the cached listing, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The functions and types `therock.rs` and `therock/update.rs` name are `pub(in crate::therock)` and reach them through `therock/resolve.rs`; `PipRuntimeResolution` and `TheRockPipPackageVersions` are `pub(in crate::therock)` with the fields they read, and `TorchaudioPin` with its `describe`, so they can use what `resolve_pip_runtime` returns without naming the types; the rest is private. - [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` and `therock/version.rs` and reaches the registry through [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (split out of `therock.rs`): the consent gate for an install that would displace the active default runtime. `active_default_runtime_relation` says what the install would replace (an upgrade, downgrade or reinstall when it is the same family and channel, otherwise the different runtime it displaces) or `None` when nothing is the active default, and it fails closed, with a relation naming what could not be resolved, when the config names an active default the registry cannot resolve; `SdkInstallConsent` and `SdkInstallApprovalSource` say how consent is obtained (asked for, or already granted by `--yes`, `--approve-replacing-active-default` or `rocm update --apply`); `sdk_install_approval` turns those and whether a terminal is attached into proceed, prompt or refuse (`SdkInstallApproval`); `preapproved_install_line`, `fresh_install_line`, `refuse_non_interactive_message` and `confirm_overwrite_existing_sdk` are the wording of each outcome, and the prompt reads its answer through `prompt_yes_no` and the shared [`confirm.rs`](../crates/rocm-core/src/confirm.rs). `SdkInstallConsent`, `SdkInstallApprovalSource` and `preapproved_install_line` are re-exported by `therock.rs`; the other items `therock.rs` calls are `pub(super)`, as is `active_default_relation_text`, which the version property tests call, and the rest is private. -- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/io.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key`, `wheel_runtime_key` and `has_nontrivial_directory_contents`, which name and inspect the install a plan targets, stay in `therock.rs`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. +- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/io.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. - [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK`. - [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_chat_rocm_command_safety`, which refuses a public-bind, non-loopback or CPU `serve` and a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_chat_rocm_command_safety` here instead of through `chat`. `chat.rs` calls into it and re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). - [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. @@ -118,7 +118,7 @@ Further `apps/rocm` modules, not part of the `main.rs` split: - [`advised_commands.rs`](../apps/rocm/src/advised_commands.rs) — test-only (`#[cfg(test)]`): checks that every `rocm …`/`rocmd …` command the repository advises — in `--help` text, production string literals, the markdown docs and skills, and the VHS tapes — exists and parses, by running it through the parser of the binary it names (`rocm`'s clap `Cli`, or `rocmd`'s `parse_args`). - [`comfyui/install.rs`](../apps/rocm/src/comfyui/install.rs) — child module of [`comfyui.rs`](../apps/rocm/src/comfyui.rs): `rocm comfyui install`, including its dry run and the reinstall that replaces ComfyUI's code (through [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs)) while refusing to do so under a running ComfyUI. It downloads and unpacks the source archive, installs ComfyUI's Python dependencies into the selected runtime with uv (filtering out and pinning the runtime's own torch stack, and streaming uv's output to the install log) and runs the AMD GPU check. The manifest, the install log paths and the GPU probe stay in `comfyui.rs`. It imports `runtime_env` and, for the running-ComfyUI guard, `process`; neither imports it. Its unit tests and the Unix-only `archive_properties` property tests for the source unpack live in the module; the fixtures they share with the tests left in `comfyui.rs` stay there. - [`comfyui/process.rs`](../apps/rocm/src/comfyui/process.rs) — child module of [`comfyui.rs`](../apps/rocm/src/comfyui.rs): the lifecycle of the ComfyUI process this CLI launches. `start` checks the installed folder, runs the GPU check, launches ComfyUI in the background (a detached child on Linux, a hidden PowerShell launcher on Windows) and waits for its port; `stop` ends the saved process tree. `evaluate_running_state` tells a running, a still-loading and a stopped ComfyUI apart for `status`, and `running_from`, `refuse_while_running` and `running_advice` are the guard that refuses a reinstall while ComfyUI runs from the folder it would replace. The saved state and the types in it stay in `comfyui.rs`. Among the `comfyui` children it imports from [`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs) and uses [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs) to check for an interrupted reinstall; neither imports it, so there is no cycle. -- [`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs) — child module of [`comfyui.rs`](../apps/rocm/src/comfyui.rs): choosing which installed ROCm runtime ComfyUI runs against (`select_runtime`, with the messages that tell the user how to pick one when none or several are ready) and building the `VIRTUAL_ENV`, `ROCM_PATH`, `PATH` and library-path environment a ComfyUI child process runs with. The types it returns (`SelectedRuntime`, `ComfyUiRuntimeEnvironment`) stay in `comfyui.rs`, which the module imports from; it imports nothing from the other `comfyui` children. +- [`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs) — child module of [`comfyui.rs`](../apps/rocm/src/comfyui.rs): choosing which installed ROCm runtime ComfyUI runs against (`select_runtime`, with the messages that tell the user how to pick one when none or several are ready) and building the `VIRTUAL_ENV`, `ROCM_PATH`, `PATH` and library-path environment a ComfyUI child process runs with. When no `--runtime-id`, active runtime or configured default says which runtime to use, `select_runtime` does not decide which runtimes count as ready itself: it asks [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs)'s `select_single_ready_runtime`, the same rule the engine paths use, and only words the answer (`single_ready_runtime_or_refusal`), so ComfyUI keeps its own remediation text. `select_runtime` takes the `SetupRecovery` to ask it with, which the install derives from `--dry-run`, so a preview does not write the registry entry the real run would restore. A choice that does name a runtime (`--runtime-id`, the active runtime, the configured default) is looked up in the registry first (`pick_from_registry`); only when the registry alone cannot answer is it looked up again among the registry plus the runtime `rocm setup` created, so the `--runtime-id` an install preview prints works from the state the preview left, and an unusable setup runtime leaves the registry's own refusal standing (with nothing chosen it is likewise set aside while the registry holds a ready runtime, and reported only when none does). The setup runtime is found and picked without writing, and a real run writes its entry back only for a pick it is going to use, so a registry that cannot take the entry is reported as that failure rather than as a missing runtime. The types it returns (`SelectedRuntime`, `ComfyUiRuntimeEnvironment`) stay in `comfyui.rs`, which the module imports from; it imports nothing from the other `comfyui` children. - [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs) — child module of [`comfyui.rs`](../apps/rocm/src/comfyui.rs): replacing ComfyUI's code in an existing `source/` folder on reinstall without touching anything else there. - [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs) — `rocm remote`: driving a remote GPU host from this machine, with SSH as the control channel and the remote's own service published onto the tailnet for inference traffic. Like [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs), it owns its command enum (`RemoteCommand`) and dispatch fn (`run`). Its `pub(crate)` submodules: - [`remote/bootstrap.rs`](../apps/rocm/src/remote/bootstrap.rs) — deciding whether a remote machine can host a model before asking it to. @@ -215,7 +215,7 @@ Further `rocm-core` modules, each with its own doc comment except where noted: - [`disk_space.rs`](../crates/rocm-core/src/disk_space.rs) — free-space preflight checks and out-of-space error reporting. - [`fake_executable.rs`](../crates/rocm-core/src/fake_executable.rs) — writing stand-in executables that tests across the workspace then run. - [`fix.rs`](../crates/rocm-core/src/fix.rs) — `rocm fix`: remediations for diagnosed failure modes. Its `RECIPES` list (in [`fix/catalog.rs`](../crates/rocm-core/src/fix/catalog.rs)), together with the checkers under `diagnose/` (their table is [`catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs)), is the authoritative failure catalog. [`fix.rs`](../crates/rocm-core/src/fix.rs) stays the module root, as [`examine.rs`](../crates/rocm-core/src/examine.rs) does for `examine/`, so links to it stay valid, with its children in `fix/`; it keeps the dispatch (`apply`, `Plan`, `plan_of`, `act_on`) and re-exports the names the rest of the crate and the apps use, so `crate::fix::exit`, `rocm_core::fix_exit`, `rocm_core::fix::FixClass` and the `crate::fix::` oracles that the tests under `diagnose/` call keep resolving. The children are private modules: [`fix/options.rs`](../crates/rocm-core/src/fix/options.rs) (the named exit codes, `FixOptions` and `FixClass`); [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs) (the consent decision and `confirm`, the run and query timeouts, `current_os`, and the `fail!` helper, which it re-exports by path for its siblings); [`fix/runners.rs`](../crates/rocm-core/src/fix/runners.rs) (the host-mutating runners, one per auto-applicable fix, with their rc-file, PATH and device-pin helpers; only the four entry points the catalog names are visible to the root, and the `*_windows` variants are ordinary functions that every platform compiles); [`fix/catalog.rs`](../crates/rocm-core/src/fix/catalog.rs) (`FixRecipe`, the per-platform class table, `RECIPES`, the lookups by fix-id, and the `cfg(test)` oracles [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) and the e2e crate hold their own copies to); [`fix/render.rs`](../crates/rocm-core/src/fix/render.rs) (`list_recipes`, a recipe's plan and verify lines, `format_flags`, and the refusal for a diagnosis position passed as a fix-id); and [`fix/manifest.rs`](../crates/rocm-core/src/fix/manifest.rs) (the published JSON contract, `CATALOG_CONTRACT_VERSION` and `catalog_manifest`). `render` and `manifest` read `RECIPES` from `catalog`; nothing in `catalog` reads either. -- [`hardware_root.rs`](../crates/rocm-core/src/hardware_root.rs) — the root the hardware probes resolve host paths through (`host_path`): the real host in a normal build, a simulated host tree for the E2E suite in an `e2e-test-hooks` build. +- [`hardware_root.rs`](../crates/rocm-core/src/hardware_root.rs) — the root the hardware probes resolve host paths through (`host_path`): the real host in a normal build, a simulated host tree for the E2E suite in an `e2e-test-hooks` build. It also owns `read_os_release` (the `/etc/os-release` read with its `/usr/lib/os-release` fallback) and `real_os_release`, the one reader every package-manager plan and install hint uses, which always reads the real host even under a simulated root; the resolver it passes is private, so a second copy of that reader cannot be written outside this module. - [`log_tail.rs`](../crates/rocm-core/src/log_tail.rs) — `tail_lines`, the bounded log-tail read shared by the lemonade and vLLM adapters (no module doc comment). - [`openmpi.rs`](../crates/rocm-core/src/openmpi.rs) — OpenMPI, libatomic and libnuma detection, the `libmpi_cxx` compatibility shim, and the per-library install plans (`build_openmpi_install_plan`, `build_libatomic_install_plan`, `build_libnuma_install_plan`) and hints, needed by vLLM's ROCm wheels. The distro-generic half of that planning lives in [`package_plan.rs`](../crates/rocm-core/src/package_plan.rs); [`openmpi.rs`](../crates/rocm-core/src/openmpi.rs) re-exports its public items, so `rocm_core::openmpi::running_as_root`, `plan_for_os_release_read` and the rest keep resolving, and `pub(crate)` re-exports `PackageManager` and `resolve_package_manager` for [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs). - [`os_release.rs`](../crates/rocm-core/src/os_release.rs) — the one `/etc/os-release` parser every reader of that file uses. @@ -223,7 +223,7 @@ Further `rocm-core` modules, each with its own doc comment except where noted: - [`proc_lifecycle.rs`](../crates/rocm-core/src/proc_lifecycle.rs) — verified, bounded termination of a process by persisted PID, refusing to signal a PID whose start-time identity no longer matches. - [`report.rs`](../crates/rocm-core/src/report.rs) — the content of a Doctor report and the rule deciding whether one may exist; it sends nothing. - [`report_delivery.rs`](../crates/rocm-core/src/report_delivery.rs) — how a report leaves the machine: building its link and deciding whether a browser may be started. -- [`runtime.rs`](../crates/rocm-core/src/runtime.rs) — the host-platform layer: `RuntimePlatform`/`RuntimeHost`, OS-specific executable and Python-environment names, default config/data/cache directories, and path-text and path-list helpers (no module doc comment). +- [`runtime.rs`](../crates/rocm-core/src/runtime.rs) — the host-platform layer: `RuntimePlatform`/`RuntimeHost`, OS-specific executable and Python-environment names (including `runtime_python_env_root_for_executable`, the one rule for finding an environment's folder from its interpreter's path: only an interpreter directly inside a `bin` or `Scripts` folder, in any case, has one, and everything else gets `None`; the adopt command, the ComfyUI runtime environment and the vLLM launch all call it rather than keeping a copy, since the engine crate cannot reach `apps/rocm`), default config/data/cache directories, path-text and path-list helpers, and `install_root_has_payload`, the one definition of a non-empty install root (dot-entries such as the runtime marker do not count; a path that does not exist holds nothing; a root that cannot be listed is an error) that tarball extraction, update repair detection and activation validation in `apps/rocm` all ask (no module doc comment). - [`terminal.rs`](../crates/rocm-core/src/terminal.rs) — one ECMA-48 walk over untrusted terminal output, shared by every caller that has to quote or re-render it (`serve_summary.rs`, `vllm/startup_diagnostics.rs`, `vllm_oom.rs`, `diagnose/engine.rs`). - [`user_path.rs`](../crates/rocm-core/src/user_path.rs) — removing the install directory from the per-user Windows `Path` that the installer added it to. - [`uv.rs`](../crates/rocm-core/src/uv.rs) — acquiring the `uv` package manager into the managed cache, plus the argument/environment helpers `apps/rocm` and the vLLM engine share. diff --git a/docs/testing.md b/docs/testing.md index 984b1e1568..370a5d8ce8 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -931,6 +931,34 @@ cargo xtask e2e -- -n "comfyui-(0[5-9]|1[0-3])" `-n` bypasses the suite's applicability resolver, so the `@requires-os:linux` gate is not applied: run it only on Linux. +`comfyui-15` to `comfyui-19` cover which runtime `rocm comfyui install` uses +when none is chosen: it recovers the runtime `rocm setup` created before +counting the ready ones, as `rocm serve` does, so a runtime whose registry entry +was lost is found. The real run (`comfyui-16`, against an existing checkout that +needs no dependencies) writes the entry back; the `--dry-run` preview +(`comfyui-15`) names the same runtime but writes nothing, because a preview +promises not to change files. The preview prints the install command with that +runtime's `--runtime-id`, and `comfyui-17` runs that command as printed from the +state the preview left: an explicit `--runtime-id`, like an active or default +runtime, finds the runtime setup created when the registry alone cannot answer, +so the command works and writes the entry back. `comfyui-18` runs the same +printed command when the registry cannot take that entry back (a plain file +stands where its directory should be, which also stops `root`; a read-only +directory would not): the preview still succeeds, and the real run fails naming +the runtime whose entry it could not restore, rather than saying "Install ROCm +first", and leaves the registry as it was. The variant with a read-only registry +directory that holds other runtimes is a unit test, because a read-only +directory is writable to `root`. `comfyui-19` covers the other way a setup folder +can matter: when its runtime is no longer usable (its interpreter is gone) and one +ready runtime is registered, the preview uses the registered one and does not +register the unusable one, because an unusable runtime is not a ready runtime and +cannot change the count; its own error is reported only when no registered +runtime is ready, which a unit test covers. All five run with no GPU: + +```bash +cargo xtask e2e -- -n "comfyui-1[5-9]" +``` + `comfyui-14` is different: it installs ComfyUI into a real, isolated managed runtime and checks the ROCm torch stack survives with no `nvidia-*` packages added. It carries `@requires-gpu @nightly @requires-os:linux @requires-bare-metal` @@ -1058,12 +1086,46 @@ rocm runtimes adopt \ --runtime-key adopted-release-pip-gfx120x-all-7-13-0 ``` +Neither writes into the folder it registers: the registry entry is the only +file created. The `.rocm-cli-runtime.json` that `rocm install sdk` leaves in an +install root is ROCm CLI's proof that it owns the folder, so an imported or +adopted runtime does not get one, and uninstall and prune never delete such a +folder. A second import or adopt of the same runtime key is refused until +`--replace` is given. `@id:runtime-lifecycle-import-rejects-duplicate-unless-replacing` +(`runtime-lifecycle-06`) checks the import half end to end: the registry entry +exists, the imported folder gains no manifest file, and the duplicate is refused +until `--replace` (the unit tests also check the refused entry is left as it +was). Adopting needs a Python environment to probe and the +recovery of the runtime `rocm setup` created (which restores a missing registry +entry from the manifest already in the setup folder) needs a setup, so neither +has a scenario; the unit tests below assert each path's file set instead, and +`write_runtime_manifest` is the one routine all four paths write through. It writes the registry entry last, after the install-root file, so a write that fails leaves no new entry and an error that says a runtime could not be registered or restored matches the registry; recovery in particular never rewrites the file it restores from, whatever shape that file has. +A source-reading test, `only_the_shared_routine_writes_a_registry_entry`, fails +if production code in the `rocm` app, `rocmd`, `rocm-core` or the engines both +names the registry directory (through the two path helpers, or by spelling the +`runtimes` and `registry` segments) and writes a file, or if a file not on its +reviewed list starts calling the path helpers; +`registry_writer_detector_sees_every_way_of_building_the_path` pins what the +detector sees. It does not catch a writer that is handed the registry path by a +function in another file, a write call that is not in its list of write +primitives, or code outside those source roots. + +`--root` may be left out when `--python` sits directly in the environment's `bin` +(or `Scripts`) folder; an interpreter anywhere else is refused until `--root` is +given (scenario `runtime-26`). + Focused unit coverage: ```bash cargo test -p rocm --bin rocm runtime_activation cargo test -p rocm --bin rocm runtime_import cargo test -p rocm --bin rocm runtime_adopt +cargo test -p rocm --bin rocm runtime_install_writes +cargo test -p rocm --bin rocm setup_recovery +cargo test -p rocm --bin rocm write_runtime_manifest +cargo test -p rocm --bin rocm a_failed_manifest_write +cargo test -p rocm --bin rocm only_the_shared_routine_writes +cargo test -p rocm --bin rocm registry_writer_detector cargo test -p rocm --bin rocm runtime_key_includes_version_for_side_by_side_installs cargo test -p rocm --bin rocm engine_plugin cargo test -p rocm --bin rocm model_registry @@ -2801,8 +2863,8 @@ rather than the real host. The `a_fallback_read_plans*`/ `an_unreadable_file*`/`distro_name_names_an_unreadable_file` tests in `rocm-core` and `rocm` construct an `OsRelease` directly — bypassing the real read entirely, since the production functions that do it -(`openmpi::real_os_release`, `system_packages::real_os_release`) -read the real machine +(`hardware_root::real_os_release`, which every caller goes through) +reads the real machine unconditionally and cannot honour a simulated root — to pin that a fallback-sourced read plans and hints identically to a primary-sourced one with the same text, and that a file which exists but cannot be read is named @@ -2823,6 +2885,19 @@ shell probe instead of merely leaving `$ID` unset, which this test does not exercise — see its own doc comment for why, and for the known limitation this leaves. +The package-manager side reads the real host through one function, +`hardware_root::real_os_release`. In the `rocm` binary's tests, +`system_packages::tests::real_os_release_ignores_a_simulated_host_root` +(`--features e2e-test-hooks`, Linux) plants a root naming another distribution +and expects the real file back, and +`system_packages::tests::the_install_hints_read_the_real_host_and_ignore_a_simulated_root` +holds the three install hints to the same rule. Both live in that process +rather than in `rocm-core`'s, so the environment variable they set cannot +overlap a `rocm-core` test that reads host paths. +`system_packages::tests::no_caller_in_this_binary_reads_the_real_os_release_by_hand` +fails when a source file of the `rocm` binary reads the file with its own +resolver instead of calling it. + `rocm examine` reads the file through the simulated host root (see Simulated Host Root in [release-trust.md](release-trust.md)), so scenarios `examine-20` to `examine-23` and `examine-27` to `examine-28` in diff --git a/engines/vllm/src/process.rs b/engines/vllm/src/process.rs index 1d6acf7c1e..87f3d4f9a5 100644 --- a/engines/vllm/src/process.rs +++ b/engines/vllm/src/process.rs @@ -459,16 +459,12 @@ fn vllm_venv_root(runtime: &VllmRuntime) -> Result { runtime.runtime_id ) })?; - python - .parent() - .and_then(Path::parent) - .map(Path::to_path_buf) - .ok_or_else(|| { - anyhow!( - "python executable `{}` is not inside a venv (bin/Scripts) layout", - python.display() - ) - }) + rocm_core::runtime_python_env_root_for_executable(python).ok_or_else(|| { + anyhow!( + "python executable `{}` is not inside a venv (bin/Scripts) layout", + python.display() + ) + }) } /// Finds `/lib/python3.*/site-packages`, the one location a venv keeps /// its packages under. Bails on anything other than exactly one `python3.*` @@ -1083,6 +1079,31 @@ mod tests { PathBuf::from("/opt/venv") ); } + /// The runtime's venv is found by the rule every caller shares (see + /// `rocm_core::runtime_python_env_root_for_executable`): an interpreter in + /// an unrelated folder is refused, not mapped to that folder's parent, + /// which would send the site-packages lookup into a folder that is not a + /// venv. + #[test] + fn vllm_venv_root_needs_a_bin_or_scripts_folder() { + for (python, expected) in [ + ("/opt/venv/bin/python", Some("/opt/venv")), + ("/opt/venv/Scripts/python.exe", Some("/opt/venv")), + ("/opt/venv/SCRIPTS/python.exe", Some("/opt/venv")), + ("/opt/venv/tools/python", None), + ("python", None), + ("/python", None), + ] { + let runtime = test_vllm_runtime(Some(PathBuf::from(python)), None); + let root = vllm_venv_root(&runtime); + if let Some(expected) = expected { + assert_eq!(root.unwrap(), PathBuf::from(expected), "{python}"); + } else { + let error = root.expect_err(python).to_string(); + assert!(error.contains("is not inside a venv"), "{python}: {error}"); + } + } + } #[test] fn venv_site_packages_dir_finds_the_single_python3_dir() -> Result<()> { let venv = tempfile::tempdir()?; diff --git a/tests/e2e-cucumber/features/comfyui.feature b/tests/e2e-cucumber/features/comfyui.feature index e6fb09bd54..18aa990947 100644 --- a/tests/e2e-cucumber/features/comfyui.feature +++ b/tests/e2e-cucumber/features/comfyui.feature @@ -333,3 +333,86 @@ Feature: ComfyUI install reports progress, makes failures actionable and preserv And ComfyUI's dependencies were installed into the runtime And the runtime's torch stack is unchanged And no nvidia-* packages were added to the runtime + + # With no runtime chosen, `rocm comfyui install` uses the one ready runtime, + # and it finds it the way `rocm serve` and `rocm engines install` do: it + # recovers the runtime `rocm setup` created first. A runtime whose registry + # entry was lost, but whose install tree still holds its own manifest and that + # setup names, therefore counts as the ready runtime, instead of the install + # refusing with "Install ROCm first" while setup has finished. The real run + # writes the lost registry entry back (comfyui-16); a `--dry-run` preview + # names the same runtime but leaves the registry as it found it, because it + # promises not to change files (this scenario). The preview prints the + # `--runtime-id` of the command it would run, which is how it shows the + # choice; comfyui-17 runs that command as printed. No GPU is needed: readiness + # is filesystem and manifest state. + @id:comfyui-install-finds-the-runtime-setup-created + Scenario: comfyui-15 - A ComfyUI install preview finds the runtime setup created without writing the registry + Given a ready ROCm runtime that setup created and the registry has forgotten + When the user previews a ComfyUI install without choosing a runtime + Then the preview selects that runtime + And the runtime registry still has no entry for it + + # The real run of the same install: it uses the runtime setup created and + # writes its lost registry entry back, so the runtime it installed into is one + # `rocm runtimes list` shows afterwards. The ComfyUI checkout is already in + # place and asks for nothing beyond the torch stack the runtime provides, so + # the install needs no network and no package index. + @id:comfyui-install-restores-the-runtime-setup-created + Scenario: comfyui-16 - A ComfyUI install restores the registry entry of the runtime setup created + Given a ready ROCm runtime that setup created and the registry has forgotten + And a ComfyUI checkout that needs nothing beyond the runtime's torch stack + When the user installs ComfyUI without choosing a runtime + Then the install uses the runtime setup created + And the runtime registry has an entry for it again + + # The preview's `install command:` line is how it shows the choice, so it has + # to be a command that works from the state the preview left, where the + # registry still has no entry for the runtime it named. Its `--runtime-id` is + # an explicit choice, which finds the runtime setup created the same way a + # missing choice does, instead of refusing with "Install ROCm first". Run as + # printed, the command installs into that runtime and writes the entry back. + @id:comfyui-install-preview-command-works-as-printed + Scenario: comfyui-17 - The install command a ComfyUI install preview prints works from the state the preview left + Given a ready ROCm runtime that setup created and the registry has forgotten + And a ComfyUI checkout that needs nothing beyond the runtime's torch stack + When the user previews a ComfyUI install without choosing a runtime + Then the runtime registry still has no entry for it + When the user runs the install command the preview printed + Then the install uses the runtime setup created + And the runtime registry has an entry for it again + + # The same printed command, when the registry cannot take the entry back (a + # file stands where its directory should be, which stops an administrator as + # well; a read-only directory would not). The preview writes nothing, so it + # still succeeds and names the runtime. The real run finds that runtime the + # same way, fails to restore its entry, and says so, rather than answering + # "Install ROCm first" as if setup had left no runtime. The registry is left as + # it was. A read-only registry directory with other runtimes in it, the other + # way to reach this, is covered by a unit test: the suite can run as root, for + # whom a read-only directory is writable. + @id:comfyui-install-reports-a-registry-it-cannot-write + Scenario: comfyui-18 - The install command a ComfyUI install preview prints reports a registry that cannot take the runtime back + Given a ready ROCm runtime that setup created and the registry has forgotten + And a ComfyUI checkout that needs nothing beyond the runtime's torch stack + And the runtime registry cannot be written + When the user previews a ComfyUI install without choosing a runtime + Then the preview selects that runtime + When the user runs the install command the preview printed + Then the install is refused because the runtime's registry entry could not be restored + And the runtime registry is as it was + + # With no runtime chosen, a setup folder whose runtime can no longer be used + # (here its interpreter is gone) is not a ready runtime and cannot change which + # runtimes are ready, so it does not stop an install that has exactly one ready + # runtime registered: the install uses that one, as a choice that names it + # does, and the unusable setup runtime is not written into the registry. Its + # own error is reported only when no registered runtime is ready (a unit + # test). No GPU is needed: readiness is filesystem and manifest state. + @id:comfyui-install-uses-the-ready-runtime-beside-an-unusable-setup-runtime + Scenario: comfyui-19 - A setup runtime that is no longer usable does not stop a ComfyUI install that has one ready runtime + Given a ready ROCm runtime in the registry + And a setup folder whose runtime is no longer usable + When the user previews a ComfyUI install without choosing a runtime + Then the preview selects the registered runtime + And the runtime registry holds only the registered runtime diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index 69873fa6b6..70b6de0eb4 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -287,9 +287,9 @@ Feature: GPU detection and system inspection # machine other than the runner. The driver plan cannot be exercised this way: # `rocm install driver` builds a plan that runs against the real machine, so it # deliberately reads the real `/etc/os-release` (`real_os_release` in - # `system_packages.rs`, whose test there pins that it ignores a simulated - # host root); the plan built from it is covered by the unit tests in - # `driver_plan.rs`. + # `hardware_root.rs`, whose test in `system_packages.rs` pins that it ignores + # a simulated host root); the plan built from it is covered by the unit tests + # in `driver_plan.rs`. # # The planted distributions are ones no CI runner or developer host here runs, # so a probe that read the real file instead would report a different one. diff --git a/tests/e2e-cucumber/features/runtime_lifecycle.feature b/tests/e2e-cucumber/features/runtime_lifecycle.feature index e5e9537edf..5f8da06c42 100644 --- a/tests/e2e-cucumber/features/runtime_lifecycle.feature +++ b/tests/e2e-cucumber/features/runtime_lifecycle.feature @@ -48,10 +48,12 @@ Feature: Runtime lifecycle state machine Given a runtime manifest to import When the user imports the runtime Then the runtime is registered as read-only + And the registry records the import and its folder gains no manifest file When the user imports the same runtime again Then the CLI refuses because it already exists When the user imports it again allowing replacement Then the import succeeds + And the registry records the import and its folder gains no manifest file # `rocm runtimes list` prefixes each row with `*`/`-`/a blank to mark active, # rollback-target, and neither, with nothing else on the page explaining what diff --git a/tests/e2e-cucumber/features/runtime_setup.feature b/tests/e2e-cucumber/features/runtime_setup.feature index d3b93cf112..bc1b24ef42 100644 --- a/tests/e2e-cucumber/features/runtime_setup.feature +++ b/tests/e2e-cucumber/features/runtime_setup.feature @@ -559,3 +559,22 @@ Feature: Runtime configuration When the user tries to preview a wheel SDK install for arch gfx1200 with no version pin Then the install fails And the preview is refused because managed Python is disabled, without a promise + + # `rocm runtimes adopt --python` takes the environment folder from where the + # interpreter sits: the folder that holds its `bin` (or `Scripts`) folder. An + # interpreter anywhere else has no environment folder to infer, and a guess (the + # interpreter's grandparent, which is what the command used to adopt) would + # register a folder that is not an environment as a runtime. The command refuses + # instead and names the flag that supplies the folder; nothing is saved. + # + # Linux-only: the stand-in interpreter is a POSIX shell script. The "adopts with + # --root" half is covered by the unit test + # `adopt_request_without_an_inferred_root_asks_for_root`, which shows an explicit + # root clears the refusal; adopting for real needs a genuine ROCm SDK environment. + @id:runtime-adopt-outside-a-venv-asks-for-root @requires-os:linux + Scenario: runtime-26 - Adopting a Python that sits outside any environment's bin folder asks for --root + Given a Python interpreter that sits outside any environment's bin folder + When the user adopts that interpreter without naming an environment folder + Then the adoption is refused + And the refusal asks for the environment folder with --root + And no runtime is registered diff --git a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs index 61b37e4faf..c0bca4e7a5 100644 --- a/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/comfyui_steps.rs @@ -34,6 +34,21 @@ //! (`crates/rocm-dash-tui/src/app/tool_approval.rs`) asserts that envelope is collapsed //! out of the chat. //! +//! `comfyui-15` to `comfyui-19` cover the other half of that selection: with +//! nothing chosen, the install recovers the runtime `rocm setup` created before +//! it counts the ready ones, as `rocm serve` does, so a runtime the registry +//! lost is found. They plant a ready runtime and the setup folder that names it, +//! but no registry entry. `comfyui-15` previews the install with `--dry-run`, +//! which prints the `--runtime-id` of the runtime it chose and writes nothing, +//! so the registry still has no entry afterwards; `comfyui-16` runs the install +//! against an existing checkout and finds the entry written back; `comfyui-17` +//! runs the command the preview printed, from the state the preview left, and +//! finds the same entry written back. `comfyui-18` runs that command with a +//! registry that cannot be written and finds the failure to restore the entry +//! reported. `comfyui-19` plants a setup folder whose runtime is no longer +//! usable next to one ready registered runtime and finds the preview using the +//! registered one without registering the unusable one. +//! //! `comfyui-04` covers the source-archive download spinner under a real PTY, //! mirroring `download_progress_pty.feature`'s tarball scenario but for //! `download_and_extract_source`'s in-process `GzDecoder`/`tar` unpack, which @@ -194,6 +209,21 @@ fn rocm_library_filename(shortname: &str) -> String { /// confusing downstream assertion instead of on the fixture. If this step starts /// failing after an edit here, suspect the manifest shape first. fn plant_ready_runtime(data: &Path, key: &str) { + let manifest = plant_ready_runtime_tree(data, key); + let registry = data.join("runtimes").join("registry"); + std::fs::create_dir_all(®istry) + .unwrap_or_else(|e| panic!("failed to create {}: {e}", registry.display())); + std::fs::write( + registry.join(format!("{key}.json")), + serde_json::to_vec_pretty(&manifest).expect("manifest serialises"), + ) + .unwrap_or_else(|e| panic!("failed to write the planted runtime manifest: {e}")); +} + +/// The on-disk half of [`plant_ready_runtime`]: the install root and what the +/// readiness check needs under it. Returns the manifest JSON that describes it, +/// leaving it to the caller whether the registry gets an entry. +fn plant_ready_runtime_tree(data: &Path, key: &str) -> serde_json::Value { let install_root = data.join("runtimes").join("roots").join(key); let sdk_root = install_root.join("sdk"); let sdk_bin = sdk_root.join("bin"); @@ -208,7 +238,7 @@ fn plant_ready_runtime(data: &Path, key: &str) { write_fixture(&amdhip, "stub"); write_fixture(&hipblas, "stub"); - let manifest = serde_json::json!({ + serde_json::json!({ "runtime_key": key, "runtime_id": key, "channel": "release", @@ -229,16 +259,7 @@ fn plant_ready_runtime(data: &Path, key: &str) { ], }, "installed_at_unix_ms": 1, - }); - - let registry = data.join("runtimes").join("registry"); - std::fs::create_dir_all(®istry) - .unwrap_or_else(|e| panic!("failed to create {}: {e}", registry.display())); - std::fs::write( - registry.join(format!("{key}.json")), - serde_json::to_vec_pretty(&manifest).expect("manifest serialises"), - ) - .unwrap_or_else(|e| panic!("failed to write the planted runtime manifest: {e}")); + }) } /// Combined stdout+stderr of the recorded `rocm` invocation. The refusal is an @@ -483,6 +504,34 @@ async fn install_comfyui_without_runtime(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +/// Runs the `install command:` line of the preview in `world.cli_output`, as a +/// user would copy it: every word after `rocm` is an argument. The runtime key +/// planted for these scenarios needs no shell quoting, so splitting on +/// whitespace is the whole parse; the first word is checked so a line in +/// another shape fails here rather than running something else. +#[when("the user runs the install command the preview printed")] +async fn run_the_install_command_the_preview_printed(world: &mut E2eWorld) { + let stdout = world.cli_output.clone().unwrap_or_default(); + let line = stdout + .lines() + .find_map(|line| line.trim().strip_prefix("install command:")) + .unwrap_or_else(|| panic!("the preview printed no install command:\n{stdout}")); + let words: Vec<&str> = line.split_whitespace().collect(); + assert_eq!( + words.first().copied(), + Some("rocm"), + "the printed command should start with `rocm`: {line}" + ); + assert!( + words.contains(&"--runtime-id") && !words.contains(&"--dry-run"), + "the printed command should name its runtime and not preview: {line}" + ); + let (stdout, stderr, rc) = crate::run_rocm(world, &words[1..]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + #[then("ComfyUI install is refused as ambiguous")] async fn comfyui_install_refused(world: &mut E2eWorld) { let text = refusal_text(world); @@ -536,6 +585,251 @@ async fn refusal_lists_both_keys(world: &mut E2eWorld) { } } +// --- comfyui-15: the install finds the runtime setup created --- + +/// The one runtime `comfyui-15` plants. Not in [`RUNTIME_KEYS`], which are the +/// two the ambiguity scenario needs. +const SETUP_RUNTIME_KEY: &str = "e2e-setup-created-runtime"; + +fn registry_dir(world: &E2eWorld) -> PathBuf { + data_dir(world).join("runtimes").join("registry") +} + +#[given("a ready ROCm runtime that setup created and the registry has forgotten")] +async fn plant_runtime_setup_created_and_registry_forgot(world: &mut E2eWorld) { + let data = data_dir(world); + let manifest = plant_ready_runtime_tree(&data, SETUP_RUNTIME_KEY); + let install_root = data.join("runtimes").join("roots").join(SETUP_RUNTIME_KEY); + // The install tree's own copy of its manifest is what makes the registry + // entry recoverable. Without it recovery finds nothing to restore and the + // scenario would pass or fail for the wrong reason. + write_fixture( + &install_root.join(".rocm-cli-runtime.json"), + &serde_json::to_string_pretty(&manifest).expect("manifest serialises"), + ); + // Black-box: plain JSON matching the CLI's on-disk config schema. `setup` + // naming the folder is what lets recovery find the tree; there is no + // registry entry, and that is the state under test. + write_fixture( + &root(world).join("config").join("config.json"), + &serde_json::to_string_pretty(&serde_json::json!({ + "setup": { "therock_venv": install_root.display().to_string() }, + })) + .expect("config serialises"), + ); + assert!( + !registry_dir(world) + .join(format!("{SETUP_RUNTIME_KEY}.json")) + .exists(), + "the registry must start without the runtime for this scenario to mean anything" + ); +} + +#[when("the user previews a ComfyUI install without choosing a runtime")] +async fn preview_comfyui_install_without_runtime(world: &mut E2eWorld) { + let (stdout, stderr, rc) = crate::run_rocm(world, &["comfyui", "install", "--dry-run"]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the preview selects that runtime")] +async fn preview_selects_setup_runtime(world: &mut E2eWorld) { + let text = refusal_text(world); + assert_eq!( + world.cli_rc, + Some(0), + "the preview should succeed, got:\n{text}" + ); + let stdout = world.cli_output.clone().unwrap_or_default(); + assert!( + stdout.lines().any(|line| line.contains("install command:") + && line.contains(&format!("--runtime-id {SETUP_RUNTIME_KEY}"))), + "the preview should name the runtime setup created in its install command, got:\n{text}" + ); +} + +#[then("the runtime registry has an entry for it again")] +async fn registry_has_the_setup_runtime_again(world: &mut E2eWorld) { + let entry = registry_dir(world).join(format!("{SETUP_RUNTIME_KEY}.json")); + assert!( + entry.is_file(), + "selecting the runtime setup created should have written {} back", + entry.display() + ); +} + +/// A preview promises not to change files, and the registry entry is a file: the +/// runtime it named is not registered by having been previewed. +#[then("the runtime registry still has no entry for it")] +async fn registry_still_lacks_the_setup_runtime(world: &mut E2eWorld) { + let entry = registry_dir(world).join(format!("{SETUP_RUNTIME_KEY}.json")); + assert!( + !entry.exists(), + "a `--dry-run` preview must not write {}", + entry.display() + ); +} + +#[given("a ComfyUI checkout that needs nothing beyond the runtime's torch stack")] +async fn comfyui_checkout_needing_no_dependencies(world: &mut E2eWorld) { + let source_dir = data_dir(world) + .join("runtimes") + .join("roots") + .join(SETUP_RUNTIME_KEY) + .join("apps") + .join("comfyui") + .join("source"); + // Every ComfyUI release has a `main.py`. The requirements name only the + // torch stack the runtime provides, which the install filters out, so it + // reaches no package index. + write_fixture(&source_dir.join("main.py"), ""); + write_fixture( + &source_dir.join("requirements.txt"), + "torch==2.4.0\ntorchvision==0.19.0\ntorchaudio==2.4.0\n", + ); +} + +/// What stands where the registry directory should be for `comfyui-18`. A plain +/// file stops every user, an administrator included, from creating the registry, +/// where a read-only directory would not stop `root`. +const UNWRITABLE_REGISTRY_CONTENTS: &str = "not a directory"; + +#[given("the runtime registry cannot be written")] +async fn runtime_registry_cannot_be_written(world: &mut E2eWorld) { + let registry = registry_dir(world); + assert!( + !registry.exists(), + "the registry must not exist yet for this step to mean anything" + ); + write_fixture(®istry, UNWRITABLE_REGISTRY_CONTENTS); +} + +/// The failing install names the entry it could not restore, so the user learns +/// why, and is not told that setup left no runtime: the runtime exists, and the +/// preview just named it. +#[then("the install is refused because the runtime's registry entry could not be restored")] +async fn install_refused_for_the_registry_entry(world: &mut E2eWorld) { + let text = refusal_text(world); + assert!( + world.cli_rc.is_some_and(|rc| rc != 0), + "the install should fail when the registry cannot take the runtime back, got:\n{text}" + ); + assert!( + text.contains(&format!( + "failed to restore setup runtime `{SETUP_RUNTIME_KEY}`" + )), + "the failure should name the runtime whose entry could not be restored, got:\n{text}" + ); + assert!( + !text.contains("Install ROCm first"), + "the failure must not tell the user to install ROCm when setup left a runtime, got:\n{text}" + ); +} + +#[then("the runtime registry is as it was")] +async fn runtime_registry_is_as_it_was(world: &mut E2eWorld) { + let registry = registry_dir(world); + assert_eq!( + std::fs::read_to_string(®istry).ok().as_deref(), + Some(UNWRITABLE_REGISTRY_CONTENTS), + "a failed restore must leave {} as it found it", + registry.display() + ); +} + +#[then("the install uses the runtime setup created")] +async fn install_uses_the_setup_runtime(world: &mut E2eWorld) { + let text = refusal_text(world); + assert_eq!( + world.cli_rc, + Some(0), + "the install should succeed, got:\n{text}" + ); + let stdout = world.cli_output.clone().unwrap_or_default(); + assert!( + stdout.contains("action: install") && stdout.contains("ROCm runtime: 7.13.0"), + "the install should report the runtime setup created, got:\n{text}" + ); +} + +// --- comfyui-19: an unusable setup runtime beside a ready registered one --- + +/// The setup runtime `comfyui-19` plants and breaks. Not [`SETUP_RUNTIME_KEY`], +/// so the two scenarios cannot be mistaken for each other in a failure. +const STALE_SETUP_RUNTIME_KEY: &str = "e2e-stale-setup-runtime"; + +#[given("a ready ROCm runtime in the registry")] +async fn plant_one_registered_ready_runtime(world: &mut E2eWorld) { + plant_ready_runtime(&data_dir(world), RUNTIME_KEY); +} + +#[given("a setup folder whose runtime is no longer usable")] +async fn plant_setup_folder_with_an_unusable_runtime(world: &mut E2eWorld) { + let data = data_dir(world); + let manifest = plant_ready_runtime_tree(&data, STALE_SETUP_RUNTIME_KEY); + let install_root = data + .join("runtimes") + .join("roots") + .join(STALE_SETUP_RUNTIME_KEY); + write_fixture( + &install_root.join(".rocm-cli-runtime.json"), + &serde_json::to_string_pretty(&manifest).expect("manifest serialises"), + ); + write_fixture( + &root(world).join("config").join("config.json"), + &serde_json::to_string_pretty(&serde_json::json!({ + "setup": { "therock_venv": install_root.display().to_string() }, + })) + .expect("config serialises"), + ); + // The interpreter is what the readiness check needs and what goes missing + // when a setup folder goes stale. Without removing it this runtime would be + // a second ready one and the scenario would assert the wrong thing. + let python = manifest["python_executable"] + .as_str() + .expect("the planted manifest records an interpreter"); + std::fs::remove_file(python).unwrap_or_else(|e| panic!("failed to remove {python}: {e}")); +} + +#[then("the preview selects the registered runtime")] +async fn preview_selects_the_registered_runtime(world: &mut E2eWorld) { + let text = refusal_text(world); + assert_eq!( + world.cli_rc, + Some(0), + "the preview should succeed, got:\n{text}" + ); + let stdout = world.cli_output.clone().unwrap_or_default(); + assert!( + stdout.lines().any(|line| line.contains("install command:") + && line.contains(&format!("--runtime-id {RUNTIME_KEY}"))), + "the preview should name the registered runtime in its install command, got:\n{text}" + ); + // The recovery error names the runtime ("setup runtime `` ... is not + // usable"); other output can say "not usable" about something else, such as + // the Python the install prepares. + assert!( + !text.contains(STALE_SETUP_RUNTIME_KEY), + "the unusable setup runtime must not be reported while another runtime is ready, got:\n{text}" + ); +} + +#[then("the runtime registry holds only the registered runtime")] +async fn registry_holds_only_the_registered_runtime(world: &mut E2eWorld) { + let mut names: Vec = std::fs::read_dir(registry_dir(world)) + .unwrap_or_else(|e| panic!("the registry should be readable: {e}")) + .flatten() + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .collect(); + names.sort(); + assert_eq!( + names, + vec![format!("{RUNTIME_KEY}.json")], + "the unusable setup runtime must not be written into the registry" + ); +} + /// Pacing knobs for `comfyui-04`'s download server, mirroring /// `therock_steps.rs`'s `PACED_TARBALL_*` constants: large enough that several /// chunk boundaries land before the transfer completes, slow enough per chunk diff --git a/tests/e2e-cucumber/tests/e2e/driver_steps.rs b/tests/e2e-cucumber/tests/e2e/driver_steps.rs index a1532cc998..cdb0d7debf 100644 --- a/tests/e2e-cucumber/tests/e2e/driver_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/driver_steps.rs @@ -41,10 +41,9 @@ async fn assert_driver_plan_matches_host_release(world: &mut E2eWorld) { // breaks the arm-selection logic pass by construction. The hosted // `ubuntu-latest` E2E image (26.04 once GitHub's migration completes) and the self-hosted GPU // lane (24.04) both take the apt arm, with their own release codename. - let os_release_text = - rocm_core::hardware_root::read_os_release(rocm_core::hardware_root::identity_path_for_str) - .text() - .to_owned(); + let os_release_text = rocm_core::hardware_root::real_os_release() + .text() + .to_owned(); let Ok(os_release) = rocm_core::os_release::parse(&os_release_text) else { panic!("host's /etc/os-release is unreadable:\n{output}"); }; diff --git a/tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs b/tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs index c127245530..1838780c1e 100644 --- a/tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/runtime_lifecycle_steps.rs @@ -273,9 +273,19 @@ fn active_runtime_key(world: &E2eWorld) -> String { /// Path to an importable manifest file (not yet in the registry) for the import /// scenario, with its install_root created so the import validates. +/// The folder the manifest to import points at: a runtime somebody else owns. +fn import_install_root(world: &E2eWorld) -> PathBuf { + world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("runtime-import") +} + fn write_import_manifest(world: &E2eWorld) -> PathBuf { let root = world.isolated_root.as_ref().expect("no isolated root"); - let install_root = root.path().join("runtime-import"); + let install_root = import_install_root(world); std::fs::create_dir_all(&install_root).expect("failed to create import install root"); std::fs::write(install_root.join("payload.txt"), "payload") .expect("failed to write import payload"); @@ -740,6 +750,27 @@ async fn imported_readonly(world: &mut E2eWorld) { ); } +/// What an import creates on disk: the registry entry, and nothing in the +/// folder it points at. The manifest ROCm CLI keeps in an install root is its +/// proof that it owns the folder, and an imported runtime is somebody else's. +/// Unit tests cover the same file set for adopt and setup recovery, which no +/// scenario reaches without a Python environment to probe. +#[then("the registry records the import and its folder gains no manifest file")] +async fn import_writes_registry_entry_only(world: &mut E2eWorld) { + let entry = registry_dir(world).join(format!("{IMPORT_KEY}.json")); + assert!( + entry.is_file(), + "expected a registry entry at {}", + entry.display() + ); + let sidecar = import_install_root(world).join(".rocm-cli-runtime.json"); + assert!( + !sidecar.exists(), + "an import must not write into the folder it registers: {}", + sidecar.display() + ); +} + #[then("the CLI refuses because it already exists")] async fn import_duplicate_refused(world: &mut E2eWorld) { let rc = world.cli_rc.expect("no command rc recorded"); diff --git a/tests/e2e-cucumber/tests/e2e/runtime_steps.rs b/tests/e2e-cucumber/tests/e2e/runtime_steps.rs index 4150c9a679..1bf377b496 100644 --- a/tests/e2e-cucumber/tests/e2e/runtime_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/runtime_steps.rs @@ -1515,3 +1515,66 @@ async fn assert_freeform_execution_prefix_is_one_word(world: &mut E2eWorld) { &freeform_prefix_with_quote(world), ); } + +/// Where scenario runtime-26 plants its interpreter: inside a folder called `tools`, +/// which is neither `bin` nor `Scripts`, so nothing about its location names an +/// environment. +fn interpreter_outside_a_venv(world: &E2eWorld) -> std::path::PathBuf { + world + .isolated_root + .as_ref() + .expect("scenario has no isolated root") + .path() + .join("not-a-venv") + .join("tools") + .join("python") +} + +/// A stand-in interpreter that answers the SDK probe with the smallest document the +/// CLI parses. The refusal under test happens after the probe and before anything is +/// read from the environment, so nothing else about the interpreter matters. +/// +/// A POSIX shell script, so the scenario is Linux-only (`@requires-os:linux`). +#[given("a Python interpreter that sits outside any environment's bin folder")] +async fn setup_interpreter_outside_a_venv(world: &mut E2eWorld) { + let python = interpreter_outside_a_venv(world); + std::fs::create_dir_all(python.parent().expect("interpreter has a folder")) + .expect("failed to create the interpreter's folder"); + rocm_core::fake_executable::write_fake_executable( + &python, + "#!/bin/sh\necho '{\"import_ok\": true}'\n", + ) + .expect("failed to write the stand-in interpreter"); +} + +#[when("the user adopts that interpreter without naming an environment folder")] +async fn user_adopts_interpreter_without_root(world: &mut E2eWorld) { + let python = interpreter_outside_a_venv(world) + .to_string_lossy() + .into_owned(); + let (stdout, stderr, rc) = crate::run_rocm(world, &["runtimes", "adopt", "--python", &python]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the refusal asks for the environment folder with --root")] +async fn assert_adopt_refusal_asks_for_root(world: &mut E2eWorld) { + let stdout = world.cli_output.as_deref().unwrap_or(""); + let stderr = world.cli_stderr.as_deref().unwrap_or(""); + let combined = format!("{stdout}{stderr}"); + assert!( + combined.contains("could not infer the Python environment folder") + && combined.contains("pass --root"), + "the refusal does not ask for --root:\n{combined}" + ); +} + +#[then("no runtime is registered")] +async fn assert_no_runtime_registered(world: &mut E2eWorld) { + let (stdout, _, _) = crate::run_rocm(world, &["runtimes", "list"]); + assert!( + stdout.contains("installed: none") || stdout.contains("managed_runtimes: 0"), + "the refused adoption left a runtime registered:\n{stdout}" + ); +} From f5b0c60e79f5e8e8e035b7e18c79edef3cfed9d2 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Fri, 6 Nov 2026 16:27:47 +0000 Subject: [PATCH 003/102] dev: skip the pre-push cargo steps that already passed on the same tree (#924) * dev: narrow the pre-push cargo hooks to the crates a push affects The three cargo pre-push hooks (clippy, the e2e harness clippy and cargo test) ran over the whole workspace whatever the push contained, so a one-file change in a leaf crate cost a full workspace build and test run. They now run through `cargo xtask pre-push-check`, which takes the range prek passes to a pre-push hook and applies the selection CI already uses (`cargo xtask affected`): the changed crates plus every crate that depends on them, and the whole workspace for the lockfile, the root manifest, the toolchain file, workflow files, a doc an xtask test reads or a path no crate owns. The directory-based selection cannot see a crate's tests reading a file from outside the crate, so the hook widens it: any Rust file also selects xtask (its contract tests scan every source file), README.md, docs/ and skills/ also select rocm and xtask, and the e2e PowerShell fixture also selects rocm-core, which includes it. A unit test scans the workspace for include_str!, include_bytes! and manifest-relative reads that leave their crate and fails when one is not covered, so a new cross-crate read cannot be missed silently. Whenever the affected crates cannot be determined (the base is not fetched, `cargo metadata` fails) a step runs over the whole workspace and prints why. A step is skipped only when the selection is known and has nothing for it to check, and it says so. The e2e harness clippy runs when e2e-cucumber is affected, and the test step never includes it, as before. `--dry-run` prints the scope and the cargo command without compiling. The step's range, `--base` override, `--dry-run` and cargo's exit status are tested through an injectable cargo runner. `affected` now lists both paths of a moved file (`--no-renames`), so a file moved out of one crate selects the crate that lost it as well as the one that gained it. Signed-off-by: Jussi Elo * dev: recognise raw-string includes and skip nested checkouts in the cross-crate read guard The drift guard that keeps the pre-push selection in step with crate tests reading files outside their crate only recognised plain `"..."` literals, so `include_str!(r#"../../../README.md"#)` passed it even though a README edit would not select the reader crate. It now reads raw literals too. It also walked every directory, including nested checkouts such as `.claude/worktrees/...`, so a rocm-core fixture include in another branch's sources was reported as an uncovered read of this tree and could fail `cargo test -p xtask`. Like the other xtask source scans, it now skips every dot-directory. Each change has a test that fails when it is reverted. Signed-off-by: Jussi Elo * dev: keep the raw-string include test from looking like an embedded data file The test added for raw-string includes spelled out `include_str!("../../../README.md")` in its source, which the ASCII output contract counts as a data file compiled into a binary, so `every_embedded_data_file_is_ascii` failed on the README's non-ASCII lines in the Linux, Windows and coverage jobs. The test now builds those arguments with `format!`, as the neighbouring scanner test does. Signed-off-by: Jussi Elo * dev: keep the rename test's git from inheriting a hook's GIT_DIR git exports an absolute `GIT_DIR` to a pre-push hook run from a linked worktree. The throwaway-repository helper and the directory-taking `diff_name_only` ran `git -C ` without clearing it, so in that setting they acted on the outer repository: the rename test could fail, and the helper's commit and staging landed on the developer's branch. Both now clear the variables git uses to select a repository, through one shared `scrub_git_env`, and a test checks that every one is cleared. Signed-off-by: Jussi Elo * dev: skip the pre-push cargo steps that already passed on the same tree The pre-push hook had no memory, so a push retried after a dropped connection, a manual run before the push, or an amend that only rewords the message re-ran clippy and the tests on content it had just checked. After a step's cargo command exits 0, `cargo xtask pre-push-check` now records a stamp, and the next run of that step looks for it first. The stamp is keyed on the step and its exact cargo arguments (so a pass for a narrow selection never lets a wider one skip), the git tree hash of HEAD (so an amend with identical content still hits), the toolchain, and a digest of the CARGO_* and RUST* environment. It is only used and written when the working tree is clean, because cargo checks the working tree rather than the pushed commit, and nothing is recorded if the run changed the tree. Stamps live under the git common directory, so every worktree shares them, expire after seven days and are capped at 64; scratch files from a write that died are cleaned up. Any error reading, writing or verifying a stamp means the step runs, and git is run with GIT_DIR and the other repository-selecting variables removed when it is pointed at a directory. A skip prints what matched, what it does not cover (user-level cargo configuration, installed tools, files git ignores, flaky tests), and how to force a run (--rerun or ROCM_PRE_PUSH_RERUN=1). CONTRIBUTING.md, docs/testing.md and the hook comment describe the skip and say it shows this tree passed earlier, not that the checks ran during this push. Signed-off-by: Jussi Elo * test(xtask): pin that an empty changed-file list runs the whole workspace on the stamped path A pushed range that lists no changed files means the range is wrong, not that nothing changed. resolve_selection already turns that into the whole workspace, and the stamped path goes through it, but nothing exercised the two together, so a change that let an empty list through to the "no Rust crate" skip would also bypass the pass stamps. The new test records a narrow pass, then runs with an empty list and expects the whole-workspace test command to run, to be recorded as its own pass and to be skipped only on its own next run, and expects the lint step to fall back the same way. It fails when the empty-range check in resolve_selection is removed. Signed-off-by: Jussi Elo * dev: drop this step's earlier pass before a --rerun so a failed rerun cannot leave it behind --rerun exists for the cases the pass key does not cover, such as a change to user-level cargo configuration or an installed tool. Before, a rerun ignored the recorded pass but left it in place, so when the rerun failed the next ordinary run still skipped on the pass the rerun was meant to double-check. A rerun that is going to run now removes exactly this step's matching stamp first, through a new Stamps::invalidate, and a successful rerun records a fresh pass as before. Stamps of other steps and other selections are different files and stay, and --dry-run still touches nothing. New tests cover pass, failed rerun and ordinary retry (which must run the step), a failed rerun leaving sibling stamps alone, a dry-run rerun leaving every stamp in place, and a passing rerun replacing the pass; each fails when its part of the change is removed. Signed-off-by: Jussi Elo * dev: read the process environment as raw bytes so a non-Unicode variable cannot abort the pre-push hook The environment part of the pass key came from std::env::vars(), which panics as soon as any variable holds bytes that are not valid Unicode, before the digest could filter out the unrelated ones. On Linux a single stray variable was enough to make `pre-push-check` abort even with --rerun, for a feature that is optional. The probe now reads std::env::vars_os(), and env_digest takes OsStr-like pairs: names are matched as text, which is enough because every prefix and exempt name is ASCII, and values are hashed as the bytes they are, so a relevant non-Unicode value still changes the digest and two different ones do not collapse into one. For Unicode values the hashed bytes are the same as before, so recorded stamps stay valid, and a test pins that. The two other environment lookups in the hook (the rerun variable and the git refs) went through std::env::var(..).ok(), which does not panic but reads a non-Unicode value as unset, so a non-Unicode ROCM_PRE_PUSH_RERUN was ignored; they now go through var_os. New tests cover unrelated and relevant non-Unicode bytes in names and values, and a child-process test that runs the real probe under non-Unicode variables; each fails when its part is reverted. Signed-off-by: Jussi Elo * dev: collapse the nested check that drops the earlier pass before a rerun Clippy's collapsible_if lint, with warnings denied as in CI, flagged the nested `if` in `execute` that drops this step's earlier pass before a `--rerun`; it is now one condition with a let chain, and behaves exactly as before. Signed-off-by: Jussi Elo * dev: test that --rerun and ROCM_PRE_PUSH_RERUN reach the stamp logic through the real wiring The tests of the stamp logic built Stamps by hand, so nothing proved that the flag parsed in main.rs, and the variable read by run(), actually arrive at it: a regression that always passed rerun=false would not have been caught. run() is now a thin wrapper over run_in(), which takes the stamp store, the probe, the clock and the environment lookup as a World and holds the wiring from the flag and the variable into Stamps; the pre-push-check arm of main.rs is a function that takes run as a parameter, so a test sees what the dispatch hands over. New tests parse real command lines and check that every flag arrives unchanged, and run run_in over a fake store with a recorded pass: without the flag or the variable the step is skipped, with --rerun or with the variable set it runs, and 0 or empty means off. Each fails when its part is broken: hard-coding rerun=false in the dispatch fails the dispatch test, hard-coding it in run_in fails the flag test, and dropping the variable lookup fails the variable test. The thin real-world run() wrapper, which builds the real git store, is not covered by a unit test. Signed-off-by: Jussi Elo * dev: say exactly what a killed write leaves behind and wrap the pass-stamp docs to the file's width A write goes through a scratch file and a rename and removes the scratch file itself when the rename fails, but a process killed mid-write leaves its scratch file behind until a later write prunes it, once it is more than an hour old; a younger one may belong to a concurrent writer. docs/testing.md and the comment on the age limit now say that, and nothing in the docs or comments claims otherwise. The paragraphs this pull request added to docs/testing.md that ran well past the rest of the file are wrapped to its width; no other text is touched. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 8a152d7db1711ff8777c85301738cbfef6e903dc) Signed-off-by: juhovainio --- .pre-commit-config.yaml | 5 + CONTRIBUTING.md | 2 + docs/testing.md | 51 +- xtask/src/main.rs | 100 ++- xtask/src/prepush.rs | 1184 +++++++++++++++++++++++++- xtask/src/prepush_stamp.rs | 1618 ++++++++++++++++++++++++++++++++++++ 6 files changed, 2922 insertions(+), 38 deletions(-) create mode 100644 xtask/src/prepush_stamp.rs diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ba014b31c7..7dba430fb0 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -161,6 +161,11 @@ repos: # cargo clippy -p e2e-cucumber --test e2e -- -D warnings # cargo test --workspace --all-targets --exclude e2e-cucumber # and `cargo xtask pre-push-check --base ` runs one by hand. + # A step that already passed on the same tree (same cargo command, toolchain + # and environment, clean working tree) is skipped and says so; a green run + # records the pass under the git directory. `ROCM_PRE_PUSH_RERUN=1` (or + # `--rerun` by hand) ignores recorded passes, and `SKIP=` still + # skips a hook. - id: cargo-clippy name: cargo clippy entry: cargo xtask pre-push-check clippy diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index efd8be82fa..35c78f3e19 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -57,6 +57,8 @@ The hooks check the whole workspace when: The hooks also add the crates whose tests read files from outside their own crate. Any Rust file adds `xtask`, whose contract tests scan every source file. `README.md`, `docs/`, and `skills/` add `rocm` and `xtask`. A push that changes no Rust file runs none of the cargo steps, whatever else it changes (a lockfile-only push included). +A step that already passed on this exact content is skipped, and says so: after a green run it records a stamp keyed on the git tree hash of `HEAD`, the exact cargo command (so a pass for a narrow selection never covers a wider one), the toolchain and the `CARGO_*` and `RUST*` environment, under the clone's `.git` directory where every worktree shares it, so a retried push, a manual run before the push or an amend that only rewords the message does not pay for the same checks twice. Stamps are only used and written when the working tree is clean (cargo checks the working tree, not the pushed commit, and an untracked file can hide a missing `git add`), expire after seven days, and do not cover your user-level cargo configuration, installed tools, files git ignores or flaky tests. A skip shows that this tree passed those checks earlier, not that they ran during this push; the `cargo test --workspace` and `cargo clippy --workspace` commands you run by hand are unaffected, and `--rerun` gives a fresh run when you want proof the checks ran for this push. `cargo xtask pre-push-check --rerun`, or `ROCM_PRE_PUSH_RERUN=1` for the hook, ignores them, drops this step's earlier pass so a failed rerun cannot leave it to skip the next run, and records a fresh pass when the step succeeds; `SKIP=` still skips a hook outright, and the commit-signature hook runs on every push. + A green hook is an early warning, not the gate. On pull requests, CI lints the whole workspace, but its Linux test job runs only the affected crates. The whole workspace's tests run in the Windows debug job, in the merge queue, and on `main`. To see what a push would check without compiling anything, run the following command (the steps are `clippy`, `clippy-e2e`, and `test`): ```bash diff --git a/docs/testing.md b/docs/testing.md index 370a5d8ce8..d1ba095afa 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -464,14 +464,61 @@ Known limits, unchanged by the narrowing: the steps check the working tree rathe commit; under prek 0.4.5 a push of several refs, a tag or a deletion runs no hooks; and `-p` builds can resolve Cargo features differently from `--workspace` ones. -Preview what a push would check without compiling: +### Pass stamps + +A step that already passed on this exact content is skipped. After a step's cargo command exits 0, +[prepush_stamp.rs](../xtask/src/prepush_stamp.rs) records a stamp, and the next run of the same step +looks for it first. The stamp stands for: + +- the step and its exact cargo arguments, so a pass recorded for `-p xtask` never lets a later run + over `--workspace` skip (the selection is part of the key, not a separate check); +- the git *tree* hash of `HEAD`, not the commit hash, so an amend that only rewords the message, or + the same content pushed to a second remote, still hits; +- the compiler and cargo identity (`rustc -vV`, `cargo -vV`) and a digest of the `CARGO_*` and + `RUST*` environment (never the values), minus a short list of per-run variables such as + `CARGO_TARGET_DIR` and `CARGO_TERM_COLOR`. The environment is read as raw bytes, so a variable + whose value is not valid Unicode never stops the hook: an unrelated one is ignored and a + relevant one is hashed as the bytes it is. + +The working tree has to be clean for a stamp to be used or written, because cargo checks the working +tree, not the pushed commit: with uncommitted changes the step runs and says why, and an untracked +file counts because it can hide a missing `git add`. The tree is probed again after the step, and +nothing is recorded if cargo changed it (a rewritten `Cargo.lock`, for example). A pass is only +written after cargo succeeded, per step, so a failed `test` does not erase the `clippy` pass that +the retry then skips. + +Stamps live in `rocm-cli-prepush-passes` under the git common directory, so every worktree of a +clone shares them, nothing lands in the work tree and `cargo clean` does not erase them. Each write +drops stamps older than seven days and keeps at most 64. A write goes through a scratch file and a +rename, and removes the scratch file itself when the rename fails; a process killed mid-write +leaves its scratch file behind until a later write prunes it, once it is more than an hour old (a +younger one may belong to a concurrent writer). Any error locating, reading or verifying a +stamp means the step runs (git is run for an explicit directory with `GIT_DIR` and the other +repository-selecting variables removed, which git and prek export to a hook run from a linked +worktree); a stamp that does not match its key exactly, is dated in the future or is +unreadable is ignored (the step names it); a failed write is reported and never fails the hook. + +The skip message says what a stamp does not cover: user-level cargo configuration, installed +tools, files git ignores (the one thing a clean `git status` cannot see) and flaky tests. Force +a run with `--rerun` or `ROCM_PRE_PUSH_RERUN=1` (the hook has no flags, so use the variable +there), or delete the directory the message names. `SKIP=` still skips a hook outright, +and `verify-commits` is a separate hook that checks commit signatures on every push. A skip +proves that this tree passed these checks earlier, not that they ran during this push; for a +fresh run before relying on the result, use `--rerun`. A rerun replaces the recorded pass: it +drops this step's earlier pass before running, so if the rerun fails the next ordinary run runs +the step instead of skipping on the pass it was meant to double-check, and a successful rerun +records a fresh one. Other steps' and other selections' passes are left alone, and `--dry-run` +never touches a stamp. + +Preview what a push would check without compiling (a dry run also says when a real run would skip): ```bash cargo xtask pre-push-check test --dry-run --base origin/main ``` `--base` replaces the range's base; `clippy` and `clippy-e2e` take the same flags. The selection -logic is unit-tested in [prepush.rs](../xtask/src/prepush.rs); as dev tooling, with no change to +logic and the stamps are unit-tested in [prepush.rs](../xtask/src/prepush.rs) and +[prepush_stamp.rs](../xtask/src/prepush_stamp.rs); as dev tooling, with no change to what the `rocm` binary does, it has no Gherkin scenario. The smoke path is the cross-platform local no-fallback acceptance surface. It diff --git a/xtask/src/main.rs b/xtask/src/main.rs index b38fa44d75..acb93f9af8 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -30,6 +30,7 @@ mod package; mod paths; mod powershell; mod prepush; +mod prepush_stamp; mod release_readiness; mod report_mailbox; mod report_rollup; @@ -43,7 +44,7 @@ mod workflow_contract; use std::path::PathBuf; use std::process::ExitCode; -use anyhow::Result; +use anyhow::{Result, bail}; use clap::{Parser, Subcommand}; #[derive(Parser)] @@ -119,6 +120,10 @@ pub(crate) enum Command { /// cannot be determined, and prints why. It is skipped only when the /// selection is known and has nothing for the step to check. CI always /// checks the whole workspace. + /// + /// A step that already passed on this exact tree (same cargo command, + /// toolchain and environment, clean working tree) is skipped, and a pass is + /// recorded after a green run; see `xtask/src/prepush_stamp.rs`. PrePushCheck { /// Which step to run. #[arg(value_enum)] @@ -130,6 +135,10 @@ pub(crate) enum Command { /// Print the scope and the cargo command without running it. #[arg(long)] dry_run: bool, + /// Run the step even when this tree already passed it, and record the + /// new result. `ROCM_PRE_PUSH_RERUN=1` does the same, for the hook. + #[arg(long)] + rerun: bool, }, /// Fail if any first-party crate has gained a normal/build dependency /// edge outside the allowlist pinned in `xtask/src/crate_edges.rs`. @@ -407,11 +416,7 @@ fn run() -> Result<()> { } => signing::verify(public_key.as_deref(), &input, &signature)?, Command::VerifyPinnedKeys => verify_pinned_keys::run()?, Command::Affected { base } => affected::run(base)?, - Command::PrePushCheck { - check, - base, - dry_run, - } => prepush::run(check, base, dry_run)?, + command @ Command::PrePushCheck { .. } => run_pre_push_check(command, prepush::run)?, Command::CheckCrateEdges => crate_edges::run()?, Command::CheckModuleSize => module_size::run()?, Command::CheckModuleSizeAdvisory => module_size::run_advisory()?, @@ -493,6 +498,25 @@ fn run() -> Result<()> { Ok(()) } +/// The `pre-push-check` arm of [`run`]: hand each argument to `run` as parsed. +/// `run` is a parameter so a test sees exactly what arrives, in particular that +/// `--rerun` is not dropped on the way to the stamp logic. +fn run_pre_push_check( + command: Command, + run: impl FnOnce(prepush::Check, Option, bool, bool) -> Result<()>, +) -> Result<()> { + let Command::PrePushCheck { + check, + base, + dry_run, + rerun, + } = command + else { + bail!("not a pre-push-check command"); + }; + run(check, base, dry_run, rerun) +} + fn main() -> ExitCode { match run() { Ok(()) => ExitCode::SUCCESS, @@ -502,3 +526,67 @@ fn main() -> ExitCode { } } } + +#[cfg(test)] +mod tests { + use clap::Parser; + + use super::{Cli, Command}; + + fn rerun(args: &[&str]) -> bool { + let cli = Cli::try_parse_from(args).expect("the arguments parse"); + let Command::PrePushCheck { rerun, .. } = cli.command else { + panic!("expected pre-push-check"); + }; + rerun + } + + /// What the dispatch hands to `prepush::run` for these arguments: the real + /// clap parse, then the `pre-push-check` arm, with `run` recording its input. + fn handed_to_run(args: &[&str]) -> (String, Option, bool, bool) { + let cli = Cli::try_parse_from(args).expect("the arguments parse"); + let mut seen = None; + super::run_pre_push_check(cli.command, |check, base, dry_run, rerun| { + seen = Some((format!("{check:?}"), base, dry_run, rerun)); + Ok(()) + }) + .expect("the dispatch succeeds"); + seen.expect("the arm calls run") + } + + #[test] + fn the_dispatch_hands_every_flag_to_run_unchanged() { + assert_eq!( + handed_to_run(&["xtask", "pre-push-check", "test"]), + ("Test".to_string(), None, false, false) + ); + assert_eq!( + handed_to_run(&[ + "xtask", + "pre-push-check", + "clippy", + "--base", + "origin/main", + "--dry-run", + "--rerun", + ]), + ( + "Clippy".to_string(), + Some("origin/main".to_string()), + true, + true + ) + ); + // `--rerun` alone, and `--dry-run` alone, are not mixed up with each other. + assert!(handed_to_run(&["xtask", "pre-push-check", "test", "--rerun"]).3); + assert!(!handed_to_run(&["xtask", "pre-push-check", "test", "--rerun"]).2); + assert!(handed_to_run(&["xtask", "pre-push-check", "test", "--dry-run"]).2); + assert!(!handed_to_run(&["xtask", "pre-push-check", "test", "--dry-run"]).3); + } + + #[test] + fn pre_push_check_takes_rerun_and_defaults_to_off() { + assert!(rerun(&["xtask", "pre-push-check", "test", "--rerun"])); + assert!(!rerun(&["xtask", "pre-push-check", "test"])); + } +} diff --git a/xtask/src/prepush.rs b/xtask/src/prepush.rs index 403bb4f3e5..9ac9fec066 100644 --- a/xtask/src/prepush.rs +++ b/xtask/src/prepush.rs @@ -31,18 +31,27 @@ //! manual run, it diffs `origin/main...HEAD` like `cargo xtask affected` does. //! `--base` overrides the from-ref. //! +//! A step that already passed on this exact tree is skipped, and a green run is +//! recorded: see [`crate::prepush_stamp`] for what a pass is keyed on (the +//! step's cargo arguments, so a narrow pass never covers a wider run; the tree +//! hash of `HEAD`; the toolchain; the environment) and why a dirty working tree +//! neither uses nor writes one. +//! //! As in [`crate::affected`], the decisions ([`push_from_env`], [`plan`], //! [`hook_selection`], [`resolve_selection`]) are pure and unit-tested; running -//! git and cargo is confined to [`run`], and [`execute`] takes both as -//! parameters so its own behavior (the range it diffs, `--dry-run`, a failing -//! cargo) is tested too. +//! git and cargo is confined to [`run`], and [`execute`] takes both, the stamps +//! and the output as parameters so its own behavior (the range it diffs, +//! `--dry-run`, a failing cargo, a skipped or recorded pass, and what it prints +//! about each) is tested too. use std::collections::BTreeSet; +use std::io::Write; use std::process::Command; use anyhow::{Context, Result, bail}; use crate::affected::{self, DEFAULT_BASE, Graph, Selection}; +use crate::prepush_stamp::{self, Lookup, Stamps}; /// The e2e harness crate. `cargo test` excludes it (its scenarios run through /// `cargo xtask e2e`), and its harness target is linted by its own step. @@ -357,39 +366,94 @@ impl Env { } } +/// What one invocation asks for. +#[derive(Debug, Clone, Default)] +pub struct Request { + /// Replaces the range's base (`--base`). + pub base: Option, + /// Print the plan without running or recording anything (`--dry-run`). + pub dry_run: bool, + /// The refs prek exported. + pub env: Env, +} + /// Run one step: resolve the range (an explicit `base` wins over prek's /// from-ref), pick the affected crates, print the scope, and hand the planned -/// cargo arguments to `run_cargo` unless the step is skipped or `dry_run`. +/// cargo arguments to `run_cargo` unless the step is skipped, `dry_run`, or a +/// pass of exactly this command on this tree is on record (`stamps`). /// -/// `load_graph`, `changed_files` (given the range's base and tip) and -/// `run_cargo` are parameters so the behavior around them is unit-tested. +/// A pass is recorded only after `run_cargo` succeeded. `load_graph`, +/// `changed_files` (given the range's base and tip), `run_cargo` and `out` are +/// parameters so the behavior around them is unit-tested. pub fn execute( check: Check, - base: Option, - dry_run: bool, - env: Env, + request: Request, + stamps: &Stamps<'_>, load_graph: impl FnOnce() -> Result, changed_files: impl FnOnce(&str, &str) -> Result>, mut run_cargo: impl FnMut(&[String]) -> Result<()>, + out: &mut impl Write, ) -> Result<()> { + let label = check.label(); + let Request { base, dry_run, env } = request; let push = push_from_env(base.or(env.from_ref), env.to_ref); let (selection, fallback) = resolve_selection(load_graph, || changed_files(push.base(), &push.to)); - println!("{}", banner(check, &push, &selection, fallback.as_deref())); + writeln!( + out, + "{}", + banner(check, &push, &selection, fallback.as_deref()) + )?; let args = match plan(check, &selection) { Plan::Skip(reason) => { - println!("pre-push-check {}: skipped, {reason}", check.label()); + writeln!(out, "pre-push-check {label}: skipped, {reason}")?; return Ok(()); } Plan::Cargo(args) => args, }; - println!("pre-push-check {}: cargo {}", check.label(), args.join(" ")); + writeln!(out, "pre-push-check {label}: cargo {}", args.join(" "))?; + let key = match stamps.lookup(label, &args) { + Lookup::Hit { name, age_secs } => { + if dry_run { + writeln!( + out, + "pre-push-check {label}: a real run would skip this step, it passed {} ago", + prepush_stamp::age_text(age_secs) + )?; + } else { + let location = stamps.location(); + for line in prepush_stamp::skip_lines(label, &args, age_secs, &name, &location) { + writeln!(out, "{line}")?; + } + } + return Ok(()); + } + Lookup::Miss { key, note } => { + if let Some(note) = note { + writeln!(out, "pre-push-check {label}: {note}")?; + } + key + } + }; if dry_run { return Ok(()); } - run_cargo(&args) + + // `--rerun` replaces the recorded pass: drop it first, so a run that fails + // leaves nothing for the next ordinary run to skip on, and write a fresh one + // only once cargo succeeded. + if stamps.rerun + && let Some(note) = key.as_ref().and_then(|key| stamps.invalidate(key)) + { + writeln!(out, "pre-push-check {label}: {note}")?; + } + run_cargo(&args)?; + if let Some(note) = key.and_then(|key| stamps.record(&key)) { + writeln!(out, "pre-push-check {label}: {note}")?; + } + Ok(()) } /// Run `cargo` with `args`, failing when it does. @@ -405,16 +469,91 @@ fn spawn_cargo(args: &[String]) -> Result<()> { Ok(()) } +/// A process environment variable as text. A value that is not valid Unicode is +/// still a set value (`std::env::var` reports it as an error, which `.ok()` would +/// read as unset, so a non-Unicode `ROCM_PRE_PUSH_RERUN` would be ignored). +fn process_env(name: &str) -> Option { + std::env::var_os(name).map(|value| value.to_string_lossy().into_owned()) +} + +/// What the `pre-push-check` command line asked for. +pub struct Flags { + pub base: Option, + pub dry_run: bool, + pub rerun: bool, +} + +/// The world one invocation runs against: where passes are kept, how the tree +/// and toolchain are probed, the clock, and how a process variable is read. +pub struct World<'a> { + pub probe: &'a dyn prepush_stamp::Probe, + pub store: &'a dyn prepush_stamp::Store, + /// Seconds since the epoch. + pub now: u64, + pub env: &'a dyn Fn(&str) -> Option, +} + /// Entry point for the `pre-push-check` subcommand. -pub fn run(check: Check, base: Option, dry_run: bool) -> Result<()> { - execute( +pub fn run(check: Check, base: Option, dry_run: bool, rerun: bool) -> Result<()> { + let store: Box = match prepush_stamp::FsStore::in_git_common_dir(None) + { + Ok(store) => Box::new(store), + Err(error) => Box::new(prepush_stamp::UnavailableStore(format!("{error:#}"))), + }; + let probe = prepush_stamp::SystemProbe::default(); + run_in( check, - base, - dry_run, - Env::from_lookup(|name| std::env::var(name).ok()), + Flags { + base, + dry_run, + rerun, + }, + &World { + probe: &probe, + store: store.as_ref(), + now: prepush_stamp::now_secs(), + env: &process_env, + }, affected::load_graph, affected::changed_files_between, spawn_cargo, + &mut std::io::stdout().lock(), + ) +} + +/// [`run`] with the world injected: this is where `--rerun` and +/// `ROCM_PRE_PUSH_RERUN` reach the stamp logic, so a test drives it with a fake +/// store and sees whether a recorded pass is honored. +pub fn run_in( + check: Check, + flags: Flags, + world: &World<'_>, + load_graph: impl FnOnce() -> Result, + changed_files: impl FnOnce(&str, &str) -> Result>, + run_cargo: impl FnMut(&[String]) -> Result<()>, + out: &mut impl Write, +) -> Result<()> { + let Flags { + base, + dry_run, + rerun, + } = flags; + let stamps = Stamps::new(world.probe, world.store, world.now, rerun, |name| { + (world.env)(name) + }); + let request = Request { + base, + dry_run, + env: Env::from_lookup(|name| (world.env)(name)), + }; + execute( + check, + request, + &stamps, + load_graph, + changed_files, + run_cargo, + out, ) } @@ -423,6 +562,10 @@ mod tests { use std::collections::{BTreeMap, BTreeSet}; use super::*; + use crate::prepush_stamp::fakes::{FakeProbe, MemStore}; + + /// A fixed clock for the stamp tests. + const NOW: u64 = 1_800_000_000; fn packages(names: &[&str]) -> Selection { Selection::Packages(names.iter().map(|name| (*name).to_string()).collect()) @@ -836,24 +979,32 @@ mod tests { } } - /// Run a step against canned inputs; `files` of `None` makes the diff fail. - /// Returns the result, every `(base, tip)` the diff was asked for, and the - /// argument lines handed to cargo. - fn run_step( + /// Everything one `execute` call did. + struct StepRun { + result: Result<()>, + ranges: Ranges, + /// The argument lines handed to cargo. + cargo: Vec, + /// What the step printed. + out: String, + } + + /// Run a step against canned inputs, with the given stamps; `files` of + /// `None` makes the diff fail. + fn run_step_with( check: Check, - base: Option<&str>, - dry_run: bool, - env: Env, + request: Request, + stamps: &Stamps<'_>, files: Option<&[&str]>, cargo_fails: bool, - ) -> (Result<()>, Ranges, Vec) { + ) -> StepRun { let ranges = std::cell::RefCell::new(Vec::new()); let cargo = std::cell::RefCell::new(Vec::new()); + let mut out = Vec::new(); let result = execute( check, - base.map(String::from), - dry_run, - env, + request, + stamps, || Ok(graph()), |base, tip| { ranges @@ -871,8 +1022,44 @@ mod tests { } Ok(()) }, + &mut out, ); - (result, ranges.into_inner(), cargo.into_inner()) + StepRun { + result, + ranges: ranges.into_inner(), + cargo: cargo.into_inner(), + out: String::from_utf8(out).expect("the step prints UTF-8"), + } + } + + /// Run a step with stamps that never apply (the working tree reads as dirty), + /// so these tests see only the range, `--dry-run` and cargo's result. + /// Returns the result, every `(base, tip)` the diff was asked for, and the + /// argument lines handed to cargo. + fn run_step( + check: Check, + base: Option<&str>, + dry_run: bool, + env: Env, + files: Option<&[&str]>, + cargo_fails: bool, + ) -> (Result<()>, Ranges, Vec) { + let probe = FakeProbe::clean("tree"); + *probe.tree.borrow_mut() = Err("the working tree is dirty".to_string()); + let store = MemStore::default(); + let stamps = Stamps { + probe: &probe, + store: &store, + now: NOW, + rerun: false, + }; + let request = Request { + base: base.map(String::from), + dry_run, + env, + }; + let run = run_step_with(check, request, &stamps, files, cargo_fails); + (run.result, run.ranges, run.cargo) } const XTASK_ONLY: Option<&[&str]> = Some(&["xtask/src/main.rs"]); @@ -982,6 +1169,943 @@ mod tests { assert_eq!(Env::from_lookup(|_| None), Env::default()); } + // ---- pass stamps: skip a step that already passed on this exact tree ---- + + const XTASK_TEST: &str = "test -p xtask --all-targets"; + const WIDE_TEST: &str = "test --workspace --all-targets --exclude e2e-cucumber"; + + fn stamps<'a>(probe: &'a FakeProbe, store: &'a MemStore, now: u64, rerun: bool) -> Stamps<'a> { + Stamps { + probe, + store, + now, + rerun, + } + } + + fn run_at( + stamps: &Stamps<'_>, + check: Check, + files: &[&str], + dry_run: bool, + cargo_fails: bool, + ) -> StepRun { + let request = Request { + dry_run, + ..Request::default() + }; + run_step_with(check, request, stamps, Some(files), cargo_fails) + } + + #[test] + fn a_green_run_is_recorded_and_the_next_identical_run_is_skipped_with_the_reason() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let first = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + first.result.expect("the first run"); + assert_eq!(first.cargo, [XTASK_TEST]); + assert_eq!(store.files.borrow().len(), 1, "a green run is recorded"); + + let second = run_at( + &stamps(&probe, &store, NOW + 90, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + second.result.expect("the skipped run"); + assert!( + second.cargo.is_empty(), + "the step must not run again: {:?}", + second.cargo + ); + assert!( + second + .out + .contains(&format!("pre-push-check test: skipped, this tree already passed `cargo {XTASK_TEST}` 1 min ago")), + "{}", + second.out + ); + // The skip names what it does not cover and how to force a run. + assert!( + second.out.contains("user-level cargo configuration"), + "{}", + second.out + ); + assert!(second.out.contains("--rerun"), "{}", second.out); + assert!(second.out.contains("files git ignores"), "{}", second.out); + assert!(second.out.contains("installed tools"), "{}", second.out); + assert!(second.out.contains("flaky tests"), "{}", second.out); + assert!( + second.out.contains(prepush_stamp::RERUN_VAR), + "{}", + second.out + ); + assert!(second.out.contains(""), "{}", second.out); + assert_eq!(store.files.borrow().len(), 1, "a skip records nothing"); + } + + /// A pushed range that lists no changed files means the range is wrong, not + /// that nothing changed, so on the stamped path too it runs the whole + /// workspace: it is never skipped as "no Rust crate", an earlier narrow pass + /// does not cover it, and it is skipped only by its own earlier pass. + #[test] + fn an_empty_changed_file_list_runs_the_whole_workspace_on_the_stamped_path() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let narrow = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert_eq!(narrow.cargo, [XTASK_TEST]); + + let empty = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &[], + false, + false, + ); + empty.result.expect("the empty-range run"); + assert_eq!(empty.cargo, [WIDE_TEST], "{}", empty.out); + assert!( + empty + .out + .contains("the pushed range lists no changed files"), + "{}", + empty.out + ); + assert!(!empty.out.contains("skipped"), "{}", empty.out); + assert_eq!(store.files.borrow().len(), 2, "the wide pass is recorded"); + + let again = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Test, + &[], + false, + false, + ); + assert!(again.cargo.is_empty(), "{:?}", again.cargo); + assert!( + again + .out + .contains(&format!("this tree already passed `cargo {WIDE_TEST}`")), + "{}", + again.out + ); + + // The lint step falls back the same way and is not skipped as "no crate". + let clippy = run_at( + &stamps(&probe, &store, NOW + 15, false), + Check::Clippy, + &[], + false, + false, + ); + assert_eq!( + clippy.cargo, + ["clippy --workspace --all-targets -- -D warnings"], + "{}", + clippy.out + ); + } + + #[test] + fn a_narrow_pass_never_lets_a_wider_selection_skip() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let narrow = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert_eq!(narrow.cargo, [XTASK_TEST]); + + // The same tree, but the push now also touches the lockfile: the whole + // workspace. The earlier `-p xtask` pass says nothing about that. + let wide = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &["Cargo.lock"], + false, + false, + ); + assert_eq!(wide.cargo, [WIDE_TEST], "{}", wide.out); + assert!(!wide.out.contains("skipped"), "{}", wide.out); + assert_eq!(store.files.borrow().len(), 2, "both passes are on record"); + + // And each is skipped on its own next run. + let again = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert!(again.cargo.is_empty(), "{}", again.out); + } + + #[test] + fn a_different_tree_runs_again() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ) + .result + .expect("first"); + *probe.tree.borrow_mut() = Ok("tree-2".to_string()); + let second = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert_eq!(second.cargo, [XTASK_TEST], "{}", second.out); + assert_eq!(store.files.borrow().len(), 2); + } + + #[test] + fn a_different_toolchain_or_environment_runs_again() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ) + .result + .expect("first"); + + *probe.toolchain.borrow_mut() = Ok("rustc 1.97.0\ncargo 1.97.0\n".to_string()); + let toolchain = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert_eq!( + toolchain.cargo, + [XTASK_TEST], + "a new toolchain must not reuse the pass" + ); + + *probe.env.borrow_mut() = "env-b".to_string(); + let env = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + assert_eq!( + env.cargo, + [XTASK_TEST], + "a new RUSTFLAGS and the like must not reuse the pass" + ); + } + + #[test] + fn a_failed_step_records_nothing_and_a_passed_sibling_still_skips_on_retry() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Clippy, + &files, + false, + false, + ) + .result + .expect("clippy passes"); + let failed = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &files, + false, + true, + ); + failed.result.expect_err("test fails"); + assert_eq!( + store.files.borrow().len(), + 1, + "only the clippy pass is on record" + ); + assert!( + store + .files + .borrow() + .keys() + .all(|name| name.starts_with("clippy-")), + "{:?}", + store.files.borrow().keys() + ); + + // The retry skips the step that passed and re-runs the one that failed. + let clippy = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Clippy, + &files, + false, + false, + ); + assert!(clippy.cargo.is_empty(), "{}", clippy.out); + let test = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Test, + &files, + false, + false, + ); + assert_eq!(test.cargo, [XTASK_TEST]); + assert_eq!(store.files.borrow().len(), 2); + } + + #[test] + fn a_dirty_working_tree_neither_skips_nor_records_and_says_so() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("clean run"); + assert_eq!(store.files.borrow().len(), 1); + + *probe.tree.borrow_mut() = + Err("the working tree has 2 uncommitted or untracked path(s)".to_string()); + let dirty = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &files, + false, + false, + ); + assert_eq!( + dirty.cargo, + [XTASK_TEST], + "a recorded pass is not honored on a dirty tree" + ); + assert!( + dirty + .out + .contains("pass stamps are not used: the working tree has 2 uncommitted"), + "{}", + dirty.out + ); + assert_eq!(*store.writes.borrow(), 1, "and no pass is recorded from it"); + } + + #[test] + fn a_tree_the_run_changed_is_not_recorded() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let stamps = stamps(&probe, &store, NOW, false); + let mut out = Vec::new(); + execute( + Check::Test, + Request::default(), + &stamps, + || Ok(graph()), + |_, _| Ok(vec!["xtask/src/main.rs".to_string()]), + |_| { + // cargo rewrote a tracked file (`Cargo.lock`, say) while it ran. + *probe.tree.borrow_mut() = + Err("the working tree has 1 uncommitted path".to_string()); + Ok(()) + }, + &mut out, + ) + .expect("the step itself passed"); + let out = String::from_utf8(out).expect("utf-8"); + assert!( + out.contains("pass not recorded: the working tree has 1 uncommitted"), + "{out}" + ); + assert!( + store.files.borrow().is_empty(), + "nothing is recorded for a tree cargo changed" + ); + } + + /// `--rerun` is used when something the key omits may have changed (user-level + /// cargo configuration, an installed tool). If that rerun fails, the earlier + /// pass must not survive to skip the next ordinary run. + #[test] + fn a_failed_rerun_does_not_leave_the_earlier_pass_to_skip_the_next_ordinary_run() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("the first, passing run"); + assert_eq!(store.files.borrow().len(), 1); + + let failed = run_at( + &stamps(&probe, &store, NOW + 100, true), + Check::Test, + &files, + false, + true, + ); + assert!(failed.result.is_err(), "the rerun fails"); + assert_eq!(failed.cargo, [XTASK_TEST], "--rerun must run the step"); + assert!( + store.files.borrow().is_empty(), + "the earlier pass is dropped before the rerun and nothing replaces it" + ); + + let retry = run_at( + &stamps(&probe, &store, NOW + 200, false), + Check::Test, + &files, + false, + false, + ); + retry.result.expect("the ordinary retry"); + assert_eq!( + retry.cargo, + [XTASK_TEST], + "the ordinary retry must run the step: {}", + retry.out + ); + assert!(!retry.out.contains("skipped"), "{}", retry.out); + assert_eq!( + store.files.borrow().len(), + 1, + "the passing retry is recorded" + ); + } + + #[test] + fn a_failed_rerun_keeps_the_passes_of_other_steps_and_other_selections() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + for (check, files) in [ + (Check::Test, &["xtask/src/main.rs"][..]), + (Check::Test, &["Cargo.lock"][..]), + (Check::Clippy, &["xtask/src/main.rs"][..]), + ] { + run_at( + &stamps(&probe, &store, NOW, false), + check, + files, + false, + false, + ) + .result + .expect("a recorded pass"); + } + assert_eq!(store.files.borrow().len(), 3); + + let failed = run_at( + &stamps(&probe, &store, NOW + 100, true), + Check::Test, + &["xtask/src/main.rs"], + false, + true, + ); + assert!(failed.result.is_err()); + assert_eq!( + store.files.borrow().len(), + 2, + "only the stamp the rerun replaces is dropped" + ); + // The sibling passes still skip. + for (check, files) in [ + (Check::Test, &["Cargo.lock"][..]), + (Check::Clippy, &["xtask/src/main.rs"][..]), + ] { + let again = run_at( + &stamps(&probe, &store, NOW + 200, false), + check, + files, + false, + false, + ); + assert!(again.cargo.is_empty(), "{:?}: {}", again.cargo, again.out); + } + } + + #[test] + fn a_dry_run_rerun_leaves_every_stamp_in_place() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("the first run"); + let before = store.files.borrow().clone(); + + let dry = run_at( + &stamps(&probe, &store, NOW + 100, true), + Check::Test, + &files, + true, + false, + ); + dry.result.expect("the dry run"); + assert!( + dry.cargo.is_empty(), + "a dry run runs nothing: {:?}", + dry.cargo + ); + assert_eq!( + *store.files.borrow(), + before, + "a dry run neither removes nor rewrites a stamp" + ); + } + + #[test] + fn a_passing_rerun_replaces_the_pass_with_a_fresh_one() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("the first run"); + run_at( + &stamps(&probe, &store, NOW + 100, true), + Check::Test, + &files, + false, + false, + ) + .result + .expect("the passing rerun"); + assert_eq!( + store.files.borrow().len(), + 1, + "one pass, replaced not added" + ); + let after = run_at( + &stamps(&probe, &store, NOW + 130, false), + Check::Test, + &files, + false, + false, + ); + assert!(after.cargo.is_empty(), "{}", after.out); + assert!( + after.out.contains( + "this tree already passed `cargo test -p xtask --all-targets` under a minute ago" + ), + "the fresh pass is the one that skips: {}", + after.out + ); + } + + #[test] + fn rerun_ignores_a_recorded_pass_and_records_a_fresh_one() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("first"); + let forced = run_at( + &stamps(&probe, &store, NOW + 100, true), + Check::Test, + &files, + false, + false, + ); + assert_eq!(forced.cargo, [XTASK_TEST], "--rerun must run the step"); + assert!( + forced.out.contains("recorded passes are ignored"), + "{}", + forced.out + ); + assert!( + forced.out.contains(prepush_stamp::RERUN_VAR), + "{}", + forced.out + ); + let recorded: Vec = store + .files + .borrow() + .values() + .map(|(text, _)| text.clone()) + .collect(); + assert_eq!(recorded.len(), 1, "the new pass replaces the old one"); + assert!( + recorded[0].contains(&format!("passed-at: {}", NOW + 100)), + "{recorded:?}" + ); + } + + #[test] + fn a_dry_run_reports_a_hit_and_never_runs_or_writes() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + let miss = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + true, + false, + ); + assert!(miss.cargo.is_empty()); + assert!(store.files.borrow().is_empty(), "a dry run records nothing"); + assert!(!miss.out.contains("would skip"), "{}", miss.out); + + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("real run"); + let writes = *store.writes.borrow(); + let hit = run_at( + &stamps(&probe, &store, NOW + 3600, false), + Check::Test, + &files, + true, + false, + ); + assert!(hit.cargo.is_empty()); + assert!( + hit.out + .contains("a real run would skip this step, it passed 1 h ago"), + "{}", + hit.out + ); + assert_eq!(*store.writes.borrow(), writes); + } + + #[test] + fn a_failed_write_is_reported_and_never_fails_the_step() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + *store.fail_writes.borrow_mut() = true; + let run = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ); + run.result + .expect("a stamp that cannot be written must not fail a green hook"); + assert_eq!(run.cargo, [XTASK_TEST]); + assert!( + run.out.contains( + "pass not recorded: could not write it under (read-only file system)" + ), + "{}", + run.out + ); + } + + #[test] + fn a_store_that_cannot_be_set_up_still_runs_the_step_and_says_why() { + let probe = FakeProbe::clean("tree-1"); + let store = prepush_stamp::UnavailableStore("not a git repository".to_string()); + let stamps = Stamps { + probe: &probe, + store: &store, + now: NOW, + rerun: false, + }; + let run = run_at(&stamps, Check::Test, &["xtask/src/main.rs"], false, false); + run.result.expect("the step runs"); + assert_eq!(run.cargo, [XTASK_TEST]); + assert!( + run.out + .contains("could not read the pass stamp (not a git repository)"), + "{}", + run.out + ); + assert!( + run.out + .contains("pass not recorded: could not write it under the stamp directory"), + "{}", + run.out + ); + } + + #[test] + fn a_corrupted_stamp_is_ignored_and_replaced_by_a_real_one() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let files = ["xtask/src/main.rs"]; + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &files, + false, + false, + ) + .result + .expect("first"); + let name = store.files.borrow().keys().next().expect("a stamp").clone(); + store + .files + .borrow_mut() + .insert(name.clone(), ("not a stamp".to_string(), NOW)); + + let second = run_at( + &stamps(&probe, &store, NOW + 5, false), + Check::Test, + &files, + false, + false, + ); + assert_eq!( + second.cargo, + [XTASK_TEST], + "garbage under the right name must not skip" + ); + assert!( + second + .out + .contains(&format!("ignoring the pass stamp {name}")), + "{}", + second.out + ); + let third = run_at( + &stamps(&probe, &store, NOW + 10, false), + Check::Test, + &files, + false, + false, + ); + assert!( + third.cargo.is_empty(), + "the replacement is a real stamp: {}", + third.out + ); + } + + #[test] + fn a_skipped_selection_neither_looks_for_nor_writes_a_stamp() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let run = run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["CHANGELOG.md"], + false, + false, + ); + assert!(run.cargo.is_empty()); + assert!( + run.out.contains("skipped, no Rust crate is affected"), + "{}", + run.out + ); + assert_eq!(*store.writes.borrow(), 0); + } + + #[test] + fn a_head_that_moved_during_the_step_is_not_recorded_and_the_note_names_head() { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let stamps = stamps(&probe, &store, NOW, false); + let mut out = Vec::new(); + execute( + Check::Test, + Request::default(), + &stamps, + || Ok(graph()), + |_, _| Ok(vec!["xtask/src/main.rs".to_string()]), + |_| { + // HEAD is a different commit by the time the step has finished. + *probe.tree.borrow_mut() = Ok("tree-2".to_string()); + Ok(()) + }, + &mut out, + ) + .expect("the step itself passed"); + let out = String::from_utf8(out).expect("utf-8"); + assert!( + out.contains("pass not recorded: HEAD moved while the step ran"), + "{out}" + ); + assert!( + store.files.borrow().is_empty(), + "nothing is written for a tree that was not checked" + ); + assert_eq!(*store.writes.borrow(), 0); + } + + /// A stamp for `xtask`'s test step on `tree-1`, written by a normal run. + fn with_a_recorded_pass() -> (FakeProbe, MemStore) { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + run_at( + &stamps(&probe, &store, NOW, false), + Check::Test, + &["xtask/src/main.rs"], + false, + false, + ) + .result + .expect("the first run"); + assert_eq!(store.files.borrow().len(), 1); + (probe, store) + } + + // ---- run_in: the flag and the variable reach the stamp logic ---- + + /// Drive the real wiring below `main`: `run_in` builds the stamps from the + /// flag and the environment, here over a fake store. With `recorded_pass` + /// a first plain run records a pass for the tree; the run that follows is + /// the one measured, and the result says whether it ran cargo. + fn measured_run_ran_cargo(rerun: bool, rerun_var: Option<&str>, recorded_pass: bool) -> bool { + let (probe, store) = (FakeProbe::clean("tree-1"), MemStore::default()); + let env = |name: &str| { + if name == prepush_stamp::RERUN_VAR { + rerun_var.map(String::from) + } else { + None + } + }; + let ran = std::cell::Cell::new(false); + let run_once = |now: u64, rerun: bool| { + ran.set(false); + run_in( + Check::Test, + Flags { + base: None, + dry_run: false, + rerun, + }, + &World { + probe: &probe, + store: &store, + now, + env: &env, + }, + || Ok(graph()), + |_, _| Ok(vec!["xtask/src/main.rs".to_string()]), + |_| { + ran.set(true); + Ok(()) + }, + &mut Vec::new(), + ) + .expect("the step"); + ran.get() + }; + if recorded_pass { + // Nothing is on record yet, so this run runs cargo and records a pass. + assert!(run_once(NOW, false), "the first run runs the step"); + assert_eq!(store.files.borrow().len(), 1, "and records a pass"); + } + run_once(NOW + 60, rerun) + } + + #[test] + fn without_the_flag_or_the_variable_a_recorded_pass_skips_the_step() { + assert!(!measured_run_ran_cargo(false, None, true)); + assert!( + !measured_run_ran_cargo(false, Some("0"), true), + "0 means off" + ); + assert!( + !measured_run_ran_cargo(false, Some(""), true), + "empty means off" + ); + } + + #[test] + fn the_flag_reaches_the_stamp_logic_and_reruns_despite_a_recorded_pass() { + assert!(measured_run_ran_cargo(true, None, true)); + } + + #[test] + fn the_variable_reaches_the_stamp_logic_and_reruns_despite_a_recorded_pass() { + assert!(measured_run_ran_cargo(false, Some("1"), true)); + } + + #[test] + fn with_no_recorded_pass_the_step_runs_either_way() { + assert!(measured_run_ran_cargo(false, None, false)); + assert!(measured_run_ran_cargo(true, None, false)); + } + + #[test] + fn the_rerun_flag_forces_a_run_despite_a_matching_stamp() { + let (probe, store) = with_a_recorded_pass(); + let forced = Stamps::new(&probe, &store, NOW + 5, true, |_| None); + let run = run_at(&forced, Check::Test, &["xtask/src/main.rs"], false, false); + assert_eq!(run.cargo, [XTASK_TEST], "{}", run.out); + assert!( + run.out.contains("recorded passes are ignored"), + "{}", + run.out + ); + assert!(!run.out.contains("skipped"), "{}", run.out); + } + + #[test] + fn the_rerun_variable_forces_a_run_despite_a_matching_stamp() { + let (probe, store) = with_a_recorded_pass(); + let lookup = |name: &str| (name == prepush_stamp::RERUN_VAR).then(|| "1".to_string()); + let forced = Stamps::new(&probe, &store, NOW + 5, false, lookup); + let run = run_at(&forced, Check::Test, &["xtask/src/main.rs"], false, false); + assert_eq!(run.cargo, [XTASK_TEST], "{}", run.out); + assert!( + run.out.contains("recorded passes are ignored"), + "{}", + run.out + ); + } + + #[test] + fn without_the_flag_or_with_the_variable_off_the_same_stamp_still_skips() { + let (probe, store) = with_a_recorded_pass(); + let off = |name: &str| (name == prepush_stamp::RERUN_VAR).then(|| "0".to_string()); + for stamps in [ + Stamps::new(&probe, &store, NOW + 5, false, |_| None), + Stamps::new(&probe, &store, NOW + 5, false, off), + ] { + let run = run_at(&stamps, Check::Test, &["xtask/src/main.rs"], false, false); + assert!(run.cargo.is_empty(), "{}", run.out); + assert!( + run.out.contains("skipped, this tree already passed"), + "{}", + run.out + ); + } + } + // ---- drift guard: every cross-crate read must be covered ---- use std::path::{Path, PathBuf}; diff --git a/xtask/src/prepush_stamp.rs b/xtask/src/prepush_stamp.rs new file mode 100644 index 0000000000..46d5d0290a --- /dev/null +++ b/xtask/src/prepush_stamp.rs @@ -0,0 +1,1618 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Pass stamps for `cargo xtask pre-push-check`: remember that a cargo step +//! already passed on exactly this content, and skip it the next time. +//! +//! The pre-push hook has no memory, so a push retried after a dropped +//! connection, a manual run before the push, an amend that only rewords the +//! message, or the same tree pushed to a second remote re-ran every step on a +//! tree it had just checked. A stamp is written after a step's cargo command +//! exits 0 and is looked up before the next run of that step. +//! +//! **What a stamp stands for.** One step (`clippy`, `clippy-e2e` or `test`), the +//! exact cargo arguments it ran, the git *tree* hash of `HEAD` (not the commit +//! hash, so an amend or a re-commit with identical content still hits), the +//! compiler and cargo identity (`rustc -vV`, `cargo -vV`), and a digest of the +//! `CARGO_*`/`RUST*` environment. Because the arguments are part of the key, a +//! pass recorded for a narrow selection can never let a later, wider selection +//! skip: `-p xtask` and `--workspace` are different stamps. +//! +//! **The working tree must be clean.** Cargo checks the working tree, not the +//! pushed commit, so a stamp keyed on `HEAD`'s tree is only true when the +//! working tree *is* that tree. With uncommitted changes or untracked files +//! (an untracked `mod` target can hide a missing `git add`) a stamp is neither +//! honored nor written, and the step says why. After the run the tree is checked +//! again: if cargo changed it (a rewritten `Cargo.lock`, say), nothing is +//! recorded. +//! +//! **Fail safe.** Every error while locating the store, reading, parsing or +//! verifying a stamp means the step runs; a stamp that does not match its key +//! byte for byte, is unreadable, belongs to a different format, is dated in the +//! future or is older than [`MAX_AGE_SECS`] is ignored; a failed write is +//! reported and never fails the hook. A stamp is only written after cargo +//! succeeded. +//! +//! **Where.** Under the git *common* directory, so every linked worktree of a +//! clone shares the stamps, nothing lands in the work tree (no ignore rule, no +//! chance of committing one), and `cargo clean` does not erase them. Each write +//! prunes stamps older than [`MAX_AGE_SECS`] and keeps at most [`MAX_STAMPS`]. +//! +//! **Not covered**, and said in the skip message: user-level cargo +//! configuration (`~/.cargo/config.toml`), installed tools, files git ignores, +//! network-dependent or flaky tests. `--rerun` or [`RERUN_VAR`] ignores recorded +//! passes: it drops this step's earlier pass before running, so a rerun that +//! fails leaves nothing for the next ordinary run to skip on, and it records a +//! fresh pass when the step succeeds. `SKIP=` still skips a hook +//! outright. The `verify-commits` hook is a separate hook and runs every time. +//! +//! As elsewhere in this crate the decisions are pure and the world is injected: +//! git and the toolchain sit behind [`Probe`], the filesystem behind [`Store`], +//! and the clock is a parameter, so the rules above are unit-tested without +//! cargo, git or a disk. + +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{Context, Result, bail}; +use sha2::{Digest, Sha256}; + +use crate::affected; + +/// Setting this to anything but empty or `0` ignores recorded passes. +pub const RERUN_VAR: &str = "ROCM_PRE_PUSH_RERUN"; + +/// Directory name under the git common directory. +const STAMP_DIR: &str = "rocm-cli-prepush-passes"; +const STAMP_EXT: &str = ".stamp"; +/// Scratch files a write goes through before it is renamed into place. +const TMP_EXT: &str = ".tmp"; +const FORMAT_LINE: &str = "rocm-cli pre-push pass stamp v1"; + +/// The revision whose hash identifies the checked content. +const HEAD_TREE: &str = "HEAD^{tree}"; + +/// A pass older than this is ignored and pruned: toolchains, dependencies and +/// the registry move on, and a week is longer than any realistic retry loop. +pub const MAX_AGE_SECS: u64 = 7 * 24 * 60 * 60; + +/// At most this many stamps are kept; the oldest go first. +pub const MAX_STAMPS: usize = 64; + +/// A write killed mid-way leaves its scratch file behind until a later write +/// prunes it, once it is older than this: an older scratch file belongs to a +/// write that died, a younger one may be a concurrent writer's and is left alone. +pub const TMP_MAX_AGE_SECS: u64 = 60 * 60; + +/// A stamp dated this far ahead of the clock is treated as corrupt. +const FUTURE_SLACK_SECS: u64 = 5 * 60; + +/// Environment variable prefixes that can change what a cargo step does. +/// Over-including only costs a re-run; leaving one out could skip a step that +/// would now fail. +const ENV_PREFIXES: &[&str] = &["CARGO_", "RUST"]; + +/// Variables under those prefixes that differ between runs without changing the +/// outcome: where artifacts go, terminal styling, and what `cargo run` sets for +/// the `xtask` binary itself. +const ENV_IGNORED: &[&str] = &[ + "CARGO_TARGET_DIR", + "CARGO_MAKEFLAGS", + "CARGO_MANIFEST_DIR", + "CARGO_MANIFEST_PATH", + "CARGO_BIN_NAME", + "CARGO_CRATE_NAME", + "CARGO_PRIMARY_PACKAGE", + "RUST_BACKTRACE", + "RUST_LIB_BACKTRACE", +]; +const ENV_IGNORED_PREFIXES: &[&str] = &["CARGO_TERM_", "CARGO_PKG_", "CARGO_BIN_EXE_"]; + +fn hex_digest(text: &str) -> String { + hex_digest_bytes(text.as_bytes()) +} + +fn hex_digest_bytes(bytes: &[u8]) -> String { + format!("{:x}", Sha256::digest(bytes)) +} + +/// Whether `--rerun` or [`RERUN_VAR`] asks for recorded passes to be ignored. +pub fn rerun_requested(flag: bool, lookup: impl Fn(&str) -> Option) -> bool { + flag || lookup(RERUN_VAR).is_some_and(|value| { + let value = value.trim(); + !value.is_empty() && value != "0" + }) +} + +/// Digest of the environment variables that can change a cargo step, in a +/// stable order. Only the digest is kept, never a value. +pub fn env_digest(vars: impl IntoIterator) -> String +where + K: AsRef, + V: AsRef, +{ + // The process environment is read as `OsString`s: a variable holding bytes + // that are not valid Unicode (on Linux, any variable at all) must not abort + // an optional check, so nothing here needs a `String`. Names are matched as + // text, which is enough because every prefix and exempt name is ASCII, and + // values are hashed as the bytes they are. + let mut relevant: Vec<(Vec, Vec)> = vars + .into_iter() + .filter(|(name, _)| { + let name = name.as_ref().to_string_lossy(); + ENV_PREFIXES.iter().any(|prefix| name.starts_with(prefix)) + && !ENV_IGNORED.contains(&name.as_ref()) + && !ENV_IGNORED_PREFIXES + .iter() + .any(|prefix| name.starts_with(prefix)) + }) + .map(|(name, value)| { + ( + name.as_ref().as_encoded_bytes().to_vec(), + value.as_ref().as_encoded_bytes().to_vec(), + ) + }) + .collect(); + relevant.sort(); + let mut bytes = Vec::new(); + for (name, value) in relevant { + bytes.extend_from_slice(&name); + bytes.push(b'='); + bytes.extend_from_slice(&value); + bytes.push(b'\n'); + } + hex_digest_bytes(&bytes) +} + +/// Everything one pass depends on. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Key { + step: String, + args: String, + tree: String, + toolchain: String, + env: String, +} + +impl Key { + fn canonical(&self) -> String { + format!( + "step: {}\ncargo: {}\ntree: {}\ntoolchain: {}\nenv: {}\n", + self.step, self.args, self.tree, self.toolchain, self.env + ) + } + + fn file_name(&self) -> String { + let digest = hex_digest(&self.canonical()); + format!("{}-{}{STAMP_EXT}", self.step, &digest[..32]) + } + + fn stamp_text(&self, passed_at: u64) -> String { + format!( + "{FORMAT_LINE}\n{}passed-at: {passed_at}\n", + self.canonical() + ) + } +} + +/// What git and the toolchain say about the state being checked. +pub trait Probe { + /// The tree hash of `HEAD`, only when the working tree is exactly that tree. + /// `Err` carries the reason stamps cannot be trusted right now. + fn clean_tree(&self) -> std::result::Result; + /// A description of the compiler and cargo that will run the step. + fn toolchain(&self) -> std::result::Result; + /// [`env_digest`] of the current environment. + fn env_digest(&self) -> String; +} + +/// Where stamps are kept. Names are file names inside the store. +pub trait Store { + fn read(&self, name: &str) -> Result>; + fn write(&self, name: &str, contents: &str) -> Result<()>; + /// Every stamp as `(name, modified time in seconds since the epoch)`. + fn list(&self) -> Result>; + fn remove(&self, name: &str) -> Result<()>; + /// Where the stamps are, for messages. + fn location(&self) -> String; +} + +/// The outcome of looking for a recorded pass. +#[derive(Debug, PartialEq, Eq)] +pub enum Lookup { + /// A matching pass is on record: skip the step. + Hit { name: String, age_secs: u64 }, + /// Run the step. `key` is `Some` when a pass can be recorded afterwards, and + /// `note` explains anything the user should know (a dirty tree, a rejected + /// stamp, `--rerun`). + Miss { + key: Option, + note: Option, + }, +} + +/// Stamp handling for one invocation, with the world injected. +pub struct Stamps<'a> { + pub probe: &'a dyn Probe, + pub store: &'a dyn Store, + /// Seconds since the epoch. + pub now: u64, + /// Ignore recorded passes (but still record a new one). + pub rerun: bool, +} + +impl<'a> Stamps<'a> { + /// The stamps for one invocation. `rerun_flag` is `--rerun`; `lookup` reads + /// the environment for [`RERUN_VAR`], so tests need no process environment. + pub fn new( + probe: &'a dyn Probe, + store: &'a dyn Store, + now: u64, + rerun_flag: bool, + lookup: impl Fn(&str) -> Option, + ) -> Self { + Self { + probe, + store, + now, + rerun: rerun_requested(rerun_flag, lookup), + } + } +} + +impl Stamps<'_> { + fn key(&self, step: &str, args: &[String]) -> std::result::Result { + let tree = self.probe.clean_tree()?; + let toolchain = hex_digest(&self.probe.toolchain()?); + Ok(Key { + step: step.to_string(), + args: args.join(" "), + tree, + toolchain, + env: self.probe.env_digest(), + }) + } + + /// Look for a pass of `step` run with `args` on the current tree. + pub fn lookup(&self, step: &str, args: &[String]) -> Lookup { + let key = match self.key(step, args) { + Ok(key) => key, + Err(why) => { + return Lookup::Miss { + key: None, + note: Some(format!("pass stamps are not used: {why}")), + }; + } + }; + if self.rerun { + return Lookup::Miss { + key: Some(key), + note: Some(format!( + "recorded passes are ignored because --rerun or {RERUN_VAR} is set" + )), + }; + } + let name = key.file_name(); + let text = match self.store.read(&name) { + Ok(Some(text)) => text, + Ok(None) => { + return Lookup::Miss { + key: Some(key), + note: None, + }; + } + Err(error) => { + return Lookup::Miss { + key: Some(key), + note: Some(format!( + "could not read the pass stamp ({error:#}); running the step" + )), + }; + } + }; + match self.verify(&key, &text) { + Ok(Some(age_secs)) => Lookup::Hit { name, age_secs }, + Ok(None) => Lookup::Miss { + key: Some(key), + note: None, + }, + Err(why) => Lookup::Miss { + key: Some(key), + note: Some(format!("ignoring the pass stamp {name}: {why}")), + }, + } + } + + /// `Ok(Some(age))` for a valid, fresh stamp, `Ok(None)` for an expired one, + /// `Err(reason)` for one that is not what its name says. + fn verify(&self, key: &Key, text: &str) -> std::result::Result, String> { + let expected = format!("{FORMAT_LINE}\n{}", key.canonical()); + let rest = text + .strip_prefix(&expected) + .ok_or("it does not match this step, tree, toolchain and environment")?; + let passed_at: u64 = rest + .strip_prefix("passed-at: ") + .and_then(|value| value.trim().parse().ok()) + .ok_or("its time stamp is unreadable")?; + if passed_at > self.now.saturating_add(FUTURE_SLACK_SECS) { + return Err("it is dated in the future".to_string()); + } + let age = self.now.saturating_sub(passed_at); + Ok((age <= MAX_AGE_SECS).then_some(age)) + } + + /// Drop the recorded pass for exactly this key, before a rerun that will + /// replace it. A rerun exists because something the key does not cover may + /// have changed (user-level cargo configuration, an installed tool), so if it + /// fails the earlier pass must not survive to skip the next ordinary run. + /// Other steps' and other trees' stamps are different files and stay. + /// Returns a note when the stamp could not be removed. + pub fn invalidate(&self, key: &Key) -> Option { + self.store.remove(&key.file_name()).err().map(|error| { + format!( + "could not remove the earlier pass stamp ({error:#}); if this run fails, the \ + next ordinary run may still skip the step" + ) + }) + } + + /// Record that the step passed. Returns a note when nothing was recorded. + /// + /// The tree is probed again first: a stamp is only true when the working + /// tree is still the one the key was computed for, and a step that rewrote + /// it (`Cargo.lock`, a generated file) did not check `HEAD`'s tree. + pub fn record(&self, key: &Key) -> Option { + match self.probe.clean_tree() { + Ok(tree) if tree == key.tree => {} + Ok(_) => return Some("pass not recorded: HEAD moved while the step ran".to_string()), + Err(why) => return Some(format!("pass not recorded: {why}")), + } + if let Err(error) = self + .store + .write(&key.file_name(), &key.stamp_text(self.now)) + { + return Some(format!( + "pass not recorded: could not write it under {} ({error:#})", + self.store.location() + )); + } + self.prune(); + None + } + + /// Drop stamps past [`MAX_AGE_SECS`], then the oldest beyond [`MAX_STAMPS`], + /// and scratch files left by a write that died ([`TMP_MAX_AGE_SECS`]). + /// Best effort: a file that cannot be removed is only wasted space. + fn prune(&self) { + let Ok(mut entries) = self.store.list() else { + return; + }; + entries.retain(|(name, modified)| { + let age = self.now.saturating_sub(*modified); + let expired = if name.ends_with(TMP_EXT) { + age > TMP_MAX_AGE_SECS + } else { + age > MAX_AGE_SECS + }; + if expired { + let _ = self.store.remove(name); + } + !expired + }); + // A scratch file is not a pass: it does not count toward the cap. + entries.retain(|(name, _)| !name.ends_with(TMP_EXT)); + entries.sort_by_key(|(_, modified)| std::cmp::Reverse(*modified)); + for (name, _) in entries.iter().skip(MAX_STAMPS) { + let _ = self.store.remove(name); + } + } + + /// Where the stamps live, for the skip message. + pub fn location(&self) -> String { + self.store.location() + } +} + +/// `3 min`, `2 h`, `1 d`: how long ago a pass was recorded. +pub fn age_text(secs: u64) -> String { + match secs { + 0..=59 => "under a minute".to_string(), + 60..=3599 => format!("{} min", secs / 60), + 3600..=86_399 => format!("{} h", secs / 3600), + _ => format!("{} d", secs / 86_400), + } +} + +/// The lines printed when a step is skipped because its pass is on record. +pub fn skip_lines( + step: &str, + args: &[String], + age_secs: u64, + name: &str, + location: &str, +) -> Vec { + vec![ + format!( + "pre-push-check {step}: skipped, this tree already passed `cargo {}` {} ago \ + (stamp {name})", + args.join(" "), + age_text(age_secs) + ), + format!( + "pre-push-check {step}: that does not cover your user-level cargo configuration, \ + installed tools, files git ignores or flaky tests; run it again with --rerun \ + or {RERUN_VAR}=1, or delete {location}" + ), + ] +} + +// ---- the real world: git, the toolchain, the filesystem ---- + +/// A `git` command for the repository in `dir`. With an explicit directory the +/// repository-selecting variables are removed ([`affected::scrub_git_env`]): git +/// and prek export an absolute `GIT_DIR` to a pre-push hook run from a linked +/// worktree, and `git -C ` would otherwise still act on the repository +/// being pushed. With `None` the environment is kept, because in the hook that +/// inherited `GIT_DIR` correctly names the repository being pushed. +fn git_command(dir: Option<&Path>) -> Command { + let mut git = Command::new("git"); + if let Some(dir) = dir { + affected::scrub_git_env(&mut git).arg("-C").arg(dir); + } + git +} + +/// Run `git` with `args` (in `dir`, or the current directory) and return its +/// trimmed stdout. +fn git_stdout(dir: Option<&Path>, args: &[&str]) -> std::result::Result { + let output = git_command(dir) + .args(args) + .output() + .map_err(|error| format!("could not run git: {error}"))?; + if !output.status.success() { + return Err(format!( + "`git {}` failed: {}", + args.join(" "), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(String::from_utf8_lossy(&output.stdout).trim().to_string()) +} + +/// ` -vV`, or the reason it could not run. +fn version_text(tool: &std::ffi::OsStr) -> std::result::Result { + let output = Command::new(tool) + .arg("-vV") + .output() + .map_err(|error| format!("could not run {}: {error}", tool.to_string_lossy()))?; + if !output.status.success() { + return Err(format!("`{} -vV` failed", tool.to_string_lossy())); + } + Ok(String::from_utf8_lossy(&output.stdout).into_owned()) +} + +/// [`Probe`] backed by the `git` and toolchain on `PATH`, for the repository in +/// `dir` (the current directory when `None`). +#[derive(Default)] +pub struct SystemProbe { + pub dir: Option, +} + +impl Probe for SystemProbe { + fn clean_tree(&self) -> std::result::Result { + let dir = self.dir.as_deref(); + let status = git_stdout(dir, &["status", "--porcelain", "--untracked-files=normal"])?; + let dirty = status + .lines() + .filter(|line| !line.trim().is_empty()) + .count(); + if dirty > 0 { + return Err(format!( + "the working tree has {dirty} uncommitted or untracked path(s), and cargo checks \ + the working tree rather than HEAD; commit or remove them (`git status` lists them)" + )); + } + git_stdout(dir, &["rev-parse", HEAD_TREE]) + } + + fn toolchain(&self) -> std::result::Result { + let cargo = std::env::var_os("CARGO").unwrap_or_else(|| "cargo".into()); + Ok(format!( + "{}\n{}", + version_text(std::ffi::OsStr::new("rustc"))?, + version_text(&cargo)? + )) + } + + fn env_digest(&self) -> String { + env_digest(std::env::vars_os()) + } +} + +/// Stamps in a directory on disk. +pub struct FsStore { + dir: PathBuf, +} + +impl FsStore { + /// The stamp directory under the git common directory of the repository in + /// `dir` (the current directory when `None`), shared by every worktree of + /// that clone. + pub fn in_git_common_dir(dir: Option<&Path>) -> Result { + let common = git_stdout(dir, &["rev-parse", "--git-common-dir"]) + .map_err(|why| anyhow::anyhow!(why))?; + let common = PathBuf::from(common); + // git prints the path relative to where it ran when it is inside there. + let common = if common.is_absolute() { + common + } else { + let base = match dir { + Some(dir) => dir.to_path_buf(), + None => std::env::current_dir().context("could not read the current directory")?, + }; + base.join(common) + }; + Ok(Self { + dir: common.join(STAMP_DIR), + }) + } + + fn path(&self, name: &str) -> Result { + if name.is_empty() || name == "." || name == ".." || name.contains(['/', '\\']) { + bail!("refusing the stamp name {name:?}"); + } + Ok(self.dir.join(name)) + } +} + +impl Store for FsStore { + fn read(&self, name: &str) -> Result> { + match std::fs::read_to_string(self.path(name)?) { + Ok(text) => Ok(Some(text)), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(error) => Err(error).context("failed to read the stamp"), + } + } + + fn write(&self, name: &str, contents: &str) -> Result<()> { + std::fs::create_dir_all(&self.dir) + .with_context(|| format!("failed to create {}", self.dir.display()))?; + let target = self.path(name)?; + // Write beside the target and rename, so a reader never sees half a stamp. + let scratch = self + .dir + .join(format!(".{name}.{}{TMP_EXT}", std::process::id())); + let result = std::fs::write(&scratch, contents) + .context("failed to write the stamp") + .and_then(|()| { + std::fs::rename(&scratch, &target).context("failed to move the stamp into place") + }); + if result.is_err() { + // Do not leave the scratch file behind; `prune` only sees what a + // later successful write lists. + let _ = std::fs::remove_file(&scratch); + } + result + } + + fn list(&self) -> Result> { + let mut entries = Vec::new(); + let read_dir = match std::fs::read_dir(&self.dir) { + Ok(read_dir) => read_dir, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(entries), + Err(error) => return Err(error).context("failed to list the stamps"), + }; + for entry in read_dir.flatten() { + let name = entry.file_name().to_string_lossy().into_owned(); + if !name.ends_with(STAMP_EXT) && !name.ends_with(TMP_EXT) { + continue; + } + let modified = entry + .metadata() + .and_then(|meta| meta.modified()) + .ok() + .and_then(|time| time.duration_since(UNIX_EPOCH).ok()) + .map_or(0, |since| since.as_secs()); + entries.push((name, modified)); + } + Ok(entries) + } + + fn remove(&self, name: &str) -> Result<()> { + match std::fs::remove_file(self.path(name)?) { + Ok(()) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error).context("failed to remove the stamp"), + } + } + + fn location(&self) -> String { + self.dir.display().to_string() + } +} + +/// A store that could not be set up: every read and write reports why, so the +/// step runs and says so. +pub struct UnavailableStore(pub String); + +impl Store for UnavailableStore { + fn read(&self, _name: &str) -> Result> { + bail!("{}", self.0) + } + fn write(&self, _name: &str, _contents: &str) -> Result<()> { + bail!("{}", self.0) + } + fn list(&self) -> Result> { + bail!("{}", self.0) + } + fn remove(&self, _name: &str) -> Result<()> { + bail!("{}", self.0) + } + fn location(&self) -> String { + "the stamp directory".to_string() + } +} + +/// Seconds since the epoch, or 0 when the clock is before it (which makes every +/// stamp look dated in the future, so none is honored). +pub fn now_secs() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |since| since.as_secs()) +} + +#[cfg(test)] +pub(crate) mod fakes { + use std::cell::RefCell; + use std::collections::BTreeMap; + + use anyhow::{Result, bail}; + + use super::{Probe, Store}; + + /// A probe whose answers a test sets, and which counts its calls. + pub struct FakeProbe { + pub tree: RefCell>, + pub toolchain: RefCell>, + pub env: RefCell, + } + + impl FakeProbe { + pub fn clean(tree: &str) -> Self { + Self { + tree: RefCell::new(Ok(tree.to_string())), + toolchain: RefCell::new(Ok("rustc 1.96.0\ncargo 1.96.0\n".to_string())), + env: RefCell::new("env-a".to_string()), + } + } + } + + impl Probe for FakeProbe { + fn clean_tree(&self) -> std::result::Result { + self.tree.borrow().clone() + } + fn toolchain(&self) -> std::result::Result { + self.toolchain.borrow().clone() + } + fn env_digest(&self) -> String { + self.env.borrow().clone() + } + } + + /// An in-memory store; `fail_writes` makes every write error. + #[derive(Default)] + pub struct MemStore { + pub files: RefCell>, + pub fail_writes: RefCell, + pub writes: RefCell, + } + + impl Store for MemStore { + fn read(&self, name: &str) -> Result> { + Ok(self.files.borrow().get(name).map(|(text, _)| text.clone())) + } + fn write(&self, name: &str, contents: &str) -> Result<()> { + if *self.fail_writes.borrow() { + bail!("read-only file system"); + } + *self.writes.borrow_mut() += 1; + // The mtime is the stamp's own `passed-at`, as on a real disk. + let modified = contents + .lines() + .find_map(|line| line.strip_prefix("passed-at: ")) + .and_then(|value| value.parse().ok()) + .unwrap_or(0); + self.files + .borrow_mut() + .insert(name.to_string(), (contents.to_string(), modified)); + Ok(()) + } + fn list(&self) -> Result> { + Ok(self + .files + .borrow() + .iter() + .map(|(name, (_, modified))| (name.clone(), *modified)) + .collect()) + } + fn remove(&self, name: &str) -> Result<()> { + self.files.borrow_mut().remove(name); + Ok(()) + } + fn location(&self) -> String { + "".to_string() + } + } +} + +#[cfg(test)] +mod tests { + use super::fakes::{FakeProbe, MemStore}; + use super::*; + + const NOW: u64 = 1_800_000_000; + + fn args(text: &str) -> Vec { + text.split_whitespace().map(String::from).collect() + } + + fn stamps<'a>(probe: &'a FakeProbe, store: &'a MemStore, now: u64) -> Stamps<'a> { + Stamps { + probe, + store, + now, + rerun: false, + } + } + + /// Record a pass of `test -p xtask` and return the stamp's file name. + fn record_pass(probe: &FakeProbe, store: &MemStore, now: u64) -> String { + let stamps = stamps(probe, store, now); + let Lookup::Miss { key: Some(key), .. } = stamps.lookup("test", &args("test -p xtask")) + else { + panic!("expected a miss with a key"); + }; + assert_eq!(stamps.record(&key), None); + key.file_name() + } + + // ---- the key ---- + + fn base_key() -> Key { + Key { + step: "test".to_string(), + args: "test -p xtask".to_string(), + tree: "tree".to_string(), + toolchain: "toolchain".to_string(), + env: "env".to_string(), + } + } + + #[test] + fn every_part_of_the_key_changes_the_stamp_name() { + let base = base_key(); + let variants = [ + Key { + step: "clippy".into(), + ..base.clone() + }, + Key { + args: "test --workspace".into(), + ..base.clone() + }, + Key { + tree: "other".into(), + ..base.clone() + }, + Key { + toolchain: "other".into(), + ..base.clone() + }, + Key { + env: "other".into(), + ..base.clone() + }, + ]; + for variant in &variants { + assert_ne!(variant.file_name(), base.file_name(), "{variant:?}"); + } + assert_eq!( + base.file_name(), + base_key().file_name(), + "the name is stable" + ); + let name = base.file_name(); + assert!( + name.starts_with("test-") && name.strip_suffix(".stamp").is_some(), + "{name}" + ); + } + + // ---- age: expiry and a clock that disagrees ---- + + #[test] + fn a_pass_is_honored_up_to_the_maximum_age_and_not_a_second_longer() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + record_pass(&probe, &store, NOW); + let at = |now| stamps(&probe, &store, now).lookup("test", &args("test -p xtask")); + assert!(matches!(at(NOW + MAX_AGE_SECS), Lookup::Hit { .. })); + // Expired: a quiet miss that is allowed to record a fresh pass. + match at(NOW + MAX_AGE_SECS + 1) { + Lookup::Miss { key, note } => { + assert!(key.is_some()); + assert_eq!(note, None); + } + Lookup::Hit { .. } => panic!("an expired pass must not skip the step"), + } + } + + #[test] + fn a_stamp_dated_in_the_future_is_ignored_and_says_so() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + let name = record_pass(&probe, &store, NOW + 10 * 60); + match stamps(&probe, &store, NOW).lookup("test", &args("test -p xtask")) { + Lookup::Miss { note, .. } => { + let note = note.expect("a note"); + assert!( + note.contains(&name) && note.contains("dated in the future"), + "{note}" + ); + } + Lookup::Hit { .. } => panic!("a future-dated stamp must not skip the step"), + } + // A little clock skew is tolerated. + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + record_pass(&probe, &store, NOW + 60); + assert!(matches!( + stamps(&probe, &store, NOW).lookup("test", &args("test -p xtask")), + Lookup::Hit { .. } + )); + } + + #[test] + fn a_stamp_from_another_format_or_with_a_bad_time_is_ignored() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + let name = record_pass(&probe, &store, NOW); + let good = store.files.borrow()[&name].0.clone(); + for (broken, why) in [ + ( + good.replace("pass stamp v1", "pass stamp v2"), + "does not match", + ), + (good.replace("tree: t", "tree: u"), "does not match"), + ( + good.replace(&format!("passed-at: {NOW}"), "passed-at: soon"), + "time stamp is unreadable", + ), + ( + good.replace(&format!("passed-at: {NOW}\n"), ""), + "time stamp is unreadable", + ), + ] { + store.files.borrow_mut().insert(name.clone(), (broken, NOW)); + match stamps(&probe, &store, NOW + 1).lookup("test", &args("test -p xtask")) { + Lookup::Miss { note, .. } => { + let note = note.expect("a note"); + assert!(note.contains(why), "{note}"); + } + Lookup::Hit { .. } => panic!("a broken stamp must not skip the step: {why}"), + } + } + } + + // ---- expiry and the size cap ---- + + #[test] + fn recording_prunes_expired_stamps_and_keeps_the_newest_up_to_the_cap() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + { + let mut files = store.files.borrow_mut(); + files.insert( + "old-a.stamp".into(), + (String::new(), NOW - MAX_AGE_SECS - 10), + ); + files.insert( + "old-b.stamp".into(), + (String::new(), NOW - MAX_AGE_SECS - 20), + ); + for index in 0..(MAX_STAMPS + 5) { + files.insert( + format!("fresh-{index:03}.stamp"), + (String::new(), NOW - 1000 + index as u64), + ); + } + } + record_pass(&probe, &store, NOW); + let files = store.files.borrow(); + assert_eq!( + files.len(), + MAX_STAMPS, + "{:?}", + files.keys().collect::>() + ); + assert!(!files.contains_key("old-a.stamp") && !files.contains_key("old-b.stamp")); + assert!( + !files.contains_key("fresh-000.stamp"), + "the oldest beyond the cap go first" + ); + assert!( + files.contains_key(&format!("fresh-{:03}.stamp", MAX_STAMPS + 4)), + "the newest stay" + ); + assert!( + files.keys().any(|name| name.starts_with("test-")), + "and so does the new pass" + ); + } + + #[test] + fn expired_stamps_are_pruned_even_when_the_store_is_far_below_the_cap() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + { + let mut files = store.files.borrow_mut(); + files.insert("old.stamp".into(), (String::new(), NOW - MAX_AGE_SECS - 1)); + files.insert("edge.stamp".into(), (String::new(), NOW - MAX_AGE_SECS)); + files.insert("fresh.stamp".into(), (String::new(), NOW - 60)); + } + record_pass(&probe, &store, NOW); + let files = store.files.borrow(); + assert!( + !files.contains_key("old.stamp"), + "{:?}", + files.keys().collect::>() + ); + assert!( + files.contains_key("edge.stamp"), + "a stamp exactly at the limit is still honored" + ); + assert!(files.contains_key("fresh.stamp")); + } + + // ---- the environment ---- + + fn vars(pairs: &[(&str, &str)]) -> Vec<(String, String)> { + pairs + .iter() + .map(|(k, v)| ((*k).to_string(), (*v).to_string())) + .collect() + } + + #[test] + fn the_environment_digest_follows_cargo_and_rust_settings_only() { + let base = env_digest(vars(&[("PATH", "/bin"), ("RUSTFLAGS", "-D warnings")])); + // Settings that can change a step's outcome change the digest ... + assert_ne!( + base, + env_digest(vars(&[("RUSTFLAGS", "-C target-cpu=native")])) + ); + assert_ne!( + base, + env_digest(vars(&[ + ("RUSTFLAGS", "-D warnings"), + ("CARGO_BUILD_TARGET", "x") + ])) + ); + // ... unrelated or per-run variables do not, whatever their order. + assert_eq!( + base, + env_digest(vars(&[ + ("HOME", "/home/a"), + ("CARGO_TARGET_DIR", "/elsewhere"), + ("CARGO_TERM_COLOR", "always"), + ("CARGO_PKG_NAME", "xtask"), + ("CARGO_MANIFEST_DIR", "/repo/xtask"), + ("RUST_BACKTRACE", "1"), + ("RUSTFLAGS", "-D warnings"), + ])) + ); + assert_eq!( + env_digest(vars(&[("RUSTFLAGS", "a"), ("CARGO_INCREMENTAL", "0")])), + env_digest(vars(&[("CARGO_INCREMENTAL", "0"), ("RUSTFLAGS", "a")])), + ); + } + + #[test] + fn no_environment_value_ends_up_in_a_stamp() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + *probe.env.borrow_mut() = env_digest(vars(&[("CARGO_REGISTRY_TOKEN", "hunter2-secret")])); + let name = record_pass(&probe, &store, NOW); + let text = store.files.borrow()[&name].0.clone(); + assert!( + !text.contains("hunter2") && !text.contains("CARGO_REGISTRY_TOKEN"), + "{text}" + ); + } + + #[test] + fn rerun_comes_from_the_flag_or_a_non_empty_non_zero_variable() { + let var = |value: Option<&'static str>| { + move |name: &str| { + (name == RERUN_VAR) + .then(|| value.map(String::from)) + .flatten() + } + }; + assert!(rerun_requested(true, var(None))); + assert!(rerun_requested(false, var(Some("1")))); + assert!(rerun_requested(false, var(Some("yes")))); + assert!(!rerun_requested(false, var(Some("0")))); + assert!(!rerun_requested(false, var(Some("")))); + assert!(!rerun_requested(false, var(Some(" ")))); + assert!(!rerun_requested(false, var(None))); + } + + #[test] + fn ages_read_as_minutes_hours_or_days() { + assert_eq!(age_text(5), "under a minute"); + assert_eq!(age_text(90), "1 min"); + assert_eq!(age_text(2 * 3600 + 5), "2 h"); + assert_eq!(age_text(3 * 86_400), "3 d"); + } + + // ---- the real store on a real disk ---- + + #[test] + fn the_file_store_round_trips_lists_only_stamps_and_removes_idempotently() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = FsStore { + dir: dir.path().join("passes"), + }; + assert_eq!( + store.read("a.stamp").expect("read"), + None, + "a missing stamp is not an error" + ); + assert!( + store.list().expect("list").is_empty(), + "nor is a missing directory" + ); + + store.write("a.stamp", "one").expect("write"); + store + .write("a.stamp", "two") + .expect("an overwrite replaces it"); + assert_eq!(store.read("a.stamp").expect("read").as_deref(), Some("two")); + std::fs::write(dir.path().join("passes/notes.txt"), "x").expect("a foreign file"); + + let listed = store.list().expect("list"); + assert_eq!(listed.len(), 1, "{listed:?}"); + assert_eq!(listed[0].0, "a.stamp"); + assert!( + listed[0].1 > 1_000_000_000, + "the modified time is read: {listed:?}" + ); + let leftovers = std::fs::read_dir(dir.path().join("passes")) + .expect("dir") + .count(); + assert_eq!( + leftovers, 2, + "no scratch file is left behind (the stamp and the foreign file)" + ); + + store.remove("a.stamp").expect("remove"); + store.remove("a.stamp").expect("removing it again is fine"); + assert_eq!(store.read("a.stamp").expect("read"), None); + } + + #[test] + fn the_file_store_refuses_names_that_leave_its_directory() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = FsStore { + dir: dir.path().join("passes"), + }; + for name in ["../escape.stamp", "a/b.stamp", "a\\b.stamp", "..", ".", ""] { + // The guard itself refuses, not whatever the filesystem then does. + let refusals = [ + store.write(name, "x").unwrap_err(), + store.read(name).unwrap_err(), + store.remove(name).unwrap_err(), + ]; + for error in refusals { + assert!( + format!("{error:#}").contains("refusing the stamp name"), + "{name}: {error:#}" + ); + } + } + assert!(!dir.path().join("escape.stamp").exists()); + } + + // ---- git: the tree hash, the dirty check and the shared location ---- + + fn git_in(dir: &Path, args: &[&str]) -> String { + let output = git_command(Some(dir)) + .args([ + "-c", + "user.name=t", + "-c", + "user.email=t@example.com", + "-c", + "commit.gpgsign=false", + ]) + .args(args) + .output() + .expect("run git"); + assert!( + output.status.success(), + "git {args:?} failed in {}", + dir.display() + ); + String::from_utf8_lossy(&output.stdout).trim().to_string() + } + + /// A repository with one commit, in a fresh directory. + fn repo() -> tempfile::TempDir { + let dir = tempfile::tempdir().expect("tempdir"); + git_in(dir.path(), &["init", "-q"]); + std::fs::write(dir.path().join("a.txt"), "one\n").expect("write"); + git_in(dir.path(), &["add", "a.txt"]); + git_in(dir.path(), &["commit", "-q", "-m", "first"]); + dir + } + + fn probe_for(dir: &Path) -> SystemProbe { + SystemProbe { + dir: Some(dir.to_path_buf()), + } + } + + #[test] + fn a_clean_tree_is_keyed_by_its_tree_hash_which_an_amend_does_not_change() { + let repo = repo(); + let probe = probe_for(repo.path()); + let tree = probe.clean_tree().expect("a clean tree"); + assert_eq!(tree, git_in(repo.path(), &["rev-parse", HEAD_TREE])); + + let commit = git_in(repo.path(), &["rev-parse", "HEAD"]); + git_in(repo.path(), &["commit", "-q", "--amend", "-m", "reworded"]); + assert_ne!( + git_in(repo.path(), &["rev-parse", "HEAD"]), + commit, + "the commit hash changed" + ); + assert_eq!( + probe.clean_tree().expect("still clean"), + tree, + "the tree hash did not" + ); + + std::fs::write(repo.path().join("a.txt"), "two\n").expect("write"); + git_in(repo.path(), &["commit", "-q", "-am", "second"]); + assert_ne!( + probe.clean_tree().expect("clean"), + tree, + "new content is a new tree" + ); + } + + #[test] + fn uncommitted_or_untracked_files_make_the_tree_unusable_for_stamps() { + let repo = repo(); + let probe = probe_for(repo.path()); + probe.clean_tree().expect("clean to start with"); + + std::fs::write(repo.path().join("a.txt"), "changed\n").expect("write"); + let why = probe.clean_tree().expect_err("a modified tracked file"); + assert!(why.contains("1 uncommitted or untracked path"), "{why}"); + assert!( + why.contains("git status"), + "the reason says how to fix it: {why}" + ); + git_in(repo.path(), &["checkout", "--", "a.txt"]); + probe.clean_tree().expect("clean again"); + + // An untracked file can hide a missing `git add`, so it counts too. + std::fs::write(repo.path().join("new.rs"), "mod x;\n").expect("write"); + assert!(probe.clean_tree().is_err()); + } + + #[test] + fn linked_worktrees_share_one_stamp_directory_outside_every_work_tree() { + let repo = repo(); + let linked = tempfile::tempdir().expect("tempdir"); + let linked_path = linked.path().join("wt"); + git_in( + repo.path(), + &[ + "worktree", + "add", + "-q", + linked_path.to_str().expect("utf-8"), + "-b", + "side", + ], + ); + + let main = FsStore::in_git_common_dir(Some(repo.path())).expect("store"); + let side = FsStore::in_git_common_dir(Some(&linked_path)).expect("store"); + let canonical = |store: &FsStore| { + std::fs::canonicalize(store.dir.parent().expect("parent")).expect("exists") + }; + assert_eq!( + canonical(&main), + canonical(&side), + "one directory for the whole clone" + ); + assert!(main.dir.ends_with(STAMP_DIR)); + assert!(canonical(&main).ends_with(".git"), "{:?}", canonical(&main)); + + // A stamp written from one worktree is read from the other, and writing + // it does not dirty either work tree. + main.write("t.stamp", "shared").expect("write"); + assert_eq!( + side.read("t.stamp").expect("read").as_deref(), + Some("shared") + ); + probe_for(repo.path()) + .clean_tree() + .expect("the main work tree is still clean"); + probe_for(&linked_path) + .clean_tree() + .expect("so is the linked one"); + } + + #[test] + fn a_directory_git_does_not_know_has_no_store_and_no_usable_tree() { + let dir = tempfile::tempdir().expect("tempdir"); + // The temp directory is not normally inside a repository; if it were, + // git would answer and there would be nothing to assert. + if git_stdout(Some(dir.path()), &["rev-parse", "--git-dir"]).is_err() { + assert!(probe_for(dir.path()).clean_tree().is_err()); + assert!(FsStore::in_git_common_dir(Some(dir.path())).is_err()); + } + } + + #[test] + fn the_toolchain_is_identified_from_rustc_and_cargo() { + let text = SystemProbe::default() + .toolchain() + .expect("rustc and cargo run in the test environment"); + assert!(text.contains("rustc") && text.contains("cargo"), "{text}"); + } + + // ---- hermetic git: a hostile ambient GIT_DIR must not redirect any of it ---- + + const HOSTILE_CHILD_VAR: &str = "ROCM_STAMP_TEST_HOSTILE_CHILD"; + + #[test] + fn an_explicit_directory_drops_the_repository_selecting_environment() { + let with_dir = git_command(Some(Path::new("scratch"))); + let envs: Vec<_> = with_dir.get_envs().collect(); + for var in [ + "GIT_DIR", + "GIT_WORK_TREE", + "GIT_INDEX_FILE", + "GIT_COMMON_DIR", + ] { + assert!( + envs.iter() + .any(|(name, value)| name.to_str() == Some(var) && value.is_none()), + "{var} must be removed for a command pointed at a directory" + ); + } + // Without a directory the hook's own environment is kept: there GIT_DIR + // correctly names the repository being pushed. + assert_eq!(git_command(None).get_envs().count(), 0); + } + + /// Does nothing on its own. The test below re-runs this binary with a hostile + /// git environment and this variable set, and this body then drives the real + /// probe, store and test helpers against a scratch repository. + #[test] + fn hostile_environment_child() { + let Some(dir) = std::env::var_os(HOSTILE_CHILD_VAR) else { + return; + }; + let dir = PathBuf::from(dir); + assert!( + std::env::var_os("GIT_DIR").is_some(), + "the child is supposed to run under a hostile GIT_DIR" + ); + let probe = probe_for(&dir); + let tree = git_in(&dir, &["rev-parse", HEAD_TREE]); + assert_eq!( + probe.clean_tree().expect("the scratch repository's tree"), + tree + ); + + // Committing through the helper lands in the scratch repository. + std::fs::write(dir.join("b.txt"), "two\n").expect("write"); + git_in(&dir, &["add", "b.txt"]); + git_in(&dir, &["commit", "-q", "-m", "second"]); + assert_ne!(probe.clean_tree().expect("clean"), tree); + + let store = FsStore::in_git_common_dir(Some(&dir)).expect("store"); + let scratch = std::fs::canonicalize(&dir).expect("canonical"); + let stamp_parent = + std::fs::canonicalize(store.dir.parent().expect("parent")).expect("exists"); + assert!( + stamp_parent.starts_with(&scratch), + "the store must be in the scratch repository, not the outer one: {stamp_parent:?}" + ); + println!("hostile-child-ran"); + } + + #[test] + fn git_driven_behaviour_ignores_a_hostile_git_environment() { + let outer = repo(); + let inner = repo(); + let outer_git = outer.path().join(".git"); + let outer_head = git_in(outer.path(), &["rev-parse", "HEAD"]); + + let output = Command::new(std::env::current_exe().expect("this test binary")) + .args([ + "--exact", + "prepush_stamp::tests::hostile_environment_child", + "--nocapture", + "--test-threads=1", + ]) + .env(HOSTILE_CHILD_VAR, inner.path()) + .env("GIT_DIR", &outer_git) + .env("GIT_WORK_TREE", outer.path()) + .env("GIT_INDEX_FILE", outer_git.join("index")) + .env("GIT_COMMON_DIR", &outer_git) + .output() + .expect("re-run the test binary"); + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(output.status.success(), "{stdout}\n{stderr}"); + assert!( + stdout.contains("hostile-child-ran"), + "the child did not run its body: {stdout}" + ); + + // The outer repository is exactly as it was: no stray commit, nothing staged. + assert_eq!(git_in(outer.path(), &["rev-parse", "HEAD"]), outer_head); + assert_eq!(git_in(outer.path(), &["rev-list", "--count", "HEAD"]), "1"); + assert_eq!(git_in(outer.path(), &["status", "--porcelain"]), ""); + } + + // ---- non-Unicode environment variables must never abort the optional stamp ---- + + #[cfg(unix)] + fn non_unicode(prefix: &str, tail: &[u8]) -> std::ffi::OsString { + use std::os::unix::ffi::OsStringExt; + let mut bytes = prefix.as_bytes().to_vec(); + bytes.extend_from_slice(tail); + std::ffi::OsString::from_vec(bytes) + } + + /// Unicode values hash to the same digest they always did, so stamps recorded + /// before the digest read the environment as bytes stay valid. + #[test] + fn the_environment_digest_of_unicode_values_is_the_digest_of_the_text() { + assert_eq!( + env_digest(vars(&[("RUSTFLAGS", "-D warnings"), ("PATH", "/bin")])), + hex_digest("RUSTFLAGS=-D warnings\n") + ); + } + + #[cfg(unix)] + #[test] + fn an_unrelated_variable_with_non_unicode_bytes_is_ignored_and_does_not_panic() { + let base = env_digest(vars(&[("RUSTFLAGS", "-D warnings")])); + let with_noise = env_digest([ + ( + std::ffi::OsString::from("RUSTFLAGS"), + std::ffi::OsString::from("-D warnings"), + ), + ( + std::ffi::OsString::from("PATH"), + non_unicode("/bin", &[0xff]), + ), + (non_unicode("HOME_", &[0xfe]), std::ffi::OsString::from("x")), + ( + std::ffi::OsString::from("LANG"), + non_unicode("", &[0x80, 0x81]), + ), + ]); + assert_eq!( + with_noise, base, + "unrelated variables never reach the digest" + ); + } + + #[cfg(unix)] + #[test] + fn a_relevant_variable_with_non_unicode_bytes_is_hashed_as_the_bytes_it_is() { + let digest = |value: &[u8]| { + env_digest([( + std::ffi::OsString::from("RUSTFLAGS"), + non_unicode("", value), + )]) + }; + let absent = env_digest(Vec::<(String, String)>::new()); + assert_ne!(digest(&[0xff]), absent, "the variable counts"); + // A lossy conversion would turn both bytes into U+FFFD and collapse them. + assert_ne!( + digest(&[0xff]), + digest(&[0xfe]), + "the bytes are hashed, not a lossy text" + ); + assert_eq!(digest(&[0xff]), digest(&[0xff]), "and the digest is stable"); + + // The same holds when the invalid bytes are in the name, after a prefix. + let named = |tail: &[u8]| { + env_digest([(non_unicode("CARGO_X", tail), std::ffi::OsString::from("1"))]) + }; + assert_ne!(named(&[0xff]), absent); + assert_ne!(named(&[0xff]), named(&[0xfe])); + } + + const NON_UNICODE_CHILD_VAR: &str = "ROCM_STAMP_TEST_NON_UNICODE_CHILD"; + + /// Does nothing on its own. The test below re-runs this binary with + /// non-Unicode bytes in unrelated and in relevant variables, and this body + /// then drives the real probe: `std::env::vars()` would panic here. + #[test] + fn non_unicode_environment_child() { + if std::env::var_os(NON_UNICODE_CHILD_VAR).is_none() { + return; + } + let probe = SystemProbe::default(); + let first = Probe::env_digest(&probe); + assert_eq!(first, Probe::env_digest(&probe), "the digest is stable"); + assert_eq!(first.len(), 64, "a SHA-256 digest: {first}"); + println!("non-unicode-child-ran {first}"); + } + + #[cfg(unix)] + #[test] + fn the_real_probe_computes_its_digest_under_non_unicode_environment_variables() { + let run = |rustflags: &[u8]| { + let output = Command::new(std::env::current_exe().expect("this test binary")) + .args([ + "--exact", + "prepush_stamp::tests::non_unicode_environment_child", + "--nocapture", + "--test-threads=1", + ]) + .env(NON_UNICODE_CHILD_VAR, "1") + .env("ROCM_STAMP_TEST_UNRELATED", non_unicode("v", &[0xff, 0xfe])) + .env(non_unicode("UNRELATED_", &[0xff]), "x") + .env("RUSTFLAGS", non_unicode("", rustflags)) + .output() + .expect("re-run the test binary"); + let stdout = String::from_utf8_lossy(&output.stdout).into_owned(); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(output.status.success(), "{stdout}\n{stderr}"); + stdout + .split_once("non-unicode-child-ran ") + .and_then(|(_, rest)| rest.split_whitespace().next()) + .unwrap_or_else(|| panic!("the child did not run its body: {stdout}")) + .to_string() + }; + let one = run(&[0xff]); + // A relevant non-Unicode value changes the digest, losslessly. + assert_ne!(one, run(&[0xfe])); + assert_eq!(one, run(&[0xff])); + } + + // ---- scratch files left by a write that died ---- + + #[test] + fn a_write_that_cannot_be_moved_into_place_leaves_no_scratch_file() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = FsStore { + dir: dir.path().join("passes"), + }; + // A non-empty directory where the stamp should go makes the rename fail. + std::fs::create_dir_all(store.dir.join("a.stamp")).expect("mkdir"); + std::fs::write(store.dir.join("a.stamp").join("inner"), "x").expect("write"); + + let error = store + .write("a.stamp", "x") + .expect_err("a directory is in the way"); + assert!( + format!("{error:#}").contains("failed to move the stamp into place"), + "{error:#}" + ); + let names: Vec = std::fs::read_dir(&store.dir) + .expect("dir") + .flatten() + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .collect(); + assert_eq!( + names, + ["a.stamp"], + "no scratch file is left behind: {names:?}" + ); + } + + #[test] + fn stale_scratch_files_are_pruned_and_fresh_ones_kept_and_none_count_toward_the_cap() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + { + let mut files = store.files.borrow_mut(); + files.insert( + ".a.stamp.1.tmp".into(), + (String::new(), NOW - TMP_MAX_AGE_SECS - 1), + ); + files.insert( + ".b.stamp.2.tmp".into(), + (String::new(), NOW - TMP_MAX_AGE_SECS), + ); + files.insert(".c.stamp.3.tmp".into(), (String::new(), NOW - 60)); + for index in 0..MAX_STAMPS { + files.insert( + format!("fresh-{index:03}.stamp"), + (String::new(), NOW - 1000 + index as u64), + ); + } + } + record_pass(&probe, &store, NOW); + let files = store.files.borrow(); + let names: Vec<&String> = files.keys().collect(); + assert!( + !files.contains_key(".a.stamp.1.tmp"), + "a dead writer's scratch file goes: {names:?}" + ); + assert!( + files.contains_key(".b.stamp.2.tmp"), + "one exactly at the bound stays" + ); + assert!( + files.contains_key(".c.stamp.3.tmp"), + "a young one may be a live writer's" + ); + let stamps = files + .keys() + .filter(|name| name.ends_with(STAMP_EXT)) + .count(); + assert_eq!( + stamps, MAX_STAMPS, + "scratch files do not push stamps out: {names:?}" + ); + } + + #[test] + fn a_real_store_prunes_a_dead_writers_scratch_file() { + let dir = tempfile::tempdir().expect("tempdir"); + let store = FsStore { + dir: dir.path().join("passes"), + }; + std::fs::create_dir_all(&store.dir).expect("mkdir"); + let now = now_secs(); + let age_of = |name: &str, secs: u64| { + let path = store.dir.join(name); + std::fs::write(&path, "x").expect("write"); + std::fs::File::options() + .write(true) + .open(&path) + .expect("open") + .set_modified(SystemTime::now() - std::time::Duration::from_secs(secs)) + .expect("set the modified time"); + path + }; + let dead = age_of(".a.stamp.1.tmp", 2 * 60 * 60); + let live = age_of(".b.stamp.2.tmp", 60); + + let probe = FakeProbe::clean("t"); + let stamps = Stamps::new(&probe, &store, now, false, |_| None); + let Lookup::Miss { key: Some(key), .. } = stamps.lookup("test", &args("test -p xtask")) + else { + panic!("expected a miss with a key"); + }; + assert_eq!(stamps.record(&key), None); + assert!( + !dead.exists(), + "the dead writer's scratch file is removed from disk" + ); + assert!(live.exists(), "the young one is left alone"); + } + + // ---- Stamps::new: the --rerun flag and the variable reach the lookup ---- + + #[test] + fn stamps_new_reads_the_flag_and_the_variable() { + let (probe, store) = (FakeProbe::clean("t"), MemStore::default()); + let var = + |value: &'static str| move |name: &str| (name == RERUN_VAR).then(|| value.to_string()); + assert!(Stamps::new(&probe, &store, NOW, true, |_| None).rerun); + assert!(Stamps::new(&probe, &store, NOW, false, var("1")).rerun); + assert!(!Stamps::new(&probe, &store, NOW, false, var("0")).rerun); + assert!(!Stamps::new(&probe, &store, NOW, false, |_| None).rerun); + } +} From 22937c37b5b617aeed604bfcc1b339361168bb2c Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Fri, 6 Nov 2026 17:34:37 +0000 Subject: [PATCH 004/102] refactor(rocm): split chat.rs into summaries, MCP bridge, intent and turn modules, and fold in four related refactors (#936) * test(rocm): pin the serve safety rules of the chat tool policy and the freeform planner A model-proposed `rocm serve` is refused for four reasons: a public bind, a CPU device, a non-loopback host, and serving that is not managed. The chat tool policy and the provider-assisted planner each enforce them in their own code today, and nothing ties the two together. Add one table of refused and accepted `serve` argvs in test_support and run it against both entry points. For each refusal it asserts the exact message and that nothing is proposed or planned (no ChatRocmCommandAction from chat, no StructuredRequestPlan from the planner); for each accepted argv it asserts the argv comes through unchanged and is held for approval. Also pinned: the order the rules fire in, and the places the two surfaces differ today (the actor named in the message, the CPU message's second half, the case of the serve word, and the --foreground/--managed pair that only the chat check ever sees because clap refuses it first). Tests only; they pass on the unmodified code. Signed-off-by: Jussi Elo * refactor(rocm): make the freeform planner call the shared serve safety check The provider-assisted planner re-implemented the four rules a model-proposed `rocm serve` must satisfy (no public bind, no CPU device, a loopback host, managed serving without --foreground), which the chat tool policy already enforced in validate_chat_rocm_command_safety. The planner's `install sdk` arm already called the shared function, so the two surfaces kept two copies of one policy that could drift apart. Move the four rules into tool_policy::validate_serve_command_safety, taking the name of the surface that proposed the call. The chat check calls it with "local assistant" and the planner's serve arm with "provider planner", so each refusal still says who was refused while the rules, their order and their wording live in one place. The planner's copy is deleted, and it still does not import chat. One user-visible difference: the planner's CPU refusal now ends "ROCm GPU execution is required", as the chat one does, where it used to end "rocm-cli requires ROCm GPU execution". Every other refusal reads as before. The serve word is still matched exactly by the planner (clap refuses `Serve` before the safety check runs) and case-insensitively by the chat check, unchanged. Tests: the shared table of unsafe and safe argvs from the previous commit now holds the unified wording, and deleting any one rule from the shared check fails both the chat and the planner suites. Four mock-lane scenarios (chat-13 to chat-16) drive a provider plan that breaks one rule each through the real binary and check that the CLI falls back to its own plan, runs nothing, and says why. docs/architecture.md, docs/llm-tool-use.md and docs/testing.md describe the shared check. Signed-off-by: Jussi Elo * refactor(rocm): make the model registry adapter note helper private again model_registry_adapter_availability_note was widened to pub(crate) when the rocm model catalog renderer moved out of examine_report_text.rs, only because one test that stayed behind called it. That test, linux_only_engine_runtime_status_is_explicit_on_native_windows, checked two things: the runtime: label from engine_runtime_status_label and the note the verbose model listing prints. Split it along that line. The label assertion keeps its name and stays in examine_report_text.rs. The note assertion moves, unchanged and with the same cfg!(windows) branches, to model_catalog_text.rs as vllm_adapter_availability_note_is_explicit_on_native_windows, next to the renderer that prints the note. The helper goes back to private, and the module docs and docs/architecture.md describe the new test location. Tests and visibility only; no behaviour change. Signed-off-by: Jussi Elo * refactor(rocm): move the background-helper note out of service_launch.rs into background_helper.rs services_text.rs imported background_helper_note from service_launch.rs while service_launch.rs imports the stop-result renderers from services_text.rs, so the two modules depended on each other. The note (HelperNote, helper_note_for, background_helper_note) and the liveness check it reads (background_helper_already_running) move verbatim into a new leaf module that imports nothing from the service modules. services_text.rs and uninstall_plan.rs read the note from there, and service_launch.rs reads background_helper_already_running from there for its autostart decision; spawning the helper stays in service_launch.rs. Pure move: no behaviour change. The tests that cover the moved items move with them and keep their names; the runtime_state and set_helper_facts fixtures become pub(crate) because the autostart and stop-output tests that stay in service_launch.rs use them too. docs/architecture.md gains the new module's entry and drops the claim that services_text.rs still reaches into service_launch.rs. Signed-off-by: Jussi Elo * refactor(rocm): hand run_attached_service the Ctrl-C stop instead of importing service_control service_launch.rs imported run_internal_sandbox_tool from service_control.rs only to stop the server when the user presses Ctrl-C in an attached serve, while service_control.rs imports the spawn helpers and the endpoint-key guard from service_launch.rs, so the two modules depended on each other. run_internal_sandbox_tool cannot move down, because it calls the stop and restart paths that live in service_control.rs, so the dependency is cut at its single call site instead: run_attached_service takes the stop as a parameter and serve_cmd.rs, which already sits above both modules, passes the same run_internal_sandbox_tool(StopServer, service id, native fallback allowed) call. The stop still runs at the same point, with the same arguments, and its answer goes to finish_attached_stop unchanged, so behaviour is the same. docs/architecture.md and the service_control.rs header now describe the one-way edge. Signed-off-by: Jussi Elo * refactor(rocm): move the chat assistant's deterministic tool summaries into chat/summaries.rs The assistant's deterministic summaries of read-only tool output (examine, model recipes, ComfyUI status, services, port status, engine inventory), the row types and parsers behind them, chat_tool_value and managed_service_running_state move out of chat.rs into a new leaf child module, chat/summaries.rs. chat.rs stays a file and gains a chat/ directory, the same layout providers.rs and comfyui.rs use, so the include_str! of the assistant skill and every path that names chat.rs are unchanged. A pure move: no behaviour change. The tests that exercise only these items move with them and keep their names; the tests that drive a summary through the turn loop's follow-up decision stay in chat.rs. Visibility widens only as far as chat.rs needs (pub(super)), and chat re-exports deterministic_comfyui_tool_summary under cfg(test) for comfyui.rs's tests. docs/architecture.md and the module doc comments are updated to match. Signed-off-by: Jussi Elo * refactor(rocm): move the comfyui dispatch function out of main.rs The `comfyui` dispatch function (the match over `ComfyuiCommand`, about 160 lines) moves from `apps/rocm/src/main.rs` into `apps/rocm/src/comfyui.rs` as `comfyui::run`, next to the code it calls, the way `remote::run` and `bootstrap::run` are named. It is a pure move: the body is unchanged apart from dropping the `comfyui::` prefix on calls that are now local, and the module reaches `record_cli_audit_event` through `crate::cli_audit` like its sibling command modules, so no new module cycle appears. The one arm of `dispatch()` now calls `comfyui::run(command)`. The clap command tree already lives in `cli.rs`, so the convention text that said the command enum and its dispatch function usually stay in `main.rs` (with `comfyui()` as the example) is updated in the same change: docs/architecture.md (module-organization convention, the `main.rs` entry and the `comfyui.rs` entry) and AGENTS.md section 6. Signed-off-by: Jussi Elo * fix(rocm): gate the chat summary re-export for comfyui's tests to unix The re-export of deterministic_comfyui_tool_summary in chat.rs was #[cfg(test)] only, but its sole readers outside summaries.rs are three comfyui.rs tests that are all #[cfg(unix)]. On Windows the test build therefore had an unused import, which -D warnings turns into a compile error ("unused import: self::summaries::deterministic_comfyui_tool_summary"). It did not show on Linux, and before the move the function was defined in chat.rs itself, so there was no import to be unused. Gate the re-export with #[cfg(all(test, unix))] to match its readers. No behaviour change. Signed-off-by: Jussi Elo * refactor(rocm): move the chat assistant's MCP bridge into chat/mcp_bridge.rs The MCP tool dispatch (run_internal_mcp_call), the in-process runner for the read-only rocm commands the assistant may ask for, the helpers that wrap a command's output as an MCP tool result, the path_exists and port_status tools, the services tool's result text, the install sdk dry-run helpers and the logs flag parsers were mixed into chat.rs with the turn loop. They move to a new leaf child, chat/mcp_bridge.rs, declared from chat.rs (chat.rs stays a file beside its chat/ directory, like summaries.rs). Pure move: 17 functions and the 18 tests that exercise only them, with their original names. run_internal_mcp_call stays reachable as chat::run_internal_mcp_call (main.rs, dash_seam.rs) through a re-export; run_chat_read_only_tool, run_chat_path_exists_tool and loopback_host_key are pub(super) because the turn loop and the command-preview property tests in chat.rs still call them. The re-exports of ChatRocmCommandAction and rocm_chat_tool_requested_args become cfg(test), since chat.rs no longer calls them in production code. The advised_commands.rs path pin for the "rocm command requires at least one argument" error follows its string, and the module docs and docs/architecture.md entries that named chat.rs as the caller of the moved items now name chat/mcp_bridge.rs. Signed-off-by: Jussi Elo * refactor(rocm): move the chat assistant's prompt-intent heuristics into chat/intent.rs The functions that decide from a user's latest message what it asks for (fallback_rocm_tool_call_for_prompt and the config, engine-list and services-list fallbacks behind it, the prompt_* phrase predicates, and the rocm install sdk approval request built from an install or setup request) were mixed into chat.rs with the turn loop. They move to a new leaf child, chat/intent.rs, declared from chat.rs. Pure move: 13 functions and the 16 tests that call them directly, with their original names. latest_user_chat_message moves with them because three of them call it; leaving it with the turn loop would have made the two modules call each other. Ten functions are pub(super) for the turn loop that stays in chat.rs; the other three are private. The advised_commands.rs path pins for "rocm install folder", "rocm installed at", "rocm installed" and "rocm please" follow their strings, and the prompt_text.rs doc comment and the docs/architecture.md entry that named chat.rs as the heuristics' home now name chat/intent.rs. Signed-off-by: Jussi Elo * refactor(rocm): move the chat turn loop and transcript into chat/turn.rs What remained in chat.rs after the summaries, MCP bridge and prompt-intent moves was the turn loop: the entry point that takes a prompt and renders the assistant's reply, the system prompt, the transcript helpers and most of the module's tests. It moves to a new child, chat/turn.rs, declared from chat.rs. chat.rs keeps the entry points other modules call (read_piped_prompt, render_chat_text, render_chat_prompt_text), the sampling-flag value parsers, the re-exports and the tests that cross the seams. Pure move: 33 production items, the 28 tests that exercise them and their one shared helper, with their original names. The assistant skill is now included from one directory deeper, so the include_str! path gains a ../ and still resolves to skills/rocm-cli-assistant/SKILL.md. render_chat_prompt_result_with_progress, ChatPromptResult and mcp_tool_result_text are pub(super); rocm_chat_tool_system_prompt_for_host stays pub(crate) and is re-exported for dash.rs. The command-preview property tests, which drive the tool-call guards across the policy, the bridge and the turn loop, stay in chat.rs and import the two helpers they use from the siblings. The advised_commands.rs path pins for "rocm tools: {}" and "rocm command" follow their strings to chat/turn.rs. AGENTS.md and the skill-evals comment in ci.yml said main.rs embeds the assistant skill; they now name chat/turn.rs. The docs/architecture.md chat.rs entry is rewritten for the file-plus-children layout, a turn.rs entry is added, and the module docs that named chat.rs as the caller of items now in a child name the child. Signed-off-by: Jussi Elo * refactor(rocm): move mcp_tool_result_text into chat/mcp_bridge.rs so the bridge has no edge back to the turn loop mcp_tool_result_text only reads the text out of an MCP tool result, the shape that chat/mcp_bridge.rs builds, and the turn loop is one of its callers. It sat in chat/turn.rs, so the bridge's own tests imported it from there: a test-only edge from the bridge back up to the turn loop. Production stayed one-way, but the "nothing below the turn loop calls back" statement in the module docs held only for non-test code. Move the function (body unchanged) and its two tests, with their names, to chat/mcp_bridge.rs. chat/turn.rs and the command-preview property tests in chat.rs now import it from there; the turn -> mcp_bridge edge already existed. The function stays pub(super) because the turn loop calls it. No behavior change. docs/architecture.md and the module doc comments are updated to match. Signed-off-by: Jussi Elo * refactor(rocm): move the background-helper test fixtures to test_support and make background_helper's tests module private again background_helper.rs declared `pub(crate) mod tests` so the tests in service_launch.rs could reuse its runtime_state and set_helper_facts fixtures. That widened the module's visibility only for tests. Move the two fixtures (bodies unchanged) to the crate-level test_support module, where the other fixtures shared across modules live, and have both test modules import them from there. background_helper's `mod tests` is private again. No behavior change and no test renamed; docs/architecture.md is updated to match. Signed-off-by: Jussi Elo * test(e2e): assert the refused planner plan is the only output, not just that no execution heading appears The step behind the planner's refused-serve scenarios (the CPU, public bind, non-local host and foreground cases) checked that nothing ran only by asserting the output lacks an "execution" heading. That passes vacuously if the heading ever changes shape. Add the positive statement: the plan renderer indents everything under its single `request plan` heading, so the output's only unindented lines must be that heading, and anything a run printed of its own would break it. Checked against a debug build with the e2e hooks: with the planner's serve arm refusal removed all four scenarios fail, and with a differently named unindented line appended to the plan the new assertion fails all four where the old heading check would not. Signed-off-by: Jussi Elo * refactor(rocm): drop a needless borrow in the attached Ctrl-C stop call After merging main, run_attached_service's `paths` is already a reference at the point where the Ctrl-C stop is handed to `stop_service`, so the extra `&` trips clippy's needless_borrow. Pass it directly. No behavior change. Signed-off-by: Jussi Elo * test(rocm): model the chat-only serve port rule in the shared serve safety fixtures The chat serve policy requires an explicit --port on any host other than 127.0.0.1, the same rule `rocm serve` applies; the provider-assisted planner's check has no port rule. The shared SERVE_SAFETY_ACCEPTED table still listed `--host=localhost` and `--host ::1` serves without a port as accepted by both surfaces, so the chat suite's accepted-case test failed. Keep the shared accepted table to argvs both surfaces accept (the two loopback-host entries now carry --port), and add SERVE_PORT_RULE_CHAT_ONLY for the argvs the planner accepts but chat refuses. One chat test asserts the exact refusal through all three entry points and that no approval action is built; one planner test asserts the planner plans them unchanged. No production code changes. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 1ab8f7f83f6cc3e083d8823d89fdfc1a49b99b96) Signed-off-by: juhovainio --- .github/workflows/ci.yml | 2 +- AGENTS.md | 4 +- apps/rocm/src/advised_commands.rs | 14 +- apps/rocm/src/automations.rs | 4 +- apps/rocm/src/background_helper.rs | 198 + apps/rocm/src/bridge_snapshot.rs | 2 +- apps/rocm/src/chat.rs | 4540 +---------------- apps/rocm/src/chat/intent.rs | 949 ++++ apps/rocm/src/chat/mcp_bridge.rs | 1305 +++++ apps/rocm/src/chat/summaries.rs | 804 +++ apps/rocm/src/chat/turn.rs | 1547 ++++++ apps/rocm/src/comfyui.rs | 165 + apps/rocm/src/config_cmd.rs | 2 +- apps/rocm/src/engine_runtime_select.rs | 7 +- apps/rocm/src/examine_report_text.rs | 20 +- apps/rocm/src/freeform_planner.rs | 228 +- apps/rocm/src/install_cmd.rs | 4 +- apps/rocm/src/main.rs | 181 +- apps/rocm/src/managed_service_status.rs | 5 +- apps/rocm/src/model_catalog_text.rs | 25 +- apps/rocm/src/prompt_text.rs | 2 +- apps/rocm/src/providers.rs | 2 +- apps/rocm/src/serve_cmd.rs | 13 +- apps/rocm/src/service_control.rs | 10 +- apps/rocm/src/service_launch.rs | 219 +- apps/rocm/src/services_text.rs | 6 +- apps/rocm/src/setup_cmd.rs | 2 +- apps/rocm/src/test_support.rs | 187 +- apps/rocm/src/tool_policy.rs | 236 +- apps/rocm/src/uninstall_plan.rs | 4 +- apps/rocm/src/update_cmd.rs | 6 +- docs/architecture.md | 49 +- docs/llm-tool-use.md | 8 + docs/testing.md | 18 + tests/e2e-cucumber/features/chat.feature | 35 + tests/e2e-cucumber/tests/e2e/planner_steps.rs | 84 + 36 files changed, 5905 insertions(+), 4982 deletions(-) create mode 100644 apps/rocm/src/background_helper.rs create mode 100644 apps/rocm/src/chat/intent.rs create mode 100644 apps/rocm/src/chat/mcp_bridge.rs create mode 100644 apps/rocm/src/chat/summaries.rs create mode 100644 apps/rocm/src/chat/turn.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 77ad26c9d2..4c6e43e5c8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1632,7 +1632,7 @@ jobs: # fires, so this job is not a substitute for them. # # skills/rocm-cli-assistant/ is deliberately out of scope. Despite its path it - # is not a published skill: apps/rocm/src/main.rs embeds it verbatim into the + # is not a published skill: apps/rocm/src/chat/turn.rs embeds it verbatim into the # chat assistant's system prompt with include_str!, so giving it the YAML # frontmatter this check requires would inject that frontmatter into a live # prompt. diff --git a/AGENTS.md b/AGENTS.md index e9db92e2f2..c02e9f866e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -274,7 +274,7 @@ dash-TUI, in `crates/rocm-dash-tui/src/ui/`: Guardrails: -- new subsystems/subcommands: default their domain implementation to its own file from day one (full domain extraction, e.g. `therock.rs`/`comfyui.rs` — private `mod` in `apps/rocm`, accessed via qualified paths; the clap command enum usually goes in [`cli.rs`](apps/rocm/src/cli.rs) with the rest of the command tree, and its dispatch function usually stays in `main.rs`; `bootstrap.rs` and `remote/mod.rs` are the exceptions that own their enum and dispatch outright — see `docs/architecture.md`'s `bootstrap.rs` note), not growth inside `main.rs`/`lib.rs` awaiting a future extraction pass; see `docs/architecture.md` for the module map and the mechanical-relocation alternative used for dispatch-adjacent clusters +- new subsystems/subcommands: default their domain implementation to its own file from day one (full domain extraction, e.g. `therock.rs`/`comfyui.rs` — private `mod` in `apps/rocm`, accessed via qualified paths; the clap command enum usually goes in [`cli.rs`](apps/rocm/src/cli.rs) with the rest of the command tree, and its dispatch function goes in the domain module (`comfyui::run`) or a `*_cmd.rs` sibling, with only the top-level `dispatch()` match staying in `main.rs`; `bootstrap.rs` and `remote/mod.rs` are the exceptions that own their enum as well — see `docs/architecture.md`'s `bootstrap.rs` note), not growth inside `main.rs`/`lib.rs` awaiting a future extraction pass; see `docs/architecture.md` for the module map and the mechanical-relocation alternative used for dispatch-adjacent clusters - `crates/rocm-engine-protocol` is a contract surface; verify all impacted engines after protocol changes - first-party crate-layering invariants (e.g. `rocmd` must never depend on `rocm`) are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`); a new first-party dependency edge failing that check means the edge needs review, not a bypass - source file size is checked by `cargo xtask check-module-size` ([`module_size.rs`](xtask/src/module_size.rs)): no source file may exceed 1,500 production lines (test-gated items excluded), with one limit for every file and no exemption mechanism. A PR that would push a file past it splits the file in the same change; do not raise the limit or opt a file out. Merely editing a file that is already over the limit does not by itself require a split: those files are split under their own tickets, and the check cannot become blocking until none is left. Today the CI step and pre-commit hook only report; see `docs/architecture.md` @@ -324,7 +324,7 @@ Required consistency points: ### `skills/rocm-doctor/` — published from here, and a test fixture `skills/rocm-cli-assistant/SKILL.md` is compiled into the binary -(`include_str!` in `apps/rocm/src/main.rs`). `skills/rocm-doctor/` is different +(`include_str!` in `apps/rocm/src/chat/turn.rs`). `skills/rocm-doctor/` is different on two counts, and both change how you edit it: - **This repo is its source of truth.** The skill is a thin driver over the diff --git a/apps/rocm/src/advised_commands.rs b/apps/rocm/src/advised_commands.rs index 8cca54df38..923eb891e3 100644 --- a/apps/rocm/src/advised_commands.rs +++ b/apps/rocm/src/advised_commands.rs @@ -1159,7 +1159,7 @@ const NOT_INVOCATIONS: &[(&str, &str, &str)] = &[ "heading of the non-interactive launcher report", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/turn.rs", "rocm tools: {}", "`rocm tools: enabled` status row", ), @@ -1169,32 +1169,32 @@ const NOT_INVOCATIONS: &[(&str, &str, &str)] = &[ "heading of the model-readiness report, `: `, naming what was run", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/intent.rs", "rocm install folder", "natural-language phrase the planner matches in a request", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/intent.rs", "rocm installed at", "natural-language phrase the planner matches in a request", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/intent.rs", "rocm installed", "natural-language phrase the planner matches in a request", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/intent.rs", "rocm please", "natural-language phrase the planner matches in a request", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/mcp_bridge.rs", "rocm command requires at least one argument", "validation error naming the chat `rocm` tool", ), ( - "apps/rocm/src/chat.rs", + "apps/rocm/src/chat/turn.rs", "rocm command", "fallback label for an unrenderable chat tool call", ), diff --git a/apps/rocm/src/automations.rs b/apps/rocm/src/automations.rs index 6f18d9f12f..382afbe526 100644 --- a/apps/rocm/src/automations.rs +++ b/apps/rocm/src/automations.rs @@ -28,8 +28,8 @@ //! `watcher_plain_trigger`, `watcher_plain_action`, `watcher_policy_note`), //! and `render_daemon_text`. `render_automations_text` and //! `render_daemon_text` are `pub(crate)`: `main.rs`'s `dispatch()` calls -//! `render_daemon_text` directly (via the re-import above), and `chat.rs`'s -//! deterministic tool summaries call both. Everything else here has no +//! `render_daemon_text` directly (via the re-import above), and `chat/mcp_bridge.rs`'s +//! read-only command runner calls both. Everything else here has no //! caller outside this module (including `watcher_policy_note`, which the //! `automations()` handler and `render_automations_text` both reach locally //! rather than through a crate-root re-import) and stays private. diff --git a/apps/rocm/src/background_helper.rs b/apps/rocm/src/background_helper.rs new file mode 100644 index 0000000000..f15e13b21c --- /dev/null +++ b/apps/rocm/src/background_helper.rs @@ -0,0 +1,198 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! What the CLI can tell about the background automation helper (`rocm daemon`): +//! whether it is running, and the note shown to the user about it after a +//! service stop or when planning an uninstall. +//! +//! Relocated verbatim from `service_launch.rs` with no behavior change, so that +//! `services_text.rs` and `uninstall_plan.rs` can read the note +//! without importing the module that spawns services. Spawning the helper +//! (`ensure_background_helper_running` and its autostart lock) stays in +//! `service_launch.rs`, which reads `background_helper_already_running` from here. +//! This module imports nothing from the service modules. + +use anyhow::Result; +use rocm_core::{AppPaths, AutomationRuntimeState}; + +/// The "should we spawn?" decision for [`ensure_background_helper_running`], +/// factored out so it is testable hermetically (no spawn side effect). Returns +/// `true` when the file-based runtime state says the daemon is `running` AND its +/// recorded `daemon_pid` is a live process — i.e. a second spawn must be guarded. +/// A missing state file, `running=false`, or a dead/zero pid returns `false`. +pub(crate) fn background_helper_already_running(paths: &AppPaths) -> Result { + Ok(AutomationRuntimeState::load(paths)? + .is_some_and(|state| state.running && rocm_core::process_is_running(state.daemon_pid))) +} + +/// What the user should be told about the background automation helper +/// (`rocm daemon`) after stopping a service or when planning an uninstall. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct HelperNote { + /// The sentence to print. + pub(crate) text: String, + /// The helper is running right now. + pub(crate) running: bool, + /// An automation watcher is enabled, so the helper stays up on its own. + pub(crate) outlives: bool, +} + +/// The note for one combination of the two facts the helper itself acts on: +/// whether a watcher is enabled +/// ([`rocm_core::RocmCliConfig::automation_daemon_enabled`] gates both the spawn +/// and the helper's exit) and whether it is running. `None` for neither. +fn helper_note_for(enabled: &[&str], running: bool) -> Option { + let text = match (enabled.is_empty(), running) { + (true, false) => return None, + (true, true) => "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds".to_owned(), + (false, true) => { + let disable = enabled + .iter() + .map(|id| format!("`rocm automations disable {id}`")) + .collect::>() + .join(", "); + format!( + "the background automation helper (`rocm daemon`) keeps running while an automation is enabled ({}); to stop it, run {disable}, after which it exits on its own within a few seconds", + enabled.join(", ") + ) + } + (false, false) => format!( + "an automation is enabled ({}) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it", + enabled.join(", ") + ), + }; + Some(HelperNote { + text, + running, + outlives: !enabled.is_empty() && running, + }) +} + +/// [`helper_note_for`] over the facts on disk. An unreadable config counts as +/// "no watcher enabled". +pub(crate) fn background_helper_note(paths: &AppPaths) -> Option { + let config = rocm_core::RocmCliConfig::load(paths).unwrap_or_default(); + let enabled: Vec<&str> = config + .automations + .watchers + .iter() + .filter(|(_, watcher)| watcher.enabled) + .map(|(id, _)| id.as_str()) + .collect(); + let running = background_helper_already_running(paths).unwrap_or(false); + helper_note_for(&enabled, running) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + use crate::test_support::{runtime_state, set_helper_facts}; + use crate::tests::test_paths; + + #[test] + fn background_helper_already_running_true_for_live_pid() { + // Phase-10 daemon no-double-spawn: a runtime-state.json with running=true + // and a LIVE daemon_pid (this very test process) means the helper is + // already up — the "should spawn?" decision must say NO (true ⇒ skip). + // Hermetic + offline: no spawn, just the file-based liveness check. + let (root, paths) = test_paths("helper-live-pid"); + runtime_state(true, std::process::id()) + .write(&paths) + .expect("write runtime state"); + assert!( + background_helper_already_running(&paths).expect("liveness check ok"), + "live recorded pid + running=true ⇒ do not spawn a second daemon" + ); + let _ = fs::remove_dir_all(&root); + } + + #[test] + fn background_helper_already_running_false_for_dead_or_missing() { + // The inverse guard cases — each must report NOT running (false ⇒ spawn): + // (1) no state file at all, (2) running=true but a dead/zero pid, + // (3) a live pid but running=false. None must spawn from this decision. + let (root, paths) = test_paths("helper-dead-or-missing"); + // (1) No state file yet. + assert!( + !background_helper_already_running(&paths).expect("missing state ⇒ ok"), + "no runtime state ⇒ not running" + ); + // (2) running=true but pid 0 is never a live process. + runtime_state(true, 0).write(&paths).expect("write state"); + assert!( + !background_helper_already_running(&paths).expect("dead pid ⇒ ok"), + "running=true + dead pid ⇒ not running (spawn)" + ); + // (3) live pid but running flag is false. + runtime_state(false, std::process::id()) + .write(&paths) + .expect("write state"); + assert!( + !background_helper_already_running(&paths).expect("not-running flag ⇒ ok"), + "running=false ⇒ not running even with a live pid" + ); + let _ = fs::remove_dir_all(&root); + } + + /// ROCMAI-827: the helper note is derived from both facts, so it cannot + /// disagree with the state it describes. Each combination: the exact wording, + /// and the running/outlives flags the uninstall plan acts on. + #[test] + fn helper_note_follows_watcher_enabled_and_helper_running() { + let (root, paths) = test_paths("helper-note-combos"); + paths.ensure().expect("create app dirs"); + + set_helper_facts(&paths, false, false); + let neither = background_helper_note(&paths); + set_helper_facts(&paths, false, true); + let idle_running = background_helper_note(&paths).expect("note"); + set_helper_facts(&paths, true, false); + let enabled_dead = background_helper_note(&paths).expect("note"); + set_helper_facts(&paths, true, true); + let enabled_running = background_helper_note(&paths).expect("note"); + let _ = fs::remove_dir_all(&root); + + assert_eq!(neither, None); + assert_eq!( + idle_running.text, + "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds" + ); + assert!(idle_running.running && !idle_running.outlives); + assert_eq!( + enabled_dead.text, + "an automation is enabled (server-recover) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it" + ); + assert!(!enabled_dead.running && !enabled_dead.outlives); + assert_eq!( + enabled_running.text, + "the background automation helper (`rocm daemon`) keeps running while an automation is enabled (server-recover); to stop it, run `rocm automations disable server-recover`, after which it exits on its own within a few seconds" + ); + assert!(enabled_running.running && enabled_running.outlives); + } + + /// The command the note names must exist and accept exactly those arguments, + /// and its effect (the helper exiting once the last watcher is off) is + /// `automations-11` in `features/automations.feature` plus the rocmd + /// `automation_loop_` tests. + #[test] + fn the_command_named_by_the_helper_note_parses() { + use clap::Parser as _; + + let (root, paths) = test_paths("helper-note-command"); + paths.ensure().expect("create app dirs"); + set_helper_facts(&paths, true, true); + let note = background_helper_note(&paths).expect("a note"); + let _ = fs::remove_dir_all(&root); + + let command = note + .text + .split('`') + .find(|part| part.starts_with("rocm automations disable")) + .expect("the note names the disable command"); + let argv: Vec<&str> = command.split_whitespace().collect(); + assert!(crate::Cli::try_parse_from(argv).is_ok(), "{command}"); + } +} diff --git a/apps/rocm/src/bridge_snapshot.rs b/apps/rocm/src/bridge_snapshot.rs index 1d00092627..62e353f4b6 100644 --- a/apps/rocm/src/bridge_snapshot.rs +++ b/apps/rocm/src/bridge_snapshot.rs @@ -14,7 +14,7 @@ //! Mechanically relocated from `main.rs` with no behavior change; see //! `docs/architecture.md`. `dispatch()`'s `InternalBridgeSnapshot` arm calls //! `build_codex_bridge_snapshot` through a re-import in `main.rs`, and -//! `chat.rs`'s internal MCP tools (`bridge_snapshot`, `gpu_snapshot` and +//! `chat/mcp_bridge.rs`'s internal MCP tools (`bridge_snapshot`, `gpu_snapshot` and //! `engines`) reach `build_codex_bridge_snapshot`, //! `build_codex_bridge_gpu_snapshot` and `builtin_codex_bridge_engine_inventory` //! as `crate::` through the same re-import, so those three are diff --git a/apps/rocm/src/chat.rs b/apps/rocm/src/chat.rs index 4ddcaa6893..e817f0ab97 100644 --- a/apps/rocm/src/chat.rs +++ b/apps/rocm/src/chat.rs @@ -2,427 +2,68 @@ // // SPDX-License-Identifier: MIT -//! Chat/assistant tool-call support: the chat/MCP-bridge cluster's types -//! (`DeterministicModelRecipe`, `DeterministicServiceRow`, -//! `DeterministicEngineRow`, and others — all module-private), its -//! tool-call execution logic, deterministic tool-output summarization, and -//! the MCP tool-bridge entry points (`run_internal_mcp_call` and its -//! helpers). The tool-call validation and approval policy -//! (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, -//! `ChatRocmCommandAction` and `ChatToolApprovalRequest`) lives in -//! `tool_policy.rs`, which this module calls into and re-exports from. -//! Callers in other modules, production and test-only, reach this module's -//! items through the `chat::` module boundary. +//! The chat assistant's entry points and the module that ties its pieces +//! together: reading a piped prompt (`read_piped_prompt`), the status screen +//! (`render_chat_text`), the one-shot reply text (`render_chat_prompt_text`) and +//! the clap value parsers for the sampling flags. The work behind them is split +//! into child modules, each a leaf of the call graph below the turn loop: `turn` +//! (the turn loop, the system prompt and the transcript), `intent` (the +//! prompt-intent heuristics), `mcp_bridge` (the MCP tool bridge and in-process +//! command execution) and `summaries` (the deterministic tool-output +//! summaries). The tool-call validation and approval policy lives in +//! `tool_policy.rs`. //! -//! Full domain extraction, mirroring `providers.rs`. Phase 6d (ROCMAI-88) of -//! the `main.rs` modularization (see `docs/architecture.md`), split across -//! three PRs given the cluster's size: this module's contents are the -//! combined slice of all three (types and leaf helpers, the orchestration -//! bodies, and the cluster's regression tests moved out of `main.rs`'s -//! shared `mod tests` block), plus the third PR's own visibility audit — -//! every function still `pub(crate)` has a confirmed caller outside this -//! module, production or test-only; none of the module's own types need to -//! be. - -use crate::automations::{render_automations_text, render_daemon_text}; -use crate::engine_runtime_select::{app_path_env_vars, service_model_names_match}; -use crate::freeform_planner::{format_structured_tool_call, plan_freeform_request_with_context}; -use crate::managed_service_status::{ - friendly_engine_label, load_managed_services, managed_service_is_live, -}; +//! Callers in other modules, production and test-only, reach the moved items +//! through the `chat::` module boundary, so this module re-exports each one +//! another module reads (`run_internal_mcp_call`, +//! `rocm_chat_tool_system_prompt_for_host`, the tool-policy items, `any_substring` +//! and `ensure_flag`, and the test-only ones under `#[cfg(test)]`). + +mod intent; +mod mcp_bridge; +mod summaries; +mod turn; + +// `deterministic_comfyui_tool_summary` lives in `summaries.rs` and stays +// reachable here as `chat::deterministic_comfyui_tool_summary` for +// `comfyui.rs`'s tests, which are its only readers outside `summaries.rs`. +// Those tests are all `#[cfg(unix)]`, so the re-export is gated the same way: +// on Windows it would be an unused import. +#[cfg(all(test, unix))] +pub(crate) use self::summaries::deterministic_comfyui_tool_summary; +// `run_internal_mcp_call` lives in `mcp_bridge.rs` and stays reachable here as +// `chat::run_internal_mcp_call` for `main.rs` and `dash_seam.rs`. +pub(crate) use self::mcp_bridge::run_internal_mcp_call; +// `rocm_chat_tool_system_prompt_for_host` lives in `turn.rs` and stays +// reachable here as `chat::rocm_chat_tool_system_prompt_for_host` for `dash.rs`. +use self::turn::render_chat_prompt_result_with_progress; +pub(crate) use self::turn::rocm_chat_tool_system_prompt_for_host; // `any_substring` lives in `prompt_text.rs` and stays reachable here as // `chat::any_substring` for `freeform_answer.rs`. pub(crate) use crate::prompt_text::any_substring; -use crate::prompt_text::{ - infer_engine_from_request, requested_install_prefix_from_prompt, - requested_therock_build_date_from_prompt, requested_therock_version_from_prompt, -}; // `ensure_flag` lives in `rocm_argv.rs` and stays reachable here as // `chat::ensure_flag` for `freeform_execution.rs`. pub(crate) use crate::rocm_argv::ensure_flag; -use crate::rocm_argv::{chat_cli_arg_value, chat_cli_has_flag}; -use crate::runtimes_cmd::render_runtimes_text; -use crate::services_text::{ - read_tail_lines, render_logs_browser_text, render_logs_text, render_service_logs_text, - render_services_text, -}; // The tool-call policy lives in `tool_policy.rs`. `chat_tool_approval_request`, // `chat_tool_call_is_read_only` and `validate_chat_tool_call` stay reachable -// here as `chat::` for `dash_seam.rs`. `ChatRocmCommandAction` and -// `rocm_chat_tool_requested_args` are used by this module's own code, so the -// import cannot be `#[cfg(test)]`; it is also `pub(crate)` because other -// modules' tests (`driver_plan.rs`, `system_packages.rs`) read both through -// `chat::`. `chat_install_driver_action` and `chat_rocm_command_action_from_args` -// are read through `chat::` only by those tests and this module does not call -// them itself, so their re-export below is `#[cfg(test)]`. -pub(crate) use crate::tool_policy::{ - ChatRocmCommandAction, chat_tool_approval_request, chat_tool_call_is_read_only, - rocm_chat_tool_requested_args, validate_chat_tool_call, -}; -use crate::tool_policy::{ - ChatToolApprovalRequest, chat_install_prefix_is_system, chat_rocm_command_action, json_string, - normalized_chat_rocm_command_args, rocm_chat_tool_requested_command, -}; +// here as `chat::` for `dash_seam.rs`. `ChatRocmCommandAction`, +// `rocm_chat_tool_requested_args`, `chat_install_driver_action` and +// `chat_rocm_command_action_from_args` are read through `chat::` only by other +// modules' tests (`driver_plan.rs`, `system_packages.rs`), and this module does +// not call them itself, so their re-export below is `#[cfg(test)]`. #[cfg(test)] pub(crate) use crate::tool_policy::{ - chat_install_driver_action, chat_rocm_command_action_from_args, -}; -use crate::uninstall_plan::render_uninstall_dry_run; -use crate::update_cmd::render_update_text; -use crate::{ - ChatInferenceParams, build_codex_bridge_gpu_snapshot, build_codex_bridge_snapshot, - builtin_codex_bridge_engine_inventory, cli_version_string, record_cli_audit_event, - render_config_text, render_engine_inventory_text_with_paths, render_examine_text, - render_examine_text_with_paths, render_model_registry_verbose_text_with_context_and_host, - render_setup_status_text, therock_install_version_selector, + ChatRocmCommandAction, chat_install_driver_action, chat_rocm_command_action_from_args, + rocm_chat_tool_requested_args, }; -use crate::{chat_host_facts, comfyui, providers, therock}; -use anyhow::{Context, Result, bail}; -use rocm_core::placeholder::empty_as_unknown; -use rocm_core::{ - AppPaths, AutomationRuntimeState, CommandCapture, DEFAULT_LOCAL_HOST, ExamineSummary, - ManagedServiceRecord, RocmCliConfig, command_capture_text, load_recent_automation_events, +pub(crate) use crate::tool_policy::{ + chat_tool_approval_request, chat_tool_call_is_read_only, validate_chat_tool_call, }; -use rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES; +use crate::{ChatInferenceParams, providers}; +use anyhow::{Context, Result}; +use rocm_core::AppPaths; use std::fmt::Write as _; use std::io::Read as _; -use std::net::{TcpStream, ToSocketAddrs}; -use std::path::{Path, PathBuf}; -use std::time::Duration; - -fn mcp_tool_result_text(value: &serde_json::Value) -> String { - value - .get("content") - .and_then(serde_json::Value::as_array) - .map(|items| { - items - .iter() - .filter_map(|item| { - (item.get("type").and_then(serde_json::Value::as_str) == Some("text")) - .then(|| item.get("text").and_then(serde_json::Value::as_str)) - .flatten() - }) - .collect::>() - .join("\n") - }) - .filter(|text| !text.trim().is_empty()) - .unwrap_or_else(|| value.to_string()) -} - -fn mcp_tool_result_is_error(value: &serde_json::Value) -> bool { - value - .get("isError") - .and_then(serde_json::Value::as_bool) - .unwrap_or(false) -} - -fn chat_tool_display_label(name: &str) -> String { - match name { - "examine" => "Checked this computer".to_owned(), - "gpu_snapshot" => "Checked GPU status".to_owned(), - "engines" => "Checked local engines".to_owned(), - "services" => "Checked model servers".to_owned(), - "bridge_snapshot" => "Checked ROCm state".to_owned(), - "service_logs" => "Read server logs".to_owned(), - "automations" => "Checked automations".to_owned(), - "natural_language_plan" => "Planned ROCm request".to_owned(), - "path_exists" => "Checked folder path".to_owned(), - "port_status" => "Checked local port".to_owned(), - "update_check" => "Checked for ROCm updates".to_owned(), - "install_sdk_dry_run" => "Previewed ROCm install".to_owned(), - "install_sdk" => "Install ROCm".to_owned(), - "install_engine" => "Install engine".to_owned(), - "launch_server" => "Start local model server".to_owned(), - "stop_server" => "Stop local model server".to_owned(), - "watcher_enable" => "Enable automation".to_owned(), - "watcher_disable" => "Disable automation".to_owned(), - other => other.replace('_', " "), - } -} - -fn chat_tool_call_display_label(call: &providers::ChatToolCall) -> String { - if call.name != "rocm_command" { - return chat_tool_display_label(&call.name); - } - let Some(args) = normalized_chat_rocm_command_args(call).ok() else { - return "rocm command".to_owned(); - }; - match args.as_slice() { - [command] if command.eq_ignore_ascii_case("examine") => "Checked this computer".to_owned(), - [command] - if command.eq_ignore_ascii_case("model") || command.eq_ignore_ascii_case("models") => - { - "Checked model recipes".to_owned() - } - [command, subcommand] - if command.eq_ignore_ascii_case("engines") - && subcommand.eq_ignore_ascii_case("list") => - { - "Checked local engines".to_owned() - } - [command] if command.eq_ignore_ascii_case("services") => "Checked model servers".to_owned(), - [command, subcommand, rest @ ..] - if command.eq_ignore_ascii_case("services") - && subcommand.eq_ignore_ascii_case("list") - && rest - .iter() - .all(|arg| matches!(arg.as_str(), "-a" | "--all")) => - { - "Checked model servers".to_owned() - } - [command] if command.eq_ignore_ascii_case("comfyui") => "Checked ComfyUI".to_owned(), - [command, subcommand] - if command.eq_ignore_ascii_case("comfyui") - && subcommand.eq_ignore_ascii_case("status") => - { - "Checked ComfyUI".to_owned() - } - [command, subcommand, ..] - if command.eq_ignore_ascii_case("comfyui") - && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => - { - "Read ComfyUI logs".to_owned() - } - [command, subcommand] - if command.eq_ignore_ascii_case("config") - && subcommand.eq_ignore_ascii_case("show") => - { - "Checked ROCm config".to_owned() - } - _ => "rocm command".to_owned(), - } -} - -fn loopback_tcp_port_is_reachable(host: &str, port: u16) -> bool { - let Ok(addresses) = (host, port).to_socket_addrs() else { - return false; - }; - addresses - .into_iter() - .any(|address| TcpStream::connect_timeout(&address, Duration::from_millis(200)).is_ok()) -} - -fn parse_optional_query(args: &[String]) -> Result> { - let mut index = 0; - while index < args.len() { - match args[index].as_str() { - "--query" | "-q" => { - let value = args - .get(index + 1) - .context("logs query flag requires a value")?; - return Ok(Some(value.to_owned())); - } - value if value.starts_with("--") => {} - value => return Ok(Some(value.to_owned())), - } - index += 1; - } - Ok(None) -} - -fn parse_optional_lines(args: &[String]) -> Result { - let mut index = 0; - while index < args.len() { - if matches!(args[index].as_str(), "--lines" | "-n") { - let value = args.get(index + 1).context("lines flag requires a value")?; - return value - .parse::() - .context("lines flag must be a positive number"); - } - index += 1; - } - Ok(DEFAULT_LOG_TAIL_LINES) -} - -#[derive(Debug, Default)] -struct DeterministicModelRecipe { - pub(crate) canonical_id: String, - pub(crate) min_gpu_mem_gib: Option, - pub(crate) engines: Vec, - pub(crate) engine_statuses: Vec, - pub(crate) warnings: Vec, -} - -fn parse_deterministic_model_recipes(tool_text: &str) -> Vec { - let mut recipes = Vec::new(); - let mut current: Option = None; - for line in tool_text.lines() { - let trimmed = line.trim(); - if let Some(recipe) = parse_model_recipe_header(trimmed) { - if let Some(previous) = current.replace(recipe) { - recipes.push(previous); - } - continue; - } - let Some(recipe) = current.as_mut() else { - continue; - }; - if is_engine_status_line(trimmed) { - recipe.engine_statuses.push(trimmed.to_owned()); - } else if let Some(warning) = trimmed.strip_prefix("warning:") { - recipe.warnings.push(warning.trim().to_owned()); - } - } - if let Some(recipe) = current { - recipes.push(recipe); - } - recipes -} - -fn parse_model_recipe_header(line: &str) -> Option { - if !line.contains(" aliases=[") || !line.contains("min_gpu_mem=") { - return None; - } - let canonical_id = line.split_whitespace().next()?.to_owned(); - let engines = parse_bracketed_csv(line, "engines=[", "]"); - let min_gpu_mem_gib = line - .split_once("min_gpu_mem=") - .and_then(|(_, rest)| rest.split_whitespace().next()) - .and_then(|value| value.parse::().ok()); - Some(DeterministicModelRecipe { - canonical_id, - min_gpu_mem_gib, - engines, - engine_statuses: Vec::new(), - warnings: Vec::new(), - }) -} - -fn parse_bracketed_csv(line: &str, start: &str, end: &str) -> Vec { - line.split_once(start) - .and_then(|(_, rest)| rest.split_once(end)) - .map(|(values, _)| { - values - .split(',') - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(str::to_owned) - .collect() - }) - .unwrap_or_default() -} - -fn is_engine_status_line(line: &str) -> bool { - matches!( - line.split_once(':').map(|(engine, _)| engine), - Some("lemonade" | "vllm" | "aim") - ) -} - -#[derive(Debug, Default)] -struct DeterministicServiceRow { - pub(crate) service_id: String, - pub(crate) engine: String, - pub(crate) model: String, - pub(crate) status: String, - pub(crate) running_state: String, - pub(crate) endpoint: String, -} - -fn parse_deterministic_service_rows(tool_text: &str) -> Vec { - let mut rows = Vec::new(); - let mut current: Option = None; - for line in tool_text.lines() { - let trimmed = line.trim(); - if let Some(rest) = trimmed.strip_prefix("- ") { - if rest.starts_with("service_id=") { - if let Some(row) = current.take() { - rows.push(row); - } - let mut row = DeterministicServiceRow::default(); - for token in rest.split_whitespace() { - if let Some((key, value)) = token.split_once('=') { - match key { - "service_id" => row.service_id = value.to_owned(), - "engine" => row.engine = value.to_owned(), - "model" => row.model = value.to_owned(), - "status" => row.status = value.to_owned(), - "running_state" => row.running_state = value.to_owned(), - "endpoint" => row.endpoint = value.to_owned(), - _ => {} - } - } - } - if row.running_state.is_empty() { - row.running_state = managed_service_running_state(&row.status).to_owned(); - } - current = Some(row); - continue; - } - if let Some(row) = current.take() { - rows.push(row); - } - current = Some(DeterministicServiceRow { - service_id: rest.to_owned(), - ..Default::default() - }); - continue; - } - let Some(row) = current.as_mut() else { - continue; - }; - if let Some(value) = trimmed.strip_prefix("status:") { - row.status = value.trim().to_owned(); - row.running_state = managed_service_running_state(&row.status).to_owned(); - } else if let Some(value) = trimmed.strip_prefix("engine:") { - row.engine = value.trim().to_owned(); - } else if let Some(value) = trimmed.strip_prefix("model:") { - row.model = value.trim().to_owned(); - } else if let Some(value) = trimmed.strip_prefix("endpoint:") { - row.endpoint = value.trim().to_owned(); - } - } - if let Some(row) = current { - rows.push(row); - } - rows -} - -#[derive(Debug, Default)] -struct DeterministicEngineRow { - pub(crate) engine: String, - pub(crate) runtime: String, - pub(crate) note: String, -} - -fn parse_deterministic_engine_rows(tool_text: &str) -> Vec { - let mut rows = Vec::new(); - let mut current: Option = None; - for line in tool_text.lines() { - let trimmed = line.trim().trim_start_matches("* ").trim_start(); - if let Some(engine) = engine_name_from_inventory_line(trimmed) { - if let Some(row) = current.take() { - rows.push(row); - } - current = Some(DeterministicEngineRow { - engine: engine.to_owned(), - ..Default::default() - }); - continue; - } - let Some(row) = current.as_mut() else { - continue; - }; - if let Some(runtime) = trimmed.strip_prefix("runtime:") { - row.runtime = runtime.trim().to_owned(); - } else if let Some(note) = trimmed.strip_prefix("note:") { - row.note = note.trim().to_owned(); - } - } - if let Some(row) = current { - rows.push(row); - } - rows -} - -fn engine_name_from_inventory_line(line: &str) -> Option<&'static str> { - ["lemonade", "vllm", "aim"].into_iter().find(|engine| { - line == *engine - || line - .strip_prefix(*engine) - .is_some_and(|rest| rest.starts_with(char::is_whitespace)) - }) -} /// Read a one-shot chat prompt from standard input when it is piped in. /// @@ -585,4086 +226,9 @@ pub(crate) fn parse_positive_u32(value: &str) -> Result { } Ok(parsed) } - -#[derive(Debug, Clone, Eq, PartialEq)] -struct ChatPromptResult { - pub rendered: String, - pub approval: Option, -} - -struct ChatToolRunResult { - pub(crate) approval: Option, - pub(crate) follow_up_text: String, - pub(crate) ran_read_only_tool: bool, - pub(crate) read_only_tool_error: bool, - pub(crate) needs_install_folder: bool, -} - -/// Convenience wrapper used by tests that do not exercise sampling controls. #[cfg(test)] -fn render_chat_prompt_result( - paths: &AppPaths, - provider: &str, - model: Option<&str>, - prompt: &str, - rocm_tools: bool, -) -> Result { - render_chat_prompt_result_with_progress( - paths, - provider, - model, - prompt, - rocm_tools, - ChatInferenceParams::default(), - None, - ) -} - -fn render_chat_prompt_result_with_progress( - paths: &AppPaths, - provider: &str, - model: Option<&str>, - prompt: &str, - rocm_tools: bool, - params: ChatInferenceParams, - progress: Option<&mut dyn FnMut(String)>, -) -> Result { - let mut progress = progress; - let user_prompt = latest_user_chat_message(prompt); - let assistant_model = local_rocm_tools_assistant_model(provider, rocm_tools).or(model); - let service_needed_model = if local_rocm_tools_assistant_model(provider, rocm_tools).is_some() { - None - } else { - model - }; - if rocm_tools && let Some(approval) = install_sdk_without_prefix_chat_approval(user_prompt) { - report_chat_tool_progress(&mut progress, "Asking the assistant."); - report_chat_tool_progress(&mut progress, "Review needed: Install ROCm"); - report_chat_tool_progress(&mut progress, "Waiting for the ROCm install folder."); - return Ok(ChatPromptResult { - rendered: render_install_sdk_folder_needed_chat_text(user_prompt, &approval.args), - approval: Some(approval), - }); - } - let mut messages = Vec::new(); - if rocm_tools { - messages.push(providers::ChatMessage { - role: "system".to_owned(), - content: rocm_chat_tool_system_prompt_for_host(Some(paths)), - }); - } - messages.push(providers::ChatMessage { - role: "user".to_owned(), - content: prompt.to_owned(), - }); - let mut response = if rocm_tools { - report_chat_tool_progress(&mut progress, "Asking the assistant."); - if let Some(call) = deterministic_mutating_tool_call_for_prompt(user_prompt)? { - report_chat_tool_progress(&mut progress, "Preparing a review card."); - providers::ChatResponse { - provider: provider.to_owned(), - model: assistant_model.unwrap_or("local").to_owned(), - content: deterministic_mutating_tool_intro(&call), - tool_calls: vec![call], - } - } else if !local_assistant_service_ready_for_chat(paths, assistant_model) - && let Some(response) = - local_read_only_fallback_response_for_prompt(provider, assistant_model, user_prompt) - { - report_chat_tool_progress(&mut progress, "Running ROCm status check."); - response - } else { - match providers::provider_chat( - paths, - provider, - &providers::ChatRequest { - model: assistant_model.map(str::to_owned), - messages: messages.clone(), - max_tokens: params.max_tokens, - temperature: params.temperature, - top_p: params.top_p, - rocm_tools, - }, - ) { - Ok(response) => response, - Err(error) - if provider == "local" && local_provider_missing_service_error(&error) => - { - if let Some(response) = local_read_only_fallback_response_for_prompt( - provider, - assistant_model, - user_prompt, - ) { - report_chat_tool_progress(&mut progress, "Running ROCm status check."); - response - } else { - return Ok(ChatPromptResult { - rendered: local_chat_service_needed_text( - service_needed_model, - user_prompt, - rocm_tools, - ), - approval: None, - }); - } - } - Err(error) => return Err(error), - } - } - } else { - report_chat_tool_progress(&mut progress, "Asking the assistant."); - match providers::provider_chat( - paths, - provider, - &providers::ChatRequest { - model: assistant_model.map(str::to_owned), - messages: messages.clone(), - max_tokens: params.max_tokens, - temperature: params.temperature, - top_p: params.top_p, - rocm_tools, - }, - ) { - Ok(response) => response, - Err(error) if provider == "local" && local_provider_missing_service_error(&error) => { - return Ok(ChatPromptResult { - rendered: local_chat_service_needed_text( - service_needed_model, - user_prompt, - rocm_tools, - ), - approval: None, - }); - } - Err(error) => return Err(error), - } - }; - let fallback_tool_call = if rocm_tools && response.tool_calls.is_empty() { - fallback_rocm_tool_call_for_prompt(user_prompt) - } else { - None - }; - let fallback_tool_call_used = fallback_tool_call.is_some(); - if let Some(call) = fallback_tool_call { - response.tool_calls.push(call); - } - if rocm_tools - && let Some(call) = - supplemental_read_only_tool_call_for_prompt(user_prompt, &response.tool_calls) - { - response.tool_calls.push(call); - } - let mut output = String::new(); - let _ = writeln!(output, "chat response"); - let _ = writeln!(output, " provider: {}", response.provider); - let _ = writeln!(output, " model: {}", response.model); - let _ = writeln!( - output, - " rocm tools: {}", - if rocm_tools { "enabled" } else { "off" } - ); - let _ = writeln!(output); - let initial_content_is_intermediate = fallback_tool_call_used - || local_tool_call_content_is_intermediate(provider, rocm_tools, &response); - let initial_content = visible_chat_content(&response.content); - if initial_content_is_intermediate { - let _ = writeln!(output, "Assistant is preparing the next step."); - } else if !initial_content.trim().is_empty() { - let _ = writeln!(output, "{initial_content}"); - } - let tool_result = if rocm_tools { - let explanation = (!fallback_tool_call_used && !initial_content.trim().is_empty()) - .then_some(initial_content.as_str()); - append_chat_tool_results(paths, &response, &mut output, explanation, &mut progress)? - } else { - ChatToolRunResult { - approval: None, - follow_up_text: String::new(), - ran_read_only_tool: false, - read_only_tool_error: false, - needs_install_folder: false, - } - }; - let deterministic_summary = deterministic_chat_tool_summary(&tool_result.follow_up_text); - if let Some(summary) = deterministic_summary.as_deref() { - let _ = writeln!(output); - let _ = writeln!(output, "ROCm CLI summary"); - let _ = writeln!(output, "{summary}"); - } - let follow_up_blocking_summary = deterministic_summary - .as_deref() - .filter(|_| deterministic_summary_can_stand_alone(&tool_result.follow_up_text)); - if should_request_local_tool_follow_up(provider, &tool_result, follow_up_blocking_summary) { - let follow_up_messages = vec![ - providers::ChatMessage { - role: "system".to_owned(), - content: "You are ROCm CLI's local assistant. ROCm tools have already been checked for this turn. Use the supplied tool results for local facts about this machine. For service results, ready/running means running, starting/recovering means starting, failed/stopped means not running, and no matching service row means ROCm CLI is not managing that service as running. If the tool results do not contain enough information to answer the whole question, say what is known from the results and then answer the rest normally from your own model knowledge. Do not request another tool call. Do not invent local paths, GPU names, versions, or install state that are not shown in the tool results. Keep the answer concise.".to_owned(), - }, - providers::ChatMessage { - role: "user".to_owned(), - content: format!( - "Original request:\n{}\n\nROCm tool results:\n{}\n\nAnswer the original request. Use the ROCm tool results for this machine's facts; if they are incomplete, answer the remaining part generally.", - user_prompt.trim(), - tool_result.follow_up_text.trim() - ), - }, - ]; - let follow_up = providers::provider_chat( - paths, - provider, - &providers::ChatRequest { - model: assistant_model.map(str::to_owned), - messages: follow_up_messages, - max_tokens: params.max_tokens.or(Some(256)), - temperature: params.temperature, - top_p: params.top_p, - rocm_tools: false, - }, - )?; - if follow_up.tool_calls.is_empty() { - let follow_up_content = visible_chat_content(&follow_up.content); - if local_follow_up_content_is_final(&follow_up) && !follow_up_content.trim().is_empty() - { - let _ = writeln!(output); - let _ = writeln!(output, "Assistant after ROCm checks"); - let _ = writeln!(output, "{}", follow_up_content.trim()); - } - } else { - let _ = writeln!(output); - let _ = writeln!( - output, - "Assistant asked for another ROCm check after the first tool results." - ); - let _ = writeln!( - output, - "rocm-cli stopped there so the answer is not based on another guess. Nothing was changed." - ); - } - } else if tool_result.read_only_tool_error { - let _ = writeln!(output); - let _ = writeln!( - output, - "Assistant did not answer from this result because a ROCm check reported an error." - ); - let _ = writeln!(output, "Nothing was changed."); - } else if tool_result.needs_install_folder { - let _ = writeln!(output); - let _ = writeln!(output, "Assistant after ROCm checks"); - let _ = writeln!( - output, - "I can install ROCm/TheRock, but I need the install folder first. Type the folder you want to use, for example D:\\ROCm\\therock_venvs. Nothing will download until ROCm CLI shows a review card and you approve it." - ); - } - if tool_result.approval.is_none() - && tool_result.ran_read_only_tool - && !output.contains("Nothing was changed.") - { - let _ = writeln!(output); - let _ = writeln!(output, "Nothing was changed."); - } - Ok(ChatPromptResult { - rendered: output, - approval: tool_result.approval, - }) -} - -fn report_chat_tool_progress(progress: &mut Option<&mut dyn FnMut(String)>, message: &str) { - if let Some(progress) = progress.as_deref_mut() { - progress(message.to_owned()); - } -} - -fn latest_user_chat_message(prompt: &str) -> &str { - const MARKER: &str = "\nNew message:\n"; - prompt - .rfind(MARKER) - .map_or(prompt, |index| &prompt[index + MARKER.len()..]) - .trim() -} - -fn deterministic_mutating_tool_call_for_prompt( - prompt: &str, -) -> Result> { - let Some(call) = fallback_rocm_tool_call_for_prompt(prompt) else { - return Ok(None); - }; - validate_chat_tool_call(&call)?; - if chat_tool_call_is_read_only(&call) { - Ok(None) - } else { - Ok(Some(call)) - } -} - -fn supplemental_read_only_tool_call_for_prompt( - prompt: &str, - existing_calls: &[providers::ChatToolCall], -) -> Option { - let normalized = latest_user_chat_message(prompt).to_ascii_lowercase(); - if prompt_mentions_serving_engine_or_service(&normalized) - && prompt_asks_running_or_status(&normalized) - && prompt_asks_engine_install_state(&normalized) - { - for call in [ - fallback_services_list_tool_call(), - fallback_engine_list_tool_call(), - ] { - if !chat_tool_calls_include_equivalent(existing_calls, &call) { - return Some(call); - } - } - return None; - } - let call = fallback_rocm_tool_call_for_prompt(prompt)?; - if !chat_tool_call_is_read_only(&call) - || chat_tool_calls_include_equivalent(existing_calls, &call) - { - return None; - } - Some(call) -} - -fn chat_tool_calls_include_equivalent( - existing_calls: &[providers::ChatToolCall], - required: &providers::ChatToolCall, -) -> bool { - existing_calls.iter().any(|existing| { - if existing.name != required.name { - return false; - } - if required.name == "rocm_command" { - return normalized_chat_rocm_command_args(existing).ok() - == normalized_chat_rocm_command_args(required).ok(); - } - if required.name == "port_status" { - return chat_port_status_tool_calls_equivalent(existing, required); - } - existing.arguments == required.arguments - }) -} - -fn chat_port_status_tool_calls_equivalent( - left: &providers::ChatToolCall, - right: &providers::ChatToolCall, -) -> bool { - let Some(left_object) = left.arguments.as_object() else { - return false; - }; - let Some(right_object) = right.arguments.as_object() else { - return false; - }; - let left_port = left_object.get("port").and_then(serde_json::Value::as_u64); - let right_port = right_object.get("port").and_then(serde_json::Value::as_u64); - if left_port != right_port { - return false; - } - let left_host = - json_string(left_object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); - let right_host = - json_string(right_object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); - loopback_host_key(&left_host) == loopback_host_key(&right_host) -} - -fn loopback_host_key(host: &str) -> String { - match host.trim().to_ascii_lowercase().as_str() { - "localhost" | "127.0.0.1" => "127.0.0.1".to_owned(), - "::1" | "[::1]" => "::1".to_owned(), - other => other.to_owned(), - } -} - -fn deterministic_mutating_tool_intro(call: &providers::ChatToolCall) -> String { - match chat_tool_approval_request(call, None) { - Ok(approval) if approval.pending_title == "Install ROCm" => { - "I can install ROCm/TheRock into the folder you chose. Review the card before anything downloads or changes." - .to_owned() - } - Ok(approval) if approval.pending_title == "Install ComfyUI" => { - "I can install ComfyUI into ROCm CLI's managed app folder. Review the card before anything downloads or changes." - .to_owned() - } - Ok(approval) if approval.pending_title == "Start ComfyUI" => { - "I can start ComfyUI for you. Review the card before ROCm CLI launches it." - .to_owned() - } - Ok(approval) if approval.pending_title == "Start local model server" => { - "I can start the recommended local model server on the GPU. Review the card before ROCm CLI launches it." - .to_owned() - } - Ok(approval) => format!( - "I can prepare this ROCm change: {}. Review the card before anything runs.", - approval.pending_title - ), - Err(_) => { - "I can prepare this ROCm change. Review the card before anything runs.".to_owned() - } - } -} - -fn local_tool_call_content_is_intermediate( - provider: &str, - rocm_tools: bool, - response: &providers::ChatResponse, -) -> bool { - provider == "local" - && rocm_tools - && !response.tool_calls.is_empty() - && !response.content.trim().is_empty() -} - -fn local_follow_up_content_is_final(response: &providers::ChatResponse) -> bool { - response.tool_calls.is_empty() && !response.content.trim().is_empty() -} - -fn local_read_only_fallback_response_for_prompt( - provider: &str, - assistant_model: Option<&str>, - prompt: &str, -) -> Option { - if provider != "local" || !prompt_can_use_read_only_without_local_assistant(prompt) { - return None; - } - let call = fallback_rocm_tool_call_for_prompt(prompt)?; - if !chat_tool_call_is_read_only(&call) { - return None; - } - Some(providers::ChatResponse { - provider: provider.to_owned(), - model: assistant_model.unwrap_or("local").to_owned(), - content: String::new(), - tool_calls: vec![call], - }) -} - -fn local_assistant_service_ready_for_chat(paths: &AppPaths, assistant_model: Option<&str>) -> bool { - let model = assistant_model.unwrap_or(providers::BUILTIN_ASSISTANT_MODEL_ID); - load_managed_services(paths).is_ok_and(|records| { - records.iter().any(|record| { - matches!(record.status.as_str(), "ready" | "running") - && (service_model_names_match(&record.canonical_model_id, model) - || service_model_names_match(&record.model_ref, model)) - }) - }) -} - -fn prompt_can_use_read_only_without_local_assistant(prompt: &str) -> bool { - let normalized = latest_user_chat_message(prompt).to_ascii_lowercase(); - let mentions_status_subject = any_substring( - &normalized, - &[ - "comfyui", - "comfy ui", - "comfy", - "vllm", - "lemonade", - "llama.cpp", - "llama cpp", - "qwen", - "model server", - "local server", - "local model server", - "assistant server", - "port", - "8188", - "therock", - "rocm", - ], - ); - mentions_status_subject - && (prompt_asks_running_or_status(&normalized) - || any_substring( - &normalized, - &["installed", "available", "detected", "engine status"], - )) -} - -fn prompt_mentions_serving_engine_or_service(normalized_prompt: &str) -> bool { - any_substring( - normalized_prompt, - &[ - "vllm", - "lemonade", - "llama.cpp", - "llama cpp", - "qwen", - "model server", - "local server", - "local model server", - "assistant server", - ], - ) -} - -fn prompt_asks_engine_install_state(normalized_prompt: &str) -> bool { - any_substring( - normalized_prompt, - &["installed", "available", "detected", "engine status"], - ) -} - -fn fallback_engine_list_tool_call() -> providers::ChatToolCall { - providers::ChatToolCall { - id: Some("fallback-engine-list".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["engines", "list"] }), - } -} - -fn fallback_services_list_tool_call() -> providers::ChatToolCall { - providers::ChatToolCall { - id: Some("fallback-services-list".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["services", "list", "--all"] }), - } -} - -fn fallback_rocm_tool_call_for_prompt(prompt: &str) -> Option { - let prompt = latest_user_chat_message(prompt); - let normalized = prompt.to_ascii_lowercase(); - let asks_running_or_status = prompt_asks_running_or_status(&normalized); - let mentions_comfyui = any_substring(&normalized, &["comfyui", "comfy ui", "comfy"]); - if asks_running_or_status - && !mentions_comfyui - && any_substring(&normalized, &["8188", "port 8188"]) - { - return Some(providers::ChatToolCall { - id: Some("fallback-port-8188-status".to_owned()), - name: "port_status".to_owned(), - arguments: serde_json::json!({ "host": DEFAULT_LOCAL_HOST, "port": 8188 }), - }); - } - let mentions_serving_engine_or_service = prompt_mentions_serving_engine_or_service(&normalized); - if mentions_serving_engine_or_service && prompt_asks_engine_install_state(&normalized) { - return Some(fallback_engine_list_tool_call()); - } - if mentions_serving_engine_or_service && asks_running_or_status { - return Some(fallback_services_list_tool_call()); - } - let mentions_llm_or_model = any_substring( - &normalized, - &["llm", "llms", "model", "models", "assistant"], - ); - let asks_support_or_fit = any_substring( - &normalized, - &[ - "support", - "supported", - "run", - "runs", - "fit", - "fits", - "can my machine", - "can this machine", - ], - ); - if mentions_llm_or_model && asks_support_or_fit { - return Some(providers::ChatToolCall { - id: Some("fallback-rocm-model".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["model"] }), - }); - } - - if mentions_comfyui { - if any_substring(&normalized, &["log", "logs"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-comfyui-logs".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["comfyui", "logs"] }), - }); - } - if asks_running_or_status { - return Some(providers::ChatToolCall { - id: Some("fallback-comfyui-status".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["comfyui", "status"] }), - }); - } - if any_substring(&normalized, &["start", "run", "launch", "open"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-comfyui-start".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["comfyui", "start"], - "reason": "Start ComfyUI locally after the user approves it." - }), - }); - } - if any_substring( - &normalized, - &[ - "can you setup", - "can you set up", - "please setup", - "please set up", - "setup comfyui for me", - "set up comfyui for me", - "install comfyui", - "download comfyui", - ], - ) { - return Some(providers::ChatToolCall { - id: Some("fallback-comfyui-install".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["comfyui", "install"], - "reason": "Install ComfyUI into ROCm CLI's managed app folder after the user approves it." - }), - }); - } - return Some(providers::ChatToolCall { - id: Some("fallback-comfyui-status".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["comfyui", "status"] }), - }); - } - - if let Some(call) = fallback_config_tool_call_for_prompt(&normalized) { - return Some(call); - } - - if mentions_llm_or_model - && any_substring( - &normalized, - &[ - "serve", - "server", - "start", - "setup and serve", - "set up and serve", - "run locally", - "local model", - "local assistant", - ], - ) - { - return Some(providers::ChatToolCall { - id: Some("fallback-serve-qwen".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["serve", "qwen", "--engine", "lemonade", "--device", "gpu_required", "--managed"], - "reason": "Start the recommended local assistant after the user approves it." - }), - }); - } - - let mentions_setup = any_substring(&normalized, &["setup", "set up", "install"]); - let asks_how = any_substring(&normalized, &["how", "help", "what do i need"]); - let mentions_rocm_or_therock = any_substring(&normalized, &["therock", "rocm"]); - let requested_install_prefix = requested_install_prefix_from_prompt(prompt); - if requested_install_prefix.is_none() - && let Some(approval) = install_sdk_without_prefix_chat_approval(prompt) - { - return Some(providers::ChatToolCall { - id: Some("fallback-therock-install-folder".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": approval.args, - "reason": "Ask the user to choose the ROCm/TheRock install folder before installing." - }), - }); - } - if mentions_rocm_or_therock - && prompt_requests_install_action(&normalized) - && let Some(prefix) = requested_install_prefix - { - let mut args = vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--prefix".to_owned(), - prefix, - ]; - if let Some(build_date) = requested_therock_build_date_from_prompt(&normalized) { - args.push("--build-date".to_owned()); - args.push(build_date); - } else if let Some(version) = requested_therock_version_from_prompt(&normalized) { - args.push("--version".to_owned()); - args.push(version); - } - return Some(providers::ChatToolCall { - id: Some("fallback-therock-install".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": args, - "reason": "Install TheRock ROCm into the user-selected folder after the user approves it." - }), - }); - } - if asks_how && mentions_setup && any_substring(&normalized, &["therock", "rocm"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-therock-examine".to_owned()), - name: "examine".to_owned(), - arguments: serde_json::json!({}), - }); - } - - let asks_where_installed = any_substring( - &normalized, - &[ - "where is rocm", - "where's rocm", - "where is therock", - "where's therock", - "where did rocm", - "where did therock", - "where rocm is installed", - "where therock is installed", - "rocm install folder", - "therock install folder", - "rocm installed at", - "therock installed at", - ], - ); - let asks_status = asks_where_installed - || any_substring( - &normalized, - &[ - "is rocm installed", - "is therock installed", - "is therock setup", - "is therock set up", - "rocm installed", - "therock installed", - "check this rocm setup", - "which gpu", - "what gpu", - "gpu is on", - "gpu do i have", - "my machine", - ], - ); - if asks_status && any_substring(&normalized, &["gpu", "rocm", "therock", "setup"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-examine".to_owned()), - name: "examine".to_owned(), - arguments: serde_json::json!({}), - }); - } - - None -} - -fn prompt_asks_running_or_status(normalized: &str) -> bool { - any_substring( - normalized, - &[ - "running", - "is it up", - "is this up", - "is there", - "are there", - "status", - "started", - "listening", - "on port", - "port ", - ], - ) -} - -fn fallback_config_tool_call_for_prompt(normalized: &str) -> Option { - if !any_substring( - normalized, - &[ - "config", - "setting", - "settings", - "default engine", - "telemetry", - ], - ) { - return None; - } - if any_substring(normalized, &["show", "check", "what", "current", "list"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-config-show".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": ["config", "show"] }), - }); - } - if any_substring(normalized, &["default engine", "set engine", "use engine"]) { - // Word-boundary inference: a bare `contains("aim")` matched "aiming". - if let Some(engine) = infer_engine_from_request(normalized) { - return Some(providers::ChatToolCall { - id: Some("fallback-config-default-engine".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["config", "set-default-engine", engine], - "reason": "Change ROCm CLI's default engine after the user approves it." - }), - }); - } - } - if normalized.contains("telemetry") { - if any_substring(normalized, &["off", "disable", "disabled"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-config-telemetry-off".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["config", "set-telemetry", "off"], - "reason": "Turn ROCm CLI telemetry off after the user approves it." - }), - }); - } - if any_substring(normalized, &["local", "on", "enable", "enabled"]) { - return Some(providers::ChatToolCall { - id: Some("fallback-config-telemetry-local".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ - "args": ["config", "set-telemetry", "local"], - "reason": "Enable local-only ROCm inspection after the user approves it." - }), - }); - } - } - None -} - -fn install_sdk_without_prefix_chat_approval(prompt: &str) -> Option { - install_sdk_chat_approval_for_prompt(prompt).and_then(|approval| { - chat_cli_arg_value(&approval.args, "--prefix") - .is_none() - .then_some(approval) - }) -} - -fn install_sdk_chat_approval_for_prompt(prompt: &str) -> Option { - let prompt = latest_user_chat_message(prompt); - let normalized = prompt.to_ascii_lowercase(); - if !prompt_requests_rocm_install_or_setup(&normalized) { - return None; - } - let mut args = vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - if normalized.contains("nightly") { - "nightly".to_owned() - } else { - "release".to_owned() - }, - "--format".to_owned(), - "wheel".to_owned(), - ]; - if let Some(prefix) = requested_install_prefix_from_prompt(prompt) { - args.push("--prefix".to_owned()); - args.push(prefix); - } - if let Some(build_date) = requested_therock_build_date_from_prompt(&normalized) { - args.push("--build-date".to_owned()); - args.push(build_date); - } else if let Some(version) = requested_therock_version_from_prompt(&normalized) { - args.push("--version".to_owned()); - args.push(version); - } - Some(ChatToolApprovalRequest { - pending_title: "Install ROCm".to_owned(), - command_title: "Install".to_owned(), - display_command: Some(format_structured_tool_call("rocm", &args)), - args, - explanation: Some( - "Install ROCm/TheRock after the user chooses the install folder.".to_owned(), - ), - }) -} - -fn render_install_sdk_folder_needed_chat_text(prompt: &str, args: &[String]) -> String { - let mut output = String::new(); - let _ = writeln!(output, "ROCm install"); - let _ = writeln!(output); - let _ = writeln!(output, "I can install ROCm/TheRock for you."); - let _ = writeln!( - output, - "First choose the folder where ROCm CLI should put the Python environment." - ); - let _ = writeln!( - output, - "Nothing will download or change until you review and approve the install." - ); - if let Some(date) = chat_cli_arg_value(args, "--build-date") { - let _ = writeln!(output); - let _ = writeln!(output, "Requested build date: {date}"); - } - if let Some(version) = chat_cli_arg_value(args, "--version") { - let _ = writeln!(output); - let _ = writeln!(output, "Requested version: {version}"); - } - let _ = writeln!(output); - let _ = writeln!(output, "Your request"); - let _ = writeln!(output, " {}", prompt.trim()); - output.trim_end().to_owned() -} - -fn prompt_requests_rocm_install_or_setup(normalized: &str) -> bool { - let mentions_rocm_stack = any_substring( - normalized, - &[ - "rocm", - "therock", - "the rock", - "amd gpu package", - "amd gpu ready", - "local ai", - ], - ); - if !mentions_rocm_stack { - return false; - } - prompt_requests_install_action(normalized) - || any_substring( - normalized, - &[ - "need rocm", - "need therock", - "need the rock", - "need to get rocm", - "need to get therock", - "get rocm installed", - "get therock installed", - "get the rock installed", - "make rocm work", - "make therock work", - "make the rock work", - "prepare rocm", - "prepare therock", - "prepare the rock", - "set up my amd gpu", - "setup my amd gpu", - "set up local ai", - "setup local ai", - "get local ai working", - "make local ai work", - "rocm please", - "make my amd gpu ready", - "make my gpu ready", - "get my amd gpu ready", - "get my gpu ready", - ], - ) -} - -fn prompt_requests_install_action(normalized: &str) -> bool { - any_substring( - normalized, - &[ - "install this", - "install specific", - "install the", - "install rocm", - "install therock", - "install the rock", - "install amd gpu package", - "install local ai", - "can you install", - "please install", - "rocm please", - "therock please", - "the rock please", - "need to install", - "want to install", - "i need rocm", - "i need therock", - "i need the rock", - "get me rocm", - "get me therock", - "get my gpu ready", - "make my gpu ready", - "make my amd gpu ready", - "get installed", - "setup for me", - "set up for me", - "setup rocm", - "set up rocm", - "setup therock", - "set up therock", - "setup the rock", - "set up the rock", - "setup my gpu", - "set up my gpu", - "setup amd gpu", - "set up amd gpu", - ], - ) -} - -fn visible_chat_content(content: &str) -> String { - let mut output = String::new(); - let mut remaining = content; - while let Some(start) = find_ascii_case_insensitive(remaining, "") { - output.push_str(&remaining[..start]); - let after_start = &remaining[start + "".len()..]; - let Some(end) = find_ascii_case_insensitive(after_start, "") else { - remaining = ""; - break; - }; - remaining = &after_start[end + "".len()..]; - } - output.push_str(remaining); - output.trim().to_owned() -} - -fn find_ascii_case_insensitive(haystack: &str, needle: &str) -> Option { - let needle = needle.as_bytes(); - haystack - .as_bytes() - .windows(needle.len()) - .position(|window| window.eq_ignore_ascii_case(needle)) -} - -/// The host-independent half of the assistant prompt. -/// -/// Statements that are only true on *some* hosts do not belong here — they used -/// to, and a WSL user was told "on native Windows, vLLM is skipped" while vLLM -/// was in fact their supported path. Anything host-dependent now comes from -/// [`chat_host_facts::HostFacts`], which knows which machine it is describing. -const ROCM_CHAT_TOOL_SYSTEM_PROMPT: &str = "You are ROCm CLI's local assistant. Speak in simple English for non-technical users. Use the provided ROCm tools when you need to inspect this machine, preview setup, read service logs, check updates, inspect automations, install or start ROCm-managed apps, or request ROCm/TheRock, config, engine, app, and local model server changes. For simple greetings or thanks like hello, hi, hey, ok, or thank you, reply normally; do not inspect ROCm, do not call tools, and do not launch or propose a model server. Tool-use rules: inspect first with read-only tools; call rocm_command only with argv-style args and no shell text; use natural_language_plan for ROCm requests that do not fit another read-only tool; ask for a mutating tool call only after explaining why it is needed; summarize tool results after they are returned. Read-only tools may run immediately. Tools that install, launch, stop, delete, or change state require user approval; request rocm_command and explain why. For 'is X running?', 'what is running?', status, or port questions, inspect before answering and do not start, stop, install, or serve anything. For ComfyUI or port 8188 use [\"comfyui\",\"status\"] or port_status. For vLLM, Lemonade, AIM, qwen, or local model servers use [\"services\",\"list\",\"--all\"] for running state and [\"engines\",\"list\"] for installed/available engine state. Treat ready/running as running, starting/recovering as starting, failed/stopped as not running, and no matching record as unknown or not managed by ROCm CLI. Interpret Examine carefully: active_runtime_status=ready means ROCm CLI has an active managed TheRock/ROCm runtime; legacy_rocm_status=not_detected only means no global system ROCm install was found. If active_runtime_status=ready, tell the user ROCm/TheRock is installed and active for ROCm CLI. For 'is TheRock installed', 'is ROCm installed', or 'which GPU is on this machine', use examine or gpu_snapshot before answering. For 'how do I setup TheRock' or install/setup requests, guide the user to choose an install folder first; do not answer with only a status check. For 'which LLMs can this machine support', use rocm_command args [\"model\"] or natural_language_plan before answering. For TheRock installs, always let the user choose the install folder. If the user names a folder or prefix, preserve that exact folder with [\"--prefix\",\"PATH\"]; you may call path_exists first to check whether that user-provided folder or its parent exists. If the user asks you to install TheRock/ROCm but has not named a folder, ask for the folder or let the guided setup folder picker collect it; do not invent a hidden default folder and do not request an install command without --prefix. Use rocm_command args [\"install\",\"sdk\",\"--channel\",\"release\",\"--format\",\"wheel\",\"--prefix\",\"PATH\"] only when the user asks you to install it and a folder is known; for a requested build date add [\"--build-date\",\"YYYY-MM-DD\"] and for a requested exact version add [\"--version\",\"VERSION\"]. For config changes, inspect with [\"config\",\"show\"] first when useful, then request config subcommands such as [\"config\",\"set-default-engine\",\"lemonade\"], [\"config\",\"set-default-runtime\",\"RUNTIME_KEY\"], or [\"config\",\"set-telemetry\",\"local\"] only after explaining why. For ComfyUI, use rocm_command with args like [\"comfyui\",\"status\"], [\"comfyui\",\"logs\"], [\"comfyui\",\"install\"], [\"comfyui\",\"start\"], or [\"comfyui\",\"stop\"]. First-time setup is the same thing as bootstrap in ROCm CLI; it is a deterministic ROCm setup flow, not a separate model chat. The built-in local assistant is fixed to qwen, which maps to Qwen3-4B-Instruct-2507-GGUF served by Lemonade with gpu_required. vLLM, Lemonade, and AIM (Docker-based) are the general serving engines; inspect or manage them when the user asks about general model serving, but do not switch the built-in assistant away from Lemonade. Use qwen-smoke only for a quick server smoke test. For vLLM management, inspect engines first and use [\"engines\",\"install\",\"vllm\"] or [\"serve\",\"MODEL\",\"--engine\",\"vllm\",\"--device\",\"gpu_required\",\"--managed\"] only where the host supports it. Do not invent shell commands and do not request CPU fallback."; -const ROCM_CHAT_TOOL_SKILL: &str = include_str!("../../../skills/rocm-cli-assistant/SKILL.md"); - -fn rocm_chat_tool_system_prompt() -> String { - format!("{ROCM_CHAT_TOOL_SYSTEM_PROMPT}\n\nROCm CLI assistant skill:\n{ROCM_CHAT_TOOL_SKILL}") -} - -/// The assistant prompt grounded in the machine it will answer for. -/// -/// The single composition point: both chat surfaces (`rocm chat --prompt -/// --tools` in this bin, and the dashboard chat via -/// [`crate::dash::resolved_args`]) send this, so neither can drift into -/// answering platform questions from pretraining alone. -pub(crate) fn rocm_chat_tool_system_prompt_for_host(paths: Option<&AppPaths>) -> String { - let facts = chat_host_facts::HostFacts::detect(paths); - format!("{}\n\n{}", rocm_chat_tool_system_prompt(), facts.render()) -} - -fn local_provider_missing_service_error(error: &anyhow::Error) -> bool { - error.chain().any(|cause| { - cause - .to_string() - .contains("local provider has no ready managed service") - }) -} - -fn local_rocm_tools_assistant_model(provider: &str, rocm_tools: bool) -> Option<&'static str> { - (provider == "local" && rocm_tools).then_some(providers::BUILTIN_ASSISTANT_MODEL_ID) -} - -fn local_chat_service_needed_text(model: Option<&str>, prompt: &str, rocm_tools: bool) -> String { - let mut output = String::new(); - let _ = writeln!(output, "No local assistant is running yet."); - let _ = writeln!(output); - let _ = writeln!( - output, - "To use LLM-assisted ROCm commands, start a ROCm GPU local model server first." - ); - let _ = writeln!( - output, - "First-time ROCm setup does not need an LLM; run `rocm` and use Set Up ROCm for that." - ); - let _ = writeln!(output); - let _ = writeln!( - output, - "Recommended path:\n run `rocm`, choose Start a local model, use the recommended assistant model, then start it" - ); - let _ = writeln!(output); - if let Some(model) = model.filter(|value| !value.trim().is_empty()) { - let args = vec![ - "serve".to_owned(), - model.to_owned(), - "--device".to_owned(), - "gpu_required".to_owned(), - "--managed".to_owned(), - ]; - let _ = writeln!( - output, - "Advanced manual command for the selected model:\n {}", - format_structured_tool_call("rocm", &args) - ); - } else { - let example_args = vec![ - "serve".to_owned(), - providers::LEMONADE_ASSISTANT_MODEL_ID.to_owned(), - "--engine".to_owned(), - "lemonade".to_owned(), - "--device".to_owned(), - "gpu_required".to_owned(), - "--managed".to_owned(), - ]; - let _ = writeln!( - output, - "Advanced manual command for the recommended assistant model:\n {}", - format_structured_tool_call("rocm", &example_args) - ); - } - let mut chat_args = vec!["chat".to_owned()]; - if rocm_tools { - chat_args.push("--tools".to_owned()); - } - chat_args.push("--provider".to_owned()); - chat_args.push("local".to_owned()); - if let Some(model) = model.filter(|value| !value.trim().is_empty()) { - chat_args.push("--model".to_owned()); - chat_args.push(model.to_owned()); - } - if !prompt.trim().is_empty() { - chat_args.push("--prompt".to_owned()); - chat_args.push(prompt.to_owned()); - } - let _ = writeln!(output); - let _ = writeln!( - output, - "After the server is ready, run:\n {}", - format_structured_tool_call("rocm", &chat_args) - ); - let _ = writeln!(output); - let _ = writeln!( - output, - "For the guided flow, run `rocm`, choose Start a local model, start a model, then choose Chat." - ); - let _ = writeln!(output, "Nothing was changed."); - output.trim_end().to_owned() -} - -fn append_chat_tool_results( - paths: &AppPaths, - response: &providers::ChatResponse, - output: &mut String, - assistant_explanation: Option<&str>, - progress: &mut Option<&mut dyn FnMut(String)>, -) -> Result { - if response.tool_calls.is_empty() { - let _ = writeln!(output, "ROCm checks used"); - let _ = writeln!(output, " none requested"); - return Ok(ChatToolRunResult { - approval: None, - follow_up_text: String::new(), - ran_read_only_tool: false, - read_only_tool_error: false, - needs_install_folder: false, - }); - } - - let _ = writeln!(output, "ROCm checks used"); - let mut approval = None; - let mut follow_up_text = String::new(); - let mut ran_read_only_tool = false; - let mut read_only_tool_error = false; - let mut needs_install_folder = false; - for call in &response.tool_calls { - if chat_tool_call_requests_therock_install_without_prefix(call) { - needs_install_folder = true; - report_chat_tool_progress(progress, "Waiting for the ROCm install folder."); - let _ = writeln!(output, " Install ROCm: needs install folder"); - let _ = writeln!( - output, - " not run: choose an install folder before the review card" - ); - continue; - } - validate_chat_tool_call(call)?; - let label = chat_tool_call_display_label(call); - if chat_tool_call_is_read_only(call) { - report_chat_tool_progress(progress, &format!("Running ROCm check: {label}.")); - let result = run_chat_read_only_tool(paths, call)?; - ran_read_only_tool = true; - let is_error = mcp_tool_result_is_error(&result); - read_only_tool_error |= is_error; - report_chat_tool_progress( - progress, - &format!( - "ROCm check finished: {label} ({})", - chat_read_only_tool_status_label(is_error) - ), - ); - let _ = writeln!( - output, - " {}: {}", - label, - chat_read_only_tool_status_label(is_error) - ); - let result_text = mcp_tool_result_text(&result); - let _ = writeln!(follow_up_text, "{}:", call.name); - let _ = writeln!(follow_up_text, "{result_text}"); - for line in result_text.lines() { - let _ = writeln!(output, " {line}"); - } - } else { - report_chat_tool_progress(progress, &format!("Review needed: {label}.")); - let _ = writeln!(output, " {label}: needs your review"); - let _ = writeln!( - output, - " not run: review the approval card before anything runs" - ); - if let Some(command) = rocm_chat_tool_requested_command(call) { - let _ = writeln!(output, " advanced manual command: {command}"); - } - if approval.is_none() { - approval = Some(chat_tool_approval_request(call, assistant_explanation)?); - } - } - } - Ok(ChatToolRunResult { - approval, - follow_up_text, - ran_read_only_tool, - read_only_tool_error, - needs_install_folder, - }) -} - -fn chat_tool_call_requests_therock_install_without_prefix(call: &providers::ChatToolCall) -> bool { - let Some(object) = call.arguments.as_object() else { - return false; - }; - match call.name.as_str() { - "install_sdk" => json_string(object, "prefix").is_none(), - "rocm_command" => rocm_command_args_install_sdk_without_prefix(object), - _ => false, - } -} - -fn rocm_command_args_install_sdk_without_prefix( - object: &serde_json::Map, -) -> bool { - let Some(args) = object.get("args").and_then(serde_json::Value::as_array) else { - return false; - }; - let args = args - .iter() - .filter_map(serde_json::Value::as_str) - .map(str::to_owned) - .collect::>(); - args.first() - .is_some_and(|arg| arg.eq_ignore_ascii_case("install")) - && args - .get(1) - .is_some_and(|arg| arg.eq_ignore_ascii_case("sdk")) - && chat_cli_arg_value(&args, "--prefix").is_none() -} - -fn managed_service_running_state(status: &str) -> &'static str { - match status { - "ready" | "running" => "running", - "starting" | "recovering" => "starting", - "failed" | "stopped" => "not_running", - _ => "unknown", - } -} - -fn render_services_tool_result_text(records: &[ManagedServiceRecord]) -> String { - let mut output = String::new(); - let _ = writeln!(output, "managed_services: {}", records.len()); - let _ = writeln!( - output, - "status_meaning: ready/running = running; starting/recovering = starting; failed/stopped = not running; no matching row = not managed by ROCm CLI" - ); - if records.is_empty() { - let _ = writeln!(output, "services: none"); - return output; - } - let _ = writeln!(output, "services:"); - for record in records { - let _ = writeln!( - output, - " - service_id={} engine={} model={} canonical_model={} status={} running_state={} endpoint={}", - record.service_id, - record.engine, - record.model_ref, - record.canonical_model_id, - record.status, - managed_service_running_state(&record.status), - record.endpoint_url - ); - } - output -} - -fn deterministic_rocm_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("examine:") { - return None; - } - let mut lines = Vec::new(); - let gpu = chat_tool_value(tool_text, "driver_detail") - .filter(|value| value != "") - .or_else(|| { - let target = chat_tool_value(tool_text, "detected_gfx_target")?; - Some(format!("AMD GPU target {target}")) - }); - if let Some(gpu) = gpu { - lines.push(format!(" GPU: {gpu}")); - } - - let active_status = chat_tool_value(tool_text, "active_runtime_status"); - if active_status.as_deref() == Some("ready") { - let family = chat_tool_value(tool_text, "active_runtime_family") - .filter(|value| value != "" && value != ""); - let version = chat_tool_value(tool_text, "active_runtime_version") - .filter(|value| value != "" && value != ""); - let mut detail = "installed and active for ROCm CLI".to_owned(); - if let Some(family) = family { - let _ = write!(detail, " ({family})"); - } - if let Some(version) = version { - let _ = write!(detail, ", {version}"); - } - lines.push(format!(" ROCm/TheRock: {detail}")); - if let Some(root) = chat_tool_value(tool_text, "active_runtime_root") - .or_else(|| chat_tool_value(tool_text, "setup_runtime_root")) - .filter(|value| value != "" && value != "" && value != "") - { - lines.push(format!(" Install folder: {root}")); - } - if let Some(cache) = chat_tool_value(tool_text, "active_runtime_pip_cache_dir") - .or_else(|| chat_tool_value(tool_text, "setup_runtime_pip_cache_dir")) - .filter(|value| value != "" && value != "" && value != "") - { - lines.push(format!(" Downloads/cache: {cache}")); - } - } else if let Some(status) = active_status.as_deref() { - lines.push(format!(" ROCm/TheRock: active runtime status is {status}")); - if let Some(root) = chat_tool_value(tool_text, "setup_runtime_root") - .filter(|value| value != "" && value != "" && value != "") - { - lines.push(format!(" Selected setup folder: {root}")); - } - if let Some(cache) = chat_tool_value(tool_text, "setup_runtime_pip_cache_dir") - .filter(|value| value != "" && value != "" && value != "") - { - lines.push(format!(" Downloads/cache: {cache}")); - } - } - - if chat_tool_value(tool_text, "legacy_rocm_status").as_deref() == Some("not_detected") - && active_status.as_deref() == Some("ready") - { - lines.push( - " Note: no global legacy ROCm install was found; ROCm CLI is using its managed TheRock runtime." - .to_owned(), - ); - } - - (!lines.is_empty()).then(|| lines.join("\n")) -} - -fn deterministic_chat_tool_summary(tool_text: &str) -> Option { - deterministic_rocm_tool_summary(tool_text) - .or_else(|| deterministic_model_tool_summary(tool_text)) - .or_else(|| deterministic_combined_status_tool_summary(tool_text)) -} - -fn deterministic_combined_status_tool_summary(tool_text: &str) -> Option { - let summaries = [ - deterministic_comfyui_tool_summary(tool_text), - deterministic_port_status_tool_summary(tool_text), - deterministic_engine_inventory_tool_summary(tool_text), - deterministic_services_tool_summary(tool_text), - ] - .into_iter() - .flatten() - .collect::>(); - (!summaries.is_empty()).then(|| summaries.join("\n")) -} - -fn deterministic_summary_can_stand_alone(tool_text: &str) -> bool { - !tool_text.contains("model recipes") -} - -fn deterministic_model_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("model recipes") { - return None; - } - let recipes = parse_deterministic_model_recipes(tool_text); - if recipes.is_empty() { - return None; - } - - let mut lines = Vec::new(); - if let Some(recipe) = recipes - .iter() - .find(|recipe| recipe.canonical_id == providers::LEMONADE_ASSISTANT_MODEL_ID) - { - lines.push(format!( - " Recommended local assistant: qwen ({})", - recipe.canonical_id - )); - lines.push(format!( - " Fit: needs about {} GPU memory.", - recipe - .min_gpu_mem_gib - .map_or_else(|| "unknown".to_owned(), |value| format!("{value} GiB")) - )); - lines.push(format!( - " Engine: {}", - deterministic_engine_fit_summary(recipe, "lemonade") - )); - } - - if let Some(recipe) = recipes - .iter() - .find(|recipe| recipe.canonical_id == "Qwen3-0.6B-GGUF") - { - lines.push(format!( - " Tiny smoke test: qwen-smoke ({})", - recipe.canonical_id - )); - lines.push( - " Use this only to check that GPU serving starts; it is not the default assistant." - .to_owned(), - ); - } - - if let Some(recipe) = recipes - .iter() - .find(|recipe| recipe.canonical_id == "meta-llama/Llama-3.2-3B-Instruct") - { - lines.push(format!( - " 8 GiB-class option: llama ({})", - recipe.canonical_id - )); - lines.push(format!( - " Fit: asks for {}; this can be tight on APUs and depends on available shared GPU memory.", - recipe - .min_gpu_mem_gib.map_or_else(|| "unknown GPU memory".to_owned(), |value| format!("{value} GiB")) - )); - lines.push(format!( - " Engines: {}", - deterministic_available_engine_names(recipe).join(", ") - )); - } - - let larger = recipes - .iter() - .filter(|recipe| recipe.min_gpu_mem_gib.is_some_and(|value| value > 8)) - .filter(|recipe| recipe.canonical_id != providers::BUILTIN_ASSISTANT_MODEL_ID) - .map(|recipe| { - format!( - "{} asks for {}", - recipe.canonical_id, - recipe.min_gpu_mem_gib.map_or_else( - || "more GPU memory".to_owned(), - |value| format!("{value} GiB") - ) - ) - }) - .collect::>(); - if !larger.is_empty() { - lines.push(format!( - " Larger recipes are not low-VRAM defaults: {}.", - larger.join("; ") - )); - } - - let linux_wsl_only = recipes - .iter() - .flat_map(|recipe| { - recipe - .engine_statuses - .iter() - .filter(|status| { - status.contains("unsupported_native_windows") - || status.to_ascii_lowercase().contains("wsl/linux") - || status.to_ascii_lowercase().contains("linux/wsl") - }) - .filter_map(|status| { - let (engine, _) = status.split_once(':')?; - Some(format!( - "{} uses {} through WSL/Linux on Windows", - recipe.canonical_id, engine - )) - }) - }) - .collect::>(); - if !linux_wsl_only.is_empty() { - lines.push(format!( - " Native Windows note: {}.", - linux_wsl_only.join("; ") - )); - } - - if !lines.is_empty() { - lines.push( - " Run `rocm examine` to refresh GPU memory details before starting anything large." - .to_owned(), - ); - } - (!lines.is_empty()).then(|| lines.join("\n")) -} - -fn deterministic_engine_fit_summary(recipe: &DeterministicModelRecipe, engine: &str) -> String { - let status = recipe - .engine_statuses - .iter() - .find(|status| status.starts_with(&format!("{engine}:"))) - .map(std::string::String::as_str); - match status { - Some(status) if status.contains("unsupported_native_windows") => { - format!("{engine}: WSL/Linux only on Windows") - } - Some(status) - if status.split_once(':').is_some_and(|(_, rest)| { - rest.trim_start().starts_with("available") - || rest.trim_start().starts_with("adapter_available") - }) => - { - format!("{engine}: available") - } - Some(status) => status.to_owned(), - None if recipe.engines.iter().any(|candidate| candidate == engine) => { - format!("{engine}: listed") - } - None => format!("{engine}: not listed"), - } -} - -fn deterministic_available_engine_names(recipe: &DeterministicModelRecipe) -> Vec { - let engines = recipe - .engine_statuses - .iter() - .filter_map(|status| { - let (engine, rest) = status.split_once(':')?; - (rest.trim_start().starts_with("available") - || rest.trim_start().starts_with("adapter_available")) - .then_some(engine.to_owned()) - }) - .collect::>(); - if engines.is_empty() { - recipe.engines.clone() - } else { - engines - } -} - -pub(crate) fn deterministic_comfyui_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("ComfyUI") || !tool_text.contains("Running") { - return None; - } - let mut lines = Vec::new(); - match chat_tool_value(tool_text, "installed").as_deref() { - Some("yes") => lines.push(" ComfyUI: installed.".to_owned()), - Some("no") => lines.push(" ComfyUI: not installed.".to_owned()), - Some(value) => lines.push(format!(" ComfyUI: installed status is {value}.")), - None => {} - } - if let Some(status) = chat_tool_value(tool_text, "status") { - let running_text = if status.eq_ignore_ascii_case("running") { - "running".to_owned() - } else if status.eq_ignore_ascii_case("starting") { - "starting".to_owned() - } else { - format!("not running ({status})") - }; - lines.push(format!(" Running: {running_text}.")); - } - if let Some(models_path) = chat_tool_value(tool_text, "models path") { - lines.push(format!(" Models folder: {models_path}.")); - } - // A note about the install itself (an interrupted reinstall, which names - // the command that finishes it) sits between `status`'s `ComfyUI` heading - // and its `Running` section, whose own notes the `Running:` line sums up. - for line in tool_text - .lines() - .skip_while(|line| line.trim() != "ComfyUI") - .take_while(|line| line.trim() != "Running") - { - if let Some(note) = line.trim().strip_prefix("note:").map(str::trim) - && !note.is_empty() - { - lines.push(format!(" Note: {note}.")); - } - } - (!lines.is_empty()).then(|| lines.join("\n")) -} - -fn deterministic_services_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("Local Servers") - && !tool_text.contains("managed_services:") - && !tool_text.contains("services:") - { - return None; - } - let rows = parse_deterministic_service_rows(tool_text); - let mut lines = Vec::new(); - let live = rows - .iter() - .filter(|row| matches!(row.running_state.as_str(), "running" | "starting")) - .collect::>(); - if live.is_empty() { - lines.push(" Local model servers: none running under ROCm CLI.".to_owned()); - } else { - lines.push(format!( - " Local model servers running/starting: {}.", - live.len() - )); - for row in live.iter().take(4) { - lines.push(format!( - " {} {} ({}) at {}.", - row.engine, - row.model, - row.running_state, - empty_as_unknown(&row.endpoint) - )); - } - } - let past = rows - .iter() - .filter(|row| !matches!(row.running_state.as_str(), "running" | "starting")) - .take(3) - .map(|row| { - format!( - "{} {} {}", - empty_as_unknown(&row.engine), - empty_as_unknown(&row.model), - empty_as_unknown(&row.status) - ) - }) - .collect::>(); - if !past.is_empty() { - lines.push(format!(" Past/non-running records: {}.", past.join("; "))); - } - (!lines.is_empty()).then(|| lines.join("\n")) -} - -fn deterministic_port_status_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("port_status:") && !tool_text.contains("listening:") { - return None; - } - let port = chat_tool_value(tool_text, "port")?; - let listening = chat_tool_value(tool_text, "listening").unwrap_or_else(|| "unknown".to_owned()); - let mut lines = Vec::new(); - let state = match listening.as_str() { - "true" => "listening", - "false" => "not listening", - _ => "unknown", - }; - lines.push(format!(" Port {port}: {state}.")); - if let Some(hint) = chat_tool_value(tool_text, "hint") { - lines.push(format!(" Hint: {hint}.")); - } - if chat_tool_value(tool_text, "managed_service").as_deref() == Some("none") { - lines.push(" Managed service: none on that endpoint.".to_owned()); - } - Some(lines.join("\n")) -} - -fn deterministic_engine_inventory_tool_summary(tool_text: &str) -> Option { - if !tool_text.contains("Local model engines") { - return None; - } - let rows = parse_deterministic_engine_rows(tool_text); - if rows.is_empty() { - return None; - } - let mut lines = vec![" Engine runtimes:".to_owned()]; - for row in rows { - let mut line = format!( - " {}: {}", - friendly_engine_label(&row.engine), - empty_as_unknown(&row.runtime) - ); - if !row.note.trim().is_empty() { - let _ = write!(line, " ({})", row.note.trim()); - } - line.push('.'); - lines.push(line); - } - Some(lines.join("\n")) -} - -fn should_request_local_tool_follow_up( - provider: &str, - tool_result: &ChatToolRunResult, - deterministic_summary: Option<&str>, -) -> bool { - provider == "local" - && deterministic_summary.is_none() - && tool_result.approval.is_none() - && tool_result.ran_read_only_tool - && !tool_result.read_only_tool_error - && !tool_result.follow_up_text.trim().is_empty() -} - -fn chat_tool_value(tool_text: &str, key: &str) -> Option { - let prefix = format!("{key}:"); - tool_text.lines().find_map(|line| { - line.trim() - .strip_prefix(&prefix) - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(str::to_owned) - }) -} - -// pub(crate): called from main.rs's own MCP dispatch and from the dash -// execution seam (dash_seam.rs) -pub(crate) fn run_internal_mcp_call( - paths: &AppPaths, - name: &str, - arguments: serde_json::Value, - allow_mutation: bool, -) -> Result { - let arguments = internal_mcp_arguments(arguments); - let call = providers::ChatToolCall { - id: None, - name: name.to_owned(), - arguments: serde_json::Value::Object(arguments.clone()), - }; - validate_chat_tool_call(&call)?; - if !chat_tool_call_is_read_only(&call) && !allow_mutation { - bail!( - "MCP tool `{name}` changes local ROCm state; rerun `rocm mcp-call {name}` with --allow-mutation only after explicit user approval" - ); - } - - match name { - "examine" | "doctor" => { - let examine = ExamineSummary::gather()?; - let text = render_examine_text()?; - Ok(internal_mcp_tool_success(text, serde_json::json!(examine))) - } - "bridge_snapshot" => { - let snapshot = build_codex_bridge_snapshot(paths)?; - Ok(internal_mcp_tool_success( - format!( - "Captured bridge snapshot for {} / {} with default engine `{}`.", - snapshot.examine.os, snapshot.examine.arch, snapshot.examine.default_engine - ), - serde_json::json!(snapshot), - )) - } - "gpu_snapshot" => { - let config = RocmCliConfig::load(paths).unwrap_or_default(); - let gpu = build_codex_bridge_gpu_snapshot(&config); - let status = if !config.telemetry.local_inspection_enabled() { - "GPU telemetry is disabled by rocm-cli config." - } else if gpu.amd_smi_available { - "Captured amd-smi GPU snapshot." - } else { - "Use `rocm examine` for the current local AMD GPU summary." - }; - Ok(internal_mcp_tool_success( - status.to_owned(), - serde_json::json!(gpu), - )) - } - "engines" => { - let engines = builtin_codex_bridge_engine_inventory(); - Ok(internal_mcp_tool_success( - format!("Found {} engine entries.", engines.len()), - serde_json::json!({ "engines": engines }), - )) - } - "services" => { - let services = load_managed_services(paths)?; - Ok(internal_mcp_tool_success( - render_services_tool_result_text(&services), - serde_json::json!({ "services": services }), - )) - } - "port_status" => run_chat_port_status_tool(paths, &call), - "service_logs" => { - let service_id = json_string(&arguments, "service_id") - .context("service_logs requires `service_id`")?; - let lines = arguments - .get("lines") - .and_then(serde_json::Value::as_u64) - .unwrap_or(80) - .clamp(1, 500) as usize; - let record = load_managed_services(paths)? - .into_iter() - .find(|service| service.service_id == service_id) - .with_context(|| format!("managed service `{service_id}` not found"))?; - let tail = read_tail_lines(&record.log_path, lines, "service log")?.join("\n"); - Ok(internal_mcp_tool_success( - format!( - "Read the last {} line(s) from service `{}`.", - lines, record.service_id - ), - serde_json::json!({ - "service": record, - "lines": lines, - "tail": tail, - }), - )) - } - "automations" => { - let event_limit = arguments - .get("event_limit") - .and_then(serde_json::Value::as_u64) - .unwrap_or(10) - .clamp(1, 64) as usize; - let runtime = AutomationRuntimeState::load(paths)?; - let events = load_recent_automation_events(paths, event_limit)?; - Ok(internal_mcp_tool_success( - format!( - "Loaded automation runtime and {} recent events.", - events.len() - ), - serde_json::json!({ - "runtime": runtime, - "recent_events": events, - }), - )) - } - "natural_language_plan" => { - let request = json_string(&arguments, "request") - .context("natural_language_plan requires `request`")?; - let config = RocmCliConfig::load(paths).unwrap_or_default(); - // Built exactly once: the text shown to the operator and the - // `action` reported as what would run next both come from the - // same `StructuredRequestPlan`, so a provider-assisted plan (not - // guaranteed deterministic across calls) cannot describe one - // command in `text` and a different one in `action`. - let (text, action) = plan_freeform_request_with_context(&request, paths, &config); - let action = action.map(|action| { - serde_json::json!({ - "args": action.args, - "approval_required": action.approval_required, - "has_placeholders": action.has_placeholders, - "provider_assisted": action.provider_assisted, - }) - }); - Ok(internal_mcp_tool_success( - "Planned the ROCm request.".to_owned(), - serde_json::json!({ - "request": request, - "text": text, - "action": action, - }), - )) - } - "rocm_command" => { - let action = chat_rocm_command_action(&call)?; - let output = match action { - ChatRocmCommandAction::ReadOnly(args) => { - run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))? - } - ChatRocmCommandAction::Approval { args, .. } if allow_mutation => { - let refs = args.iter().map(String::as_str).collect::>(); - run_rocm_capture_for_paths(paths, &refs, Duration::from_mins(2))? - } - ChatRocmCommandAction::Approval { .. } => { - bail!("rocm_command changes local ROCm state and needs approval") - } - }; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm` command.", - output, - false, - )) - } - "update_check" => { - let output = - run_rocm_command_for_paths(paths, &["update".to_owned()], Duration::from_mins(1))?; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm update`.", - output, - false, - )) - } - "install_sdk_dry_run" => { - let args = internal_mcp_install_sdk_args(&arguments, true)?; - let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm install sdk --dry-run`.", - output, - false, - )) - } - "path_exists" => run_chat_path_exists_tool(&call), - "proposal_action" => { - // Executes IN-PROCESS (no CLI subprocess): show loads a proposal; - // approve/reject mutate its status (allow_mutation already enforced - // by the read-only split above). proposal_action executes in-process - // (via update_automation_proposal_status); it never delegates to the - // subprocess arm. - let proposal_id = json_string(&arguments, "proposal_id") - .context("proposal_action requires `proposal_id`")?; - let action = - json_string(&arguments, "action").context("proposal_action requires `action`")?; - match action.as_str() { - "show" => { - let proposal = rocm_core::find_automation_proposal(paths, &proposal_id) - .with_context(|| { - format!("automation proposal `{proposal_id}` not found") - })?; - Ok(internal_mcp_tool_success( - format!( - "Proposal {} ({}): {}", - proposal.proposal_id, proposal.status, proposal.title - ), - serde_json::json!({ - "id": proposal.proposal_id, - "status": proposal.status, - "summary": proposal.title, - "reason": proposal.message, - }), - )) - } - "approve" | "reject" => { - let status = if action == "approve" { - "approved" - } else { - "rejected" - }; - let updated = - rocm_core::update_automation_proposal_status(paths, &proposal_id, status)?; - record_cli_audit_event( - paths, - "automations", - if action == "approve" { - "proposal_approved" - } else { - "proposal_rejected" - }, - "info", - format!("proposal {proposal_id} {status}"), - None, - ); - Ok(internal_mcp_tool_success( - format!("Proposal {proposal_id} {status}."), - serde_json::json!({ - "id": updated.proposal_id, - "status": updated.status, - "summary": updated.title, - "reason": updated.message, - }), - )) - } - other => bail!("proposal_action `{other}` is not supported"), - } - } - "install_sdk" | "install_engine" | "launch_server" | "stop_server" | "watcher_enable" - | "watcher_disable" => { - let args = rocm_chat_tool_requested_args(&call) - .with_context(|| format!("MCP tool `{name}` is missing required arguments"))?; - let refs = args.iter().map(String::as_str).collect::>(); - let output = run_rocm_capture_for_paths(paths, &refs, Duration::from_mins(2))?; - Ok(internal_mcp_tool_result_from_command( - "Ran approved `rocm` command.", - output, - false, - )) - } - other => bail!("unsupported MCP tool `{other}`"), - } -} - -fn internal_mcp_arguments(value: serde_json::Value) -> serde_json::Map { - if let Some(arguments) = value - .get("arguments") - .and_then(serde_json::Value::as_object) - { - return arguments.clone(); - } - value.as_object().cloned().unwrap_or_default() -} - -fn internal_mcp_tool_success(text: String, structured: serde_json::Value) -> serde_json::Value { - serde_json::json!({ - "content": [ - { - "type": "text", - "text": text, - } - ], - "structuredContent": structured, - "isError": false, - }) -} - -fn internal_mcp_tool_result_from_command( - prefix: &str, - output: CommandCapture, - is_error: bool, -) -> serde_json::Value { - let text = format!("{prefix}\n\n{}", command_capture_text(&output)); - serde_json::json!({ - "content": [ - { - "type": "text", - "text": text, - } - ], - "structuredContent": { - "argv": output.argv, - "exit_status": output.exit_status, - "stdout": output.stdout, - "stderr": output.stderr, - }, - "isError": is_error || output.exit_status != 0, - }) -} - -fn run_rocm_capture_for_paths( - paths: &AppPaths, - args: &[&str], - timeout: Duration, -) -> Result { - rocm_core::run_rocm_capture_with_env(args, &app_path_env_vars(paths), timeout) -} - -fn run_rocm_command_for_paths( - paths: &AppPaths, - args: &[String], - _timeout: Duration, -) -> Result { - match run_rocm_read_only_in_process(paths, args) { - Ok(stdout) => Ok(CommandCapture { - argv: std::iter::once("rocm".to_owned()) - .chain(args.iter().cloned()) - .collect(), - exit_status: 0, - stdout, - stderr: String::new(), - }), - Err(error) => Err(error).with_context(|| { - format!( - "read-only assistant command is not implemented in-process: {}", - format_structured_tool_call("rocm", args) - ) - }), - } -} - -fn run_rocm_read_only_in_process(paths: &AppPaths, args: &[String]) -> Result { - let config = RocmCliConfig::load(paths).unwrap_or_default(); - match args { - [] => bail!("rocm command requires at least one argument"), - [command] if command.eq_ignore_ascii_case("examine") => { - render_examine_text_with_paths(paths, &config) - } - [command] - if command.eq_ignore_ascii_case("version") - || command == "--version" - || command == "-V" => - { - Ok(format!("rocm-cli {}\n", cli_version_string())) - } - [command] - if command.eq_ignore_ascii_case("model") || command.eq_ignore_ascii_case("models") => - { - Ok(render_model_registry_verbose_text_with_context_and_host( - Some(paths), - None, - None, - )) - } - [command] if command.eq_ignore_ascii_case("daemon") => { - Ok(render_daemon_text(paths, &config)) - } - [command] if command.eq_ignore_ascii_case("logs") => Ok(render_logs_text(paths)), - [command, rest @ ..] if command.eq_ignore_ascii_case("logs") => { - let query = parse_optional_query(rest)?; - Ok(render_logs_browser_text(paths, query.as_deref())) - } - [command] if command.eq_ignore_ascii_case("runtimes") => { - render_runtimes_text(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("runtimes") - && subcommand.eq_ignore_ascii_case("list") => - { - render_runtimes_text(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("engines") - && subcommand.eq_ignore_ascii_case("list") => - { - Ok(render_engine_inventory_text_with_paths(Some(paths))) - } - [command] if command.eq_ignore_ascii_case("services") => render_services_text(paths, false), - [command, subcommand] - if command.eq_ignore_ascii_case("services") - && subcommand.eq_ignore_ascii_case("list") => - { - render_services_text(paths, false) - } - [command, subcommand, flag] - if command.eq_ignore_ascii_case("services") - && subcommand.eq_ignore_ascii_case("list") - && matches!(flag.as_str(), "-a" | "--all") => - { - render_services_text(paths, true) - } - [command, subcommand, service_id] - if command.eq_ignore_ascii_case("services") - && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => - { - render_service_logs_text(paths, service_id) - } - [command] if command.eq_ignore_ascii_case("automations") => { - render_automations_text(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("automations") - && subcommand.eq_ignore_ascii_case("list") => - { - render_automations_text(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("config") - && subcommand.eq_ignore_ascii_case("show") => - { - Ok(render_config_text(paths, &config)) - } - [command] if command.eq_ignore_ascii_case("comfyui") => { - comfyui::render_status(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("comfyui") - && subcommand.eq_ignore_ascii_case("status") => - { - comfyui::render_status(paths, &config) - } - [command, subcommand, rest @ ..] - if command.eq_ignore_ascii_case("comfyui") - && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => - { - comfyui::render_logs( - paths, - parse_optional_lines(rest).unwrap_or(DEFAULT_LOG_TAIL_LINES), - ) - } - [command, rest @ ..] - if command.eq_ignore_ascii_case("update") - && !rest.iter().any(|arg| arg.eq_ignore_ascii_case("--apply")) => - { - render_update_text(paths) - } - [command, subcommand, rest @ ..] - if command.eq_ignore_ascii_case("install") - && subcommand.eq_ignore_ascii_case("sdk") - && rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dry-run")) => - { - render_install_sdk_dry_run_for_args(paths, rest) - } - [command, rest @ ..] - if command.eq_ignore_ascii_case("uninstall") - && rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dry-run")) => - { - render_uninstall_dry_run(paths) - } - [command] if command.eq_ignore_ascii_case("setup") => { - render_setup_status_text(paths, &config) - } - [command, subcommand] - if command.eq_ignore_ascii_case("setup") - && subcommand.eq_ignore_ascii_case("status") => - { - render_setup_status_text(paths, &config) - } - // `install driver` is a read-only call where the command only reports - // (`chat_install_driver_action`). It runs as a dry run that never - // executes anything, so a call that reaches it elsewhere cannot change - // the host either. `--dry-run` is accepted because it is what this arm - // already does; `--reconcile` is not, since it rewrites the saved driver - // install state and the command refuses to combine it with the other - // flags. - [command, subcommand, rest @ ..] - if command.eq_ignore_ascii_case("install") - && subcommand.eq_ignore_ascii_case("driver") - && rest.iter().all(|arg| { - arg.eq_ignore_ascii_case("--dkms") || arg.eq_ignore_ascii_case("--dry-run") - }) => - { - let dkms = rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dkms")); - crate::driver_install::install_driver(paths, dkms, false, true) - .map(|result| result.output) - .map_err(|error| error.source) - } - _ => bail!( - "unsupported in-process read-only rocm command: {}", - format_structured_tool_call("rocm", args) - ), - } -} - -fn render_install_sdk_dry_run_for_args(paths: &AppPaths, args: &[String]) -> Result { - let channel = chat_cli_arg_value(args, "--channel").unwrap_or("release"); - let format = chat_cli_arg_value(args, "--format").unwrap_or("wheel"); - let prefix = chat_cli_arg_value(args, "--prefix").map(PathBuf::from); - let version = chat_cli_arg_value(args, "--version").map(str::to_owned); - let build_date = chat_cli_arg_value(args, "--build-date").map(str::to_owned); - let selector = therock_install_version_selector(version, build_date)?; - let devel = chat_cli_has_flag(args, "--devel"); - // Dry run, so nothing is displaced and the consent gate is never reached; - // the narrow consent is what this chat surface would pass for a real - // install, and passing `--yes`'s source here would be a lie waiting to be - // printed if the preview ever grew a gate. - Ok(therock::install_sdk( - paths, - therock::SdkInstallRequest { - channel, - format, - prefix, - version_selector: selector, - dry_run: true, - include_devel: devel, - consent: therock::SdkInstallConsent::Preapproved( - therock::SdkInstallApprovalSource::ApproveReplacingActiveDefault, - ), - ..therock::SdkInstallRequest::default() - }, - )? - .output) -} - -fn internal_mcp_install_sdk_args( - arguments: &serde_json::Map, - dry_run: bool, -) -> Result> { - let channel = json_string(arguments, "channel").unwrap_or_else(|| "release".to_owned()); - let format = json_string(arguments, "format").unwrap_or_else(|| "wheel".to_owned()); - let mut argv = vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - channel, - "--format".to_owned(), - format, - ]; - if let Some(prefix) = json_string(arguments, "prefix") { - let prefix_path = Path::new(&prefix); - if chat_install_prefix_is_system(prefix_path) { - bail!( - "install_sdk prefix `{}` is a system folder; choose a user folder instead", - prefix_path.display() - ); - } - argv.push("--prefix".to_owned()); - argv.push(prefix); - } - if let Some(version) = json_string(arguments, "version") { - argv.push("--version".to_owned()); - argv.push(version); - } - if let Some(build_date) = json_string(arguments, "build_date") { - argv.push("--build-date".to_owned()); - argv.push(build_date); - } - if dry_run { - argv.push("--dry-run".to_owned()); - } - Ok(argv) -} - -fn run_chat_read_only_tool( - paths: &AppPaths, - call: &providers::ChatToolCall, -) -> Result { - match call.name.as_str() { - "path_exists" => run_chat_path_exists_tool(call), - "port_status" => run_chat_port_status_tool(paths, call), - "rocm_command" => { - let action = chat_rocm_command_action(call)?; - let ChatRocmCommandAction::ReadOnly(args) = action else { - bail!("assistant read-only path cannot run mutating rocm_command"); - }; - let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm` command.", - output, - false, - )) - } - "update_check" => { - let output = - run_rocm_command_for_paths(paths, &["update".to_owned()], Duration::from_mins(1))?; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm update`.", - output, - false, - )) - } - "install_sdk_dry_run" => { - let arguments = internal_mcp_arguments(call.arguments.clone()); - let args = internal_mcp_install_sdk_args(&arguments, true)?; - let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; - Ok(internal_mcp_tool_result_from_command( - "Ran `rocm install sdk --dry-run`.", - output, - false, - )) - } - _ => run_internal_mcp_call(paths, &call.name, call.arguments.clone(), false), - } -} - -fn run_chat_path_exists_tool(call: &providers::ChatToolCall) -> Result { - let object = call - .arguments - .as_object() - .context("path_exists arguments must be a JSON object")?; - let path = json_string(object, "path").context("path_exists requires path")?; - let path = Path::new(&path); - let metadata = path.metadata().ok(); - let path_kind = metadata.as_ref().map_or("missing", |metadata| { - if metadata.is_dir() { - "directory" - } else if metadata.is_file() { - "file" - } else { - "other" - } - }); - let parent = path.parent(); - let parent_exists = parent.is_some_and(Path::exists); - let parent_display = parent.map_or_else( - || "".to_owned(), - |parent| parent.display().to_string(), - ); - let text = format!( - "path: {}\nexists: {}\nkind: {}\nparent: {}\nparent_exists: {}", - path.display(), - metadata.is_some(), - path_kind, - parent_display, - parent_exists - ); - Ok(serde_json::json!({ - "content": [{ - "type": "text", - "text": text, - }] - })) -} - -fn run_chat_port_status_tool( - paths: &AppPaths, - call: &providers::ChatToolCall, -) -> Result { - let object = call - .arguments - .as_object() - .context("port_status arguments must be a JSON object")?; - let host = json_string(object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); - let port = object - .get("port") - .and_then(serde_json::Value::as_u64) - .context("port_status requires port")? as u16; - let reachable = loopback_tcp_port_is_reachable(&host, port); - let host_key = loopback_host_key(&host); - let matching_services = load_managed_services(paths)? - .into_iter() - .filter(|record| { - record.port == port - && loopback_host_key(&record.host) == host_key - && managed_service_is_live(record) - }) - .collect::>(); - let app_hint = if port == comfyui::default_port() { - Some("ComfyUI default port") - } else if port == rocm_core::DEFAULT_LOCAL_PORT { - Some("ROCm local model server default port") - } else { - None - }; - let mut text = String::new(); - let _ = writeln!(text, "host: {host}"); - let _ = writeln!(text, "port: {port}"); - let _ = writeln!(text, "listening: {reachable}"); - if let Some(app_hint) = app_hint { - let _ = writeln!(text, "hint: {app_hint}"); - } - if matching_services.is_empty() { - let _ = writeln!(text, "managed_service: none"); - } else { - let _ = writeln!(text, "managed_services:"); - for service in &matching_services { - let _ = writeln!( - text, - " - service_id={} engine={} model={} status={} running_state={} endpoint={}", - service.service_id, - service.engine, - service.model_ref, - service.status, - managed_service_running_state(&service.status), - service.endpoint_url - ); - } - } - Ok(serde_json::json!({ - "content": [{ - "type": "text", - "text": text, - }], - "structuredContent": { - "host": host, - "port": port, - "listening": reachable, - "hint": app_hint, - "managed_services": matching_services, - }, - "isError": false, - })) -} - -const fn chat_read_only_tool_status_label(is_error: bool) -> &'static str { - if is_error { - "reported an error" - } else { - "done" - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::tests::test_paths; - use rocm_core::ManagedServiceRecord; - use std::fs; - use std::net::TcpListener; - - fn rocm_command_call(args: &[&str]) -> providers::ChatToolCall { - providers::ChatToolCall { - id: Some("test".to_owned()), - name: "rocm_command".to_owned(), - arguments: serde_json::json!({ "args": args }), - } - } - - #[test] - fn mcp_tool_result_text_joins_text_content_items() { - let value = serde_json::json!({ - "content": [ - {"type": "text", "text": "line one"}, - {"type": "image", "text": "ignored"}, - {"type": "text", "text": "line two"}, - ] - }); - assert_eq!(mcp_tool_result_text(&value), "line one\nline two"); - } - - #[test] - fn mcp_tool_result_text_falls_back_to_raw_value_without_text_content() { - let value = serde_json::json!({"content": []}); - assert_eq!(mcp_tool_result_text(&value), value.to_string()); - } - - #[test] - fn mcp_tool_result_is_error_reads_the_is_error_flag() { - assert!(mcp_tool_result_is_error( - &serde_json::json!({"isError": true}) - )); - assert!(!mcp_tool_result_is_error( - &serde_json::json!({"isError": false}) - )); - assert!(!mcp_tool_result_is_error(&serde_json::json!({}))); - } - - #[test] - fn chat_tool_display_label_maps_known_tools_and_humanizes_others() { - assert_eq!(chat_tool_display_label("examine"), "Checked this computer"); - assert_eq!( - chat_tool_display_label("gpu_snapshot"), - "Checked GPU status" - ); - assert_eq!(chat_tool_display_label("install_sdk"), "Install ROCm"); - assert_eq!( - chat_tool_display_label("some_unmapped_tool"), - "some unmapped tool" - ); - } - - #[test] - fn chat_tool_call_display_label_delegates_for_non_rocm_command_tools() { - let call = providers::ChatToolCall { - id: None, - name: "examine".to_owned(), - arguments: serde_json::json!({}), - }; - assert_eq!(chat_tool_call_display_label(&call), "Checked this computer"); - } - - #[test] - fn chat_tool_call_display_label_recognizes_rocm_command_shapes() { - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["examine"])), - "Checked this computer" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["models"])), - "Checked model recipes" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["model"])), - "Checked model recipes" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["ENGINES", "List"])), - "Checked local engines" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["services"])), - "Checked model servers" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["services", "list", "-a"])), - "Checked model servers" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["services", "list", "--all"])), - "Checked model servers" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["comfyui"])), - "Checked ComfyUI" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["comfyui", "status"])), - "Checked ComfyUI" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["comfyui", "logs", "-n", "50"])), - "Read ComfyUI logs" - ); - assert_eq!( - chat_tool_call_display_label(&rocm_command_call(&["config", "show"])), - "Checked ROCm config" - ); - } - - #[test] - fn chat_tool_call_display_label_falls_back_for_unparseable_args() { - let call = providers::ChatToolCall { - id: None, - name: "rocm_command".to_owned(), - arguments: serde_json::json!({"args": []}), - }; - assert_eq!(chat_tool_call_display_label(&call), "rocm command"); - } - - #[test] - fn loopback_tcp_port_is_reachable_detects_listening_and_closed_ports() { - let listener = TcpListener::bind("127.0.0.1:0").expect("bind loopback listener"); - let port = listener.local_addr().expect("local addr").port(); - assert!(loopback_tcp_port_is_reachable("127.0.0.1", port)); - drop(listener); - - // Port 0 can never be connected to; stands in for "closed" without - // depending on a specific unused port staying unused. - assert!(!loopback_tcp_port_is_reachable("127.0.0.1", 0)); - } - - #[test] - fn parse_optional_query_reads_flag_and_positional_forms() { - assert_eq!( - parse_optional_query(&["--query".to_owned(), "gpu".to_owned()]).unwrap(), - Some("gpu".to_owned()) - ); - assert_eq!( - parse_optional_query(&["-q".to_owned(), "gpu".to_owned()]).unwrap(), - Some("gpu".to_owned()) - ); - assert_eq!( - parse_optional_query(&["gpu".to_owned()]).unwrap(), - Some("gpu".to_owned()) - ); - assert_eq!(parse_optional_query(&[]).unwrap(), None); - } - - #[test] - fn parse_optional_query_rejects_a_dangling_flag() { - assert!(parse_optional_query(&["--query".to_owned()]).is_err()); - } - - #[test] - fn parse_optional_query_skips_unknown_flags_to_find_the_positional_value() { - assert_eq!( - parse_optional_query(&["--unknown-flag".to_owned(), "gpu".to_owned()]).unwrap(), - Some("gpu".to_owned()) - ); - } - - #[test] - fn parse_optional_lines_reads_flag_or_defaults() { - assert_eq!( - parse_optional_lines(&["--lines".to_owned(), "25".to_owned()]).unwrap(), - 25 - ); - assert_eq!( - parse_optional_lines(&["-n".to_owned(), "25".to_owned()]).unwrap(), - 25 - ); - assert_eq!(parse_optional_lines(&[]).unwrap(), DEFAULT_LOG_TAIL_LINES); - } - - #[test] - fn parse_optional_lines_rejects_a_non_numeric_value() { - assert!(parse_optional_lines(&["--lines".to_owned(), "nope".to_owned()]).is_err()); - } - - #[test] - fn parse_deterministic_model_recipes_reads_header_fields_and_notes() { - let recipes = parse_deterministic_model_recipes( - "\ - Qwen3-4B-Instruct-2507-GGUF aliases=[qwen, lemonade-qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] -lemonade: ready -warning: slow on this GPU - Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] -vllm: not_installed -", - ); - assert_eq!(recipes.len(), 2); - assert_eq!(recipes[0].canonical_id, "Qwen3-4B-Instruct-2507-GGUF"); - assert_eq!(recipes[0].min_gpu_mem_gib, Some(4)); - assert_eq!(recipes[0].engines, vec!["lemonade".to_owned()]); - assert_eq!( - recipes[0].engine_statuses, - vec!["lemonade: ready".to_owned()] - ); - assert_eq!(recipes[0].warnings, vec!["slow on this GPU".to_owned()]); - assert_eq!(recipes[1].canonical_id, "Qwen/Qwen3.5-4B"); - assert_eq!(recipes[1].min_gpu_mem_gib, Some(12)); - } - - #[test] - fn parse_deterministic_model_recipes_requires_aliases_bracket_in_header() { - let recipes = parse_deterministic_model_recipes( - "\ - NotAHeader task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] - Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] -", - ); - assert_eq!(recipes.len(), 1); - assert_eq!(recipes[0].canonical_id, "Qwen/Qwen3.5-4B"); - } - - #[test] - fn parse_deterministic_service_rows_reads_inline_key_value_form() { - let rows = parse_deterministic_service_rows( - " - service_id=svc-1 engine=vllm model=qwen status=ready running_state=running endpoint=127.0.0.1:8000", - ); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].service_id, "svc-1"); - assert_eq!(rows[0].engine, "vllm"); - assert_eq!(rows[0].status, "ready"); - assert_eq!(rows[0].running_state, "running"); - assert_eq!(rows[0].endpoint, "127.0.0.1:8000"); - } - - #[test] - fn parse_deterministic_service_rows_derives_running_state_when_absent() { - let rows = parse_deterministic_service_rows( - "\ -- svc-2 -status: starting -engine: lemonade -model: qwen -endpoint: 127.0.0.1:8001 -", - ); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].service_id, "svc-2"); - assert_eq!(rows[0].running_state, "starting"); - } - - #[test] - fn parse_deterministic_service_rows_derives_running_state_for_inline_form_when_absent() { - let rows = parse_deterministic_service_rows( - " - service_id=svc-3 engine=vllm model=qwen status=starting endpoint=127.0.0.1:8002", - ); - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].running_state, "starting"); - } - - #[test] - fn parse_deterministic_engine_rows_reads_runtime_and_note() { - let rows = parse_deterministic_engine_rows( - "\ -* lemonade - runtime: gfx120X-all - note: default engine -* vllm - runtime: gfx110X-all -", - ); - assert_eq!(rows.len(), 2); - assert_eq!(rows[0].engine, "lemonade"); - assert_eq!(rows[0].runtime, "gfx120X-all"); - assert_eq!(rows[0].note, "default engine"); - assert_eq!(rows[1].engine, "vllm"); - assert_eq!(rows[1].note, ""); - } - - #[test] - fn managed_service_running_state_maps_every_known_status() { - assert_eq!(managed_service_running_state("ready"), "running"); - assert_eq!(managed_service_running_state("running"), "running"); - assert_eq!(managed_service_running_state("starting"), "starting"); - assert_eq!(managed_service_running_state("recovering"), "starting"); - assert_eq!(managed_service_running_state("failed"), "not_running"); - assert_eq!(managed_service_running_state("stopped"), "not_running"); - assert_eq!(managed_service_running_state("anything-else"), "unknown"); - } - - #[test] - fn services_tool_result_text_includes_running_interpretation() { - let (_root, paths) = test_paths("services-tool-text"); - let mut record = ManagedServiceRecord::new( - &paths, - "svc-vllm", - "vllm", - "Qwen/Qwen3.5", - "Qwen/Qwen3.5", - "127.0.0.1", - 11435, - "managed", - std::process::id(), - Some("therock-release".to_owned()), - None, - Some("gpu_required".to_owned()), - ); - record.status = "ready".to_owned(); - - let rendered = render_services_tool_result_text(&[record]); - - assert!(rendered.contains("status_meaning: ready/running = running")); - assert!(rendered.contains("engine=vllm")); - assert!(rendered.contains("running_state=running")); - } - - #[test] - fn local_assistant_prompt_instructions_cover_core_support_questions() { - let prompt = rocm_chat_tool_system_prompt(); - for expected in [ - "is TheRock installed", - "which GPU is on this machine", - "active_runtime_status=ready", - "legacy_rocm_status=not_detected", - "[\"model\"]", - "--build-date", - "always let the user choose the install folder", - "--prefix", - "do not invent a hidden default folder", - "config", - "comfyui", - "First-time setup is the same thing as bootstrap", - "vllm", - "Qwen3-4B-Instruct-2507-GGUF", - "fixed to qwen", - "served by Lemonade", - "port_status", - "[\"services\",\"list\",\"--all\"]", - "qwen-smoke", - "Do not invent shell commands", - "ROCm CLI Assistant Skill", - "Treat `localhost` and `127.0.0.1` as the same loopback endpoint", - ] { - assert!( - prompt.contains(expected), - "system prompt should mention {expected}" - ); - } - } - - /// The reported bug: asked "What can ROCm do on Windows?", the assistant - /// answered that ROCm is Windows-incompatible and suggested CUDA/DirectX — - /// because nothing ever told it which machine it was on. The prompt the CLI - /// actually sends must carry the host. - #[test] - fn assistant_prompt_states_the_host_it_is_answering_for() { - let prompt = rocm_chat_tool_system_prompt_for_host(None); - - // The tool-use rules survive the composition (this is the same prompt, - // grounded — not a replacement for it). - assert!( - prompt.contains("You are ROCm CLI's local assistant"), - "the ROCm tool-use prompt must still be there:\n{prompt}" - ); - - // …and it now names this machine's OS and GPU state. - let expected_os = if cfg!(windows) { - "- Operating system: Windows" - } else { - "- Operating system: Linux" - }; - assert!( - prompt.contains(expected_os), - "the prompt must state this machine's operating system:\n{prompt}" - ); - assert!( - prompt.contains("- AMD GPU: "), - "the prompt must state what GPU was detected (or that none was):\n{prompt}" - ); - assert!( - prompt.contains("Never tell the user ROCm is unavailable on their platform"), - "the prompt must refuse the reported answer:\n{prompt}" - ); - } - - /// The static prompt asserted two Windows-only facts at every host, which is - /// how a WSL user — where vLLM IS the supported path — was told to go use - /// WSL. Platform claims now come from the detected facts instead. - #[test] - fn assistant_prompt_makes_no_unconditional_windows_claims() { - let prompt = rocm_chat_tool_system_prompt(); - assert!( - !prompt.contains("non-technical Windows users"), - "the audience is not assumed to be on Windows:\n{prompt}" - ); - assert!( - !prompt.contains("On native Windows, vLLM is skipped"), - "the vLLM caveat belongs in the host facts, not the static prompt:\n{prompt}" - ); - - // Stated only where it is true: present on Windows, absent elsewhere. - let grounded = rocm_chat_tool_system_prompt_for_host(None); - assert_eq!( - grounded.contains("vLLM and AIM are skipped on native Windows"), - cfg!(windows), - "the vLLM caveat must track the host:\n{grounded}" - ); - } - - /// The prompt tells the model to "use examine … before answering". The dash - /// registers its machine check as `doctor`, so before the alias that - /// sentence named a tool absent from the dash's schema. - #[test] - fn every_tool_the_prompt_names_exists_in_the_dash_schema() { - let prompt = rocm_chat_tool_system_prompt(); - for named in [ - "examine", - "gpu_snapshot", - "port_status", - "natural_language_plan", - ] { - assert!(prompt.contains(named), "prompt should mention {named}"); - assert!( - rocm_dash_tui::agent::ROCM_READ_TOOL_NAMES.contains(&named), - "the prompt names `{named}` but the dash never registers it, so a \ - model that obeys the prompt calls a tool the schema does not offer" - ); - } - } - - #[test] - fn deterministic_rocm_tool_summary_interprets_managed_runtime_as_installed() { - let summary = deterministic_rocm_tool_summary( - "\ -examine: - driver_detail: AMD Radeon RX 9070 XT driver 32.0.23033.1002 - legacy_rocm_status: not_detected -runtime_state: - active_runtime_status: ready - active_runtime_root: D:\\ROCm\\therock_venvs - active_runtime_pip_cache_dir: D:\\ROCm\\therock_venvs\\pip-cache - active_runtime_version: 7.13.0a20260511 (build 2026-05-11) - active_runtime_family: gfx120X-all -", - ) - .expect("examine output should summarize"); - - assert!(summary.contains("GPU: AMD Radeon RX 9070 XT driver 32.0.23033.1002")); - assert!(summary.contains("ROCm/TheRock: installed and active for ROCm CLI")); - assert!(summary.contains("gfx120X-all")); - assert!(summary.contains(r"Install folder: D:\ROCm\therock_venvs")); - assert!(summary.contains(r"Downloads/cache: D:\ROCm\therock_venvs\pip-cache")); - assert!(summary.contains("no global legacy ROCm install was found")); - } - - #[test] - fn fallback_tool_call_routes_where_installed_to_read_only_examine() { - for prompt in [ - "where is rocm installed?", - "where is TheRock installed?", - "what is the ROCm install folder?", - "where did rocm install to?", - ] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "examine", "{prompt}"); - assert!(chat_tool_call_is_read_only(&call), "{prompt}"); - } - } - - #[test] - fn deterministic_rocm_tool_summary_suppresses_extra_local_model_follow_up() { - let tool_result = ChatToolRunResult { - approval: None, - follow_up_text: "\ -examine: - legacy_rocm_status: not_detected -runtime_state: - active_runtime_status: ready -" - .to_owned(), - ran_read_only_tool: true, - read_only_tool_error: false, - needs_install_folder: false, - }; - let summary = deterministic_rocm_tool_summary(&tool_result.follow_up_text); - - assert!(summary.is_some()); - assert!(!should_request_local_tool_follow_up( - "local", - &tool_result, - summary.as_deref() - )); - - let mut model_list_result = tool_result; - model_list_result.follow_up_text = "rocm_command:\nmodel recipes\n qwen\n".to_owned(); - assert!(should_request_local_tool_follow_up( - "local", - &model_list_result, - None - )); - } - - #[test] - fn deterministic_model_tool_summary_identifies_low_vram_assistant() { - let summary = deterministic_model_tool_summary( - "\ -rocm_command: -model recipes - Qwen3-4B-Instruct-2507-GGUF aliases=[qwen, lemonade-qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] - engine_support: - lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe - warning: recommended Lemonade GGUF assistant for ROCm machines - Qwen3-0.6B-GGUF aliases=[qwen-smoke, lemonade-tiny] task=chat dtype=gguf device=gpu_required min_gpu_mem=2 GiB engines=[lemonade] - engine_support: - lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe - warning: tiny Lemonade GGUF smoke-test model; not the default assistant - Qwen/Qwen2.5-0.5B-Instruct aliases=[qwen-tiny] task=chat dtype=float16 device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] - engine_support: - lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe - Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] - engine_support: - vllm: adapter_available path=D:\\rocm\\rocm-engine-vllm.exe runtime_status=unsupported_native_windows reason=native Windows skipped; use WSL/Linux vLLM ROCm - meta-llama/Llama-3.2-3B-Instruct aliases=[llama] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=8 GiB engines=[lemonade, vllm] - engine_support: - lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe - vllm: available path=D:\\rocm\\rocm-engine-vllm.exe -", - ) - .expect("model output should summarize"); - - assert!(summary.contains("Recommended local assistant: qwen")); - assert!(summary.contains("Qwen3-4B-Instruct-2507-GGUF")); - assert!(summary.contains("4 GiB")); - assert!(summary.contains("Tiny smoke test: qwen-smoke")); - assert!(summary.contains("Qwen3-0.6B-GGUF")); - assert!(summary.contains("8 GiB-class option: llama")); - assert!(summary.contains("lemonade, vllm")); - assert!(summary.contains("Qwen/Qwen3.5-4B asks for 12 GiB")); - assert!(summary.contains("Native Windows note")); - assert!(summary.contains("Run `rocm examine`")); - } - - #[test] - fn deterministic_model_tool_summary_suppresses_extra_local_model_follow_up() { - let tool_result = ChatToolRunResult { - approval: None, - follow_up_text: "\ -rocm_command: -model recipes - Qwen3-4B-Instruct-2507-GGUF aliases=[qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] - engine_support: - lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe -" - .to_owned(), - ran_read_only_tool: true, - read_only_tool_error: false, - needs_install_folder: false, - }; - let summary = deterministic_chat_tool_summary(&tool_result.follow_up_text); - - assert!(summary.is_some()); - assert!(!should_request_local_tool_follow_up( - "local", - &tool_result, - summary.as_deref() - )); - } - - #[test] - fn default_engine_fallback_matches_aim_only_as_a_word() { - let proposed = |prompt: &str| { - fallback_config_tool_call_for_prompt(prompt).map(|call| { - call.arguments["args"][2] - .as_str() - .expect("engine arg") - .to_owned() - }) - }; - assert_eq!( - proposed("change the default engine to aim").as_deref(), - Some("aim") - ); - assert_eq!( - proposed("change the default engine to vllm, aiming for low latency").as_deref(), - Some("vllm") - ); - assert_eq!( - proposed( - "change the default engine so I can maintain my claim, aiming for low latency" - ), - None - ); - } - - #[test] - fn proposal_action_approve_updates_status() { - let (root, paths) = test_paths("proposal-approve"); - // Seed a pending proposal. - let proposal = rocm_core::AutomationProposalRecord { - at_unix_ms: rocm_core::unix_time_millis(), - proposal_id: "prop-approve-1".to_owned(), - watcher_id: "therock-update".to_owned(), - action: "prepare_driver_plan".to_owned(), - title: "Apply driver plan".to_owned(), - message: "A reviewed driver plan is ready.".to_owned(), - status: "pending".to_owned(), - service_id: None, - tool: None, - arguments: serde_json::Value::Null, - reviewed_at_unix_ms: None, - }; - rocm_core::append_automation_proposal(&paths, &proposal).expect("seed proposal"); - - // show is read-only and returns the proposal. - let shown = run_internal_mcp_call( - &paths, - "proposal_action", - serde_json::json!({ "proposal_id": "prop-approve-1", "action": "show" }), - false, - ) - .expect("show ok"); - assert_eq!(shown["structuredContent"]["status"], "pending"); - - // approve requires allow_mutation. - assert!( - run_internal_mcp_call( - &paths, - "proposal_action", - serde_json::json!({ "proposal_id": "prop-approve-1", "action": "approve" }), - false, - ) - .is_err(), - "approve without allow_mutation must bail" - ); - - // approve with allow_mutation sets status to approved. - let approved = run_internal_mcp_call( - &paths, - "proposal_action", - serde_json::json!({ "proposal_id": "prop-approve-1", "action": "approve" }), - true, - ) - .expect("approve ok"); - assert_eq!(approved["structuredContent"]["status"], "approved"); - let stored = rocm_core::find_automation_proposal(&paths, "prop-approve-1") - .expect("proposal still present"); - assert_eq!(stored.status, "approved"); - - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn proposal_action_reject_updates_status() { - let (root, paths) = test_paths("proposal-reject"); - let proposal = rocm_core::AutomationProposalRecord { - at_unix_ms: rocm_core::unix_time_millis(), - proposal_id: "prop-reject-1".to_owned(), - watcher_id: "server-recover".to_owned(), - action: "queue_stop_server_proposal".to_owned(), - title: "Stop overheating server".to_owned(), - message: "GPU thermal pressure detected.".to_owned(), - status: "pending".to_owned(), - service_id: None, - tool: None, - arguments: serde_json::Value::Null, - reviewed_at_unix_ms: None, - }; - rocm_core::append_automation_proposal(&paths, &proposal).expect("seed proposal"); - - let rejected = run_internal_mcp_call( - &paths, - "proposal_action", - serde_json::json!({ "proposal_id": "prop-reject-1", "action": "reject" }), - true, - ) - .expect("reject ok"); - assert_eq!(rejected["structuredContent"]["status"], "rejected"); - let stored = rocm_core::find_automation_proposal(&paths, "prop-reject-1") - .expect("proposal still present"); - assert_eq!(stored.status, "rejected"); - - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn proposal_action_show_missing_proposal_errors() { - let (root, paths) = test_paths("proposal-missing"); - assert!( - run_internal_mcp_call( - &paths, - "proposal_action", - serde_json::json!({ "proposal_id": "nope", "action": "show" }), - false, - ) - .is_err(), - "showing a missing proposal must error" - ); - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn assistant_read_only_rocm_commands_do_not_fallback_to_child_process() { - let (_root, paths) = test_paths("readonly-rocm-in-process-only"); - let args = vec!["services".to_owned(), "status".to_owned()]; - - let error = run_rocm_command_for_paths(&paths, &args, Duration::from_secs(1)) - .unwrap_err() - .to_string(); - - assert!(error.contains("read-only assistant command is not implemented in-process")); - assert!(error.contains("rocm services status")); - assert!(!paths.data_dir.join("logs").exists()); - } - - #[test] - fn internal_mcp_read_only_rocm_command_runs_in_process() { - let (_root, paths) = test_paths("mcp-readonly-rocm-in-process"); - - let result = run_internal_mcp_call( - &paths, - "rocm_command", - serde_json::json!({ "args": ["version"] }), - false, - ) - .expect("read-only rocm mcp-call should run"); - - assert_eq!( - result.get("isError").and_then(serde_json::Value::as_bool), - Some(false) - ); - assert_eq!( - result - .pointer("/structuredContent/argv/0") - .and_then(serde_json::Value::as_str), - Some("rocm") - ); - assert!(mcp_tool_result_text(&result).contains(env!("CARGO_PKG_VERSION"))); - } - - #[test] - fn natural_language_plan_returns_structured_mutating_action() { - let (_root, paths) = test_paths("nl-plan-structured-mutating"); - - let result = run_internal_mcp_call( - &paths, - "natural_language_plan", - serde_json::json!({ - "request": "install TheRock into /opt/rocm-target" - }), - false, - ) - .expect("natural_language_plan should plan the request"); - - assert_eq!( - result.get("isError").and_then(serde_json::Value::as_bool), - Some(false) - ); - // text (rendered plan) preserved. - assert!( - !result - .pointer("/structuredContent/text") - .and_then(serde_json::Value::as_str) - .unwrap_or_default() - .trim() - .is_empty() - ); - // A known install folder yields a mutating, placeholder-free action. - assert_eq!( - result - .pointer("/structuredContent/action/approval_required") - .and_then(serde_json::Value::as_bool), - Some(true) - ); - assert_eq!( - result - .pointer("/structuredContent/action/has_placeholders") - .and_then(serde_json::Value::as_bool), - Some(false) - ); - let args = result - .pointer("/structuredContent/action/args") - .and_then(serde_json::Value::as_array) - .expect("action args should be present"); - assert!(args.iter().any(|arg| arg.as_str() == Some("install"))); - assert!(args.iter().any(|arg| arg.as_str() == Some("--prefix"))); - } - - #[test] - fn natural_language_plan_returns_placeholder_action_when_incomplete() { - let (_root, paths) = test_paths("nl-plan-structured-placeholder"); - - let result = run_internal_mcp_call( - &paths, - "natural_language_plan", - serde_json::json!({ "request": "serve" }), - false, - ) - .expect("natural_language_plan should plan the request"); - - assert_eq!( - result - .pointer("/structuredContent/action/has_placeholders") - .and_then(serde_json::Value::as_bool), - Some(true) - ); - } - - /// On Windows `rocm install driver` only reports, and chat plans it as a - /// read-only call. That call must run in process: before this arm existed it - /// failed with "unsupported in-process read-only rocm command". It is a dry - /// run, so it never executes anything on any platform. - #[test] - fn install_driver_read_only_call_runs_in_process() { - let (_root, paths) = test_paths("install-driver-in-process"); - let args = vec!["install".to_owned(), "driver".to_owned()]; - let text = run_rocm_read_only_in_process(&paths, &args) - .expect("the read-only install driver call should run in process"); - assert!(text.contains("approval:"), "{text}"); - assert!(!text.contains("driver install completed"), "{text}"); - assert!( - run_rocm_read_only_in_process( - &paths, - &[ - "install".to_owned(), - "driver".to_owned(), - "--yes".to_owned() - ] - ) - .is_err(), - "a flag the read-only path does not know is still refused" - ); - } - - /// `--dry-run` is what the in-process arm already does, so it is accepted - /// and the call reports a dry run. `--reconcile` writes driver install state - /// and stays refused, as does any flag the arm does not know. - #[test] - fn install_driver_read_only_call_accepts_dry_run_and_refuses_reconcile() { - let (_root, paths) = test_paths("install-driver-in-process-dry-run"); - let run = |extra: &[&str]| { - let mut args = vec!["install".to_owned(), "driver".to_owned()]; - args.extend(extra.iter().map(|arg| (*arg).to_owned())); - run_rocm_read_only_in_process(&paths, &args) - }; - for extra in [&["--dry-run"][..], &["--dkms", "--dry-run"], &["--DRY-RUN"]] { - let text = run(extra).unwrap_or_else(|error| panic!("{extra:?} must run: {error}")); - assert!(text.contains("dry_run: true"), "{extra:?}: {text}"); - assert!( - !text.contains("driver install completed"), - "{extra:?}: {text}" - ); - } - for extra in [&["--reconcile"][..], &["--dry-run", "--reconcile"]] { - assert!(run(extra).is_err(), "{extra:?} must stay refused"); - } - // Refusing `--reconcile` changed nothing on disk: the state file is - // written only by a reconcile or an executed install. - assert!( - !crate::driver_install::driver_install_state_path(&paths).exists(), - "a refused call must not create driver install state" - ); - } - - #[test] - fn setup_status_renders_in_process() { - let (_root, paths) = test_paths("setup-status-in-process"); - - for args in [ - vec!["setup".to_owned()], - vec!["setup".to_owned(), "status".to_owned()], - ] { - let text = run_rocm_read_only_in_process(&paths, &args) - .expect("setup status read path should render in-process"); - assert!( - !text.trim().is_empty(), - "setup status text should be non-empty for {args:?}" - ); - assert!( - text.contains("ROCm setup"), - "setup status text should be reached for {args:?}: {text}" - ); - } - } - - #[test] - fn chat_tool_call_refusal_prioritizes_safe_review_wording() -> Result<()> { - let mut output = String::new(); - let (_root, paths) = test_paths("chat-tool-refusal"); - let response = providers::ChatResponse { - provider: "local".to_owned(), - model: "tiny.gguf".to_owned(), - content: "I can install ROCm.".to_owned(), - tool_calls: vec![providers::ChatToolCall { - id: None, - name: "install_sdk".to_owned(), - arguments: serde_json::json!({ - "channel": "release", - "format": "wheel", - "prefix": "D:\\ROCm\\therock_venvs" - }), - }], - }; - - let mut progress = None; - append_chat_tool_results( - &paths, - &response, - &mut output, - Some("I can install ROCm."), - &mut progress, - )?; - - assert!(output.contains("Install ROCm: needs your review")); - assert!(!output.contains("install_sdk: approval required")); - assert!(output.contains("not run: review the approval card before anything runs")); - assert!(output.contains("advanced manual command: rocm install sdk")); - assert!(!output.contains("run the shown ROCm command")); - Ok(()) - } - - #[test] - fn fallback_tool_call_runs_model_support_checks() { - let call = - fallback_rocm_tool_call_for_prompt("Which LLMs can this machine support?").unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec!["model".to_owned()] - ); - - assert!(fallback_rocm_tool_call_for_prompt("Which LLM are you using?").is_none()); - } - - #[test] - fn local_rocm_tools_chat_uses_fixed_lemonade_qwen_assistant() { - assert_eq!( - local_rocm_tools_assistant_model("local", true), - Some(providers::BUILTIN_ASSISTANT_MODEL_ID) - ); - assert_eq!(local_rocm_tools_assistant_model("local", false), None); - assert_eq!(local_rocm_tools_assistant_model("openai", true), None); - } - - #[test] - fn fallback_tool_call_routes_running_questions_to_status_tools() { - let comfy = fallback_rocm_tool_call_for_prompt("Is ComfyUI running?").unwrap(); - assert_eq!(comfy.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&comfy).unwrap(), - vec!["comfyui".to_owned(), "status".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&comfy)); - - let comfy_port = - fallback_rocm_tool_call_for_prompt("Is ComfyUI running on port 8188?").unwrap(); - assert_eq!(comfy_port.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&comfy_port).unwrap(), - vec!["comfyui".to_owned(), "status".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&comfy_port)); - - for prompt in [ - "Is vLLM running?", - "is lemonade running?", - "is the model server running?", - "is qwen running?", - ] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "rocm_command", "{prompt}"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec!["services".to_owned(), "list".to_owned(), "--all".to_owned(),], - "{prompt}" - ); - assert!(chat_tool_call_is_read_only(&call), "{prompt}"); - } - - let port = fallback_rocm_tool_call_for_prompt("what is running on port 8188?").unwrap(); - assert_eq!(port.name, "port_status"); - assert_eq!( - port.arguments, - serde_json::json!({ "host": DEFAULT_LOCAL_HOST, "port": 8188 }) - ); - assert!(chat_tool_call_is_read_only(&port)); - } - - #[test] - fn fallback_tool_call_routes_engine_install_state_to_engines_list() { - for prompt in ["is vLLM installed?", "is Lemonade available?"] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "rocm_command", "{prompt}"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec!["engines".to_owned(), "list".to_owned()], - "{prompt}" - ); - assert!(chat_tool_call_is_read_only(&call), "{prompt}"); - } - } - - #[test] - fn supplemental_tool_call_adds_missing_specific_status_check() { - let generic_services = providers::ChatToolCall { - id: Some("model-picked-services".to_owned()), - name: "services".to_owned(), - arguments: serde_json::json!({}), - }; - - let call = - supplemental_read_only_tool_call_for_prompt("Is ComfyUI running?", &[generic_services]) - .unwrap(); - - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec!["comfyui".to_owned(), "status".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&call)); - } - - #[test] - fn supplemental_tool_call_does_not_duplicate_equivalent_status_check() { - let comfy_status = fallback_rocm_tool_call_for_prompt("Is ComfyUI running?").unwrap(); - - assert!( - supplemental_read_only_tool_call_for_prompt("Is ComfyUI running?", &[comfy_status]) - .is_none() - ); - } - - #[test] - fn supplemental_tool_call_adds_running_state_for_engine_install_question() { - let engine_inventory = - fallback_rocm_tool_call_for_prompt("Is vLLM installed and is it running?").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&engine_inventory).unwrap(), - vec!["engines".to_owned(), "list".to_owned()] - ); - - let services = supplemental_read_only_tool_call_for_prompt( - "Is vLLM installed and is it running?", - &[engine_inventory], - ) - .unwrap(); - - assert_eq!(services.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&services).unwrap(), - vec!["services".to_owned(), "list".to_owned(), "--all".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&services)); - } - - #[test] - fn supplemental_tool_call_treats_loopback_port_checks_as_equivalent() { - let model_port_check = providers::ChatToolCall { - id: Some("model-picked-port".to_owned()), - name: "port_status".to_owned(), - arguments: serde_json::json!({ "host": "localhost", "port": 8188 }), - }; - - assert!( - supplemental_read_only_tool_call_for_prompt( - "what is running on port 8188?", - &[model_port_check] - ) - .is_none() - ); - } - - #[test] - fn fallback_tool_call_runs_status_checks() { - for prompt in [ - "Which GPU is on my machine, and is ROCm installed?", - "Is TheRock setup on my machine?", - "Check this ROCm setup.", - ] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "examine"); - assert_eq!(call.arguments, serde_json::json!({})); - } - } - - #[test] - fn fallback_tool_call_routes_therock_setup_to_install_folder_flow() { - let call = fallback_rocm_tool_call_for_prompt("How do I setup TheRock?").unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&call)); - } - - #[test] - fn fallback_tool_call_preserves_requested_therock_install_prefix() { - for (prompt, expected_prefix) in [ - ( - "install TheRock for me in D:\\ROCm\\therock_venvs", - "D:\\ROCm\\therock_venvs", - ), - ("install ROCm to D:\\ROCm\\temp", "D:\\ROCm\\temp"), - ] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--prefix".to_owned(), - expected_prefix.to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&call)); - } - } - - #[test] - fn fallback_tool_call_routes_requested_therock_build_date_install() { - for (prompt, expected_prefix) in [ - ( - "Install this specific TheRock wheel from date 06052026 into D:\\ROCm\\therock_venvs", - "D:\\ROCm\\therock_venvs", - ), - ( - "install ROCm at D:\\ROCm\\temp with build date 2026-06-05", - "D:\\ROCm\\temp", - ), - ] { - let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--prefix".to_owned(), - expected_prefix.to_owned(), - "--build-date".to_owned(), - "2026-06-05".to_owned(), - ], - "{prompt}" - ); - assert!(!chat_tool_call_is_read_only(&call)); - } - } - - #[test] - fn fallback_tool_call_does_not_install_therock_without_folder() { - let call = fallback_rocm_tool_call_for_prompt( - "Install this specific TheRock wheel from date 06052026", - ) - .unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--build-date".to_owned(), - "2026-06-05".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&call)); - } - - #[test] - fn chat_install_intent_without_folder_uses_folder_picker_not_rocm_check() -> Result<()> { - let (_root, paths) = test_paths("chat-install-intent-folder-picker"); - for prompt in [ - "i need to install rocm", - "How do I setup TheRock?", - "install therock", - "How do I get it to install therock?", - "rocm please", - "make my AMD GPU ready", - "get TheRock installed for local AI", - "set up my AMD GPU for local AI", - "install this specific TheRock wheel from date 06052026", - ] { - let result = render_chat_prompt_result(&paths, "local", None, prompt, true)?; - let approval = result - .approval - .as_ref() - .expect("install should need folder"); - assert_eq!(approval.pending_title, "Install ROCm", "{prompt}"); - assert_eq!( - approval.args[..6], - [ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - ], - "{prompt}" - ); - assert!( - !approval.args.iter().any(|arg| arg == "--prefix"), - "{prompt}" - ); - assert!( - result.rendered.contains("First choose the folder"), - "{prompt}: {}", - result.rendered - ); - assert!( - !result.rendered.contains("I checked ROCm"), - "{prompt}: {}", - result.rendered - ); - assert!( - !result.rendered.contains("ROCm CLI summary"), - "{prompt}: {}", - result.rendered - ); - if prompt.contains("06052026") { - assert!(approval.args.contains(&"--build-date".to_owned())); - assert!(approval.args.contains(&"2026-06-05".to_owned())); - } - } - Ok(()) - } - - #[test] - fn chat_install_intent_ignores_old_conversation_words() -> Result<()> { - let (_root, paths) = test_paths("chat-install-intent-latest-message"); - let prompt = "\ -Conversation so far: -Assistant: Use /examine to refresh actual GPU memory fit before starting anything large. -Assistant: Native Windows note: models may use WSL/Linux through Windows. - -New message: -install therock"; - - let result = render_chat_prompt_result(&paths, "local", None, prompt, true)?; - let approval = result - .approval - .as_ref() - .expect("direct latest install request should need a folder"); - - assert_eq!(approval.pending_title, "Install ROCm"); - assert!(!approval.args.iter().any(|arg| arg == "--prefix")); - assert!(result.rendered.contains("I can install ROCm/TheRock")); - assert!(result.rendered.contains("First choose the folder")); - assert!(!result.rendered.contains("I checked ROCm")); - assert!(!result.rendered.contains("ROCm CLI summary")); - Ok(()) - } - - #[test] - fn chat_how_to_setup_question_opens_install_folder_flow() { - assert!( - install_sdk_without_prefix_chat_approval("How do I setup TheRock?").is_some(), - "a setup question should ask for the install folder" - ); - assert!( - install_sdk_without_prefix_chat_approval("install therock").is_some(), - "a direct install command should open the folder picker" - ); - } - - #[test] - fn chat_install_intent_preserves_bare_folder_path() { - let approval = - install_sdk_chat_approval_for_prompt("install therock D:\\ROCm\\therock_venvs") - .expect("direct install prompt should be recognized"); - - assert_eq!( - approval.args, - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--prefix".to_owned(), - "D:\\ROCm\\therock_venvs".to_owned(), - ] - ); - } - - #[test] - fn fallback_tool_call_routes_requested_therock_exact_version_install() { - let call = fallback_rocm_tool_call_for_prompt( - "Install the TheRock ROCm wheel version 7.13.0a20260605 into D:\\ROCm\\therock_venvs", - ) - .unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "install".to_owned(), - "sdk".to_owned(), - "--channel".to_owned(), - "release".to_owned(), - "--format".to_owned(), - "wheel".to_owned(), - "--prefix".to_owned(), - "D:\\ROCm\\therock_venvs".to_owned(), - "--version".to_owned(), - "7.13.0a20260605".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&call)); - } - - #[test] - fn port_status_matches_loopback_managed_services() -> Result<()> { - let (root, paths) = test_paths("port-status-loopback"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "svc-comfyui", - "comfyui", - "ComfyUI", - "ComfyUI", - "127.0.0.1", - 18188, - "managed", - std::process::id(), - Some("therock-release".to_owned()), - None, - Some("gpu_required".to_owned()), - ); - record.status = "ready".to_owned(); - record.write()?; - - let call = providers::ChatToolCall { - id: Some("port-check".to_owned()), - name: "port_status".to_owned(), - arguments: serde_json::json!({ "host": "localhost", "port": 18188 }), - }; - let result = run_chat_port_status_tool(&paths, &call)?; - let text = mcp_tool_result_text(&result); - let managed_service_count = result - .get("structuredContent") - .and_then(|content| content.get("managed_services")) - .and_then(serde_json::Value::as_array) - .map_or(0, Vec::len); - let _ = fs::remove_dir_all(root); - - assert_eq!(managed_service_count, 1); - assert!(text.contains("managed_services:"), "{text}"); - assert!(text.contains("service_id=svc-comfyui"), "{text}"); - assert!(text.contains("running_state=starting"), "{text}"); - Ok(()) - } - - #[test] - fn fallback_tool_call_routes_simple_config_changes() { - let show = fallback_rocm_tool_call_for_prompt("Show current ROCm CLI config").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&show).unwrap(), - vec!["config".to_owned(), "show".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&show)); - - let engine = fallback_rocm_tool_call_for_prompt("Set the default engine to vllm").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&engine).unwrap(), - vec![ - "config".to_owned(), - "set-default-engine".to_owned(), - "vllm".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&engine)); - - let telemetry = - fallback_rocm_tool_call_for_prompt("Disable telemetry in settings").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&telemetry).unwrap(), - vec![ - "config".to_owned(), - "set-telemetry".to_owned(), - "off".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&telemetry)); - } - - #[test] - fn fallback_tool_call_routes_comfyui_support_and_actions() { - let status = fallback_rocm_tool_call_for_prompt("How do I setup ComfyUI?").unwrap(); - assert_eq!(status.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&status).unwrap(), - vec!["comfyui".to_owned(), "status".to_owned()] - ); - assert!(chat_tool_call_is_read_only(&status)); - - let install = fallback_rocm_tool_call_for_prompt("Can you setup ComfyUI for me?").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&install).unwrap(), - vec!["comfyui".to_owned(), "install".to_owned()] - ); - assert!(!chat_tool_call_is_read_only(&install)); - let approval = - chat_tool_approval_request(&install, Some("Install ComfyUI after approval.")).unwrap(); - assert_eq!(approval.pending_title, "Install ComfyUI"); - - let start = fallback_rocm_tool_call_for_prompt("Can you start ComfyUI?").unwrap(); - assert_eq!( - normalized_chat_rocm_command_args(&start).unwrap(), - vec!["comfyui".to_owned(), "start".to_owned()] - ); - assert!(!chat_tool_call_is_read_only(&start)); - let approval = - chat_tool_approval_request(&start, Some("Start ComfyUI after approval.")).unwrap(); - assert_eq!(approval.pending_title, "Start ComfyUI"); - } - - #[test] - fn fallback_tool_call_routes_local_llm_serve_requests() { - let call = - fallback_rocm_tool_call_for_prompt("Can you setup and serve an LLM for me?").unwrap(); - assert_eq!(call.name, "rocm_command"); - assert_eq!( - normalized_chat_rocm_command_args(&call).unwrap(), - vec![ - "serve".to_owned(), - "qwen".to_owned(), - "--engine".to_owned(), - "lemonade".to_owned(), - "--device".to_owned(), - "gpu_required".to_owned(), - "--managed".to_owned(), - ] - ); - assert!(!chat_tool_call_is_read_only(&call)); - let approval = - chat_tool_approval_request(&call, Some("Start qwen after approval.")).unwrap(); - assert_eq!(approval.pending_title, "Start local model server"); - assert_eq!( - rocm_chat_tool_requested_command(&call).as_deref(), - Some("rocm serve qwen --engine lemonade --device gpu_required --managed") - ); - } - - #[test] - fn local_chat_tool_call_content_is_treated_as_intermediate() { - let response = providers::ChatResponse { - provider: "local".to_owned(), - model: "Qwen/Qwen3-0.6B".to_owned(), - content: "The active runtime root is /opt/rocm.".to_owned(), - tool_calls: vec![providers::ChatToolCall { - id: Some("call-1".to_owned()), - name: "examine".to_owned(), - arguments: serde_json::json!({}), - }], - }; - - assert!(local_tool_call_content_is_intermediate( - "local", true, &response - )); - assert!(!local_tool_call_content_is_intermediate( - "openai", true, &response - )); - assert!(!local_tool_call_content_is_intermediate( - "local", false, &response - )); - - let without_tools = providers::ChatResponse { - tool_calls: Vec::new(), - ..response - }; - assert!(!local_tool_call_content_is_intermediate( - "local", - true, - &without_tools - )); - } - - #[test] - fn local_chat_follow_up_with_tool_call_is_not_final_answer() { - let response = providers::ChatResponse { - provider: "local".to_owned(), - model: "Qwen/Qwen3-0.6B".to_owned(), - content: "The runtime root is /opt/rocml.".to_owned(), - tool_calls: vec![providers::ChatToolCall { - id: Some("call-2".to_owned()), - name: "examine".to_owned(), - arguments: serde_json::json!({}), - }], - }; - - assert!(!local_follow_up_content_is_final(&response)); - - let final_answer = providers::ChatResponse { - tool_calls: Vec::new(), - content: "The runtime root is D:\\ROCm\\therock_venvs.".to_owned(), - ..response - }; - assert!(local_follow_up_content_is_final(&final_answer)); - } - - #[test] - fn visible_chat_content_removes_reasoning_blocks() { - assert_eq!( - visible_chat_content( - "\nchecking the tool output\n\nThe runtime root is D:\\ROCm\\temp." - ), - "The runtime root is D:\\ROCm\\temp." - ); - assert_eq!( - visible_chat_content("Before\nhidden\nAfter"), - "Before\n\nAfter" - ); - assert_eq!(visible_chat_content("unfinished"), ""); - } - - #[test] - fn chat_tool_result_errors_use_plain_failure_wording() { - assert_eq!(chat_read_only_tool_status_label(false), "done"); - assert_eq!(chat_read_only_tool_status_label(true), "reported an error"); - assert_eq!(chat_tool_display_label("examine"), "Checked this computer"); - assert_eq!( - chat_tool_display_label("gpu_snapshot"), - "Checked GPU status" - ); - assert_eq!(chat_tool_display_label("install_sdk"), "Install ROCm"); - assert!(mcp_tool_result_is_error(&serde_json::json!({ - "isError": true - }))); - assert!(!mcp_tool_result_is_error(&serde_json::json!({ - "isError": false - }))); - assert!(!mcp_tool_result_is_error(&serde_json::json!({}))); - } - - #[test] - fn local_chat_without_service_explains_serve_before_chat_without_llm_setup() { - let (_root, paths) = test_paths("local-chat-no-service-guidance"); - let result = - render_chat_prompt_result(&paths, "local", None, "Check this ROCm setup", true) - .expect("missing local assistant should render guidance"); - assert!(result.approval.is_none()); - let rendered = result.rendered; - - assert!(rendered.contains("No local assistant is running yet.")); - assert!(rendered.contains("First-time ROCm setup does not need an LLM")); - assert!(rendered.contains("Recommended path:")); - assert!(rendered.contains("Advanced manual command")); - assert!(rendered.contains( - "rocm serve Qwen3-4B-Instruct-2507-GGUF --engine lemonade --device gpu_required --managed" - )); - assert!(!rendered.contains("sshleifer/tiny-gpt2")); - assert!(rendered.contains("rocm chat --tools --provider local --prompt")); - assert!(rendered.contains("Nothing was changed.")); - assert!(!rendered.contains("install sdk")); - assert!(!rendered.contains("setup TheRock with an LLM")); - } - - #[test] - fn local_chat_status_prompts_use_read_only_tools_without_assistant() -> Result<()> { - let (_root, paths) = test_paths("local-chat-status-fallback"); - - let running = - render_chat_prompt_result(&paths, "local", None, "Is vLLM running?", true)?.rendered; - assert!(!running.contains("No local assistant is running yet.")); - assert!(running.contains("Checked model servers: done"), "{running}"); - assert!(running.contains("ROCm CLI summary"), "{running}"); - assert!( - running.contains("Local model servers: none running under ROCm CLI."), - "{running}" - ); - assert!(running.contains("Nothing was changed.")); - - let installed = - render_chat_prompt_result(&paths, "local", None, "Is vLLM installed?", true)?.rendered; - assert!(!installed.contains("No local assistant is running yet.")); - assert!(installed.contains("Engine runtimes:"), "{installed}"); - assert!(installed.contains("vLLM:"), "{installed}"); - - let installed_and_running = render_chat_prompt_result( - &paths, - "local", - None, - "Is vLLM installed and is it running?", - true, - )? - .rendered; - assert!( - installed_and_running.contains("Checked local engines: done"), - "{installed_and_running}" - ); - assert!( - installed_and_running.contains("Checked model servers: done"), - "{installed_and_running}" - ); - assert!( - installed_and_running.contains("Engine runtimes:"), - "{installed_and_running}" - ); - assert!( - installed_and_running.contains("Local model servers: none running under ROCm CLI."), - "{installed_and_running}" - ); - - let port = render_chat_prompt_result( - &paths, - "local", - None, - "What is running on port 8188?", - true, - )? - .rendered; - assert!(!port.contains("No local assistant is running yet.")); - assert!(port.contains("Checked local port: done"), "{port}"); - assert!(port.contains("Port 8188:"), "{port}"); - Ok(()) - } - - #[test] - fn chat_tools_anthropic_reaches_provider_opt_in_boundary() { - let (_root, paths) = test_paths("anthropic-chat-tools-opt-in"); - - let error = render_chat_prompt_result( - &paths, - "anthropic", - Some("claude-test"), - "Check this ROCm setup", - true, - ) - .unwrap_err() - .to_string(); - - assert!(error.contains("cloud provider `anthropic` is disabled")); - assert!(error.contains("rocm config enable-provider anthropic")); - assert!(!error.contains("OpenAI-compatible provider")); - } +mod tests { + use super::*; #[test] fn piped_prompt_keeps_everything_but_the_trailing_line_ending() { @@ -4734,11 +298,13 @@ install therock"; /// mostly metacharacters. #[cfg(test)] mod command_preview_properties { + use super::mcp_bridge::{mcp_tool_result_text, run_chat_path_exists_tool}; use super::*; use crate::freeform_planner::{ - ProviderPlannerToolCall, planner_failure_note, provider_planner_response_to_plan, - validate_provider_planner_tool_call, + ProviderPlannerToolCall, format_structured_tool_call, planner_failure_note, + provider_planner_response_to_plan, validate_provider_planner_tool_call, }; + use crate::tool_policy::rocm_chat_tool_requested_command; // The one generator, shared with `shell_quote`'s own suite so both are // covered by its reach guard. See `crate::shell_quote::hostile`. use crate::shell_quote::has_terminal_control_character; diff --git a/apps/rocm/src/chat/intent.rs b/apps/rocm/src/chat/intent.rs new file mode 100644 index 0000000000..e665f666d4 --- /dev/null +++ b/apps/rocm/src/chat/intent.rs @@ -0,0 +1,949 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The chat assistant's prompt-intent heuristics: what a user's latest message +//! asks for, decided from its words alone. `fallback_rocm_tool_call_for_prompt` +//! maps a prompt to the read-only or approval-gated tool call the assistant +//! proposes when it does not need the local model to choose one (with +//! `fallback_config_tool_call_for_prompt`, `fallback_engine_list_tool_call` and +//! `fallback_services_list_tool_call` behind it), +//! `prompt_can_use_read_only_without_local_assistant` says whether a status +//! question can be answered before any model server is up, and `install_sdk_chat_approval_for_prompt` (with +//! `install_sdk_without_prefix_chat_approval`) turns an install or setup +//! request into the `rocm install sdk` approval card. The `prompt_asks_*`, +//! `prompt_mentions_*` and `prompt_requests_*` predicates hold the phrase +//! lists, and `latest_user_chat_message` cuts the newest message out of the +//! conversation text the chat builds around it, so older words in the +//! conversation never steer these heuristics. +//! +//! A leaf of the `chat` module: nothing here calls back into `chat.rs`. The +//! edges out are `prompt_text.rs` (the phrase and install-detail extraction), +//! `rocm_argv.rs`, `tool_policy.rs` (`ChatToolApprovalRequest`), +//! `freeform_planner.rs` (`format_structured_tool_call`) and `providers` (the +//! tool-call type). The turn loop in `chat/turn.rs` reads this module's items +//! through `super::intent`. + +use crate::freeform_planner::format_structured_tool_call; +use crate::prompt_text::{ + any_substring, infer_engine_from_request, requested_install_prefix_from_prompt, + requested_therock_build_date_from_prompt, requested_therock_version_from_prompt, +}; +use crate::providers; +use crate::rocm_argv::chat_cli_arg_value; +use crate::tool_policy::ChatToolApprovalRequest; +use rocm_core::DEFAULT_LOCAL_HOST; + +pub(super) fn latest_user_chat_message(prompt: &str) -> &str { + const MARKER: &str = "\nNew message:\n"; + prompt + .rfind(MARKER) + .map_or(prompt, |index| &prompt[index + MARKER.len()..]) + .trim() +} + +pub(super) fn prompt_can_use_read_only_without_local_assistant(prompt: &str) -> bool { + let normalized = latest_user_chat_message(prompt).to_ascii_lowercase(); + let mentions_status_subject = any_substring( + &normalized, + &[ + "comfyui", + "comfy ui", + "comfy", + "vllm", + "lemonade", + "llama.cpp", + "llama cpp", + "qwen", + "model server", + "local server", + "local model server", + "assistant server", + "port", + "8188", + "therock", + "rocm", + ], + ); + mentions_status_subject + && (prompt_asks_running_or_status(&normalized) + || any_substring( + &normalized, + &["installed", "available", "detected", "engine status"], + )) +} + +pub(super) fn prompt_mentions_serving_engine_or_service(normalized_prompt: &str) -> bool { + any_substring( + normalized_prompt, + &[ + "vllm", + "lemonade", + "llama.cpp", + "llama cpp", + "qwen", + "model server", + "local server", + "local model server", + "assistant server", + ], + ) +} + +pub(super) fn prompt_asks_engine_install_state(normalized_prompt: &str) -> bool { + any_substring( + normalized_prompt, + &["installed", "available", "detected", "engine status"], + ) +} + +pub(super) fn fallback_engine_list_tool_call() -> providers::ChatToolCall { + providers::ChatToolCall { + id: Some("fallback-engine-list".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["engines", "list"] }), + } +} + +pub(super) fn fallback_services_list_tool_call() -> providers::ChatToolCall { + providers::ChatToolCall { + id: Some("fallback-services-list".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["services", "list", "--all"] }), + } +} + +pub(super) fn fallback_rocm_tool_call_for_prompt(prompt: &str) -> Option { + let prompt = latest_user_chat_message(prompt); + let normalized = prompt.to_ascii_lowercase(); + let asks_running_or_status = prompt_asks_running_or_status(&normalized); + let mentions_comfyui = any_substring(&normalized, &["comfyui", "comfy ui", "comfy"]); + if asks_running_or_status + && !mentions_comfyui + && any_substring(&normalized, &["8188", "port 8188"]) + { + return Some(providers::ChatToolCall { + id: Some("fallback-port-8188-status".to_owned()), + name: "port_status".to_owned(), + arguments: serde_json::json!({ "host": DEFAULT_LOCAL_HOST, "port": 8188 }), + }); + } + let mentions_serving_engine_or_service = prompt_mentions_serving_engine_or_service(&normalized); + if mentions_serving_engine_or_service && prompt_asks_engine_install_state(&normalized) { + return Some(fallback_engine_list_tool_call()); + } + if mentions_serving_engine_or_service && asks_running_or_status { + return Some(fallback_services_list_tool_call()); + } + let mentions_llm_or_model = any_substring( + &normalized, + &["llm", "llms", "model", "models", "assistant"], + ); + let asks_support_or_fit = any_substring( + &normalized, + &[ + "support", + "supported", + "run", + "runs", + "fit", + "fits", + "can my machine", + "can this machine", + ], + ); + if mentions_llm_or_model && asks_support_or_fit { + return Some(providers::ChatToolCall { + id: Some("fallback-rocm-model".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["model"] }), + }); + } + + if mentions_comfyui { + if any_substring(&normalized, &["log", "logs"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-comfyui-logs".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["comfyui", "logs"] }), + }); + } + if asks_running_or_status { + return Some(providers::ChatToolCall { + id: Some("fallback-comfyui-status".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["comfyui", "status"] }), + }); + } + if any_substring(&normalized, &["start", "run", "launch", "open"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-comfyui-start".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["comfyui", "start"], + "reason": "Start ComfyUI locally after the user approves it." + }), + }); + } + if any_substring( + &normalized, + &[ + "can you setup", + "can you set up", + "please setup", + "please set up", + "setup comfyui for me", + "set up comfyui for me", + "install comfyui", + "download comfyui", + ], + ) { + return Some(providers::ChatToolCall { + id: Some("fallback-comfyui-install".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["comfyui", "install"], + "reason": "Install ComfyUI into ROCm CLI's managed app folder after the user approves it." + }), + }); + } + return Some(providers::ChatToolCall { + id: Some("fallback-comfyui-status".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["comfyui", "status"] }), + }); + } + + if let Some(call) = fallback_config_tool_call_for_prompt(&normalized) { + return Some(call); + } + + if mentions_llm_or_model + && any_substring( + &normalized, + &[ + "serve", + "server", + "start", + "setup and serve", + "set up and serve", + "run locally", + "local model", + "local assistant", + ], + ) + { + return Some(providers::ChatToolCall { + id: Some("fallback-serve-qwen".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["serve", "qwen", "--engine", "lemonade", "--device", "gpu_required", "--managed"], + "reason": "Start the recommended local assistant after the user approves it." + }), + }); + } + + let mentions_setup = any_substring(&normalized, &["setup", "set up", "install"]); + let asks_how = any_substring(&normalized, &["how", "help", "what do i need"]); + let mentions_rocm_or_therock = any_substring(&normalized, &["therock", "rocm"]); + let requested_install_prefix = requested_install_prefix_from_prompt(prompt); + if requested_install_prefix.is_none() + && let Some(approval) = install_sdk_without_prefix_chat_approval(prompt) + { + return Some(providers::ChatToolCall { + id: Some("fallback-therock-install-folder".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": approval.args, + "reason": "Ask the user to choose the ROCm/TheRock install folder before installing." + }), + }); + } + if mentions_rocm_or_therock + && prompt_requests_install_action(&normalized) + && let Some(prefix) = requested_install_prefix + { + let mut args = vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--prefix".to_owned(), + prefix, + ]; + if let Some(build_date) = requested_therock_build_date_from_prompt(&normalized) { + args.push("--build-date".to_owned()); + args.push(build_date); + } else if let Some(version) = requested_therock_version_from_prompt(&normalized) { + args.push("--version".to_owned()); + args.push(version); + } + return Some(providers::ChatToolCall { + id: Some("fallback-therock-install".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": args, + "reason": "Install TheRock ROCm into the user-selected folder after the user approves it." + }), + }); + } + if asks_how && mentions_setup && any_substring(&normalized, &["therock", "rocm"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-therock-examine".to_owned()), + name: "examine".to_owned(), + arguments: serde_json::json!({}), + }); + } + + let asks_where_installed = any_substring( + &normalized, + &[ + "where is rocm", + "where's rocm", + "where is therock", + "where's therock", + "where did rocm", + "where did therock", + "where rocm is installed", + "where therock is installed", + "rocm install folder", + "therock install folder", + "rocm installed at", + "therock installed at", + ], + ); + let asks_status = asks_where_installed + || any_substring( + &normalized, + &[ + "is rocm installed", + "is therock installed", + "is therock setup", + "is therock set up", + "rocm installed", + "therock installed", + "check this rocm setup", + "which gpu", + "what gpu", + "gpu is on", + "gpu do i have", + "my machine", + ], + ); + if asks_status && any_substring(&normalized, &["gpu", "rocm", "therock", "setup"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-examine".to_owned()), + name: "examine".to_owned(), + arguments: serde_json::json!({}), + }); + } + + None +} + +pub(super) fn prompt_asks_running_or_status(normalized: &str) -> bool { + any_substring( + normalized, + &[ + "running", + "is it up", + "is this up", + "is there", + "are there", + "status", + "started", + "listening", + "on port", + "port ", + ], + ) +} + +fn fallback_config_tool_call_for_prompt(normalized: &str) -> Option { + if !any_substring( + normalized, + &[ + "config", + "setting", + "settings", + "default engine", + "telemetry", + ], + ) { + return None; + } + if any_substring(normalized, &["show", "check", "what", "current", "list"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-config-show".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": ["config", "show"] }), + }); + } + if any_substring(normalized, &["default engine", "set engine", "use engine"]) { + // Word-boundary inference: a bare `contains("aim")` matched "aiming". + if let Some(engine) = infer_engine_from_request(normalized) { + return Some(providers::ChatToolCall { + id: Some("fallback-config-default-engine".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["config", "set-default-engine", engine], + "reason": "Change ROCm CLI's default engine after the user approves it." + }), + }); + } + } + if normalized.contains("telemetry") { + if any_substring(normalized, &["off", "disable", "disabled"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-config-telemetry-off".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["config", "set-telemetry", "off"], + "reason": "Turn ROCm CLI telemetry off after the user approves it." + }), + }); + } + if any_substring(normalized, &["local", "on", "enable", "enabled"]) { + return Some(providers::ChatToolCall { + id: Some("fallback-config-telemetry-local".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ + "args": ["config", "set-telemetry", "local"], + "reason": "Enable local-only ROCm inspection after the user approves it." + }), + }); + } + } + None +} + +pub(super) fn install_sdk_without_prefix_chat_approval( + prompt: &str, +) -> Option { + install_sdk_chat_approval_for_prompt(prompt).and_then(|approval| { + chat_cli_arg_value(&approval.args, "--prefix") + .is_none() + .then_some(approval) + }) +} + +pub(super) fn install_sdk_chat_approval_for_prompt( + prompt: &str, +) -> Option { + let prompt = latest_user_chat_message(prompt); + let normalized = prompt.to_ascii_lowercase(); + if !prompt_requests_rocm_install_or_setup(&normalized) { + return None; + } + let mut args = vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + if normalized.contains("nightly") { + "nightly".to_owned() + } else { + "release".to_owned() + }, + "--format".to_owned(), + "wheel".to_owned(), + ]; + if let Some(prefix) = requested_install_prefix_from_prompt(prompt) { + args.push("--prefix".to_owned()); + args.push(prefix); + } + if let Some(build_date) = requested_therock_build_date_from_prompt(&normalized) { + args.push("--build-date".to_owned()); + args.push(build_date); + } else if let Some(version) = requested_therock_version_from_prompt(&normalized) { + args.push("--version".to_owned()); + args.push(version); + } + Some(ChatToolApprovalRequest { + pending_title: "Install ROCm".to_owned(), + command_title: "Install".to_owned(), + display_command: Some(format_structured_tool_call("rocm", &args)), + args, + explanation: Some( + "Install ROCm/TheRock after the user chooses the install folder.".to_owned(), + ), + }) +} + +fn prompt_requests_rocm_install_or_setup(normalized: &str) -> bool { + let mentions_rocm_stack = any_substring( + normalized, + &[ + "rocm", + "therock", + "the rock", + "amd gpu package", + "amd gpu ready", + "local ai", + ], + ); + if !mentions_rocm_stack { + return false; + } + prompt_requests_install_action(normalized) + || any_substring( + normalized, + &[ + "need rocm", + "need therock", + "need the rock", + "need to get rocm", + "need to get therock", + "get rocm installed", + "get therock installed", + "get the rock installed", + "make rocm work", + "make therock work", + "make the rock work", + "prepare rocm", + "prepare therock", + "prepare the rock", + "set up my amd gpu", + "setup my amd gpu", + "set up local ai", + "setup local ai", + "get local ai working", + "make local ai work", + "rocm please", + "make my amd gpu ready", + "make my gpu ready", + "get my amd gpu ready", + "get my gpu ready", + ], + ) +} + +fn prompt_requests_install_action(normalized: &str) -> bool { + any_substring( + normalized, + &[ + "install this", + "install specific", + "install the", + "install rocm", + "install therock", + "install the rock", + "install amd gpu package", + "install local ai", + "can you install", + "please install", + "rocm please", + "therock please", + "the rock please", + "need to install", + "want to install", + "i need rocm", + "i need therock", + "i need the rock", + "get me rocm", + "get me therock", + "get my gpu ready", + "make my gpu ready", + "make my amd gpu ready", + "get installed", + "setup for me", + "set up for me", + "setup rocm", + "set up rocm", + "setup therock", + "set up therock", + "setup the rock", + "set up the rock", + "setup my gpu", + "set up my gpu", + "setup amd gpu", + "set up amd gpu", + ], + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tool_policy::{ + chat_tool_approval_request, chat_tool_call_is_read_only, normalized_chat_rocm_command_args, + rocm_chat_tool_requested_command, + }; + + #[test] + fn fallback_tool_call_routes_where_installed_to_read_only_examine() { + for prompt in [ + "where is rocm installed?", + "where is TheRock installed?", + "what is the ROCm install folder?", + "where did rocm install to?", + ] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "examine", "{prompt}"); + assert!(chat_tool_call_is_read_only(&call), "{prompt}"); + } + } + + #[test] + fn default_engine_fallback_matches_aim_only_as_a_word() { + let proposed = |prompt: &str| { + fallback_config_tool_call_for_prompt(prompt).map(|call| { + call.arguments["args"][2] + .as_str() + .expect("engine arg") + .to_owned() + }) + }; + assert_eq!( + proposed("change the default engine to aim").as_deref(), + Some("aim") + ); + assert_eq!( + proposed("change the default engine to vllm, aiming for low latency").as_deref(), + Some("vllm") + ); + assert_eq!( + proposed( + "change the default engine so I can maintain my claim, aiming for low latency" + ), + None + ); + } + + #[test] + fn fallback_tool_call_runs_model_support_checks() { + let call = + fallback_rocm_tool_call_for_prompt("Which LLMs can this machine support?").unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec!["model".to_owned()] + ); + + assert!(fallback_rocm_tool_call_for_prompt("Which LLM are you using?").is_none()); + } + + #[test] + fn fallback_tool_call_routes_running_questions_to_status_tools() { + let comfy = fallback_rocm_tool_call_for_prompt("Is ComfyUI running?").unwrap(); + assert_eq!(comfy.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&comfy).unwrap(), + vec!["comfyui".to_owned(), "status".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&comfy)); + + let comfy_port = + fallback_rocm_tool_call_for_prompt("Is ComfyUI running on port 8188?").unwrap(); + assert_eq!(comfy_port.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&comfy_port).unwrap(), + vec!["comfyui".to_owned(), "status".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&comfy_port)); + + for prompt in [ + "Is vLLM running?", + "is lemonade running?", + "is the model server running?", + "is qwen running?", + ] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "rocm_command", "{prompt}"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec!["services".to_owned(), "list".to_owned(), "--all".to_owned(),], + "{prompt}" + ); + assert!(chat_tool_call_is_read_only(&call), "{prompt}"); + } + + let port = fallback_rocm_tool_call_for_prompt("what is running on port 8188?").unwrap(); + assert_eq!(port.name, "port_status"); + assert_eq!( + port.arguments, + serde_json::json!({ "host": DEFAULT_LOCAL_HOST, "port": 8188 }) + ); + assert!(chat_tool_call_is_read_only(&port)); + } + + #[test] + fn fallback_tool_call_routes_engine_install_state_to_engines_list() { + for prompt in ["is vLLM installed?", "is Lemonade available?"] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "rocm_command", "{prompt}"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec!["engines".to_owned(), "list".to_owned()], + "{prompt}" + ); + assert!(chat_tool_call_is_read_only(&call), "{prompt}"); + } + } + + #[test] + fn fallback_tool_call_runs_status_checks() { + for prompt in [ + "Which GPU is on my machine, and is ROCm installed?", + "Is TheRock setup on my machine?", + "Check this ROCm setup.", + ] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "examine"); + assert_eq!(call.arguments, serde_json::json!({})); + } + } + + #[test] + fn fallback_tool_call_routes_therock_setup_to_install_folder_flow() { + let call = fallback_rocm_tool_call_for_prompt("How do I setup TheRock?").unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&call)); + } + + #[test] + fn fallback_tool_call_preserves_requested_therock_install_prefix() { + for (prompt, expected_prefix) in [ + ( + "install TheRock for me in D:\\ROCm\\therock_venvs", + "D:\\ROCm\\therock_venvs", + ), + ("install ROCm to D:\\ROCm\\temp", "D:\\ROCm\\temp"), + ] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--prefix".to_owned(), + expected_prefix.to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&call)); + } + } + + #[test] + fn fallback_tool_call_routes_requested_therock_build_date_install() { + for (prompt, expected_prefix) in [ + ( + "Install this specific TheRock wheel from date 06052026 into D:\\ROCm\\therock_venvs", + "D:\\ROCm\\therock_venvs", + ), + ( + "install ROCm at D:\\ROCm\\temp with build date 2026-06-05", + "D:\\ROCm\\temp", + ), + ] { + let call = fallback_rocm_tool_call_for_prompt(prompt).unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--prefix".to_owned(), + expected_prefix.to_owned(), + "--build-date".to_owned(), + "2026-06-05".to_owned(), + ], + "{prompt}" + ); + assert!(!chat_tool_call_is_read_only(&call)); + } + } + + #[test] + fn fallback_tool_call_does_not_install_therock_without_folder() { + let call = fallback_rocm_tool_call_for_prompt( + "Install this specific TheRock wheel from date 06052026", + ) + .unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--build-date".to_owned(), + "2026-06-05".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&call)); + } + + #[test] + fn chat_how_to_setup_question_opens_install_folder_flow() { + assert!( + install_sdk_without_prefix_chat_approval("How do I setup TheRock?").is_some(), + "a setup question should ask for the install folder" + ); + assert!( + install_sdk_without_prefix_chat_approval("install therock").is_some(), + "a direct install command should open the folder picker" + ); + } + + #[test] + fn chat_install_intent_preserves_bare_folder_path() { + let approval = + install_sdk_chat_approval_for_prompt("install therock D:\\ROCm\\therock_venvs") + .expect("direct install prompt should be recognized"); + + assert_eq!( + approval.args, + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--prefix".to_owned(), + "D:\\ROCm\\therock_venvs".to_owned(), + ] + ); + } + + #[test] + fn fallback_tool_call_routes_requested_therock_exact_version_install() { + let call = fallback_rocm_tool_call_for_prompt( + "Install the TheRock ROCm wheel version 7.13.0a20260605 into D:\\ROCm\\therock_venvs", + ) + .unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + "--prefix".to_owned(), + "D:\\ROCm\\therock_venvs".to_owned(), + "--version".to_owned(), + "7.13.0a20260605".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&call)); + } + + #[test] + fn fallback_tool_call_routes_simple_config_changes() { + let show = fallback_rocm_tool_call_for_prompt("Show current ROCm CLI config").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&show).unwrap(), + vec!["config".to_owned(), "show".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&show)); + + let engine = fallback_rocm_tool_call_for_prompt("Set the default engine to vllm").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&engine).unwrap(), + vec![ + "config".to_owned(), + "set-default-engine".to_owned(), + "vllm".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&engine)); + + let telemetry = + fallback_rocm_tool_call_for_prompt("Disable telemetry in settings").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&telemetry).unwrap(), + vec![ + "config".to_owned(), + "set-telemetry".to_owned(), + "off".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&telemetry)); + } + + #[test] + fn fallback_tool_call_routes_comfyui_support_and_actions() { + let status = fallback_rocm_tool_call_for_prompt("How do I setup ComfyUI?").unwrap(); + assert_eq!(status.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&status).unwrap(), + vec!["comfyui".to_owned(), "status".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&status)); + + let install = fallback_rocm_tool_call_for_prompt("Can you setup ComfyUI for me?").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&install).unwrap(), + vec!["comfyui".to_owned(), "install".to_owned()] + ); + assert!(!chat_tool_call_is_read_only(&install)); + let approval = + chat_tool_approval_request(&install, Some("Install ComfyUI after approval.")).unwrap(); + assert_eq!(approval.pending_title, "Install ComfyUI"); + + let start = fallback_rocm_tool_call_for_prompt("Can you start ComfyUI?").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&start).unwrap(), + vec!["comfyui".to_owned(), "start".to_owned()] + ); + assert!(!chat_tool_call_is_read_only(&start)); + let approval = + chat_tool_approval_request(&start, Some("Start ComfyUI after approval.")).unwrap(); + assert_eq!(approval.pending_title, "Start ComfyUI"); + } + + #[test] + fn fallback_tool_call_routes_local_llm_serve_requests() { + let call = + fallback_rocm_tool_call_for_prompt("Can you setup and serve an LLM for me?").unwrap(); + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec![ + "serve".to_owned(), + "qwen".to_owned(), + "--engine".to_owned(), + "lemonade".to_owned(), + "--device".to_owned(), + "gpu_required".to_owned(), + "--managed".to_owned(), + ] + ); + assert!(!chat_tool_call_is_read_only(&call)); + let approval = + chat_tool_approval_request(&call, Some("Start qwen after approval.")).unwrap(); + assert_eq!(approval.pending_title, "Start local model server"); + assert_eq!( + rocm_chat_tool_requested_command(&call).as_deref(), + Some("rocm serve qwen --engine lemonade --device gpu_required --managed") + ); + } +} diff --git a/apps/rocm/src/chat/mcp_bridge.rs b/apps/rocm/src/chat/mcp_bridge.rs new file mode 100644 index 0000000000..85af28b5aa --- /dev/null +++ b/apps/rocm/src/chat/mcp_bridge.rs @@ -0,0 +1,1305 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The chat assistant's MCP bridge and in-process command execution: the +//! entry point `run_internal_mcp_call` (the tool dispatch behind `rocm +//! mcp-call`, the dash execution seam and the assistant's read-only tool +//! path), the in-process runner for the read-only `rocm` commands the assistant +//! may ask for (`run_rocm_read_only_in_process`), the helpers that wrap a +//! command's output as an MCP tool result, and the `path_exists` and +//! `port_status` tools. `render_services_tool_result_text`, the text of the +//! `services` tool, lives here too, as does `mcp_tool_result_text`, which reads +//! the text back out of an MCP tool result for the turn loop. +//! +//! A child of the `chat` module. It calls into `tool_policy.rs` (the validators +//! and the approval split), `rocm_argv.rs`, `chat/summaries.rs` +//! (`managed_service_running_state`) and the crate-root renderers; nothing here +//! calls back into the turn loop in `chat/turn.rs`, tests included; the turn +//! loop reads this module's items through `super::mcp_bridge`. +//! `run_internal_mcp_call` stays reachable as `chat::run_internal_mcp_call` for +//! `main.rs` and `dash_seam.rs`. + +use super::summaries::managed_service_running_state; +use crate::automations::{render_automations_text, render_daemon_text}; +use crate::engine_runtime_select::app_path_env_vars; +use crate::freeform_planner::{format_structured_tool_call, plan_freeform_request_with_context}; +use crate::managed_service_status::{load_managed_services, managed_service_is_live}; +use crate::rocm_argv::{chat_cli_arg_value, chat_cli_has_flag}; +use crate::runtimes_cmd::render_runtimes_text; +use crate::services_text::{ + read_tail_lines, render_logs_browser_text, render_logs_text, render_service_logs_text, + render_services_text, +}; +use crate::tool_policy::{ + ChatRocmCommandAction, chat_install_prefix_is_system, chat_rocm_command_action, + chat_tool_call_is_read_only, json_string, rocm_chat_tool_requested_args, + validate_chat_tool_call, +}; +use crate::uninstall_plan::render_uninstall_dry_run; +use crate::update_cmd::render_update_text; +use crate::{ + build_codex_bridge_gpu_snapshot, build_codex_bridge_snapshot, + builtin_codex_bridge_engine_inventory, cli_version_string, record_cli_audit_event, + render_config_text, render_engine_inventory_text_with_paths, render_examine_text, + render_examine_text_with_paths, render_model_registry_verbose_text_with_context_and_host, + render_setup_status_text, therock_install_version_selector, +}; +use crate::{comfyui, providers, therock}; +use anyhow::{Context, Result, bail}; +use rocm_core::{ + AppPaths, AutomationRuntimeState, CommandCapture, DEFAULT_LOCAL_HOST, ExamineSummary, + ManagedServiceRecord, RocmCliConfig, command_capture_text, load_recent_automation_events, +}; +use rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES; +use std::fmt::Write as _; +use std::net::{TcpStream, ToSocketAddrs}; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +fn loopback_tcp_port_is_reachable(host: &str, port: u16) -> bool { + let Ok(addresses) = (host, port).to_socket_addrs() else { + return false; + }; + addresses + .into_iter() + .any(|address| TcpStream::connect_timeout(&address, Duration::from_millis(200)).is_ok()) +} + +fn parse_optional_query(args: &[String]) -> Result> { + let mut index = 0; + while index < args.len() { + match args[index].as_str() { + "--query" | "-q" => { + let value = args + .get(index + 1) + .context("logs query flag requires a value")?; + return Ok(Some(value.to_owned())); + } + value if value.starts_with("--") => {} + value => return Ok(Some(value.to_owned())), + } + index += 1; + } + Ok(None) +} + +fn parse_optional_lines(args: &[String]) -> Result { + let mut index = 0; + while index < args.len() { + if matches!(args[index].as_str(), "--lines" | "-n") { + let value = args.get(index + 1).context("lines flag requires a value")?; + return value + .parse::() + .context("lines flag must be a positive number"); + } + index += 1; + } + Ok(DEFAULT_LOG_TAIL_LINES) +} + +pub(super) fn loopback_host_key(host: &str) -> String { + match host.trim().to_ascii_lowercase().as_str() { + "localhost" | "127.0.0.1" => "127.0.0.1".to_owned(), + "::1" | "[::1]" => "::1".to_owned(), + other => other.to_owned(), + } +} + +fn render_services_tool_result_text(records: &[ManagedServiceRecord]) -> String { + let mut output = String::new(); + let _ = writeln!(output, "managed_services: {}", records.len()); + let _ = writeln!( + output, + "status_meaning: ready/running = running; starting/recovering = starting; failed/stopped = not running; no matching row = not managed by ROCm CLI" + ); + if records.is_empty() { + let _ = writeln!(output, "services: none"); + return output; + } + let _ = writeln!(output, "services:"); + for record in records { + let _ = writeln!( + output, + " - service_id={} engine={} model={} canonical_model={} status={} running_state={} endpoint={}", + record.service_id, + record.engine, + record.model_ref, + record.canonical_model_id, + record.status, + managed_service_running_state(&record.status), + record.endpoint_url + ); + } + output +} + +// pub(crate): called from main.rs's own MCP dispatch and from the dash +// execution seam (dash_seam.rs) +pub(crate) fn run_internal_mcp_call( + paths: &AppPaths, + name: &str, + arguments: serde_json::Value, + allow_mutation: bool, +) -> Result { + let arguments = internal_mcp_arguments(arguments); + let call = providers::ChatToolCall { + id: None, + name: name.to_owned(), + arguments: serde_json::Value::Object(arguments.clone()), + }; + validate_chat_tool_call(&call)?; + if !chat_tool_call_is_read_only(&call) && !allow_mutation { + bail!( + "MCP tool `{name}` changes local ROCm state; rerun `rocm mcp-call {name}` with --allow-mutation only after explicit user approval" + ); + } + + match name { + "examine" | "doctor" => { + let examine = ExamineSummary::gather()?; + let text = render_examine_text()?; + Ok(internal_mcp_tool_success(text, serde_json::json!(examine))) + } + "bridge_snapshot" => { + let snapshot = build_codex_bridge_snapshot(paths)?; + Ok(internal_mcp_tool_success( + format!( + "Captured bridge snapshot for {} / {} with default engine `{}`.", + snapshot.examine.os, snapshot.examine.arch, snapshot.examine.default_engine + ), + serde_json::json!(snapshot), + )) + } + "gpu_snapshot" => { + let config = RocmCliConfig::load(paths).unwrap_or_default(); + let gpu = build_codex_bridge_gpu_snapshot(&config); + let status = if !config.telemetry.local_inspection_enabled() { + "GPU telemetry is disabled by rocm-cli config." + } else if gpu.amd_smi_available { + "Captured amd-smi GPU snapshot." + } else { + "Use `rocm examine` for the current local AMD GPU summary." + }; + Ok(internal_mcp_tool_success( + status.to_owned(), + serde_json::json!(gpu), + )) + } + "engines" => { + let engines = builtin_codex_bridge_engine_inventory(); + Ok(internal_mcp_tool_success( + format!("Found {} engine entries.", engines.len()), + serde_json::json!({ "engines": engines }), + )) + } + "services" => { + let services = load_managed_services(paths)?; + Ok(internal_mcp_tool_success( + render_services_tool_result_text(&services), + serde_json::json!({ "services": services }), + )) + } + "port_status" => run_chat_port_status_tool(paths, &call), + "service_logs" => { + let service_id = json_string(&arguments, "service_id") + .context("service_logs requires `service_id`")?; + let lines = arguments + .get("lines") + .and_then(serde_json::Value::as_u64) + .unwrap_or(80) + .clamp(1, 500) as usize; + let record = load_managed_services(paths)? + .into_iter() + .find(|service| service.service_id == service_id) + .with_context(|| format!("managed service `{service_id}` not found"))?; + let tail = read_tail_lines(&record.log_path, lines, "service log")?.join("\n"); + Ok(internal_mcp_tool_success( + format!( + "Read the last {} line(s) from service `{}`.", + lines, record.service_id + ), + serde_json::json!({ + "service": record, + "lines": lines, + "tail": tail, + }), + )) + } + "automations" => { + let event_limit = arguments + .get("event_limit") + .and_then(serde_json::Value::as_u64) + .unwrap_or(10) + .clamp(1, 64) as usize; + let runtime = AutomationRuntimeState::load(paths)?; + let events = load_recent_automation_events(paths, event_limit)?; + Ok(internal_mcp_tool_success( + format!( + "Loaded automation runtime and {} recent events.", + events.len() + ), + serde_json::json!({ + "runtime": runtime, + "recent_events": events, + }), + )) + } + "natural_language_plan" => { + let request = json_string(&arguments, "request") + .context("natural_language_plan requires `request`")?; + let config = RocmCliConfig::load(paths).unwrap_or_default(); + // Built exactly once: the text shown to the operator and the + // `action` reported as what would run next both come from the + // same `StructuredRequestPlan`, so a provider-assisted plan (not + // guaranteed deterministic across calls) cannot describe one + // command in `text` and a different one in `action`. + let (text, action) = plan_freeform_request_with_context(&request, paths, &config); + let action = action.map(|action| { + serde_json::json!({ + "args": action.args, + "approval_required": action.approval_required, + "has_placeholders": action.has_placeholders, + "provider_assisted": action.provider_assisted, + }) + }); + Ok(internal_mcp_tool_success( + "Planned the ROCm request.".to_owned(), + serde_json::json!({ + "request": request, + "text": text, + "action": action, + }), + )) + } + "rocm_command" => { + let action = chat_rocm_command_action(&call)?; + let output = match action { + ChatRocmCommandAction::ReadOnly(args) => { + run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))? + } + ChatRocmCommandAction::Approval { args, .. } if allow_mutation => { + let refs = args.iter().map(String::as_str).collect::>(); + run_rocm_capture_for_paths(paths, &refs, Duration::from_mins(2))? + } + ChatRocmCommandAction::Approval { .. } => { + bail!("rocm_command changes local ROCm state and needs approval") + } + }; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm` command.", + output, + false, + )) + } + "update_check" => { + let output = + run_rocm_command_for_paths(paths, &["update".to_owned()], Duration::from_mins(1))?; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm update`.", + output, + false, + )) + } + "install_sdk_dry_run" => { + let args = internal_mcp_install_sdk_args(&arguments, true)?; + let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm install sdk --dry-run`.", + output, + false, + )) + } + "path_exists" => run_chat_path_exists_tool(&call), + "proposal_action" => { + // Executes IN-PROCESS (no CLI subprocess): show loads a proposal; + // approve/reject mutate its status (allow_mutation already enforced + // by the read-only split above). proposal_action executes in-process + // (via update_automation_proposal_status); it never delegates to the + // subprocess arm. + let proposal_id = json_string(&arguments, "proposal_id") + .context("proposal_action requires `proposal_id`")?; + let action = + json_string(&arguments, "action").context("proposal_action requires `action`")?; + match action.as_str() { + "show" => { + let proposal = rocm_core::find_automation_proposal(paths, &proposal_id) + .with_context(|| { + format!("automation proposal `{proposal_id}` not found") + })?; + Ok(internal_mcp_tool_success( + format!( + "Proposal {} ({}): {}", + proposal.proposal_id, proposal.status, proposal.title + ), + serde_json::json!({ + "id": proposal.proposal_id, + "status": proposal.status, + "summary": proposal.title, + "reason": proposal.message, + }), + )) + } + "approve" | "reject" => { + let status = if action == "approve" { + "approved" + } else { + "rejected" + }; + let updated = + rocm_core::update_automation_proposal_status(paths, &proposal_id, status)?; + record_cli_audit_event( + paths, + "automations", + if action == "approve" { + "proposal_approved" + } else { + "proposal_rejected" + }, + "info", + format!("proposal {proposal_id} {status}"), + None, + ); + Ok(internal_mcp_tool_success( + format!("Proposal {proposal_id} {status}."), + serde_json::json!({ + "id": updated.proposal_id, + "status": updated.status, + "summary": updated.title, + "reason": updated.message, + }), + )) + } + other => bail!("proposal_action `{other}` is not supported"), + } + } + "install_sdk" | "install_engine" | "launch_server" | "stop_server" | "watcher_enable" + | "watcher_disable" => { + let args = rocm_chat_tool_requested_args(&call) + .with_context(|| format!("MCP tool `{name}` is missing required arguments"))?; + let refs = args.iter().map(String::as_str).collect::>(); + let output = run_rocm_capture_for_paths(paths, &refs, Duration::from_mins(2))?; + Ok(internal_mcp_tool_result_from_command( + "Ran approved `rocm` command.", + output, + false, + )) + } + other => bail!("unsupported MCP tool `{other}`"), + } +} + +fn internal_mcp_arguments(value: serde_json::Value) -> serde_json::Map { + if let Some(arguments) = value + .get("arguments") + .and_then(serde_json::Value::as_object) + { + return arguments.clone(); + } + value.as_object().cloned().unwrap_or_default() +} + +fn internal_mcp_tool_success(text: String, structured: serde_json::Value) -> serde_json::Value { + serde_json::json!({ + "content": [ + { + "type": "text", + "text": text, + } + ], + "structuredContent": structured, + "isError": false, + }) +} + +fn internal_mcp_tool_result_from_command( + prefix: &str, + output: CommandCapture, + is_error: bool, +) -> serde_json::Value { + let text = format!("{prefix}\n\n{}", command_capture_text(&output)); + serde_json::json!({ + "content": [ + { + "type": "text", + "text": text, + } + ], + "structuredContent": { + "argv": output.argv, + "exit_status": output.exit_status, + "stdout": output.stdout, + "stderr": output.stderr, + }, + "isError": is_error || output.exit_status != 0, + }) +} + +pub(super) fn mcp_tool_result_text(value: &serde_json::Value) -> String { + value + .get("content") + .and_then(serde_json::Value::as_array) + .map(|items| { + items + .iter() + .filter_map(|item| { + (item.get("type").and_then(serde_json::Value::as_str) == Some("text")) + .then(|| item.get("text").and_then(serde_json::Value::as_str)) + .flatten() + }) + .collect::>() + .join("\n") + }) + .filter(|text| !text.trim().is_empty()) + .unwrap_or_else(|| value.to_string()) +} + +fn run_rocm_capture_for_paths( + paths: &AppPaths, + args: &[&str], + timeout: Duration, +) -> Result { + rocm_core::run_rocm_capture_with_env(args, &app_path_env_vars(paths), timeout) +} + +fn run_rocm_command_for_paths( + paths: &AppPaths, + args: &[String], + _timeout: Duration, +) -> Result { + match run_rocm_read_only_in_process(paths, args) { + Ok(stdout) => Ok(CommandCapture { + argv: std::iter::once("rocm".to_owned()) + .chain(args.iter().cloned()) + .collect(), + exit_status: 0, + stdout, + stderr: String::new(), + }), + Err(error) => Err(error).with_context(|| { + format!( + "read-only assistant command is not implemented in-process: {}", + format_structured_tool_call("rocm", args) + ) + }), + } +} + +fn run_rocm_read_only_in_process(paths: &AppPaths, args: &[String]) -> Result { + let config = RocmCliConfig::load(paths).unwrap_or_default(); + match args { + [] => bail!("rocm command requires at least one argument"), + [command] if command.eq_ignore_ascii_case("examine") => { + render_examine_text_with_paths(paths, &config) + } + [command] + if command.eq_ignore_ascii_case("version") + || command == "--version" + || command == "-V" => + { + Ok(format!("rocm-cli {}\n", cli_version_string())) + } + [command] + if command.eq_ignore_ascii_case("model") || command.eq_ignore_ascii_case("models") => + { + Ok(render_model_registry_verbose_text_with_context_and_host( + Some(paths), + None, + None, + )) + } + [command] if command.eq_ignore_ascii_case("daemon") => { + Ok(render_daemon_text(paths, &config)) + } + [command] if command.eq_ignore_ascii_case("logs") => Ok(render_logs_text(paths)), + [command, rest @ ..] if command.eq_ignore_ascii_case("logs") => { + let query = parse_optional_query(rest)?; + Ok(render_logs_browser_text(paths, query.as_deref())) + } + [command] if command.eq_ignore_ascii_case("runtimes") => { + render_runtimes_text(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("runtimes") + && subcommand.eq_ignore_ascii_case("list") => + { + render_runtimes_text(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("engines") + && subcommand.eq_ignore_ascii_case("list") => + { + Ok(render_engine_inventory_text_with_paths(Some(paths))) + } + [command] if command.eq_ignore_ascii_case("services") => render_services_text(paths, false), + [command, subcommand] + if command.eq_ignore_ascii_case("services") + && subcommand.eq_ignore_ascii_case("list") => + { + render_services_text(paths, false) + } + [command, subcommand, flag] + if command.eq_ignore_ascii_case("services") + && subcommand.eq_ignore_ascii_case("list") + && matches!(flag.as_str(), "-a" | "--all") => + { + render_services_text(paths, true) + } + [command, subcommand, service_id] + if command.eq_ignore_ascii_case("services") + && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => + { + render_service_logs_text(paths, service_id) + } + [command] if command.eq_ignore_ascii_case("automations") => { + render_automations_text(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("automations") + && subcommand.eq_ignore_ascii_case("list") => + { + render_automations_text(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("config") + && subcommand.eq_ignore_ascii_case("show") => + { + Ok(render_config_text(paths, &config)) + } + [command] if command.eq_ignore_ascii_case("comfyui") => { + comfyui::render_status(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("comfyui") + && subcommand.eq_ignore_ascii_case("status") => + { + comfyui::render_status(paths, &config) + } + [command, subcommand, rest @ ..] + if command.eq_ignore_ascii_case("comfyui") + && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => + { + comfyui::render_logs( + paths, + parse_optional_lines(rest).unwrap_or(DEFAULT_LOG_TAIL_LINES), + ) + } + [command, rest @ ..] + if command.eq_ignore_ascii_case("update") + && !rest.iter().any(|arg| arg.eq_ignore_ascii_case("--apply")) => + { + render_update_text(paths) + } + [command, subcommand, rest @ ..] + if command.eq_ignore_ascii_case("install") + && subcommand.eq_ignore_ascii_case("sdk") + && rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dry-run")) => + { + render_install_sdk_dry_run_for_args(paths, rest) + } + [command, rest @ ..] + if command.eq_ignore_ascii_case("uninstall") + && rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dry-run")) => + { + render_uninstall_dry_run(paths) + } + [command] if command.eq_ignore_ascii_case("setup") => { + render_setup_status_text(paths, &config) + } + [command, subcommand] + if command.eq_ignore_ascii_case("setup") + && subcommand.eq_ignore_ascii_case("status") => + { + render_setup_status_text(paths, &config) + } + // `install driver` is a read-only call where the command only reports + // (`chat_install_driver_action`). It runs as a dry run that never + // executes anything, so a call that reaches it elsewhere cannot change + // the host either. `--dry-run` is accepted because it is what this arm + // already does; `--reconcile` is not, since it rewrites the saved driver + // install state and the command refuses to combine it with the other + // flags. + [command, subcommand, rest @ ..] + if command.eq_ignore_ascii_case("install") + && subcommand.eq_ignore_ascii_case("driver") + && rest.iter().all(|arg| { + arg.eq_ignore_ascii_case("--dkms") || arg.eq_ignore_ascii_case("--dry-run") + }) => + { + let dkms = rest.iter().any(|arg| arg.eq_ignore_ascii_case("--dkms")); + crate::driver_install::install_driver(paths, dkms, false, true) + .map(|result| result.output) + .map_err(|error| error.source) + } + _ => bail!( + "unsupported in-process read-only rocm command: {}", + format_structured_tool_call("rocm", args) + ), + } +} + +fn render_install_sdk_dry_run_for_args(paths: &AppPaths, args: &[String]) -> Result { + let channel = chat_cli_arg_value(args, "--channel").unwrap_or("release"); + let format = chat_cli_arg_value(args, "--format").unwrap_or("wheel"); + let prefix = chat_cli_arg_value(args, "--prefix").map(PathBuf::from); + let version = chat_cli_arg_value(args, "--version").map(str::to_owned); + let build_date = chat_cli_arg_value(args, "--build-date").map(str::to_owned); + let selector = therock_install_version_selector(version, build_date)?; + let devel = chat_cli_has_flag(args, "--devel"); + // Dry run, so nothing is displaced and the consent gate is never reached; + // the narrow consent is what this chat surface would pass for a real + // install, and passing `--yes`'s source here would be a lie waiting to be + // printed if the preview ever grew a gate. + Ok(therock::install_sdk( + paths, + therock::SdkInstallRequest { + channel, + format, + prefix, + version_selector: selector, + dry_run: true, + include_devel: devel, + consent: therock::SdkInstallConsent::Preapproved( + therock::SdkInstallApprovalSource::ApproveReplacingActiveDefault, + ), + ..therock::SdkInstallRequest::default() + }, + )? + .output) +} + +fn internal_mcp_install_sdk_args( + arguments: &serde_json::Map, + dry_run: bool, +) -> Result> { + let channel = json_string(arguments, "channel").unwrap_or_else(|| "release".to_owned()); + let format = json_string(arguments, "format").unwrap_or_else(|| "wheel".to_owned()); + let mut argv = vec![ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + channel, + "--format".to_owned(), + format, + ]; + if let Some(prefix) = json_string(arguments, "prefix") { + let prefix_path = Path::new(&prefix); + if chat_install_prefix_is_system(prefix_path) { + bail!( + "install_sdk prefix `{}` is a system folder; choose a user folder instead", + prefix_path.display() + ); + } + argv.push("--prefix".to_owned()); + argv.push(prefix); + } + if let Some(version) = json_string(arguments, "version") { + argv.push("--version".to_owned()); + argv.push(version); + } + if let Some(build_date) = json_string(arguments, "build_date") { + argv.push("--build-date".to_owned()); + argv.push(build_date); + } + if dry_run { + argv.push("--dry-run".to_owned()); + } + Ok(argv) +} + +pub(super) fn run_chat_read_only_tool( + paths: &AppPaths, + call: &providers::ChatToolCall, +) -> Result { + match call.name.as_str() { + "path_exists" => run_chat_path_exists_tool(call), + "port_status" => run_chat_port_status_tool(paths, call), + "rocm_command" => { + let action = chat_rocm_command_action(call)?; + let ChatRocmCommandAction::ReadOnly(args) = action else { + bail!("assistant read-only path cannot run mutating rocm_command"); + }; + let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm` command.", + output, + false, + )) + } + "update_check" => { + let output = + run_rocm_command_for_paths(paths, &["update".to_owned()], Duration::from_mins(1))?; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm update`.", + output, + false, + )) + } + "install_sdk_dry_run" => { + let arguments = internal_mcp_arguments(call.arguments.clone()); + let args = internal_mcp_install_sdk_args(&arguments, true)?; + let output = run_rocm_command_for_paths(paths, &args, Duration::from_mins(2))?; + Ok(internal_mcp_tool_result_from_command( + "Ran `rocm install sdk --dry-run`.", + output, + false, + )) + } + _ => run_internal_mcp_call(paths, &call.name, call.arguments.clone(), false), + } +} + +pub(super) fn run_chat_path_exists_tool( + call: &providers::ChatToolCall, +) -> Result { + let object = call + .arguments + .as_object() + .context("path_exists arguments must be a JSON object")?; + let path = json_string(object, "path").context("path_exists requires path")?; + let path = Path::new(&path); + let metadata = path.metadata().ok(); + let path_kind = metadata.as_ref().map_or("missing", |metadata| { + if metadata.is_dir() { + "directory" + } else if metadata.is_file() { + "file" + } else { + "other" + } + }); + let parent = path.parent(); + let parent_exists = parent.is_some_and(Path::exists); + let parent_display = parent.map_or_else( + || "".to_owned(), + |parent| parent.display().to_string(), + ); + let text = format!( + "path: {}\nexists: {}\nkind: {}\nparent: {}\nparent_exists: {}", + path.display(), + metadata.is_some(), + path_kind, + parent_display, + parent_exists + ); + Ok(serde_json::json!({ + "content": [{ + "type": "text", + "text": text, + }] + })) +} + +fn run_chat_port_status_tool( + paths: &AppPaths, + call: &providers::ChatToolCall, +) -> Result { + let object = call + .arguments + .as_object() + .context("port_status arguments must be a JSON object")?; + let host = json_string(object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); + let port = object + .get("port") + .and_then(serde_json::Value::as_u64) + .context("port_status requires port")? as u16; + let reachable = loopback_tcp_port_is_reachable(&host, port); + let host_key = loopback_host_key(&host); + let matching_services = load_managed_services(paths)? + .into_iter() + .filter(|record| { + record.port == port + && loopback_host_key(&record.host) == host_key + && managed_service_is_live(record) + }) + .collect::>(); + let app_hint = if port == comfyui::default_port() { + Some("ComfyUI default port") + } else if port == rocm_core::DEFAULT_LOCAL_PORT { + Some("ROCm local model server default port") + } else { + None + }; + let mut text = String::new(); + let _ = writeln!(text, "host: {host}"); + let _ = writeln!(text, "port: {port}"); + let _ = writeln!(text, "listening: {reachable}"); + if let Some(app_hint) = app_hint { + let _ = writeln!(text, "hint: {app_hint}"); + } + if matching_services.is_empty() { + let _ = writeln!(text, "managed_service: none"); + } else { + let _ = writeln!(text, "managed_services:"); + for service in &matching_services { + let _ = writeln!( + text, + " - service_id={} engine={} model={} status={} running_state={} endpoint={}", + service.service_id, + service.engine, + service.model_ref, + service.status, + managed_service_running_state(&service.status), + service.endpoint_url + ); + } + } + Ok(serde_json::json!({ + "content": [{ + "type": "text", + "text": text, + }], + "structuredContent": { + "host": host, + "port": port, + "listening": reachable, + "hint": app_hint, + "managed_services": matching_services, + }, + "isError": false, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tests::test_paths; + use std::fs; + use std::net::TcpListener; + + #[test] + fn mcp_tool_result_text_joins_text_content_items() { + let value = serde_json::json!({ + "content": [ + {"type": "text", "text": "line one"}, + {"type": "image", "text": "ignored"}, + {"type": "text", "text": "line two"}, + ] + }); + assert_eq!(mcp_tool_result_text(&value), "line one\nline two"); + } + + #[test] + fn mcp_tool_result_text_falls_back_to_raw_value_without_text_content() { + let value = serde_json::json!({"content": []}); + assert_eq!(mcp_tool_result_text(&value), value.to_string()); + } + + #[test] + fn loopback_tcp_port_is_reachable_detects_listening_and_closed_ports() { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind loopback listener"); + let port = listener.local_addr().expect("local addr").port(); + assert!(loopback_tcp_port_is_reachable("127.0.0.1", port)); + drop(listener); + + // Port 0 can never be connected to; stands in for "closed" without + // depending on a specific unused port staying unused. + assert!(!loopback_tcp_port_is_reachable("127.0.0.1", 0)); + } + + #[test] + fn parse_optional_query_reads_flag_and_positional_forms() { + assert_eq!( + parse_optional_query(&["--query".to_owned(), "gpu".to_owned()]).unwrap(), + Some("gpu".to_owned()) + ); + assert_eq!( + parse_optional_query(&["-q".to_owned(), "gpu".to_owned()]).unwrap(), + Some("gpu".to_owned()) + ); + assert_eq!( + parse_optional_query(&["gpu".to_owned()]).unwrap(), + Some("gpu".to_owned()) + ); + assert_eq!(parse_optional_query(&[]).unwrap(), None); + } + + #[test] + fn parse_optional_query_rejects_a_dangling_flag() { + assert!(parse_optional_query(&["--query".to_owned()]).is_err()); + } + + #[test] + fn parse_optional_query_skips_unknown_flags_to_find_the_positional_value() { + assert_eq!( + parse_optional_query(&["--unknown-flag".to_owned(), "gpu".to_owned()]).unwrap(), + Some("gpu".to_owned()) + ); + } + + #[test] + fn parse_optional_lines_reads_flag_or_defaults() { + assert_eq!( + parse_optional_lines(&["--lines".to_owned(), "25".to_owned()]).unwrap(), + 25 + ); + assert_eq!( + parse_optional_lines(&["-n".to_owned(), "25".to_owned()]).unwrap(), + 25 + ); + assert_eq!(parse_optional_lines(&[]).unwrap(), DEFAULT_LOG_TAIL_LINES); + } + + #[test] + fn parse_optional_lines_rejects_a_non_numeric_value() { + assert!(parse_optional_lines(&["--lines".to_owned(), "nope".to_owned()]).is_err()); + } + + #[test] + fn services_tool_result_text_includes_running_interpretation() { + let (_root, paths) = test_paths("services-tool-text"); + let mut record = ManagedServiceRecord::new( + &paths, + "svc-vllm", + "vllm", + "Qwen/Qwen3.5", + "Qwen/Qwen3.5", + "127.0.0.1", + 11435, + "managed", + std::process::id(), + Some("therock-release".to_owned()), + None, + Some("gpu_required".to_owned()), + ); + record.status = "ready".to_owned(); + + let rendered = render_services_tool_result_text(&[record]); + + assert!(rendered.contains("status_meaning: ready/running = running")); + assert!(rendered.contains("engine=vllm")); + assert!(rendered.contains("running_state=running")); + } + + #[test] + fn proposal_action_approve_updates_status() { + let (root, paths) = test_paths("proposal-approve"); + // Seed a pending proposal. + let proposal = rocm_core::AutomationProposalRecord { + at_unix_ms: rocm_core::unix_time_millis(), + proposal_id: "prop-approve-1".to_owned(), + watcher_id: "therock-update".to_owned(), + action: "prepare_driver_plan".to_owned(), + title: "Apply driver plan".to_owned(), + message: "A reviewed driver plan is ready.".to_owned(), + status: "pending".to_owned(), + service_id: None, + tool: None, + arguments: serde_json::Value::Null, + reviewed_at_unix_ms: None, + }; + rocm_core::append_automation_proposal(&paths, &proposal).expect("seed proposal"); + + // show is read-only and returns the proposal. + let shown = run_internal_mcp_call( + &paths, + "proposal_action", + serde_json::json!({ "proposal_id": "prop-approve-1", "action": "show" }), + false, + ) + .expect("show ok"); + assert_eq!(shown["structuredContent"]["status"], "pending"); + + // approve requires allow_mutation. + assert!( + run_internal_mcp_call( + &paths, + "proposal_action", + serde_json::json!({ "proposal_id": "prop-approve-1", "action": "approve" }), + false, + ) + .is_err(), + "approve without allow_mutation must bail" + ); + + // approve with allow_mutation sets status to approved. + let approved = run_internal_mcp_call( + &paths, + "proposal_action", + serde_json::json!({ "proposal_id": "prop-approve-1", "action": "approve" }), + true, + ) + .expect("approve ok"); + assert_eq!(approved["structuredContent"]["status"], "approved"); + let stored = rocm_core::find_automation_proposal(&paths, "prop-approve-1") + .expect("proposal still present"); + assert_eq!(stored.status, "approved"); + + let _ = fs::remove_dir_all(&root); + } + + #[test] + fn proposal_action_reject_updates_status() { + let (root, paths) = test_paths("proposal-reject"); + let proposal = rocm_core::AutomationProposalRecord { + at_unix_ms: rocm_core::unix_time_millis(), + proposal_id: "prop-reject-1".to_owned(), + watcher_id: "server-recover".to_owned(), + action: "queue_stop_server_proposal".to_owned(), + title: "Stop overheating server".to_owned(), + message: "GPU thermal pressure detected.".to_owned(), + status: "pending".to_owned(), + service_id: None, + tool: None, + arguments: serde_json::Value::Null, + reviewed_at_unix_ms: None, + }; + rocm_core::append_automation_proposal(&paths, &proposal).expect("seed proposal"); + + let rejected = run_internal_mcp_call( + &paths, + "proposal_action", + serde_json::json!({ "proposal_id": "prop-reject-1", "action": "reject" }), + true, + ) + .expect("reject ok"); + assert_eq!(rejected["structuredContent"]["status"], "rejected"); + let stored = rocm_core::find_automation_proposal(&paths, "prop-reject-1") + .expect("proposal still present"); + assert_eq!(stored.status, "rejected"); + + let _ = fs::remove_dir_all(&root); + } + + #[test] + fn proposal_action_show_missing_proposal_errors() { + let (root, paths) = test_paths("proposal-missing"); + assert!( + run_internal_mcp_call( + &paths, + "proposal_action", + serde_json::json!({ "proposal_id": "nope", "action": "show" }), + false, + ) + .is_err(), + "showing a missing proposal must error" + ); + let _ = fs::remove_dir_all(&root); + } + + #[test] + fn assistant_read_only_rocm_commands_do_not_fallback_to_child_process() { + let (_root, paths) = test_paths("readonly-rocm-in-process-only"); + let args = vec!["services".to_owned(), "status".to_owned()]; + + let error = run_rocm_command_for_paths(&paths, &args, Duration::from_secs(1)) + .unwrap_err() + .to_string(); + + assert!(error.contains("read-only assistant command is not implemented in-process")); + assert!(error.contains("rocm services status")); + assert!(!paths.data_dir.join("logs").exists()); + } + + #[test] + fn internal_mcp_read_only_rocm_command_runs_in_process() { + let (_root, paths) = test_paths("mcp-readonly-rocm-in-process"); + + let result = run_internal_mcp_call( + &paths, + "rocm_command", + serde_json::json!({ "args": ["version"] }), + false, + ) + .expect("read-only rocm mcp-call should run"); + + assert_eq!( + result.get("isError").and_then(serde_json::Value::as_bool), + Some(false) + ); + assert_eq!( + result + .pointer("/structuredContent/argv/0") + .and_then(serde_json::Value::as_str), + Some("rocm") + ); + assert!(mcp_tool_result_text(&result).contains(env!("CARGO_PKG_VERSION"))); + } + + #[test] + fn natural_language_plan_returns_structured_mutating_action() { + let (_root, paths) = test_paths("nl-plan-structured-mutating"); + + let result = run_internal_mcp_call( + &paths, + "natural_language_plan", + serde_json::json!({ + "request": "install TheRock into /opt/rocm-target" + }), + false, + ) + .expect("natural_language_plan should plan the request"); + + assert_eq!( + result.get("isError").and_then(serde_json::Value::as_bool), + Some(false) + ); + // text (rendered plan) preserved. + assert!( + !result + .pointer("/structuredContent/text") + .and_then(serde_json::Value::as_str) + .unwrap_or_default() + .trim() + .is_empty() + ); + // A known install folder yields a mutating, placeholder-free action. + assert_eq!( + result + .pointer("/structuredContent/action/approval_required") + .and_then(serde_json::Value::as_bool), + Some(true) + ); + assert_eq!( + result + .pointer("/structuredContent/action/has_placeholders") + .and_then(serde_json::Value::as_bool), + Some(false) + ); + let args = result + .pointer("/structuredContent/action/args") + .and_then(serde_json::Value::as_array) + .expect("action args should be present"); + assert!(args.iter().any(|arg| arg.as_str() == Some("install"))); + assert!(args.iter().any(|arg| arg.as_str() == Some("--prefix"))); + } + + #[test] + fn natural_language_plan_returns_placeholder_action_when_incomplete() { + let (_root, paths) = test_paths("nl-plan-structured-placeholder"); + + let result = run_internal_mcp_call( + &paths, + "natural_language_plan", + serde_json::json!({ "request": "serve" }), + false, + ) + .expect("natural_language_plan should plan the request"); + + assert_eq!( + result + .pointer("/structuredContent/action/has_placeholders") + .and_then(serde_json::Value::as_bool), + Some(true) + ); + } + + /// On Windows `rocm install driver` only reports, and chat plans it as a + /// read-only call. That call must run in process: before this arm existed it + /// failed with "unsupported in-process read-only rocm command". It is a dry + /// run, so it never executes anything on any platform. + #[test] + fn install_driver_read_only_call_runs_in_process() { + let (_root, paths) = test_paths("install-driver-in-process"); + let args = vec!["install".to_owned(), "driver".to_owned()]; + let text = run_rocm_read_only_in_process(&paths, &args) + .expect("the read-only install driver call should run in process"); + assert!(text.contains("approval:"), "{text}"); + assert!(!text.contains("driver install completed"), "{text}"); + assert!( + run_rocm_read_only_in_process( + &paths, + &[ + "install".to_owned(), + "driver".to_owned(), + "--yes".to_owned() + ] + ) + .is_err(), + "a flag the read-only path does not know is still refused" + ); + } + + /// `--dry-run` is what the in-process arm already does, so it is accepted + /// and the call reports a dry run. `--reconcile` writes driver install state + /// and stays refused, as does any flag the arm does not know. + #[test] + fn install_driver_read_only_call_accepts_dry_run_and_refuses_reconcile() { + let (_root, paths) = test_paths("install-driver-in-process-dry-run"); + let run = |extra: &[&str]| { + let mut args = vec!["install".to_owned(), "driver".to_owned()]; + args.extend(extra.iter().map(|arg| (*arg).to_owned())); + run_rocm_read_only_in_process(&paths, &args) + }; + for extra in [&["--dry-run"][..], &["--dkms", "--dry-run"], &["--DRY-RUN"]] { + let text = run(extra).unwrap_or_else(|error| panic!("{extra:?} must run: {error}")); + assert!(text.contains("dry_run: true"), "{extra:?}: {text}"); + assert!( + !text.contains("driver install completed"), + "{extra:?}: {text}" + ); + } + for extra in [&["--reconcile"][..], &["--dry-run", "--reconcile"]] { + assert!(run(extra).is_err(), "{extra:?} must stay refused"); + } + // Refusing `--reconcile` changed nothing on disk: the state file is + // written only by a reconcile or an executed install. + assert!( + !crate::driver_install::driver_install_state_path(&paths).exists(), + "a refused call must not create driver install state" + ); + } + + #[test] + fn setup_status_renders_in_process() { + let (_root, paths) = test_paths("setup-status-in-process"); + + for args in [ + vec!["setup".to_owned()], + vec!["setup".to_owned(), "status".to_owned()], + ] { + let text = run_rocm_read_only_in_process(&paths, &args) + .expect("setup status read path should render in-process"); + assert!( + !text.trim().is_empty(), + "setup status text should be non-empty for {args:?}" + ); + assert!( + text.contains("ROCm setup"), + "setup status text should be reached for {args:?}: {text}" + ); + } + } + + #[test] + fn port_status_matches_loopback_managed_services() -> Result<()> { + let (root, paths) = test_paths("port-status-loopback"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "svc-comfyui", + "comfyui", + "ComfyUI", + "ComfyUI", + "127.0.0.1", + 18188, + "managed", + std::process::id(), + Some("therock-release".to_owned()), + None, + Some("gpu_required".to_owned()), + ); + record.status = "ready".to_owned(); + record.write()?; + + let call = providers::ChatToolCall { + id: Some("port-check".to_owned()), + name: "port_status".to_owned(), + arguments: serde_json::json!({ "host": "localhost", "port": 18188 }), + }; + let result = run_chat_port_status_tool(&paths, &call)?; + let text = mcp_tool_result_text(&result); + let managed_service_count = result + .get("structuredContent") + .and_then(|content| content.get("managed_services")) + .and_then(serde_json::Value::as_array) + .map_or(0, Vec::len); + let _ = fs::remove_dir_all(root); + + assert_eq!(managed_service_count, 1); + assert!(text.contains("managed_services:"), "{text}"); + assert!(text.contains("service_id=svc-comfyui"), "{text}"); + assert!(text.contains("running_state=starting"), "{text}"); + Ok(()) + } +} diff --git a/apps/rocm/src/chat/summaries.rs b/apps/rocm/src/chat/summaries.rs new file mode 100644 index 0000000000..7415ec0ba9 --- /dev/null +++ b/apps/rocm/src/chat/summaries.rs @@ -0,0 +1,804 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Deterministic summaries of the read-only tool output the chat assistant +//! reads back: `examine`, the model recipes, ComfyUI status, the managed +//! services, a port check and the engine inventory. Each +//! `deterministic_*_tool_summary` function recognises one tool's text, reads +//! the values it needs out of it and returns a short plain-language summary, +//! or `None` when the text is not that tool's. `deterministic_chat_tool_summary` +//! is the entry point the turn loop calls; when it returns a summary the chat +//! shows it instead of asking the local model to restate the output. +//! +//! The parsers that turn tool text back into rows (`DeterministicModelRecipe`, +//! `DeterministicServiceRow`, `DeterministicEngineRow`) and +//! `managed_service_running_state`, the mapping from a managed service's status +//! to `running`, `starting`, `not_running` or `unknown`, live here too. The +//! services tool text itself is rendered by `chat/mcp_bridge.rs` +//! (`render_services_tool_result_text`), which reads +//! `managed_service_running_state` from this module. +//! +//! A leaf of the `chat` module: nothing here calls back into `chat.rs`. The +//! only edges out are `providers` (the assistant model ids) and +//! `managed_service_status::friendly_engine_label`. + +use crate::managed_service_status::friendly_engine_label; +use crate::providers; +use rocm_core::placeholder::empty_as_unknown; +use std::fmt::Write as _; + +#[derive(Debug, Default)] +struct DeterministicModelRecipe { + pub(crate) canonical_id: String, + pub(crate) min_gpu_mem_gib: Option, + pub(crate) engines: Vec, + pub(crate) engine_statuses: Vec, + pub(crate) warnings: Vec, +} + +fn parse_deterministic_model_recipes(tool_text: &str) -> Vec { + let mut recipes = Vec::new(); + let mut current: Option = None; + for line in tool_text.lines() { + let trimmed = line.trim(); + if let Some(recipe) = parse_model_recipe_header(trimmed) { + if let Some(previous) = current.replace(recipe) { + recipes.push(previous); + } + continue; + } + let Some(recipe) = current.as_mut() else { + continue; + }; + if is_engine_status_line(trimmed) { + recipe.engine_statuses.push(trimmed.to_owned()); + } else if let Some(warning) = trimmed.strip_prefix("warning:") { + recipe.warnings.push(warning.trim().to_owned()); + } + } + if let Some(recipe) = current { + recipes.push(recipe); + } + recipes +} + +fn parse_model_recipe_header(line: &str) -> Option { + if !line.contains(" aliases=[") || !line.contains("min_gpu_mem=") { + return None; + } + let canonical_id = line.split_whitespace().next()?.to_owned(); + let engines = parse_bracketed_csv(line, "engines=[", "]"); + let min_gpu_mem_gib = line + .split_once("min_gpu_mem=") + .and_then(|(_, rest)| rest.split_whitespace().next()) + .and_then(|value| value.parse::().ok()); + Some(DeterministicModelRecipe { + canonical_id, + min_gpu_mem_gib, + engines, + engine_statuses: Vec::new(), + warnings: Vec::new(), + }) +} + +fn parse_bracketed_csv(line: &str, start: &str, end: &str) -> Vec { + line.split_once(start) + .and_then(|(_, rest)| rest.split_once(end)) + .map(|(values, _)| { + values + .split(',') + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) + .collect() + }) + .unwrap_or_default() +} + +fn is_engine_status_line(line: &str) -> bool { + matches!( + line.split_once(':').map(|(engine, _)| engine), + Some("lemonade" | "vllm" | "aim") + ) +} + +#[derive(Debug, Default)] +struct DeterministicServiceRow { + pub(crate) service_id: String, + pub(crate) engine: String, + pub(crate) model: String, + pub(crate) status: String, + pub(crate) running_state: String, + pub(crate) endpoint: String, +} + +fn parse_deterministic_service_rows(tool_text: &str) -> Vec { + let mut rows = Vec::new(); + let mut current: Option = None; + for line in tool_text.lines() { + let trimmed = line.trim(); + if let Some(rest) = trimmed.strip_prefix("- ") { + if rest.starts_with("service_id=") { + if let Some(row) = current.take() { + rows.push(row); + } + let mut row = DeterministicServiceRow::default(); + for token in rest.split_whitespace() { + if let Some((key, value)) = token.split_once('=') { + match key { + "service_id" => row.service_id = value.to_owned(), + "engine" => row.engine = value.to_owned(), + "model" => row.model = value.to_owned(), + "status" => row.status = value.to_owned(), + "running_state" => row.running_state = value.to_owned(), + "endpoint" => row.endpoint = value.to_owned(), + _ => {} + } + } + } + if row.running_state.is_empty() { + row.running_state = managed_service_running_state(&row.status).to_owned(); + } + current = Some(row); + continue; + } + if let Some(row) = current.take() { + rows.push(row); + } + current = Some(DeterministicServiceRow { + service_id: rest.to_owned(), + ..Default::default() + }); + continue; + } + let Some(row) = current.as_mut() else { + continue; + }; + if let Some(value) = trimmed.strip_prefix("status:") { + row.status = value.trim().to_owned(); + row.running_state = managed_service_running_state(&row.status).to_owned(); + } else if let Some(value) = trimmed.strip_prefix("engine:") { + row.engine = value.trim().to_owned(); + } else if let Some(value) = trimmed.strip_prefix("model:") { + row.model = value.trim().to_owned(); + } else if let Some(value) = trimmed.strip_prefix("endpoint:") { + row.endpoint = value.trim().to_owned(); + } + } + if let Some(row) = current { + rows.push(row); + } + rows +} + +#[derive(Debug, Default)] +struct DeterministicEngineRow { + pub(crate) engine: String, + pub(crate) runtime: String, + pub(crate) note: String, +} + +fn parse_deterministic_engine_rows(tool_text: &str) -> Vec { + let mut rows = Vec::new(); + let mut current: Option = None; + for line in tool_text.lines() { + let trimmed = line.trim().trim_start_matches("* ").trim_start(); + if let Some(engine) = engine_name_from_inventory_line(trimmed) { + if let Some(row) = current.take() { + rows.push(row); + } + current = Some(DeterministicEngineRow { + engine: engine.to_owned(), + ..Default::default() + }); + continue; + } + let Some(row) = current.as_mut() else { + continue; + }; + if let Some(runtime) = trimmed.strip_prefix("runtime:") { + row.runtime = runtime.trim().to_owned(); + } else if let Some(note) = trimmed.strip_prefix("note:") { + row.note = note.trim().to_owned(); + } + } + if let Some(row) = current { + rows.push(row); + } + rows +} + +fn engine_name_from_inventory_line(line: &str) -> Option<&'static str> { + ["lemonade", "vllm", "aim"].into_iter().find(|engine| { + line == *engine + || line + .strip_prefix(*engine) + .is_some_and(|rest| rest.starts_with(char::is_whitespace)) + }) +} + +pub(super) fn managed_service_running_state(status: &str) -> &'static str { + match status { + "ready" | "running" => "running", + "starting" | "recovering" => "starting", + "failed" | "stopped" => "not_running", + _ => "unknown", + } +} + +pub(super) fn deterministic_rocm_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("examine:") { + return None; + } + let mut lines = Vec::new(); + let gpu = chat_tool_value(tool_text, "driver_detail") + .filter(|value| value != "") + .or_else(|| { + let target = chat_tool_value(tool_text, "detected_gfx_target")?; + Some(format!("AMD GPU target {target}")) + }); + if let Some(gpu) = gpu { + lines.push(format!(" GPU: {gpu}")); + } + + let active_status = chat_tool_value(tool_text, "active_runtime_status"); + if active_status.as_deref() == Some("ready") { + let family = chat_tool_value(tool_text, "active_runtime_family") + .filter(|value| value != "" && value != ""); + let version = chat_tool_value(tool_text, "active_runtime_version") + .filter(|value| value != "" && value != ""); + let mut detail = "installed and active for ROCm CLI".to_owned(); + if let Some(family) = family { + let _ = write!(detail, " ({family})"); + } + if let Some(version) = version { + let _ = write!(detail, ", {version}"); + } + lines.push(format!(" ROCm/TheRock: {detail}")); + if let Some(root) = chat_tool_value(tool_text, "active_runtime_root") + .or_else(|| chat_tool_value(tool_text, "setup_runtime_root")) + .filter(|value| value != "" && value != "" && value != "") + { + lines.push(format!(" Install folder: {root}")); + } + if let Some(cache) = chat_tool_value(tool_text, "active_runtime_pip_cache_dir") + .or_else(|| chat_tool_value(tool_text, "setup_runtime_pip_cache_dir")) + .filter(|value| value != "" && value != "" && value != "") + { + lines.push(format!(" Downloads/cache: {cache}")); + } + } else if let Some(status) = active_status.as_deref() { + lines.push(format!(" ROCm/TheRock: active runtime status is {status}")); + if let Some(root) = chat_tool_value(tool_text, "setup_runtime_root") + .filter(|value| value != "" && value != "" && value != "") + { + lines.push(format!(" Selected setup folder: {root}")); + } + if let Some(cache) = chat_tool_value(tool_text, "setup_runtime_pip_cache_dir") + .filter(|value| value != "" && value != "" && value != "") + { + lines.push(format!(" Downloads/cache: {cache}")); + } + } + + if chat_tool_value(tool_text, "legacy_rocm_status").as_deref() == Some("not_detected") + && active_status.as_deref() == Some("ready") + { + lines.push( + " Note: no global legacy ROCm install was found; ROCm CLI is using its managed TheRock runtime." + .to_owned(), + ); + } + + (!lines.is_empty()).then(|| lines.join("\n")) +} + +pub(super) fn deterministic_chat_tool_summary(tool_text: &str) -> Option { + deterministic_rocm_tool_summary(tool_text) + .or_else(|| deterministic_model_tool_summary(tool_text)) + .or_else(|| deterministic_combined_status_tool_summary(tool_text)) +} + +fn deterministic_combined_status_tool_summary(tool_text: &str) -> Option { + let summaries = [ + deterministic_comfyui_tool_summary(tool_text), + deterministic_port_status_tool_summary(tool_text), + deterministic_engine_inventory_tool_summary(tool_text), + deterministic_services_tool_summary(tool_text), + ] + .into_iter() + .flatten() + .collect::>(); + (!summaries.is_empty()).then(|| summaries.join("\n")) +} + +pub(super) fn deterministic_summary_can_stand_alone(tool_text: &str) -> bool { + !tool_text.contains("model recipes") +} + +fn deterministic_model_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("model recipes") { + return None; + } + let recipes = parse_deterministic_model_recipes(tool_text); + if recipes.is_empty() { + return None; + } + + let mut lines = Vec::new(); + if let Some(recipe) = recipes + .iter() + .find(|recipe| recipe.canonical_id == providers::LEMONADE_ASSISTANT_MODEL_ID) + { + lines.push(format!( + " Recommended local assistant: qwen ({})", + recipe.canonical_id + )); + lines.push(format!( + " Fit: needs about {} GPU memory.", + recipe + .min_gpu_mem_gib + .map_or_else(|| "unknown".to_owned(), |value| format!("{value} GiB")) + )); + lines.push(format!( + " Engine: {}", + deterministic_engine_fit_summary(recipe, "lemonade") + )); + } + + if let Some(recipe) = recipes + .iter() + .find(|recipe| recipe.canonical_id == "Qwen3-0.6B-GGUF") + { + lines.push(format!( + " Tiny smoke test: qwen-smoke ({})", + recipe.canonical_id + )); + lines.push( + " Use this only to check that GPU serving starts; it is not the default assistant." + .to_owned(), + ); + } + + if let Some(recipe) = recipes + .iter() + .find(|recipe| recipe.canonical_id == "meta-llama/Llama-3.2-3B-Instruct") + { + lines.push(format!( + " 8 GiB-class option: llama ({})", + recipe.canonical_id + )); + lines.push(format!( + " Fit: asks for {}; this can be tight on APUs and depends on available shared GPU memory.", + recipe + .min_gpu_mem_gib.map_or_else(|| "unknown GPU memory".to_owned(), |value| format!("{value} GiB")) + )); + lines.push(format!( + " Engines: {}", + deterministic_available_engine_names(recipe).join(", ") + )); + } + + let larger = recipes + .iter() + .filter(|recipe| recipe.min_gpu_mem_gib.is_some_and(|value| value > 8)) + .filter(|recipe| recipe.canonical_id != providers::BUILTIN_ASSISTANT_MODEL_ID) + .map(|recipe| { + format!( + "{} asks for {}", + recipe.canonical_id, + recipe.min_gpu_mem_gib.map_or_else( + || "more GPU memory".to_owned(), + |value| format!("{value} GiB") + ) + ) + }) + .collect::>(); + if !larger.is_empty() { + lines.push(format!( + " Larger recipes are not low-VRAM defaults: {}.", + larger.join("; ") + )); + } + + let linux_wsl_only = recipes + .iter() + .flat_map(|recipe| { + recipe + .engine_statuses + .iter() + .filter(|status| { + status.contains("unsupported_native_windows") + || status.to_ascii_lowercase().contains("wsl/linux") + || status.to_ascii_lowercase().contains("linux/wsl") + }) + .filter_map(|status| { + let (engine, _) = status.split_once(':')?; + Some(format!( + "{} uses {} through WSL/Linux on Windows", + recipe.canonical_id, engine + )) + }) + }) + .collect::>(); + if !linux_wsl_only.is_empty() { + lines.push(format!( + " Native Windows note: {}.", + linux_wsl_only.join("; ") + )); + } + + if !lines.is_empty() { + lines.push( + " Run `rocm examine` to refresh GPU memory details before starting anything large." + .to_owned(), + ); + } + (!lines.is_empty()).then(|| lines.join("\n")) +} + +fn deterministic_engine_fit_summary(recipe: &DeterministicModelRecipe, engine: &str) -> String { + let status = recipe + .engine_statuses + .iter() + .find(|status| status.starts_with(&format!("{engine}:"))) + .map(std::string::String::as_str); + match status { + Some(status) if status.contains("unsupported_native_windows") => { + format!("{engine}: WSL/Linux only on Windows") + } + Some(status) + if status.split_once(':').is_some_and(|(_, rest)| { + rest.trim_start().starts_with("available") + || rest.trim_start().starts_with("adapter_available") + }) => + { + format!("{engine}: available") + } + Some(status) => status.to_owned(), + None if recipe.engines.iter().any(|candidate| candidate == engine) => { + format!("{engine}: listed") + } + None => format!("{engine}: not listed"), + } +} + +fn deterministic_available_engine_names(recipe: &DeterministicModelRecipe) -> Vec { + let engines = recipe + .engine_statuses + .iter() + .filter_map(|status| { + let (engine, rest) = status.split_once(':')?; + (rest.trim_start().starts_with("available") + || rest.trim_start().starts_with("adapter_available")) + .then_some(engine.to_owned()) + }) + .collect::>(); + if engines.is_empty() { + recipe.engines.clone() + } else { + engines + } +} + +pub(crate) fn deterministic_comfyui_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("ComfyUI") || !tool_text.contains("Running") { + return None; + } + let mut lines = Vec::new(); + match chat_tool_value(tool_text, "installed").as_deref() { + Some("yes") => lines.push(" ComfyUI: installed.".to_owned()), + Some("no") => lines.push(" ComfyUI: not installed.".to_owned()), + Some(value) => lines.push(format!(" ComfyUI: installed status is {value}.")), + None => {} + } + if let Some(status) = chat_tool_value(tool_text, "status") { + let running_text = if status.eq_ignore_ascii_case("running") { + "running".to_owned() + } else if status.eq_ignore_ascii_case("starting") { + "starting".to_owned() + } else { + format!("not running ({status})") + }; + lines.push(format!(" Running: {running_text}.")); + } + if let Some(models_path) = chat_tool_value(tool_text, "models path") { + lines.push(format!(" Models folder: {models_path}.")); + } + // A note about the install itself (an interrupted reinstall, which names + // the command that finishes it) sits between `status`'s `ComfyUI` heading + // and its `Running` section, whose own notes the `Running:` line sums up. + for line in tool_text + .lines() + .skip_while(|line| line.trim() != "ComfyUI") + .take_while(|line| line.trim() != "Running") + { + if let Some(note) = line.trim().strip_prefix("note:").map(str::trim) + && !note.is_empty() + { + lines.push(format!(" Note: {note}.")); + } + } + (!lines.is_empty()).then(|| lines.join("\n")) +} + +fn deterministic_services_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("Local Servers") + && !tool_text.contains("managed_services:") + && !tool_text.contains("services:") + { + return None; + } + let rows = parse_deterministic_service_rows(tool_text); + let mut lines = Vec::new(); + let live = rows + .iter() + .filter(|row| matches!(row.running_state.as_str(), "running" | "starting")) + .collect::>(); + if live.is_empty() { + lines.push(" Local model servers: none running under ROCm CLI.".to_owned()); + } else { + lines.push(format!( + " Local model servers running/starting: {}.", + live.len() + )); + for row in live.iter().take(4) { + lines.push(format!( + " {} {} ({}) at {}.", + row.engine, + row.model, + row.running_state, + empty_as_unknown(&row.endpoint) + )); + } + } + let past = rows + .iter() + .filter(|row| !matches!(row.running_state.as_str(), "running" | "starting")) + .take(3) + .map(|row| { + format!( + "{} {} {}", + empty_as_unknown(&row.engine), + empty_as_unknown(&row.model), + empty_as_unknown(&row.status) + ) + }) + .collect::>(); + if !past.is_empty() { + lines.push(format!(" Past/non-running records: {}.", past.join("; "))); + } + (!lines.is_empty()).then(|| lines.join("\n")) +} + +fn deterministic_port_status_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("port_status:") && !tool_text.contains("listening:") { + return None; + } + let port = chat_tool_value(tool_text, "port")?; + let listening = chat_tool_value(tool_text, "listening").unwrap_or_else(|| "unknown".to_owned()); + let mut lines = Vec::new(); + let state = match listening.as_str() { + "true" => "listening", + "false" => "not listening", + _ => "unknown", + }; + lines.push(format!(" Port {port}: {state}.")); + if let Some(hint) = chat_tool_value(tool_text, "hint") { + lines.push(format!(" Hint: {hint}.")); + } + if chat_tool_value(tool_text, "managed_service").as_deref() == Some("none") { + lines.push(" Managed service: none on that endpoint.".to_owned()); + } + Some(lines.join("\n")) +} + +fn deterministic_engine_inventory_tool_summary(tool_text: &str) -> Option { + if !tool_text.contains("Local model engines") { + return None; + } + let rows = parse_deterministic_engine_rows(tool_text); + if rows.is_empty() { + return None; + } + let mut lines = vec![" Engine runtimes:".to_owned()]; + for row in rows { + let mut line = format!( + " {}: {}", + friendly_engine_label(&row.engine), + empty_as_unknown(&row.runtime) + ); + if !row.note.trim().is_empty() { + let _ = write!(line, " ({})", row.note.trim()); + } + line.push('.'); + lines.push(line); + } + Some(lines.join("\n")) +} + +fn chat_tool_value(tool_text: &str, key: &str) -> Option { + let prefix = format!("{key}:"); + tool_text.lines().find_map(|line| { + line.trim() + .strip_prefix(&prefix) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parse_deterministic_model_recipes_reads_header_fields_and_notes() { + let recipes = parse_deterministic_model_recipes( + "\ + Qwen3-4B-Instruct-2507-GGUF aliases=[qwen, lemonade-qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] +lemonade: ready +warning: slow on this GPU + Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] +vllm: not_installed +", + ); + assert_eq!(recipes.len(), 2); + assert_eq!(recipes[0].canonical_id, "Qwen3-4B-Instruct-2507-GGUF"); + assert_eq!(recipes[0].min_gpu_mem_gib, Some(4)); + assert_eq!(recipes[0].engines, vec!["lemonade".to_owned()]); + assert_eq!( + recipes[0].engine_statuses, + vec!["lemonade: ready".to_owned()] + ); + assert_eq!(recipes[0].warnings, vec!["slow on this GPU".to_owned()]); + assert_eq!(recipes[1].canonical_id, "Qwen/Qwen3.5-4B"); + assert_eq!(recipes[1].min_gpu_mem_gib, Some(12)); + } + + #[test] + fn parse_deterministic_model_recipes_requires_aliases_bracket_in_header() { + let recipes = parse_deterministic_model_recipes( + "\ + NotAHeader task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] + Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] +", + ); + assert_eq!(recipes.len(), 1); + assert_eq!(recipes[0].canonical_id, "Qwen/Qwen3.5-4B"); + } + + #[test] + fn parse_deterministic_service_rows_reads_inline_key_value_form() { + let rows = parse_deterministic_service_rows( + " - service_id=svc-1 engine=vllm model=qwen status=ready running_state=running endpoint=127.0.0.1:8000", + ); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].service_id, "svc-1"); + assert_eq!(rows[0].engine, "vllm"); + assert_eq!(rows[0].status, "ready"); + assert_eq!(rows[0].running_state, "running"); + assert_eq!(rows[0].endpoint, "127.0.0.1:8000"); + } + + #[test] + fn parse_deterministic_service_rows_derives_running_state_when_absent() { + let rows = parse_deterministic_service_rows( + "\ +- svc-2 +status: starting +engine: lemonade +model: qwen +endpoint: 127.0.0.1:8001 +", + ); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].service_id, "svc-2"); + assert_eq!(rows[0].running_state, "starting"); + } + + #[test] + fn parse_deterministic_service_rows_derives_running_state_for_inline_form_when_absent() { + let rows = parse_deterministic_service_rows( + " - service_id=svc-3 engine=vllm model=qwen status=starting endpoint=127.0.0.1:8002", + ); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].running_state, "starting"); + } + + #[test] + fn parse_deterministic_engine_rows_reads_runtime_and_note() { + let rows = parse_deterministic_engine_rows( + "\ +* lemonade + runtime: gfx120X-all + note: default engine +* vllm + runtime: gfx110X-all +", + ); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0].engine, "lemonade"); + assert_eq!(rows[0].runtime, "gfx120X-all"); + assert_eq!(rows[0].note, "default engine"); + assert_eq!(rows[1].engine, "vllm"); + assert_eq!(rows[1].note, ""); + } + + #[test] + fn managed_service_running_state_maps_every_known_status() { + assert_eq!(managed_service_running_state("ready"), "running"); + assert_eq!(managed_service_running_state("running"), "running"); + assert_eq!(managed_service_running_state("starting"), "starting"); + assert_eq!(managed_service_running_state("recovering"), "starting"); + assert_eq!(managed_service_running_state("failed"), "not_running"); + assert_eq!(managed_service_running_state("stopped"), "not_running"); + assert_eq!(managed_service_running_state("anything-else"), "unknown"); + } + + #[test] + fn deterministic_rocm_tool_summary_interprets_managed_runtime_as_installed() { + let summary = deterministic_rocm_tool_summary( + "\ +examine: + driver_detail: AMD Radeon RX 9070 XT driver 32.0.23033.1002 + legacy_rocm_status: not_detected +runtime_state: + active_runtime_status: ready + active_runtime_root: D:\\ROCm\\therock_venvs + active_runtime_pip_cache_dir: D:\\ROCm\\therock_venvs\\pip-cache + active_runtime_version: 7.13.0a20260511 (build 2026-05-11) + active_runtime_family: gfx120X-all +", + ) + .expect("examine output should summarize"); + + assert!(summary.contains("GPU: AMD Radeon RX 9070 XT driver 32.0.23033.1002")); + assert!(summary.contains("ROCm/TheRock: installed and active for ROCm CLI")); + assert!(summary.contains("gfx120X-all")); + assert!(summary.contains(r"Install folder: D:\ROCm\therock_venvs")); + assert!(summary.contains(r"Downloads/cache: D:\ROCm\therock_venvs\pip-cache")); + assert!(summary.contains("no global legacy ROCm install was found")); + } + + #[test] + fn deterministic_model_tool_summary_identifies_low_vram_assistant() { + let summary = deterministic_model_tool_summary( + "\ +rocm_command: +model recipes + Qwen3-4B-Instruct-2507-GGUF aliases=[qwen, lemonade-qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] + engine_support: + lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe + warning: recommended Lemonade GGUF assistant for ROCm machines + Qwen3-0.6B-GGUF aliases=[qwen-smoke, lemonade-tiny] task=chat dtype=gguf device=gpu_required min_gpu_mem=2 GiB engines=[lemonade] + engine_support: + lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe + warning: tiny Lemonade GGUF smoke-test model; not the default assistant + Qwen/Qwen2.5-0.5B-Instruct aliases=[qwen-tiny] task=chat dtype=float16 device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] + engine_support: + lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe + Qwen/Qwen3.5-4B aliases=[qwen3.5] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=12 GiB engines=[vllm] + engine_support: + vllm: adapter_available path=D:\\rocm\\rocm-engine-vllm.exe runtime_status=unsupported_native_windows reason=native Windows skipped; use WSL/Linux vLLM ROCm + meta-llama/Llama-3.2-3B-Instruct aliases=[llama] task=chat dtype=bfloat16 device=gpu_preferred min_gpu_mem=8 GiB engines=[lemonade, vllm] + engine_support: + lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe + vllm: available path=D:\\rocm\\rocm-engine-vllm.exe +", + ) + .expect("model output should summarize"); + + assert!(summary.contains("Recommended local assistant: qwen")); + assert!(summary.contains("Qwen3-4B-Instruct-2507-GGUF")); + assert!(summary.contains("4 GiB")); + assert!(summary.contains("Tiny smoke test: qwen-smoke")); + assert!(summary.contains("Qwen3-0.6B-GGUF")); + assert!(summary.contains("8 GiB-class option: llama")); + assert!(summary.contains("lemonade, vllm")); + assert!(summary.contains("Qwen/Qwen3.5-4B asks for 12 GiB")); + assert!(summary.contains("Native Windows note")); + assert!(summary.contains("Run `rocm examine`")); + } +} diff --git a/apps/rocm/src/chat/turn.rs b/apps/rocm/src/chat/turn.rs new file mode 100644 index 0000000000..fcef7a8b39 --- /dev/null +++ b/apps/rocm/src/chat/turn.rs @@ -0,0 +1,1547 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The chat assistant's turn loop and transcript: one prompt in, the rendered +//! reply (and the approval request, if the assistant asked for a change) out. +//! `render_chat_prompt_result_with_progress` is the entry point. It builds the +//! assistant's system prompt (`ROCM_CHAT_TOOL_SYSTEM_PROMPT` with the compiled-in +//! assistant skill, grounded in the host by `rocm_chat_tool_system_prompt_for_host`), +//! asks the provider, runs the read-only tool calls the reply requests through +//! the bridge, summarizes their output, and renders the transcript +//! (`append_chat_tool_results`, the `local_*` and display-label helpers). It +//! owns the turn's result types, `ChatPromptResult` and `ChatToolRunResult`. +//! +//! A child of the `chat` module and the top of its call graph: it calls into +//! the prompt-intent heuristics (`chat/intent.rs`), the MCP bridge +//! (`chat/mcp_bridge.rs`), the summaries (`chat/summaries.rs`) and +//! `tool_policy.rs`, and none of them calls back into it. `chat.rs` reads +//! `render_chat_prompt_result_with_progress` from here, and re-exports +//! `rocm_chat_tool_system_prompt_for_host` for `dash.rs`. + +use super::intent::{ + fallback_engine_list_tool_call, fallback_rocm_tool_call_for_prompt, + fallback_services_list_tool_call, install_sdk_without_prefix_chat_approval, + latest_user_chat_message, prompt_asks_engine_install_state, prompt_asks_running_or_status, + prompt_can_use_read_only_without_local_assistant, prompt_mentions_serving_engine_or_service, +}; +use super::mcp_bridge::{loopback_host_key, mcp_tool_result_text, run_chat_read_only_tool}; +use super::summaries::{deterministic_chat_tool_summary, deterministic_summary_can_stand_alone}; +use crate::engine_runtime_select::service_model_names_match; +use crate::freeform_planner::format_structured_tool_call; +use crate::managed_service_status::load_managed_services; +use crate::rocm_argv::chat_cli_arg_value; +use crate::tool_policy::{ + ChatToolApprovalRequest, chat_tool_approval_request, chat_tool_call_is_read_only, json_string, + normalized_chat_rocm_command_args, rocm_chat_tool_requested_command, validate_chat_tool_call, +}; +use crate::{ChatInferenceParams, chat_host_facts, providers}; +use anyhow::Result; +use rocm_core::{AppPaths, DEFAULT_LOCAL_HOST}; +use std::fmt::Write as _; + +fn mcp_tool_result_is_error(value: &serde_json::Value) -> bool { + value + .get("isError") + .and_then(serde_json::Value::as_bool) + .unwrap_or(false) +} + +fn chat_tool_display_label(name: &str) -> String { + match name { + "examine" => "Checked this computer".to_owned(), + "gpu_snapshot" => "Checked GPU status".to_owned(), + "engines" => "Checked local engines".to_owned(), + "services" => "Checked model servers".to_owned(), + "bridge_snapshot" => "Checked ROCm state".to_owned(), + "service_logs" => "Read server logs".to_owned(), + "automations" => "Checked automations".to_owned(), + "natural_language_plan" => "Planned ROCm request".to_owned(), + "path_exists" => "Checked folder path".to_owned(), + "port_status" => "Checked local port".to_owned(), + "update_check" => "Checked for ROCm updates".to_owned(), + "install_sdk_dry_run" => "Previewed ROCm install".to_owned(), + "install_sdk" => "Install ROCm".to_owned(), + "install_engine" => "Install engine".to_owned(), + "launch_server" => "Start local model server".to_owned(), + "stop_server" => "Stop local model server".to_owned(), + "watcher_enable" => "Enable automation".to_owned(), + "watcher_disable" => "Disable automation".to_owned(), + other => other.replace('_', " "), + } +} + +fn chat_tool_call_display_label(call: &providers::ChatToolCall) -> String { + if call.name != "rocm_command" { + return chat_tool_display_label(&call.name); + } + let Some(args) = normalized_chat_rocm_command_args(call).ok() else { + return "rocm command".to_owned(); + }; + match args.as_slice() { + [command] if command.eq_ignore_ascii_case("examine") => "Checked this computer".to_owned(), + [command] + if command.eq_ignore_ascii_case("model") || command.eq_ignore_ascii_case("models") => + { + "Checked model recipes".to_owned() + } + [command, subcommand] + if command.eq_ignore_ascii_case("engines") + && subcommand.eq_ignore_ascii_case("list") => + { + "Checked local engines".to_owned() + } + [command] if command.eq_ignore_ascii_case("services") => "Checked model servers".to_owned(), + [command, subcommand, rest @ ..] + if command.eq_ignore_ascii_case("services") + && subcommand.eq_ignore_ascii_case("list") + && rest + .iter() + .all(|arg| matches!(arg.as_str(), "-a" | "--all")) => + { + "Checked model servers".to_owned() + } + [command] if command.eq_ignore_ascii_case("comfyui") => "Checked ComfyUI".to_owned(), + [command, subcommand] + if command.eq_ignore_ascii_case("comfyui") + && subcommand.eq_ignore_ascii_case("status") => + { + "Checked ComfyUI".to_owned() + } + [command, subcommand, ..] + if command.eq_ignore_ascii_case("comfyui") + && matches!(subcommand.to_ascii_lowercase().as_str(), "logs" | "log") => + { + "Read ComfyUI logs".to_owned() + } + [command, subcommand] + if command.eq_ignore_ascii_case("config") + && subcommand.eq_ignore_ascii_case("show") => + { + "Checked ROCm config".to_owned() + } + _ => "rocm command".to_owned(), + } +} + +#[derive(Debug, Clone, Eq, PartialEq)] +pub(super) struct ChatPromptResult { + pub rendered: String, + pub approval: Option, +} + +struct ChatToolRunResult { + pub(crate) approval: Option, + pub(crate) follow_up_text: String, + pub(crate) ran_read_only_tool: bool, + pub(crate) read_only_tool_error: bool, + pub(crate) needs_install_folder: bool, +} + +/// Convenience wrapper used by tests that do not exercise sampling controls. +#[cfg(test)] +fn render_chat_prompt_result( + paths: &AppPaths, + provider: &str, + model: Option<&str>, + prompt: &str, + rocm_tools: bool, +) -> Result { + render_chat_prompt_result_with_progress( + paths, + provider, + model, + prompt, + rocm_tools, + ChatInferenceParams::default(), + None, + ) +} + +pub(super) fn render_chat_prompt_result_with_progress( + paths: &AppPaths, + provider: &str, + model: Option<&str>, + prompt: &str, + rocm_tools: bool, + params: ChatInferenceParams, + progress: Option<&mut dyn FnMut(String)>, +) -> Result { + let mut progress = progress; + let user_prompt = latest_user_chat_message(prompt); + let assistant_model = local_rocm_tools_assistant_model(provider, rocm_tools).or(model); + let service_needed_model = if local_rocm_tools_assistant_model(provider, rocm_tools).is_some() { + None + } else { + model + }; + if rocm_tools && let Some(approval) = install_sdk_without_prefix_chat_approval(user_prompt) { + report_chat_tool_progress(&mut progress, "Asking the assistant."); + report_chat_tool_progress(&mut progress, "Review needed: Install ROCm"); + report_chat_tool_progress(&mut progress, "Waiting for the ROCm install folder."); + return Ok(ChatPromptResult { + rendered: render_install_sdk_folder_needed_chat_text(user_prompt, &approval.args), + approval: Some(approval), + }); + } + let mut messages = Vec::new(); + if rocm_tools { + messages.push(providers::ChatMessage { + role: "system".to_owned(), + content: rocm_chat_tool_system_prompt_for_host(Some(paths)), + }); + } + messages.push(providers::ChatMessage { + role: "user".to_owned(), + content: prompt.to_owned(), + }); + let mut response = if rocm_tools { + report_chat_tool_progress(&mut progress, "Asking the assistant."); + if let Some(call) = deterministic_mutating_tool_call_for_prompt(user_prompt)? { + report_chat_tool_progress(&mut progress, "Preparing a review card."); + providers::ChatResponse { + provider: provider.to_owned(), + model: assistant_model.unwrap_or("local").to_owned(), + content: deterministic_mutating_tool_intro(&call), + tool_calls: vec![call], + } + } else if !local_assistant_service_ready_for_chat(paths, assistant_model) + && let Some(response) = + local_read_only_fallback_response_for_prompt(provider, assistant_model, user_prompt) + { + report_chat_tool_progress(&mut progress, "Running ROCm status check."); + response + } else { + match providers::provider_chat( + paths, + provider, + &providers::ChatRequest { + model: assistant_model.map(str::to_owned), + messages: messages.clone(), + max_tokens: params.max_tokens, + temperature: params.temperature, + top_p: params.top_p, + rocm_tools, + }, + ) { + Ok(response) => response, + Err(error) + if provider == "local" && local_provider_missing_service_error(&error) => + { + if let Some(response) = local_read_only_fallback_response_for_prompt( + provider, + assistant_model, + user_prompt, + ) { + report_chat_tool_progress(&mut progress, "Running ROCm status check."); + response + } else { + return Ok(ChatPromptResult { + rendered: local_chat_service_needed_text( + service_needed_model, + user_prompt, + rocm_tools, + ), + approval: None, + }); + } + } + Err(error) => return Err(error), + } + } + } else { + report_chat_tool_progress(&mut progress, "Asking the assistant."); + match providers::provider_chat( + paths, + provider, + &providers::ChatRequest { + model: assistant_model.map(str::to_owned), + messages: messages.clone(), + max_tokens: params.max_tokens, + temperature: params.temperature, + top_p: params.top_p, + rocm_tools, + }, + ) { + Ok(response) => response, + Err(error) if provider == "local" && local_provider_missing_service_error(&error) => { + return Ok(ChatPromptResult { + rendered: local_chat_service_needed_text( + service_needed_model, + user_prompt, + rocm_tools, + ), + approval: None, + }); + } + Err(error) => return Err(error), + } + }; + let fallback_tool_call = if rocm_tools && response.tool_calls.is_empty() { + fallback_rocm_tool_call_for_prompt(user_prompt) + } else { + None + }; + let fallback_tool_call_used = fallback_tool_call.is_some(); + if let Some(call) = fallback_tool_call { + response.tool_calls.push(call); + } + if rocm_tools + && let Some(call) = + supplemental_read_only_tool_call_for_prompt(user_prompt, &response.tool_calls) + { + response.tool_calls.push(call); + } + let mut output = String::new(); + let _ = writeln!(output, "chat response"); + let _ = writeln!(output, " provider: {}", response.provider); + let _ = writeln!(output, " model: {}", response.model); + let _ = writeln!( + output, + " rocm tools: {}", + if rocm_tools { "enabled" } else { "off" } + ); + let _ = writeln!(output); + let initial_content_is_intermediate = fallback_tool_call_used + || local_tool_call_content_is_intermediate(provider, rocm_tools, &response); + let initial_content = visible_chat_content(&response.content); + if initial_content_is_intermediate { + let _ = writeln!(output, "Assistant is preparing the next step."); + } else if !initial_content.trim().is_empty() { + let _ = writeln!(output, "{initial_content}"); + } + let tool_result = if rocm_tools { + let explanation = (!fallback_tool_call_used && !initial_content.trim().is_empty()) + .then_some(initial_content.as_str()); + append_chat_tool_results(paths, &response, &mut output, explanation, &mut progress)? + } else { + ChatToolRunResult { + approval: None, + follow_up_text: String::new(), + ran_read_only_tool: false, + read_only_tool_error: false, + needs_install_folder: false, + } + }; + let deterministic_summary = deterministic_chat_tool_summary(&tool_result.follow_up_text); + if let Some(summary) = deterministic_summary.as_deref() { + let _ = writeln!(output); + let _ = writeln!(output, "ROCm CLI summary"); + let _ = writeln!(output, "{summary}"); + } + let follow_up_blocking_summary = deterministic_summary + .as_deref() + .filter(|_| deterministic_summary_can_stand_alone(&tool_result.follow_up_text)); + if should_request_local_tool_follow_up(provider, &tool_result, follow_up_blocking_summary) { + let follow_up_messages = vec![ + providers::ChatMessage { + role: "system".to_owned(), + content: "You are ROCm CLI's local assistant. ROCm tools have already been checked for this turn. Use the supplied tool results for local facts about this machine. For service results, ready/running means running, starting/recovering means starting, failed/stopped means not running, and no matching service row means ROCm CLI is not managing that service as running. If the tool results do not contain enough information to answer the whole question, say what is known from the results and then answer the rest normally from your own model knowledge. Do not request another tool call. Do not invent local paths, GPU names, versions, or install state that are not shown in the tool results. Keep the answer concise.".to_owned(), + }, + providers::ChatMessage { + role: "user".to_owned(), + content: format!( + "Original request:\n{}\n\nROCm tool results:\n{}\n\nAnswer the original request. Use the ROCm tool results for this machine's facts; if they are incomplete, answer the remaining part generally.", + user_prompt.trim(), + tool_result.follow_up_text.trim() + ), + }, + ]; + let follow_up = providers::provider_chat( + paths, + provider, + &providers::ChatRequest { + model: assistant_model.map(str::to_owned), + messages: follow_up_messages, + max_tokens: params.max_tokens.or(Some(256)), + temperature: params.temperature, + top_p: params.top_p, + rocm_tools: false, + }, + )?; + if follow_up.tool_calls.is_empty() { + let follow_up_content = visible_chat_content(&follow_up.content); + if local_follow_up_content_is_final(&follow_up) && !follow_up_content.trim().is_empty() + { + let _ = writeln!(output); + let _ = writeln!(output, "Assistant after ROCm checks"); + let _ = writeln!(output, "{}", follow_up_content.trim()); + } + } else { + let _ = writeln!(output); + let _ = writeln!( + output, + "Assistant asked for another ROCm check after the first tool results." + ); + let _ = writeln!( + output, + "rocm-cli stopped there so the answer is not based on another guess. Nothing was changed." + ); + } + } else if tool_result.read_only_tool_error { + let _ = writeln!(output); + let _ = writeln!( + output, + "Assistant did not answer from this result because a ROCm check reported an error." + ); + let _ = writeln!(output, "Nothing was changed."); + } else if tool_result.needs_install_folder { + let _ = writeln!(output); + let _ = writeln!(output, "Assistant after ROCm checks"); + let _ = writeln!( + output, + "I can install ROCm/TheRock, but I need the install folder first. Type the folder you want to use, for example D:\\ROCm\\therock_venvs. Nothing will download until ROCm CLI shows a review card and you approve it." + ); + } + if tool_result.approval.is_none() + && tool_result.ran_read_only_tool + && !output.contains("Nothing was changed.") + { + let _ = writeln!(output); + let _ = writeln!(output, "Nothing was changed."); + } + Ok(ChatPromptResult { + rendered: output, + approval: tool_result.approval, + }) +} + +fn report_chat_tool_progress(progress: &mut Option<&mut dyn FnMut(String)>, message: &str) { + if let Some(progress) = progress.as_deref_mut() { + progress(message.to_owned()); + } +} + +fn deterministic_mutating_tool_call_for_prompt( + prompt: &str, +) -> Result> { + let Some(call) = fallback_rocm_tool_call_for_prompt(prompt) else { + return Ok(None); + }; + validate_chat_tool_call(&call)?; + if chat_tool_call_is_read_only(&call) { + Ok(None) + } else { + Ok(Some(call)) + } +} + +fn supplemental_read_only_tool_call_for_prompt( + prompt: &str, + existing_calls: &[providers::ChatToolCall], +) -> Option { + let normalized = latest_user_chat_message(prompt).to_ascii_lowercase(); + if prompt_mentions_serving_engine_or_service(&normalized) + && prompt_asks_running_or_status(&normalized) + && prompt_asks_engine_install_state(&normalized) + { + for call in [ + fallback_services_list_tool_call(), + fallback_engine_list_tool_call(), + ] { + if !chat_tool_calls_include_equivalent(existing_calls, &call) { + return Some(call); + } + } + return None; + } + let call = fallback_rocm_tool_call_for_prompt(prompt)?; + if !chat_tool_call_is_read_only(&call) + || chat_tool_calls_include_equivalent(existing_calls, &call) + { + return None; + } + Some(call) +} + +fn chat_tool_calls_include_equivalent( + existing_calls: &[providers::ChatToolCall], + required: &providers::ChatToolCall, +) -> bool { + existing_calls.iter().any(|existing| { + if existing.name != required.name { + return false; + } + if required.name == "rocm_command" { + return normalized_chat_rocm_command_args(existing).ok() + == normalized_chat_rocm_command_args(required).ok(); + } + if required.name == "port_status" { + return chat_port_status_tool_calls_equivalent(existing, required); + } + existing.arguments == required.arguments + }) +} + +fn chat_port_status_tool_calls_equivalent( + left: &providers::ChatToolCall, + right: &providers::ChatToolCall, +) -> bool { + let Some(left_object) = left.arguments.as_object() else { + return false; + }; + let Some(right_object) = right.arguments.as_object() else { + return false; + }; + let left_port = left_object.get("port").and_then(serde_json::Value::as_u64); + let right_port = right_object.get("port").and_then(serde_json::Value::as_u64); + if left_port != right_port { + return false; + } + let left_host = + json_string(left_object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); + let right_host = + json_string(right_object, "host").unwrap_or_else(|| DEFAULT_LOCAL_HOST.to_owned()); + loopback_host_key(&left_host) == loopback_host_key(&right_host) +} + +fn deterministic_mutating_tool_intro(call: &providers::ChatToolCall) -> String { + match chat_tool_approval_request(call, None) { + Ok(approval) if approval.pending_title == "Install ROCm" => { + "I can install ROCm/TheRock into the folder you chose. Review the card before anything downloads or changes." + .to_owned() + } + Ok(approval) if approval.pending_title == "Install ComfyUI" => { + "I can install ComfyUI into ROCm CLI's managed app folder. Review the card before anything downloads or changes." + .to_owned() + } + Ok(approval) if approval.pending_title == "Start ComfyUI" => { + "I can start ComfyUI for you. Review the card before ROCm CLI launches it." + .to_owned() + } + Ok(approval) if approval.pending_title == "Start local model server" => { + "I can start the recommended local model server on the GPU. Review the card before ROCm CLI launches it." + .to_owned() + } + Ok(approval) => format!( + "I can prepare this ROCm change: {}. Review the card before anything runs.", + approval.pending_title + ), + Err(_) => { + "I can prepare this ROCm change. Review the card before anything runs.".to_owned() + } + } +} + +fn local_tool_call_content_is_intermediate( + provider: &str, + rocm_tools: bool, + response: &providers::ChatResponse, +) -> bool { + provider == "local" + && rocm_tools + && !response.tool_calls.is_empty() + && !response.content.trim().is_empty() +} + +fn local_follow_up_content_is_final(response: &providers::ChatResponse) -> bool { + response.tool_calls.is_empty() && !response.content.trim().is_empty() +} + +fn local_read_only_fallback_response_for_prompt( + provider: &str, + assistant_model: Option<&str>, + prompt: &str, +) -> Option { + if provider != "local" || !prompt_can_use_read_only_without_local_assistant(prompt) { + return None; + } + let call = fallback_rocm_tool_call_for_prompt(prompt)?; + if !chat_tool_call_is_read_only(&call) { + return None; + } + Some(providers::ChatResponse { + provider: provider.to_owned(), + model: assistant_model.unwrap_or("local").to_owned(), + content: String::new(), + tool_calls: vec![call], + }) +} + +fn local_assistant_service_ready_for_chat(paths: &AppPaths, assistant_model: Option<&str>) -> bool { + let model = assistant_model.unwrap_or(providers::BUILTIN_ASSISTANT_MODEL_ID); + load_managed_services(paths).is_ok_and(|records| { + records.iter().any(|record| { + matches!(record.status.as_str(), "ready" | "running") + && (service_model_names_match(&record.canonical_model_id, model) + || service_model_names_match(&record.model_ref, model)) + }) + }) +} + +fn render_install_sdk_folder_needed_chat_text(prompt: &str, args: &[String]) -> String { + let mut output = String::new(); + let _ = writeln!(output, "ROCm install"); + let _ = writeln!(output); + let _ = writeln!(output, "I can install ROCm/TheRock for you."); + let _ = writeln!( + output, + "First choose the folder where ROCm CLI should put the Python environment." + ); + let _ = writeln!( + output, + "Nothing will download or change until you review and approve the install." + ); + if let Some(date) = chat_cli_arg_value(args, "--build-date") { + let _ = writeln!(output); + let _ = writeln!(output, "Requested build date: {date}"); + } + if let Some(version) = chat_cli_arg_value(args, "--version") { + let _ = writeln!(output); + let _ = writeln!(output, "Requested version: {version}"); + } + let _ = writeln!(output); + let _ = writeln!(output, "Your request"); + let _ = writeln!(output, " {}", prompt.trim()); + output.trim_end().to_owned() +} + +fn visible_chat_content(content: &str) -> String { + let mut output = String::new(); + let mut remaining = content; + while let Some(start) = find_ascii_case_insensitive(remaining, "") { + output.push_str(&remaining[..start]); + let after_start = &remaining[start + "".len()..]; + let Some(end) = find_ascii_case_insensitive(after_start, "") else { + remaining = ""; + break; + }; + remaining = &after_start[end + "".len()..]; + } + output.push_str(remaining); + output.trim().to_owned() +} + +fn find_ascii_case_insensitive(haystack: &str, needle: &str) -> Option { + let needle = needle.as_bytes(); + haystack + .as_bytes() + .windows(needle.len()) + .position(|window| window.eq_ignore_ascii_case(needle)) +} + +/// The host-independent half of the assistant prompt. +/// +/// Statements that are only true on *some* hosts do not belong here — they used +/// to, and a WSL user was told "on native Windows, vLLM is skipped" while vLLM +/// was in fact their supported path. Anything host-dependent now comes from +/// [`chat_host_facts::HostFacts`], which knows which machine it is describing. +const ROCM_CHAT_TOOL_SYSTEM_PROMPT: &str = "You are ROCm CLI's local assistant. Speak in simple English for non-technical users. Use the provided ROCm tools when you need to inspect this machine, preview setup, read service logs, check updates, inspect automations, install or start ROCm-managed apps, or request ROCm/TheRock, config, engine, app, and local model server changes. For simple greetings or thanks like hello, hi, hey, ok, or thank you, reply normally; do not inspect ROCm, do not call tools, and do not launch or propose a model server. Tool-use rules: inspect first with read-only tools; call rocm_command only with argv-style args and no shell text; use natural_language_plan for ROCm requests that do not fit another read-only tool; ask for a mutating tool call only after explaining why it is needed; summarize tool results after they are returned. Read-only tools may run immediately. Tools that install, launch, stop, delete, or change state require user approval; request rocm_command and explain why. For 'is X running?', 'what is running?', status, or port questions, inspect before answering and do not start, stop, install, or serve anything. For ComfyUI or port 8188 use [\"comfyui\",\"status\"] or port_status. For vLLM, Lemonade, AIM, qwen, or local model servers use [\"services\",\"list\",\"--all\"] for running state and [\"engines\",\"list\"] for installed/available engine state. Treat ready/running as running, starting/recovering as starting, failed/stopped as not running, and no matching record as unknown or not managed by ROCm CLI. Interpret Examine carefully: active_runtime_status=ready means ROCm CLI has an active managed TheRock/ROCm runtime; legacy_rocm_status=not_detected only means no global system ROCm install was found. If active_runtime_status=ready, tell the user ROCm/TheRock is installed and active for ROCm CLI. For 'is TheRock installed', 'is ROCm installed', or 'which GPU is on this machine', use examine or gpu_snapshot before answering. For 'how do I setup TheRock' or install/setup requests, guide the user to choose an install folder first; do not answer with only a status check. For 'which LLMs can this machine support', use rocm_command args [\"model\"] or natural_language_plan before answering. For TheRock installs, always let the user choose the install folder. If the user names a folder or prefix, preserve that exact folder with [\"--prefix\",\"PATH\"]; you may call path_exists first to check whether that user-provided folder or its parent exists. If the user asks you to install TheRock/ROCm but has not named a folder, ask for the folder or let the guided setup folder picker collect it; do not invent a hidden default folder and do not request an install command without --prefix. Use rocm_command args [\"install\",\"sdk\",\"--channel\",\"release\",\"--format\",\"wheel\",\"--prefix\",\"PATH\"] only when the user asks you to install it and a folder is known; for a requested build date add [\"--build-date\",\"YYYY-MM-DD\"] and for a requested exact version add [\"--version\",\"VERSION\"]. For config changes, inspect with [\"config\",\"show\"] first when useful, then request config subcommands such as [\"config\",\"set-default-engine\",\"lemonade\"], [\"config\",\"set-default-runtime\",\"RUNTIME_KEY\"], or [\"config\",\"set-telemetry\",\"local\"] only after explaining why. For ComfyUI, use rocm_command with args like [\"comfyui\",\"status\"], [\"comfyui\",\"logs\"], [\"comfyui\",\"install\"], [\"comfyui\",\"start\"], or [\"comfyui\",\"stop\"]. First-time setup is the same thing as bootstrap in ROCm CLI; it is a deterministic ROCm setup flow, not a separate model chat. The built-in local assistant is fixed to qwen, which maps to Qwen3-4B-Instruct-2507-GGUF served by Lemonade with gpu_required. vLLM, Lemonade, and AIM (Docker-based) are the general serving engines; inspect or manage them when the user asks about general model serving, but do not switch the built-in assistant away from Lemonade. Use qwen-smoke only for a quick server smoke test. For vLLM management, inspect engines first and use [\"engines\",\"install\",\"vllm\"] or [\"serve\",\"MODEL\",\"--engine\",\"vllm\",\"--device\",\"gpu_required\",\"--managed\"] only where the host supports it. Do not invent shell commands and do not request CPU fallback."; + +const ROCM_CHAT_TOOL_SKILL: &str = include_str!("../../../../skills/rocm-cli-assistant/SKILL.md"); + +fn rocm_chat_tool_system_prompt() -> String { + format!("{ROCM_CHAT_TOOL_SYSTEM_PROMPT}\n\nROCm CLI assistant skill:\n{ROCM_CHAT_TOOL_SKILL}") +} + +/// The assistant prompt grounded in the machine it will answer for. +/// +/// The single composition point: both chat surfaces (`rocm chat --prompt +/// --tools` in this bin, and the dashboard chat via +/// [`crate::dash::resolved_args`]) send this, so neither can drift into +/// answering platform questions from pretraining alone. +pub(crate) fn rocm_chat_tool_system_prompt_for_host(paths: Option<&AppPaths>) -> String { + let facts = chat_host_facts::HostFacts::detect(paths); + format!("{}\n\n{}", rocm_chat_tool_system_prompt(), facts.render()) +} + +fn local_provider_missing_service_error(error: &anyhow::Error) -> bool { + error.chain().any(|cause| { + cause + .to_string() + .contains("local provider has no ready managed service") + }) +} + +fn local_rocm_tools_assistant_model(provider: &str, rocm_tools: bool) -> Option<&'static str> { + (provider == "local" && rocm_tools).then_some(providers::BUILTIN_ASSISTANT_MODEL_ID) +} + +fn local_chat_service_needed_text(model: Option<&str>, prompt: &str, rocm_tools: bool) -> String { + let mut output = String::new(); + let _ = writeln!(output, "No local assistant is running yet."); + let _ = writeln!(output); + let _ = writeln!( + output, + "To use LLM-assisted ROCm commands, start a ROCm GPU local model server first." + ); + let _ = writeln!( + output, + "First-time ROCm setup does not need an LLM; run `rocm` and use Set Up ROCm for that." + ); + let _ = writeln!(output); + let _ = writeln!( + output, + "Recommended path:\n run `rocm`, choose Start a local model, use the recommended assistant model, then start it" + ); + let _ = writeln!(output); + if let Some(model) = model.filter(|value| !value.trim().is_empty()) { + let args = vec![ + "serve".to_owned(), + model.to_owned(), + "--device".to_owned(), + "gpu_required".to_owned(), + "--managed".to_owned(), + ]; + let _ = writeln!( + output, + "Advanced manual command for the selected model:\n {}", + format_structured_tool_call("rocm", &args) + ); + } else { + let example_args = vec![ + "serve".to_owned(), + providers::LEMONADE_ASSISTANT_MODEL_ID.to_owned(), + "--engine".to_owned(), + "lemonade".to_owned(), + "--device".to_owned(), + "gpu_required".to_owned(), + "--managed".to_owned(), + ]; + let _ = writeln!( + output, + "Advanced manual command for the recommended assistant model:\n {}", + format_structured_tool_call("rocm", &example_args) + ); + } + let mut chat_args = vec!["chat".to_owned()]; + if rocm_tools { + chat_args.push("--tools".to_owned()); + } + chat_args.push("--provider".to_owned()); + chat_args.push("local".to_owned()); + if let Some(model) = model.filter(|value| !value.trim().is_empty()) { + chat_args.push("--model".to_owned()); + chat_args.push(model.to_owned()); + } + if !prompt.trim().is_empty() { + chat_args.push("--prompt".to_owned()); + chat_args.push(prompt.to_owned()); + } + let _ = writeln!(output); + let _ = writeln!( + output, + "After the server is ready, run:\n {}", + format_structured_tool_call("rocm", &chat_args) + ); + let _ = writeln!(output); + let _ = writeln!( + output, + "For the guided flow, run `rocm`, choose Start a local model, start a model, then choose Chat." + ); + let _ = writeln!(output, "Nothing was changed."); + output.trim_end().to_owned() +} + +fn append_chat_tool_results( + paths: &AppPaths, + response: &providers::ChatResponse, + output: &mut String, + assistant_explanation: Option<&str>, + progress: &mut Option<&mut dyn FnMut(String)>, +) -> Result { + if response.tool_calls.is_empty() { + let _ = writeln!(output, "ROCm checks used"); + let _ = writeln!(output, " none requested"); + return Ok(ChatToolRunResult { + approval: None, + follow_up_text: String::new(), + ran_read_only_tool: false, + read_only_tool_error: false, + needs_install_folder: false, + }); + } + + let _ = writeln!(output, "ROCm checks used"); + let mut approval = None; + let mut follow_up_text = String::new(); + let mut ran_read_only_tool = false; + let mut read_only_tool_error = false; + let mut needs_install_folder = false; + for call in &response.tool_calls { + if chat_tool_call_requests_therock_install_without_prefix(call) { + needs_install_folder = true; + report_chat_tool_progress(progress, "Waiting for the ROCm install folder."); + let _ = writeln!(output, " Install ROCm: needs install folder"); + let _ = writeln!( + output, + " not run: choose an install folder before the review card" + ); + continue; + } + validate_chat_tool_call(call)?; + let label = chat_tool_call_display_label(call); + if chat_tool_call_is_read_only(call) { + report_chat_tool_progress(progress, &format!("Running ROCm check: {label}.")); + let result = run_chat_read_only_tool(paths, call)?; + ran_read_only_tool = true; + let is_error = mcp_tool_result_is_error(&result); + read_only_tool_error |= is_error; + report_chat_tool_progress( + progress, + &format!( + "ROCm check finished: {label} ({})", + chat_read_only_tool_status_label(is_error) + ), + ); + let _ = writeln!( + output, + " {}: {}", + label, + chat_read_only_tool_status_label(is_error) + ); + let result_text = mcp_tool_result_text(&result); + let _ = writeln!(follow_up_text, "{}:", call.name); + let _ = writeln!(follow_up_text, "{result_text}"); + for line in result_text.lines() { + let _ = writeln!(output, " {line}"); + } + } else { + report_chat_tool_progress(progress, &format!("Review needed: {label}.")); + let _ = writeln!(output, " {label}: needs your review"); + let _ = writeln!( + output, + " not run: review the approval card before anything runs" + ); + if let Some(command) = rocm_chat_tool_requested_command(call) { + let _ = writeln!(output, " advanced manual command: {command}"); + } + if approval.is_none() { + approval = Some(chat_tool_approval_request(call, assistant_explanation)?); + } + } + } + Ok(ChatToolRunResult { + approval, + follow_up_text, + ran_read_only_tool, + read_only_tool_error, + needs_install_folder, + }) +} + +fn chat_tool_call_requests_therock_install_without_prefix(call: &providers::ChatToolCall) -> bool { + let Some(object) = call.arguments.as_object() else { + return false; + }; + match call.name.as_str() { + "install_sdk" => json_string(object, "prefix").is_none(), + "rocm_command" => rocm_command_args_install_sdk_without_prefix(object), + _ => false, + } +} + +fn rocm_command_args_install_sdk_without_prefix( + object: &serde_json::Map, +) -> bool { + let Some(args) = object.get("args").and_then(serde_json::Value::as_array) else { + return false; + }; + let args = args + .iter() + .filter_map(serde_json::Value::as_str) + .map(str::to_owned) + .collect::>(); + args.first() + .is_some_and(|arg| arg.eq_ignore_ascii_case("install")) + && args + .get(1) + .is_some_and(|arg| arg.eq_ignore_ascii_case("sdk")) + && chat_cli_arg_value(&args, "--prefix").is_none() +} + +fn should_request_local_tool_follow_up( + provider: &str, + tool_result: &ChatToolRunResult, + deterministic_summary: Option<&str>, +) -> bool { + provider == "local" + && deterministic_summary.is_none() + && tool_result.approval.is_none() + && tool_result.ran_read_only_tool + && !tool_result.read_only_tool_error + && !tool_result.follow_up_text.trim().is_empty() +} + +const fn chat_read_only_tool_status_label(is_error: bool) -> &'static str { + if is_error { + "reported an error" + } else { + "done" + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::chat::summaries::deterministic_rocm_tool_summary; + use crate::tests::test_paths; + + fn rocm_command_call(args: &[&str]) -> providers::ChatToolCall { + providers::ChatToolCall { + id: Some("test".to_owned()), + name: "rocm_command".to_owned(), + arguments: serde_json::json!({ "args": args }), + } + } + + #[test] + fn mcp_tool_result_is_error_reads_the_is_error_flag() { + assert!(mcp_tool_result_is_error( + &serde_json::json!({"isError": true}) + )); + assert!(!mcp_tool_result_is_error( + &serde_json::json!({"isError": false}) + )); + assert!(!mcp_tool_result_is_error(&serde_json::json!({}))); + } + + #[test] + fn chat_tool_display_label_maps_known_tools_and_humanizes_others() { + assert_eq!(chat_tool_display_label("examine"), "Checked this computer"); + assert_eq!( + chat_tool_display_label("gpu_snapshot"), + "Checked GPU status" + ); + assert_eq!(chat_tool_display_label("install_sdk"), "Install ROCm"); + assert_eq!( + chat_tool_display_label("some_unmapped_tool"), + "some unmapped tool" + ); + } + + #[test] + fn chat_tool_call_display_label_delegates_for_non_rocm_command_tools() { + let call = providers::ChatToolCall { + id: None, + name: "examine".to_owned(), + arguments: serde_json::json!({}), + }; + assert_eq!(chat_tool_call_display_label(&call), "Checked this computer"); + } + + #[test] + fn chat_tool_call_display_label_recognizes_rocm_command_shapes() { + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["examine"])), + "Checked this computer" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["models"])), + "Checked model recipes" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["model"])), + "Checked model recipes" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["ENGINES", "List"])), + "Checked local engines" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["services"])), + "Checked model servers" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["services", "list", "-a"])), + "Checked model servers" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["services", "list", "--all"])), + "Checked model servers" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["comfyui"])), + "Checked ComfyUI" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["comfyui", "status"])), + "Checked ComfyUI" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["comfyui", "logs", "-n", "50"])), + "Read ComfyUI logs" + ); + assert_eq!( + chat_tool_call_display_label(&rocm_command_call(&["config", "show"])), + "Checked ROCm config" + ); + } + + #[test] + fn chat_tool_call_display_label_falls_back_for_unparseable_args() { + let call = providers::ChatToolCall { + id: None, + name: "rocm_command".to_owned(), + arguments: serde_json::json!({"args": []}), + }; + assert_eq!(chat_tool_call_display_label(&call), "rocm command"); + } + + #[test] + fn local_assistant_prompt_instructions_cover_core_support_questions() { + let prompt = rocm_chat_tool_system_prompt(); + for expected in [ + "is TheRock installed", + "which GPU is on this machine", + "active_runtime_status=ready", + "legacy_rocm_status=not_detected", + "[\"model\"]", + "--build-date", + "always let the user choose the install folder", + "--prefix", + "do not invent a hidden default folder", + "config", + "comfyui", + "First-time setup is the same thing as bootstrap", + "vllm", + "Qwen3-4B-Instruct-2507-GGUF", + "fixed to qwen", + "served by Lemonade", + "port_status", + "[\"services\",\"list\",\"--all\"]", + "qwen-smoke", + "Do not invent shell commands", + "ROCm CLI Assistant Skill", + "Treat `localhost` and `127.0.0.1` as the same loopback endpoint", + ] { + assert!( + prompt.contains(expected), + "system prompt should mention {expected}" + ); + } + } + + /// The reported bug: asked "What can ROCm do on Windows?", the assistant + /// answered that ROCm is Windows-incompatible and suggested CUDA/DirectX — + /// because nothing ever told it which machine it was on. The prompt the CLI + /// actually sends must carry the host. + #[test] + fn assistant_prompt_states_the_host_it_is_answering_for() { + let prompt = rocm_chat_tool_system_prompt_for_host(None); + + // The tool-use rules survive the composition (this is the same prompt, + // grounded — not a replacement for it). + assert!( + prompt.contains("You are ROCm CLI's local assistant"), + "the ROCm tool-use prompt must still be there:\n{prompt}" + ); + + // …and it now names this machine's OS and GPU state. + let expected_os = if cfg!(windows) { + "- Operating system: Windows" + } else { + "- Operating system: Linux" + }; + assert!( + prompt.contains(expected_os), + "the prompt must state this machine's operating system:\n{prompt}" + ); + assert!( + prompt.contains("- AMD GPU: "), + "the prompt must state what GPU was detected (or that none was):\n{prompt}" + ); + assert!( + prompt.contains("Never tell the user ROCm is unavailable on their platform"), + "the prompt must refuse the reported answer:\n{prompt}" + ); + } + + /// The static prompt asserted two Windows-only facts at every host, which is + /// how a WSL user — where vLLM IS the supported path — was told to go use + /// WSL. Platform claims now come from the detected facts instead. + #[test] + fn assistant_prompt_makes_no_unconditional_windows_claims() { + let prompt = rocm_chat_tool_system_prompt(); + assert!( + !prompt.contains("non-technical Windows users"), + "the audience is not assumed to be on Windows:\n{prompt}" + ); + assert!( + !prompt.contains("On native Windows, vLLM is skipped"), + "the vLLM caveat belongs in the host facts, not the static prompt:\n{prompt}" + ); + + // Stated only where it is true: present on Windows, absent elsewhere. + let grounded = rocm_chat_tool_system_prompt_for_host(None); + assert_eq!( + grounded.contains("vLLM and AIM are skipped on native Windows"), + cfg!(windows), + "the vLLM caveat must track the host:\n{grounded}" + ); + } + + /// The prompt tells the model to "use examine … before answering". The dash + /// registers its machine check as `doctor`, so before the alias that + /// sentence named a tool absent from the dash's schema. + #[test] + fn every_tool_the_prompt_names_exists_in_the_dash_schema() { + let prompt = rocm_chat_tool_system_prompt(); + for named in [ + "examine", + "gpu_snapshot", + "port_status", + "natural_language_plan", + ] { + assert!(prompt.contains(named), "prompt should mention {named}"); + assert!( + rocm_dash_tui::agent::ROCM_READ_TOOL_NAMES.contains(&named), + "the prompt names `{named}` but the dash never registers it, so a \ + model that obeys the prompt calls a tool the schema does not offer" + ); + } + } + + #[test] + fn deterministic_rocm_tool_summary_suppresses_extra_local_model_follow_up() { + let tool_result = ChatToolRunResult { + approval: None, + follow_up_text: "\ +examine: + legacy_rocm_status: not_detected +runtime_state: + active_runtime_status: ready +" + .to_owned(), + ran_read_only_tool: true, + read_only_tool_error: false, + needs_install_folder: false, + }; + let summary = deterministic_rocm_tool_summary(&tool_result.follow_up_text); + + assert!(summary.is_some()); + assert!(!should_request_local_tool_follow_up( + "local", + &tool_result, + summary.as_deref() + )); + + let mut model_list_result = tool_result; + model_list_result.follow_up_text = "rocm_command:\nmodel recipes\n qwen\n".to_owned(); + assert!(should_request_local_tool_follow_up( + "local", + &model_list_result, + None + )); + } + + #[test] + fn deterministic_model_tool_summary_suppresses_extra_local_model_follow_up() { + let tool_result = ChatToolRunResult { + approval: None, + follow_up_text: "\ +rocm_command: +model recipes + Qwen3-4B-Instruct-2507-GGUF aliases=[qwen] task=chat dtype=gguf device=gpu_required min_gpu_mem=4 GiB engines=[lemonade] + engine_support: + lemonade: available path=D:\\rocm\\rocm-engine-lemonade.exe +" + .to_owned(), + ran_read_only_tool: true, + read_only_tool_error: false, + needs_install_folder: false, + }; + let summary = deterministic_chat_tool_summary(&tool_result.follow_up_text); + + assert!(summary.is_some()); + assert!(!should_request_local_tool_follow_up( + "local", + &tool_result, + summary.as_deref() + )); + } + + #[test] + fn chat_tool_call_refusal_prioritizes_safe_review_wording() -> Result<()> { + let mut output = String::new(); + let (_root, paths) = test_paths("chat-tool-refusal"); + let response = providers::ChatResponse { + provider: "local".to_owned(), + model: "tiny.gguf".to_owned(), + content: "I can install ROCm.".to_owned(), + tool_calls: vec![providers::ChatToolCall { + id: None, + name: "install_sdk".to_owned(), + arguments: serde_json::json!({ + "channel": "release", + "format": "wheel", + "prefix": "D:\\ROCm\\therock_venvs" + }), + }], + }; + + let mut progress = None; + append_chat_tool_results( + &paths, + &response, + &mut output, + Some("I can install ROCm."), + &mut progress, + )?; + + assert!(output.contains("Install ROCm: needs your review")); + assert!(!output.contains("install_sdk: approval required")); + assert!(output.contains("not run: review the approval card before anything runs")); + assert!(output.contains("advanced manual command: rocm install sdk")); + assert!(!output.contains("run the shown ROCm command")); + Ok(()) + } + + #[test] + fn local_rocm_tools_chat_uses_fixed_lemonade_qwen_assistant() { + assert_eq!( + local_rocm_tools_assistant_model("local", true), + Some(providers::BUILTIN_ASSISTANT_MODEL_ID) + ); + assert_eq!(local_rocm_tools_assistant_model("local", false), None); + assert_eq!(local_rocm_tools_assistant_model("openai", true), None); + } + + #[test] + fn supplemental_tool_call_adds_missing_specific_status_check() { + let generic_services = providers::ChatToolCall { + id: Some("model-picked-services".to_owned()), + name: "services".to_owned(), + arguments: serde_json::json!({}), + }; + + let call = + supplemental_read_only_tool_call_for_prompt("Is ComfyUI running?", &[generic_services]) + .unwrap(); + + assert_eq!(call.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&call).unwrap(), + vec!["comfyui".to_owned(), "status".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&call)); + } + + #[test] + fn supplemental_tool_call_does_not_duplicate_equivalent_status_check() { + let comfy_status = fallback_rocm_tool_call_for_prompt("Is ComfyUI running?").unwrap(); + + assert!( + supplemental_read_only_tool_call_for_prompt("Is ComfyUI running?", &[comfy_status]) + .is_none() + ); + } + + #[test] + fn supplemental_tool_call_adds_running_state_for_engine_install_question() { + let engine_inventory = + fallback_rocm_tool_call_for_prompt("Is vLLM installed and is it running?").unwrap(); + assert_eq!( + normalized_chat_rocm_command_args(&engine_inventory).unwrap(), + vec!["engines".to_owned(), "list".to_owned()] + ); + + let services = supplemental_read_only_tool_call_for_prompt( + "Is vLLM installed and is it running?", + &[engine_inventory], + ) + .unwrap(); + + assert_eq!(services.name, "rocm_command"); + assert_eq!( + normalized_chat_rocm_command_args(&services).unwrap(), + vec!["services".to_owned(), "list".to_owned(), "--all".to_owned()] + ); + assert!(chat_tool_call_is_read_only(&services)); + } + + #[test] + fn supplemental_tool_call_treats_loopback_port_checks_as_equivalent() { + let model_port_check = providers::ChatToolCall { + id: Some("model-picked-port".to_owned()), + name: "port_status".to_owned(), + arguments: serde_json::json!({ "host": "localhost", "port": 8188 }), + }; + + assert!( + supplemental_read_only_tool_call_for_prompt( + "what is running on port 8188?", + &[model_port_check] + ) + .is_none() + ); + } + + #[test] + fn chat_install_intent_without_folder_uses_folder_picker_not_rocm_check() -> Result<()> { + let (_root, paths) = test_paths("chat-install-intent-folder-picker"); + for prompt in [ + "i need to install rocm", + "How do I setup TheRock?", + "install therock", + "How do I get it to install therock?", + "rocm please", + "make my AMD GPU ready", + "get TheRock installed for local AI", + "set up my AMD GPU for local AI", + "install this specific TheRock wheel from date 06052026", + ] { + let result = render_chat_prompt_result(&paths, "local", None, prompt, true)?; + let approval = result + .approval + .as_ref() + .expect("install should need folder"); + assert_eq!(approval.pending_title, "Install ROCm", "{prompt}"); + assert_eq!( + approval.args[..6], + [ + "install".to_owned(), + "sdk".to_owned(), + "--channel".to_owned(), + "release".to_owned(), + "--format".to_owned(), + "wheel".to_owned(), + ], + "{prompt}" + ); + assert!( + !approval.args.iter().any(|arg| arg == "--prefix"), + "{prompt}" + ); + assert!( + result.rendered.contains("First choose the folder"), + "{prompt}: {}", + result.rendered + ); + assert!( + !result.rendered.contains("I checked ROCm"), + "{prompt}: {}", + result.rendered + ); + assert!( + !result.rendered.contains("ROCm CLI summary"), + "{prompt}: {}", + result.rendered + ); + if prompt.contains("06052026") { + assert!(approval.args.contains(&"--build-date".to_owned())); + assert!(approval.args.contains(&"2026-06-05".to_owned())); + } + } + Ok(()) + } + + #[test] + fn chat_install_intent_ignores_old_conversation_words() -> Result<()> { + let (_root, paths) = test_paths("chat-install-intent-latest-message"); + let prompt = "\ +Conversation so far: +Assistant: Use /examine to refresh actual GPU memory fit before starting anything large. +Assistant: Native Windows note: models may use WSL/Linux through Windows. + +New message: +install therock"; + + let result = render_chat_prompt_result(&paths, "local", None, prompt, true)?; + let approval = result + .approval + .as_ref() + .expect("direct latest install request should need a folder"); + + assert_eq!(approval.pending_title, "Install ROCm"); + assert!(!approval.args.iter().any(|arg| arg == "--prefix")); + assert!(result.rendered.contains("I can install ROCm/TheRock")); + assert!(result.rendered.contains("First choose the folder")); + assert!(!result.rendered.contains("I checked ROCm")); + assert!(!result.rendered.contains("ROCm CLI summary")); + Ok(()) + } + + #[test] + fn local_chat_tool_call_content_is_treated_as_intermediate() { + let response = providers::ChatResponse { + provider: "local".to_owned(), + model: "Qwen/Qwen3-0.6B".to_owned(), + content: "The active runtime root is /opt/rocm.".to_owned(), + tool_calls: vec![providers::ChatToolCall { + id: Some("call-1".to_owned()), + name: "examine".to_owned(), + arguments: serde_json::json!({}), + }], + }; + + assert!(local_tool_call_content_is_intermediate( + "local", true, &response + )); + assert!(!local_tool_call_content_is_intermediate( + "openai", true, &response + )); + assert!(!local_tool_call_content_is_intermediate( + "local", false, &response + )); + + let without_tools = providers::ChatResponse { + tool_calls: Vec::new(), + ..response + }; + assert!(!local_tool_call_content_is_intermediate( + "local", + true, + &without_tools + )); + } + + #[test] + fn local_chat_follow_up_with_tool_call_is_not_final_answer() { + let response = providers::ChatResponse { + provider: "local".to_owned(), + model: "Qwen/Qwen3-0.6B".to_owned(), + content: "The runtime root is /opt/rocml.".to_owned(), + tool_calls: vec![providers::ChatToolCall { + id: Some("call-2".to_owned()), + name: "examine".to_owned(), + arguments: serde_json::json!({}), + }], + }; + + assert!(!local_follow_up_content_is_final(&response)); + + let final_answer = providers::ChatResponse { + tool_calls: Vec::new(), + content: "The runtime root is D:\\ROCm\\therock_venvs.".to_owned(), + ..response + }; + assert!(local_follow_up_content_is_final(&final_answer)); + } + + #[test] + fn visible_chat_content_removes_reasoning_blocks() { + assert_eq!( + visible_chat_content( + "\nchecking the tool output\n\nThe runtime root is D:\\ROCm\\temp." + ), + "The runtime root is D:\\ROCm\\temp." + ); + assert_eq!( + visible_chat_content("Before\nhidden\nAfter"), + "Before\n\nAfter" + ); + assert_eq!(visible_chat_content("unfinished"), ""); + } + + #[test] + fn chat_tool_result_errors_use_plain_failure_wording() { + assert_eq!(chat_read_only_tool_status_label(false), "done"); + assert_eq!(chat_read_only_tool_status_label(true), "reported an error"); + assert_eq!(chat_tool_display_label("examine"), "Checked this computer"); + assert_eq!( + chat_tool_display_label("gpu_snapshot"), + "Checked GPU status" + ); + assert_eq!(chat_tool_display_label("install_sdk"), "Install ROCm"); + assert!(mcp_tool_result_is_error(&serde_json::json!({ + "isError": true + }))); + assert!(!mcp_tool_result_is_error(&serde_json::json!({ + "isError": false + }))); + assert!(!mcp_tool_result_is_error(&serde_json::json!({}))); + } + + #[test] + fn local_chat_without_service_explains_serve_before_chat_without_llm_setup() { + let (_root, paths) = test_paths("local-chat-no-service-guidance"); + let result = + render_chat_prompt_result(&paths, "local", None, "Check this ROCm setup", true) + .expect("missing local assistant should render guidance"); + assert!(result.approval.is_none()); + let rendered = result.rendered; + + assert!(rendered.contains("No local assistant is running yet.")); + assert!(rendered.contains("First-time ROCm setup does not need an LLM")); + assert!(rendered.contains("Recommended path:")); + assert!(rendered.contains("Advanced manual command")); + assert!(rendered.contains( + "rocm serve Qwen3-4B-Instruct-2507-GGUF --engine lemonade --device gpu_required --managed" + )); + assert!(!rendered.contains("sshleifer/tiny-gpt2")); + assert!(rendered.contains("rocm chat --tools --provider local --prompt")); + assert!(rendered.contains("Nothing was changed.")); + assert!(!rendered.contains("install sdk")); + assert!(!rendered.contains("setup TheRock with an LLM")); + } + + #[test] + fn local_chat_status_prompts_use_read_only_tools_without_assistant() -> Result<()> { + let (_root, paths) = test_paths("local-chat-status-fallback"); + + let running = + render_chat_prompt_result(&paths, "local", None, "Is vLLM running?", true)?.rendered; + assert!(!running.contains("No local assistant is running yet.")); + assert!(running.contains("Checked model servers: done"), "{running}"); + assert!(running.contains("ROCm CLI summary"), "{running}"); + assert!( + running.contains("Local model servers: none running under ROCm CLI."), + "{running}" + ); + assert!(running.contains("Nothing was changed.")); + + let installed = + render_chat_prompt_result(&paths, "local", None, "Is vLLM installed?", true)?.rendered; + assert!(!installed.contains("No local assistant is running yet.")); + assert!(installed.contains("Engine runtimes:"), "{installed}"); + assert!(installed.contains("vLLM:"), "{installed}"); + + let installed_and_running = render_chat_prompt_result( + &paths, + "local", + None, + "Is vLLM installed and is it running?", + true, + )? + .rendered; + assert!( + installed_and_running.contains("Checked local engines: done"), + "{installed_and_running}" + ); + assert!( + installed_and_running.contains("Checked model servers: done"), + "{installed_and_running}" + ); + assert!( + installed_and_running.contains("Engine runtimes:"), + "{installed_and_running}" + ); + assert!( + installed_and_running.contains("Local model servers: none running under ROCm CLI."), + "{installed_and_running}" + ); + + let port = render_chat_prompt_result( + &paths, + "local", + None, + "What is running on port 8188?", + true, + )? + .rendered; + assert!(!port.contains("No local assistant is running yet.")); + assert!(port.contains("Checked local port: done"), "{port}"); + assert!(port.contains("Port 8188:"), "{port}"); + Ok(()) + } + + #[test] + fn chat_tools_anthropic_reaches_provider_opt_in_boundary() { + let (_root, paths) = test_paths("anthropic-chat-tools-opt-in"); + + let error = render_chat_prompt_result( + &paths, + "anthropic", + Some("claude-test"), + "Check this ROCm setup", + true, + ) + .unwrap_err() + .to_string(); + + assert!(error.contains("cloud provider `anthropic` is disabled")); + assert!(error.contains("rocm config enable-provider anthropic")); + assert!(!error.contains("OpenAI-compatible provider")); + } +} diff --git a/apps/rocm/src/comfyui.rs b/apps/rocm/src/comfyui.rs index 2d68ad6b81..bf3de3349a 100644 --- a/apps/rocm/src/comfyui.rs +++ b/apps/rocm/src/comfyui.rs @@ -2,6 +2,8 @@ // // SPDX-License-Identifier: MIT +use crate::cli::ComfyuiCommand; +use crate::cli_audit::record_cli_audit_event; use crate::therock; use anyhow::{Context, Result, bail}; use process::{comfyui_run_state_cli_label, endpoint_is_reachable, evaluate_running_state}; @@ -60,6 +62,169 @@ pub(crate) struct ComfyUiStartOptions { pub no_open_browser: bool, } +pub(crate) fn run(command: Option) -> Result<()> { + let paths = AppPaths::discover()?; + let config = RocmCliConfig::load(&paths).unwrap_or_default(); + match command.unwrap_or(ComfyuiCommand::Status) { + ComfyuiCommand::Status => { + print!("{}", render_status(&paths, &config)?); + record_cli_audit_event( + &paths, + "app", + "comfyui_status", + "info", + "rendered ComfyUI status", + None, + ); + Ok(()) + } + ComfyuiCommand::ModelsPath => { + print!("{}", render_models_path(&paths)?); + record_cli_audit_event( + &paths, + "app", + "comfyui_models_path", + "info", + "rendered ComfyUI models path", + None, + ); + Ok(()) + } + ComfyuiCommand::Logs { lines } => { + print!("{}", render_logs(&paths, lines)?); + record_cli_audit_event( + &paths, + "app", + "comfyui_logs", + "info", + "rendered ComfyUI logs", + None, + ); + Ok(()) + } + ComfyuiCommand::Install { + runtime_id, + reinstall, + dry_run, + yes: _, + } => { + match install( + &paths, + &config, + ComfyUiInstallOptions { + runtime_id: runtime_id.clone(), + reinstall, + dry_run, + }, + ) { + Ok(text) => { + print!("{text}"); + record_cli_audit_event( + &paths, + "app", + if dry_run { + "comfyui_install_dry_run" + } else { + "comfyui_install" + }, + "info", + format!( + "ComfyUI install completed runtime={} reinstall={} dry_run={}", + runtime_id.as_deref().unwrap_or(""), + reinstall, + dry_run + ), + None, + ); + Ok(()) + } + Err(error) => { + record_cli_audit_event( + &paths, + "app", + if dry_run { + "comfyui_install_dry_run" + } else { + "comfyui_install" + }, + "error", + format!( + "ComfyUI install failed runtime={} reinstall={} dry_run={}: {error}", + runtime_id.as_deref().unwrap_or(""), + reinstall, + dry_run + ), + None, + ); + Err(error) + } + } + } + ComfyuiCommand::Start { + host, + port, + no_open_browser, + yes: _, + } => match start( + &paths, + ComfyUiStartOptions { + host, + port, + no_open_browser, + }, + ) { + Ok(text) => { + print!("{text}"); + record_cli_audit_event( + &paths, + "app", + "comfyui_start", + "info", + "ComfyUI start requested", + None, + ); + Ok(()) + } + Err(error) => { + record_cli_audit_event( + &paths, + "app", + "comfyui_start", + "error", + format!("ComfyUI start failed: {error}"), + None, + ); + Err(error) + } + }, + ComfyuiCommand::Stop { yes: _ } => match stop(&paths) { + Ok(text) => { + print!("{text}"); + record_cli_audit_event( + &paths, + "app", + "comfyui_stop", + "info", + "ComfyUI stop requested", + None, + ); + Ok(()) + } + Err(error) => { + record_cli_audit_event( + &paths, + "app", + "comfyui_stop", + "error", + format!("ComfyUI stop failed: {error}"), + None, + ); + Err(error) + } + }, + } +} + #[derive(Debug, Clone, Serialize, Deserialize)] struct ComfyUiManifest { app_id: String, diff --git a/apps/rocm/src/config_cmd.rs b/apps/rocm/src/config_cmd.rs index 576a0813df..ffc81f3cd6 100644 --- a/apps/rocm/src/config_cmd.rs +++ b/apps/rocm/src/config_cmd.rs @@ -13,7 +13,7 @@ //! on, lives in `cli.rs` with the rest of the `Command`/`Cli` definition, and //! `dispatch()` calls `config` through the re-import `use //! crate::config_cmd::{...};`. `render_config_text` is also `pub(crate)`: -//! `chat.rs`'s read-only handling of `rocm config show` reaches it as +//! `chat/mcp_bridge.rs`'s read-only handling of `rocm config show` reaches it as //! `crate::render_config_text` through that re-import. `read_provider_key_from_user` //! and `telemetry_policy_summary` have no caller outside this module and stay //! private. diff --git a/apps/rocm/src/engine_runtime_select.rs b/apps/rocm/src/engine_runtime_select.rs index 73589e7967..793a82f3ab 100644 --- a/apps/rocm/src/engine_runtime_select.rs +++ b/apps/rocm/src/engine_runtime_select.rs @@ -66,8 +66,9 @@ //! `engine_request`; `parse_engine_recipe_json_arg` is `pub(crate)`, read //! from `main.rs`'s `dispatch()` and `service_control.rs`'s `restart_child_argv`. //! `app_path_env_vars`/`apply_app_path_env` are `pub(crate)`, read from -//! `engine_ipc.rs`, `app_path_env_vars` is also read from `chat.rs`, and -//! `apply_app_path_env` is also read from `engines_cmd.rs`, +//! `engine_ipc.rs`, `app_path_env_vars` is also read from +//! `chat/mcp_bridge.rs`, and `apply_app_path_env` is also read from +//! `engines_cmd.rs`, //! `service_launch.rs`, and `service_control.rs`'s //! `restart_internal_managed_service_with`; `app_path_env_var_values`/ //! `app_path_env_var_refs` are `pub(crate)`, read from `service_launch.rs` @@ -79,7 +80,7 @@ //! `restart_internal_managed_service_with`; `wait_for_service_http_ready_with_progress` //! is `pub(crate)`, read from `service_launch.rs`. `http_post_local_service_json` //! is `pub(crate)`, read from `service_control.rs`'s `unload_lemonade_service_model`. -//! `service_model_names_match` is `pub(crate)`, read from `chat.rs`. +//! `service_model_names_match` is `pub(crate)`, read from `chat/turn.rs`. //! `service_http_readiness_paths`, `http_get_local_service`, //! `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`, //! `model_list_ready_for_model`, `service_model_gpu_ready`, and diff --git a/apps/rocm/src/examine_report_text.rs b/apps/rocm/src/examine_report_text.rs index 2674b2a7d5..ec67be84b1 100644 --- a/apps/rocm/src/examine_report_text.rs +++ b/apps/rocm/src/examine_report_text.rs @@ -9,7 +9,7 @@ //! Relocated from `main.rs` with no behavior change (see //! `docs/architecture.md`). The `rocm model` catalog renderer used to live //! here too; it moved to `model_catalog_text.rs`, which this module does not -//! call into outside its own tests. +//! call into. //! //! This module owns: the launch-summary/examine text surfaces //! (`render_launch_summary`, `render_examine_text`, @@ -29,8 +29,8 @@ //! `launch_default` calls `render_launch_summary`, re-imported there. //! `render_examine_text`, `render_examine_text_with_paths` and //! `render_engine_inventory_text_with_paths` are `pub(crate)`, read from -//! [`chat.rs`](../apps/rocm/src/chat.rs)'s deterministic tool-output -//! summaries; `render_engine_inventory_text_with_paths` is also read from +//! [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only +//! command runner; `render_engine_inventory_text_with_paths` is also read from //! [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s //! `render_engine_inventory_text`. `append_examine_runtime_state` is also //! `pub(crate)`, read only from @@ -49,11 +49,9 @@ //! The regression tests for these surfaces live in this module's own //! `mod tests`, reaching the shared `test_paths` fixture through //! `crate::tests::` and `write_test_pip_runtime` through -//! `crate::test_support::`. One of them, -//! `linux_only_engine_runtime_status_is_explicit_on_native_windows`, also -//! calls `model_catalog_text.rs`'s `model_registry_adapter_availability_note`, -//! which is `pub(crate)` for that reason; its primary subject is -//! `engine_runtime_status_label`, so it stays here. +//! `crate::test_support::`. The note the `rocm model --verbose` listing prints +//! for a rule-out engine is tested in `model_catalog_text.rs`, next to the +//! renderer that prints it. use std::fmt::Write as _; use std::path::PathBuf; @@ -578,7 +576,6 @@ mod tests { use std::fs; use super::*; - use crate::model_catalog_text::model_registry_adapter_availability_note; use crate::runtimes_cmd::runtime_manifest_path; use crate::test_support::write_test_pip_runtime; use crate::tests::test_paths; @@ -891,13 +888,8 @@ mod tests { engine_runtime_status_label("vllm", &detect), "unsupported_native_windows" ); - assert!( - model_registry_adapter_availability_note("vllm", &|| true) - .is_some_and(|note| note.contains("unsupported_native_windows")) - ); } else { assert_eq!(engine_runtime_status_label("vllm", &detect), "not found"); - assert!(model_registry_adapter_availability_note("vllm", &|| true).is_none()); } } diff --git a/apps/rocm/src/freeform_planner.rs b/apps/rocm/src/freeform_planner.rs index 42f4593052..877c91e904 100644 --- a/apps/rocm/src/freeform_planner.rs +++ b/apps/rocm/src/freeform_planner.rs @@ -47,8 +47,6 @@ use std::fmt::Write as _; // as `freeform_planner::format_structured_tool_call` (read by `comfyui.rs`, // `freeform_execution.rs` and `providers.rs`'s tests). pub(crate) use crate::rocm_argv::format_structured_tool_call; -use crate::rocm_argv::serve_args_request_cpu_device; -use crate::service_launch::is_loopback_host; use crate::shell_quote::has_terminal_control_character; use crate::{Cli, prompt_text, providers, rocm_argv, tool_policy}; @@ -88,7 +86,7 @@ pub(crate) fn freeform_plan_next_action( /// Builds the freeform plan exactly once, returning both the rendered plan /// text and the next action derived from that same plan. /// -/// Used by `chat.rs`'s `natural_language_plan` MCP tool, which used to call +/// Used by `chat/mcp_bridge.rs`'s `natural_language_plan` MCP tool, which used to call /// `render_freeform_plan` and (the now-removed) /// `freeform_plan_next_action_with_context` independently — each building its /// own `StructuredRequestPlan`. When the request is ambiguous and a cloud @@ -1016,24 +1014,7 @@ pub(crate) fn validate_provider_planner_tool_call( [command, subcommand, ..] if command == "install" && subcommand == "driver" => {} [command, ..] if command == "update" => {} [command, ..] if command == "serve" => { - if rocm_argv::chat_cli_has_flag(&call.args, "--allow-public-bind") { - bail!("provider planner cannot request public network binding"); - } - if serve_args_request_cpu_device(&call.args) { - bail!( - "provider planner cannot request CPU execution; rocm-cli requires ROCm GPU execution" - ); - } - if let Some(host) = rocm_argv::chat_cli_arg_value(&call.args, "--host") - && !is_loopback_host(host) - { - bail!("provider planner cannot request non-local host `{host}`"); - } - if rocm_argv::chat_cli_has_flag(&call.args, "--foreground") - || !rocm_argv::chat_cli_has_flag(&call.args, "--managed") - { - bail!("provider planner must request managed serving with --managed"); - } + tool_policy::validate_serve_command_safety(&call.args, "provider planner")?; } [command, ..] if command == "uninstall" => {} _ => bail!( @@ -2020,4 +2001,209 @@ mod tests { ); Ok(()) } + + /// The planner's response for a `serve` call, as the provider would send it. + fn serve_planner_response(args: &[&str]) -> String { + serde_json::json!({ + "intent": "serve", + "tool_call": {"tool": "rocm", "args": args}, + }) + .to_string() + } + + /// The serve safety rules as the provider-assisted planner meets them: every + /// refusal in the shared table, at the validator and at the response parser + /// that builds the plan. A refused response yields no `StructuredRequestPlan`, + /// so there is no planned action for anything to run, and the operator sees + /// the refusal as the planner's failure note. + /// + /// The chat suite (`tool_policy`) runs the same table. + #[test] + fn planner_refuses_each_unsafe_serve_and_plans_nothing() { + use crate::test_support::{SERVE_SAFETY_REFUSALS, owned_argv}; + + for case in SERVE_SAFETY_REFUSALS { + let context = format!("{}: {:?}", case.rule, case.args); + let call = ProviderPlannerToolCall { + tool: "rocm".to_owned(), + args: owned_argv(case.args), + }; + let error = validate_provider_planner_tool_call(&call).expect_err(&context); + assert_eq!(error.to_string(), case.planner_error, "{context}"); + + let error = provider_planner_response_to_plan( + "serve a model", + "local", + &serve_planner_response(case.args), + ) + .expect_err(&context); + assert_eq!(error.to_string(), case.planner_error, "{context}"); + assert_eq!( + planner_failure_note("local", &error), + format!( + "Provider-assisted planner `local` could not resolve this request: {}", + case.planner_error + ), + "{context}" + ); + } + } + + #[test] + fn planner_plans_a_safe_serve_for_approval_with_its_argv_unchanged() -> Result<()> { + use crate::test_support::{SERVE_SAFETY_ACCEPTED, owned_argv}; + + for args in SERVE_SAFETY_ACCEPTED { + let plan = provider_planner_response_to_plan( + "serve a model", + "local", + &serve_planner_response(args), + )?; + assert_eq!(plan.intent, PlannerIntent::Serve, "{args:?}"); + // A serve starts a server, so the plan always asks for approval. + assert_eq!(plan.approval, "required before execution", "{args:?}"); + assert_eq!(plan.actions.len(), 1, "{args:?}"); + assert_eq!(plan.actions[0].args, owned_argv(args), "{args:?}"); + } + Ok(()) + } + + /// The planner has no port rule: a loopback host other than `127.0.0.1` + /// without `--port` is planned, where the chat policy refuses it + /// (`chat_requires_a_port_on_a_loopback_host_other_than_the_default`). + #[test] + fn planner_accepts_a_loopback_host_without_a_port() -> Result<()> { + use crate::test_support::{SERVE_PORT_RULE_CHAT_ONLY, owned_argv}; + + for (args, _) in SERVE_PORT_RULE_CHAT_ONLY { + let plan = provider_planner_response_to_plan( + "serve a model", + "local", + &serve_planner_response(args), + )?; + assert_eq!(plan.actions[0].args, owned_argv(args), "{args:?}"); + } + Ok(()) + } + + /// The rules fire in a fixed order, so an argv that breaks several reports + /// the first one: public bind, then CPU, then host, then managed. + #[test] + fn planner_reports_the_first_broken_serve_rule() { + use crate::test_support::owned_argv; + + for (args, expected) in [ + ( + &[ + "serve", + "qwen", + "--allow-public-bind", + "--device", + "cpu", + "--host", + "0.0.0.0", + "--foreground", + ][..], + "provider planner cannot request public network binding", + ), + ( + &[ + "serve", + "qwen", + "--device", + "cpu", + "--host", + "0.0.0.0", + "--foreground", + ][..], + "provider planner cannot request CPU execution; ROCm GPU execution is required", + ), + ( + &["serve", "qwen", "--host", "0.0.0.0", "--foreground"][..], + "provider planner cannot request non-local host `0.0.0.0`", + ), + ( + &["serve", "qwen", "--foreground"][..], + "provider planner must request managed serving with --managed", + ), + ] { + let call = ProviderPlannerToolCall { + tool: "rocm".to_owned(), + args: owned_argv(args), + }; + let error = validate_provider_planner_tool_call(&call) + .expect_err("unsafe") + .to_string(); + assert_eq!(error, expected, "{args:?}"); + } + } + + /// The planner and the chat tool policy enforce one set of serve rules, so + /// they refuse an argv with the same words and differ only in who is named + /// at the front of the message. + #[test] + fn planner_and_chat_serve_refusals_differ_in_the_actor_only() { + use crate::test_support::SERVE_SAFETY_REFUSALS; + + for case in SERVE_SAFETY_REFUSALS { + let chat = case + .chat_error + .strip_prefix("local assistant ") + .unwrap_or_else(|| panic!("chat refusal names the assistant: {}", case.chat_error)); + let planner = case + .planner_error + .strip_prefix("provider planner ") + .unwrap_or_else(|| { + panic!("planner refusal names the planner: {}", case.planner_error) + }); + assert_eq!(chat, planner, "{}: {:?}", case.rule, case.args); + } + } + + /// `--foreground` next to `--managed` is refused by clap before the planner's + /// safety check runs, where the chat check refuses it for the managed rule. + #[test] + fn planner_refuses_foreground_with_managed_as_an_invalid_command() { + use crate::test_support::{SERVE_FOREGROUND_AND_MANAGED, owned_argv}; + + let call = ProviderPlannerToolCall { + tool: "rocm".to_owned(), + args: owned_argv(SERVE_FOREGROUND_AND_MANAGED), + }; + let error = validate_provider_planner_tool_call(&call) + .expect_err("clap refuses the conflicting flags") + .to_string(); + assert_eq!( + error, + "provider planner returned a rocm command that is not valid" + ); + } + + /// The planner matches the `serve` word exactly, where the chat check + /// ignores case, so `Serve` and `SERVE` never reach the planner's safety + /// check: the clap parse that precedes it refuses them. Either way an + /// uppercase `serve` is refused; only the reason differs. + #[test] + fn planner_refuses_an_uppercase_serve_word_before_the_safety_check() { + use crate::test_support::owned_argv; + + for word in ["Serve", "SERVE"] { + for args in [ + &[word, "qwen", "--device", "cpu", "--managed"][..], + &[word, "qwen", "--managed"][..], + ] { + let call = ProviderPlannerToolCall { + tool: "rocm".to_owned(), + args: owned_argv(args), + }; + let error = validate_provider_planner_tool_call(&call) + .expect_err("an uppercase serve word is not a rocm command") + .to_string(); + assert_eq!( + error, "provider planner returned a rocm command that is not valid", + "{args:?}" + ); + } + } + } } diff --git a/apps/rocm/src/install_cmd.rs b/apps/rocm/src/install_cmd.rs index 76d0c97194..f7174945cd 100644 --- a/apps/rocm/src/install_cmd.rs +++ b/apps/rocm/src/install_cmd.rs @@ -15,8 +15,8 @@ //! live in `cli.rs` with the rest of the `Command`/`Cli` definition. //! `dispatch()` calls `install` through the re-import `use //! crate::install_cmd::{...};`. `therock_install_version_selector` is also -//! `pub(crate)`: `chat.rs`'s `render_install_sdk_dry_run_for_args` calls it as -//! `crate::therock_install_version_selector` through that re-import, which is +//! `pub(crate)`: `chat/mcp_bridge.rs`'s `render_install_sdk_dry_run_for_args` +//! calls it as `crate::therock_install_version_selector` through that re-import, which is //! why `sdk_install_request` moved here rather than staying in `chat.rs` — //! this module does not depend on `chat.rs`. `sdk_install_request` keeps the //! `pub(crate)` it had in `chat.rs`, although only `install` and this module's diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 573044fb9a..e879060b5c 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -10,6 +10,7 @@ mod advised_commands; #[path = "../../../crates/rocm-core/src/archive_props.rs"] mod archive_props; mod automations; +mod background_helper; mod bootstrap; mod bridge_snapshot; mod chat; @@ -107,12 +108,13 @@ mod update_cmd; // reaches them via its own imports instead. // `record_cli_audit_event`, `cli_lifecycle_log_path` and `sanitize_log_value` // (from `cli_audit.rs`) are re-imported so other modules keep reaching them as -// `crate::`; `main.rs` itself calls `record_cli_audit_event`. +// `crate::`. // `therock_install_version_selector` (from `install_cmd.rs`) is re-imported for -// the same reason: `chat.rs` reaches it as `crate::therock_install_version_selector`. +// the same reason: `chat/mcp_bridge.rs` reaches it as +// `crate::therock_install_version_selector`. // `build_codex_bridge_gpu_snapshot` and `builtin_codex_bridge_engine_inventory` -// (from `bridge_snapshot.rs`) are re-imported only so `chat.rs` keeps reaching -// them as `crate::`; `build_codex_bridge_snapshot` is called from +// (from `bridge_snapshot.rs`) are re-imported only so `chat/mcp_bridge.rs` keeps +// reaching them as `crate::`; `build_codex_bridge_snapshot` is called from // `dispatch()` as well. // The clap command tree and its value enums (from `cli.rs`) are re-imported so // every module keeps reaching them as `crate::Command`, `crate::Provider` and so @@ -124,9 +126,9 @@ use crate::bridge_snapshot::{ }; use crate::chat_cmd::run_chat_command; use crate::cli::{ - AutomationsCommand, BenchCommand, Cli, ComfyuiCommand, Command, ConfigCommand, DevicePolicyArg, - EnginesCommand, InstallFormat, InstallTarget, Provider, RuntimesCommand, SandboxToolArg, - ServicesCommand, SetupCommand, StorageCommand, + AutomationsCommand, BenchCommand, Cli, Command, ConfigCommand, DevicePolicyArg, EnginesCommand, + InstallFormat, InstallTarget, Provider, RuntimesCommand, SandboxToolArg, ServicesCommand, + SetupCommand, StorageCommand, }; // Reached only as `super::SUPPORTED_ENGINES` from `advised_commands.rs`'s test, // so a non-test build does not carry it. @@ -860,7 +862,7 @@ fn dispatch(cli: Cli) -> Result<()> { api_key, engine_env, }), - Some(Command::Comfyui { command }) => comfyui(command), + Some(Command::Comfyui { command }) => comfyui::run(command), Some(Command::Services { command }) => services(command), Some(Command::Remote { command }) => remote::run(command), Some(Command::Automations { command }) => automations(command), @@ -1098,169 +1100,6 @@ fn platform_refusal(ruled_out: PlatformRuleOut, action: GatedEngineAction) -> Op } } -fn comfyui(command: Option) -> Result<()> { - let paths = AppPaths::discover()?; - let config = RocmCliConfig::load(&paths).unwrap_or_default(); - match command.unwrap_or(ComfyuiCommand::Status) { - ComfyuiCommand::Status => { - print!("{}", comfyui::render_status(&paths, &config)?); - record_cli_audit_event( - &paths, - "app", - "comfyui_status", - "info", - "rendered ComfyUI status", - None, - ); - Ok(()) - } - ComfyuiCommand::ModelsPath => { - print!("{}", comfyui::render_models_path(&paths)?); - record_cli_audit_event( - &paths, - "app", - "comfyui_models_path", - "info", - "rendered ComfyUI models path", - None, - ); - Ok(()) - } - ComfyuiCommand::Logs { lines } => { - print!("{}", comfyui::render_logs(&paths, lines)?); - record_cli_audit_event( - &paths, - "app", - "comfyui_logs", - "info", - "rendered ComfyUI logs", - None, - ); - Ok(()) - } - ComfyuiCommand::Install { - runtime_id, - reinstall, - dry_run, - yes: _, - } => { - match comfyui::install( - &paths, - &config, - comfyui::ComfyUiInstallOptions { - runtime_id: runtime_id.clone(), - reinstall, - dry_run, - }, - ) { - Ok(text) => { - print!("{text}"); - record_cli_audit_event( - &paths, - "app", - if dry_run { - "comfyui_install_dry_run" - } else { - "comfyui_install" - }, - "info", - format!( - "ComfyUI install completed runtime={} reinstall={} dry_run={}", - runtime_id.as_deref().unwrap_or(""), - reinstall, - dry_run - ), - None, - ); - Ok(()) - } - Err(error) => { - record_cli_audit_event( - &paths, - "app", - if dry_run { - "comfyui_install_dry_run" - } else { - "comfyui_install" - }, - "error", - format!( - "ComfyUI install failed runtime={} reinstall={} dry_run={}: {error}", - runtime_id.as_deref().unwrap_or(""), - reinstall, - dry_run - ), - None, - ); - Err(error) - } - } - } - ComfyuiCommand::Start { - host, - port, - no_open_browser, - yes: _, - } => match comfyui::start( - &paths, - comfyui::ComfyUiStartOptions { - host, - port, - no_open_browser, - }, - ) { - Ok(text) => { - print!("{text}"); - record_cli_audit_event( - &paths, - "app", - "comfyui_start", - "info", - "ComfyUI start requested", - None, - ); - Ok(()) - } - Err(error) => { - record_cli_audit_event( - &paths, - "app", - "comfyui_start", - "error", - format!("ComfyUI start failed: {error}"), - None, - ); - Err(error) - } - }, - ComfyuiCommand::Stop { yes: _ } => match comfyui::stop(&paths) { - Ok(text) => { - print!("{text}"); - record_cli_audit_event( - &paths, - "app", - "comfyui_stop", - "info", - "ComfyUI stop requested", - None, - ); - Ok(()) - } - Err(error) => { - record_cli_audit_event( - &paths, - "app", - "comfyui_stop", - "error", - format!("ComfyUI stop failed: {error}"), - None, - ); - Err(error) - } - }, - } -} - /// Shown after any activation (direct `runtimes activate`, `update --apply /// --activate`, or an `install sdk` that switches the active runtime) that /// recorded a previous runtime, so every path that reaches this state gives diff --git a/apps/rocm/src/managed_service_status.rs b/apps/rocm/src/managed_service_status.rs index f0fd2b5bff..586adccf94 100644 --- a/apps/rocm/src/managed_service_status.rs +++ b/apps/rocm/src/managed_service_status.rs @@ -22,13 +22,14 @@ //! attempts counts and one-line status text //! (`ManagedServiceSidebarCounts`/`managed_service_sidebar_counts`/ //! `local_server_sidebar_status`) plus the engine display-name helper -//! (`friendly_engine_label`), which only `chat.rs` reads. `service_records.rs` +//! (`friendly_engine_label`), which only `chat/summaries.rs` reads. `service_records.rs` //! calls into it for the in-use check behind service-record removal and pruning //! (`managed_service_is_live`, `recorded_service_pids`) and for //! `load_managed_services` in prune planning, and `bridge_snapshot.rs` calls //! `load_managed_services` for the codex-bridge snapshot; //! `services_text.rs` reads the sidebar helpers for the local-servers view; -//! `chat.rs` calls into it for the deterministic services-tool summary; +//! `chat/turn.rs` and `chat/mcp_bridge.rs` call into it for the services tool +//! and the local-assistant readiness check; //! `runtime_services.rs` calls into it when reconciling live servers against a //! runtime activation; `service_launch.rs` is the only caller of the //! `existing_live_managed_service_in` duplicate-launch guard it runs before diff --git a/apps/rocm/src/model_catalog_text.rs b/apps/rocm/src/model_catalog_text.rs index c9cafd54e1..c10b270e5b 100644 --- a/apps/rocm/src/model_catalog_text.rs +++ b/apps/rocm/src/model_catalog_text.rs @@ -20,10 +20,8 @@ //! `select_serve_engine`. //! //! The two renderers and `format_bytes` are `pub(crate)`; `main.rs` imports them -//! and `chat.rs`, `storage.rs` and `automations.rs` reach them as -//! `crate::`. `model_registry_adapter_availability_note` is `pub(crate)` -//! only so the engine-status test that stays in `examine_report_text.rs` can -//! call it. Every other item is private. +//! and `chat/mcp_bridge.rs`, `storage.rs` and `automations.rs` reach them as +//! `crate::`. Every other item is private. use std::collections::BTreeMap; use std::fmt::Write as _; @@ -819,7 +817,7 @@ fn append_model_engine_support_lines( } #[allow(dead_code)] -pub(crate) fn model_registry_adapter_availability_note( +fn model_registry_adapter_availability_note( engine: &str, docker_reachable: &dyn Fn() -> bool, ) -> Option<&'static str> { @@ -1407,4 +1405,21 @@ mod tests { assert!(!rendered.contains("CPU fallback")); assert!(rendered.contains("engine_action: use /engine install ")); } + + /// The adapter-availability note the verbose listing appends for vLLM: a + /// native Windows host gets the explicit `unsupported_native_windows` + /// status, every other host gets no note. Moved here from the + /// `engine_runtime_status_label` test in `examine_report_text.rs`, which + /// keeps the `runtime:` label half of the same check. + #[test] + fn vllm_adapter_availability_note_is_explicit_on_native_windows() { + if cfg!(windows) { + assert!( + model_registry_adapter_availability_note("vllm", &|| true) + .is_some_and(|note| note.contains("unsupported_native_windows")) + ); + } else { + assert!(model_registry_adapter_availability_note("vllm", &|| true).is_none()); + } + } } diff --git a/apps/rocm/src/prompt_text.rs b/apps/rocm/src/prompt_text.rs index 7647c532f7..8324c1fd89 100644 --- a/apps/rocm/src/prompt_text.rs +++ b/apps/rocm/src/prompt_text.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Text extraction from a natural-language request, shared by `chat.rs`'s +//! Text extraction from a natural-language request, shared by `chat/intent.rs`'s //! prompt-intent heuristics (`fallback_rocm_tool_call_for_prompt`, //! `fallback_config_tool_call_for_prompt`, `install_sdk_chat_approval_for_prompt` //! and the `prompt_*` predicates) and the freeform planner: which of a list of diff --git a/apps/rocm/src/providers.rs b/apps/rocm/src/providers.rs index 87d6c9ff44..b40c3209c2 100644 --- a/apps/rocm/src/providers.rs +++ b/apps/rocm/src/providers.rs @@ -1068,7 +1068,7 @@ mod tests { #[test] fn freeform_plan_dedup_prevents_provider_assisted_divergence() -> Result<()> { - // Regression for the `chat.rs` `natural_language_plan` call site, + // Regression for the `chat/mcp_bridge.rs` `natural_language_plan` call site, // which used to call `render_freeform_plan` and the now-removed // `freeform_plan_next_action_with_context` independently -- each // calling `build_freeform_plan_with_context` on its own. diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 9f5895d3e3..7147245f2c 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -74,6 +74,7 @@ use crate::serve_recipe::{ parse_gpu_memory_utilization, protocol_engine_recipe_hint, render_serve_engine_recipe_lines, }; use crate::serve_summary; +use crate::service_control::run_internal_sandbox_tool; use crate::service_launch::{ drop_orphaned_endpoint_key_on_already_running, e2e_simulate_oom_launch, ensure_background_helper_running_quiet, ensure_public_bind_engine_supported, @@ -82,8 +83,8 @@ use crate::service_launch::{ wait_begin_client_config, }; use crate::{ - GatedEngineAction, cli_progress, device_policy_name, ensure_engine_allowed_on_os, - ensure_engine_allowed_on_platform, + GatedEngineAction, SandboxToolArg, cli_progress, device_policy_name, + ensure_engine_allowed_on_os, ensure_engine_allowed_on_platform, }; #[derive(Debug, Clone, Eq, PartialEq)] @@ -1314,6 +1315,14 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { launch_lock, require_api_key, gpu_refusal_waived_for_reuse, + |paths, service_id| { + run_internal_sandbox_tool( + paths, + SandboxToolArg::StopServer, + Some(service_id.to_owned()), + true, + ) + }, ) } diff --git a/apps/rocm/src/service_control.rs b/apps/rocm/src/service_control.rs index 7bc72177c3..2f4db4617e 100644 --- a/apps/rocm/src/service_control.rs +++ b/apps/rocm/src/service_control.rs @@ -22,7 +22,9 @@ //! Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are //! `pub(crate)`. `run_internal_sandbox_tool` is read from `main.rs`'s //! `dispatch()`, `services_cmd.rs`'s `run_approved_service_action` and -//! `service_launch.rs`; `restart_internal_managed_service` from +//! `serve_cmd.rs`, which hands it to `service_launch.rs`'s +//! `run_attached_service` as the stop for a Ctrl-C; +//! `restart_internal_managed_service` from //! `runtime_services.rs`. Every other item has no caller outside this module and //! stays private. //! @@ -30,9 +32,9 @@ //! (`load_managed_service`, `read_managed_service_manifest`) and shapes the //! `stop_server` answer with its `internal_stop_server_value`, which stays there //! next to `service_stop_unconfirmed` and the renderer that reads the same -//! `stop_unconfirmed` status. `service_launch.rs` and this module import each -//! other: this module takes `ensure_public_service_has_endpoint_key` and the -//! spawn helpers from it, and it takes `run_internal_sandbox_tool` from here. +//! `stop_unconfirmed` status. This module takes +//! `ensure_public_service_has_endpoint_key` and the spawn helpers from +//! `service_launch.rs`, which does not import this module back. //! //! The regression tests for the stop and restart paths live in this module's own //! `mod tests`. They reach shared fixtures (`test_paths`, `ScopedTestEnv`, diff --git a/apps/rocm/src/service_launch.rs b/apps/rocm/src/service_launch.rs index 720d9312c5..53f9b5c41d 100644 --- a/apps/rocm/src/service_launch.rs +++ b/apps/rocm/src/service_launch.rs @@ -39,10 +39,11 @@ use std::thread; use std::time::Duration; use anyhow::{Context, Result, anyhow, bail}; -use rocm_core::{AppPaths, AutomationRuntimeState, EndpointReadiness, ManagedServiceRecord}; +use rocm_core::{AppPaths, EndpointReadiness, ManagedServiceRecord}; use rocm_core::{format_http_base_url, process_is_running}; use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse}; +use crate::background_helper::background_helper_already_running; use crate::endpoint_keys; use crate::engine_ipc::builtin_engine_serve_http_args; #[cfg(not(windows))] @@ -57,14 +58,13 @@ use crate::managed_service_status::{ existing_live_managed_service_in, load_managed_services, managed_service_is_live, }; use crate::serve_port; -use crate::service_control::run_internal_sandbox_tool; use crate::service_hints; use crate::services_text::read_optional_tail_lines; use crate::services_text::{ render_service_action_result, render_service_action_result_with_helper_note, service_stop_unconfirmed, }; -use crate::{SandboxToolArg, device_policy_name, record_cli_audit_event, status_for_readiness}; +use crate::{device_policy_name, record_cli_audit_event, status_for_readiness}; pub(crate) fn validate_bind_host(host: &str, allow_public_bind: bool) -> Result<()> { if !is_loopback_host(host) && !allow_public_bind { @@ -1574,74 +1574,6 @@ fn render_endpoint_client_config(endpoint_url: &str, api_key: &str) -> String { out } -/// The "should we spawn?" decision for [`ensure_background_helper_running`], -/// factored out so it is testable hermetically (no spawn side effect). Returns -/// `true` when the file-based runtime state says the daemon is `running` AND its -/// recorded `daemon_pid` is a live process — i.e. a second spawn must be guarded. -/// A missing state file, `running=false`, or a dead/zero pid returns `false`. -fn background_helper_already_running(paths: &AppPaths) -> Result { - Ok(AutomationRuntimeState::load(paths)? - .is_some_and(|state| state.running && rocm_core::process_is_running(state.daemon_pid))) -} - -/// What the user should be told about the background automation helper -/// (`rocm daemon`) after stopping a service or when planning an uninstall. -#[derive(Debug, Clone, PartialEq, Eq)] -pub(crate) struct HelperNote { - /// The sentence to print. - pub(crate) text: String, - /// The helper is running right now. - pub(crate) running: bool, - /// An automation watcher is enabled, so the helper stays up on its own. - pub(crate) outlives: bool, -} - -/// The note for one combination of the two facts the helper itself acts on: -/// whether a watcher is enabled -/// ([`rocm_core::RocmCliConfig::automation_daemon_enabled`] gates both the spawn -/// and the helper's exit) and whether it is running. `None` for neither. -fn helper_note_for(enabled: &[&str], running: bool) -> Option { - let text = match (enabled.is_empty(), running) { - (true, false) => return None, - (true, true) => "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds".to_owned(), - (false, true) => { - let disable = enabled - .iter() - .map(|id| format!("`rocm automations disable {id}`")) - .collect::>() - .join(", "); - format!( - "the background automation helper (`rocm daemon`) keeps running while an automation is enabled ({}); to stop it, run {disable}, after which it exits on its own within a few seconds", - enabled.join(", ") - ) - } - (false, false) => format!( - "an automation is enabled ({}) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it", - enabled.join(", ") - ), - }; - Some(HelperNote { - text, - running, - outlives: !enabled.is_empty() && running, - }) -} - -/// [`helper_note_for`] over the facts on disk. An unreadable config counts as -/// "no watcher enabled". -pub(crate) fn background_helper_note(paths: &AppPaths) -> Option { - let config = rocm_core::RocmCliConfig::load(paths).unwrap_or_default(); - let enabled: Vec<&str> = config - .automations - .watchers - .iter() - .filter(|(_, watcher)| watcher.enabled) - .map(|(id, _)| id.as_str()) - .collect(); - let running = background_helper_already_running(paths).unwrap_or(false); - helper_note_for(&enabled, running) -} - /// Shared daemon-lifecycle entrypoint: ensures the background automation helper /// (`rocm daemon`) is running when an automation watcher needs it, spawning it /// detached if not. Liveness is read from the file-based automation runtime @@ -1861,6 +1793,11 @@ enum AttachOutcome { /// detached managed child (the same child the background path spawns) and stream /// its log in this terminal. Unlike the old in-process foreground, the server /// survives the session — Ctrl-D detaches and leaves it running, Ctrl-C stops it. +/// +/// `stop_service` is how Ctrl-C stops the server: the caller passes the managed +/// stop, so this module does not import the stop path, which itself imports this +/// module's spawn helpers. It returns the same `stop_server` answer +/// [`finish_attached_stop`] reads. #[allow(clippy::too_many_arguments)] pub(crate) fn run_attached_service( paths: &AppPaths, @@ -1877,6 +1814,7 @@ pub(crate) fn run_attached_service( launch_lock: rocm_core::FileLock, require_api_key: bool, gpu_refusal_waived_for_reuse: bool, + stop_service: impl FnOnce(&AppPaths, &str) -> Result, ) -> Result<()> { let spawn = spawn_managed_engine_child( paths, @@ -1986,12 +1924,7 @@ pub(crate) fn run_attached_service( } AttachOutcome::Stop => { println!("stopping server..."); - let stop = run_internal_sandbox_tool( - paths, - SandboxToolArg::StopServer, - Some(service_id.clone()), - true, - ); + let stop = stop_service(paths, &service_id); finish_attached_stop( paths, &service_id, @@ -2287,6 +2220,7 @@ mod tests { } } use super::*; + use crate::test_support::{runtime_state, set_helper_facts}; #[test] fn a_child_that_exited_during_the_startup_window_is_reported_with_its_log() -> Result<()> { @@ -3606,6 +3540,7 @@ mod tests { launch_lock, false, false, + |_, _| unreachable!("the launch fails before a Ctrl-C could stop it"), ); let Err(error) = result else { let _ = fs::remove_dir_all(root); @@ -3679,6 +3614,7 @@ mod tests { launch_lock, false, false, + |_, _| unreachable!("the spawn fails before a Ctrl-C could stop it"), ) } else { start_managed_service( @@ -3920,64 +3856,6 @@ mod tests { assert!(rendered.contains("shown only now"), "{rendered}"); } - /// Build an `AutomationRuntimeState` for the no-double-spawn guard tests. - fn runtime_state(running: bool, daemon_pid: u32) -> AutomationRuntimeState { - AutomationRuntimeState { - running, - automations_enabled: true, - daemon_pid, - started_at_unix_ms: 1, - last_tick_unix_ms: 1, - local_webhook_endpoint: None, - active_watchers: Vec::new(), - } - } - - #[test] - fn background_helper_already_running_true_for_live_pid() { - // Phase-10 daemon no-double-spawn: a runtime-state.json with running=true - // and a LIVE daemon_pid (this very test process) means the helper is - // already up — the "should spawn?" decision must say NO (true ⇒ skip). - // Hermetic + offline: no spawn, just the file-based liveness check. - let (root, paths) = test_paths("helper-live-pid"); - runtime_state(true, std::process::id()) - .write(&paths) - .expect("write runtime state"); - assert!( - background_helper_already_running(&paths).expect("liveness check ok"), - "live recorded pid + running=true ⇒ do not spawn a second daemon" - ); - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn background_helper_already_running_false_for_dead_or_missing() { - // The inverse guard cases — each must report NOT running (false ⇒ spawn): - // (1) no state file at all, (2) running=true but a dead/zero pid, - // (3) a live pid but running=false. None must spawn from this decision. - let (root, paths) = test_paths("helper-dead-or-missing"); - // (1) No state file yet. - assert!( - !background_helper_already_running(&paths).expect("missing state ⇒ ok"), - "no runtime state ⇒ not running" - ); - // (2) running=true but pid 0 is never a live process. - runtime_state(true, 0).write(&paths).expect("write state"); - assert!( - !background_helper_already_running(&paths).expect("dead pid ⇒ ok"), - "running=true + dead pid ⇒ not running (spawn)" - ); - // (3) live pid but running flag is false. - runtime_state(false, std::process::id()) - .write(&paths) - .expect("write state"); - assert!( - !background_helper_already_running(&paths).expect("not-running flag ⇒ ok"), - "running=false ⇒ not running even with a live pid" - ); - let _ = fs::remove_dir_all(&root); - } - /// ROCMAI-827: `rocm serve` started the background daemon on every launch, /// and with no automation watcher enabled that daemon had nothing to do yet /// kept running after every service was stopped. With the default config @@ -4107,54 +3985,6 @@ mod tests { }}}) } - /// Put the on-disk facts in one of the four combinations: whether a watcher - /// is enabled, and whether the helper is running (this process stands in for - /// a live daemon). - fn set_helper_facts(paths: &AppPaths, watcher_enabled: bool, helper_running: bool) { - let mut config = rocm_core::RocmCliConfig::default(); - config.watcher_config_mut("server-recover").enabled = watcher_enabled; - config.save(paths).expect("save config"); - runtime_state(helper_running, std::process::id()) - .write(paths) - .expect("write runtime state"); - } - - /// ROCMAI-827: the helper note is derived from both facts, so it cannot - /// disagree with the state it describes. Each combination: the exact wording, - /// and the running/outlives flags the uninstall plan acts on. - #[test] - fn helper_note_follows_watcher_enabled_and_helper_running() { - let (root, paths) = test_paths("helper-note-combos"); - paths.ensure().expect("create app dirs"); - - set_helper_facts(&paths, false, false); - let neither = background_helper_note(&paths); - set_helper_facts(&paths, false, true); - let idle_running = background_helper_note(&paths).expect("note"); - set_helper_facts(&paths, true, false); - let enabled_dead = background_helper_note(&paths).expect("note"); - set_helper_facts(&paths, true, true); - let enabled_running = background_helper_note(&paths).expect("note"); - let _ = fs::remove_dir_all(&root); - - assert_eq!(neither, None); - assert_eq!( - idle_running.text, - "the background automation helper (`rocm daemon`) is still running; no automation is enabled, so it exits on its own within a few seconds" - ); - assert!(idle_running.running && !idle_running.outlives); - assert_eq!( - enabled_dead.text, - "an automation is enabled (server-recover) but the background automation helper (`rocm daemon`) is not running; `rocm serve` or `rocm automations enable` starts it" - ); - assert!(!enabled_dead.running && !enabled_dead.outlives); - assert_eq!( - enabled_running.text, - "the background automation helper (`rocm daemon`) keeps running while an automation is enabled (server-recover); to stop it, run `rocm automations disable server-recover`, after which it exits on its own within a few seconds" - ); - assert!(enabled_running.running && enabled_running.outlives); - } - /// `rocm services stop` prints the note for a stop only, and only the variant /// that matches the state. #[test] @@ -4189,29 +4019,6 @@ mod tests { ); } - /// The command the note names must exist and accept exactly those arguments, - /// and its effect (the helper exiting once the last watcher is off) is - /// `automations-11` in `features/automations.feature` plus the rocmd - /// `automation_loop_` tests. - #[test] - fn the_command_named_by_the_helper_note_parses() { - use clap::Parser as _; - - let (root, paths) = test_paths("helper-note-command"); - paths.ensure().expect("create app dirs"); - set_helper_facts(&paths, true, true); - let note = background_helper_note(&paths).expect("a note"); - let _ = fs::remove_dir_all(&root); - - let command = note - .text - .split('`') - .find(|part| part.starts_with("rocm automations disable")) - .expect("the note names the disable command"); - let argv: Vec<&str> = command.split_whitespace().collect(); - assert!(crate::Cli::try_parse_from(argv).is_ok(), "{command}"); - } - /// `serve` calls the helper autostart after its child is already running. An /// unreadable config must not turn that launch into an error: it reads as "no /// watcher enabled" and nothing is spawned. diff --git a/apps/rocm/src/services_text.rs b/apps/rocm/src/services_text.rs index 6bae507e7f..f1bee48f29 100644 --- a/apps/rocm/src/services_text.rs +++ b/apps/rocm/src/services_text.rs @@ -56,8 +56,8 @@ //! `process_check_command` are `pub(crate)` for `service_control.rs` (the last //! one only for its tests); `read_tail_lines`, `render_logs_text`, //! `render_logs_browser_text`, `render_services_text` and -//! `render_service_logs_text` are also `pub(crate)` and read from `chat.rs`'s -//! deterministic tool-output summaries; `validate_service_id` is also +//! `render_service_logs_text` are also `pub(crate)` and read from +//! `chat/mcp_bridge.rs`'s read-only command runner; `validate_service_id` is also //! `pub(crate)` and read from `tool_policy.rs`'s service-tool validation, //! `services_cmd.rs` and `service_records.rs`; `read_optional_tail_lines` and //! `render_service_action_result` are also `pub(crate)` and read from @@ -526,7 +526,7 @@ pub(crate) fn render_service_action_result_with_helper_note( ) -> String { let mut text = render_service_action_result(tool, value); if tool == "stop_server" - && let Some(note) = crate::service_launch::background_helper_note(paths) + && let Some(note) = crate::background_helper::background_helper_note(paths) { let _ = writeln!(text, " note: {}", note.text); } diff --git a/apps/rocm/src/setup_cmd.rs b/apps/rocm/src/setup_cmd.rs index 4fb933cb44..35a3c706a8 100644 --- a/apps/rocm/src/setup_cmd.rs +++ b/apps/rocm/src/setup_cmd.rs @@ -11,7 +11,7 @@ //! `docs/architecture.md`. `SetupCommand`, the clap enum `setup` matches on, //! lives in `cli.rs` with the rest of the `Command`/`Cli` definition. //! `dispatch()` calls `setup` through the re-import `use -//! crate::setup_cmd::{...};`, and `chat.rs`'s read-only handling of `rocm setup` +//! crate::setup_cmd::{...};`, and `chat/mcp_bridge.rs`'s read-only handling of `rocm setup` //! and `rocm setup status` reaches `render_setup_status_text` as //! `crate::render_setup_status_text` through the same re-import; both are //! `pub(crate)`. `reset_setup_prompt_state` has no caller outside this module diff --git a/apps/rocm/src/test_support.rs b/apps/rocm/src/test_support.rs index e4f0d406cb..1987de3679 100644 --- a/apps/rocm/src/test_support.rs +++ b/apps/rocm/src/test_support.rs @@ -9,7 +9,7 @@ use std::path::PathBuf; use std::sync::atomic::{AtomicU64, Ordering}; use anyhow::Result; -use rocm_core::AppPaths; +use rocm_core::{AppPaths, AutomationRuntimeState}; use crate::runtimes_cmd::{runtime_manifest_path, runtime_registry_dir}; use crate::therock; @@ -229,6 +229,191 @@ pub(crate) fn write_test_pip_runtime( Ok(manifest) } +/// A `rocm serve` argv the serve safety policy refuses, with the refusal each +/// surface reports for it. +/// +/// Two surfaces enforce the policy on a model-proposed `serve`: the chat tool +/// policy (`tool_policy::validate_chat_rocm_command_safety`) and the +/// provider-assisted planner (`freeform_planner::validate_provider_planner_tool_call`). +/// Both suites run this one table, so a rule missing from either surface fails +/// that surface's tests, and the same argv cannot be refused by one for a +/// different reason than the other without the table saying so. +pub(crate) struct ServeSafetyCase { + /// The rule the argv breaks. + pub(crate) rule: &'static str, + pub(crate) args: &'static [&'static str], + /// The full message the chat tool policy refuses it with. + pub(crate) chat_error: &'static str, + /// The full message the planner refuses it with. + pub(crate) planner_error: &'static str, +} + +/// Every refusal the serve safety policy has, each rule reached by more than one +/// spelling of its trigger. A case breaks exactly one rule, so the message names +/// the rule that fired and nothing else could have. +pub(crate) const SERVE_SAFETY_REFUSALS: &[ServeSafetyCase] = &[ + ServeSafetyCase { + rule: "public bind", + args: &[ + "serve", + "qwen", + "--engine", + "vllm", + "--allow-public-bind", + "--managed", + ], + chat_error: "local assistant cannot request public network binding", + planner_error: "provider planner cannot request public network binding", + }, + ServeSafetyCase { + rule: "public bind", + args: &["serve", "qwen", "--managed", "--allow-public-bind"], + chat_error: "local assistant cannot request public network binding", + planner_error: "provider planner cannot request public network binding", + }, + ServeSafetyCase { + rule: "cpu device", + args: &[ + "serve", + "qwen", + "--engine", + "vllm", + "--device", + "cpu", + "--managed", + ], + chat_error: "local assistant cannot request CPU execution; ROCm GPU execution is required", + planner_error: "provider planner cannot request CPU execution; ROCm GPU execution is required", + }, + ServeSafetyCase { + rule: "cpu device", + args: &["serve", "qwen", "--device=cpu", "--managed"], + chat_error: "local assistant cannot request CPU execution; ROCm GPU execution is required", + planner_error: "provider planner cannot request CPU execution; ROCm GPU execution is required", + }, + ServeSafetyCase { + rule: "cpu device", + args: &["serve", "qwen", "--device", "cpu_only", "--managed"], + chat_error: "local assistant cannot request CPU execution; ROCm GPU execution is required", + planner_error: "provider planner cannot request CPU execution; ROCm GPU execution is required", + }, + ServeSafetyCase { + rule: "non-loopback host", + args: &["serve", "qwen", "--host", "0.0.0.0", "--managed"], + chat_error: "local assistant cannot request non-local host `0.0.0.0`", + planner_error: "provider planner cannot request non-local host `0.0.0.0`", + }, + ServeSafetyCase { + rule: "non-loopback host", + args: &["serve", "qwen", "--host=example.test", "--managed"], + chat_error: "local assistant cannot request non-local host `example.test`", + planner_error: "provider planner cannot request non-local host `example.test`", + }, + ServeSafetyCase { + // Classified public on purpose: only the three exact loopback spellings + // pass (see `is_public_bind_host`). + rule: "non-loopback host", + args: &["serve", "qwen", "--host", "LOCALHOST", "--managed"], + chat_error: "local assistant cannot request non-local host `LOCALHOST`", + planner_error: "provider planner cannot request non-local host `LOCALHOST`", + }, + ServeSafetyCase { + rule: "managed required", + args: &["serve", "qwen", "--engine", "vllm"], + chat_error: "local assistant must request managed serving with --managed", + planner_error: "provider planner must request managed serving with --managed", + }, + ServeSafetyCase { + rule: "managed required", + args: &["serve", "qwen", "--engine", "vllm", "--foreground"], + chat_error: "local assistant must request managed serving with --managed", + planner_error: "provider planner must request managed serving with --managed", + }, +]; + +/// `--foreground` next to `--managed`: the safety check refuses the pair, but +/// clap does too (the flags conflict), so the planner never reaches its safety +/// check with it and only the chat check is what refuses it first. +pub(crate) const SERVE_FOREGROUND_AND_MANAGED: &[&str] = + &["serve", "qwen", "--managed", "--foreground"]; + +/// `rocm serve` argvs the serve safety policy lets through on both surfaces, and +/// that also parse as a `rocm` command, so the planner reaches its safety check +/// with them. A host other than `127.0.0.1` carries an explicit `--port`, which +/// the chat policy requires there (see [`SERVE_PORT_RULE_CHAT_ONLY`]). +pub(crate) const SERVE_SAFETY_ACCEPTED: &[&[&str]] = &[ + &["serve", "qwen", "--managed"], + &[ + "serve", + "qwen", + "--engine", + "vllm", + "--device", + "gpu_required", + "--managed", + ], + &["serve", "qwen", "--device=gpu_required", "--managed"], + &["serve", "qwen", "--host", "127.0.0.1", "--managed"], + &[ + "serve", + "qwen", + "--host=localhost", + "--port", + "8000", + "--managed", + ], + &["serve", "qwen", "--host", "::1", "--port=8000", "--managed"], +]; + +/// `rocm serve` argvs the planner lets through but the chat policy refuses: a +/// loopback host other than `127.0.0.1` without an explicit `--port`. The chat +/// policy applies `rocm serve`'s own port rule so that an approval card is never +/// offered for a command the CLI then refuses; the planner's check has no port +/// rule. Each entry is the argv and the full message the chat policy refuses it +/// with. +pub(crate) const SERVE_PORT_RULE_CHAT_ONLY: &[(&[&str], &str)] = &[ + ( + &["serve", "qwen", "--host=localhost", "--managed"], + "`rocm serve --host localhost` needs an explicit `--port <1-65535>`. Automatic port selection is supported only on the default host 127.0.0.1, where ROCm can prove a port is free before launching; on any other address the engine owns the bind.", + ), + ( + &["serve", "qwen", "--host", "::1", "--managed"], + "`rocm serve --host ::1` needs an explicit `--port <1-65535>`. Automatic port selection is supported only on the default host 127.0.0.1, where ROCm can prove a port is free before launching; on any other address the engine owns the bind.", + ), +]; + +/// `args` as the owned argv both surfaces take. +pub(crate) fn owned_argv(args: &[&str]) -> Vec { + args.iter().map(|arg| (*arg).to_owned()).collect() +} + +/// Build an `AutomationRuntimeState` for the background-helper tests (the +/// no-double-spawn guard in `background_helper.rs` and the autostart tests in +/// `service_launch.rs`). +pub(crate) fn runtime_state(running: bool, daemon_pid: u32) -> AutomationRuntimeState { + AutomationRuntimeState { + running, + automations_enabled: true, + daemon_pid, + started_at_unix_ms: 1, + last_tick_unix_ms: 1, + local_webhook_endpoint: None, + active_watchers: Vec::new(), + } +} + +/// Put the on-disk facts in one of the four combinations: whether a watcher +/// is enabled, and whether the helper is running (this process stands in for +/// a live daemon). +pub(crate) fn set_helper_facts(paths: &AppPaths, watcher_enabled: bool, helper_running: bool) { + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = watcher_enabled; + config.save(paths).expect("save config"); + runtime_state(helper_running, std::process::id()) + .write(paths) + .expect("write runtime state"); +} + /// Leave `paths` as `rocm setup` followed by a lost registry entry would: a ready /// wheel runtime whose install tree holds its own `.rocm-cli-runtime.json` and /// that `config.json` names as the setup folder, but that the runtime registry diff --git a/apps/rocm/src/tool_policy.rs b/apps/rocm/src/tool_policy.rs index 678dbdaec8..d4cae125fb 100644 --- a/apps/rocm/src/tool_policy.rs +++ b/apps/rocm/src/tool_policy.rs @@ -18,8 +18,8 @@ //! `rocm_chat_tool_requested_command` turn a tool call into the argv and the //! one-line command it stands for. //! -//! The module was relocated verbatim from `chat.rs`, which calls into it and -//! re-exports the items other modules reach as `chat::`. It calls +//! The module was relocated verbatim from `chat.rs`, whose child modules call +//! into it and which re-exports the items other modules reach as `chat::`. It calls //! nothing in `chat.rs` and nothing in `freeform_planner.rs`, which is what //! lets the planner call into it without importing `chat`. @@ -849,23 +849,44 @@ fn canonicalize_chat_rocm_command(args: &mut [String]) -> Result<()> { Ok(()) } +/// The rules a model-proposed `rocm serve` must satisfy, whoever proposes it. +/// +/// Four, checked in this order and reported first-failure-wins: no +/// `--allow-public-bind`, no CPU device, a loopback `--host` if one is given, +/// and `--managed` without `--foreground`. `args` is the whole argv, starting +/// at `serve`; the caller has already decided it is one. +/// +/// `actor` names who proposed the call and opens every refusal (`{actor} +/// cannot request ...`), so each surface keeps saying who was refused while the +/// rules, their order and the rest of the wording stay in this one place. The +/// chat policy passes `local assistant`; the provider-assisted planner passes +/// `provider planner`. Deleting a rule here removes it from both. +pub(crate) fn validate_serve_command_safety(args: &[String], actor: &str) -> Result<()> { + if chat_cli_has_flag(args, "--allow-public-bind") { + bail!("{actor} cannot request public network binding"); + } + if serve_args_request_cpu_device(args) { + bail!("{actor} cannot request CPU execution; ROCm GPU execution is required"); + } + if let Some(host) = chat_cli_arg_value(args, "--host") + && !is_loopback_host(host) + { + bail!("{actor} cannot request non-local host `{host}`"); + } + if chat_cli_has_flag(args, "--foreground") || !chat_cli_has_flag(args, "--managed") { + bail!("{actor} must request managed serving with --managed"); + } + Ok(()) +} + pub(crate) fn validate_chat_rocm_command_safety(args: &[String]) -> Result<()> { if let Some(first) = args.first() && first.eq_ignore_ascii_case("serve") { - if chat_cli_has_flag(args, "--allow-public-bind") { - bail!("local assistant cannot request public network binding"); - } - if serve_args_request_cpu_device(args) { - bail!("local assistant cannot request CPU execution; ROCm GPU execution is required"); - } - if let Some(host) = chat_cli_arg_value(args, "--host") - && !is_loopback_host(host) - { - bail!("local assistant cannot request non-local host `{host}`"); - } + validate_serve_command_safety(args, "local assistant")?; // The same rule `rocm serve` applies, so an approval card is never offered - // for a command the CLI then refuses. + // for a command the CLI then refuses. Only the chat policy applies it; the + // provider-assisted planner's check has no port rule. let port = chat_cli_arg_value(args, "--port") .map(crate::serve_port::parse_serve_port) .transpose() @@ -874,9 +895,6 @@ pub(crate) fn validate_chat_rocm_command_safety(args: &[String]) -> Result<()> { chat_cli_arg_value(args, "--host").unwrap_or(rocm_core::DEFAULT_LOCAL_HOST), crate::serve_port::PortRequest::from_flag(port), )?; - if chat_cli_has_flag(args, "--foreground") || !chat_cli_has_flag(args, "--managed") { - bail!("local assistant must request managed serving with --managed"); - } } if args .first() @@ -2629,4 +2647,188 @@ mod tests { "accepted as a user install folder: {accepted:?}" ); } + + /// The serve safety rules as the chat assistant meets them: every refusal in + /// the shared table, through the safety check itself and through the two + /// entry points a model's `rocm_command` call goes by. A refused call yields + /// no `ChatRocmCommandAction`, so no approval card is built and nothing runs. + /// + /// The planner suite (`freeform_planner`) runs the same table. + #[test] + fn chat_refuses_each_unsafe_serve_and_proposes_nothing() { + use crate::test_support::{SERVE_SAFETY_REFUSALS, owned_argv}; + + for case in SERVE_SAFETY_REFUSALS { + let context = format!("{}: {:?}", case.rule, case.args); + let error = validate_chat_rocm_command_safety(&owned_argv(case.args)) + .expect_err(&context) + .to_string(); + assert_eq!(error, case.chat_error, "{context}"); + + let call = rocm_command_call(case.args); + let error = chat_rocm_command_action(&call) + .expect_err(&context) + .to_string(); + assert_eq!(error, case.chat_error, "{context}"); + let error = validate_chat_tool_call(&call) + .expect_err(&context) + .to_string(); + assert_eq!(error, case.chat_error, "{context}"); + } + + // `--foreground` refuses even next to `--managed`. The chat check is the + // only one that sees this pair; the planner's clap parse refuses it first. + let error = validate_chat_rocm_command_safety(&owned_argv( + crate::test_support::SERVE_FOREGROUND_AND_MANAGED, + )) + .expect_err("foreground serving is refused even with --managed") + .to_string(); + assert_eq!( + error, + "local assistant must request managed serving with --managed" + ); + } + + #[test] + fn chat_proposes_a_safe_serve_for_approval_with_its_argv_unchanged() { + use crate::test_support::{SERVE_SAFETY_ACCEPTED, owned_argv}; + + for args in SERVE_SAFETY_ACCEPTED { + let argv = owned_argv(args); + validate_chat_rocm_command_safety(&argv) + .unwrap_or_else(|error| panic!("{args:?} must pass the safety check: {error}")); + let action = chat_rocm_command_action(&rocm_command_call(args)) + .unwrap_or_else(|error| panic!("{args:?} must be proposed: {error}")); + // A serve starts a server, so it is never read-only. + assert_eq!( + action, + ChatRocmCommandAction::Approval { + args: argv, + pending_title: "Start local model server".to_owned(), + command_title: "Serve".to_owned(), + }, + "{args:?}" + ); + } + } + + /// The chat policy also applies `rocm serve`'s port rule, which the planner's + /// check does not: a loopback host other than `127.0.0.1` needs an explicit + /// `--port`. Each argv is refused with the exact message and proposes nothing, + /// and the same argv with a port is accepted (the shared accepted table). + /// `planner_accepts_a_loopback_host_without_a_port` holds the planner side. + #[test] + fn chat_requires_a_port_on_a_loopback_host_other_than_the_default() { + use crate::test_support::{SERVE_PORT_RULE_CHAT_ONLY, owned_argv}; + + for (args, expected) in SERVE_PORT_RULE_CHAT_ONLY { + let error = validate_chat_rocm_command_safety(&owned_argv(args)) + .expect_err("a non-default host needs a port") + .to_string(); + assert_eq!(error, *expected, "{args:?}"); + let call = rocm_command_call(args); + let error = chat_rocm_command_action(&call) + .expect_err("no approval card for a command the CLI refuses") + .to_string(); + assert_eq!(error, *expected, "{args:?}"); + let error = validate_chat_tool_call(&call) + .expect_err("the tool-call entry refuses it too") + .to_string(); + assert_eq!(error, *expected, "{args:?}"); + } + } + + /// The rules fire in a fixed order, so an argv that breaks several reports + /// the first one: public bind, then CPU, then host, then managed. + #[test] + fn chat_reports_the_first_broken_serve_rule() { + use crate::test_support::owned_argv; + + for (args, expected) in [ + ( + &[ + "serve", + "qwen", + "--allow-public-bind", + "--device", + "cpu", + "--host", + "0.0.0.0", + "--foreground", + ][..], + "local assistant cannot request public network binding", + ), + ( + &[ + "serve", + "qwen", + "--device", + "cpu", + "--host", + "0.0.0.0", + "--foreground", + ][..], + "local assistant cannot request CPU execution; ROCm GPU execution is required", + ), + ( + &["serve", "qwen", "--host", "0.0.0.0", "--foreground"][..], + "local assistant cannot request non-local host `0.0.0.0`", + ), + ( + &["serve", "qwen", "--foreground"][..], + "local assistant must request managed serving with --managed", + ), + ] { + let error = validate_chat_rocm_command_safety(&owned_argv(args)) + .expect_err("unsafe") + .to_string(); + assert_eq!(error, expected, "{args:?}"); + } + } + + /// The shared serve check opens every refusal with the actor its caller + /// names, so each surface keeps saying who it refused. + #[test] + fn serve_safety_refusals_open_with_the_actor_the_caller_names() { + use crate::test_support::{SERVE_SAFETY_REFUSALS, owned_argv}; + + for case in SERVE_SAFETY_REFUSALS { + let error = validate_serve_command_safety(&owned_argv(case.args), "some caller") + .expect_err(case.rule) + .to_string(); + assert_eq!( + error, + case.chat_error + .replacen("local assistant", "some caller", 1), + "{}: {:?}", + case.rule, + case.args + ); + } + } + + /// The chat check matches the `serve` word without regard to case, so a + /// model that writes `SERVE` does not step around the rules. + #[test] + fn chat_enforces_the_serve_rules_whatever_the_case_of_the_serve_word() { + use crate::test_support::owned_argv; + + for word in ["Serve", "SERVE"] { + let error = validate_chat_rocm_command_safety(&owned_argv(&[ + word, + "qwen", + "--device", + "cpu", + "--managed", + ])) + .expect_err("a CPU serve is refused in any case") + .to_string(); + assert_eq!( + error, + "local assistant cannot request CPU execution; ROCm GPU execution is required" + ); + validate_chat_rocm_command_safety(&owned_argv(&[word, "qwen", "--managed"])) + .expect("a managed serve passes in any case"); + } + } } diff --git a/apps/rocm/src/uninstall_plan.rs b/apps/rocm/src/uninstall_plan.rs index e450fde189..ea210c2eae 100644 --- a/apps/rocm/src/uninstall_plan.rs +++ b/apps/rocm/src/uninstall_plan.rs @@ -33,7 +33,7 @@ //! `render_uninstall_plan`; `storage.rs` calls into it for //! `UninstallPlan`/`UninstallPlanEntry`/`confirm_uninstall`; `runtimes_cmd.rs` //! calls into it for `confirm_uninstall` (the same `[y/N]` prompt, reused for -//! `rocm runtimes uninstall`); `chat.rs` calls into it for +//! `rocm runtimes uninstall`); `chat/mcp_bridge.rs` calls into it for //! `render_uninstall_dry_run` (the deterministic uninstall-preview tool //! result); `deletion_properties.rs`'s recursive-deletion property tests call //! into it for `UninstallOptions`/`build_uninstall_plan`/`remove_path`/ @@ -137,7 +137,7 @@ pub(crate) fn build_uninstall_plan( // Only a helper that is running can be left behind. "Uninstall does not stop // it" is said only where that is the whole story: with a watcher enabled the // helper stays up; an idle one is already on its way out. - if let Some(note) = crate::service_launch::background_helper_note(paths) + if let Some(note) = crate::background_helper::background_helper_note(paths) && note.running { plan.warnings.push(if note.outlives { diff --git a/apps/rocm/src/update_cmd.rs b/apps/rocm/src/update_cmd.rs index 49cf65942c..1b64a253c8 100644 --- a/apps/rocm/src/update_cmd.rs +++ b/apps/rocm/src/update_cmd.rs @@ -17,8 +17,8 @@ //! body was that arm's own and which was relocated here verbatim, re-imported //! via `use crate::update_cmd::update;`; `update` in turn calls //! `update_should_preview_or_apply(apply, dry_run)`, -//! `apply_runtime_update(...)`, and `render_update_text(&paths)`. `chat.rs`'s -//! deterministic `rocm update` tool summary also calls `render_update_text`, +//! `apply_runtime_update(...)`, and `render_update_text(&paths)`. `chat/mcp_bridge.rs`'s +//! read-only `rocm update` handling also calls `render_update_text`, //! outside `dispatch()`. //! //! One item physically interleaved in this span in `main.rs` stayed there @@ -31,7 +31,7 @@ //! from here the same way via `use crate::ROLLBACK_RECOVERY_HINT;`. //! //! `update` is `pub(crate)` for `dispatch()` and `render_update_text` for -//! `chat.rs`; `apply_runtime_update` and `update_should_preview_or_apply` keep +//! `chat/mcp_bridge.rs`; `apply_runtime_update` and `update_should_preview_or_apply` keep //! the `pub(crate)` they had when `dispatch()` called them directly, though //! only `update` calls them now. `select_runtime_update_source` is also //! `pub(crate)`: it is diff --git a/docs/architecture.md b/docs/architecture.md index 07209fb95a..487fe32658 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,8 +16,8 @@ A second check runs the other way: `cargo xtask check-architecture-doc` ([`archi New subcommands and subsystems default to their own file from day one — they should not grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. Two extraction patterns already exist in the codebase; use whichever fits: -- **Full domain extraction** — a subsystem's domain implementation moves into its own file that owns its own types (structs/enums), not just relocated functions; that ownership is usually what distinguishes this pattern from mechanical relocation below, though the Phase 6a variant described next shows ownership alone doesn't fully tell them apart — every `apps/rocm` module is a private `mod x;` regardless of pattern, so module privacy alone doesn't tell the two apart either. Full-domain-extraction modules are typically reached via qualified paths (e.g. `comfyui::render_status(...)`); mechanically relocated modules with no owned types are reached the same way (e.g. [`endpoint_keys.rs`](../apps/rocm/src/endpoint_keys.rs), [`logging.rs`](../apps/rocm/src/logging.rs)). The exception — reached unqualified through `use crate::x::{...}` re-imports, matching their dispatch call sites' pre-extraction form — is limited to the clusters that moved as a dispatch-adjacent group; the block of `use crate::x::{...}` re-imports near the top of [`main.rs`](../apps/rocm/src/main.rs) is the current list of the modules it reaches that way (a few of them re-imported for reasons other than dispatch, as the comment above the block explains). [`service_launch.rs`](../apps/rocm/src/service_launch.rs) (as of Phase 6b, ROCMAI-91) uses the same unqualified re-import form for its cross-module callers, but it is not a dispatch-adjacent cluster in the same sense — one of its callers, `automations.rs`, is not a dispatch call site, and its pre-extraction form was a qualified `crate::` call rather than an unqualified import, so the re-import is new rather than kept. This call-site form, not type ownership or dispatch-fn placement, is what actually separates the Phase 6a variant below from full domain extraction. Where a subsystem has a dedicated clap subcommand, its command *enum* usually lives in [`cli.rs`](../apps/rocm/src/cli.rs) as part of the `Command`/`Cli` definition (e.g. `ComfyuiCommand`, `RuntimesCommand`), the way [`apps/rocmd`](../apps/rocmd/src/cli.rs) keeps its command tree in a `cli.rs` of its own — but the dispatch function, the match over that tree, does not always stay in `main.rs` with it: `comfyui()` does, while `engines()` (Phase 6a) and `runtimes()` (Phase 6c, ROCMAI-89) moved out to `engines_cmd.rs`/`runtimes_cmd.rs` respectively, following the mechanical-relocation re-import form below rather than full domain extraction's qualified-path form. Not every domain-extracted module has a dedicated command enum at all (`providers.rs` has none; it's invoked from the existing chat/config command flows). In library crates (`crates/rocm-core`) the module is `pub mod x;` plus a `pub use x::{...};` re-export, since it's part of the crate's public API. This is the default for new subsystems. Examples: [`apps/rocm/src/therock.rs`](../apps/rocm/src/therock.rs) (TheRock SDK install orchestration, with its update planning and other clusters in `therock/` child modules and no command enum of its own), [`comfyui.rs`](../apps/rocm/src/comfyui.rs) (`ComfyuiCommand`), [`providers.rs`](../apps/rocm/src/providers.rs) (no dedicated command enum); `crates/rocm-core`'s [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs)/[`examine.rs`](../crates/rocm-core/src/examine.rs). -- **Mechanical relocation** — a `pub(crate) fn` moves out verbatim, with shared types/config staying at the crate root and reached via `crate::`. Used for dispatch-adjacent clusters where a minimal, easy-to-review diff matters more than full extraction. Example: [`apps/rocm/src/update_cmd.rs`](../apps/rocm/src/update_cmd.rs). Variant: when the cluster is large enough to own types (part of `main.rs`'s Phase 6a split, ROCMAI-82, `main.rs`'s Phase 6b split, ROCMAI-91, `main.rs`'s Phase 6c split, ROCMAI-89, and Phase 6e's follow-up, ROCMAI-726), those types move with it instead of staying in `main.rs` — [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s types are used nowhere else, but [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s `DriverInstallResult`/`DriverInstallError` are `pub(crate)` and read from [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs)'s `install()` (`result.output`/`result.executed`, `error.source`/`error.executed`); [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs)'s `FreeformExecution` is `pub(crate)` and read from `main.rs`'s `execute_freeform_next_action()` (`execution.action`) — it was already `pub(crate)` before the split, unlike the two functions that return/consume it, which widened from module-private; [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `ServeEngineSelection`/`ServeArgs` both stay `pub(crate)` and are used from `main.rs` (`ServeArgs`) and [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`assess_model_for_host` calls `select_serve_engine` and reads the resulting `ServeEngineSelection`'s `.engine`/`.source` fields) — and `ServeGenerationDefaults` moved out with the engine-recipe overrides to [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs), where it and its fields are `pub(crate)` because `serve()` builds it; [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are private, and its `ManagedLaunchReport` is `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) for the interactive summary; [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimeActivationResult` and `ActiveRuntimeMarker` are `pub(crate)` and read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary` and from [`storage.rs`](../apps/rocm/src/storage.rs) respectively; its sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`. The pattern is still "mechanical relocation" rather than full domain extraction because call sites keep their pre-extraction unqualified-re-import form — the exception named above — rather than switching to the qualified paths full-domain-extraction modules use. The top-level dispatch fn (`dispatch()`) staying in `main.rs` is not what makes it so: full domain extraction usually keeps its dispatch fn in `main.rs` too (see `comfyui()` above). +- **Full domain extraction** — a subsystem's domain implementation moves into its own file that owns its own types (structs/enums), not just relocated functions; that ownership is usually what distinguishes this pattern from mechanical relocation below, though the Phase 6a variant described next shows ownership alone doesn't fully tell them apart — every `apps/rocm` module is a private `mod x;` regardless of pattern, so module privacy alone doesn't tell the two apart either. Full-domain-extraction modules are typically reached via qualified paths (e.g. `comfyui::render_status(...)`); mechanically relocated modules with no owned types are reached the same way (e.g. [`endpoint_keys.rs`](../apps/rocm/src/endpoint_keys.rs), [`logging.rs`](../apps/rocm/src/logging.rs)). The exception — reached unqualified through `use crate::x::{...}` re-imports, matching their dispatch call sites' pre-extraction form — is limited to the clusters that moved as a dispatch-adjacent group; the block of `use crate::x::{...}` re-imports near the top of [`main.rs`](../apps/rocm/src/main.rs) is the current list of the modules it reaches that way (a few of them re-imported for reasons other than dispatch, as the comment above the block explains). [`service_launch.rs`](../apps/rocm/src/service_launch.rs) (as of Phase 6b, ROCMAI-91) uses the same unqualified re-import form for its cross-module callers, but it is not a dispatch-adjacent cluster in the same sense — one of its callers, `automations.rs`, is not a dispatch call site, and its pre-extraction form was a qualified `crate::` call rather than an unqualified import, so the re-import is new rather than kept. This call-site form, not type ownership or dispatch-fn placement, is what actually separates the Phase 6a variant below from full domain extraction. Where a subsystem has a dedicated clap subcommand, its command *enum* usually lives in [`cli.rs`](../apps/rocm/src/cli.rs) as part of the `Command`/`Cli` definition (e.g. `ComfyuiCommand`, `RuntimesCommand`), the way [`apps/rocmd`](../apps/rocmd/src/cli.rs) keeps its command tree in a `cli.rs` of its own — but the dispatch function, the match over that tree, does not stay in `main.rs` with it: `comfyui::run` lives in `comfyui.rs`, reached by the qualified path full domain extraction uses, while `engines()` and `runtimes()` moved out to `engines_cmd.rs`/`runtimes_cmd.rs` respectively, following the mechanical-relocation re-import form below. Only the top-level `dispatch()`, the match that sends each `Command` variant to its handler, stays in `main.rs`. Not every domain-extracted module has a dedicated command enum at all (`providers.rs` has none; it's invoked from the existing chat/config command flows). In library crates (`crates/rocm-core`) the module is `pub mod x;` plus a `pub use x::{...};` re-export, since it's part of the crate's public API. This is the default for new subsystems. Examples: [`apps/rocm/src/therock.rs`](../apps/rocm/src/therock.rs) (TheRock SDK install orchestration, with its update planning and other clusters in `therock/` child modules and no command enum of its own), [`comfyui.rs`](../apps/rocm/src/comfyui.rs) (`ComfyuiCommand`), [`providers.rs`](../apps/rocm/src/providers.rs) (no dedicated command enum); `crates/rocm-core`'s [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs)/[`examine.rs`](../crates/rocm-core/src/examine.rs). +- **Mechanical relocation** — a `pub(crate) fn` moves out verbatim, with shared types/config staying at the crate root and reached via `crate::`. Used for dispatch-adjacent clusters where a minimal, easy-to-review diff matters more than full extraction. Example: [`apps/rocm/src/update_cmd.rs`](../apps/rocm/src/update_cmd.rs). Variant: when the cluster is large enough to own types (part of `main.rs`'s Phase 6a split, ROCMAI-82, `main.rs`'s Phase 6b split, ROCMAI-91, `main.rs`'s Phase 6c split, ROCMAI-89, and Phase 6e's follow-up, ROCMAI-726), those types move with it instead of staying in `main.rs` — [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s types are used nowhere else, but [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s `DriverInstallResult`/`DriverInstallError` are `pub(crate)` and read from [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs)'s `install()` (`result.output`/`result.executed`, `error.source`/`error.executed`); [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs)'s `FreeformExecution` is `pub(crate)` and read from `main.rs`'s `execute_freeform_next_action()` (`execution.action`) — it was already `pub(crate)` before the split, unlike the two functions that return/consume it, which widened from module-private; [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `ServeEngineSelection`/`ServeArgs` both stay `pub(crate)` and are used from `main.rs` (`ServeArgs`) and [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`assess_model_for_host` calls `select_serve_engine` and reads the resulting `ServeEngineSelection`'s `.engine`/`.source` fields) — and `ServeGenerationDefaults` moved out with the engine-recipe overrides to [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs), where it and its fields are `pub(crate)` because `serve()` builds it; [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are private, and its `ManagedLaunchReport` is `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) for the interactive summary; [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimeActivationResult` and `ActiveRuntimeMarker` are `pub(crate)` and read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary` and from [`storage.rs`](../apps/rocm/src/storage.rs) respectively; its sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`. The pattern is still "mechanical relocation" rather than full domain extraction because call sites keep their pre-extraction unqualified-re-import form — the exception named above — rather than switching to the qualified paths full-domain-extraction modules use. Where a subcommand's dispatch fn lives is not what makes it so: a full-domain-extraction module can host its own dispatch fn too (see `comfyui::run` above). A flat size limit backs this convention: `cargo xtask check-module-size` ([`module_size.rs`](../xtask/src/module_size.rs)) reports every source file with more than 1,500 production lines, and there is one limit for every file, with no per-file ceilings, baseline or exemption. Production lines exclude test-gated items (inline `#[cfg(test)]` modules and the like), and test-only files are not source files; the exact definition is at the top of that file. Today the check is advisory: the `clippy` CI job and the `cargo-module-size` pre-commit hook print the offenders without failing, until the files over the limit have been split. A PR that would push a file past 1,500 production lines splits it in the same change (a pure move, the tests moving with the code, this document updated in the same PR); raising the limit or exempting a file is not an option for a feature PR. A change that merely edits a file that is already over the limit does not by itself have to split it: those files are split under their own tickets, and the check becomes blocking only once none is left. `too_many_lines = "allow"` in the workspace `Cargo.toml` stays a deliberate function-level choice; this gate is about files. @@ -29,20 +29,25 @@ Each per-crate module inventory lists one file per line. Flat inventories (every ### `apps/rocm` — main CLI binary -[`main.rs`](../apps/rocm/src/main.rs) is the crate root. Besides smaller helpers, it holds `main()`, `run()` and `dispatch()`, the match over the clap `Cli`/`Command` tree that [`cli.rs`](../apps/rocm/src/cli.rs) defines; the builders that turn that tree into a `clap::Command` (`cli_command()`, `completion_command()`); the command handlers still defined here (`comfyui()`, `version()` and `launch_default()`, the no-subcommand arm); freeform-invocation routing (`parse_freeform_invocation`, `should_treat_as_freeform`, `run_freeform`, `execute_freeform_next_action`, `treat_as_natural_language`); a few crate-root items several modules share (`status_for_readiness`, `device_policy_name`, `engine_inventory`, `ROLLBACK_RECOVERY_HINT`); the `use crate::x::{...}` re-import block described above, which also re-imports the `cli.rs` types so other modules keep reaching them as `crate::Command`, `crate::Provider` and so on; and the crate's shared `mod tests` block. +[`main.rs`](../apps/rocm/src/main.rs) is the crate root. Besides smaller helpers, it holds `main()`, `run()` and `dispatch()`, the match over the clap `Cli`/`Command` tree that [`cli.rs`](../apps/rocm/src/cli.rs) defines; the builders that turn that tree into a `clap::Command` (`cli_command()`, `completion_command()`); the command handlers still defined here (`version()` and `launch_default()`, the no-subcommand arm); freeform-invocation routing (`parse_freeform_invocation`, `should_treat_as_freeform`, `run_freeform`, `execute_freeform_next_action`, `treat_as_natural_language`); a few crate-root items several modules share (`status_for_readiness`, `device_policy_name`, `engine_inventory`, `ROLLBACK_RECOVERY_HINT`); the `use crate::x::{...}` re-import block described above, which also re-imports the `cli.rs` types so other modules keep reaching them as `crate::Command`, `crate::Provider` and so on; and the crate's shared `mod tests` block. -- [`automations.rs`](../apps/rocm/src/automations.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-210): the `rocm automations` dispatch handler (`automations()`) plus the `rocm automations` status report it prints (`render_automations_text`), the `ProposalKind` classification that report renders through (`proposal_kind`, `fallback_tool_for_automation_action`, `proposal_plain_summary`, `proposal_plain_reason`), the small plain-language label helpers (`proposal_status_label`, `proposal_bool_argument`, `automation_event_plain_summary`, `audit_event_plain_summary`, `watcher_mode_plain_label`, `watcher_last_check`, `watcher_plain_name`, `watcher_plain_trigger`, `watcher_plain_action`, `watcher_policy_note`), and the unrelated `rocm daemon --status` summary (`render_daemon_text`). `render_automations_text` and `render_daemon_text` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `render_daemon_text` directly (re-imported alongside `automations` via `use crate::automations::{automations, render_daemon_text};`; `render_automations_text` has no `dispatch()` call site of its own, reached only through `automations()` above, now local to this file), and [`chat.rs`](../apps/rocm/src/chat.rs)'s deterministic tool summaries call both. Every other item in this list, including `watcher_policy_note` (called from `automations()`'s `Enable` arm and from `render_automations_text`, both in this same file), has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do. +- [`automations.rs`](../apps/rocm/src/automations.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-210): the `rocm automations` dispatch handler (`automations()`) plus the `rocm automations` status report it prints (`render_automations_text`), the `ProposalKind` classification that report renders through (`proposal_kind`, `fallback_tool_for_automation_action`, `proposal_plain_summary`, `proposal_plain_reason`), the small plain-language label helpers (`proposal_status_label`, `proposal_bool_argument`, `automation_event_plain_summary`, `audit_event_plain_summary`, `watcher_mode_plain_label`, `watcher_last_check`, `watcher_plain_name`, `watcher_plain_trigger`, `watcher_plain_action`, `watcher_policy_note`), and the unrelated `rocm daemon --status` summary (`render_daemon_text`). `render_automations_text` and `render_daemon_text` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `render_daemon_text` directly (re-imported alongside `automations` via `use crate::automations::{automations, render_daemon_text};`; `render_automations_text` has no `dispatch()` call site of its own, reached only through `automations()` above, now local to this file), and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner calls both. Every other item in this list, including `watcher_policy_note` (called from `automations()`'s `Enable` arm and from `render_automations_text`, both in this same file), has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do. +- [`background_helper.rs`](../apps/rocm/src/background_helper.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), owns one `pub(crate)` type: what the CLI can tell about the background automation helper (`rocm daemon`). It holds `background_helper_already_running` (the file-based liveness check behind the no-double-spawn decision), the `HelperNote` type and `background_helper_note` with its `helper_note_for` wording, which [`services_text.rs`](../apps/rocm/src/services_text.rs) prints after a stop and [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) turns into an uninstall warning. Spawning the helper (`ensure_background_helper_running` and its autostart lock) stays in [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which reads `background_helper_already_running` from here. The module is a leaf: it imports nothing from the service modules, so the note can be read from [`services_text.rs`](../apps/rocm/src/services_text.rs) without that module importing the one that spawns services. The cluster's regression tests live in this module's own `mod tests`, which stays private: the `runtime_state` and `set_helper_facts` fixtures they share with the helper-autostart and stop-output tests in [`service_launch.rs`](../apps/rocm/src/service_launch.rs) live in the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs). - [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs) — a further-extracted variant of full domain extraction: it owns its clap command enum (`BootstrapCommand`) and dispatch function too, rather than leaving the enum in [`cli.rs`](../apps/rocm/src/cli.rs) and the dispatch in [`main.rs`](../apps/rocm/src/main.rs). -- [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) — mechanical relocation, no owned types: the host snapshot behind the internal bridge surface (`build_codex_bridge_snapshot`, `build_codex_bridge_gpu_snapshot`, `builtin_codex_bridge_engine_inventory`), whose engine list is derived from `main.rs`'s `engine_inventory`, the single table of built-in engine ids that `rocm examine`, `rocm update` and the `--engine` help test read as well. All three are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `build_codex_bridge_snapshot` through a re-import, [`chat.rs`](../apps/rocm/src/chat.rs)'s internal MCP tools reach them as `crate::` through the same re-import. It has no tests of its own. -- [`chat.rs`](../apps/rocm/src/chat.rs) — full domain extraction (Phase 6d, ROCMAI-88), mirrors [`providers.rs`](../apps/rocm/src/providers.rs). Owns the chat/MCP-bridge cluster's types (`DeterministicModelRecipe`, `DeterministicServiceRow`, `DeterministicEngineRow`, and others — all module-private), its execution logic, deterministic tool-output summarization (including `render_services_tool_result_text`), and the MCP tool-bridge entry points (`run_internal_mcp_call` and its helpers). The tool-call validation and approval policy (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `ChatRocmCommandAction`, `ChatToolApprovalRequest`) lives in [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs), which this module calls into and re-exports from. Callers in other modules, production and test-only, reach this module's items, and the re-exported ones, through the `chat::` module boundary. As a child module of the crate root (`mod chat;` in `main.rs`), Rust's module-privacy rule already makes most of `main.rs`'s private helpers visible to `chat.rs` without any visibility change, when `chat.rs` calls into `main.rs`. The other direction needs real widening: `chat.rs`'s own functions reached from other modules — `run_internal_mcp_call` (a production call from `main.rs`), `deterministic_comfyui_tool_summary` (reached only by `comfyui.rs`'s own test module), among others — stay `pub(crate)`; the third PR's visibility audit tightened every other function to module-private, so each function still `pub(crate)` has a confirmed external caller, production or test-only. The cluster's regression tests live in `chat.rs`'s own `mod tests`, moved out of `main.rs`'s shared `mod tests` block across the second and third PRs; `serve_negative_sampling_space_form_reaches_range_validator`, a shared clap value-parser wiring test living alongside its sibling command-parsing tests, has since moved with them to the test module in [`cli.rs`](../apps/rocm/src/cli.rs); the four structural checks that read [`chat_cmd.rs`](../apps/rocm/src/chat_cmd.rs)'s source for the `Command::Chat` handler's wiring (`command_chat_interactive_routes_to_dash_run_chat_not_tui`, `command_chat_interactive_notes_dropped_provider_flag`, `command_chat_honors_chat_mock_and_keeps_prompt_passthrough`, `command_chat_reads_prompt_from_piped_stdin`) live in that module's own `mod tests`. +- [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) — mechanical relocation, no owned types: the host snapshot behind the internal bridge surface (`build_codex_bridge_snapshot`, `build_codex_bridge_gpu_snapshot`, `builtin_codex_bridge_engine_inventory`), whose engine list is derived from `main.rs`'s `engine_inventory`, the single table of built-in engine ids that `rocm examine`, `rocm update` and the `--engine` help test read as well. All three are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `build_codex_bridge_snapshot` through a re-import, [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s internal MCP tools reach them as `crate::` through the same re-import. It has no tests of its own. +- [`chat.rs`](../apps/rocm/src/chat.rs) — the chat assistant's entry points and the module that ties its pieces together, a file beside a `chat/` directory of child modules (as [`providers.rs`](../apps/rocm/src/providers.rs) is beside `providers/`). It keeps `read_piped_prompt` and `piped_prompt_from_input` (the `echo "…" | rocm chat` path), `render_chat_text` (the status screen), `render_chat_prompt_text` (the one-shot reply text, which calls into `chat/turn.rs`), `plain_provider_auth_status` and the clap value parsers for the sampling flags (`parse_non_negative_temperature`, `parse_unit_interval`, `parse_positive_u32`), plus the re-exports below. The work behind them lives in four leaf children: [`chat/turn.rs`](../apps/rocm/src/chat/turn.rs) (the turn loop, the system prompt and the transcript), [`chat/intent.rs`](../apps/rocm/src/chat/intent.rs) (the prompt-intent heuristics), [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the MCP tool bridge and in-process command execution) and [`chat/summaries.rs`](../apps/rocm/src/chat/summaries.rs) (the deterministic tool-output summaries); `turn` is the top of that call graph and calls the other three, none of which calls back. The tool-call validation and approval policy (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `ChatRocmCommandAction`, `ChatToolApprovalRequest`) lives in [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs), which the children call into. Callers in other modules, production and test-only, reach the moved items through the `chat::` module boundary, so `chat.rs` re-exports each one another module reads: `run_internal_mcp_call` (`main.rs`, `dash_seam.rs`), `rocm_chat_tool_system_prompt_for_host` (`dash.rs`), `chat_tool_approval_request`, `chat_tool_call_is_read_only` and `validate_chat_tool_call` (`dash_seam.rs`), `any_substring` and `ensure_flag` (from `prompt_text.rs` and `rocm_argv.rs`, for `freeform_answer.rs` and `freeform_execution.rs`), and, only under `#[cfg(test)]`, `deterministic_comfyui_tool_summary` (`comfyui.rs`'s tests, unix only), `ChatRocmCommandAction`, `rocm_chat_tool_requested_args`, `chat_install_driver_action` and `chat_rocm_command_action_from_args` (`driver_plan.rs`'s and `system_packages.rs`'s tests). Each child's tests live in its own `mod tests`, with their original names; the two piped-prompt tests and the command-preview property tests (which drive the tool-call guards, the planner's guards and the rendered command lines across the policy, the bridge and the turn loop) stay in `chat.rs`. `serve_negative_sampling_space_form_reaches_range_validator`, a shared clap value-parser wiring test living alongside its sibling command-parsing tests, has moved with them to the test module in [`cli.rs`](../apps/rocm/src/cli.rs); the four structural checks that read [`chat_cmd.rs`](../apps/rocm/src/chat_cmd.rs)'s source for the `Command::Chat` handler's wiring (`command_chat_interactive_routes_to_dash_run_chat_not_tui`, `command_chat_interactive_notes_dropped_provider_flag`, `command_chat_honors_chat_mock_and_keeps_prompt_passthrough`, `command_chat_reads_prompt_from_piped_stdin`) live in that module's own `mod tests`. +- [`chat/turn.rs`](../apps/rocm/src/chat/turn.rs) — mechanical relocation out of `chat.rs`, a child of `chat.rs` and the top of its call graph: the chat turn loop and transcript. `render_chat_prompt_result_with_progress` is the entry point: it picks the assistant model, answers an install request that names no folder with the folder-needed text, builds the system prompt, asks the provider (or takes a tool call the intent heuristics propose when no model server is ready), runs the read-only tool calls the reply requests through the bridge, summarizes their output (through `chat/summaries.rs`, or by asking the local model for a follow-up) and renders the transcript, returning a `ChatPromptResult` (the rendered text and the approval request, if the assistant asked for a change). It owns `ChatPromptResult` and `ChatToolRunResult`, the system prompt (`ROCM_CHAT_TOOL_SYSTEM_PROMPT`, which `rocm_chat_tool_system_prompt` joins to the compiled-in assistant skill at `skills/rocm-cli-assistant/SKILL.md`, and `rocm_chat_tool_system_prompt_for_host`, which grounds it in the machine it answers for), `append_chat_tool_results` and the helpers around it (the `local_*` decisions about intermediate and final replies and the missing-service text, `deterministic_mutating_tool_call_for_prompt` and `supplemental_read_only_tool_call_for_prompt`, the tool-call display labels, `mcp_tool_result_is_error`, `visible_chat_content` which strips reasoning blocks, and `render_install_sdk_folder_needed_chat_text`). `render_chat_prompt_result_with_progress` and `ChatPromptResult` are `pub(super)` (called from `chat.rs`), `rocm_chat_tool_system_prompt_for_host` is `pub(crate)` and re-exported from `chat.rs` for `dash.rs`, and the rest is private. Its edges out are `chat/intent.rs`, `chat/mcp_bridge.rs`, `chat/summaries.rs`, `tool_policy.rs`, `rocm_argv.rs`, `freeform_planner.rs` (`format_structured_tool_call`), `engine_runtime_select.rs`, `managed_service_status.rs`, `chat_host_facts.rs` and `providers`; nothing below it calls back. Most of the former `chat.rs` tests moved with it into its own `mod tests`, with their original names. +- [`chat/summaries.rs`](../apps/rocm/src/chat/summaries.rs) — full domain extraction, a leaf child of `chat.rs`: the deterministic summaries of the read-only tool output the assistant reads back. `deterministic_chat_tool_summary` is the entry point the turn loop calls; it tries `deterministic_rocm_tool_summary` (`examine`), `deterministic_model_tool_summary` (model recipes, with `deterministic_engine_fit_summary` and `deterministic_available_engine_names`) and `deterministic_combined_status_tool_summary` (ComfyUI, port status, engine inventory and services, via the `deterministic_*_tool_summary` functions) in that order, and `deterministic_summary_can_stand_alone` says whether a summary may replace the model's own follow-up. The row types and parsers behind them (`DeterministicModelRecipe`, `DeterministicServiceRow`, `DeterministicEngineRow`, the `parse_deterministic_*` functions and their line helpers), `chat_tool_value` and `managed_service_running_state`, the status-to-running-state mapping that the services tool text in `chat/mcp_bridge.rs` also uses, live here too. `deterministic_chat_tool_summary`, `deterministic_summary_can_stand_alone` and `managed_service_running_state` are `pub(super)`, called from `chat/turn.rs` (and `chat/mcp_bridge.rs` for the last); `deterministic_rocm_tool_summary` is `pub(super)` for `chat/turn.rs`'s tests, and `deterministic_comfyui_tool_summary` stays `pub(crate)` for `comfyui.rs`'s tests, reached through `chat.rs`; the rest is private. Its only edges out are `providers` (the assistant model ids), `managed_service_status::friendly_engine_label` and `rocm_core::placeholder::empty_as_unknown`; nothing here calls back into `chat.rs`. The tests of the parsers, `managed_service_running_state` and the summary builders that need no chat turn live in this module's own `mod tests`; the tests that run a summary through the turn loop's follow-up decision stay in `chat/turn.rs`. +- [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) — mechanical relocation out of `chat.rs`, a leaf child of `chat.rs`: the MCP bridge and in-process command execution behind the assistant's tools. `run_internal_mcp_call` is the tool dispatch (`rocm mcp-call`, the dash execution seam and the assistant's read-only path all end in it): it validates the call through `tool_policy.rs`, refuses a state-changing tool unless mutation was allowed, then runs the tool in process or hands an approved `rocm` command to a child process. `run_rocm_read_only_in_process` is the in-process runner for the read-only `rocm` commands the assistant may ask for, `run_rocm_command_for_paths` and `run_rocm_capture_for_paths` wrap a command's output as a `CommandCapture`, and `internal_mcp_arguments`, `internal_mcp_tool_success` and `internal_mcp_tool_result_from_command` build the MCP tool result, and `mcp_tool_result_text` reads the text back out of one. It also holds `run_chat_read_only_tool` (the assistant's own read-only path), the `path_exists` and `port_status` tools (`run_chat_path_exists_tool`, `run_chat_port_status_tool`, with `loopback_tcp_port_is_reachable` and `loopback_host_key`), `render_services_tool_result_text` (the text of the `services` tool), the `install sdk --dry-run` argument and preview helpers (`internal_mcp_install_sdk_args`, `render_install_sdk_dry_run_for_args`) and the `logs` flag parsers (`parse_optional_query`, `parse_optional_lines`). `run_internal_mcp_call` is `pub(crate)` and re-exported from `chat.rs` for `main.rs` and `dash_seam.rs`; `run_chat_read_only_tool`, `loopback_host_key` and `mcp_tool_result_text` are `pub(super)` for the turn loop in `chat/turn.rs`, and `run_chat_path_exists_tool` and `mcp_tool_result_text` are also read by the command-preview property tests in `chat.rs`; the rest is private. Its edges out are `tool_policy.rs`, `rocm_argv.rs`, `chat/summaries.rs` (`managed_service_running_state`), `freeform_planner.rs` (the `natural_language_plan` tool), `providers` (the tool-call type) and the crate-root renderers and bridge builders; nothing in it calls back into the turn loop, its tests included. Its tests (the `logs` flag parsers, `mcp_tool_result_text`, the services text, the `proposal_action` tool, the in-process read-only commands, `natural_language_plan` and the port check) live in its own `mod tests`, with their original names. +- [`chat/intent.rs`](../apps/rocm/src/chat/intent.rs) — mechanical relocation out of `chat.rs`, a leaf child of `chat.rs`: the prompt-intent heuristics, which decide from a user's latest message alone what it asks for. `fallback_rocm_tool_call_for_prompt` maps a prompt to the read-only or approval-gated tool call the assistant proposes without asking the local model (with `fallback_config_tool_call_for_prompt`, `fallback_engine_list_tool_call` and `fallback_services_list_tool_call` behind it), `prompt_can_use_read_only_without_local_assistant` says whether a status question can be answered before any model server is up, and `install_sdk_chat_approval_for_prompt` (with `install_sdk_without_prefix_chat_approval`) turns an install or setup request into the `rocm install sdk` approval request. The phrase lists live in the `prompt_asks_running_or_status`, `prompt_asks_engine_install_state`, `prompt_mentions_serving_engine_or_service`, `prompt_requests_rocm_install_or_setup` and `prompt_requests_install_action` predicates, and `latest_user_chat_message` cuts the newest message out of the conversation text the chat builds around it, so older words never steer these heuristics; it moved here with the heuristics because three of them call it, and leaving it with the turn loop would make the two modules call each other. Ten of the functions are `pub(super)`, called from `chat/turn.rs`; the other three are private. Its edges out are [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs), [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `tool_policy.rs` (`ChatToolApprovalRequest`), `freeform_planner.rs` (`format_structured_tool_call`) and `providers` (the tool-call type); nothing in it calls back into `chat/turn.rs` or `chat.rs`. Its tests, the `fallback_tool_call_*` family and the install-intent checks that call the heuristics directly, live in its own `mod tests`, with their original names. - [`chat_cmd.rs`](../apps/rocm/src/chat_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm chat`'s command handler (`run_chat_command`, the body of the `Command::Chat` arm of `dispatch()`), which hands off to the dashboard chat (`dash::run_chat`) in a terminal with no `--prompt` and otherwise prints the reply or status screen through `chat::render_chat_prompt_text`/`chat::render_chat_text`. It is a module of its own, not part of [`chat.rs`](../apps/rocm/src/chat.rs), because it calls into both `chat.rs` and [`dash.rs`](../apps/rocm/src/dash.rs) while `dash.rs` already calls into `chat.rs`; this module depends on both and neither depends on it. `Provider` lives in [`cli.rs`](../apps/rocm/src/cli.rs) and `ChatInferenceParams` stays in [`main.rs`](../apps/rocm/src/main.rs); `run_chat_command` is `pub(crate)` and re-imported there for `dispatch()`. Four source-text tests in this module's own `mod tests` read this file and assert on the handler's text, so a rename of the function or of the calls they look for has to be made there too. - [`chat_host_facts.rs`](../apps/rocm/src/chat_host_facts.rs) — full domain extraction. - [`cli.rs`](../apps/rocm/src/cli.rs) — the `rocm` command line definition: `Cli`, `Command`, the subcommand enums nested under it (`BenchCommand`, `InstallTarget`, `EnginesCommand`, `RuntimesCommand`, `StorageCommand`, `ComfyuiCommand`, `ServicesCommand`, `AutomationsCommand`, `ConfigCommand`, `SetupCommand`) and the clap value enums behind `--provider`, `--device`, `--framework` and similar arguments (`Provider`, `DevicePolicyArg`, `FrameworkArg`, `InstallFormat`, `WatcherModeArg`, `TelemetryModeArg`, `SandboxToolArg`, `PermissionsModeArg`), with `SUPPORTED_ENGINES` (the `--engine` choices) and the conversions from a value enum to its domain type. Like [`apps/rocmd`](../apps/rocmd/src/cli.rs)'s `cli.rs`, it holds definitions only: the `dispatch()` that matches on `Command` stays in [`main.rs`](../apps/rocm/src/main.rs). Declaration order sets the order of every nested `--help` command list (the top-level list is sorted alphabetically by `cli_command()` in `main.rs`), so reordering variants reorders help output. Its items are `pub(crate)` and re-imported in `main.rs`, so other modules keep reaching them as `crate::Command`, `crate::Provider` and so on; [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs) and [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs) own their enums and appear in `Command` as a single variant each. The tests that parse the tree or render its help live in its own `mod tests`. - [`cli_audit.rs`](../apps/rocm/src/cli_audit.rs) — mechanical relocation, no owned types: the CLI audit and lifecycle log (`record_cli_audit_event`, plus the helpers behind it that write `logs/cli-lifecycle.log` and the per-action files under `logs/cli/`). Its production code depends only on `rocm-core`. `record_cli_audit_event`, `cli_lifecycle_log_path` and `sanitize_log_value` are `pub(crate)` and reached from other modules as `crate::` through a re-import in [`main.rs`](../apps/rocm/src/main.rs); every other item is private. The two regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`. - [`cli_progress.rs`](../apps/rocm/src/cli_progress.rs) — shared CLI-output component (`Spinner`, `AnimatedSpinner`). - [`cli_report.rs`](../apps/rocm/src/cli_report.rs) — shared CLI-output component (`ActionReport`, `write_marked_row`, `JsonOrText`/`print_json_or_text`). -- [`comfyui.rs`](../apps/rocm/src/comfyui.rs) — full domain extraction; serves the `comfyui` subcommand, whose command enum (`ComfyuiCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs) and dispatch fn stays in [`main.rs`](../apps/rocm/src/main.rs) (unlike [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs) above, which owns both). It owns the ComfyUI types and constants, the install and state paths, the manifest and state files, `status` and `logs` rendering, and the helpers that run a Python child and capture its output. Choosing the ROCm runtime and building its environment ([`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs)), the process lifecycle ([`comfyui/process.rs`](../apps/rocm/src/comfyui/process.rs)) and the install ([`comfyui/install.rs`](../apps/rocm/src/comfyui/install.rs)) are in child modules, listed below with the other `apps/rocm` modules; `start`, `stop` and `install` are re-exported from them so callers still write `comfyui::start`, `comfyui::stop` and `comfyui::install`. -- [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm config`'s command handler (`config()`), the provider-key prompt it uses (`read_provider_key_from_user`) and the text `rocm config show` prints (`render_config_text`, `telemetry_policy_summary`). `ConfigCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `config()` and `render_config_text` are `pub(crate)`: `dispatch()` calls `config()` and [`chat.rs`](../apps/rocm/src/chat.rs)'s read-only handling of `rocm config show` calls `render_config_text`, both through re-imports in `main.rs`; the other two functions are private. The two `render_config_text` regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`. +- [`comfyui.rs`](../apps/rocm/src/comfyui.rs) — full domain extraction; serves the `comfyui` subcommand, whose command enum (`ComfyuiCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs) while its dispatch fn `run`, the body of the `Comfyui` arm of `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), is defined here (unlike [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs) above, which owns the enum as well). It owns that dispatch fn, which also records the audit event for each ComfyUI action, the ComfyUI types and constants, the install and state paths, the manifest and state files, `status` and `logs` rendering, and the helpers that run a Python child and capture its output. Choosing the ROCm runtime and building its environment ([`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs)), the process lifecycle ([`comfyui/process.rs`](../apps/rocm/src/comfyui/process.rs)) and the install ([`comfyui/install.rs`](../apps/rocm/src/comfyui/install.rs)) are in child modules, listed below with the other `apps/rocm` modules; `start`, `stop` and `install` are re-exported from them so callers still write `comfyui::start`, `comfyui::stop` and `comfyui::install`. +- [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm config`'s command handler (`config()`), the provider-key prompt it uses (`read_provider_key_from_user`) and the text `rocm config show` prints (`render_config_text`, `telemetry_policy_summary`). `ConfigCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `config()` and `render_config_text` are `pub(crate)`: `dispatch()` calls `config()` and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only handling of `rocm config show` calls `render_config_text`, both through re-imports in `main.rs`; the other two functions are private. The two `render_config_text` regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`. - [`dash.rs`](../apps/rocm/src/dash.rs) — full domain extraction. - [`dash_seam.rs`](../apps/rocm/src/dash_seam.rs) — full domain extraction. - [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: the command handlers for `rocm diagnose` and `rocm fix` (`diagnose()`, `fix()`) and the report step behind `rocm diagnose --report` (`show_prepared_report`, `perform_delivery`, `established_entry`). `diagnose()` and `fix()` are `pub(crate)` and re-imported in [`main.rs`](../apps/rocm/src/main.rs) for `dispatch()`; the other three functions are private. `diagnose()` gets its `--model` verdict from [`model_fit.rs`](../apps/rocm/src/model_fit.rs), and `fix()` returns a failing exit code through `FixExitCode`, which stays in `main.rs` with the other exit-code plumbing. The report and delivery regression tests live in this module's own `mod tests`; the exit-code tests stay in `main.rs`. @@ -55,17 +60,17 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-215): deciding which runtime/env a `serve`/`engines install` invocation resolves to (`EngineSelection`, `resolve_engine_selection`, `validate_engine_selection_runtime`, `single_ready_runtime_key`, `resolve_runtime_selector_to_exact_key`), small argv-building helpers used when relaunching a managed engine (`optional_arg`, `engine_recipe_json_arg`, `parse_engine_recipe_json_arg`, `daemon_run_argv`, `managed_service_launcher_path`), the `ROCM_CLI_*` environment variables every managed child inherits (`app_path_env_vars`/`app_path_env_var_values`/`app_path_env_var_refs`/`apply_app_path_env`), and polling a freshly launched service until it can answer inference requests (`wait_for_service_http_ready`, `wait_for_service_http_ready_with_progress`, `service_http_readiness_paths`, `http_get_local_service`, `http_post_local_service_json`, `service_http_readiness_response_ready`, `lemonade_health_ready_for_model`, `model_list_ready_for_model`, `service_model_gpu_ready`, `service_model_reports_rocm_backend`, `service_model_names_match`). `single_ready_runtime_key` does not decide which runtimes count as ready: it asks [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs)'s `select_single_ready_runtime`, which `rocm comfyui install` shares. `const fn status_for_readiness` sits physically interleaved in this span in `main.rs` but stayed there instead of moving — it has callers outside this cluster ([`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `record_restart_readiness`, which `restart_internal_managed_service_with` calls), so it stays a shared crate-root function, reached only from this module's own tests, via `use crate::status_for_readiness;`. Which items are `pub(crate)` and who calls them is documented in the module's own doc comment rather than repeated here. The cluster's regression tests live in the module's own `mod tests`. What remains in `main.rs` immediately after this cluster is `treat_as_natural_language`, dispatch-adjacent natural-language-invocation routing left over from an earlier phase (ROCMAI-87), explicitly out of scope and untouched by this split. - [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs) — mechanical relocation, owns private types (Phase 6a): `engines()` (the `rocm engines` command), `ensure_self_managed_engine_ready` (the readiness check [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) runs for engines that bring their own runtime) and `env_root_for_service` (read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs)). Which runtime an engine installs into and where its environment lives is answered in [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs), which this module imports from. This module also calls into [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) for `settle_engine_install` and the host system-package installs; that module does not import this one. - [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) — mechanical relocation, dispatch-adjacent, owns private types: `rocm examine`'s command handler (`examine()`), which prints the human report through [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `examine_human_report`, or with `--json` the document built from the private `ExamineJson`/`ExamineJsonSummary` types (the `Examination` verbatim plus a summary of what the CLI knows but the probe cannot). `examine()` is `pub(crate)` and re-imported in [`main.rs`](../apps/rocm/src/main.rs) for `dispatch()`. No test in the crate calls it or builds the document directly. -- [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) — mechanical relocation: the launch-summary/examine text surfaces (`render_launch_summary`, `render_examine_text`, `render_examine_text_with_paths`, `examine_human_report`, `render_examine_plain_header`, `plain_status_label`) and the engine-inventory listing shared by `rocm engines` and `rocm examine` (`write_default_engine_legend`, `render_engine_inventory_text_with_paths`, `append_examine_runtime_state`, `append_examine_engine_inventory`, `append_engine_detect_summary`, `friendly_engine_runtime_kind`, `friendly_engine_detect_notes`, `friendly_engine_detect_note_fallback`, `engine_runtime_status_label`). The `DEFAULT_ENGINE_MARKER` constant is shared by both engine-inventory renderers so the two copies cannot drift apart. The `rocm model` catalog renderer is not here; it lives in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), and nothing in this file calls into it outside its own tests. [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs)'s `examine()`, which the `Command::Examine` arm calls, imports and calls `examine_human_report`; `main.rs`'s own `launch_default` calls `render_launch_summary`, re-imported there. `render_examine_text`, `render_examine_text_with_paths` and `render_engine_inventory_text_with_paths` are `pub(crate)`, read from [`chat.rs`](../apps/rocm/src/chat.rs)'s deterministic tool-output summaries; `render_engine_inventory_text_with_paths` is also read from [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s `render_engine_inventory_text`; `append_examine_runtime_state` is also `pub(crate)`, read only from [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s own test module, so `main.rs`'s re-import of it is itself `#[cfg(test)]` rather than carrying an otherwise-unused import into a non-test build. Every other item in this list has no caller outside this module and stays private. This module also reaches items that live at the crate root because they have other callers: `engine_inventory()` (also read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) and [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` (defined in [`model_fit.rs`](../apps/rocm/src/model_fit.rs) and re-imported in `main.rs`) — all reached via `use crate::{...};`. The regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime`/`runtime_manifest_path` through `crate::test_support::`/`crate::runtimes_cmd::`. One test here, `linux_only_engine_runtime_status_is_explicit_on_native_windows`, also calls `model_registry_adapter_availability_note` from `model_catalog_text.rs`, which is `pub(crate)` for that reason; its primary subject is `engine_runtime_status_label`, so it stays. +- [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) — mechanical relocation: the launch-summary/examine text surfaces (`render_launch_summary`, `render_examine_text`, `render_examine_text_with_paths`, `examine_human_report`, `render_examine_plain_header`, `plain_status_label`) and the engine-inventory listing shared by `rocm engines` and `rocm examine` (`write_default_engine_legend`, `render_engine_inventory_text_with_paths`, `append_examine_runtime_state`, `append_examine_engine_inventory`, `append_engine_detect_summary`, `friendly_engine_runtime_kind`, `friendly_engine_detect_notes`, `friendly_engine_detect_note_fallback`, `engine_runtime_status_label`). The `DEFAULT_ENGINE_MARKER` constant is shared by both engine-inventory renderers so the two copies cannot drift apart. The `rocm model` catalog renderer is not here; it lives in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), and nothing in this file calls into it. [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs)'s `examine()`, which the `Command::Examine` arm calls, imports and calls `examine_human_report`; `main.rs`'s own `launch_default` calls `render_launch_summary`, re-imported there. `render_examine_text`, `render_examine_text_with_paths` and `render_engine_inventory_text_with_paths` are `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `render_engine_inventory_text_with_paths` is also read from [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s `render_engine_inventory_text`; `append_examine_runtime_state` is also `pub(crate)`, read only from [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s own test module, so `main.rs`'s re-import of it is itself `#[cfg(test)]` rather than carrying an otherwise-unused import into a non-test build. Every other item in this list has no caller outside this module and stays private. This module also reaches items that live at the crate root because they have other callers: `engine_inventory()` (also read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) and [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` (defined in [`model_fit.rs`](../apps/rocm/src/model_fit.rs) and re-imported in `main.rs`) — all reached via `use crate::{...};`. The regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime`/`runtime_manifest_path` through `crate::test_support::`/`crate::runtimes_cmd::`. Its tests do not call into `model_catalog_text.rs`: the test of the note the verbose `rocm model` listing prints for a rule-out engine lives beside that renderer. - [`freeform_answer.rs`](../apps/rocm/src/freeform_answer.rs) — mechanical relocation, no owned types: the answers a freeform request can get without running a command — `render_freeform_read_only_answer`, which answers a single approval-free `examine`, `comfyui status` or `comfyui logs` action, and `render_freeform_examine_answer`, the plain-language ROCm status it prints for `examine`. `run_freeform` in [`main.rs`](../apps/rocm/src/main.rs) is the only caller of the first, which is `pub(crate)` and re-imported there; the second is private. Besides `chat.rs` (`any_substring`, re-exported there from [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs)), it depends on `comfyui.rs`, `runtimes_cmd.rs` and `therock.rs`, none of which calls back into it. No unit test calls either function directly. - [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) — mechanical relocation, dispatch-adjacent, owns a `pub(crate)` type (as of Phase 6e's follow-up, ROCMAI-726; see the Phase 6a variant note above for the owned-types pattern). Holds the freeform natural-language planner's plan-*execution*/consent half that [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) below deliberately left out: `validate_freeform_execution_action()`, `prepare_freeform_execution()`, `apply_freeform_execution_consent()`, `render_freeform_execution_header()`, and the `pub(crate)` `FreeformExecution` struct that `prepare_freeform_execution` returns and `render_freeform_execution_header` consumes. `main.rs`'s `execute_freeform_next_action()` and `run_freeform()` stay in `main.rs` — both are dispatch-level, not part of deciding whether a plan's next action is safe to execute — and `execute_freeform_next_action()` calls in via the unqualified re-import `use crate::freeform_execution::{prepare_freeform_execution, render_freeform_execution_header};` (`run_freeform()` only calls `execute_freeform_next_action()`, never this module directly), matching [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s dispatch-adjacent form rather than `freeform_planner.rs`'s own qualified-path form. `prepare_freeform_execution` takes the already-built `plan` by value rather than rebuilding it from `request`/`paths`/`config` (ROCMAI-725's single-build guarantee, inherited unchanged from `main.rs`); `prepare_freeform_execution`/`render_freeform_execution_header` are `pub(crate)` for that cross-module call; `FreeformExecution` was already `pub(crate)` (read directly by `main.rs` and this module's own tests); `validate_freeform_execution_action`/`apply_freeform_execution_consent` have no caller outside this module and stay module-private, unchanged from before the move. The cluster's six regression tests moved out of `main.rs`'s shared `mod tests` into this module's own `mod tests`, inheriting the qualified `freeform_planner::` path they already used to reach `freeform_planner.rs`'s items as `main.rs`'s former tests — this module reaches in from outside using the same qualified path `main.rs` and `providers.rs`'s tests use (unlike `chat.rs`/`comfyui.rs`, which import it unqualified). -- [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) — full domain extraction (Phase 6e, ROCMAI-87). Owns the freeform natural-language planner cluster's types (`StructuredRequestPlan`, `PlannerIntent`, `PlannedToolCall`, `FreeformPlanAction`, `ProviderPlannerResponse`, `ProviderPlannerToolCall`, all module-private except `FreeformPlanAction`, `StructuredRequestPlan` and `PlannedToolCall`, which are `pub(crate)`, and `ProviderPlannerToolCall`, `pub(crate)` together with `validate_provider_planner_tool_call` and `planner_failure_note` so [`chat.rs`](../apps/rocm/src/chat.rs)'s command-preview property tests can drive the planner's guards), the deterministic plan builder (`build_freeform_plan*`), the provider-assisted path (`resolve_freeform_plan_with_provider`, `build_provider_planner_prompt`, `provider_planner_response_to_plan`, `validate_provider_planner_tool_call`), and plan rendering (`render_structured_request_plan`). [`main.rs`](../apps/rocm/src/main.rs) calls `build_freeform_plan_with_context`/`render_structured_request_plan` directly for the dispatch/review flow ([`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `render_launch_summary` → `host_default_engine` is another call into this module, unrelated to planning: bare `rocm`'s non-interactive launch summary), while [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) above calls `plan_next_action`/`format_structured_tool_call` for the execution flow and reads `FreeformPlanAction`/`StructuredRequestPlan` by type for its execution-validation functions — those live there, not here, because they decide whether a plan's next action is safe to *execute*, a dispatch-adjacent concern distinct from *building* the plan; `main.rs`'s `run_freeform` builds a plan exactly once via `build_freeform_plan_with_context` and threads that same value through `execute_freeform_next_action` to `prepare_freeform_execution` rather than rebuilding it (ROCMAI-725: building a plan can call a configured cloud `planner_provider`, which is not guaranteed deterministic, so a second independent build could silently disagree with the first). [`chat.rs`](../apps/rocm/src/chat.rs)'s `natural_language_plan` MCP tool gets the same single-build guarantee from `plan_freeform_request_with_context`, which builds once and returns both the rendered text and the next action; `chat.rs` also calls `format_structured_tool_call` directly, like [`comfyui.rs`](../apps/rocm/src/comfyui.rs) does, while [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs)'s `validate_chat_rocm_command_safety` reads `serve_args_request_cpu_device` from [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) directly as a no-CPU-fallback safety check on provider-proposed `serve` args; `validate_provider_planner_tool_call` calls that same function in its own `serve` branch, and hands an `install sdk` call to `tool_policy::validate_chat_rocm_command_safety`. `host_default_engine`, `build_freeform_plan_with_context`, `render_structured_request_plan`, `plan_next_action`, `plan_freeform_request_with_context` and `provider_planner_response_to_plan` are `pub(crate)` for these reasons, as is `plan_driver_request` (read from [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs)'s test module); `format_structured_tool_call` lives in [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) and is re-exported here with `pub(crate) use`, so the callers above keep their `freeform_planner::` path, while the planner's own argv lookups call `rocm_argv`, its phrase and engine readers call [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) and its safety check calls `tool_policy.rs`, directly: the planner no longer imports `chat`, and only `chat` imports the planner (`plan_freeform_request_with_context`); `freeform_plan_next_action` and `render_freeform_plan` are test-only (`#[cfg(test)]`) and `FreeformPlanAction` was already `pub(crate)` before the split. `parse_freeform_invocation`/`should_treat_as_freeform` (deciding whether an invocation is freeform at all) stay in `main.rs`, a dispatch-level concern outside both this module and `freeform_execution.rs`. The cluster's regression tests live in `freeform_planner.rs`'s own `mod tests`; the handful of `main.rs` tests that exercise the invocation classifier/clap-suggestion-avoidance stay in `main.rs`'s shared `mod tests`, since neither is specific to this module or to `freeform_execution.rs` (and neither touches `freeform_planner::` at all), and the ROCMAI-725 regression tests proving the single-build guarantee live in `providers.rs`'s `mod tests` (reusing its mock local-provider server), reaching this module's items through the qualified `freeform_planner::` path like any other external caller. +- [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) — full domain extraction (Phase 6e, ROCMAI-87). Owns the freeform natural-language planner cluster's types (`StructuredRequestPlan`, `PlannerIntent`, `PlannedToolCall`, `FreeformPlanAction`, `ProviderPlannerResponse`, `ProviderPlannerToolCall`, all module-private except `FreeformPlanAction`, `StructuredRequestPlan` and `PlannedToolCall`, which are `pub(crate)`, and `ProviderPlannerToolCall`, `pub(crate)` together with `validate_provider_planner_tool_call` and `planner_failure_note` so [`chat.rs`](../apps/rocm/src/chat.rs)'s command-preview property tests can drive the planner's guards), the deterministic plan builder (`build_freeform_plan*`), the provider-assisted path (`resolve_freeform_plan_with_provider`, `build_provider_planner_prompt`, `provider_planner_response_to_plan`, `validate_provider_planner_tool_call`), and plan rendering (`render_structured_request_plan`). [`main.rs`](../apps/rocm/src/main.rs) calls `build_freeform_plan_with_context`/`render_structured_request_plan` directly for the dispatch/review flow ([`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `render_launch_summary` → `host_default_engine` is another call into this module, unrelated to planning: bare `rocm`'s non-interactive launch summary), while [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) above calls `plan_next_action`/`format_structured_tool_call` for the execution flow and reads `FreeformPlanAction`/`StructuredRequestPlan` by type for its execution-validation functions — those live there, not here, because they decide whether a plan's next action is safe to *execute*, a dispatch-adjacent concern distinct from *building* the plan; `main.rs`'s `run_freeform` builds a plan exactly once via `build_freeform_plan_with_context` and threads that same value through `execute_freeform_next_action` to `prepare_freeform_execution` rather than rebuilding it (ROCMAI-725: building a plan can call a configured cloud `planner_provider`, which is not guaranteed deterministic, so a second independent build could silently disagree with the first). [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s `natural_language_plan` MCP tool gets the same single-build guarantee from `plan_freeform_request_with_context`, which builds once and returns both the rendered text and the next action; `chat/turn.rs` and `chat/mcp_bridge.rs` also call `format_structured_tool_call` directly, like [`comfyui.rs`](../apps/rocm/src/comfyui.rs) does, while [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs)'s `validate_serve_command_safety` holds the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), reading `serve_args_request_cpu_device` from [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) for the no-CPU-fallback rule; `validate_provider_planner_tool_call` calls it from its `serve` branch with the actor `provider planner` (`validate_chat_rocm_command_safety` calls it with `local assistant`, so the two refusals differ only in that name), and hands an `install sdk` call to `tool_policy::validate_chat_rocm_command_safety`. `host_default_engine`, `build_freeform_plan_with_context`, `render_structured_request_plan`, `plan_next_action`, `plan_freeform_request_with_context` and `provider_planner_response_to_plan` are `pub(crate)` for these reasons, as is `plan_driver_request` (read from [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs)'s test module); `format_structured_tool_call` lives in [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) and is re-exported here with `pub(crate) use`, so the callers above keep their `freeform_planner::` path, while the planner's own argv lookups call `rocm_argv`, its phrase and engine readers call [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) and its safety check calls `tool_policy.rs`, directly: the planner no longer imports `chat`, and only `chat` imports the planner (`plan_freeform_request_with_context`); `freeform_plan_next_action` and `render_freeform_plan` are test-only (`#[cfg(test)]`) and `FreeformPlanAction` was already `pub(crate)` before the split. `parse_freeform_invocation`/`should_treat_as_freeform` (deciding whether an invocation is freeform at all) stay in `main.rs`, a dispatch-level concern outside both this module and `freeform_execution.rs`. The cluster's regression tests live in `freeform_planner.rs`'s own `mod tests`; the handful of `main.rs` tests that exercise the invocation classifier/clap-suggestion-avoidance stay in `main.rs`'s shared `mod tests`, since neither is specific to this module or to `freeform_execution.rs` (and neither touches `freeform_planner::` at all), and the ROCMAI-725 regression tests proving the single-build guarantee live in `providers.rs`'s `mod tests` (reusing its mock local-provider server), reaching this module's items through the qualified `freeform_planner::` path like any other external caller. - [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-214): `--device`/`--gpu` parsing (`parse_device_policy`, `parse_gpu_selection`, `parse_gpu_indices_arg`), the managed-launch-lock GPU selection (`select_gpu_indices_under_launch_lock`, `validate_pinned_gpu_index`, `effective_gpu_count`, `auto_select_gpu_indices`, `select_auto_gpu_index`, `busy_gpu_indices`), per-GPU VRAM telemetry (the `GpuVramUsage` type, `gpu_vram_usage`, `simulated_low_vram_usage`, `gpu_vram_usage_amd_smi`, `gpu_vram_usage_sysfs`, `read_drm_vram_usage`, `read_sysfs_u64`, `drm_device_is_amd`, `parse_gpu_vram_usage`), and the serve-plan low-VRAM warning (`gpu_low_memory_warning`, `vram_capacity_is_meaningful`, `serve_gpu_low_memory_warning`), plus the best-effort `amd-smi`-based GPU count (`detect_gpu_count`).Both `amd-smi` probes go through `rocm_core::amd_smi_json` (the `/dev/kfd` pre-flight) with this module's own `AMD_SMI_SERVE_PROBE_TIMEOUT`, whose doc records the resulting worst-case `serve` latency. `const fn device_policy_name` sits immediately after this cluster's last function in the original file but stayed in `main.rs` rather than moving — it already had callers outside the cluster (`engine_ipc.rs`, `service_launch.rs`, `serve_cmd.rs`) plus its own test. `GpuVramUsage` (and its fields), `gpu_vram_usage`, `parse_gpu_indices_arg`, `serve_gpu_low_memory_warning`, and `vram_capacity_is_meaningful` are `pub(crate)` and read from [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`classify_accelerator_memory`, `host_accelerator_memory`) [`main.rs`](../apps/rocm/src/main.rs) (the `dispatch()` `Command::EngineServeHttp` arm) and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) (`collect_serve_notes`); `detect_gpu_count`, `gpu_vram_usage`, `parse_device_policy`, `parse_gpu_selection`, `select_gpu_indices_under_launch_lock`, `serve_gpu_low_memory_warning`, and `validate_pinned_gpu_index` are also `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs); `parse_device_policy` is also read from [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs)'s `run_builtin_engine_serve_http` and, as of Phase 6f (ROCMAI-209), from [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `restart_child_argv` (moved out of `main.rs` in that same phase, and since relocated from `services_text.rs`). `effective_gpu_count`, `auto_select_gpu_indices`, `select_auto_gpu_index`, `simulated_low_vram_usage`, `gpu_vram_usage_amd_smi`, `gpu_vram_usage_sysfs` (both the Linux and non-Linux `cfg` variants), `read_drm_vram_usage`, `read_sysfs_u64`, `drm_device_is_amd`, `parse_gpu_vram_usage`, `gpu_low_memory_warning`, and `busy_gpu_indices` have no callers outside this module and stay private. The cluster's regression tests live in `gpu_policy.rs`'s own `mod tests`; two small test-only fixture builders (`vram`, `host_gpu`) are duplicated rather than shared, since `serve_cmd.rs`'s own `serve_notes_pair_low_vram_with_the_vllm_utilization_hint` (testing `collect_serve_notes`) needs them too. -- [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm install`'s command handler (`install()`, for the `sdk` and `driver` targets), the `--version`/`--build-date` selector parsing and display it uses (`therock_install_version_selector`, `therock_install_version_selector_display`) and `sdk_install_request`, which maps the parsed `install sdk` arguments onto a `therock::SdkInstallRequest` and moved here from [`chat.rs`](../apps/rocm/src/chat.rs) with its two tests. `InstallTarget` and `InstallFormat` live in [`cli.rs`](../apps/rocm/src/cli.rs). `install()` is `pub(crate)` and re-imported in `main.rs` for `dispatch()`; `therock_install_version_selector` is `pub(crate)` too, and `chat.rs`'s `render_install_sdk_dry_run_for_args` reaches it as `crate::therock_install_version_selector` through the same kind of re-import, which is why `sdk_install_request` lives here and not in `chat.rs`: this module does not depend on `chat.rs`. `sdk_install_request` keeps the `pub(crate)` it had, although only `install()` and the tests call it now. +- [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm install`'s command handler (`install()`, for the `sdk` and `driver` targets), the `--version`/`--build-date` selector parsing and display it uses (`therock_install_version_selector`, `therock_install_version_selector_display`) and `sdk_install_request`, which maps the parsed `install sdk` arguments onto a `therock::SdkInstallRequest` and moved here from [`chat.rs`](../apps/rocm/src/chat.rs) with its two tests. `InstallTarget` and `InstallFormat` live in [`cli.rs`](../apps/rocm/src/cli.rs). `install()` is `pub(crate)` and re-imported in `main.rs` for `dispatch()`; `therock_install_version_selector` is `pub(crate)` too, and `chat/mcp_bridge.rs`'s `render_install_sdk_dry_run_for_args` reaches it as `crate::therock_install_version_selector` through the same kind of re-import, which is why `sdk_install_request` lives here and not in `chat.rs`: this module does not depend on `chat.rs`. `sdk_install_request` keeps the `pub(crate)` it had, although only `install()` and the tests call it now. - [`logging.rs`](../apps/rocm/src/logging.rs) — mechanical relocation, no owned types, reached by qualified path. -- [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-211): managed-service liveness classification (`managed_service_is_live`, `refresh_managed_service_runtime_liveness`, `existing_live_managed_service_in`'s idempotency guard) and the dashboard-sidebar status/counts helpers derived from it (`ManagedServiceSidebarCounts`/`managed_service_sidebar_counts`/`local_server_sidebar_status`), plus the engine display-name helper `friendly_engine_label`, read only by [`chat.rs`](../apps/rocm/src/chat.rs). `ManagedServiceSidebarCounts` is `pub(crate)` to match the visibility of the `pub(crate)` functions that return/take it; `friendly_engine_label`, `existing_live_managed_service_in`, `recorded_service_pids`, `refresh_managed_service_runtime_liveness`, `local_server_sidebar_status`, `managed_service_sidebar_counts`, and `load_managed_services` are `pub(crate)` for the same reason — each has a confirmed caller outside this module. [`service_records.rs`](../apps/rocm/src/service_records.rs) uses its liveness helpers (`managed_service_is_live`, `recorded_service_pids`) for the in-use check behind service-record removal and pruning, and `load_managed_services` for prune planning, while [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) uses `load_managed_services` for the codex-bridge snapshot; [`service_launch.rs`](../apps/rocm/src/service_launch.rs) is the only caller of the `existing_live_managed_service_in` duplicate-launch guard; [`services_text.rs`](../apps/rocm/src/services_text.rs) reads the sidebar helpers; other modules, including [`chat.rs`](../apps/rocm/src/chat.rs) and [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs), read the registry and liveness helpers. -- [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs) — mechanical relocation out of `examine_report_text.rs`: the `rocm model` (alias `models`) catalog renderer. `render_model_registry_text_with_context_and_host` prints the curated list grouped by hardware target (`model_catalog_platform_heading`, `model_catalog_header`, `model_recipe_memory_label`, `model_recipe_gpu_fit_label`), and `render_model_registry_verbose_text_with_context_and_host` prints the per-recipe diagnostics behind `rocm model --verbose` (`append_model_recipe_registry_source`, `append_model_recipe_metadata_lines`, `append_model_engine_recipe_settings_lines`, `append_model_recipe_staleness_lines`, `append_model_artifact_lines`, `append_artifact_source_policy_lines`, `artifact_source_policy_label`, `append_model_host_ram_fit_lines`, `append_model_fit_lines`, `append_manual_alternative_lines`, `manual_alternative_recommendations`, `recipe_is_manual_fit`, `append_model_engine_support_lines`, `model_registry_adapter_availability_note`, `recipe_display_ref`, `preferred_engine_action_target`, `model_registry_reason`, plus the small formatters `format_list_or_none`, `format_string_map_or_none`, `format_gib` and `once_per_call`). `format_bytes`, the human-readable size formatter, lives here too. The two renderers and `format_bytes` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs) imports them for the `Command::Model` arm, and [`chat.rs`](../apps/rocm/src/chat.rs) (the verbose renderer), [`storage.rs`](../apps/rocm/src/storage.rs) and [`automations.rs`](../apps/rocm/src/automations.rs) (`format_bytes`) reach them as `crate::`. `model_registry_adapter_availability_note` is `pub(crate)` only because a test in `examine_report_text.rs` calls it. The module has no edge to `examine_report_text.rs`; it reaches the engine-availability helpers in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs), `PlatformRuleOut` and `engine_ruled_out_by_platform_with` at the crate root, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `select_serve_engine`. The catalog's regression tests live in this module's own `mod tests`; they keep the names they had in `examine_report_text.rs`. The `format_bytes` property test's saved failure case lives in [`model_catalog_text.txt`](../apps/rocm/proptest-regressions/model_catalog_text.txt), which proptest finds by the test's source file, so that file has to be renamed whenever the test moves. +- [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-211): managed-service liveness classification (`managed_service_is_live`, `refresh_managed_service_runtime_liveness`, `existing_live_managed_service_in`'s idempotency guard) and the dashboard-sidebar status/counts helpers derived from it (`ManagedServiceSidebarCounts`/`managed_service_sidebar_counts`/`local_server_sidebar_status`), plus the engine display-name helper `friendly_engine_label`, read only by [`chat/summaries.rs`](../apps/rocm/src/chat/summaries.rs). `ManagedServiceSidebarCounts` is `pub(crate)` to match the visibility of the `pub(crate)` functions that return/take it; `friendly_engine_label`, `existing_live_managed_service_in`, `recorded_service_pids`, `refresh_managed_service_runtime_liveness`, `local_server_sidebar_status`, `managed_service_sidebar_counts`, and `load_managed_services` are `pub(crate)` for the same reason — each has a confirmed caller outside this module. [`service_records.rs`](../apps/rocm/src/service_records.rs) uses its liveness helpers (`managed_service_is_live`, `recorded_service_pids`) for the in-use check behind service-record removal and pruning, and `load_managed_services` for prune planning, while [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) uses `load_managed_services` for the codex-bridge snapshot; [`service_launch.rs`](../apps/rocm/src/service_launch.rs) is the only caller of the `existing_live_managed_service_in` duplicate-launch guard; [`services_text.rs`](../apps/rocm/src/services_text.rs) reads the sidebar helpers; other modules, including [`chat/turn.rs`](../apps/rocm/src/chat/turn.rs), [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) and [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs), read the registry and liveness helpers. +- [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs) — mechanical relocation out of `examine_report_text.rs`: the `rocm model` (alias `models`) catalog renderer. `render_model_registry_text_with_context_and_host` prints the curated list grouped by hardware target (`model_catalog_platform_heading`, `model_catalog_header`, `model_recipe_memory_label`, `model_recipe_gpu_fit_label`), and `render_model_registry_verbose_text_with_context_and_host` prints the per-recipe diagnostics behind `rocm model --verbose` (`append_model_recipe_registry_source`, `append_model_recipe_metadata_lines`, `append_model_engine_recipe_settings_lines`, `append_model_recipe_staleness_lines`, `append_model_artifact_lines`, `append_artifact_source_policy_lines`, `artifact_source_policy_label`, `append_model_host_ram_fit_lines`, `append_model_fit_lines`, `append_manual_alternative_lines`, `manual_alternative_recommendations`, `recipe_is_manual_fit`, `append_model_engine_support_lines`, `model_registry_adapter_availability_note`, `recipe_display_ref`, `preferred_engine_action_target`, `model_registry_reason`, plus the small formatters `format_list_or_none`, `format_string_map_or_none`, `format_gib` and `once_per_call`). `format_bytes`, the human-readable size formatter, lives here too. The two renderers and `format_bytes` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs) imports them for the `Command::Model` arm, and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the verbose renderer), [`storage.rs`](../apps/rocm/src/storage.rs) and [`automations.rs`](../apps/rocm/src/automations.rs) (`format_bytes`) reach them as `crate::`. Every other item, including `model_registry_adapter_availability_note`, is private. The module has no edge to `examine_report_text.rs`; it reaches the engine-availability helpers in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs), `PlatformRuleOut` and `engine_ruled_out_by_platform_with` at the crate root, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `select_serve_engine`. The catalog's regression tests live in this module's own `mod tests`; they keep the names they had in `examine_report_text.rs`. The `format_bytes` property test's saved failure case lives in [`model_catalog_text.txt`](../apps/rocm/proptest-regressions/model_catalog_text.txt), which proptest finds by the test's source file, so that file has to be renamed whenever the test moves. - [`model_fit.rs`](../apps/rocm/src/model_fit.rs) — mechanical relocation, owns a `pub(crate)` type: answering whether a host can run a model for `rocm diagnose --model` (`assess_model_on_this_host`, `assess_model_with_host_paths`, `assess_model_for_host`), the accelerator-memory fact it rests on (`host_accelerator_memory`, `classify_accelerator_memory`, `examination_reports_amd_gpu`) and the platform gate that rules an engine out (`PlatformRuleOut`, `engine_ruled_out_by_platform`, `unsupported_here_for`). It is a module of its own, rather than part of the `diagnose` handler, so [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) and [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), which read `PlatformRuleOut` and the platform gate (`engine_ruled_out_by_platform` and `engine_ruled_out_by_platform_with` respectively), do not depend on a command module. It depends on [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) (`select_serve_engine`) and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs), neither of which depends on it. `assess_model_on_this_host` (called from [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` are `pub(crate)`, and [`main.rs`](../apps/rocm/src/main.rs) re-imports the last two so `examine_report_text.rs` reaches them as `crate::` (`model_catalog_text.rs` reaches `PlatformRuleOut` and `engine_ruled_out_by_platform_with` the same way); every other item is private. The cluster's regression tests live in this module's own `mod tests`; the one that re-runs the test binary on a helper test by its full path (`engine_ruled_out_by_platform_gates_aim_on_docker_reachability`) names the helper's new path, `model_fit::tests::docker_unreachable_subprocess_helper`. -- [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) — text extraction from a natural-language request, shared by `chat.rs`'s prompt-intent heuristics (`fallback_rocm_tool_call_for_prompt`, `fallback_config_tool_call_for_prompt`, `install_sdk_chat_approval_for_prompt` and the `prompt_*` predicates) and the freeform planner, in a leaf module that depends only on `therock::RuntimeVersionSelector` (`tool_policy.rs` does not use it): `any_substring`, the install-folder, TheRock build-date and TheRock version readers (`requested_install_prefix_from_prompt`, `requested_therock_build_date_from_prompt`, `requested_therock_version_from_prompt`, with `requested_bare_install_prefix_from_prompt`, `clean_requested_install_prefix`, `looks_like_user_path` and `looks_like_therock_runtime_version` private beside them) from `chat.rs`, and `infer_engine_from_request` from `freeform_planner.rs`. They moved verbatim, together with `infer_engine_from_request`'s regression test, so that `freeform_planner.rs` no longer imports `chat` for them; `chat.rs` re-exports `any_substring`, which [`freeform_answer.rs`](../apps/rocm/src/freeform_answer.rs) still reaches as `chat::any_substring`. +- [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) — text extraction from a natural-language request, shared by `chat/intent.rs`'s prompt-intent heuristics (`fallback_rocm_tool_call_for_prompt`, `fallback_config_tool_call_for_prompt`, `install_sdk_chat_approval_for_prompt` and the `prompt_*` predicates) and the freeform planner, in a leaf module that depends only on `therock::RuntimeVersionSelector` (`tool_policy.rs` does not use it): `any_substring`, the install-folder, TheRock build-date and TheRock version readers (`requested_install_prefix_from_prompt`, `requested_therock_build_date_from_prompt`, `requested_therock_version_from_prompt`, with `requested_bare_install_prefix_from_prompt`, `clean_requested_install_prefix`, `looks_like_user_path` and `looks_like_therock_runtime_version` private beside them) from `chat.rs`, and `infer_engine_from_request` from `freeform_planner.rs`. They moved verbatim, together with `infer_engine_from_request`'s regression test, so that `freeform_planner.rs` no longer imports `chat` for them; `chat.rs` re-exports `any_substring`, which [`freeform_answer.rs`](../apps/rocm/src/freeform_answer.rs) still reaches as `chat::any_substring`. - [`provider_keys.rs`](../apps/rocm/src/provider_keys.rs) — full domain extraction. - [`providers.rs`](../apps/rocm/src/providers.rs) — full domain extraction (no dedicated command enum; invoked from the existing chat/config command flows). Keeps the request/response types, the `ProviderAdapter` trait and the `provider_*` dispatch, the `LocalProvider` and `RemoteProvider` implementations, local-service selection (`select_local_chat_service`, `ready_local_services`), the remote model and endpoint resolution, the `ureq` remote transport and the `SseBytesEmitter` trait. The ROCm tool schema, the local HTTP/SSE transport and the provider wire formats live in child modules under `providers/`, below; `providers.rs` imports the request-body builders, response parsers and SSE emitters it calls from `providers/wire.rs`. - [`providers/tool_schema.rs`](../apps/rocm/src/providers/tool_schema.rs) — full domain extraction, a leaf child of `providers.rs`: the ROCm tool catalog a provider advertises to a model, as OpenAI function definitions (`rocm_openai_tool_definitions`) and reshaped for Anthropic's `tools` field (`rocm_anthropic_tool_definitions`). Both are `pub(super)`, read by the request-body builders in `providers/wire.rs`; `rocm_openai_tool`, the one-tool constructor behind them, is private. Depends on nothing else in `providers`. @@ -82,14 +87,14 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) — mechanical relocation, owns types (Phase 6a): `serve()`, plus the engine selection that runs ahead of it (`select_serve_engine`, `recipe_preferred_engine`, `serve_model_ref_for_engine`) and the AIM pre-flight that runs inside it: the Hugging Face token check, the GPU-family profile check and pin, the multi-GPU refusal (`aim_gpu_preflight`), and the board note. The engine-recipe overrides `serve()` applies to the launch hint live in [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs); this module imports them by name. It also holds `collect_serve_notes`, the GPU/device warnings folded into the interactive summary, which was relocated here from `main.rs` with its two tests. Its tests cover engine selection, the AIM pre-flight and the reuse pre-gate; the AIM tests build their hint with `protocol_engine_recipe_hint` and `engine_recipe_with_extra_env` from `serve_recipe.rs`, which `serve()` also calls. - [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs) — mechanical relocation, owns a type (split out of [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which imports the entry points by name): the pure transforms `serve()` applies to the selected engine's launch recipe hint. `protocol_engine_recipe_hint` maps the catalog's engine record onto the protocol hint, and `render_serve_engine_recipe_lines` prints it for the serve plan. The explicit controls are then layered on top: `--tool-call-parser` (`engine_recipe_with_tool_call_override`), `--gpu-memory-utilization` (`parse_gpu_memory_utilization`, `engine_recipe_with_gpu_memory_utilization_override`), `--temperature`/`--top-p`/`--max-tokens` (`ServeGenerationDefaults`, `engine_recipe_with_generation_defaults`) and `--engine-env` (`engine_recipe_with_extra_env`, `engine_env_ignored_note`, with the env names it refuses to override in `RESERVED_ENGINE_ENV_KEYS`); `engine_recipe_enables_tool_choice` reads the result back. The per-flag rewriters (`set_vllm_tool_call_parser`, `set_vllm_gpu_memory_utilization`, `set_vllm_override_generation_config`, `set_lemonade_generation_defaults`) are private. The entry points above, and `ServeGenerationDefaults`, are `pub(crate)` for `serve()` and for the AIM tests in `serve_cmd.rs`; `RESERVED_ENGINE_ENV_KEYS` is private. Among first-party crates the module depends only on `rocm_core` and `rocm_engine_protocol`, and nothing in it calls back into `serve_cmd.rs`. Its tests cover each override on its own. - [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) — full domain extraction. -- [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`terminate_recorded_service_pids`, the `MANAGED_STOP_GRACE` constant, the `RecordedPidOutcome` and `RecordedStopTermination` types, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. [`service_launch.rs`](../apps/rocm/src/service_launch.rs) and this module do import each other (this module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from it, and `service_launch.rs` takes `run_internal_sandbox_tool` from here), the pairing `services_text.rs` had with `service_launch.rs` before the split; `services_text.rs` still reaches `background_helper_note` in `service_launch.rs`, so that pair remains too, and breaking either is a structure change rather than a move. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. +- [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`terminate_recorded_service_pids`, the `MANAGED_STOP_GRACE` constant, the `RecordedPidOutcome` and `RecordedStopTermination` types, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which passes it to [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `run_attached_service` as the stop for a Ctrl-C, the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. This module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which does not import it back. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. - [`serve_port.rs`](../apps/rocm/src/serve_port.rs) — full domain extraction: the central `rocm serve` port policy (automatic vs explicit request, reservation by live managed services, OS preflight lease), used by [`service_launch.rs`](../apps/rocm/src/service_launch.rs); the CLI restart path shares the `rocm-core` lease primitives (`lease_loopback_port_or_bind_error`, `is_canonical_loopback_host`) rather than this module. - [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. -- [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, owns types (Phase 6b, ROCMAI-91): spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind — `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are its private types; `ManagedLaunchReport` is `pub(crate)`, since [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads its fields directly for the interactive summary. Most of this module's functions are `pub(crate)` for the same reason — other modules, including [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs), call into it; `main.rs` does not. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path, out of scope for Phase 6b. +- [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, owns types (Phase 6b, ROCMAI-91): spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind — `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are its private types; `ManagedLaunchReport` is `pub(crate)`, since [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads its fields directly for the interactive summary. Most of this module's functions are `pub(crate)` for the same reason — other modules, including [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs), call into it; `main.rs` does not. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path, out of scope for Phase 6b. This module does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it: `run_attached_service` takes the stop for a Ctrl-C as a parameter, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) passes `run_internal_sandbox_tool`. - [`service_records.rs`](../apps/rocm/src/service_records.rs) — mechanical relocation, owns types: removing one managed-service record and every file it owns (`remove_managed_service_record`, `service_record_artifacts`) and the `rocm services prune` plan/render/apply steps (`build_service_prune_plan`, `render_service_prune_plan`, `apply_service_prune_plan`, `prune_managed_service_records`) with the age gate behind them (`DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS`, `service_prune_min_age_hours`). It reads record state through [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) and [`services_text.rs`](../apps/rocm/src/services_text.rs) and builds its advice text with [`service_hints.rs`](../apps/rocm/src/service_hints.rs); none of the three depends on it. `remove_managed_service_record`, `prune_managed_service_records`, `service_prune_min_age_hours` and `DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS` are `pub(crate)` (`services()` in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) calls the first three; the `ServicesCommand::Prune` clap definition in [`cli.rs`](../apps/rocm/src/cli.rs) uses the constant), and `ServicePruneOutcome`, the return type of `prune_managed_service_records`, is `pub(crate)` with `pub(crate)` `text`, `removed_records`, `removed_files` and `failures` fields that `services()` reads. Every other item is private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths`, `plant_service_record` and `plant_service_record_without_its_key` fixtures through `crate::tests::` (the last two widened to `pub(crate)` for that). - [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm services`' command handler (`services()`, which dispatches `list`, `logs`, `stop`, `restart`, `remove` and `prune`) and `run_approved_service_action`, the approval-gated stop/restart path. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). Both functions are `pub(crate)`: `dispatch()` calls `services()` through a re-import, and [`advised_commands.rs`](../apps/rocm/src/advised_commands.rs)'s test reaches `run_approved_service_action` through a test-only re-import in `main.rs`. The module reads from [`services_text.rs`](../apps/rocm/src/services_text.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`service_hints.rs`](../apps/rocm/src/service_hints.rs) and [`cli_audit.rs`](../apps/rocm/src/cli_audit.rs); none of them calls back into it. The cluster's one regression test lives in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`. -- [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service --follow` and `rocm services logs --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_tail_lines`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_tail_lines`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat.rs`](../apps/rocm/src/chat.rs)'s deterministic tool-output summaries; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead. -- [`setup_cmd.rs`](../apps/rocm/src/setup_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm setup`'s command handler (`setup()`, which runs `setup status` by default and `setup reset`), the text `rocm setup status` prints (`render_setup_status_text`) and the reset that clears the saved first-time prompt state (`reset_setup_prompt_state`). `SetupCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `setup()` and `render_setup_status_text` are `pub(crate)` and re-imported in `main.rs`: `dispatch()` calls the first, and [`chat.rs`](../apps/rocm/src/chat.rs)'s read-only handling of `rocm setup` and `rocm setup status` reaches the second as `crate::render_setup_status_text`; `reset_setup_prompt_state` is private. The three regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime` through `crate::test_support::`. +- [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service --follow` and `rocm services logs --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_tail_lines`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_tail_lines`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead. +- [`setup_cmd.rs`](../apps/rocm/src/setup_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm setup`'s command handler (`setup()`, which runs `setup status` by default and `setup reset`), the text `rocm setup status` prints (`render_setup_status_text`) and the reset that clears the saved first-time prompt state (`reset_setup_prompt_state`). `SetupCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `setup()` and `render_setup_status_text` are `pub(crate)` and re-imported in `main.rs`: `dispatch()` calls the first, and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only handling of `rocm setup` and `rocm setup status` reaches the second as `crate::render_setup_status_text`; `reset_setup_prompt_state` is private. The three regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime` through `crate::test_support::`. - [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) — shared CLI-output component (`shell_quote`): the CLI's name for `rocm-core`'s POSIX quoter, [`posix_shell_quote`](../crates/rocm-core/src/quoting.rs). `rocm-dash-tui` keeps a copy as `ui::exec::quote_display_arg` in [`exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs), since the crate-edge check forbids it `rocm-core`; a per-character walk and a property test here hold the two to the same output. It covers the remote `ssh` command line, the chat approval card and the plan for a free-form request, the install dry-run `command:` line, the [`service_hints.rs`](../apps/rocm/src/service_hints.rs) hints, the runtime-activation report, the `rocm services logs ` lines `rocm services` lists and the `rocm comfyui install --runtime-id` advice a failed ComfyUI reinstall prints — not every `rocm …` hint the CLI prints; older ones such as `rocm logs ` still interpolate their value bare. Also holds `has_terminal_control_character`, the predicate the chat tool-call guards refuse on. - [`storage.rs`](../apps/rocm/src/storage.rs) — full domain extraction. - [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and reads the os-release every one of these plans is built from through `rocm_core::hardware_root::real_os_release`, which has no copy here. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs). `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `rocm_core::hardware_root::real_os_release` and the three OpenMPI/libatomic/libnuma install hints ignore a simulated host root, and a scan that no source file of the binary reads os-release by hand. @@ -107,11 +112,11 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/family.rs`, `therock/fetch.rs`, `therock/process.rs`, `therock/python.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for `WheelRuntimeComposition` (split out of `therock.rs`): resolving the wheel stack a request installs. `resolve_pip_runtime` (and `resolve_pip_runtime_with_timeout`, which takes the metadata download timeout and an optional source layout) resolves the family and the layout, reads the aggregate root, decides the device payload and returns a `PipRuntimeResolution`; on the canonical layout `select_matching_pip_package_versions` then takes the newest `rocm` whose `torch`, `torchvision` and `torchaudio` listings agree, and on the next layout `torchaudio_pin` pins a torchaudio first (the hardware-verified release when it is published, read from the vLLM engine's table) and `resolve_published_pip_package_versions` has `uv pip compile` choose the rest. `wheel_runtime_composition` turns a resolution into the exact package specs (`therock_pip_package_specs`) an install gives `uv` and records in the manifest, `published_pip_requirements` is the matching requirement set for `uv pip compile`, and `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` listing so the interpreter can be picked before resolution. Beneath that, `parse_aggregate_package_links` and `parse_aggregate_device_targets` read the package links of the canonical aggregate root and the exact device targets it publishes, and `validate_aggregate_index_layout` refuses a root that lacks the expected packages or any device payload; `AggregateDeviceTarget::resolve` picks the one exact target a host gets, or records why it could not (`Undetermined`); `load_simple_index_versions` fetches a package's simple-index listing through the cached metadata download and `parse_simple_index_versions` reads the versions out of it, dropping wheels the interpreter's tags do not accept when given them (`wheel_stem_matches_compatibility` and the `wheel_*_tag_matches` helpers); `package_release_key` orders the PEP 440 versions found there (`PackageReleaseKey`). `slugify`, which keys the cached listing, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The functions and types `therock.rs` and `therock/update.rs` name are `pub(in crate::therock)` and reach them through `therock/resolve.rs`; `PipRuntimeResolution` and `TheRockPipPackageVersions` are `pub(in crate::therock)` with the fields they read, and `TorchaudioPin` with its `describe`, so they can use what `resolve_pip_runtime` returns without naming the types; the rest is private. - [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` and `therock/version.rs` and reaches the registry through [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (split out of `therock.rs`): the consent gate for an install that would displace the active default runtime. `active_default_runtime_relation` says what the install would replace (an upgrade, downgrade or reinstall when it is the same family and channel, otherwise the different runtime it displaces) or `None` when nothing is the active default, and it fails closed, with a relation naming what could not be resolved, when the config names an active default the registry cannot resolve; `SdkInstallConsent` and `SdkInstallApprovalSource` say how consent is obtained (asked for, or already granted by `--yes`, `--approve-replacing-active-default` or `rocm update --apply`); `sdk_install_approval` turns those and whether a terminal is attached into proceed, prompt or refuse (`SdkInstallApproval`); `preapproved_install_line`, `fresh_install_line`, `refuse_non_interactive_message` and `confirm_overwrite_existing_sdk` are the wording of each outcome, and the prompt reads its answer through `prompt_yes_no` and the shared [`confirm.rs`](../crates/rocm-core/src/confirm.rs). `SdkInstallConsent`, `SdkInstallApprovalSource` and `preapproved_install_line` are re-exported by `therock.rs`; the other items `therock.rs` calls are `pub(super)`, as is `active_default_relation_text`, which the version property tests call, and the rest is private. - [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/io.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. -- [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK`. -- [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_chat_rocm_command_safety`, which refuses a public-bind, non-loopback or CPU `serve` and a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_chat_rocm_command_safety` here instead of through `chat`. `chat.rs` calls into it and re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). +- [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. +- [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_serve_command_safety`, the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), and `validate_chat_rocm_command_safety`, which applies them to a `serve` argv and also refuses a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_serve_command_safety` and `validate_chat_rocm_command_safety` here instead of through `chat`, and keeps no copy of the serve rules. `chat.rs`'s child modules call into it and `chat.rs` re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). - [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. -- [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) — mechanical relocation, shares crate-visible types (Phase 6f, ROCMAI-212): the uninstall-plan types (`UninstallOptions`, `UninstallPlan`, `UninstallPlanEntry`), building the plan (`build_uninstall_plan`, `plan_installed_binaries`, and the installed-binary/shared-cache discovery helpers behind them), rendering it (`render_uninstall_dry_run`, `render_uninstall_plan`), the `[y/N]` confirmation prompt (`confirm_uninstall`), and the filesystem removal primitive both [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`storage.rs`](../apps/rocm/src/storage.rs) call to carry a plan out (`remove_path`, and the entry-resolution helpers behind it). Kept separate from `uninstall.rs` rather than folded into it, since `uninstall.rs`'s own doc comment and name both describe `uninstall()`'s exclusive logic (the installer `PATH` undo and running-executable deletion), while this module's plan/render/removal helpers are shared with `storage.rs`. `main.rs` (`UninstallOptions`), `uninstall.rs`, `storage.rs`, `runtimes_cmd.rs`, `chat.rs`, and [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) (Unix-only property tests) each import the subset of these `pub(crate)` types and helpers they need; see this module's own doc comment for which symbol each one uses. -- [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types (Phase 6f, ROCMAI-210): `rocm update`'s command handler (`update()`, the body of the `update` arm of `dispatch()`) and its rendering/activation logic — the read-only status report (`render_update_text`, `append_update_surfaces`), the predicate deciding whether `update` should route into the mutating path instead of that report (`update_should_preview_or_apply`), applying a runtime update (`apply_runtime_update`, `append_update_activate_summary`), and the update-source/installed-runtime pickers behind it (`select_runtime_update_source`, `select_installed_update_runtime`). `const ROLLBACK_RECOVERY_HINT` stays in `main.rs` as a shared crate-root constant, because callers outside this module also use it ([`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimesCommand::Activate` handler and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `render_sdk_install_success`); this module reaches it via its `use crate::{...};` import, the same way `gpu_policy.rs`'s `device_policy_name` stays in `main.rs`. `update()` is `pub(crate)` and called from [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` through a re-import (`use crate::update_cmd::update;`), and calls `update_should_preview_or_apply`, `apply_runtime_update` and `render_update_text` itself; `render_update_text` is also `pub(crate)` and read from [`chat.rs`](../apps/rocm/src/chat.rs) (the deterministic `rocm update` tool summary), while `apply_runtime_update` and `update_should_preview_or_apply` keep the `pub(crate)` they had when `dispatch()` called them directly, though only `update()` calls them now; `select_runtime_update_source` is also `pub(crate)`, read from [`therock_version_proptests.rs`](../apps/rocm/src/therock_version_proptests.rs)'s property tests via `crate::update_cmd::select_runtime_update_source`. `append_update_surfaces`, `append_update_activate_summary`, and `select_installed_update_runtime` have no callers outside this module and stay private. The cluster's regression tests live in `update_cmd.rs`'s own `mod tests`, reaching shared fixtures (`test_paths`, `test_runtime_manifest_for_update`, `write_test_pip_runtime`) through `crate::tests::`/`crate::test_support::` the same way other mechanically relocated modules do. +- [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) — mechanical relocation, shares crate-visible types (Phase 6f, ROCMAI-212): the uninstall-plan types (`UninstallOptions`, `UninstallPlan`, `UninstallPlanEntry`), building the plan (`build_uninstall_plan`, `plan_installed_binaries`, and the installed-binary/shared-cache discovery helpers behind them), rendering it (`render_uninstall_dry_run`, `render_uninstall_plan`), the `[y/N]` confirmation prompt (`confirm_uninstall`), and the filesystem removal primitive both [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`storage.rs`](../apps/rocm/src/storage.rs) call to carry a plan out (`remove_path`, and the entry-resolution helpers behind it). Kept separate from `uninstall.rs` rather than folded into it, since `uninstall.rs`'s own doc comment and name both describe `uninstall()`'s exclusive logic (the installer `PATH` undo and running-executable deletion), while this module's plan/render/removal helpers are shared with `storage.rs`. `main.rs` (`UninstallOptions`), `uninstall.rs`, `storage.rs`, `runtimes_cmd.rs`, `chat/mcp_bridge.rs`, and [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) (Unix-only property tests) each import the subset of these `pub(crate)` types and helpers they need; see this module's own doc comment for which symbol each one uses. +- [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types (Phase 6f, ROCMAI-210): `rocm update`'s command handler (`update()`, the body of the `update` arm of `dispatch()`) and its rendering/activation logic — the read-only status report (`render_update_text`, `append_update_surfaces`), the predicate deciding whether `update` should route into the mutating path instead of that report (`update_should_preview_or_apply`), applying a runtime update (`apply_runtime_update`, `append_update_activate_summary`), and the update-source/installed-runtime pickers behind it (`select_runtime_update_source`, `select_installed_update_runtime`). `const ROLLBACK_RECOVERY_HINT` stays in `main.rs` as a shared crate-root constant, because callers outside this module also use it ([`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimesCommand::Activate` handler and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `render_sdk_install_success`); this module reaches it via its `use crate::{...};` import, the same way `gpu_policy.rs`'s `device_policy_name` stays in `main.rs`. `update()` is `pub(crate)` and called from [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` through a re-import (`use crate::update_cmd::update;`), and calls `update_should_preview_or_apply`, `apply_runtime_update` and `render_update_text` itself; `render_update_text` is also `pub(crate)` and read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the read-only `rocm update` handling), while `apply_runtime_update` and `update_should_preview_or_apply` keep the `pub(crate)` they had when `dispatch()` called them directly, though only `update()` calls them now; `select_runtime_update_source` is also `pub(crate)`, read from [`therock_version_proptests.rs`](../apps/rocm/src/therock_version_proptests.rs)'s property tests via `crate::update_cmd::select_runtime_update_source`. `append_update_surfaces`, `append_update_activate_summary`, and `select_installed_update_runtime` have no callers outside this module and stay private. The cluster's regression tests live in `update_cmd.rs`'s own `mod tests`, reaching shared fixtures (`test_paths`, `test_runtime_manifest_for_update`, `write_test_pip_runtime`) through `crate::tests::`/`crate::test_support::` the same way other mechanically relocated modules do. Further `apps/rocm` modules, not part of the `main.rs` split: diff --git a/docs/llm-tool-use.md b/docs/llm-tool-use.md index 97c6080daf..46fffe82ab 100644 --- a/docs/llm-tool-use.md +++ b/docs/llm-tool-use.md @@ -30,6 +30,14 @@ rocm-cli local assistants use structured tools, not shell commands. note carrying such a character is left out and counted. - CPU fallback is not a supported path. GPU-required ROCm commands must fail loudly when the ROCm GPU path is not ready. +- A `rocm serve` the model proposes through `rocm_command`, or the + provider-assisted planner proposes in its tool call, is refused unless it + asks for no public bind (`--allow-public-bind`), no CPU device, a loopback + `--host` if it names one, and managed serving (`--managed`, never + `--foreground`). One shared check applies those four rules to both, in that + order, and a refused call is neither proposed for approval nor planned; the + two refusals read alike and differ only in naming the local assistant or the + provider planner. - The built-in local assistant is fixed to `qwen` (`Qwen3-4B-Instruct-2507-GGUF`) served by Lemonade. vLLM, Lemonade and AIM (a Docker-based engine) are the general serving engines; the assistant may inspect or manage them for model diff --git a/docs/testing.md b/docs/testing.md index d1ba095afa..eb0fb8671f 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1311,10 +1311,28 @@ provider-assisted plan before dispatching it, so it never runs the plan's command and shows only the plan text; for that surface the reuse of the rendered plan is covered by the unit tests below. +A `serve` the provider proposes must also pass the four serve safety rules the +chat assistant's `rocm_command` is held to (no CPU device, no public bind, a +loopback host, managed serving), which one shared check in +[tool_policy.rs](../apps/rocm/src/tool_policy.rs) applies to both. Four more +mock-lane scenarios in the same feature file run `rocm "serve a model"` against +a provider whose plan breaks one rule each: `chat-13` +(`chat-planning-assistant-cpu-serve-refused`), `chat-14` +(`chat-planning-assistant-public-bind-serve-refused`), `chat-15` +(`chat-planning-assistant-non-local-host-serve-refused`) and `chat-16` +(`chat-planning-assistant-unmanaged-serve-refused`). Each asks the provider +exactly once, prints the CLI's own plan rather than the provider's, runs +nothing, and says why the provider's plan was not used. The two unit suites +that run one shared table of refused and accepted `serve` argvs are +`chat_refuses_each_unsafe_serve_and_proposes_nothing` and +`planner_refuses_each_unsafe_serve_and_plans_nothing`; deleting a rule from the +shared check fails both. + Focused coverage: ```bash cargo test -p rocm --bin rocm provider_planner +cargo test -p rocm --bin rocm unsafe_serve cargo test -p rocm --bin rocm freeform_execution_validation_rejects_provider_assisted_plans cargo test -p rocm --bin rocm freeform_plan_dedup_prevents_provider_assisted_divergence cargo test -p rocm --bin rocm freeform_execution_reuses_the_rendered_plan_without_rebuilding diff --git a/tests/e2e-cucumber/features/chat.feature b/tests/e2e-cucumber/features/chat.feature index 3f83c17592..f882460da1 100644 --- a/tests/e2e-cucumber/features/chat.feature +++ b/tests/e2e-cucumber/features/chat.feature @@ -150,3 +150,38 @@ Feature: Chat and endpoint detection Then the planning assistant was asked exactly once And the plan text returned is the one the planning assistant gave And the action returned is the one the planning assistant gave + + # A `serve` the planning assistant proposes is held to the same four rules as + # one the chat assistant proposes: no CPU device, no public bind, loopback host + # only, and managed serving. The CLI does not show an unsafe plan from the + # assistant, it falls back to its own plan and says why, and nothing is run. + # The message is asserted with the plan shown, which is the behavior it + # describes: the assistant's serve names a model the CLI's own plan never + # would, so that model must not appear anywhere in the output. + @id:chat-planning-assistant-cpu-serve-refused + Scenario: chat-13 - A plan that serves on the CPU is refused, not shown + Given a local planning assistant that suggests serving a model with "--device cpu --managed" + When the user asks the CLI to plan a vague request + Then the planning assistant was asked exactly once + And the CLI plans the request itself and says the planning assistant's plan "cannot request CPU execution; ROCm GPU execution is required" + + @id:chat-planning-assistant-public-bind-serve-refused + Scenario: chat-14 - A plan that binds the server publicly is refused, not shown + Given a local planning assistant that suggests serving a model with "--allow-public-bind --managed" + When the user asks the CLI to plan a vague request + Then the planning assistant was asked exactly once + And the CLI plans the request itself and says the planning assistant's plan "cannot request public network binding" + + @id:chat-planning-assistant-non-local-host-serve-refused + Scenario: chat-15 - A plan that binds a non-local host is refused, not shown + Given a local planning assistant that suggests serving a model with "--host 0.0.0.0 --managed" + When the user asks the CLI to plan a vague request + Then the planning assistant was asked exactly once + And the CLI plans the request itself and says the planning assistant's plan "cannot request non-local host `0.0.0.0`" + + @id:chat-planning-assistant-unmanaged-serve-refused + Scenario: chat-16 - A plan that serves in the foreground is refused, not shown + Given a local planning assistant that suggests serving a model with "--foreground" + When the user asks the CLI to plan a vague request + Then the planning assistant was asked exactly once + And the CLI plans the request itself and says the planning assistant's plan "must request managed serving with --managed" diff --git a/tests/e2e-cucumber/tests/e2e/planner_steps.rs b/tests/e2e-cucumber/tests/e2e/planner_steps.rs index 17ffe01299..eef456809c 100644 --- a/tests/e2e-cucumber/tests/e2e/planner_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/planner_steps.rs @@ -168,6 +168,30 @@ async fn planning_assistant_with_distinct_plans(world: &mut E2eWorld) { start_planning_assistant(world, vec![examine_plan_reply(), engines_list_plan_reply()]).await; } +/// The model the planning assistant names in an unsafe serve plan. The CLI's own +/// plan for the vague request names no model, so this appearing in the output +/// means the assistant's plan was shown. +const UNSAFE_PLAN_MODEL: &str = "sshleifer/tiny-gpt2"; + +#[given(expr = "a local planning assistant that suggests serving a model with {string}")] +async fn planning_assistant_suggests_serve_with(world: &mut E2eWorld, flags: String) { + let mut args = vec!["serve", UNSAFE_PLAN_MODEL]; + args.extend(flags.split_whitespace()); + let plan = json!({ + "intent": "serve", + "tool_call": {"tool": "rocm", "args": args} + }); + // One reply: a second planning request would be an unscripted HTTP 500. + start_planning_assistant( + world, + vec![chat_text_reply( + BUILTIN_ASSISTANT_MODEL_ID, + &plan.to_string(), + )], + ) + .await; +} + #[given( "a local chat assistant that hands a vague request to the planning assistant, which suggests a different plan each time it is asked" )] @@ -207,6 +231,15 @@ async fn user_approves_vague_request(world: &mut E2eWorld) { world.cli_rc = Some(rc); } +#[when("the user asks the CLI to plan a vague request")] +async fn user_asks_cli_to_plan_vague_request(world: &mut E2eWorld) { + // No `--yes`: the CLI prints the plan and runs nothing. + let (stdout, stderr, rc) = crate::run_rocm(world, &[VAGUE_REQUEST]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + #[when("a tool-using client asks the CLI to plan a vague request")] async fn client_asks_for_a_plan(world: &mut E2eWorld) { // The planning tool is read-only, so no `--allow-mutation`. The exit code is @@ -308,6 +341,57 @@ fn assert_plan_text_is_the_first_answer(text: &str) { ); } +#[then(expr = "the CLI plans the request itself and says the planning assistant's plan {string}")] +async fn cli_plans_itself_and_says_why(world: &mut E2eWorld, reason: String) { + let out = cli_output(world); + let rc = world.cli_rc.expect("no command was run"); + assert_eq!( + rc, + 0, + "planning must still succeed with the CLI's own plan:\n{out}\n{}", + world.cli_stderr.as_deref().unwrap_or_default() + ); + // The behavior: the assistant's serve is not the plan shown, and nothing + // ran. The plan comes from the CLI's own planner alone (no provider in its + // name), names none of the assistant's model, and has no execution section. + assert!( + !out.contains(UNSAFE_PLAN_MODEL), + "the planning assistant's unsafe plan must not be shown:\n{out}" + ); + assert!( + out.lines() + .any(|line| line.trim() == "planner: hybrid-parser-v1"), + "the plan shown must be the CLI's own:\n{out}" + ); + assert!( + !out.contains("\nexecution\n"), + "nothing may be executed:\n{out}" + ); + // The same fact, stated positively so it cannot pass vacuously if the + // execution heading ever changes shape: the plan renderer indents everything + // under its one `request plan` heading, so a run that printed anything of its + // own (an `execution` section, a launch report, an error) would add another + // unindented line. + let sections: Vec<&str> = out + .lines() + .filter(|line| !line.trim().is_empty() && !line.starts_with(char::is_whitespace)) + .collect(); + assert_eq!( + sections, + ["request plan"], + "the output must be the plan alone, with nothing run after it:\n{out}" + ); + // The message, asserted together with the behavior above: it gives the + // rule's reason and names the assistant's plan as what was refused. + let note = format!( + "Provider-assisted planner `local` could not resolve this request: provider planner {reason}" + ); + assert!( + out.lines().any(|line| line.contains(¬e)), + "the CLI must say why the assistant's plan was not used (`{note}`):\n{out}" + ); +} + #[then("the plan shown is the one the planning assistant gave")] async fn plan_shown_is_the_assistants(world: &mut E2eWorld) { assert_plan_text_is_the_first_answer(cli_output(world)); From d46589b549237ed5645f931a52c79d813edd21e3 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Fri, 6 Nov 2026 18:44:39 +0000 Subject: [PATCH 005/102] test: guard orphaned proptest seeds and the planner's chat imports, and give the JSONL store tests their own record type (#932) * test(rocm-core): give the JSONL store tests a record type of their own The tests in jsonl_store.rs drove the generic store engine through the automation audit store: they imported AuditEventRecord, append_audit_event and load_recent_audit_events and seeded files with helpers shaped around that record. That is a test-only edge from the generic engine back to one of its users, so the engine's own tests could not be read or run without its first customer. They now use a small local record (Entry, padded so a serialized line is about as long as a real store's, which the rotation caps and the torn-tail cuts are sized against) and local helpers over append_jsonl_record and load_recent_jsonl_records. automation.rs keeps its audit fixtures in test_support.rs and its own tests are unchanged. Test-only: every leaf test name and assertion is kept, and no production code changes. docs/architecture.md and docs/testing.md no longer describe the test-only edge. Signed-off-by: Jussi Elo * test(xtask): fail when the freeform planner or its leaf modules import chat chat.rs calls the freeform planner, so a planner that imported chat back would re-form a module cycle. The cycle was broken by moving what the planner used into leaf modules (prompt_text.rs, rocm_argv.rs, tool_policy.rs), but check-crate-edges compares crates, so nothing stopped a later change from putting the import back. Add a source-scan contract test in the style of the environment-mutation and ASCII-output contracts. It reads freeform_planner.rs and the three leaf modules, blanks comments and literals with the environment contract's existing stripper, drops #[cfg(test)] items by brace matching, and reports file:line for any remaining crate::chat / super::chat / self::chat path, chat member of a use crate::{..} group, or mod chat. The stripper is made pub(crate) for this; nothing else in that file changes. Tooling only: no change to the shipped binaries. Signed-off-by: Jussi Elo * test(xtask): fail on a proptest regression file no test reads proptest keys a test's saved failure cases by the source file of the proptest! block: /src/.rs reads /proptest-regressions/.txt. Moving a property test to another file therefore leaves its seed file behind and the saved counterexamples silently stop being replayed; this happened to a seed in apps/rocm during a module split. Add a source-scan contract in the style of the other xtask contract tests: it walks the tree, maps every file under a proptest-regressions directory back to the source file proptest derives it from, and fails when that file is missing or holds no proptest! block, naming the seed, the expected source, the likely cause and the fix. Fixture tests cover the mapping, a moved test, a deleted source, a stray non-seed file and a comment that merely mentions proptest!. Document the rule in docs/testing.md and in the module organization convention in docs/architecture.md: moving a property test moves its regression file with it. Signed-off-by: Jussi Elo * test(xtask): share the literal and comment stripper between the contracts `env_mutation_contract` carried a tokenizer that blanks string literals, raw strings, char literals and (nested) comments while keeping line structure, private to its own test module. The proptest regression guard needs the same answer to the same question (is this text code?), so the tokenizer moves to `xtask/src/rust_source.rs` and both contracts call it. The move is verbatim apart from the `pub(crate)` on the entry point and two comment sentences that named the one caller; its tests stay with `env_mutation_contract`, which exercises it through the scan, and `cargo test -p xtask` still runs 475 tests. Signed-off-by: Jussi Elo * test(xtask): see through trailing and block comments when looking for proptest! The guard that fails on a proptest regression file no test reads recognised a property block with a line check: a line that did not start with `//` and contained `proptest!`. So `fn main() {} // proptest! moved elsewhere`, a `/* ... */` note, or a property block disabled by wrapping it in a block comment still counted as a test that reads the seed, and the orphan went unreported. The check now runs on the text after the shared `strip_literals_and_comments` has blanked comments (nested block comments included), string literals, raw strings and char literals, so a `proptest!` only counts when it is in code. Fixtures expect `NoPropertyTest` for each of those sources (a trailing line comment, a block comment on one line, one around the whole block, a nested one, one that opens after code, a string literal, a raw string), and a second test holds the other direction: a real block after a comment, a lifetime, a char literal or a string still counts, so a seed something reads is not reported. The failure text no longer advises an unconditional `git mv`: it says to move the file when the destination has no seeds yet and to append the lines when it does. Signed-off-by: Jussi Elo * docs: carry proptest saved cases to a split's new file, do not only git mv The advice for a moved property test was an unconditional `git mv` of the regression file. That is right when every property test of the source file moves and the new file has no regression file yet. When only some move, the tests left behind still read the old file (proptest keys the saved cases by the source file and does not record which test a line belongs to), so `git mv` silently stops replaying their cases, and the orphan guard cannot see it: no file is orphaned, one is just missing. When the new file already has a regression file, `git mv` would replace or refuse it. `docs/testing.md` and `docs/architecture.md` now say: `git mv` for a complete move to a new path; for a partial split keep the original and copy the cases to the new file's regression file, appending when it exists. They also say what the guard does and does not cover, and that a mention of `proptest!` in a comment or a string literal does not count as a test. The per-module note on the `format_bytes` seed no longer says its file must be renamed whenever the test moves. Signed-off-by: Jussi Elo * test(xtask): see a raw-identifier spelling of chat in the planner import guard The guard's tokenizer split r#chat into the identifier r, a # and the identifier chat, so use crate::r#chat;, use crate::{r#chat}; and mod r#chat; named the chat module and passed. A raw identifier is now one token carrying the plain name, so every form the guard flags has a raw twin that fails it too. A raw identifier that spells a keyword (r#type) keeps its prefix, so it is never mistaken for the keyword. The fixtures cover the raw forms of a path, a group member, a nested group, a module declaration and an inline call, each expecting an offense; the look-alikes that are not the module (a raw chat under another path, a local variable, a raw identifier that merely starts with chat) stay unflagged; and a tokenizer test pins the single-token and raw-keyword behaviour. The raw-form fixtures are red without the change. Signed-off-by: Jussi Elo * test(xtask): end a cfg(test) exemption at the field, variant or statement it gates The planner import guard removed everything a #[cfg(test)] attribute covered by treating it as a gate on an item: it ran to the first semicolon or to the end of the first brace block that followed. On a struct field, an enum variant, a parameter, a tail expression, a struct-literal field or a match arm there is neither, so the exemption ran on past the closing brace of the struct or the parenthesis of the parameter list and swallowed the next function. struct State { #[cfg(test)] marker: () } followed by fn production() { crate::chat::f(); } produced no offense. A gate now covers one thing and ends with it: at a semicolon, at the comma that separates a list element from the next (not a comma inside generic arguments), at the end of the first brace block, or, without consuming it, at a closer that has no opener inside the gate, which is the closing brace of the struct, enum, block or match the element sits in or the closing parenthesis of its parameter list. Whether the gate is on a list element or on an item or statement is read from the first token after the attributes and visibility, because the commas in an item's generics and where clause are not its end. The fixtures cover a test-only field (last, with a comma, with stacked attributes and a visibility, with generic arguments, in a tuple struct), a variant (bare, tuple, struct-like), a parameter, a tail expression, a struct-literal field and a match arm, each followed by production code that must still be reported; the item and statement forms that must stay exempt despite commas in their generics; and a field called union. Eleven of the fifteen field and variant cases are red on the previous scanner. The module docs and docs/testing.md say what a gate covers and list the routes to chat that a path scan does not see (an alias, a glob import, a path attribute). Signed-off-by: Jussi Elo * test(xtask): read a cfg(test) gate inside parentheses or brackets as one list element A gate on a tuple-struct or tuple-variant field whose type starts with fn, unsafe or extern (a function pointer) was read as an item: the scanner ended it at the next semicolon or block and skipped the separating comma, so the production field after it, such as crate::chat::T, went unscanned. Decide list element versus item from the bracket enclosing the gate: inside parentheses or square brackets nothing is an item, so the exemption ends at the comma. Inside braces the first word of the gated thing still decides. Fixtures cover fn, unsafe fn, extern fn, unsafe extern fn, pub unsafe fn and for<'a> fn fields, the enum-variant form, and the cases that already held. The module docs name what stays blind. Signed-off-by: Jussi Elo * test(xtask): recognise proptest invocations written with whitespace or comments before the bang The orphaned-seed guard searched the stripped source for the contiguous text proptest!, so a valid block such as proptest ! { .. } or proptest /* note */ ! { .. } (the stripper blanks a comment to spaces) was not found and its live seed file was reported as an orphan. Match the name, the bang and an opening delimiter as separate tokens with whitespace allowed between them. The name must not be glued to a longer identifier, and a bang that opens no delimiter (proptest != x) is not an invocation, so my_proptest! no longer keeps a seed alive either. Fixtures cover the spaced, tabbed, newline, block-comment and line-comment forms and the near misses. Signed-off-by: Jussi Elo * test(xtask): skip only cfg(test) modules in the planner's chat-import guard The guard decided where a #[cfg(test)] gate on a field, variant, parameter, match arm or statement ended with its own bracket and generics tracking. For a rule over four files that was most of the file's 880 lines, and it could fail open: a comparison `<` after an identifier inside a gated list element was read as generics, so the exemption ran on to the closing bracket and hid later production elements. It now removes `#[cfg(test)] mod name { .. }` blocks (or `mod name;`) by bracket matching over the stripped text and scans everything else, so a gate on any other item exempts nothing. That can only fail toward a false alarm, and the real tree has no such use of chat. The tokenizer and the path rules are unchanged, and the fixtures that pinned each gate-end heuristic are replaced by one that shows a gate on a non-module hides nothing. Signed-off-by: Jussi Elo * docs: say which proptest seed layouts the orphan guard does not cover proptest derives a seed file from the nearest ancestor directory of the test's source file that holds a main.rs or lib.rs, not from a literal src/. That is src/ for every crate in this tree, but the docs and the guard's failure message stated the mapping as /src/.rs without saying so. Word it as the directory with main.rs or lib.rs (src/ in this tree) in the guard's module docs and failure text, docs/testing.md and docs/architecture.md, and list the two layouts the guard gets wrong or cannot see: a src/bin//main.rs crate with a property test, which would be reported as an orphan, and property tests under tests/, whose seed proptest writes next to the test file. Avoiding the false alarm for src/bin//main.rs needs a directory listing to find the main.rs, which the pure seed-to-source mapping does not have, and no crate here has that layout, so it is documented rather than handled. Signed-off-by: Jussi Elo * ci: run the proptest seed guard when a pull request moves a property test The guard that fails on an orphaned proptest seed lives in xtask, and the affected-crates selection reaches xtask only through the crates it depends on. A property test moved inside apps/rocm alone therefore met the guard only in the merge queue. cargo xtask affected now also selects xtask when a changed file is under a proptest-regressions directory, or when the diff of a changed Rust file adds or removes a line containing proptest! (git diff -G, so both ends of a move are found, including a move whose seed was forgotten). With no xtask member in the graph the answer is the whole workspace. The rust paths-filter in ci.yml gains the proptest-regressions glob, since a seed-only rename or deletion moves no Rust file and would otherwise skip every rust-gated job. Signed-off-by: Jussi Elo * test(rocm-core): cover a torn live file reaching the rotation cap, and name the generations coupling The JSONL store tests that append with no rotation policy never put a torn tail and a rotation on the same store, and the rotation tests never seed debris, so the documented order (a live file at the cap is rotated aside whole, debris and all, before its tail is inspected) had no test. The new test seeds a file at the cap that ends in torn debris, appends under a rotation policy, and asserts the rotated copy still holds the debris untouched, the live file holds only the new record, and a reader spans both. The 4 generations the tests read through mirror EVENT_LOG_ROTATED_GENERATIONS in automation.rs; the comment now says so, and why it stays a literal (the engine's tests keep no edge to automation.rs). Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 78867173bfa6c9c2c60edfe6ebbe2eebbd351cb3) Signed-off-by: juhovainio --- .github/workflows/ci.yml | 5 + crates/rocm-core/src/jsonl_store.rs | 342 ++++++----- crates/rocm-core/src/test_support.rs | 5 +- docs/architecture.md | 8 +- docs/testing.md | 103 +++- xtask/src/affected.rs | 198 ++++++- xtask/src/chat_import_contract.rs | 568 ++++++++++++++++++ xtask/src/env_mutation_contract.rs | 214 +------ xtask/src/main.rs | 4 + xtask/src/proptest_regressions_contract.rs | 642 +++++++++++++++++++++ xtask/src/rust_source.rs | 225 ++++++++ 11 files changed, 1946 insertions(+), 368 deletions(-) create mode 100644 xtask/src/chat_import_contract.rs create mode 100644 xtask/src/proptest_regressions_contract.rs create mode 100644 xtask/src/rust_source.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4c6e43e5c8..197adf9548 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -144,6 +144,11 @@ jobs: # rocm-deps compiles the pins into constants, so a pin change is a # source change even though no *.rs file moved. - 'runtime-deps.toml' + # A proptest saved-failure file is read by the seed guard in xtask + # (`cargo xtask affected` adds xtask when one changes). Renaming or + # deleting one moves no *.rs file, so without this entry a seed-only + # change would skip every rust-gated job. + - '**/proptest-regressions/**' # pinned_tool_versions_match_across_docs_and_workflows (xtask) reads # this file's hawkeye/cargo-about pins. This entry only gates # whether the rust-flagged jobs run at all; `cargo xtask affected` diff --git a/crates/rocm-core/src/jsonl_store.rs b/crates/rocm-core/src/jsonl_store.rs index 497664c844..c0b9dd0bd6 100644 --- a/crates/rocm-core/src/jsonl_store.rs +++ b/crates/rocm-core/src/jsonl_store.rs @@ -841,10 +841,68 @@ mod test_hooks { #[cfg(test)] mod tests { use super::*; - use crate::test_support::{ - audit_line, audit_messages, audit_record, seed_audit_store, temp_app_paths, - }; - use crate::{AppPaths, AuditEventRecord, append_audit_event, load_recent_audit_events}; + use crate::test_support::temp_app_paths; + use serde::Deserialize; + + /// The record these tests store. The engine is generic over its record + /// type, so its tests carry their own rather than borrow one of its users': + /// `automation.rs` covers the stores it builds on this engine. + #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] + struct Entry { + tag: String, + /// Padding, so a line is about as long as a real store's. The + /// rotation tests size their byte caps, and the torn-tail tests cut + /// their debris, against that length. + detail: String, + } + + fn entry(tag: &str) -> Entry { + Entry { + tag: tag.to_owned(), + detail: "d".repeat(110), + } + } + + /// One serialized record, without its newline. + fn entry_line(tag: &str) -> String { + serde_json::to_string(&entry(tag)).expect("serialize entry") + } + + fn entry_tags(entries: &[Entry]) -> Vec<&str> { + entries.iter().map(|entry| entry.tag.as_str()).collect() + } + + /// A throwaway root, and the path of a store inside it, whose directory + /// exists. + fn temp_store(name: &str) -> Result<(PathBuf, PathBuf)> { + let (root, _paths) = temp_app_paths(name); + let path = root.join("store").join("events.jsonl"); + fs::create_dir_all(path.parent().context("a store has a directory")?)?; + Ok((root, path)) + } + + /// Write `contents` as the store, as an out-of-band edit or a killed + /// writer would have left it. + fn seed_store(path: &Path, contents: &str) -> Result<()> { + fs::write(path, contents)?; + Ok(()) + } + + /// Append the record tagged `tag` to `path`, with no rotation. + fn append_entry(path: &Path, tag: &str) -> Result<()> { + append_jsonl_record(path, &entry(tag), None, "entry record") + } + + /// The most recent `limit` records of the store at `path`, reading up to + /// the generations a real store keeps. The 4 mirrors + /// `EVENT_LOG_ROTATED_GENERATIONS` in `automation.rs`, which the audit + /// reader these tests replaced used. It is a literal, not that constant, + /// so these tests keep no edge to `automation.rs`; if the production + /// value changes, change this one on purpose, since the tests reading + /// through it then no longer see the span a real store keeps. + fn load_recent_entries(path: &Path, limit: usize) -> Result> { + load_recent_jsonl_records(path, 4, limit, "entry") + } #[test] fn a_rotation_failure_is_reported_once_per_store() { @@ -919,18 +977,16 @@ mod tests { // it — sees the file end early; that is not an error. #[test] fn a_scan_that_finds_the_file_shorter_than_it_was_still_succeeds() -> Result<()> { - let (root, paths) = temp_app_paths("scan-shrunk"); - seed_audit_store( - &paths, - &format!("{}\n{}\n", audit_line("a"), audit_line("b")), + let (root, path) = temp_store("scan-shrunk")?; + seed_store( + &path, + &format!("{}\n{}\n", entry_line("a"), entry_line("b")), )?; - let path = paths.audit_events_path(); let file = fs::File::open(&path)?; let stale_len = file.metadata()?.len() + 40; - let recent = - scan_recent_jsonl_records::(&path, &file, stale_len, 5, "audit"); + let recent = scan_recent_jsonl_records::(&path, &file, stale_len, 5, "entry"); fs::remove_dir_all(root).ok(); - assert_eq!(audit_messages(&recent?), ["a", "b"]); + assert_eq!(entry_tags(&recent?), ["a", "b"]); Ok(()) } @@ -946,11 +1002,10 @@ mod tests { // `automation.rs`, just complete JSON rather than torn). #[test] fn load_recent_retries_past_a_shrink_mid_scan_and_succeeds() -> Result<()> { - let (root, paths) = temp_app_paths("scan-retry-after-shrink"); - let seed = format!("{}\n", audit_line("seed")); - let giant = audit_line(&"x".repeat(200_000)); - seed_audit_store(&paths, &format!("{seed}{giant}"))?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("scan-retry-after-shrink")?; + let seed = format!("{}\n", entry_line("seed")); + let giant = entry_line(&"x".repeat(200_000)); + seed_store(&path, &format!("{seed}{giant}"))?; let seed_len = seed.len() as u64; let hook_path = path.clone(); test_hooks::set_before_scan_read( @@ -964,10 +1019,10 @@ mod tests { .expect("truncate mid-scan"); }), ); - let recent = load_recent_audit_events(&paths, 5); + let recent = load_recent_entries(&path, 5); fs::remove_dir_all(root).ok(); assert_eq!( - audit_messages(&recent?), + entry_tags(&recent?), ["seed"], "a shrink strictly between two chunked reads must be retried, not failed" ); @@ -1044,21 +1099,20 @@ mod tests { )) } - /// Run the torn-tail repair on the seeded audit store with the replace and - /// truncate steps supplied by the test, appending the record `b`. - fn repair_audit_store_with( - paths: &AppPaths, + /// Run the torn-tail repair on the seeded store at `path` with the replace + /// and truncate steps supplied by the test, appending the record `b`. + fn repair_store_with( + path: &Path, replace: impl FnOnce(&Path, &[u8]) -> Result<()>, truncate: impl FnOnce(&Path, u64) -> Result<()>, ) -> Result<()> { - let path = paths.audit_events_path(); - let file = open_jsonl_for_append(&path)?; - let TailState::Torn(tail_start) = inspect_tail(&path, &file)? else { + let file = open_jsonl_for_append(path)?; + let TailState::Torn(tail_start) = inspect_tail(path, &file)? else { bail!("the seeded store has no torn tail"); }; - let mut line = audit_line("b").into_bytes(); + let mut line = entry_line("b").into_bytes(); line.push(b'\n'); - repair_torn_tail_and_append(&path, file, tail_start, &line, replace, truncate) + repair_torn_tail_and_append(path, file, tail_start, &line, replace, truncate) } // The fallback the whole-store rewrite lacked: when it cannot complete @@ -1069,15 +1123,14 @@ mod tests { // in `automation.rs` pins. #[test] fn a_torn_tail_is_cut_in_place_when_replacing_the_store_fails() -> Result<()> { - let (root, paths) = temp_app_paths("torn-tail-fallback-truncates"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("torn-tail-fallback-truncates")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; let before = file_identity(&path)?; let mut attempted = Vec::new(); - repair_audit_store_with( - &paths, + repair_store_with( + &path, |_, content| { attempted = content.to_vec(); bail!("no space left on device") @@ -1087,9 +1140,9 @@ mod tests { let after = file_identity(&path)?; let text = fs::read_to_string(&path); - let recent = load_recent_audit_events(&paths, 5); + let recent = load_recent_entries(&path, 5); fs::remove_dir_all(root).ok(); - let expected = format!("{}\n{}\n", audit_line("a"), audit_line("b")); + let expected = format!("{}\n{}\n", entry_line("a"), entry_line("b")); assert_eq!( String::from_utf8(attempted)?, expected, @@ -1097,7 +1150,7 @@ mod tests { ); assert_eq!(text?, expected); assert_eq!(before, after, "the fallback must cut the debris in place"); - assert_eq!(audit_messages(&recent?), ["a", "b"]); + assert_eq!(entry_tags(&recent?), ["a", "b"]); Ok(()) } @@ -1106,14 +1159,13 @@ mod tests { // every repair. #[test] fn a_torn_tail_is_not_truncated_when_replacing_the_store_works() -> Result<()> { - let (root, paths) = temp_app_paths("torn-tail-fallback-unused"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("torn-tail-fallback-unused")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; let before = file_identity(&path)?; - repair_audit_store_with( - &paths, + repair_store_with( + &path, |path, content| crate::write_atomic(path, content, "jsonl"), |_, _| bail!("truncate must not run when the replace succeeded"), )?; @@ -1121,24 +1173,23 @@ mod tests { let after = file_identity(&path)?; let text = fs::read_to_string(&path); fs::remove_dir_all(root).ok(); - assert_eq!(text?, format!("{}\n{}\n", audit_line("a"), audit_line("b"))); + assert_eq!(text?, format!("{}\n{}\n", entry_line("a"), entry_line("b"))); assert_ne!(before, after, "a working replace publishes a new file"); Ok(()) } #[test] fn a_torn_tail_that_can_be_neither_replaced_nor_cut_reports_both_causes() -> Result<()> { - let (root, paths) = temp_app_paths("torn-tail-fallback-both-fail"); - let torn = &audit_line("torn")[..20]; - let seeded = format!("{}\n{torn}", audit_line("a")); - seed_audit_store(&paths, &seeded)?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("torn-tail-fallback-both-fail")?; + let torn = &entry_line("torn")[..20]; + let seeded = format!("{}\n{torn}", entry_line("a")); + seed_store(&path, &seeded)?; // Layered like the real steps (`write_atomic` and // `truncate_jsonl_in_place` both wrap the OS error in a context), so // the assertion fails if either cause is formatted without its chain. - let result = repair_audit_store_with( - &paths, + let result = repair_store_with( + &path, |_, _| { Err(anyhow::anyhow!("no space left on device")) .context("failed to replace the store") @@ -1149,7 +1200,7 @@ mod tests { ); let text = fs::read_to_string(&path); - let recent = load_recent_audit_events(&paths, 5); + let recent = load_recent_entries(&path, 5); fs::remove_dir_all(root).ok(); assert_eq!( format!("{:#}", result.expect_err("both steps failed")), @@ -1164,7 +1215,7 @@ mod tests { text?, seeded, "a failed repair must leave the store as it was" ); - assert_eq!(audit_messages(&recent?), ["a"]); + assert_eq!(entry_tags(&recent?), ["a"]); Ok(()) } @@ -1173,13 +1224,12 @@ mod tests { // stub swaps the store for a directory, which makes reopening it fail. #[test] fn a_failed_append_after_the_cut_still_reports_why_the_replace_failed() -> Result<()> { - let (root, paths) = temp_app_paths("torn-tail-fallback-append-fails"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("torn-tail-fallback-append-fails")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; - let result = repair_audit_store_with( - &paths, + let result = repair_store_with( + &path, |_, _| { Err(anyhow::anyhow!("no space left on device")) .context("failed to replace the store") @@ -1208,18 +1258,17 @@ mod tests { // one-line append: no second repair, no new file. #[test] fn appends_after_a_fallback_repair_are_plain_appends() -> Result<()> { - let (root, paths) = temp_app_paths("torn-tail-fallback-then-append"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); - repair_audit_store_with( - &paths, + let (root, path) = temp_store("torn-tail-fallback-then-append")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; + repair_store_with( + &path, |_, _| bail!("no space left on device"), truncate_jsonl_in_place, )?; let repaired = file_identity(&path)?; - append_audit_event(&paths, &audit_record("c"))?; + append_entry(&path, "c")?; let after = file_identity(&path)?; let text = fs::read_to_string(&path); @@ -1228,9 +1277,9 @@ mod tests { text?, format!( "{}\n{}\n{}\n", - audit_line("a"), - audit_line("b"), - audit_line("c") + entry_line("a"), + entry_line("b"), + entry_line("c") ) ); assert_eq!(repaired, after); @@ -1250,11 +1299,10 @@ mod tests { fn append_repairs_a_torn_tail_in_a_directory_that_takes_no_new_files() -> Result<()> { use std::os::unix::fs::PermissionsExt; - let (root, paths) = temp_app_paths("torn-tail-read-only-dir"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); - let dir = path.parent().context("the audit store has a directory")?; + let (root, path) = temp_store("torn-tail-read-only-dir")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; + let dir = path.parent().context("the store has a directory")?; // Create the lock file now: the append reuses an existing one, but // could not create it inside the read-only directory. drop(FileLock::acquire(jsonl_lock_path(&path))?); @@ -1262,7 +1310,7 @@ mod tests { fs::set_permissions(dir, fs::Permissions::from_mode(0o555))?; let read_only_holds = fs::File::create(dir.join("probe")).is_err(); - let result = read_only_holds.then(|| append_audit_event(&paths, &audit_record("b"))); + let result = read_only_holds.then(|| append_entry(&path, "b")); fs::set_permissions(dir, fs::Permissions::from_mode(0o755))?; let after = file_identity(&path)?; @@ -1274,7 +1322,7 @@ mod tests { return Ok(()); }; result?; - assert_eq!(text?, format!("{}\n{}\n", audit_line("a"), audit_line("b"))); + assert_eq!(text?, format!("{}\n{}\n", entry_line("a"), entry_line("b"))); assert_eq!(before, after); Ok(()) } @@ -1288,24 +1336,23 @@ mod tests { use std::os::windows::fs::OpenOptionsExt; const FILE_SHARE_READ_WRITE: u32 = 0x1 | 0x2; - let (root, paths) = temp_app_paths("torn-tail-share-violation"); - let torn = &audit_line("torn")[..20]; - seed_audit_store(&paths, &format!("{}\n{torn}", audit_line("a")))?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("torn-tail-share-violation")?; + let torn = &entry_line("torn")[..20]; + seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; let before = file_identity(&path)?; let blocker = fs::OpenOptions::new() .read(true) .share_mode(FILE_SHARE_READ_WRITE) .open(&path)?; - let result = append_audit_event(&paths, &audit_record("b")); + let result = append_entry(&path, "b"); drop(blocker); let after = file_identity(&path)?; let text = fs::read_to_string(&path); fs::remove_dir_all(root).ok(); result?; - assert_eq!(text?, format!("{}\n{}\n", audit_line("a"), audit_line("b"))); + assert_eq!(text?, format!("{}\n{}\n", entry_line("a"), entry_line("b"))); assert_eq!(before, after); Ok(()) } @@ -1343,47 +1390,41 @@ mod tests { #[test] fn the_store_lock_is_a_sibling_of_the_store() { - let paths = AppPaths { - config_dir: PathBuf::from("c"), - data_dir: PathBuf::from("d"), - cache_dir: PathBuf::from("k"), - }; + let events = PathBuf::from("d").join("first").join("events.jsonl"); + let proposals = PathBuf::from("d").join("second").join("proposals.jsonl"); assert_eq!( - jsonl_lock_path(&paths.audit_events_path()), - paths.audit_dir().join("events.jsonl.lock") + jsonl_lock_path(&events), + PathBuf::from("d").join("first").join("events.jsonl.lock") ); assert_eq!( - jsonl_lock_path(&paths.automation_proposals_path()), - paths.automations_dir().join("proposals.jsonl.lock") + jsonl_lock_path(&proposals), + PathBuf::from("d") + .join("second") + .join("proposals.jsonl.lock") ); } #[test] fn rotated_generations_are_numbered_siblings_of_the_store() { - let path = PathBuf::from("d").join("audit").join("events.jsonl"); + let path = PathBuf::from("d").join("store").join("events.jsonl"); assert_eq!(jsonl_generation_path(&path, 0), path); assert_eq!( jsonl_generation_path(&path, 1), - PathBuf::from("d").join("audit").join("events.jsonl.1") + PathBuf::from("d").join("store").join("events.jsonl.1") ); assert_eq!( jsonl_generation_path(&path, 4), - PathBuf::from("d").join("audit").join("events.jsonl.4") + PathBuf::from("d").join("store").join("events.jsonl.4") ); } - /// Append an audit record tagged `tag` to `path` under `rotation`. + /// Append the record tagged `tag` to `path` under `rotation`. fn append_rotating(path: &Path, tag: &str, rotation: JsonlRotation) -> Result<()> { - append_jsonl_record( - path, - &audit_record(tag), - Some(rotation), - "audit event record", - ) + append_jsonl_record(path, &entry(tag), Some(rotation), "entry record") } - /// The tags in every generation of the audit-shaped store at `path`, - /// oldest first, read straight from the files. + /// The tags in every generation of the store at `path`, oldest first, read + /// straight from the files. fn tags_in_every_generation(path: &Path, generations: usize) -> Result> { let mut tags = Vec::new(); for generation in (0..=generations).rev() { @@ -1392,7 +1433,7 @@ mod tests { continue; } for line in fs::read_to_string(&generation_path)?.lines() { - tags.push(serde_json::from_str::(line)?.message); + tags.push(serde_json::from_str::(line)?.tag); } } Ok(tags) @@ -1400,18 +1441,16 @@ mod tests { fn recent_tags(path: &Path, generations: usize, limit: usize) -> Result> { Ok( - load_recent_jsonl_records::(path, generations, limit, "audit")? + load_recent_jsonl_records::(path, generations, limit, "entry")? .into_iter() - .map(|record| record.message) + .map(|record| record.tag) .collect(), ) } #[test] fn rotation_keeps_the_live_file_and_the_configured_generations() -> Result<()> { - let (root, paths) = temp_app_paths("rotation-generations"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-generations")?; // One record per file: every append after the first rotates. let rotation = JsonlRotation { max_bytes: 1, @@ -1439,10 +1478,8 @@ mod tests { // asks for; the rest must come from the rotated generations, in order. #[test] fn load_recent_spans_a_rollover() -> Result<()> { - let (root, paths) = temp_app_paths("rotation-spans"); - paths.ensure()?; - let path = paths.audit_events_path(); - let line_len = audit_line("00").len() as u64 + 1; + let (root, path) = temp_store("rotation-spans")?; + let line_len = entry_line("00").len() as u64 + 1; // Rotate once a file holds five records. let rotation = JsonlRotation { max_bytes: 5 * line_len, @@ -1473,11 +1510,9 @@ mod tests { // where `.1` was shifted to `.2` but the live file not yet renamed. #[test] fn load_recent_reads_a_store_caught_mid_rotation_in_order() -> Result<()> { - let (root, paths) = temp_app_paths("rotation-mid"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-mid")?; let write = |generation: usize, tags: &[&str]| -> Result<()> { - let text: String = tags.iter().map(|tag| audit_line(tag) + "\n").collect(); + let text: String = tags.iter().map(|tag| entry_line(tag) + "\n").collect(); fs::write(jsonl_generation_path(&path, generation), text)?; Ok(()) }; @@ -1499,11 +1534,9 @@ mod tests { // nor rename the live file over a `.1` that was never shifted away. #[test] fn a_rotation_that_fails_still_appends_the_record() -> Result<()> { - let (root, paths) = temp_app_paths("rotation-fails"); - paths.ensure()?; - let path = paths.audit_events_path(); - fs::write(&path, audit_line("a") + "\n")?; - fs::write(jsonl_generation_path(&path, 1), audit_line("old") + "\n")?; + let (root, path) = temp_store("rotation-fails")?; + fs::write(&path, entry_line("a") + "\n")?; + fs::write(jsonl_generation_path(&path, 1), entry_line("old") + "\n")?; // `.1` cannot be renamed over a non-empty directory. let blocker = jsonl_generation_path(&path, 2); fs::create_dir_all(&blocker)?; @@ -1518,12 +1551,47 @@ mod tests { let recent = recent_tags(&path, 2, 10); fs::remove_dir_all(root).ok(); appended?; - assert_eq!(live?, format!("{}\n{}\n", audit_line("a"), audit_line("b"))); - assert_eq!(first?, audit_line("old") + "\n"); + assert_eq!(live?, format!("{}\n{}\n", entry_line("a"), entry_line("b"))); + assert_eq!(first?, entry_line("old") + "\n"); assert_eq!(recent?, ["old", "a", "b"]); Ok(()) } + // A live file at the cap that also ends in torn debris is rotated aside + // whole, before the tail is inspected, so the repair never rewrites a copy + // that is about to be moved away. The torn-tail tests above append with no + // rotation policy and the rotation tests never seed debris, so this is the + // one test that has repair and rotation meet on the same store. + #[test] + fn a_torn_live_file_at_the_cap_is_rotated_whole_and_the_new_record_starts_a_clean_file() + -> Result<()> { + let (root, path) = temp_store("rotation-of-a-torn-file")?; + let torn = &entry_line("torn")[..20]; + let seeded = format!("{}\n{torn}", entry_line("a")); + seed_store(&path, &seeded)?; + let rotation = JsonlRotation { + max_bytes: seeded.len() as u64, + generations: 2, + }; + let appended = append_rotating(&path, "b", rotation); + let live = fs::read_to_string(&path); + let rotated = fs::read_to_string(jsonl_generation_path(&path, 1)); + let recent = recent_tags(&path, 2, 10); + fs::remove_dir_all(root).ok(); + appended?; + assert_eq!( + rotated?, seeded, + "the rotated copy keeps its debris untouched: it was not repaired first" + ); + assert_eq!( + live?, + entry_line("b") + "\n", + "the live file starts clean, with only the new record" + ); + assert_eq!(recent?, ["a", "b"], "readers skip the debris and span both"); + Ok(()) + } + // `rotation_failure_warning` is unit-tested on its own; this pins that // `append_jsonl_record` actually prints it, once for a store that keeps // failing to rotate. Run in a child process to read its stderr, which the @@ -1542,11 +1610,9 @@ mod tests { return Ok(()); } - let (root, paths) = temp_app_paths("rotation-warning"); - paths.ensure()?; - let path = paths.audit_events_path(); - fs::write(&path, audit_line("a") + "\n")?; - fs::write(jsonl_generation_path(&path, 1), audit_line("old") + "\n")?; + let (root, path) = temp_store("rotation-warning")?; + fs::write(&path, entry_line("a") + "\n")?; + fs::write(jsonl_generation_path(&path, 1), entry_line("old") + "\n")?; // `.1` cannot be renamed over a non-empty directory. let blocker = jsonl_generation_path(&path, 2); fs::create_dir_all(&blocker)?; @@ -1589,13 +1655,11 @@ mod tests { // live file open without delete sharing). #[test] fn a_rotation_after_one_left_part_way_drops_no_generation() -> Result<()> { - let (root, paths) = temp_app_paths("rotation-part-way"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-part-way")?; let write = |generation: usize, tag: &str| -> Result<()> { fs::write( jsonl_generation_path(&path, generation), - audit_line(tag) + "\n", + entry_line(tag) + "\n", )?; Ok(()) }; @@ -1656,9 +1720,7 @@ mod tests { fn rotation_under_concurrent_threads_loses_no_records() -> Result<()> { const WRITERS: usize = 8; const APPENDS_PER_WRITER: usize = 25; - let (root, paths) = temp_app_paths("rotation-threads"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-threads")?; let barrier = std::sync::Barrier::new(WRITERS); std::thread::scope(|scope| { let handles: Vec<_> = (0..WRITERS) @@ -1708,9 +1770,7 @@ mod tests { return Ok(()); } - let (root, paths) = temp_app_paths("rotation-processes"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-processes")?; let exe = std::env::current_exe().context("current test executable")?; let children = (0..WRITERS) .map(|writer| { @@ -1757,9 +1817,7 @@ mod tests { max_bytes: 2048, generations: 3, }; - let (root, paths) = temp_app_paths("rotation-read-race"); - paths.ensure()?; - let path = paths.audit_events_path(); + let (root, path) = temp_store("rotation-read-race")?; let done = std::sync::atomic::AtomicBool::new(false); let outcome = std::thread::scope(|scope| { let reader = scope.spawn(|| -> Result { diff --git a/crates/rocm-core/src/test_support.rs b/crates/rocm-core/src/test_support.rs index 518787ee5f..f5f2c18410 100644 --- a/crates/rocm-core/src/test_support.rs +++ b/crates/rocm-core/src/test_support.rs @@ -3,8 +3,9 @@ // SPDX-License-Identifier: MIT //! Shared fixtures for building a throwaway [`AppPaths`] under a per-test, -//! per-process directory, and for seeding the automation audit store that both -//! `automation.rs`'s and `jsonl_store.rs`'s tests drive. +//! per-process directory, and for seeding the automation audit store that +//! `automation.rs`'s tests drive. `jsonl_store.rs`'s tests use only the +//! throwaway directory: the engine is generic, so they carry their own record. //! //! Used from every module split out of this crate's former `lib.rs` god file //! so the naming/uniqueness scheme for test artifact directories stays in one diff --git a/docs/architecture.md b/docs/architecture.md index 487fe32658..9508db8415 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,6 +19,8 @@ New subcommands and subsystems default to their own file from day one — they s - **Full domain extraction** — a subsystem's domain implementation moves into its own file that owns its own types (structs/enums), not just relocated functions; that ownership is usually what distinguishes this pattern from mechanical relocation below, though the Phase 6a variant described next shows ownership alone doesn't fully tell them apart — every `apps/rocm` module is a private `mod x;` regardless of pattern, so module privacy alone doesn't tell the two apart either. Full-domain-extraction modules are typically reached via qualified paths (e.g. `comfyui::render_status(...)`); mechanically relocated modules with no owned types are reached the same way (e.g. [`endpoint_keys.rs`](../apps/rocm/src/endpoint_keys.rs), [`logging.rs`](../apps/rocm/src/logging.rs)). The exception — reached unqualified through `use crate::x::{...}` re-imports, matching their dispatch call sites' pre-extraction form — is limited to the clusters that moved as a dispatch-adjacent group; the block of `use crate::x::{...}` re-imports near the top of [`main.rs`](../apps/rocm/src/main.rs) is the current list of the modules it reaches that way (a few of them re-imported for reasons other than dispatch, as the comment above the block explains). [`service_launch.rs`](../apps/rocm/src/service_launch.rs) (as of Phase 6b, ROCMAI-91) uses the same unqualified re-import form for its cross-module callers, but it is not a dispatch-adjacent cluster in the same sense — one of its callers, `automations.rs`, is not a dispatch call site, and its pre-extraction form was a qualified `crate::` call rather than an unqualified import, so the re-import is new rather than kept. This call-site form, not type ownership or dispatch-fn placement, is what actually separates the Phase 6a variant below from full domain extraction. Where a subsystem has a dedicated clap subcommand, its command *enum* usually lives in [`cli.rs`](../apps/rocm/src/cli.rs) as part of the `Command`/`Cli` definition (e.g. `ComfyuiCommand`, `RuntimesCommand`), the way [`apps/rocmd`](../apps/rocmd/src/cli.rs) keeps its command tree in a `cli.rs` of its own — but the dispatch function, the match over that tree, does not stay in `main.rs` with it: `comfyui::run` lives in `comfyui.rs`, reached by the qualified path full domain extraction uses, while `engines()` and `runtimes()` moved out to `engines_cmd.rs`/`runtimes_cmd.rs` respectively, following the mechanical-relocation re-import form below. Only the top-level `dispatch()`, the match that sends each `Command` variant to its handler, stays in `main.rs`. Not every domain-extracted module has a dedicated command enum at all (`providers.rs` has none; it's invoked from the existing chat/config command flows). In library crates (`crates/rocm-core`) the module is `pub mod x;` plus a `pub use x::{...};` re-export, since it's part of the crate's public API. This is the default for new subsystems. Examples: [`apps/rocm/src/therock.rs`](../apps/rocm/src/therock.rs) (TheRock SDK install orchestration, with its update planning and other clusters in `therock/` child modules and no command enum of its own), [`comfyui.rs`](../apps/rocm/src/comfyui.rs) (`ComfyuiCommand`), [`providers.rs`](../apps/rocm/src/providers.rs) (no dedicated command enum); `crates/rocm-core`'s [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs)/[`examine.rs`](../crates/rocm-core/src/examine.rs). - **Mechanical relocation** — a `pub(crate) fn` moves out verbatim, with shared types/config staying at the crate root and reached via `crate::`. Used for dispatch-adjacent clusters where a minimal, easy-to-review diff matters more than full extraction. Example: [`apps/rocm/src/update_cmd.rs`](../apps/rocm/src/update_cmd.rs). Variant: when the cluster is large enough to own types (part of `main.rs`'s Phase 6a split, ROCMAI-82, `main.rs`'s Phase 6b split, ROCMAI-91, `main.rs`'s Phase 6c split, ROCMAI-89, and Phase 6e's follow-up, ROCMAI-726), those types move with it instead of staying in `main.rs` — [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s types are used nowhere else, but [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s `DriverInstallResult`/`DriverInstallError` are `pub(crate)` and read from [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs)'s `install()` (`result.output`/`result.executed`, `error.source`/`error.executed`); [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs)'s `FreeformExecution` is `pub(crate)` and read from `main.rs`'s `execute_freeform_next_action()` (`execution.action`) — it was already `pub(crate)` before the split, unlike the two functions that return/consume it, which widened from module-private; [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `ServeEngineSelection`/`ServeArgs` both stay `pub(crate)` and are used from `main.rs` (`ServeArgs`) and [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`assess_model_for_host` calls `select_serve_engine` and reads the resulting `ServeEngineSelection`'s `.engine`/`.source` fields) — and `ServeGenerationDefaults` moved out with the engine-recipe overrides to [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs), where it and its fields are `pub(crate)` because `serve()` builds it; [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are private, and its `ManagedLaunchReport` is `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) for the interactive summary; [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimeActivationResult` and `ActiveRuntimeMarker` are `pub(crate)` and read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary` and from [`storage.rs`](../apps/rocm/src/storage.rs) respectively; its sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`. The pattern is still "mechanical relocation" rather than full domain extraction because call sites keep their pre-extraction unqualified-re-import form — the exception named above — rather than switching to the qualified paths full-domain-extraction modules use. Where a subcommand's dispatch fn lives is not what makes it so: a full-domain-extraction module can host its own dispatch fn too (see `comfyui::run` above). +Moving a `proptest!` block to another file leaves its saved failure cases behind unless they are carried over: proptest keys them by the test's source file (`/src/.rs` reads `/proptest-regressions/.txt`, where `src/` is the directory holding `main.rs` or `lib.rs`) and does not warn when the file stops matching. Carry them in the same commit, and how depends on how much moved. When every property test of the file moves and the new file has no regression file yet, `git mv` the old one to the name the new file needs. When only some move, the tests left behind still read the old file, so keep it, and copy the saved cases to the new file's regression file (appending to it when it already exists, so its own cases stay). An `xtask` test fails the build on a regression file no test reads; it cannot notice a regression file that was moved away from tests that still need it, so check a split's diff for that. See [Property tests](testing.md#property-tests). + A flat size limit backs this convention: `cargo xtask check-module-size` ([`module_size.rs`](../xtask/src/module_size.rs)) reports every source file with more than 1,500 production lines, and there is one limit for every file, with no per-file ceilings, baseline or exemption. Production lines exclude test-gated items (inline `#[cfg(test)]` modules and the like), and test-only files are not source files; the exact definition is at the top of that file. Today the check is advisory: the `clippy` CI job and the `cargo-module-size` pre-commit hook print the offenders without failing, until the files over the limit have been split. A PR that would push a file past 1,500 production lines splits it in the same change (a pure move, the tests moving with the code, this document updated in the same PR); raising the limit or exempting a file is not an option for a feature PR. A change that merely edits a file that is already over the limit does not by itself have to split it: those files are split under their own tickets, and the check becomes blocking only once none is left. `too_many_lines = "allow"` in the workspace `Cargo.toml` stays a deliberate function-level choice; this gate is about files. ## Module map @@ -63,12 +65,12 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) — mechanical relocation: the launch-summary/examine text surfaces (`render_launch_summary`, `render_examine_text`, `render_examine_text_with_paths`, `examine_human_report`, `render_examine_plain_header`, `plain_status_label`) and the engine-inventory listing shared by `rocm engines` and `rocm examine` (`write_default_engine_legend`, `render_engine_inventory_text_with_paths`, `append_examine_runtime_state`, `append_examine_engine_inventory`, `append_engine_detect_summary`, `friendly_engine_runtime_kind`, `friendly_engine_detect_notes`, `friendly_engine_detect_note_fallback`, `engine_runtime_status_label`). The `DEFAULT_ENGINE_MARKER` constant is shared by both engine-inventory renderers so the two copies cannot drift apart. The `rocm model` catalog renderer is not here; it lives in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), and nothing in this file calls into it. [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs)'s `examine()`, which the `Command::Examine` arm calls, imports and calls `examine_human_report`; `main.rs`'s own `launch_default` calls `render_launch_summary`, re-imported there. `render_examine_text`, `render_examine_text_with_paths` and `render_engine_inventory_text_with_paths` are `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `render_engine_inventory_text_with_paths` is also read from [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s `render_engine_inventory_text`; `append_examine_runtime_state` is also `pub(crate)`, read only from [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s own test module, so `main.rs`'s re-import of it is itself `#[cfg(test)]` rather than carrying an otherwise-unused import into a non-test build. Every other item in this list has no caller outside this module and stays private. This module also reaches items that live at the crate root because they have other callers: `engine_inventory()` (also read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) and [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` (defined in [`model_fit.rs`](../apps/rocm/src/model_fit.rs) and re-imported in `main.rs`) — all reached via `use crate::{...};`. The regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime`/`runtime_manifest_path` through `crate::test_support::`/`crate::runtimes_cmd::`. Its tests do not call into `model_catalog_text.rs`: the test of the note the verbose `rocm model` listing prints for a rule-out engine lives beside that renderer. - [`freeform_answer.rs`](../apps/rocm/src/freeform_answer.rs) — mechanical relocation, no owned types: the answers a freeform request can get without running a command — `render_freeform_read_only_answer`, which answers a single approval-free `examine`, `comfyui status` or `comfyui logs` action, and `render_freeform_examine_answer`, the plain-language ROCm status it prints for `examine`. `run_freeform` in [`main.rs`](../apps/rocm/src/main.rs) is the only caller of the first, which is `pub(crate)` and re-imported there; the second is private. Besides `chat.rs` (`any_substring`, re-exported there from [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs)), it depends on `comfyui.rs`, `runtimes_cmd.rs` and `therock.rs`, none of which calls back into it. No unit test calls either function directly. - [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) — mechanical relocation, dispatch-adjacent, owns a `pub(crate)` type (as of Phase 6e's follow-up, ROCMAI-726; see the Phase 6a variant note above for the owned-types pattern). Holds the freeform natural-language planner's plan-*execution*/consent half that [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) below deliberately left out: `validate_freeform_execution_action()`, `prepare_freeform_execution()`, `apply_freeform_execution_consent()`, `render_freeform_execution_header()`, and the `pub(crate)` `FreeformExecution` struct that `prepare_freeform_execution` returns and `render_freeform_execution_header` consumes. `main.rs`'s `execute_freeform_next_action()` and `run_freeform()` stay in `main.rs` — both are dispatch-level, not part of deciding whether a plan's next action is safe to execute — and `execute_freeform_next_action()` calls in via the unqualified re-import `use crate::freeform_execution::{prepare_freeform_execution, render_freeform_execution_header};` (`run_freeform()` only calls `execute_freeform_next_action()`, never this module directly), matching [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s dispatch-adjacent form rather than `freeform_planner.rs`'s own qualified-path form. `prepare_freeform_execution` takes the already-built `plan` by value rather than rebuilding it from `request`/`paths`/`config` (ROCMAI-725's single-build guarantee, inherited unchanged from `main.rs`); `prepare_freeform_execution`/`render_freeform_execution_header` are `pub(crate)` for that cross-module call; `FreeformExecution` was already `pub(crate)` (read directly by `main.rs` and this module's own tests); `validate_freeform_execution_action`/`apply_freeform_execution_consent` have no caller outside this module and stay module-private, unchanged from before the move. The cluster's six regression tests moved out of `main.rs`'s shared `mod tests` into this module's own `mod tests`, inheriting the qualified `freeform_planner::` path they already used to reach `freeform_planner.rs`'s items as `main.rs`'s former tests — this module reaches in from outside using the same qualified path `main.rs` and `providers.rs`'s tests use (unlike `chat.rs`/`comfyui.rs`, which import it unqualified). -- [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) — full domain extraction (Phase 6e, ROCMAI-87). Owns the freeform natural-language planner cluster's types (`StructuredRequestPlan`, `PlannerIntent`, `PlannedToolCall`, `FreeformPlanAction`, `ProviderPlannerResponse`, `ProviderPlannerToolCall`, all module-private except `FreeformPlanAction`, `StructuredRequestPlan` and `PlannedToolCall`, which are `pub(crate)`, and `ProviderPlannerToolCall`, `pub(crate)` together with `validate_provider_planner_tool_call` and `planner_failure_note` so [`chat.rs`](../apps/rocm/src/chat.rs)'s command-preview property tests can drive the planner's guards), the deterministic plan builder (`build_freeform_plan*`), the provider-assisted path (`resolve_freeform_plan_with_provider`, `build_provider_planner_prompt`, `provider_planner_response_to_plan`, `validate_provider_planner_tool_call`), and plan rendering (`render_structured_request_plan`). [`main.rs`](../apps/rocm/src/main.rs) calls `build_freeform_plan_with_context`/`render_structured_request_plan` directly for the dispatch/review flow ([`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `render_launch_summary` → `host_default_engine` is another call into this module, unrelated to planning: bare `rocm`'s non-interactive launch summary), while [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) above calls `plan_next_action`/`format_structured_tool_call` for the execution flow and reads `FreeformPlanAction`/`StructuredRequestPlan` by type for its execution-validation functions — those live there, not here, because they decide whether a plan's next action is safe to *execute*, a dispatch-adjacent concern distinct from *building* the plan; `main.rs`'s `run_freeform` builds a plan exactly once via `build_freeform_plan_with_context` and threads that same value through `execute_freeform_next_action` to `prepare_freeform_execution` rather than rebuilding it (ROCMAI-725: building a plan can call a configured cloud `planner_provider`, which is not guaranteed deterministic, so a second independent build could silently disagree with the first). [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s `natural_language_plan` MCP tool gets the same single-build guarantee from `plan_freeform_request_with_context`, which builds once and returns both the rendered text and the next action; `chat/turn.rs` and `chat/mcp_bridge.rs` also call `format_structured_tool_call` directly, like [`comfyui.rs`](../apps/rocm/src/comfyui.rs) does, while [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs)'s `validate_serve_command_safety` holds the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), reading `serve_args_request_cpu_device` from [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) for the no-CPU-fallback rule; `validate_provider_planner_tool_call` calls it from its `serve` branch with the actor `provider planner` (`validate_chat_rocm_command_safety` calls it with `local assistant`, so the two refusals differ only in that name), and hands an `install sdk` call to `tool_policy::validate_chat_rocm_command_safety`. `host_default_engine`, `build_freeform_plan_with_context`, `render_structured_request_plan`, `plan_next_action`, `plan_freeform_request_with_context` and `provider_planner_response_to_plan` are `pub(crate)` for these reasons, as is `plan_driver_request` (read from [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs)'s test module); `format_structured_tool_call` lives in [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) and is re-exported here with `pub(crate) use`, so the callers above keep their `freeform_planner::` path, while the planner's own argv lookups call `rocm_argv`, its phrase and engine readers call [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) and its safety check calls `tool_policy.rs`, directly: the planner no longer imports `chat`, and only `chat` imports the planner (`plan_freeform_request_with_context`); `freeform_plan_next_action` and `render_freeform_plan` are test-only (`#[cfg(test)]`) and `FreeformPlanAction` was already `pub(crate)` before the split. `parse_freeform_invocation`/`should_treat_as_freeform` (deciding whether an invocation is freeform at all) stay in `main.rs`, a dispatch-level concern outside both this module and `freeform_execution.rs`. The cluster's regression tests live in `freeform_planner.rs`'s own `mod tests`; the handful of `main.rs` tests that exercise the invocation classifier/clap-suggestion-avoidance stay in `main.rs`'s shared `mod tests`, since neither is specific to this module or to `freeform_execution.rs` (and neither touches `freeform_planner::` at all), and the ROCMAI-725 regression tests proving the single-build guarantee live in `providers.rs`'s `mod tests` (reusing its mock local-provider server), reaching this module's items through the qualified `freeform_planner::` path like any other external caller. +- [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) — full domain extraction (Phase 6e, ROCMAI-87). Owns the freeform natural-language planner cluster's types (`StructuredRequestPlan`, `PlannerIntent`, `PlannedToolCall`, `FreeformPlanAction`, `ProviderPlannerResponse`, `ProviderPlannerToolCall`, all module-private except `FreeformPlanAction`, `StructuredRequestPlan` and `PlannedToolCall`, which are `pub(crate)`, and `ProviderPlannerToolCall`, `pub(crate)` together with `validate_provider_planner_tool_call` and `planner_failure_note` so [`chat.rs`](../apps/rocm/src/chat.rs)'s command-preview property tests can drive the planner's guards), the deterministic plan builder (`build_freeform_plan*`), the provider-assisted path (`resolve_freeform_plan_with_provider`, `build_provider_planner_prompt`, `provider_planner_response_to_plan`, `validate_provider_planner_tool_call`), and plan rendering (`render_structured_request_plan`). [`main.rs`](../apps/rocm/src/main.rs) calls `build_freeform_plan_with_context`/`render_structured_request_plan` directly for the dispatch/review flow ([`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs)'s `render_launch_summary` → `host_default_engine` is another call into this module, unrelated to planning: bare `rocm`'s non-interactive launch summary), while [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs) above calls `plan_next_action`/`format_structured_tool_call` for the execution flow and reads `FreeformPlanAction`/`StructuredRequestPlan` by type for its execution-validation functions — those live there, not here, because they decide whether a plan's next action is safe to *execute*, a dispatch-adjacent concern distinct from *building* the plan; `main.rs`'s `run_freeform` builds a plan exactly once via `build_freeform_plan_with_context` and threads that same value through `execute_freeform_next_action` to `prepare_freeform_execution` rather than rebuilding it (ROCMAI-725: building a plan can call a configured cloud `planner_provider`, which is not guaranteed deterministic, so a second independent build could silently disagree with the first). [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s `natural_language_plan` MCP tool gets the same single-build guarantee from `plan_freeform_request_with_context`, which builds once and returns both the rendered text and the next action; `chat/turn.rs` and `chat/mcp_bridge.rs` also call `format_structured_tool_call` directly, like [`comfyui.rs`](../apps/rocm/src/comfyui.rs) does, while [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs)'s `validate_serve_command_safety` holds the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), reading `serve_args_request_cpu_device` from [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) for the no-CPU-fallback rule; `validate_provider_planner_tool_call` calls it from its `serve` branch with the actor `provider planner` (`validate_chat_rocm_command_safety` calls it with `local assistant`, so the two refusals differ only in that name), and hands an `install sdk` call to `tool_policy::validate_chat_rocm_command_safety`. `host_default_engine`, `build_freeform_plan_with_context`, `render_structured_request_plan`, `plan_next_action`, `plan_freeform_request_with_context` and `provider_planner_response_to_plan` are `pub(crate)` for these reasons, as is `plan_driver_request` (read from [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs)'s test module); `format_structured_tool_call` lives in [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) and is re-exported here with `pub(crate) use`, so the callers above keep their `freeform_planner::` path, while the planner's own argv lookups call `rocm_argv`, its phrase and engine readers call [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) and its safety check calls `tool_policy.rs`, directly: the planner no longer imports `chat`, and only `chat` imports the planner (`plan_freeform_request_with_context`); a source-scan test in `xtask` ([`chat_import_contract.rs`](../xtask/src/chat_import_contract.rs)) fails if this file or the three leaf modules above import `chat` in non-test code; `freeform_plan_next_action` and `render_freeform_plan` are test-only (`#[cfg(test)]`) and `FreeformPlanAction` was already `pub(crate)` before the split. `parse_freeform_invocation`/`should_treat_as_freeform` (deciding whether an invocation is freeform at all) stay in `main.rs`, a dispatch-level concern outside both this module and `freeform_execution.rs`. The cluster's regression tests live in `freeform_planner.rs`'s own `mod tests`; the handful of `main.rs` tests that exercise the invocation classifier/clap-suggestion-avoidance stay in `main.rs`'s shared `mod tests`, since neither is specific to this module or to `freeform_execution.rs` (and neither touches `freeform_planner::` at all), and the ROCMAI-725 regression tests proving the single-build guarantee live in `providers.rs`'s `mod tests` (reusing its mock local-provider server), reaching this module's items through the qualified `freeform_planner::` path like any other external caller. - [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-214): `--device`/`--gpu` parsing (`parse_device_policy`, `parse_gpu_selection`, `parse_gpu_indices_arg`), the managed-launch-lock GPU selection (`select_gpu_indices_under_launch_lock`, `validate_pinned_gpu_index`, `effective_gpu_count`, `auto_select_gpu_indices`, `select_auto_gpu_index`, `busy_gpu_indices`), per-GPU VRAM telemetry (the `GpuVramUsage` type, `gpu_vram_usage`, `simulated_low_vram_usage`, `gpu_vram_usage_amd_smi`, `gpu_vram_usage_sysfs`, `read_drm_vram_usage`, `read_sysfs_u64`, `drm_device_is_amd`, `parse_gpu_vram_usage`), and the serve-plan low-VRAM warning (`gpu_low_memory_warning`, `vram_capacity_is_meaningful`, `serve_gpu_low_memory_warning`), plus the best-effort `amd-smi`-based GPU count (`detect_gpu_count`).Both `amd-smi` probes go through `rocm_core::amd_smi_json` (the `/dev/kfd` pre-flight) with this module's own `AMD_SMI_SERVE_PROBE_TIMEOUT`, whose doc records the resulting worst-case `serve` latency. `const fn device_policy_name` sits immediately after this cluster's last function in the original file but stayed in `main.rs` rather than moving — it already had callers outside the cluster (`engine_ipc.rs`, `service_launch.rs`, `serve_cmd.rs`) plus its own test. `GpuVramUsage` (and its fields), `gpu_vram_usage`, `parse_gpu_indices_arg`, `serve_gpu_low_memory_warning`, and `vram_capacity_is_meaningful` are `pub(crate)` and read from [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`classify_accelerator_memory`, `host_accelerator_memory`) [`main.rs`](../apps/rocm/src/main.rs) (the `dispatch()` `Command::EngineServeHttp` arm) and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) (`collect_serve_notes`); `detect_gpu_count`, `gpu_vram_usage`, `parse_device_policy`, `parse_gpu_selection`, `select_gpu_indices_under_launch_lock`, `serve_gpu_low_memory_warning`, and `validate_pinned_gpu_index` are also `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs); `parse_device_policy` is also read from [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs)'s `run_builtin_engine_serve_http` and, as of Phase 6f (ROCMAI-209), from [`service_control.rs`](../apps/rocm/src/service_control.rs)'s `restart_child_argv` (moved out of `main.rs` in that same phase, and since relocated from `services_text.rs`). `effective_gpu_count`, `auto_select_gpu_indices`, `select_auto_gpu_index`, `simulated_low_vram_usage`, `gpu_vram_usage_amd_smi`, `gpu_vram_usage_sysfs` (both the Linux and non-Linux `cfg` variants), `read_drm_vram_usage`, `read_sysfs_u64`, `drm_device_is_amd`, `parse_gpu_vram_usage`, `gpu_low_memory_warning`, and `busy_gpu_indices` have no callers outside this module and stay private. The cluster's regression tests live in `gpu_policy.rs`'s own `mod tests`; two small test-only fixture builders (`vram`, `host_gpu`) are duplicated rather than shared, since `serve_cmd.rs`'s own `serve_notes_pair_low_vram_with_the_vllm_utilization_hint` (testing `collect_serve_notes`) needs them too. - [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm install`'s command handler (`install()`, for the `sdk` and `driver` targets), the `--version`/`--build-date` selector parsing and display it uses (`therock_install_version_selector`, `therock_install_version_selector_display`) and `sdk_install_request`, which maps the parsed `install sdk` arguments onto a `therock::SdkInstallRequest` and moved here from [`chat.rs`](../apps/rocm/src/chat.rs) with its two tests. `InstallTarget` and `InstallFormat` live in [`cli.rs`](../apps/rocm/src/cli.rs). `install()` is `pub(crate)` and re-imported in `main.rs` for `dispatch()`; `therock_install_version_selector` is `pub(crate)` too, and `chat/mcp_bridge.rs`'s `render_install_sdk_dry_run_for_args` reaches it as `crate::therock_install_version_selector` through the same kind of re-import, which is why `sdk_install_request` lives here and not in `chat.rs`: this module does not depend on `chat.rs`. `sdk_install_request` keeps the `pub(crate)` it had, although only `install()` and the tests call it now. - [`logging.rs`](../apps/rocm/src/logging.rs) — mechanical relocation, no owned types, reached by qualified path. - [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) — mechanical relocation, owns types (Phase 6f, ROCMAI-211): managed-service liveness classification (`managed_service_is_live`, `refresh_managed_service_runtime_liveness`, `existing_live_managed_service_in`'s idempotency guard) and the dashboard-sidebar status/counts helpers derived from it (`ManagedServiceSidebarCounts`/`managed_service_sidebar_counts`/`local_server_sidebar_status`), plus the engine display-name helper `friendly_engine_label`, read only by [`chat/summaries.rs`](../apps/rocm/src/chat/summaries.rs). `ManagedServiceSidebarCounts` is `pub(crate)` to match the visibility of the `pub(crate)` functions that return/take it; `friendly_engine_label`, `existing_live_managed_service_in`, `recorded_service_pids`, `refresh_managed_service_runtime_liveness`, `local_server_sidebar_status`, `managed_service_sidebar_counts`, and `load_managed_services` are `pub(crate)` for the same reason — each has a confirmed caller outside this module. [`service_records.rs`](../apps/rocm/src/service_records.rs) uses its liveness helpers (`managed_service_is_live`, `recorded_service_pids`) for the in-use check behind service-record removal and pruning, and `load_managed_services` for prune planning, while [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) uses `load_managed_services` for the codex-bridge snapshot; [`service_launch.rs`](../apps/rocm/src/service_launch.rs) is the only caller of the `existing_live_managed_service_in` duplicate-launch guard; [`services_text.rs`](../apps/rocm/src/services_text.rs) reads the sidebar helpers; other modules, including [`chat/turn.rs`](../apps/rocm/src/chat/turn.rs), [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) and [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs), read the registry and liveness helpers. -- [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs) — mechanical relocation out of `examine_report_text.rs`: the `rocm model` (alias `models`) catalog renderer. `render_model_registry_text_with_context_and_host` prints the curated list grouped by hardware target (`model_catalog_platform_heading`, `model_catalog_header`, `model_recipe_memory_label`, `model_recipe_gpu_fit_label`), and `render_model_registry_verbose_text_with_context_and_host` prints the per-recipe diagnostics behind `rocm model --verbose` (`append_model_recipe_registry_source`, `append_model_recipe_metadata_lines`, `append_model_engine_recipe_settings_lines`, `append_model_recipe_staleness_lines`, `append_model_artifact_lines`, `append_artifact_source_policy_lines`, `artifact_source_policy_label`, `append_model_host_ram_fit_lines`, `append_model_fit_lines`, `append_manual_alternative_lines`, `manual_alternative_recommendations`, `recipe_is_manual_fit`, `append_model_engine_support_lines`, `model_registry_adapter_availability_note`, `recipe_display_ref`, `preferred_engine_action_target`, `model_registry_reason`, plus the small formatters `format_list_or_none`, `format_string_map_or_none`, `format_gib` and `once_per_call`). `format_bytes`, the human-readable size formatter, lives here too. The two renderers and `format_bytes` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs) imports them for the `Command::Model` arm, and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the verbose renderer), [`storage.rs`](../apps/rocm/src/storage.rs) and [`automations.rs`](../apps/rocm/src/automations.rs) (`format_bytes`) reach them as `crate::`. Every other item, including `model_registry_adapter_availability_note`, is private. The module has no edge to `examine_report_text.rs`; it reaches the engine-availability helpers in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs), `PlatformRuleOut` and `engine_ruled_out_by_platform_with` at the crate root, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `select_serve_engine`. The catalog's regression tests live in this module's own `mod tests`; they keep the names they had in `examine_report_text.rs`. The `format_bytes` property test's saved failure case lives in [`model_catalog_text.txt`](../apps/rocm/proptest-regressions/model_catalog_text.txt), which proptest finds by the test's source file, so that file has to be renamed whenever the test moves. +- [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs) — mechanical relocation out of `examine_report_text.rs`: the `rocm model` (alias `models`) catalog renderer. `render_model_registry_text_with_context_and_host` prints the curated list grouped by hardware target (`model_catalog_platform_heading`, `model_catalog_header`, `model_recipe_memory_label`, `model_recipe_gpu_fit_label`), and `render_model_registry_verbose_text_with_context_and_host` prints the per-recipe diagnostics behind `rocm model --verbose` (`append_model_recipe_registry_source`, `append_model_recipe_metadata_lines`, `append_model_engine_recipe_settings_lines`, `append_model_recipe_staleness_lines`, `append_model_artifact_lines`, `append_artifact_source_policy_lines`, `artifact_source_policy_label`, `append_model_host_ram_fit_lines`, `append_model_fit_lines`, `append_manual_alternative_lines`, `manual_alternative_recommendations`, `recipe_is_manual_fit`, `append_model_engine_support_lines`, `model_registry_adapter_availability_note`, `recipe_display_ref`, `preferred_engine_action_target`, `model_registry_reason`, plus the small formatters `format_list_or_none`, `format_string_map_or_none`, `format_gib` and `once_per_call`). `format_bytes`, the human-readable size formatter, lives here too. The two renderers and `format_bytes` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs) imports them for the `Command::Model` arm, and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the verbose renderer), [`storage.rs`](../apps/rocm/src/storage.rs) and [`automations.rs`](../apps/rocm/src/automations.rs) (`format_bytes`) reach them as `crate::`. Every other item, including `model_registry_adapter_availability_note`, is private. The module has no edge to `examine_report_text.rs`; it reaches the engine-availability helpers in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs), `PlatformRuleOut` and `engine_ruled_out_by_platform_with` at the crate root, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `select_serve_engine`. The catalog's regression tests live in this module's own `mod tests`; they keep the names they had in `examine_report_text.rs`. The `format_bytes` property test's saved failure case lives in [`model_catalog_text.txt`](../apps/rocm/proptest-regressions/model_catalog_text.txt), which proptest finds by the test's source file, so the saved case has to follow the test when it moves (see [Property tests](testing.md#property-tests) for how, which depends on whether the whole file moves). - [`model_fit.rs`](../apps/rocm/src/model_fit.rs) — mechanical relocation, owns a `pub(crate)` type: answering whether a host can run a model for `rocm diagnose --model` (`assess_model_on_this_host`, `assess_model_with_host_paths`, `assess_model_for_host`), the accelerator-memory fact it rests on (`host_accelerator_memory`, `classify_accelerator_memory`, `examination_reports_amd_gpu`) and the platform gate that rules an engine out (`PlatformRuleOut`, `engine_ruled_out_by_platform`, `unsupported_here_for`). It is a module of its own, rather than part of the `diagnose` handler, so [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs) and [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), which read `PlatformRuleOut` and the platform gate (`engine_ruled_out_by_platform` and `engine_ruled_out_by_platform_with` respectively), do not depend on a command module. It depends on [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) (`select_serve_engine`) and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs), neither of which depends on it. `assess_model_on_this_host` (called from [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs)), `PlatformRuleOut` and `engine_ruled_out_by_platform` are `pub(crate)`, and [`main.rs`](../apps/rocm/src/main.rs) re-imports the last two so `examine_report_text.rs` reaches them as `crate::` (`model_catalog_text.rs` reaches `PlatformRuleOut` and `engine_ruled_out_by_platform_with` the same way); every other item is private. The cluster's regression tests live in this module's own `mod tests`; the one that re-runs the test binary on a helper test by its full path (`engine_ruled_out_by_platform_gates_aim_on_docker_reachability`) names the helper's new path, `model_fit::tests::docker_unreachable_subprocess_helper`. - [`prompt_text.rs`](../apps/rocm/src/prompt_text.rs) — text extraction from a natural-language request, shared by `chat/intent.rs`'s prompt-intent heuristics (`fallback_rocm_tool_call_for_prompt`, `fallback_config_tool_call_for_prompt`, `install_sdk_chat_approval_for_prompt` and the `prompt_*` predicates) and the freeform planner, in a leaf module that depends only on `therock::RuntimeVersionSelector` (`tool_policy.rs` does not use it): `any_substring`, the install-folder, TheRock build-date and TheRock version readers (`requested_install_prefix_from_prompt`, `requested_therock_build_date_from_prompt`, `requested_therock_version_from_prompt`, with `requested_bare_install_prefix_from_prompt`, `clean_requested_install_prefix`, `looks_like_user_path` and `looks_like_therock_runtime_version` private beside them) from `chat.rs`, and `infer_engine_from_request` from `freeform_planner.rs`. They moved verbatim, together with `infer_engine_from_request`'s regression test, so that `freeform_planner.rs` no longer imports `chat` for them; `chat.rs` re-exports `any_substring`, which [`freeform_answer.rs`](../apps/rocm/src/freeform_answer.rs) still reaches as `chat::any_substring`. - [`provider_keys.rs`](../apps/rocm/src/provider_keys.rs) — full domain extraction. @@ -183,7 +185,7 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`gfx_target.rs`](../crates/rocm-core/src/gfx_target.rs) — gfx-target identification and the TheRock package-family tables, split out of [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) so [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) can read the family mapping without importing the module that probes the host: `extract_first_gfx_token`, `normalize_therock_family`/`known_therock_families`, `gfx_target_has_no_rocm_build`, the AMD PCI-device-id and marketing-name tables behind `gfx_target_from_pnp_device_id`, and the parsers for the Windows display text and the kernel's gfx-target version. The module is private; `lib.rs` re-exports four of the public names (all but `gfx_target_from_pnp_device_id`) and `host_gpu.rs`, as before, re-exports all five. - [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) — (ROCMAI-85, Phase 7a of ROCMAI-27's sequencing) GPU/host hardware detection — see the module's own doc comment for what it owns. Its leaf concerns are child modules under `host_gpu/`, with `host_gpu.rs` kept as the module root: [`capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs) (running an optional tool under a timeout and the tool-on-`PATH` lookup), [`os_facts.rs`](../crates/rocm-core/src/host_gpu/os_facts.rs) (kernel version, distro name, CPU model and installed RAM), [`linux.rs`](../crates/rocm-core/src/host_gpu/linux.rs) (the sysfs, KFD and DRM probes), [`gpu_visibility.rs`](../crates/rocm-core/src/host_gpu/gpu_visibility.rs) (which AMD GPUs this process may use, from the KFD/DRM counts and the `HIP_VISIBLE_DEVICES`/`ROCR_VISIBLE_DEVICES` masks, plus `windows_amd_gpu_listing`, the native-Windows adapter bound `rocm serve --gpu ` is checked against), [`windows_inventory.rs`](../crates/rocm-core/src/host_gpu/windows_inventory.rs) (the Windows display-adapter inventory and its parsers, including the AMD-adapter count behind `windows_amd_gpu_listing`), [`wsl.rs`](../crates/rocm-core/src/host_gpu/wsl.rs) (WSL detection, the librocdxg chain and the Windows host driver as seen from the guest), [`render.rs`](../crates/rocm-core/src/host_gpu/render.rs) (the `rocm examine` text of an `ExamineSummary`, as a second `impl` block), [`diagnostics.rs`](../crates/rocm-core/src/host_gpu/diagnostics.rs) (the GPU detection diagnostics text) and [`engine_default.rs`](../crates/rocm-core/src/host_gpu/engine_default.rs) (the default serve engine, and the vLLM preference for the TheRock families that serve through it). - [`host-probing.md`](host-probing.md) — a design note, not a module: [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) and [`examine.rs`](../crates/rocm-core/src/examine.rs) are two separate host-probing stacks that still read some of the same facts twice (kernel version, CPU model, the Windows GPU query, the marketing-name table, the `PATH` lookup and the subprocess runners). It proposes which of each pair should survive, how the two differ today and the order in which they could converge; it changes no code. It is a snapshot, to be retired when the convergence tickets close. -- [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) — the JSONL store engine behind the automation stores: the sibling `.lock` file (`jsonl_lock_path`, re-exported at the crate root), single-line appends, torn-tail repair, size-based rotation (`JsonlRotation`) and the tail-first readers that span rotated generations. It is generic over the record type; [`automation.rs`](../crates/rocm-core/src/automation.rs) supplies the record types and each store's policy, and in production nothing in this module depends on it; only this module's own tests do, because they drive the engine through the audit-store records (a test-only edge). The module is private; its engine items are `pub(crate)`. +- [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) — the JSONL store engine behind the automation stores: the sibling `.lock` file (`jsonl_lock_path`, re-exported at the crate root), single-line appends, torn-tail repair, size-based rotation (`JsonlRotation`) and the tail-first readers that span rotated generations. It is generic over the record type; [`automation.rs`](../crates/rocm-core/src/automation.rs) supplies the record types and each store's policy, and nothing in this module depends on it, in production or in its own tests: those drive the engine with a small record type of their own. The module is private; its engine items are `pub(crate)`. - [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) — (ROCMAI-85, Phase 7a) the TheRock managed-runtime-environment surface — see the module's own doc comment for what it owns. - [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs) — (ROCMAI-152, Phase 7c) the curated model catalog's types, built-in catalog JSON loading, signed external recipe indexes, and artifact cache-marker bookkeeping — see the module's own doc comment for what it owns. - [`model_readiness.rs`](../crates/rocm-core/src/model_readiness.rs) — the `rocm diagnose --model` fit assessment: curated-recipe lookup against a host's measured GPU/RAM. `rocm model --verbose` renders its own fit labels in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), which uses exactly two items from this module: `curated_alternatives`, called directly, and `recipe_display_ref`, reached through a private one-line wrapper of the same name in that file. diff --git a/docs/testing.md b/docs/testing.md index eb0fb8671f..1cbf599274 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -153,6 +153,44 @@ all of which the scan reports nothing for: `cfg_attr`. Write `#[cfg(unix)]` above `#[test]` instead, which the scan arms on. +### The freeform planner never imports `chat` + +`chat.rs` calls the freeform planner, so a planner that imported `chat` back +would be a module cycle. The cycle is broken by three leaf modules +([`prompt_text.rs`](../apps/rocm/src/prompt_text.rs), +[`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) and +[`tool_policy.rs`](../apps/rocm/src/tool_policy.rs)) that the planner and `chat` +both call into. `cargo xtask check-crate-edges` compares crates and cannot see +a rule between modules of one crate, so, as with the environment rule above, a +test enforces it rather than an `xtask` subcommand: +`the_freeform_planner_and_its_leaf_modules_never_import_chat` in `xtask` +([`chat_import_contract.rs`](../xtask/src/chat_import_contract.rs)) reads +[`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs) and those three +files and **fails the build** if any of them reaches `chat` in non-test code, +naming each as `file:line`. Run it on its own with: + +```bash +cargo test -p xtask chat_import_contract +``` + +It flags `crate::chat`, `super::chat` and `self::chat` as a path, `chat` as a +member of a `use crate::{..}` group, and `mod chat`, and a raw identifier +(`r#chat`) counts as `chat`. The one thing it skips is a +`#[cfg(test)] mod name { .. }` block, so a `mod tests` may use `chat`; a +`#[cfg(test)]` on anything else (a `use`, a function, a field, an expression) +skips nothing, and a test that needs `chat` belongs in the test module. A route +that never writes the `crate::chat` path is not seen: an alias +(`use crate as root; root::chat::f()`), a glob (`use crate::*;` then +`chat::f()`) or a `#[path = ".."] mod` that points at `chat.rs`; and +`#[cfg(all(test, ..))]` is read as production code, so it can only cause a false +alarm. It does not flag the +identifiers and prose that merely contain the word, such as +`chat_cli_arg_value`. It reads the four listed files only, not the modules they +import. If one of them moves or is renamed, update `GUARDED` in the test; the +test fails on a path that no longer exists rather than dropping it from the +guard. When the planner needs something that lives in `chat.rs`, move it into +a leaf module and import it from there. + ### Tests that run a fake executable Plant a stand-in `python`, `uv`, `amd-smi`, ... with @@ -288,6 +326,63 @@ share of cases and would pass with it present on a short run. The uninstall prop knob instead, `ROCM_DELETION_PROP_CASES`, described under Uninstall And Download Removal below; it follows the same rule. +When a property fails, proptest saves the shrunk case as a `cc # shrinks +to ...` line in a file named after the source file the `proptest!` block is +written in, and replays it before any new case on every later run: a block in +`/src/.rs` uses `/proptest-regressions/.txt`, where +`src/` is the nearest ancestor directory holding a `main.rs` or `lib.rs` (it is +`src/` for every crate in this tree). Moving +the test to another file therefore leaves its saved cases behind, and proptest +does not warn: they just stop being replayed. **Carry the saved cases to the +test's new file in the same commit**; what that takes depends on how much moved: + +- When every property test of the source file moves, and the new file has no + regression file yet, `git mv` the regression file to the name the new file + needs. +- When only some of them move (a split), the tests left behind still read the + old regression file, and proptest does not record which test a saved case + belongs to: all the `proptest!` blocks of a file share one. Keep the old file + where it is and copy the `cc` lines to the new file's regression file. If that + file already exists, append to it so its own cases stay; do not overwrite it + and do not `git mv` onto it. A case copied to a test it did not come from is + only one more input for that test to run. + +This is enforced in part by a test, not by an `xtask` subcommand: +`every_proptest_regression_file_belongs_to_a_property_test_that_reads_it` in +`xtask` walks the tree and **fails the build** on a file under any +`proptest-regressions/` directory whose `/src/.rs` (`src/` being the +directory with `main.rs` or `lib.rs`) is missing or +holds no `proptest!` block in code (a mention in a comment or a string literal +does not count; whitespace or a comment between `proptest` and the `!`, as in +`proptest ! { .. }`, still does), naming the file and the source file it expected. It cannot see +the opposite mistake, a split that `git mv`d the regression file away from tests +that still need it: no file is orphaned then, one is just missing, so read the +diff of a split for it. Two more things it does not check: a crate with a +`src/bin//main.rs` that has a property test would be reported as an orphan by +mistake, because proptest then keeps its seeds in `src/bin/proptest-regressions/` +(nothing in this tree has one); and a property test under a crate's `tests/` +directory is invisible to it, since proptest cannot find a `main.rs` or `lib.rs` +above such a file and writes the seed next to the test file instead. Run the +check on its own with: + +```bash +cargo test -p xtask every_proptest_regression_file_belongs_to_a_property_test_that_reads_it +``` + +The pull-request affected-crates selection (see [CI test selection](#ci-test-selection)) +adds `xtask` when a changed file is under a `proptest-regressions` directory or +when the diff of a changed Rust file adds or removes a line containing +`proptest!`, so a property test moved inside `apps/rocm` alone still runs this +check on the pull request, whether or not its seed moved with it. The other +source-scan contracts do not have this rule; they still meet a change confined to +a crate that `xtask` does not depend on in the merge queue, which runs the whole +workspace. To prove a moved seed is read, +append a deliberately unparsable line to it, run the test with `--nocapture` +(`cargo test -p -- --nocapture`), look for proptest's +`unparsable line, ignoring` warning naming that path, and remove the line. +The check does not tell a seed that still reproduces from one that no longer +does; it only confirms that something reads the file. + ### Known-gap reproducers Most `#[ignore]` attributes in the tree gate on the environment: a GPU, a live @@ -412,7 +507,10 @@ or nothing when no crate is affected). It falls back to `--workspace` for any change that can't be confined to specific crates (the lockfile, the toolchain file, the workspace root manifest, CI config, or a doc an xtask test reads — `MANIFEST.md`, `CONTRIBUTING.md`, `docs/ci-hardware-testing.md`, -`tests/e2e-cucumber/README.md`). On `main` +`tests/e2e-cucumber/README.md`). Two further rules add `xtask`, which holds +the proptest seed guard and which no other crate depends on: a changed file under +a `proptest-regressions` directory, and a changed Rust file whose diff adds or +removes a line containing `proptest!` (found with `git diff -G`). On `main` and in the merge queue the full workspace always runs. For a full local verification, keep using `cargo test --workspace --all-targets` above — `affected` is a CI optimization, and it also narrows the local pre-push hooks (next section). @@ -1999,7 +2097,8 @@ compaction keeping every pending proposal, the newest reviewed ones, and legacy proposal ids. The rotation tests in `jsonl_store::tests` drive `append_jsonl_record`/`load_recent_jsonl_records` directly with a custom -rotation policy; two further tests in `automation::tests` go through the public +rotation policy and a record type of their own, so the engine's tests do not +depend on `automation.rs`; two further tests in `automation::tests` go through the public `append_automation_event`/`append_audit_event` and `load_recent_automation_events`/`load_recent_audit_events` wrappers instead, at the real size cap, so the wiring between them and the rotation policy diff --git a/xtask/src/affected.rs b/xtask/src/affected.rs index 850549b324..699ea6623f 100644 --- a/xtask/src/affected.rs +++ b/xtask/src/affected.rs @@ -16,11 +16,17 @@ //! Skipping a test that should have run is the failure mode this guards //! against, so when in doubt it runs everything. //! +//! One rule adds a crate nothing depends on: a changed `proptest-regressions` +//! file, or a Rust file whose diff adds or removes a `proptest!` line, selects +//! `xtask`, which holds the seed guard that reads the whole tree (see +//! `PROPTEST_GUARD_CRATE`). +//! //! As with [`crate::verify_commits`], the decision logic ([`select`], //! [`owning_crate`], [`reverse_closure`]) is pure and unit-tested without git or //! cargo; the I/O lives at the edges ([`load_graph`], [`changed_files`]). use std::collections::{BTreeMap, BTreeSet}; +use std::path::Path; use std::process::Command; use anyhow::{Context, Result, bail}; @@ -149,6 +155,21 @@ fn forces_full_workspace(file: &str) -> bool { || file.starts_with(".github/workflows/") } +/// The workspace member whose tests include the proptest seed guard +/// (`every_proptest_regression_file_belongs_to_a_property_test_that_reads_it` +/// in `proptest_regressions_contract.rs`). That guard reads the whole tree, so +/// a change in another crate can fail it: moving a `proptest!` block inside +/// `apps/rocm` leaves its seed behind in `apps/rocm/proptest-regressions/`, and +/// nothing in the reverse-dependency closure of `apps/rocm` runs the guard. +const PROPTEST_GUARD_CRATE: &str = "xtask"; + +/// True for a file under a `proptest-regressions` directory: the saved-failure +/// files the guard above checks. +fn is_proptest_seed_file(file: &str) -> bool { + file.split('/') + .any(|segment| segment == "proptest-regressions") +} + /// True for changed files that cannot affect any Rust crate's build or tests, so /// they neither select a crate nor force a full run. fn is_ignorable(file: &str) -> bool { @@ -158,13 +179,32 @@ fn is_ignorable(file: &str) -> bool { is_markdown || file.starts_with("docs/") || file == "LICENSE" || file.starts_with("LICENSE.") } +/// [`select_with_proptest_sources`] for a caller that has only the changed-file +/// names: a changed file under a `proptest-regressions` directory still selects +/// the `xtask` seed guard, but a Rust file that moved a `proptest!` block is not +/// recognised without its diff. +pub fn select(changed_files: &[String], graph: &Graph) -> Selection { + select_with_proptest_sources(changed_files, &[], graph) +} + /// Pure selection logic: turn the changed-file list into a [`Selection`]. /// +/// `proptest_sources` is the subset of changed Rust files whose diff adds or +/// removes a `proptest!` line ([`proptest_source_files`]). It, and any changed +/// file under a `proptest-regressions` directory, adds `xtask` to the selection +/// so the seed guard runs on the pull request instead of only in the merge +/// queue. When the graph has no `xtask` member the guard cannot be selected, so +/// the answer is the whole workspace. +/// /// Order matters — a single force-full file (e.g. `Cargo.lock`) outweighs any /// number of crate-attributable files. Files owned by a crate seed the changed /// set; ignorable files are dropped; anything left unattributed is treated as /// force-full, because an unrecognized path might affect anything. -pub fn select(changed_files: &[String], graph: &Graph) -> Selection { +pub fn select_with_proptest_sources( + changed_files: &[String], + proptest_sources: &[String], + graph: &Graph, +) -> Selection { let mut changed_crates: BTreeSet = BTreeSet::new(); let mut saw_relevant = false; @@ -172,6 +212,17 @@ pub fn select(changed_files: &[String], graph: &Graph) -> Selection { if forces_full_workspace(file) { return Selection::Workspace; } + if is_proptest_seed_file(file) || proptest_sources.contains(file) { + if !graph + .crate_dirs + .iter() + .any(|(name, _)| name == PROPTEST_GUARD_CRATE) + { + return Selection::Workspace; + } + changed_crates.insert(PROPTEST_GUARD_CRATE.to_string()); + saw_relevant = true; + } if let Some(owner) = owning_crate(file, &graph.crate_dirs) { changed_crates.insert(owner.to_string()); saw_relevant = true; @@ -348,15 +399,38 @@ pub(crate) fn scrub_git_env(cmd: &mut Command) -> &mut Command { /// [`changed_files_between`] in `dir` (the current directory when `None`). fn diff_name_only(dir: Option<&std::path::Path>, base: &str, head: &str) -> Result> { + diff_names(dir, base, head, &[], &[]) +} + +/// List the Rust files changed in `base...HEAD` whose diff adds or removes a +/// line containing `proptest!` (git's `-G` pickaxe, so a block moved out of one +/// file and into another lists both). The pattern also matches a longer name +/// such as `my_proptest!`; an extra hit only adds `xtask` to the selection. +fn proptest_source_files(dir: Option<&Path>, base: &str) -> Result> { + diff_names(dir, base, "HEAD", &["-G", "proptest!"], &["*.rs"]) +} + +/// `git diff --name-only --no-renames base...head [-- ]` in +/// `dir` (the current directory when `None`). +fn diff_names( + dir: Option<&Path>, + base: &str, + head: &str, + options: &[&str], + pathspec: &[&str], +) -> Result> { let range = format!("{base}...{head}"); let mut git = Command::new("git"); if let Some(dir) = dir { scrub_git_env(&mut git).arg("-C").arg(dir); } - let output = git - .args(["diff", "--name-only", "--no-renames", &range]) - .output() - .context("failed to run `git diff`")?; + git.args(["diff", "--name-only", "--no-renames"]) + .args(options) + .arg(&range); + if !pathspec.is_empty() { + git.arg("--").args(pathspec); + } + let output = git.output().context("failed to run `git diff`")?; if !output.status.success() { bail!( "`git diff {range}` failed: {}", @@ -376,7 +450,8 @@ pub fn run(base: Option) -> Result<()> { let base = base.unwrap_or_else(|| DEFAULT_BASE.to_string()); let graph = load_graph()?; let files = changed_files(&base)?; - let selection = select(&files, &graph); + let proptest_sources = proptest_source_files(None, &base)?; + let selection = select_with_proptest_sources(&files, &proptest_sources, &graph); println!("{}", selection.to_cargo_args()); Ok(()) } @@ -385,6 +460,10 @@ pub fn run(base: Option) -> Result<()> { mod tests { use super::*; + fn names(pkgs: &[&str]) -> BTreeSet { + pkgs.iter().map(|p| (*p).to_string()).collect() + } + fn graph() -> Graph { // A small stand-in for the real workspace: core <- protocol <- engine, // plus an independent leaf and the top app depending on everything. @@ -394,6 +473,7 @@ mod tests { ("engine".to_string(), "engines/engine".to_string()), ("leaf".to_string(), "crates/leaf".to_string()), ("app".to_string(), "apps/app".to_string()), + ("xtask".to_string(), "xtask".to_string()), ]; // forward: protocol->core, engine->{core,protocol}, app->{engine,leaf} // reverse: core->{protocol,engine}, protocol->{engine}, engine->{app}, leaf->{app} @@ -403,6 +483,7 @@ mod tests { rev_deps.insert("engine".into(), vec!["app".into()]); rev_deps.insert("leaf".into(), vec!["app".into()]); rev_deps.insert("app".into(), vec![]); + rev_deps.insert("xtask".into(), vec![]); Graph { crate_dirs, rev_deps, @@ -668,6 +749,111 @@ mod tests { assert_eq!(files, ["crates/a/src/moved.rs", "crates/b/src/moved.rs"]); } + #[test] + fn a_changed_proptest_seed_file_selects_the_xtask_guard() { + // The seed guard lives in xtask and reads every crate's seeds; `app` + // has no edge to xtask, so without this rule a seed edit in `app` + // selects only `app`. Both the seed's own crate and the guard run. + let sel = select( + &["apps/app/proptest-regressions/a/b.txt".to_string()], + &graph(), + ); + assert_eq!(sel, Selection::Packages(names(&["app", "xtask"]))); + assert_eq!(sel.to_cargo_args(), "-p app -p xtask"); + } + + #[test] + fn a_changed_proptest_source_selects_the_xtask_guard() { + // The case the guard exists for: a `proptest!` block moves between + // files inside one crate and the seed is forgotten, so no seed file is + // in the change at all; only the diff of the Rust files says so. + let files = [ + "apps/app/src/old.rs".to_string(), + "apps/app/src/new.rs".to_string(), + ]; + assert_eq!( + select_with_proptest_sources(&files, &files, &graph()), + Selection::Packages(names(&["app", "xtask"])) + ); + // The same Rust change with no `proptest!` line in its diff does not. + assert_eq!( + select_with_proptest_sources(&files, &[], &graph()), + Selection::Packages(names(&["app"])) + ); + } + + #[test] + fn a_proptest_change_without_an_xtask_member_runs_the_workspace() { + // Nothing to select the guard with, so fall back rather than skip it. + let mut g = graph(); + g.crate_dirs.retain(|(name, _)| name != "xtask"); + g.rev_deps.remove("xtask"); + let seed = ["apps/app/proptest-regressions/x.txt".to_string()]; + assert_eq!(select(&seed, &g), Selection::Workspace); + let src = ["apps/app/src/x.rs".to_string()]; + assert_eq!( + select_with_proptest_sources(&src, &src, &g), + Selection::Workspace + ); + } + + #[test] + fn only_a_proptest_regressions_directory_counts_as_a_seed_path() { + assert!(is_proptest_seed_file( + "apps/rocm/proptest-regressions/main.txt" + )); + assert!(is_proptest_seed_file( + "crates/c/proptest-regressions/ui/format.txt" + )); + // A file merely named alike, or a Rust file, is not a seed. + assert!(!is_proptest_seed_file( + "apps/rocm/proptest-regressions-notes.md" + )); + assert!(!is_proptest_seed_file( + "apps/rocm/src/proptest-regressions.rs" + )); + assert!(!is_proptest_seed_file("apps/rocm/src/main.rs")); + // Unrelated crate changes still leave xtask out. + assert_eq!( + select(&["apps/app/src/lib.rs".to_string()], &graph()), + Selection::Packages(names(&["app"])) + ); + } + + #[test] + fn proptest_source_files_lists_rust_files_whose_diff_touches_a_proptest_block() { + let dir = tempfile::tempdir().expect("tempdir"); + let repo = dir.path(); + git_in(repo, &["init", "-q", "-b", "main"]); + std::fs::write( + repo.join("moved_from.rs"), + "proptest! { fn a() {} }\nfn keep() {}\n", + ) + .unwrap(); + std::fs::write(repo.join("plain.rs"), "fn plain() {}\n").unwrap(); + std::fs::write(repo.join("notes.md"), "proptest! is mentioned\n").unwrap(); + git_in(repo, &["add", "."]); + git_in(repo, &["commit", "-q", "-m", "base"]); + git_in(repo, &["checkout", "-q", "-b", "topic"]); + // The block moves out of one Rust file and into another. A Rust file + // whose diff has no proptest line, and a Markdown note that does but + // is not Rust, change too; neither may be listed. + std::fs::write(repo.join("moved_from.rs"), "fn keep() {}\n").unwrap(); + std::fs::write(repo.join("moved_to.rs"), "proptest! { fn a() {} }\n").unwrap(); + std::fs::write(repo.join("plain.rs"), "fn plain() { let _ = 1; }\n").unwrap(); + std::fs::write(repo.join("notes.md"), "proptest! is mentioned again\n").unwrap(); + git_in(repo, &["add", "."]); + git_in(repo, &["commit", "-q", "-m", "move"]); + + let mut got = proptest_source_files(Some(repo), "main").expect("diff"); + got.sort(); + assert_eq!(got, ["moved_from.rs", "moved_to.rs"]); + // The plain name-only diff, which the selector otherwise relies on, + // sees all four files: the narrowing above is what the `-G` adds. + let all = diff_name_only(Some(repo), "main", "topic").expect("diff"); + assert_eq!(all.len(), 4, "{all:?}"); + } + #[test] fn cargo_args_rendering() { assert_eq!(Selection::Workspace.to_cargo_args(), "--workspace"); diff --git a/xtask/src/chat_import_contract.rs b/xtask/src/chat_import_contract.rs new file mode 100644 index 0000000000..b3fbc9f739 --- /dev/null +++ b/xtask/src/chat_import_contract.rs @@ -0,0 +1,568 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Static contract test: the freeform planner and the leaf modules it shares +//! with `chat.rs` must not import `chat` in production code. +//! +//! `chat.rs` calls the freeform planner (`plan_freeform_request_with_context`), +//! so a planner that also imported `chat` would be a module cycle. The cycle was +//! broken by moving what the planner used from `chat.rs` into leaf modules +//! (`prompt_text.rs`, `rocm_argv.rs`, `tool_policy.rs`) that import nothing from +//! either side. `cargo xtask check-crate-edges` cannot see that rule, because it +//! compares crates, not modules inside `apps/rocm`, so without this test a later +//! change can put the import back and nothing fails. +//! +//! The scan is on `use` paths, not on the word `chat`: modules in this crate are +//! full of identifiers such as `chat_cli_arg_value` and of prose about the chat +//! assistant, and neither is an import. What it flags in production code is +//! `chat` reached as a module: `crate::chat`, `super::chat` or `self::chat` as a +//! path (a `use`, or an inline call such as `crate::chat::f()`), `chat` as a +//! member of a `use crate::{..}` group, and `mod chat`. A raw identifier +//! (`r#chat`) is the same name and is flagged the same way. +//! +//! Test code is exempt, one way only: a `#[cfg(test)] mod name { .. }` block +//! (or `mod name;`, with any attributes and a visibility between the gate and +//! `mod`) is removed before the scan, by matching brackets over text from which +//! comments and literals have already been blanked by [`crate::rust_source`]'s +//! `strip_literals_and_comments`, so a brace inside a string cannot move where +//! the block appears to end. A `#[cfg(test)]` on anything else (a `use`, a +//! function, a field, a variant, an expression) exempts nothing: what it covers +//! is scanned like production code, which fails toward a false alarm rather +//! than a missed import. A test that needs `chat` goes in the test module. +//! +//! Known limits, stated rather than hidden: +//! +//! * Only `#[cfg(test)]` exactly is recognised as a test gate. `#[cfg(all(test, +//! ..))]` reads as production code, which fails toward a false alarm. +//! * The scan covers the files in `GUARDED`, not the modules they import. A +//! module the planner depends on that imports `chat` would re-form the cycle +//! one step away; add it to `GUARDED` if that ever needs pinning. +//! * `chat` is found only as a path segment after `crate::`, `super::` or +//! `self::`, as a member of a group opened by one of those, or as `mod chat`. +//! A route that never writes that path is not seen: an alias (`use crate as +//! root; root::chat::f()`), a glob (`use crate::*;` then `chat::f()`) or a +//! `#[path = ".."] mod` that points at `chat.rs`. None is used in these files. + +#[cfg(test)] +mod tests { + use crate::rust_source::strip_literals_and_comments; + use std::path::Path; + + /// The files that must not import `chat` in production code, relative to + /// the repository root. + /// + /// The planner itself, then the leaf modules it reaches the shared tool + /// policy, argv helpers and prompt text through. The test below fails if + /// one of these no longer exists, so a rename has to update this list + /// rather than quietly dropping the file out of the guard. + const GUARDED: &[&str] = &[ + "apps/rocm/src/freeform_planner.rs", + "apps/rocm/src/prompt_text.rs", + "apps/rocm/src/rocm_argv.rs", + "apps/rocm/src/tool_policy.rs", + ]; + + /// The reserved words, strict and future. Only these keep their `r#` in + /// [`tokenize`]; any other raw identifier is the plain name. + const KEYWORDS: &[&str] = &[ + "abstract", "as", "async", "await", "become", "box", "break", "const", "continue", "crate", + "do", "dyn", "else", "enum", "extern", "false", "final", "fn", "for", "gen", "if", "impl", + "in", "let", "loop", "macro", "match", "mod", "move", "mut", "override", "priv", "pub", + "ref", "return", "self", "Self", "static", "struct", "super", "trait", "true", "try", + "type", "typeof", "unsafe", "unsized", "use", "virtual", "where", "while", "yield", + ]; + + #[derive(Clone, Debug, PartialEq, Eq)] + enum Tok { + Ident(String), + /// `::` + PathSep, + Punct(char), + } + + #[derive(Clone, Debug)] + struct Token { + tok: Tok, + line: usize, + } + + /// A production-code import of `chat`. + #[derive(Clone, Debug, PartialEq, Eq)] + struct Offense { + line: usize, + /// What was found, for the failure message. + what: &'static str, + } + + /// Split already-stripped source into identifiers, `::` and single + /// punctuation characters, keeping the 1-based line of each. + fn tokenize(stripped: &str) -> Vec { + let chars: Vec = stripped.chars().collect(); + let mut tokens = Vec::new(); + let mut line = 1; + let mut i = 0; + while i < chars.len() { + let c = chars[i]; + if c == '\n' { + line += 1; + i += 1; + } else if c.is_whitespace() { + i += 1; + } else if c == 'r' + && chars.get(i + 1) == Some(&'#') + && chars + .get(i + 2) + .is_some_and(|next| next.is_alphabetic() || *next == '_') + { + // A raw identifier, `r#chat`: one token that names `chat`. + let start = i + 2; + i = start; + while i < chars.len() && (chars[i].is_alphanumeric() || chars[i] == '_') { + i += 1; + } + let name: String = chars[start..i].iter().collect(); + // `r#type` is a name that happens to spell a keyword, so it + // keeps its prefix and cannot be mistaken for the keyword. + let name = if KEYWORDS.contains(&name.as_str()) { + format!("r#{name}") + } else { + name + }; + tokens.push(Token { + tok: Tok::Ident(name), + line, + }); + } else if c.is_alphanumeric() || c == '_' { + let start = i; + while i < chars.len() && (chars[i].is_alphanumeric() || chars[i] == '_') { + i += 1; + } + tokens.push(Token { + tok: Tok::Ident(chars[start..i].iter().collect()), + line, + }); + } else if c == ':' && chars.get(i + 1) == Some(&':') { + tokens.push(Token { + tok: Tok::PathSep, + line, + }); + i += 2; + } else { + tokens.push(Token { + tok: Tok::Punct(c), + line, + }); + i += 1; + } + } + tokens + } + + fn is_punct(tokens: &[Token], index: usize, wanted: char) -> bool { + tokens + .get(index) + .is_some_and(|t| t.tok == Tok::Punct(wanted)) + } + + fn is_ident(tokens: &[Token], index: usize, wanted: &str) -> bool { + tokens + .get(index) + .is_some_and(|t| matches!(&t.tok, Tok::Ident(name) if name == wanted)) + } + + /// The length of a `#[cfg(test)]` attribute starting at `index`, if one does. + fn cfg_test_attribute_len(tokens: &[Token], index: usize) -> Option { + let matches = is_punct(tokens, index, '#') + && is_punct(tokens, index + 1, '[') + && is_ident(tokens, index + 2, "cfg") + && is_punct(tokens, index + 3, '(') + && is_ident(tokens, index + 4, "test") + && is_punct(tokens, index + 5, ')') + && is_punct(tokens, index + 6, ']'); + matches.then_some(7) + } + + /// The index just past the `(`, `[` or `{` at `open`, once its matching + /// closer has been consumed (the end of the input when it has none). + fn past_balanced(tokens: &[Token], open: usize) -> usize { + let mut depth = 0usize; + let mut i = open; + while i < tokens.len() { + match tokens[i].tok { + Tok::Punct('(' | '[' | '{') => depth += 1, + Tok::Punct(')' | ']' | '}') => { + depth = depth.saturating_sub(1); + if depth == 0 { + return i + 1; + } + } + _ => {} + } + i += 1; + } + tokens.len() + } + + /// The first token of what a gate covers: past any attributes stacked under + /// the gate and a `pub` or `pub(..)` visibility. + fn first_token_gated(tokens: &[Token], mut index: usize) -> usize { + loop { + if is_punct(tokens, index, '#') && is_punct(tokens, index + 1, '[') { + index = past_balanced(tokens, index + 1); + } else if is_ident(tokens, index, "pub") { + index += 1; + if is_punct(tokens, index, '(') { + index = past_balanced(tokens, index); + } + } else { + return index; + } + } + } + + /// The index just past the `#[cfg(test)] mod name { .. }` (or `mod name;`) + /// whose gate ends just before `start`, or `None` when the gate is on + /// anything else. + fn past_test_module(tokens: &[Token], start: usize) -> Option { + let module = first_token_gated(tokens, start); + if !is_ident(tokens, module, "mod") { + return None; + } + let after_name = module + 2; + if is_punct(tokens, after_name, '{') { + Some(past_balanced(tokens, after_name)) + } else if is_punct(tokens, after_name, ';') { + Some(after_name + 1) + } else { + None + } + } + + /// The tokens left once every `#[cfg(test)] mod` block is removed. A gate on + /// anything else removes nothing, so what it covers is scanned as the + /// production code it may be. + fn without_test_modules(tokens: &[Token]) -> Vec { + let mut kept = Vec::new(); + let mut i = 0; + while i < tokens.len() { + if let Some(len) = cfg_test_attribute_len(tokens, i) + && let Some(end) = past_test_module(tokens, i + len) + { + i = end; + continue; + } + kept.push(tokens[i].clone()); + i += 1; + } + kept + } + + fn is_path_root(tokens: &[Token], index: usize) -> bool { + ["crate", "super", "self"] + .iter() + .any(|root| is_ident(tokens, index, root)) + } + + /// Every place `source` reaches the `chat` module outside test code. + /// + /// Takes the raw file text, so a caller (the real-tree test, the fixtures) + /// cannot forget to strip comments and literals first. + fn chat_imports_in_production_code(source: &str) -> Vec { + let tokens = without_test_modules(&tokenize(&strip_literals_and_comments(source))); + let mut offenses = Vec::new(); + // One entry per open `{`: whether it directly follows `crate::`, + // `super::` or `self::`, so its members are children of the root. + let mut groups: Vec = Vec::new(); + for (i, token) in tokens.iter().enumerate() { + match &token.tok { + Tok::Punct('{') => { + let after_root = + i >= 2 && tokens[i - 1].tok == Tok::PathSep && is_path_root(&tokens, i - 2); + groups.push(after_root); + } + Tok::Punct('}') => { + groups.pop(); + } + Tok::Ident(name) if name == "chat" => { + let after_root_path = + i >= 2 && tokens[i - 1].tok == Tok::PathSep && is_path_root(&tokens, i - 2); + let group_member = groups.last() == Some(&true) + && (is_punct(&tokens, i - 1, '{') || is_punct(&tokens, i - 1, ',')); + let module_decl = i >= 1 && is_ident(&tokens, i - 1, "mod"); + let what = if after_root_path { + Some("a path through `crate::chat` (or `super::`/`self::`)") + } else if group_member { + Some("`chat` as a member of a `use crate::{..}` group") + } else if module_decl { + Some("a `mod chat` declaration") + } else { + None + }; + if let Some(what) = what { + offenses.push(Offense { + line: token.line, + what, + }); + } + } + _ => {} + } + } + offenses + } + + /// One failure line per offense, locating it as `file:line`. + fn report(file: &str, offenses: &[Offense]) -> Vec { + offenses + .iter() + .map(|o| format!("{file}:{}: {}", o.line, o.what)) + .collect() + } + + #[test] + fn the_freeform_planner_and_its_leaf_modules_never_import_chat() { + let root = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("xtask crate has a parent directory"); + let mut failures = Vec::new(); + for relative in GUARDED { + let source = std::fs::read_to_string(root.join(relative)).unwrap_or_else(|err| { + panic!( + "cannot read {relative}: {err}. If the module moved or was renamed, \ + update GUARDED in xtask/src/chat_import_contract.rs so it stays guarded." + ) + }); + failures.extend(report(relative, &chat_imports_in_production_code(&source))); + } + assert!( + failures.is_empty(), + "these modules import `chat` in non-test code, which closes the cycle \ + `chat` -> freeform planner -> `chat` again:\n {}\n\ + Move whatever the module needs from `chat.rs` into a leaf module \ + (`prompt_text.rs`, `rocm_argv.rs` or `tool_policy.rs`, or a new one that \ + does not import `chat`) and import it from there. `chat` may be \ + referenced from a `#[cfg(test)]` item.", + failures.join("\n ") + ); + } + + #[test] + fn the_guard_finds_an_import_in_the_real_planner_source() { + // Liveness: the fixtures below prove the scan on small strings; this + // proves it on the real file, whose test items and braces it has to + // get past before it reaches the production code. + let root = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("xtask crate has a parent directory"); + let source = std::fs::read_to_string(root.join(GUARDED[0])).expect("planner source"); + // Counted against the file's own result, so this stays a statement + // about the scanner when the real-tree test above is red. + let baseline = chat_imports_in_production_code(&source).len(); + let with_import = format!("use crate::chat;\n{source}"); + let hits = chat_imports_in_production_code(&with_import); + assert_eq!(hits.len(), baseline + 1, "{hits:?}"); + assert_eq!(hits[0].line, 1, "{hits:?}"); + let at_the_end = format!("{source}\nfn late() {{ crate::chat::f(); }}\n"); + let hits = chat_imports_in_production_code(&at_the_end); + assert_eq!( + hits.len(), + baseline + 1, + "a call appended after the file's `mod tests` is production code: {hits:?}" + ); + } + + fn lines_of(source: &str) -> Vec { + chat_imports_in_production_code(source) + .iter() + .map(|o| o.line) + .collect() + } + + #[test] + fn a_plain_use_of_chat_is_reported_with_its_line() { + let source = "use anyhow::Result;\n\nuse crate::chat;\n"; + let offenses = chat_imports_in_production_code(source); + assert_eq!(offenses.len(), 1, "{offenses:?}"); + assert_eq!( + report("apps/rocm/src/freeform_planner.rs", &offenses), + vec![ + "apps/rocm/src/freeform_planner.rs:3: \ + a path through `crate::chat` (or `super::`/`self::`)" + .to_string() + ] + ); + } + + #[test] + fn every_spelling_of_the_import_is_reported() { + for (source, line) in [ + ("use crate::chat::any_substring;\n", 1), + ("use crate::chat as c;\n", 1), + ("use super::chat::ensure_flag;\n", 1), + ("use self::chat;\n", 1), + ("use crate::{Cli, chat, providers};\n", 1), + ("use crate::{Cli, providers, chat};\n", 1), + ("use crate::{\n Cli,\n chat,\n providers,\n};\n", 3), + ("use crate::{chat::any_substring, Cli};\n", 1), + ("fn f() {\n let _ = crate::chat::f();\n}\n", 2), + ("mod chat;\n", 1), + ("#[cfg(not(test))]\nuse crate::chat;\n", 2), + // A re-export, and the shape the cycle had before the leaf modules: + // `chat` in the import group, then bare `chat::f()` calls. + ("pub use crate::chat::any_substring;\n", 1), + ("pub(crate) use crate::chat::{A, B};\n", 1), + ( + "use crate::{Cli, chat, providers};\nfn f() { chat::any_substring(); }\n", + 1, + ), + ] { + assert_eq!(lines_of(source), vec![line], "{source:?}"); + } + } + + #[test] + fn a_chat_import_inside_a_test_module_is_allowed() { + for source in [ + "fn prod() {}\n\n#[cfg(test)]\nmod tests {\n use crate::chat;\n use crate::{Cli, chat as c};\n #[test]\n fn t() {\n crate::chat::f();\n }\n}\n", + "#[cfg(test)]\n#[allow(dead_code)]\npub(crate) mod helpers {\n pub fn f() -> bool {\n crate::chat::f()\n }\n}\n", + "#[cfg(test)]\n#[path = \"planner_tests.rs\"]\nmod tests;\n", + ] { + assert!( + chat_imports_in_production_code(source).is_empty(), + "{source:?}" + ); + } + } + + #[test] + fn production_code_around_a_test_module_is_still_scanned() { + let source = "use crate::chat::A;\n\ + #[cfg(test)]\nmod tests {\n use crate::chat::B;\n}\n\ + use crate::chat::C;\n\ + #[cfg(test)]\nmod more {\n fn f() { crate::chat::D; }\n}\n\ + fn prod() {\n crate::chat::E;\n}\n"; + assert_eq!(lines_of(source), vec![1, 6, 12]); + } + + #[test] + fn a_test_module_ends_at_its_own_closing_brace_not_at_a_brace_in_a_literal() { + // The `use` between the two literals is test code and stays exempt: a + // `}` read out of the first literal would close the item before it. + let source = "#[cfg(test)]\nmod tests {\n const S: &str = \"}\";\n use crate::chat;\n const T: &str = \"{\";\n}\n\nuse crate::chat;\n"; + assert_eq!(lines_of(source), vec![8]); + } + + #[test] + fn mentions_of_chat_that_are_not_imports_are_ignored() { + for source in [ + "// use crate::chat;\n", + "/// Moved out of `crate::chat` so the planner never imports it.\n", + "/* use crate::chat; */\n", + "const NOTE: &str = \"crate::chat::f\";\n", + "use crate::rocm_argv::chat_cli_arg_value;\n", + "use crate::{providers, tool_policy};\nfn f() { chat_tool_call_is_read_only(); }\n", + "fn f() {\n let chat = 1;\n let _ = chat;\n}\n", + "use crate::providers::chat;\n", + "use crate::{providers::{chat}, Cli};\n", + "use crate::{a::{b, chat}};\n", + "fn f(s: &str) -> bool {\n s.contains(\"chat\")\n}\n", + ] { + assert!( + chat_imports_in_production_code(source).is_empty(), + "{source:?}" + ); + } + } + + #[test] + fn a_raw_identifier_spelling_of_chat_is_reported() { + // `r#chat` names the same module as `chat`, so every form above has a + // raw twin that must fail the guard too. + for (source, line) in [ + ("use crate::r#chat;\n", 1), + ("use crate::{r#chat};\n", 1), + ("use crate::{Cli, r#chat, providers};\n", 1), + ("use crate::{Cli, r#chat as c};\n", 1), + ("use crate::{\n Cli,\n r#chat,\n};\n", 3), + ("use crate::{r#chat::any_substring, Cli};\n", 1), + ("use super::r#chat::ensure_flag;\n", 1), + ("use self::r#chat;\n", 1), + ("mod r#chat;\n", 1), + ("pub mod r#chat;\n", 1), + ("fn f() {\n let _ = crate::r#chat::f();\n}\n", 2), + ("#[cfg(not(test))]\nuse crate::r#chat;\n", 2), + ] { + assert_eq!(lines_of(source), vec![line], "{source:?}"); + } + } + + #[test] + fn raw_identifiers_that_are_not_the_chat_module_are_ignored() { + for source in [ + "use crate::providers::r#chat;\n", + "use crate::r#chat_cli_arg_value;\n", + "fn f() {\n let r#chat = 1;\n let _ = r#chat;\n}\n", + "use crate::{providers::{r#chat}, Cli};\n", + "struct S {\n r#type: u8,\n}\n", + ] { + assert!( + chat_imports_in_production_code(source).is_empty(), + "{source:?}" + ); + } + } + + #[test] + fn a_raw_identifier_is_one_token_and_a_raw_keyword_stays_distinct() { + let names = |source: &str| -> Vec { + tokenize(source) + .into_iter() + .filter_map(|t| match t.tok { + Tok::Ident(name) => Some(name), + _ => None, + }) + .collect() + }; + assert_eq!(names("r#chat"), vec!["chat"]); + assert_eq!(names("crate::r#chat::f"), vec!["crate", "chat", "f"]); + // `r#type` is a field or variable called `type`, not the keyword that + // starts a type alias, so it must not read as one. + assert_eq!(names("r#type"), vec!["r#type"]); + assert_eq!(names("r#match r#fn"), vec!["r#match", "r#fn"]); + // An `r` that is only the start of an ordinary identifier is untouched. + assert_eq!(names("r rest #[x]"), vec!["r", "rest", "x"]); + } + + #[test] + fn a_test_gate_on_anything_but_a_module_exempts_nothing() { + // Only a `#[cfg(test)] mod` block is exempt. A gate on a `use`, a + // function, a field, a variant, a parameter or an expression covers + // nothing the scan skips, so it cannot hide what follows it either: + // this is the shape that let a comparison `<` inside a gated element + // swallow the production elements after it when the scan tried to find + // where each such gate ends. Every line below must still be reported. + for (source, lines) in [ + ("#[cfg(test)]\nuse crate::chat::any_substring;\n", &[2][..]), + ( + "#[cfg(test)]\nfn only_in_tests() {\n crate::chat::f();\n}\n", + &[3], + ), + ( + "struct S {\n #[cfg(test)]\n m: crate::chat::T,\n n: crate::chat::U,\n}\n", + &[3, 4], + ), + ( + "fn f() -> S {\n S {\n #[cfg(test)]\n a: x < 1,\n b: crate::chat::B,\n }\n}\n", + &[5], + ), + ( + "fn f(x: u8) {\n match x {\n #[cfg(test)]\n 0 => crate::chat::f(),\n _ => crate::chat::g(),\n }\n}\n", + &[4, 5], + ), + ("#[cfg(all(test, unix))]\nuse crate::chat;\n", &[2]), + ] { + assert_eq!(lines_of(source), lines, "{source:?}"); + } + } +} diff --git a/xtask/src/env_mutation_contract.rs b/xtask/src/env_mutation_contract.rs index f87709918a..41f2353a63 100644 --- a/xtask/src/env_mutation_contract.rs +++ b/xtask/src/env_mutation_contract.rs @@ -40,6 +40,7 @@ #[cfg(test)] mod tests { + use crate::rust_source::strip_literals_and_comments; use std::path::{Path, PathBuf}; fn repo_root() -> PathBuf { @@ -218,219 +219,6 @@ mod tests { out } - /// Replace the CONTENTS of string literals, char literals and comments with - /// spaces, preserving line structure and everything outside them. - /// - /// The scan below counts braces and looks for call text, and both are - /// wrong if a literal can contribute either. Two real files in this repo - /// break the naive version: - /// - /// * `engines/lemonade/src/lib.rs` contains - /// `ensure_cached_archive("test://archive", ..., |_, destination| {`. - /// Truncating the line at `//` throws away the closure's opening brace, - /// so the enclosing test module reads as closed hundreds of lines early - /// and every mutation after it becomes invisible. - /// * `tests/e2e-cucumber/src/mock_server.rs` contains format strings such - /// as `vllm:num_requests_running{{model=...}}`. Those braces inflate the - /// depth, which drags PRODUCTION code into the scanned region and makes - /// the guard fail a build for a mutation it should not be watching. - /// - /// Both directions are silent, which is the one failure mode a permanent - /// guard must not have, so literals are tokenized rather than approximated. - fn strip_literals_and_comments(text: &str) -> String { - #[derive(Clone, Copy)] - enum State { - Code, - LineComment, - BlockComment(usize), - Str, - RawStr(usize), - Char, - } - - let mut out = String::with_capacity(text.len()); - let mut state = State::Code; - let chars: Vec = text.chars().collect(); - let mut i = 0; - while i < chars.len() { - let c = chars[i]; - let next = chars.get(i + 1).copied(); - match state { - State::Code => { - // A raw string opener: `r"`, `r#"`, `br##"` and so on. Checked - // before the plain-string case so the hashes are counted. - if (c == 'r' || c == 'b') - && let Some((hashes, consumed)) = raw_string_opener(&chars, i) - { - out.extend(std::iter::repeat_n(' ', consumed)); - state = State::RawStr(hashes); - i += consumed; - continue; - } - if c == '/' && next == Some('/') { - state = State::LineComment; - out.push_str(" "); - i += 2; - continue; - } - if c == '/' && next == Some('*') { - state = State::BlockComment(1); - out.push_str(" "); - i += 2; - continue; - } - if c == '"' { - state = State::Str; - out.push(' '); - i += 1; - continue; - } - // A lifetime (`'a`) is not a char literal. A char literal is - // `'x'` or `'\n'`, so require a closing quote nearby. - if c == '\'' && is_char_literal(&chars, i) { - state = State::Char; - out.push(' '); - i += 1; - continue; - } - out.push(c); - i += 1; - } - State::LineComment => { - if c == '\n' { - state = State::Code; - out.push('\n'); - } else { - out.push(' '); - } - i += 1; - } - State::BlockComment(depth) => { - if c == '*' && next == Some('/') { - state = if depth == 1 { - State::Code - } else { - State::BlockComment(depth - 1) - }; - out.push_str(" "); - i += 2; - continue; - } - if c == '/' && next == Some('*') { - state = State::BlockComment(depth + 1); - out.push_str(" "); - i += 2; - continue; - } - out.push(if c == '\n' { '\n' } else { ' ' }); - i += 1; - } - State::Str => { - if c == '\\' { - // The escaped character cannot close the literal, so - // it is consumed here. When it is a NEWLINE -- a line - // continuation, which several strings in this tree use, - // including this file's own assertion message -- the - // newline is not part of the string's value but it is - // part of the file's line structure. Replacing it with - // a space merges two source lines and shifts the line - // number of every offense reported after it, which is - // the `file:line` the assertion promises. - out.push(' '); - out.push(if next == Some('\n') { '\n' } else { ' ' }); - i += 2; - continue; - } - if c == '"' { - state = State::Code; - } - out.push(if c == '\n' { '\n' } else { ' ' }); - i += 1; - } - State::RawStr(hashes) => { - if c == '"' && closing_hashes(&chars, i + 1) >= hashes { - state = State::Code; - out.extend(std::iter::repeat_n(' ', hashes + 1)); - i += hashes + 1; - continue; - } - out.push(if c == '\n' { '\n' } else { ' ' }); - i += 1; - } - State::Char => { - if c == '\\' { - out.push_str(" "); - i += 2; - continue; - } - if c == '\'' { - state = State::Code; - } - out.push(' '); - i += 1; - } - } - } - out - } - - /// `Some((hash_count, chars_consumed))` when a raw-string literal opens at - /// `i` (`r"`, `r#"`, `br##"`, ...). - fn raw_string_opener(chars: &[char], i: usize) -> Option<(usize, usize)> { - let mut j = i; - // Accepting the byte-string `b` here is cosmetic, and deliberately has - // no fixture: removing it only means the caller declines at the `b` and - // matches one character later at the `r`, which opens the same literal - // and strips the same span. The single character of difference is the - // `b` itself, emitted as code rather than as a space. Four things read - // the stripped text — the brace counter, the names in `MUTATIONS`, - // [`serializer_column`] and [`has_test_attribute`] — and a `b` whose only - // neighbour on the right is the stripped literal's run of spaces cannot - // complete a brace, a `set_var(`, a `_TEST_LOCK`/`ScopedTestEnv`, or a - // `#[`. Equivalent on every path that reads it, so a test for it would - // assert nothing. - if chars.get(j) == Some(&'b') { - j += 1; - } - if chars.get(j) != Some(&'r') { - return None; - } - j += 1; - let hash_start = j; - while chars.get(j) == Some(&'#') { - j += 1; - } - if chars.get(j) == Some(&'"') { - Some((j - hash_start, j - i + 1)) - } else { - None - } - } - - fn closing_hashes(chars: &[char], mut i: usize) -> usize { - let start = i; - while chars.get(i) == Some(&'#') { - i += 1; - } - i - start - } - - /// Distinguish `'x'` from a lifetime such as `'a` in `&'a str`. - fn is_char_literal(chars: &[char], i: usize) -> bool { - match chars.get(i + 1) { - // `'\n'`, `'\''`, `'\\'` -- always a literal. - Some('\\') => true, - Some(_) => chars.get(i + 2) == Some(&'\''), - // A `'` as the final character of the file. Deliberately untested: - // the two answers are indistinguishable, because the caller's loop - // ends on the next step either way and the one character of output - // that differs (`'` versus a space) is neither a brace nor part of - // a mutating call's name. Flipping this arm is an equivalent - // mutant, so a fixture for it would assert nothing. - None => false, - } - } - /// Whether `code` carries a test attribute. /// /// The rule is "the attribute path's last segment is `test`", which covers diff --git a/xtask/src/main.rs b/xtask/src/main.rs index acb93f9af8..3d3581e2d5 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -15,6 +15,7 @@ mod amd_smi_launch_contract; mod architecture_doc; mod ascii_output_contract; mod catalog; +mod chat_import_contract; mod coverage; mod crate_edges; mod demos; @@ -31,10 +32,13 @@ mod paths; mod powershell; mod prepush; mod prepush_stamp; +mod proptest_regressions_contract; mod release_readiness; mod report_mailbox; mod report_rollup; mod rocm_releases; +#[cfg(test)] +mod rust_source; mod signing; mod tpn; mod verify_commits; diff --git a/xtask/src/proptest_regressions_contract.rs b/xtask/src/proptest_regressions_contract.rs new file mode 100644 index 0000000000..883d85eeea --- /dev/null +++ b/xtask/src/proptest_regressions_contract.rs @@ -0,0 +1,642 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Static contract test: every saved proptest failure file must sit where +//! proptest will read it. +//! +//! proptest finds a test's saved failure cases (`cc # shrinks to ...` +//! lines) by the SOURCE FILE the `proptest!` block is written in. With the +//! default `FileFailurePersistence::SourceParallel("proptest-regressions")`, +//! a block in `/src/.rs` reads and writes +//! `/proptest-regressions/.txt`, where `src/` stands for the +//! nearest ancestor directory of the source file that holds a `main.rs` or +//! `lib.rs` (it is `src/` for every crate in this tree). Move the test to another file +//! and its seed file is no longer read: no warning, no failure, and the +//! recorded counterexample is silently never replayed again. That happened +//! twice during module splits in `apps/rocm` -- once for real +//! (`proptest-regressions/main.txt`, whose test had moved to another file) and +//! once caught in review before it merged. +//! +//! This guard walks the whole tree, finds every file under a +//! `proptest-regressions/` directory, and maps it back to the source file +//! proptest would have derived it from. A seed file is an orphan when that +//! source file does not exist, or exists but holds no `proptest!` block. A +//! block is the name, the `!` and an opening delimiter as separate tokens, so +//! `proptest ! { .. }` and `proptest /* note */ ! { .. }` count; a mention in a +//! comment or a string literal, a longer name (`my_proptest!`) and a bang that +//! opens nothing (`proptest != x`) do not. The failure names the file, says +//! which source file it expected, and says what to do about it. +//! +//! What it deliberately does not check: that the seed still reproduces +//! anything (a fixed bug leaves a harmless, still-replayed seed), or which +//! `proptest!` block in a file owns which line (proptest does not record +//! that; all blocks in a file share one seed file). It also assumes the +//! default persistence and that layout. A crate with a `src/bin//main.rs` +//! would be reported as an orphan by mistake: proptest takes `src/bin//` as +//! the directory holding `main.rs` and writes `src/bin/proptest-regressions/main.txt`, +//! which this maps to `src/bin/src/main.rs`. No crate here has such a binary +//! with a property test; avoiding the false alarm needs a directory listing +//! to find the `main.rs`, which the pure mapping does not have. Property tests +//! under a crate's `tests/` directory are not seen either: no ancestor of such a +//! file holds `main.rs` or `lib.rs`, so proptest warns and writes the seed next +//! to the test file, not under `proptest-regressions/`. A test configured with `failure_persistence: None` +//! never writes a seed file, and one with a different `FileFailurePersistence` +//! would put its seeds outside `proptest-regressions/`, where this does not +//! look. `cfg(test)` only, like the other source-scan contracts in this +//! crate: `cargo test -p xtask` runs it, and the merge queue (which runs the +//! whole workspace) is where a change confined to another crate meets it. + +#[cfg(test)] +mod tests { + use crate::rust_source::strip_literals_and_comments; + use std::path::{Path, PathBuf}; + + /// The directory name proptest's default persistence uses. + const REGRESSIONS_DIR: &str = "proptest-regressions"; + + /// The name of the macro that marks a file as holding a property test, + /// which `proptest!` spells with a `!` and a delimiter after it. + const PROPTEST_MACRO: &str = "proptest"; + + fn repo_root() -> PathBuf { + // CARGO_MANIFEST_DIR is the xtask/ crate dir; its parent is the repo + // root (same idiom as workflow_contract::repo_root). + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("xtask crate has a parent directory") + .to_path_buf() + } + + /// Every file under `root`, as root-relative `/` paths, sorted. + /// + /// A whole-tree walk rather than a list of crate directories, so a crate + /// added under a new top-level directory is covered without anyone + /// remembering to extend this. `target/` and dot-directories are skipped: + /// the latter include the `.claude/worktrees` checkouts of other + /// branches, and scanning those would make the verdict depend on state + /// outside the commit under test. + fn workspace_files(root: &Path) -> Vec { + let mut found = Vec::new(); + collect_files(root, root, &mut found); + found.sort(); + found + } + + fn collect_files(root: &Path, dir: &Path, found: &mut Vec) { + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + let skip = path + .file_name() + .and_then(|name| name.to_str()) + .is_none_or(|name| name == "target" || name.starts_with('.')); + if !skip { + collect_files(root, &path, found); + } + } else { + let relative = path + .strip_prefix(root) + .unwrap_or(&path) + .to_string_lossy() + .replace('\\', "/"); + found.push(relative); + } + } + } + + /// The files that live under a `proptest-regressions/` directory. + fn regression_files(all_files: &[String]) -> Vec { + all_files + .iter() + .filter(|file| file.split('/').any(|part| part == REGRESSIONS_DIR)) + .cloned() + .collect() + } + + /// The source file proptest derives `seed` from, or `None` when `seed` is + /// not shaped like one of its seed files (`/proptest-regressions/.txt`). + /// + /// The crate directory is everything before the `proptest-regressions` + /// component, which is empty for a crate at the repository root. The + /// inverse of proptest's own rule: strip the directory, swap `.txt` for + /// `.rs`, and put the result under the crate's `src/`. proptest's rule is + /// the nearest ancestor holding `main.rs` or `lib.rs`, which is `src/` for + /// every crate in this tree; see the module docs for the layouts where it + /// is not. + fn expected_source(seed: &str) -> Option { + let parts: Vec<&str> = seed.split('/').collect(); + let at = parts.iter().position(|part| *part == REGRESSIONS_DIR)?; + let (crate_dir, below) = (&parts[..at], &parts[at + 1..]); + let (last, dirs) = below.split_last()?; + let stem = last.strip_suffix(".txt").filter(|stem| !stem.is_empty())?; + let mut path: Vec = crate_dir.iter().map(|part| (*part).to_string()).collect(); + path.push("src".to_string()); + path.extend(dirs.iter().map(|part| (*part).to_string())); + path.push(format!("{stem}.rs")); + Some(path.join("/")) + } + + /// True when `source` has a `proptest!` block in code. + /// + /// A `proptest!` named in a comment or a string literal must not keep an + /// orphaned seed alive: a note about where a test moved, whether a `//` + /// line, a trailing `// ...` after code, or a `/* ... */` block (nested + /// ones included), and a test disabled by wrapping it in a block comment, are + /// not property tests. So the text is tokenized first, with the same + /// [`strip_literals_and_comments`] the other source-scan contracts use, + /// which blanks the contents of comments, string literals, raw strings and + /// char literals, and only what is left is searched. + /// + /// What is searched for is the name, the `!` and an opening delimiter as + /// separate tokens, not the text `proptest!`: Rust allows whitespace and + /// comments between them (`proptest ! { .. }`, `proptest /* note */ ! { .. }`), + /// and proptest reads the seed file of such a block all the same. The + /// stripper leaves whitespace alone and blanks a comment to spaces, so + /// skipping whitespace after the name, and again after the `!`, covers + /// both. The name must not be glued to a longer identifier (`my_proptest!` + /// is another macro), and a `!` that does not open a delimiter + /// (`proptest != x`, a variable compared with something) is not an + /// invocation. + /// + /// It still does not parse Rust: a `proptest!` inside another macro's + /// arguments counts, and so does one in a block that is compiled out by a + /// `cfg`. Nothing in this tree does either. + fn has_proptest_block(source: &str) -> bool { + let code = strip_literals_and_comments(source); + code.match_indices(PROPTEST_MACRO).any(|(at, name)| { + let glued_to_a_longer_name = code[..at] + .chars() + .next_back() + .is_some_and(|before| before.is_alphanumeric() || before == '_'); + if glued_to_a_longer_name { + return false; + } + code[at + name.len()..] + .trim_start() + .strip_prefix('!') + .is_some_and(|after_bang| after_bang.trim_start().starts_with(['{', '(', '['])) + }) + } + + /// Why a seed file is not read by any test. + #[derive(Debug, Clone, PartialEq, Eq)] + enum Orphan { + /// Not `.txt` under a `proptest-regressions` directory, so + /// proptest would never produce or read it. + NotASeedFile { seed: String }, + /// The source file proptest would read this seed for does not exist. + SourceMissing { seed: String, expected: String }, + /// The source file exists but no longer holds a `proptest!` block. + NoPropertyTest { seed: String, expected: String }, + } + + /// The seed files nothing reads. + /// + /// `read_source` returns a file's text by root-relative path, or `None` + /// when it does not exist; taking it as a parameter keeps the decision + /// pure so fixtures can stand in for the tree. + fn orphaned_seeds( + seeds: &[String], + read_source: impl Fn(&str) -> Option, + ) -> Vec { + let mut orphans = Vec::new(); + for seed in seeds { + let Some(expected) = expected_source(seed) else { + orphans.push(Orphan::NotASeedFile { seed: seed.clone() }); + continue; + }; + match read_source(&expected) { + None => orphans.push(Orphan::SourceMissing { + seed: seed.clone(), + expected, + }), + Some(text) if !has_proptest_block(&text) => orphans.push(Orphan::NoPropertyTest { + seed: seed.clone(), + expected, + }), + Some(_) => {} + } + } + orphans + } + + /// The failure text: the file, what proptest expected, the likely cause, + /// and the fix. The cause and fix are the same for every kind, so they are + /// written once. + fn describe(orphans: &[Orphan]) -> String { + let lines: Vec = orphans + .iter() + .map(|orphan| match orphan { + Orphan::NotASeedFile { seed } => format!( + " {seed}\n not shaped like a proptest seed file \ + (/{REGRESSIONS_DIR}/.txt), so proptest never reads it" + ), + Orphan::SourceMissing { seed, expected } => format!( + " {seed}\n proptest would read it for {expected}, which does not exist" + ), + Orphan::NoPropertyTest { seed, expected } => format!( + " {seed}\n proptest would read it for {expected}, which has no \ + `{PROPTEST_MACRO}` block" + ), + }) + .collect(); + format!( + "{} proptest seed file(s) that no test reads. proptest names a seed file after the \ + source file of the test (/src/.rs reads \ + /{REGRESSIONS_DIR}/.txt, with src/ the directory holding \ + main.rs or lib.rs), so the likely cause is that the property \ + test moved to another file, or the file was renamed: the saved counterexample is \ + then never replayed again, with no warning. Fix: put its `cc` lines in \ + /{REGRESSIONS_DIR}/: `git mv` the file when nothing is there yet, and append the lines to the \ + existing file when something is (a destination that already has seeds of its own \ + keeps them). Then check it is read by appending a deliberately unparsable line and \ + running the test with `--nocapture` (proptest prints `unparsable line, \ + ignoring`); remove the line again. If the test is gone, delete the seed file.\n{}", + orphans.len(), + lines.join("\n") + ) + } + + /// Orphans among the real files under `root`. + fn orphans_under(root: &Path) -> Vec { + let seeds = regression_files(&workspace_files(root)); + orphaned_seeds(&seeds, |path| std::fs::read_to_string(root.join(path)).ok()) + } + + /// The contract: every seed file is read by a test that still exists. + #[test] + fn every_proptest_regression_file_belongs_to_a_property_test_that_reads_it() { + let orphans = orphans_under(&repo_root()); + assert!(orphans.is_empty(), "{}", describe(&orphans)); + } + + /// The walk reaches the crates' sources, so an empty seed list means + /// "nothing to check" and not "the scan looked in the wrong place". + #[test] + fn the_scan_sees_the_workspace_it_is_guarding() { + let files = workspace_files(&repo_root()); + assert!( + files.iter().any(|file| file == "apps/rocm/src/main.rs"), + "the walk did not reach apps/rocm/src/main.rs -- the walk is broken, not the tree" + ); + } + + fn fixture(files: &[(&str, &str)]) -> impl Fn(&str) -> Option { + let files: Vec<(String, String)> = files + .iter() + .map(|(path, text)| ((*path).to_string(), (*text).to_string())) + .collect(); + move |path| { + files + .iter() + .find(|(candidate, _)| candidate == path) + .map(|(_, text)| text.clone()) + } + } + + fn strings(items: &[&str]) -> Vec { + items.iter().map(|item| (*item).to_string()).collect() + } + + #[test] + fn a_seed_maps_to_the_source_file_proptest_derives_it_from() { + assert_eq!( + expected_source("apps/rocm/proptest-regressions/model_catalog_text.txt").as_deref(), + Some("apps/rocm/src/model_catalog_text.rs") + ); + assert_eq!( + expected_source("apps/rocm/proptest-regressions/providers/local_http/proptests.txt") + .as_deref(), + Some("apps/rocm/src/providers/local_http/proptests.rs") + ); + assert_eq!( + expected_source("crates/rocm-dash-tui/proptest-regressions/ui/format.txt").as_deref(), + Some("crates/rocm-dash-tui/src/ui/format.rs") + ); + // A crate at the repository root has an empty crate directory. + assert_eq!( + expected_source("proptest-regressions/a.txt").as_deref(), + Some("src/a.rs") + ); + } + + #[test] + fn a_file_that_is_not_a_txt_under_a_regressions_directory_has_no_source() { + assert_eq!(expected_source("apps/rocm/proptest-regressions/a.rs"), None); + assert_eq!(expected_source("apps/rocm/proptest-regressions/.txt"), None); + assert_eq!(expected_source("apps/rocm/proptest-regressions/"), None); + assert_eq!(expected_source("apps/rocm/src/a.txt"), None); + } + + #[test] + fn a_seed_whose_test_moved_to_another_file_is_an_orphan() { + // The shape of the real failure: the seed is still named for the file + // the test used to be in, and that file no longer has the test. + let read = fixture(&[ + ("apps/rocm/src/main.rs", "fn main() {}\n"), + ( + "apps/rocm/src/examine_report_text.rs", + "proptest! { fn format_bytes_renders_a_size_in_its_own_unit() {} }\n", + ), + ]); + let orphans = orphaned_seeds(&strings(&["apps/rocm/proptest-regressions/main.txt"]), read); + assert_eq!( + orphans, + vec![Orphan::NoPropertyTest { + seed: "apps/rocm/proptest-regressions/main.txt".to_string(), + expected: "apps/rocm/src/main.rs".to_string(), + }] + ); + } + + #[test] + fn a_seed_whose_source_file_is_gone_is_an_orphan() { + let read = fixture(&[("apps/rocm/src/other.rs", "proptest! {}\n")]); + let orphans = orphaned_seeds(&strings(&["apps/rocm/proptest-regressions/gone.txt"]), read); + assert_eq!( + orphans, + vec![Orphan::SourceMissing { + seed: "apps/rocm/proptest-regressions/gone.txt".to_string(), + expected: "apps/rocm/src/gone.rs".to_string(), + }] + ); + } + + #[test] + fn a_stray_file_in_a_regressions_directory_is_an_orphan() { + let read = fixture(&[]); + let orphans = orphaned_seeds(&strings(&["apps/rocm/proptest-regressions/notes.md"]), read); + assert_eq!( + orphans, + vec![Orphan::NotASeedFile { + seed: "apps/rocm/proptest-regressions/notes.md".to_string(), + }] + ); + } + + #[test] + fn a_seed_beside_its_property_test_is_not_an_orphan() { + let read = fixture(&[ + ( + "apps/rocm/src/model_catalog_text.rs", + "mod tests {\n proptest! {\n #[test] fn p() {}\n }\n}\n", + ), + ( + "crates/rocm-core/src/runtime.rs", + " ::proptest::proptest! { }\n", + ), + ]); + let seeds = strings(&[ + "apps/rocm/proptest-regressions/model_catalog_text.txt", + "crates/rocm-core/proptest-regressions/runtime.txt", + ]); + assert_eq!(orphaned_seeds(&seeds, read), Vec::new()); + } + + #[test] + fn a_comment_naming_proptest_does_not_keep_a_seed_alive() { + // A note about where the test moved to is not a property test. + assert!(!has_proptest_block( + "// the proptest! that was here moved to examine_report_text.rs\n/// proptest! docs\n" + )); + assert!(has_proptest_block(" proptest! {\n")); + let read = fixture(&[( + "apps/rocm/src/main.rs", + "// the proptest! moved to examine_report_text.rs\nfn main() {}\n", + )]); + let orphans = orphaned_seeds(&strings(&["apps/rocm/proptest-regressions/main.txt"]), read); + assert_eq!(orphans.len(), 1); + } + + /// The seed of `apps/rocm/src/main.rs`, whose property test moved away and + /// left only `source` behind: it is an orphan whenever `source` has no + /// `proptest!` in code. + fn orphans_when_main_holds(source: &str) -> Vec { + let files = [("apps/rocm/src/main.rs", source)]; + let seeds = strings(&["apps/rocm/proptest-regressions/main.txt"]); + orphaned_seeds(&seeds, fixture(&files)) + } + + #[test] + fn a_proptest_named_outside_code_does_not_keep_a_seed_alive() { + // Everything a note about a moved test, or a disabled one, can look like + // without being a property test. Each used to be accepted: only a line + // that STARTED with `//` was excluded. + let expected = vec![Orphan::NoPropertyTest { + seed: "apps/rocm/proptest-regressions/main.txt".to_string(), + expected: "apps/rocm/src/main.rs".to_string(), + }]; + let mut accepted = Vec::new(); + for (what, source) in [ + ( + "a trailing line comment", + "fn main() {} // proptest! moved elsewhere\n", + ), + ( + "a block comment on one line", + "fn main() {} /* proptest! moved elsewhere */\n", + ), + ( + "a block comment around the whole property block", + "/*\nproptest! {\n #[test]\n fn p() {}\n}\n*/\nfn main() {}\n", + ), + ( + "a nested block comment", + "/* outer /* inner */ proptest! { fn p() {} } still outer */\nfn main() {}\n", + ), + ( + "a block comment that opens after code", + "fn main() {} /* proptest! {\n fn p() {}\n} */\n", + ), + ( + "a string literal", + "const NOTE: &str = \"the proptest! moved elsewhere\";\n", + ), + ( + "a raw string literal", + "const NOTE: &str = r#\"the \"proptest!\" moved\"#;\n", + ), + ] { + if orphans_when_main_holds(source) != expected { + accepted.push(what); + } + } + assert!( + accepted.is_empty(), + "the guard counted a proptest! outside code as a property test: {accepted:?}" + ); + } + + #[test] + fn text_that_only_resembles_the_macro_does_not_keep_a_seed_alive() { + // The other side of reading `proptest`, `!` and a delimiter as separate + // tokens: the looser match must not accept what is not an invocation. + let expected = vec![Orphan::NoPropertyTest { + seed: "apps/rocm/proptest-regressions/main.txt".to_string(), + expected: "apps/rocm/src/main.rs".to_string(), + }]; + let mut accepted = Vec::new(); + for (what, source) in [ + ("a longer macro name", "my_proptest! { }\n"), + ("a longer macro name, spaced", "my_proptest ! { }\n"), + ("a name that continues", "proptest_cases! { }\n"), + ("the name alone", "use proptest::prelude::*;\n"), + ("a function of that name", "fn proptest() {}\n"), + ( + "a comparison", + "fn f(proptest: u8) -> bool { proptest != 3 }\n", + ), + ("a bang with no delimiter", "const X: u8 = proptest ! 3;\n"), + ("a sibling macro", "proptest::prop_compose! { }\n"), + ("a trailing name", "proptest\n"), + ] { + if orphans_when_main_holds(source) != expected { + accepted.push(what); + } + } + assert!( + accepted.is_empty(), + "the guard counted text that is not a proptest! invocation: {accepted:?}" + ); + } + + #[test] + fn a_property_block_next_to_comments_and_literals_still_counts() { + // The other direction: stripping must not hide a real block. Each is + // a source the guard has to accept, so the stricter scan cannot turn a + // seed that something reads into a false orphan. + let mut hidden = Vec::new(); + for (what, source) in [ + ("after a block comment", "/* note */ proptest! { }\n"), + ("after a line comment", "// note\nproptest! { }\n"), + ( + "after a nested block comment", + "/* a /* b */ c */ proptest! { }\n", + ), + ( + "after a string literal", + "const A: &str = \"x\"; proptest! { }\n", + ), + ( + "after a lifetime", + "fn f<'a>(_: &'a str) {} proptest! { }\n", + ), + ( + "after a char literal", + "const C: char = '\"'; proptest! { }\n", + ), + ( + "with a comment inside the block", + "proptest! {\n // a note\n #[test] fn p() {}\n}\n", + ), + ("through its path", "::proptest::proptest! { }\n"), + ("with a space before the bang", "proptest ! { }\n"), + ("with a tab before the bang", "proptest\t! { }\n"), + ("with the bang on the next line", "proptest\n ! {\n}\n"), + ( + "with a block comment before the bang", + "proptest /* note */ ! { }\n", + ), + ( + "with a nested block comment before the bang", + "proptest/* a /* b */ c */! { }\n", + ), + ( + "with a line comment before the bang", + "proptest // note\n ! { }\n", + ), + ( + "with a space before the bang, through its path", + "::proptest::proptest ! { }\n", + ), + ("with spaces around the bang", "proptest ! {\n}\n"), + ("with parentheses as its delimiter", "proptest! (\n);\n"), + ("with brackets as its delimiter", "proptest![];\n"), + ( + "with a comment between the bang and its delimiter", + "proptest! /* note */ { }\n", + ), + ] { + if !orphans_when_main_holds(source).is_empty() { + hidden.push(what); + } + } + assert!( + hidden.is_empty(), + "the guard did not see a real proptest! block: {hidden:?}" + ); + } + + #[test] + fn the_failure_names_the_file_the_likely_cause_and_the_fix() { + let text = describe(&[Orphan::NoPropertyTest { + seed: "apps/rocm/proptest-regressions/main.txt".to_string(), + expected: "apps/rocm/src/main.rs".to_string(), + }]); + assert!( + text.contains("apps/rocm/proptest-regressions/main.txt"), + "{text}" + ); + assert!(text.contains("apps/rocm/src/main.rs"), "{text}"); + assert!(text.contains("moved to another file"), "{text}"); + assert!(text.contains("git mv"), "{text}"); + // A destination that already has seeds keeps them: the advice must not + // be an unconditional `git mv`, which would replace or refuse there. + assert!(text.contains("append the lines"), "{text}"); + assert!(text.contains("unparsable line, ignoring"), "{text}"); + } + + /// The walk and the mapping together, on a tree on disk: a seed beside its + /// test passes, and the same tree with the test moved out from under it + /// fails. Also covers a nested seed path and a skipped dot-directory. + #[test] + fn a_tree_on_disk_is_orphaned_exactly_when_the_test_moves() { + let tmp = tempfile::tempdir().expect("tempdir"); + let write = |relative: &str, text: &str| { + let path = tmp.path().join(relative); + std::fs::create_dir_all(path.parent().expect("parent")).expect("mkdir"); + std::fs::write(path, text).expect("write"); + }; + write("apps/demo/src/main.rs", "fn main() {}\n"); + write("apps/demo/src/sizes.rs", "proptest! { }\n"); + write("apps/demo/src/nested/deep.rs", "proptest! { }\n"); + write( + "apps/demo/proptest-regressions/sizes.txt", + "cc 00 # shrinks to n = 1\n", + ); + write( + "apps/demo/proptest-regressions/nested/deep.txt", + "cc 00 # shrinks to n = 1\n", + ); + // Another checkout's worktree: skipped, so its orphan does not count. + write( + ".claude/worktrees/x/apps/demo/proptest-regressions/stale.txt", + "cc 00\n", + ); + assert_eq!(orphans_under(tmp.path()), Vec::new()); + + // The test moves out of sizes.rs; the seed stays behind. + write("apps/demo/src/sizes.rs", "pub fn size() {}\n"); + write("apps/demo/src/report.rs", "proptest! { }\n"); + assert_eq!( + orphans_under(tmp.path()), + vec![Orphan::NoPropertyTest { + seed: "apps/demo/proptest-regressions/sizes.txt".to_string(), + expected: "apps/demo/src/sizes.rs".to_string(), + }] + ); + + // Re-homing the seed to the file the test is in now clears it. + std::fs::rename( + tmp.path().join("apps/demo/proptest-regressions/sizes.txt"), + tmp.path().join("apps/demo/proptest-regressions/report.txt"), + ) + .expect("git mv stand-in"); + assert_eq!(orphans_under(tmp.path()), Vec::new()); + } +} diff --git a/xtask/src/rust_source.rs b/xtask/src/rust_source.rs new file mode 100644 index 0000000000..48afd297f7 --- /dev/null +++ b/xtask/src/rust_source.rs @@ -0,0 +1,225 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Lexical helpers shared by the source-scan contract tests. +//! +//! The contracts in this crate read Rust sources as text. Each of them has to +//! look at code only: a brace, a call or a macro name inside a string literal +//! or a comment must not count. [`strip_literals_and_comments`] blanks those +//! out while keeping line structure, so every contract that needs the question +//! answered gets the same answer. + +/// Replace the CONTENTS of string literals, char literals and comments with +/// spaces, preserving line structure and everything outside them. +/// +/// A scan that counts braces or looks for call text is wrong if a literal can +/// contribute either. Two real files in this repo break the naive version, as +/// `env_mutation_contract.rs` found: +/// +/// * `engines/lemonade/src/lib.rs` contains +/// `ensure_cached_archive("test://archive", ..., |_, destination| {`. +/// Truncating the line at `//` throws away the closure's opening brace, +/// so the enclosing test module reads as closed hundreds of lines early +/// and every mutation after it becomes invisible. +/// * `tests/e2e-cucumber/src/mock_server.rs` contains format strings such +/// as `vllm:num_requests_running{{model=...}}`. Those braces inflate the +/// depth, which drags PRODUCTION code into the scanned region and makes +/// the guard fail a build for a mutation it should not be watching. +/// +/// Both directions are silent, which is the one failure mode a permanent +/// guard must not have, so literals are tokenized rather than approximated. +pub(crate) fn strip_literals_and_comments(text: &str) -> String { + #[derive(Clone, Copy)] + enum State { + Code, + LineComment, + BlockComment(usize), + Str, + RawStr(usize), + Char, + } + + let mut out = String::with_capacity(text.len()); + let mut state = State::Code; + let chars: Vec = text.chars().collect(); + let mut i = 0; + while i < chars.len() { + let c = chars[i]; + let next = chars.get(i + 1).copied(); + match state { + State::Code => { + // A raw string opener: `r"`, `r#"`, `br##"` and so on. Checked + // before the plain-string case so the hashes are counted. + if (c == 'r' || c == 'b') + && let Some((hashes, consumed)) = raw_string_opener(&chars, i) + { + out.extend(std::iter::repeat_n(' ', consumed)); + state = State::RawStr(hashes); + i += consumed; + continue; + } + if c == '/' && next == Some('/') { + state = State::LineComment; + out.push_str(" "); + i += 2; + continue; + } + if c == '/' && next == Some('*') { + state = State::BlockComment(1); + out.push_str(" "); + i += 2; + continue; + } + if c == '"' { + state = State::Str; + out.push(' '); + i += 1; + continue; + } + // A lifetime (`'a`) is not a char literal. A char literal is + // `'x'` or `'\n'`, so require a closing quote nearby. + if c == '\'' && is_char_literal(&chars, i) { + state = State::Char; + out.push(' '); + i += 1; + continue; + } + out.push(c); + i += 1; + } + State::LineComment => { + if c == '\n' { + state = State::Code; + out.push('\n'); + } else { + out.push(' '); + } + i += 1; + } + State::BlockComment(depth) => { + if c == '*' && next == Some('/') { + state = if depth == 1 { + State::Code + } else { + State::BlockComment(depth - 1) + }; + out.push_str(" "); + i += 2; + continue; + } + if c == '/' && next == Some('*') { + state = State::BlockComment(depth + 1); + out.push_str(" "); + i += 2; + continue; + } + out.push(if c == '\n' { '\n' } else { ' ' }); + i += 1; + } + State::Str => { + if c == '\\' { + // The escaped character cannot close the literal, so + // it is consumed here. When it is a NEWLINE -- a line + // continuation, which several strings in this tree use, + // including this file's own assertion message -- the + // newline is not part of the string's value but it is + // part of the file's line structure. Replacing it with + // a space merges two source lines and shifts the line + // number of every offense reported after it, which is + // the `file:line` the assertion promises. + out.push(' '); + out.push(if next == Some('\n') { '\n' } else { ' ' }); + i += 2; + continue; + } + if c == '"' { + state = State::Code; + } + out.push(if c == '\n' { '\n' } else { ' ' }); + i += 1; + } + State::RawStr(hashes) => { + if c == '"' && closing_hashes(&chars, i + 1) >= hashes { + state = State::Code; + out.extend(std::iter::repeat_n(' ', hashes + 1)); + i += hashes + 1; + continue; + } + out.push(if c == '\n' { '\n' } else { ' ' }); + i += 1; + } + State::Char => { + if c == '\\' { + out.push_str(" "); + i += 2; + continue; + } + if c == '\'' { + state = State::Code; + } + out.push(' '); + i += 1; + } + } + } + out +} + +/// `Some((hash_count, chars_consumed))` when a raw-string literal opens at +/// `i` (`r"`, `r#"`, `br##"`, ...). +fn raw_string_opener(chars: &[char], i: usize) -> Option<(usize, usize)> { + let mut j = i; + // Accepting the byte-string `b` here is cosmetic, and deliberately has + // no fixture: removing it only means the caller declines at the `b` and + // matches one character later at the `r`, which opens the same literal + // and strips the same span. The single character of difference is the + // `b` itself, emitted as code rather than as a space. In + // `env_mutation_contract.rs`, four things read + // the stripped text — the brace counter, the names in `MUTATIONS`, + // [`serializer_column`] and [`has_test_attribute`] — and a `b` whose only + // neighbour on the right is the stripped literal's run of spaces cannot + // complete a brace, a `set_var(`, a `_TEST_LOCK`/`ScopedTestEnv`, or a + // `#[`. Equivalent on every path that reads it, so a test for it would + // assert nothing. + if chars.get(j) == Some(&'b') { + j += 1; + } + if chars.get(j) != Some(&'r') { + return None; + } + j += 1; + let hash_start = j; + while chars.get(j) == Some(&'#') { + j += 1; + } + if chars.get(j) == Some(&'"') { + Some((j - hash_start, j - i + 1)) + } else { + None + } +} + +fn closing_hashes(chars: &[char], mut i: usize) -> usize { + let start = i; + while chars.get(i) == Some(&'#') { + i += 1; + } + i - start +} + +/// Distinguish `'x'` from a lifetime such as `'a` in `&'a str`. +fn is_char_literal(chars: &[char], i: usize) -> bool { + match chars.get(i + 1) { + // `'\n'`, `'\''`, `'\\'` -- always a literal. + Some('\\') => true, + Some(_) => chars.get(i + 2) == Some(&'\''), + // A `'` as the final character of the file. Deliberately untested: + // the two answers are indistinguishable, because the caller's loop + // ends on the next step either way and the one character of output + // that differs (`'` versus a space) is neither a brace nor part of + // a mutating call's name. Flipping this arm is an equivalent + // mutant, so a fixture for it would assert nothing. + None => false, + } +} From f825bc70498e426d523c8130021fcf8574d0f078 Mon Sep 17 00:00:00 2001 From: Tomas Saaristola <144015745+tomastola@users.noreply.github.com> Date: Sat, 7 Nov 2026 00:02:31 +0000 Subject: [PATCH 006/102] feat(diagnose): estimate context size and memory for a GGUF model (#826) * feat(diagnose): estimate context size and memory for a GGUF model `rocm diagnose --model` answered UNDETERMINED for every model outside the curated catalog, and nothing in it took the context size into account. For a GGUF outside the catalog (a local path, or owner/repo:QUANT on Hugging Face) it now reads the file's header, by HTTP range request for a remote file, without downloading the weights. It estimates the weights, the KV cache at a context size and KV-cache type, the compute buffer and a runtime reserve, and compares them with the dedicated VRAM of the GPU with the most of it. The result is labelled ESTIMATED, never READY or BLOCKED. It gives the largest context that fits, and when the model does not fit, what would: a smaller context, a q8_0/q4_0 KV cache, a smaller quantization, or partial CPU offload. The verdict, the largest context and the suggestions are all decided on the figures as shown, so they cannot contradict each other. New flags: --context-size (default 4096) and --kv-cache-type (f16, q8_0, q4_0). The compute buffer and runtime reserve are calibrated on real loads of two dense models on an MI300X with llama.cpp b9752 and flash attention on. In all sixteen measured loads the estimate was above the real VRAM growth, by 0.05% to 5.5%. The report states this caveat and gives per-model notes where the figure may be off. Split files and latent attention are refused with a reason. The header is untrusted input. Header size, layer count, kept metadata and the remote read time are all bounded. A listed file name that could leave the repository or hide in a terminal is dropped. Under --report the model is not assessed and nothing is fetched; a note on stderr says which flags were ignored. Part of #781: the `rocm serve` context option is left for a follow-up, so #781 stays open. Signed-off-by: Tomas Saaristola * fix(diagnose): pass the ASCII contract and bound the range read The bidirectional and zero-width check in the listed-name filter used `\u{..}` char literals, which the xtask ASCII output contract rejects. Compare code points instead, and test both ends of each range. Clamp the range reader's buffer capacity to the window size where it is allocated. `wanted` was already bounded by the window, but only through arithmetic a static analyser does not follow. Signed-off-by: Tomas Saaristola * refactor(diagnose): split the GGUF estimate glue and state calibration once No change in behaviour; this answers review points on maintainability. - The calibration figures (which loads, how close the estimate came) are written down only in the gguf_fit module documentation. The README, docs/testing.md and docs/manual-testing.md point there instead of repeating them. - One fixed_width() table of GGUF type ids serves both a scalar value and an array element, so a new type id is added in one place. - attach_context_estimate is now plan, read and apply steps; the plan step is tested on its own. - The binary's diagnose --model glue (the kv-cache-type flag enum, the model query, the --report note, the device-mask caveat) moves out of main.rs into diagnose_model.rs, reached by qualified path. Signed-off-by: Tomas Saaristola * fix(diagnose): answer a named projector as one, and test the zero guard A Hugging Face file name that matches only a multimodal projector (mmproj) was filtered out before matching, so it read as a file the repository did not have, and the listing in the error left it out. It now raises a typed refusal: the estimate's reason is not_a_model, the detail names the repository's model files, and the plan asks about the model (by its exact name when there is one) instead of offering to serve the projector. A projector is still never picked as the model, and a bare name a model shares with a projector picks the model. Scenario diagnose-51 covers it against a stand-in hub that serves the listing. max_context's `context > 0` guard had no test that failed without it. The new test puts the card's memory between the exact and shown zero-context totals, where even zero context does not fit. docs/testing.md also records why the device-mask cases are unit-tested only. Signed-off-by: Tomas Saaristola * docs(diagnose): reflow the --model help and widen the model field's lead The --model help wrapped a sentence across two doc lines, which clap rendered ragged, and repeated the device-mask detail the README already gives. The rocm-doctor reference opened the model field as curated-only, though the same entry goes on to describe the GGUF estimate. Signed-off-by: Tomas Saaristola * test(diagnose): pin the GGUF estimate's branches by mutation A review mutated the estimate's new branches and twelve survived the whole suite. Each now has a test that fails with it applied alone: the percent-encoding of listed names, the deadline on the listing and HEAD requests, `_` as a label boundary, the widest per-layer feed-forward width, the estimate's `note:` label, the 4096 default, a ceiling equal to the request, the trained-context cap on the smaller-cache suggestion, either missing width, the CUDA_VISIBLE_DEVICES mask, and the order of estimate and mask caveat (the environment lookup is now injectable). Three tests could not fail on what they named: the size-bound test now reaches the running header-size check, the skipped-arrays test reads a kept key back after the arrays, and the "no weights fetched" step is dropped from the local-file scenarios, where nothing could download. Scenarios diagnose-52 to 56 cover the refusals and the no-match listing, and diagnose-49's label is now checked against whether memory was compared rather than accepting either. A projector refusal names the file the reference resolved to, and a no-match error lists every GGUF with projectors marked. A local header is read through a buffer that survives the parser's relative seeks, which made a large vocabulary seek the file once per token. with_hub and fetched are test-only. Signed-off-by: Tomas Saaristola * fix(diagnose): keep the amd-smi plan when an estimate cannot compare A GGUF whose header was sized on a host whose GPU memory could not be read ended with no next step: the estimate cleared the plan, though a curated model on the same host is told to look at amd-smi. It now keeps that plan. No GPU and an APU's unreadable pool still get none, as for a curated model, since no command would change the answer. Signed-off-by: Tomas Saaristola * fix(diagnose): give no fit verdict where the figure can be too low A state-space or hybrid model's recurrent state is not counted, and a mixture-of-experts model's compute buffer was not measured, so the total can be too low; yet the report still headlined "ESTIMATED: fits", with only a note line saying the figure could be higher. A confident fits that is wrong costs a failed load, and a margin with no measurement behind it would be a guess. These layouts now get their figures and this GPU's memory, with no verdict: cannot_estimate, reason unmeasured_layout, and the detail says why. A sliding-window model, whose figure can only be too high, keeps its verdict. Signed-off-by: Tomas Saaristola * refactor(diagnose): show the host's MiB through one rounding helper The no-verdict detail floored the GiB figure without the 1e-6 the evidence line uses, so a figure a hair under a whole MiB could read one MiB lower there than on the line beside it. Both now go through whole_mib_offered, and a test pins that they agree. Signed-off-by: Tomas Saaristola * fix(diagnose): withhold the verdict when a width is missing; test what was promised A header without the feed-forward or embedding width sized the compute buffer's fixed part as zero, said the figure could be too low, and still gave a fits / does-not-fit verdict. It now gets its figures with no verdict (reason missing_metadata), as a state-space or mixture-of-experts model does. Tests for what the code promised but nothing checked: - `rocm diagnose --model owner/repo:QUANT` against a stand-in hub, with every Hugging Face token variable set, sends no Authorization or Cookie header on the listing, HEAD or range requests. - Every fixed-width GGUF value type is read at its own width, as a scalar and as an array element, kept and skipped. - A 206 with no Content-Range and a HEAD with no Content-Length are refused. - F16 does not match BF16. A shard is now split at the last `-of-`, so a model named `Chain-of-Thought-...-00001-of-00003.gguf` is one split model rather than several matching files. The `rocm diagnose` command a GGUF plan offers keeps --context-size and --kv-cache-type and quotes a path a shell would split. Comments and docs that disagreed with the code are corrected, and docs/llm-tool-use.md says partial GPU offload is not CPU fallback. Signed-off-by: Tomas Saaristola * fix(diagnose): give a GGUF estimate with no visible GPU the curated no-GPU plan A sized GGUF estimate on a host with no visible GPU offered no plan, while a curated model there offers `rocm diagnose` to find out why. Both now share one plan, and the GGUF one asks again with the same flags. Signed-off-by: Tomas Saaristola * fix(diagnose): count a tied embedding in VRAM The weights left the whole input embedding out of VRAM. A model with tied embeddings has no output.weight: llama.cpp uses token_embd.weight for the output layer as well and places that use with the output layer, on the GPU, so leaving it out could understate the figure and say "fits" when it does not. The parser now records whether output.weight exists, and the embedding is left out only when it does. The tied Qwen-shaped calibration model is now estimated 3.6% to 9.5% above its measured loads (was 0.05% to 2.5%); the module documentation, README and docs/testing.md give the new margins and name the two calibrated shapes. Signed-off-by: Tomas Saaristola * fix(diagnose): quote the re-run command per platform; pin the new branches The `rocm diagnose --model` line a GGUF plan offers was quoted for a POSIX shell on every platform, which breaks in cmd.exe and PowerShell. It now leaves a reference with no shell-special character as it is (so a plain Windows path works in both), and otherwise quotes it with the shared PowerShell or POSIX quoter. A reference starting with `-` is joined to the flag so it is not read as another option. Tests now pin the flags on the amd-smi, no-GPU and projector re-run lines, that a float under a kept key is never read as an integer, and that a tensor after output.weight does not clear it. Comments and docs that still said the input embedding always stays in host memory, gave the old calibration range, or said a no-GPU estimate gets no plan are corrected. Signed-off-by: Tomas Saaristola * test(e2e): renumber the GGUF diagnose scenarios after main's diagnose-48 main added diagnose-48 (the Windows HIP SDK scenario), so this branch's scenarios move up by one: diagnose-49 to diagnose-57. Signed-off-by: Tomas Saaristola * fix(diagnose): price Q2_K as bits per weight; tidy the review's loose ends The Q2_K row of the bits-per-weight table held 2.63, llama.cpp's published LLaMA-v1-7B size in GiB rather than its bits per weight (about 3.35), so a "Q2_K would fit" suggestion was about 20% light in the tight case where it is offered. A test now holds every K-quant and Q8_0 row to the published sizes. Also: - An unsized local GGUF's plan asks the question again instead of offering `rocm serve `, which takes no path. - The hub listing is read as bytes, so a cut inside a multi-byte character reports the size limit rather than unreadable text. - An exact file name may carry `+`; hub file names do. - File-name segments go through the one percent-encoder in report_delivery. - RelativeSeekBuf's doc says why it tracks its position, and a test pins it. - The assistant skill says partial GPU offload is not CPU fallback, matching docs/llm-tool-use.md, since that skill is the assistant's system prompt. - Tests for a context flag on a reference that is not a GGUF, and a comment on the guard that keeps a recipe-less answer from being estimated. - MissingMetadata's doc covers implausible figures; docs/testing.md points at the module docs for the calibration margins instead of restating them. Signed-off-by: Tomas Saaristola * feat(diagnose): show a spinner while a GGUF header is read from the hub A remote header read (a listing, a HEAD, then range requests, within a two-minute budget) printed nothing until it finished. The shared stderr spinner now covers it, only for a catalog miss named owner/repo:QUANT and only when stderr is a terminal, so --json and piped output are unchanged. Signed-off-by: Tomas Saaristola * fix(diagnose): give the Qwen calibration fixture the output layer its file has The Qwen2.5-1.5B Q8_0 GGUF the allowances were calibrated on carries its own output.weight (29 `output.weight` tensor names in the header: 28 attention outputs and the model's own), a second embedding-sized tensor that accounts for its 1 888 581 632 bytes of tensor data. The fixture left it out, so the test treated the model as tied, counted the embedding on the GPU and restated the margins (3.6% to 9.5%) and an "unexplained overhead" around a fixture error. With the tensor in place the estimate is again 0.05% to 5.5% above all sixteen measured loads. The tied-embedding rule stays, and the docs now say it is not measured: both calibration files have their own output layer, and counting the embedding can only overstate. The architecture name, the one header string the report prints, now has its terminal-unsafe characters replaced, and KvCacheType's FromStr is test-only. Signed-off-by: Tomas Saaristola * fix(diagnose): check the name and network when a hub header cannot be read An unreadable hub header (offline, gated, misspelt) still offered `rocm serve owner/repo:QUANT`, which would start a multi-GB download to fail the same way. It now asks to check the name and the network and ask again, and says a gated repository can be read by downloading the file and passing its local path. A pasted URL, read as a path, is told which form is read. Also: listed hub names are held to the shared terminal predicates (adding the line separators, invisible operators, soft hyphen and tag characters); the header reads send User-Agent: rocm-cli as the CLI's other downloads do; the re-run command leaves a mid-path `~` (Windows short names) unquoted; the no-token test also checks the estimate it produced; `--context-size 1` is tested as accepted. Signed-off-by: Tomas Saaristola * test(e2e): accept the GGUF path's re-run command bare or quoted On the Windows lane the scenario's temp path decides whether the command quotes it, so diagnose-55 accepts either form instead of asserting one. Signed-off-by: Tomas Saaristola * fix(diagnose): an APU hidden by a device mask is "no GPU is visible" The APU branch of the memory classification ignored the device-visibility mask, so a masked-out Strix Halo still read as an APU whose pool cannot be read, with no plan, while a discrete card hidden the same way already read as "no GPU is visible". Both now give the answer a command can change. Tests: a hub that redirects every request to another host (as resolve/main does to its CDN) is followed for the listing, the HEAD and the range requests, with Range kept; a state-space, hybrid or mixture-of-experts model with no GPU visible gets "no GPU" with its figures. Signed-off-by: Tomas Saaristola * test(diagnose): pin the parser and reader bounds that no test could fail Each new test fails when its guard is removed: the metadata and tensor count bounds (by message), a tensor name past the key-length bound (now refused as a metadata key is), dimensions that overflow, a range answer that runs past the bytes asked for, and a listing that never ends. The sliding- window key no longer takes one of the kept slots, and redirects are bounded at five explicitly. Signed-off-by: Tomas Saaristola * fix(diagnose): say what rocm serve does with the size, and test the outcome - A compared estimate now says `rocm serve` passes no context size, so a served model gets the engine's default; a test over serve's clap args fails when serve gains one of the options the notes name. - The ESTIMATED label says the allowances were calibrated on one MI300X. - A zero feed-forward or embedding width counts as missing. - A header that could not be read gets a plan that names the possible causes (name, network, busy hub, not a GGUF). - An unsized local GGUF is offered to `rocm serve` again, which does take a local path, quoted for the shell. - A layout with no verdict has its no-plan arm named and an integration test. - diagnose-49/50 derive the outcome they expect from the reported figures instead of accepting either; two vacuous assertions now check something. - Docs: --help example and HF_ENDPOINT; max_context null cases; the module docs of model_fit.rs and diagnose_cmd.rs; gguf* entries in alphabetical order. Signed-off-by: Tomas Saaristola * docs(architecture): keep the GGUF estimate in the model_readiness entry main rewrote the model_readiness.rs entry, and the rebase took its version, dropping the sentence that points at the gguf* modules. Signed-off-by: Tomas Saaristola * fix(diagnose): test the GGUF guards and refuse https-to-http redirects A self-review mutated the new GGUF modules one branch at a time and found guards that no test could fail on. Each now has a test that fails when the guard is removed: the zero head count, the U+2028/U+2029 separators, the total deadline `open` hands the reader (the source takes a budget so a test can shorten it), the eight-segment variant limit, the 403 "gated or private" arm, the sliding-window key taking no kept slot, and a repeated key at the kept-key cap. The bits-per-weight floor is not among them: the smallest quantization in the table is 3.35, so no value under 1.0 can yield a suggestion and the floor cannot change an outcome. Redirects were followed by the HTTP agent, which cannot say where one goes before going there, so a mirror set through HF_ENDPOINT could move a request from https to plain http. The agent now follows none; `send` follows up to five by hand, keeps the Range header on each hop, and refuses a hop from https to http or to a scheme that is not http(s). Also restore the model_readiness entry in docs/architecture.md that a rebase had carried over from before main renamed the file it names, point the serve-options note at cli.rs where its test lives, and rewrap a 125-column module doc line. Signed-off-by: Tomas Saaristola --------- Signed-off-by: Tomas Saaristola (cherry picked from commit 3dc4a31aad079e1ee0e875c208f1c346a3032235) Signed-off-by: juhovainio --- README.md | 119 +- apps/rocm/src/cli.rs | 163 +- apps/rocm/src/diagnose_cmd.rs | 29 +- apps/rocm/src/diagnose_model.rs | 501 +++ apps/rocm/src/main.rs | 10 +- apps/rocm/src/model_fit.rs | 121 +- .../tests/diagnose_model_sends_no_token.rs | 220 + crates/rocm-core/src/download.rs | 2 +- crates/rocm-core/src/gguf.rs | 1611 +++++++ crates/rocm-core/src/gguf_fit.rs | 3739 +++++++++++++++++ crates/rocm-core/src/gguf_source.rs | 2508 +++++++++++ crates/rocm-core/src/lib.rs | 3 + crates/rocm-core/src/model_readiness.rs | 1721 +++++++- crates/rocm-core/src/report_delivery.rs | 8 +- docs/architecture.md | 6 +- docs/llm-tool-use.md | 5 +- docs/manual-testing.md | 63 + docs/testing.md | 104 +- skills/rocm-cli-assistant/SKILL.md | 2 +- skills/rocm-doctor/reference.md | 44 +- tests/e2e-cucumber/features/diagnose.feature | 103 +- tests/e2e-cucumber/tests/e2e.rs | 5 + .../e2e-cucumber/tests/e2e/diagnose_steps.rs | 679 +++ 23 files changed, 11674 insertions(+), 92 deletions(-) create mode 100644 apps/rocm/src/diagnose_model.rs create mode 100644 apps/rocm/tests/diagnose_model_sends_no_token.rs create mode 100644 crates/rocm-core/src/gguf.rs create mode 100644 crates/rocm-core/src/gguf_fit.rs create mode 100644 crates/rocm-core/src/gguf_source.rs diff --git a/README.md b/README.md index 545d4347f7..bf24be610c 100644 --- a/README.md +++ b/README.md @@ -385,7 +385,7 @@ form works depends on the engine your GPU selects. | `rocm` | Open the launcher menu (setup, serve, diagnose, chat, dashboard) | | `rocm examine` | Check GPU, ROCm install, engines, and managed folders | | `rocm diagnose` | Match this machine against known ROCm, PyTorch, and llama.cpp failure modes | -| `rocm diagnose --model ` | Say whether a model will run here, before downloading it | +| `rocm diagnose --model ` | Say whether a model will run here, before downloading it; for a GGUF, estimate the largest context that fits | | `rocm diagnose --report` | Show what this machine would contribute to a problem report, and send nothing | | `rocm fix []` | Apply a fix reported by `rocm diagnose` | | `rocm install sdk` | Install TheRock ROCm wheels into a managed Python environment | @@ -438,7 +438,7 @@ Windows. The Vulkan driver version is not shown. ``` rocm diagnose [--symptom TEXT] [--top N] [--json] [--distro [NAME] | --report [--send]] -rocm diagnose --model [--json] +rocm diagnose --model [--context-size N] [--kv-cache-type f16|q8_0|q4_0] [--json] rocm fix [] [--yes] [--dry-run] [--device-index N] [--json] ``` @@ -474,7 +474,9 @@ position. those. - `--report` shows exactly what this machine would contribute to a problem report, and sends nothing. There is no automatic transport, and there will be - none: a report leaves a machine only by its owner's own action. + none: a report leaves a machine only by its owner's own action. A `--model` + given with it is not assessed (a report carries no model verdict), so a GGUF + reference is not looked up on Hugging Face for it. - `--send`, which requires `--report`, additionally offers a prefilled mail carrying that report. It still sends nothing: the mail opens already filled in with the content `--report` just printed, addressed to `ROCmCLI@amd.com`, @@ -915,14 +917,14 @@ Windows these installs are skipped. Ask before downloading anything: ``` -rocm diagnose --model [--json] +rocm diagnose --model [--context-size N] [--kv-cache-type f16|q8_0|q4_0] [--json] ``` -Answers in seconds, from the curated recipe and this machine's GPU. It -fetches no weights and makes no network call. The verdict is `ready`, -`degraded`, `blocked`, or `undetermined`. A `ready` answer also names the -engine `rocm serve` would use; a `blocked` one names curated models that would -run here instead. +Answers from the curated recipe and this machine's GPU, in seconds. It +downloads no weights. For a curated model it makes no network call. The +verdict is `ready`, `degraded`, `blocked`, or `undetermined`. A `ready` answer +also names the engine `rocm serve` would use; a `blocked` one names curated +models that would run here instead. `undetermined` is a real answer, not a failure. You get it when any of these is true: @@ -935,7 +937,104 @@ true: None of these says anything about whether the model fits, so none of them is reported as though it did. `rocm serve` still accepts a model outside the -catalog, but `diagnose` cannot tell you in advance how it will go. +catalog, but for anything other than a GGUF, `diagnose` cannot tell you in +advance how it will go. + +#### A GGUF model outside the catalog + +Pass a local `.gguf` path, or `owner/repo:QUANT` for a file on Hugging Face +(for example `owner/repo:Q4_K_M`, or an exact file name ending in `.gguf`, with +its directory when the file is in one and another file shares its name: +`owner/repo:dir/model-Q4_K_M.gguf`; a bare name means the file at the top of the +repository when there is one). On Hugging Face, a multimodal projector (an `mmproj` +file shipped beside a vision model) is never picked as the model, and naming one +by its file name is answered as such, with the repository's model files named +instead. A local projector file is not recognised as one; it is read like any +other GGUF. A file that +exists on this machine is read as that file even when its name looks like +`owner/repo:QUANT`. A bare +`owner/repo`, with no quantization, names a repository rather than a file and +is answered like any other model outside the catalog: undetermined, with +nothing read. +Instead of a recipe's minimum, `diagnose` reads the file's header, which holds +the layer count, the KV head count and head width, and the tensor shapes. For +a remote file that is one listing request for the repository (up to 16 MiB, to +find the file and its size), a `HEAD` request for a file the listing does not +size, then a few megabytes fetched by HTTP range request (set +`HF_ENDPOINT` to use a mirror); the weights are never downloaded. A remote read +has two minutes in all, so on a slow link it can take that long to fail. No +token is sent, so a gated or private repository cannot be read. +It then estimates weights, plus the KV cache at `--context-size` (default 4096) +and `--kv-cache-type` (default `f16`), plus a compute buffer and a runtime +reserve, against this GPU's dedicated VRAM: + +``` +rocm diagnose --model owner/repo:Q4_K_M --context-size 32768 --kv-cache-type q8_0 +``` + +The answer is labelled `ESTIMATED: fits at N tokens` or `ESTIMATED: does not +fit at N tokens`, never `ready` or `blocked`: in `--json` the `verdict` stays +`undetermined` and the figures are under `model.estimate`. It reports the +largest context that fits with that KV cache type. When the file does not fit +at the size you asked for, it lists what would make it fit: a smaller context, +a q8_0 or q4_0 KV cache (llama.cpp needs flash attention for a quantized V +cache, and the estimate assumes it is on), a smaller quantization, or keeping +some layers on the CPU. Those are llama.cpp options (`--ctx-size`, `--cache-type-k`/`-v`, +`--n-gpu-layers`): **`rocm serve` does not take a context size, KV cache type +or GPU layer count yet**, so none of them is printed as a `rocm serve` command. + +The KV cache is exact for a plain full-attention model, and the weights are the +file's own size from its tensor offsets, less the input embedding, which +llama.cpp keeps in host memory -- except for a model with tied embeddings (no +separate output layer), where llama.cpp uses the same tensor for the output +layer on the GPU, so there it is counted. That case was not measured, and +counting it can only overstate the figure. Only the compute buffer and the +runtime reserve are allowances, calibrated against real loads of two dense +full-attention models on one MI300X with llama.cpp b9752, flash attention on: +a Llama-3.1-8B-shaped and a Qwen2.5-1.5B-shaped GGUF, each with its own output +layer. +No mixture-of-experts, state-space, hybrid or sliding-window model was among +them. How close the estimate came to those +loads, and why that describes the fit rather than a bound, is written down once, +in the +[`gguf_fit` module documentation](https://github.com/ROCm/rocm-cli/blob/main/crates/rocm-core/src/gguf_fit.rs). +A model with sliding-window, state-space or mixture-of-experts layers was not +measured. A sliding-window model's KV cache is sized as if every layer attended +over the whole context, which can only overstate it, so it still gets a verdict. +A state-space or hybrid model's recurrent state is not counted, and a +mixture-of-experts model's compute buffer was not measured, so for those the +figures are shown with this GPU's memory but with no fits / does-not-fit verdict +(`cannot_estimate`, reason `unmeasured_layout`): a "fits" that is too low would +be worse than no answer. Many current open models are mixture-of-experts or +hybrid, so for a large share of them this gives the memory figures and leaves +the comparison to the reader. A header without the feed-forward or embedding width, +which size the compute buffer, is treated the same way (reason +`missing_metadata`, with the figures). Another +GPU, engine build or batch size can miss by more, so leave headroom. The figures +are compared with the total dedicated VRAM of the GPU with the most of it that +ROCm can see, the same figure a curated model's minimum is compared with, not +with what other processes have left free. When a device-visibility variable +(`HIP_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`) is set on +a host whose GPUs differ in memory, the report says that the mapping between the +mask and `amd-smi`'s numbering has not been verified on such a host, so the +figure may be for a different card; that mapping has not been checked on a +multi-GPU machine. Where memory cannot be read, the memory the model needs is +shown and nothing is compared. The memory figures are whole MiB, each rounded +up, and the verdict is decided on their sum, so what is printed always adds up. +The largest context is also capped at the context length the model was trained +for. `--context-size` takes 1 to 16777216 tokens. Passing `--context-size` or +`--kv-cache-type` for a curated model changes nothing, because a recipe's memory +minimum is fixed; the report says so. Under `--report` the model is not assessed +at all, so nothing is fetched for it, and `--context-size` and `--kv-cache-type` +are ignored along with `--model`; a note on stderr names the flags that were +ignored. + +On an integrated GPU (APU), where the CLI can't read the memory +pool the engine uses, and on a host with no GPU or an unreadable one, the +answer stays `undetermined` but still shows what the file needs. Layouts the +formula does not describe (a file that is one shard of a split model, latent +attention, a model with no attention layers) are reported as such rather than +estimated. ### Curated models diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 222a5ef726..eeed2127f8 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -111,22 +111,70 @@ pub(crate) enum Command { /// Also answer whether a model would run on this machine, before /// downloading it. /// - /// Takes a curated model name or alias (see `rocm model`). Nothing is - /// fetched: the answer comes from the recipe and this host's GPU. A - /// model the catalog does not carry is reported as undetermined rather - /// than blocked - `rocm serve` still accepts it, this just cannot say - /// in advance whether it fits. + /// Takes a curated model name or alias (see `rocm model`), or a GGUF + /// model: a local `.gguf` path, or `owner/repo:QUANT` on Hugging Face. + /// No weights are downloaded. A curated model is answered from its + /// recipe and this host's GPU. A GGUF outside the catalog is estimated + /// from its file header (for a remote file: one repository listing, a + /// `HEAD` request for a file the listing does not size, then a few MB by + /// range request; no token is sent, so a gated or private repository + /// cannot be read): weights plus KV cache at `--context-size` plus a + /// compute buffer and a runtime reserve, against the dedicated VRAM of + /// the GPU with the most of it, labelled ESTIMATED rather than READY or + /// BLOCKED. A device-visibility mask on a host whose GPUs differ in + /// memory is flagged in the report, since which card it selects is not + /// verified there. Any other model the + /// catalog does not carry is reported as undetermined rather than + /// blocked - `rocm serve` still accepts it, this just cannot say in + /// advance whether it fits. + /// + /// For example: `rocm diagnose --model Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M + /// --context-size 32768 --kv-cache-type q8_0`. `HF_ENDPOINT` points the + /// header read at a Hugging Face mirror. /// /// A flag rather than a positional: `diagnose` takes no positional /// today, and one added now would be ambiguous against `--symptom`. #[arg(long, value_name = "MODEL")] model: Option, + /// Context size, in tokens, to estimate a GGUF model for (1 to + /// 16777216; default 4096). + /// + /// The largest context that fits (never more than the context the model + /// was trained for) is reported as well, but only when this machine's + /// GPU memory could be read; without a reading, the memory the model + /// needs is shown and nothing is compared, so no largest context is + /// given. The comparison is against the total dedicated VRAM of the GPU + /// with the most of it that ROCm can see, as for a curated model, not + /// what other processes have left free. Applies to a GGUF outside the + /// catalog (a curated model's memory minimum is fixed, and says so); + /// `rocm serve` has no context-size option yet, so this answers the + /// question without changing what serve does. + #[arg( + long, + value_name = "TOKENS", + requires = "model", + value_parser = clap::value_parser!(u64).range(1..=rocm_core::gguf_fit::MAX_CONTEXT_SIZE) + )] + context_size: Option, + /// KV cache element type to estimate for (default f16). + /// + /// A quantized cache is smaller. In llama.cpp a quantized V cache needs + /// flash attention (q8_0 and q4_0 here quantize both K and V), and the + /// estimate assumes flash attention is on. Applies to a GGUF outside the + /// catalog. + #[arg(long, value_enum, value_name = "TYPE", requires = "model")] + kv_cache_type: Option, /// Show the report this machine would contribute, and send nothing. /// /// Nothing leaves the machine: this prints the exact content so it can /// be read before any of it is shared. Hardware that is not on AMD's /// published compatibility matrix produces no report at all. /// + /// `--model` is not consulted with `--report`: a report carries no model + /// verdict, so nothing is looked up for it and nothing is fetched. A + /// note on stderr says that `--model` (and `--context-size` and + /// `--kv-cache-type`, if given) were ignored. + /// /// Not combinable with `--distro`: a report describes this machine, and /// a WSL distribution reached remotely is not fully examined (see /// `--distro`'s own help), so it cannot back the disclosure guard's @@ -1892,6 +1940,111 @@ mod tests { } } + /// `rocm diagnose --model` tells the user that `rocm serve` takes no context + /// size, KV cache type or GPU layer count and passes none to the engine + /// (`SERVE_OPTIONS_NOTE` and `SERVE_CONTEXT_NOTE` in `gguf_fit.rs`). This + /// fails when `rocm serve` gains one of them, so the notes are updated with it. + #[test] + fn serve_takes_no_estimate_options_yet() { + let serve = |extra: &[&str]| { + let mut argv = vec!["rocm", "serve", "o/r:Q4_K_M"]; + argv.extend_from_slice(extra); + Cli::try_parse_from(argv) + }; + assert!(serve(&[]).is_ok(), "the model alone parses"); + for option in [ + &["--context-size", "4096"][..], + &["--ctx-size", "4096"][..], + &["--kv-cache-type", "q8_0"][..], + &["--n-gpu-layers", "20"][..], + ] { + assert!(serve(option).is_err(), "rocm serve now takes {option:?}"); + } + } + + fn parse_diagnose(args: &[&str]) -> Result { + let mut argv = vec!["rocm", "diagnose"]; + argv.extend_from_slice(args); + Cli::try_parse_from(argv) + } + + // The context options describe the model being asked about, so without + // `--model` they would be accepted and silently do nothing. + #[test] + fn diagnose_context_options_are_refused_without_a_model() { + for args in [ + &["--context-size", "8192"][..], + &["--kv-cache-type", "q8_0"][..], + ] { + let error = parse_diagnose(args).expect_err("an option with nothing to apply to"); + assert_eq!( + error.kind(), + clap::error::ErrorKind::MissingRequiredArgument, + "{args:?}" + ); + assert!(error.to_string().contains("--model"), "{args:?}: {error}"); + } + } + + #[test] + fn diagnose_context_options_reach_the_request_and_bad_values_are_refused() { + let cli = parse_diagnose(&[ + "--model", + "o/r:Q4_K_M", + "--context-size", + "32768", + "--kv-cache-type", + "q4_0", + ]) + .expect("a model with both options parses"); + match cli.command { + Some(Command::Diagnose { + context_size, + kv_cache_type, + .. + }) => { + assert_eq!(context_size, Some(32768)); + assert_eq!( + kv_cache_type.map(rocm_core::gguf_fit::KvCacheType::from), + Some(rocm_core::gguf_fit::KvCacheType::Q4_0) + ); + } + other => panic!("expected diagnose, got {other:?}"), + } + // The smallest size accepted, beside the 0 refused below. + assert!(parse_diagnose(&["--model", "o/r:Q4_K_M", "--context-size", "1"]).is_ok()); + for bad in [ + &["--context-size", "0"][..], + &["--context-size", "-1"][..], + &["--context-size", "lots"][..], + &["--context-size", "18446744073709551615"][..], + &["--context-size", "16777217"][..], + &["--kv-cache-type", "q5_0"][..], + ] { + let mut args = vec!["--model", "o/r:Q4_K_M"]; + args.extend_from_slice(bad); + assert!(parse_diagnose(&args).is_err(), "{bad:?} must not parse"); + } + } + + // The largest context size the flag takes is the one the estimate is made for, + // not a number of its own: a figure past it would print an absurd memory + // need, so clap refuses it before anything is estimated. + #[test] + fn the_largest_context_size_accepted_is_the_largest_the_estimate_makes() { + let largest = rocm_core::gguf_fit::MAX_CONTEXT_SIZE.to_string(); + let cli = parse_diagnose(&["--model", "o/r:Q4_K_M", "--context-size", &largest]) + .expect("the cap itself parses"); + match cli.command { + Some(Command::Diagnose { context_size, .. }) => { + assert_eq!(context_size, Some(rocm_core::gguf_fit::MAX_CONTEXT_SIZE)); + } + other => panic!("expected diagnose, got {other:?}"), + } + let over = (rocm_core::gguf_fit::MAX_CONTEXT_SIZE + 1).to_string(); + assert!(parse_diagnose(&["--model", "o/r:Q4_K_M", "--context-size", &over]).is_err()); + } + // `--yes` approves a restart, so on its own it asks for something that was // never requested. Mutation: removing `requires = "restart_services"` from // either subcommand's `yes` makes that subcommand's argv parse and fails diff --git a/apps/rocm/src/diagnose_cmd.rs b/apps/rocm/src/diagnose_cmd.rs index 0d53e615a6..1ee5fc2a07 100644 --- a/apps/rocm/src/diagnose_cmd.rs +++ b/apps/rocm/src/diagnose_cmd.rs @@ -8,11 +8,12 @@ //! which acts on a delivery decision with an injected `Opener`, and //! `established_entry` picks the catalog entry a report names. //! -//! Mechanically relocated from `main.rs` with no behavior change; see -//! `docs/architecture.md`. `diagnose` and `fix` are `pub(crate)`: `dispatch()` -//! calls them through the re-import `use crate::diagnose_cmd::{diagnose, -//! fix};`. The other three functions have no caller outside this module and -//! stay private. `diagnose` gets its `--model` verdict from +//! Relocated from `main.rs` with no behavior change; see +//! `docs/architecture.md`. Since then `--model` has gained the GGUF estimate's +//! flags, and `--report` ignores a model question and says so on stderr. +//! `diagnose` and `fix` are `pub(crate)`: `dispatch()` calls them through the +//! re-import `use crate::diagnose_cmd::{diagnose, fix};`. The other three +//! functions have no caller outside this module and stay private. `diagnose` gets its `--model` verdict from //! `model_fit.rs`'s `assess_model_on_this_host`, and `fix` reports a failing //! exit code to `main()` through `FixExitCode`, which stays in `main.rs` with //! the rest of the exit-code plumbing and is reached as `crate::FixExitCode`. @@ -31,6 +32,7 @@ use serde::Serialize; use crate::FixExitCode; use crate::cli_report::{JsonOrText, print_json_or_text}; +use crate::diagnose_model; use crate::model_fit::assess_model_on_this_host; pub(crate) fn diagnose( @@ -38,7 +40,7 @@ pub(crate) fn diagnose( top: usize, json: bool, distro: Option, - model: Option, + model: Option, report_requested: bool, send: bool, ) -> Result<()> { @@ -95,8 +97,18 @@ pub(crate) fn diagnose( .as_ref() .is_some_and(|wsl| !wsl.locally_probed); let mut report = rocm_core::run_diagnose(&examination, &symptom.unwrap_or_default()); - if let Some(model_ref) = &model { - report.model = Some(assess_model_on_this_host(model_ref, &examination)); + // A prepared report carries no model verdict, and a GGUF estimate may read + // the model's header from the network -- which `--report` promises it never + // does -- so under `--report` the model is not assessed at all. + if report_requested && let Some(query) = &model { + eprintln!("{}", diagnose_model::report_ignores_model_note(query)); + } + if let Some(query) = model.as_ref().filter(|_| !report_requested) { + report.model = Some(assess_model_on_this_host( + &query.model_ref, + &examination, + &query.context, + )); } if report_requested { return show_prepared_report(&examination, &report, json, send); @@ -570,6 +582,7 @@ mod tests { route: None, undetermined_reason: None, warnings: Vec::new(), + estimate: None, }; let readiness_text = rocm_core::model_readiness::render_model_readiness_text(&readiness); report.model = Some(readiness); diff --git a/apps/rocm/src/diagnose_model.rs b/apps/rocm/src/diagnose_model.rs new file mode 100644 index 0000000000..789b61136f --- /dev/null +++ b/apps/rocm/src/diagnose_model.rs @@ -0,0 +1,501 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The binary's side of `rocm diagnose --model`'s GGUF context-size estimate. +//! +//! The estimate itself lives in `rocm-core` ([`rocm_core::model_readiness`] and +//! the `gguf*` modules). What is here is what only the binary knows: the flags +//! as clap parsed them ([`KvCacheTypeArg`], [`ModelQuery`]), what `--report` +//! says when it is handed a model question it will not answer, and the mask +//! caveat: whether a device-visibility variable is set (read here), over the +//! `amd-smi` rows the host's memory was classified from (passed in). +//! +//! The `Diagnose` command variant and its dispatch arm stay in `main.rs` with the +//! rest of the command tree; `diagnose()` is in `diagnose_cmd.rs`, and the host +//! assessment that calls [`attach_gguf_estimate`] in `model_fit.rs`. + +use crate::gpu_policy::GpuVramUsage; +use clap::ValueEnum; +use rocm_core::gguf_fit::KvCacheType; +use rocm_core::gguf_source::GgufSource; +use rocm_core::model_readiness::{ + ContextRequest, HostFacts, ModelReadiness, UndeterminedReason, attach_context_estimate, + note_unverified_mask, +}; + +/// KV cache element type for `rocm diagnose --model`'s GGUF estimate. +#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)] +pub(crate) enum KvCacheTypeArg { + /// 16-bit floats (llama.cpp's default). + #[value(name = "f16")] + F16, + /// 8-bit quantized: about half the size of f16. + #[value(name = "q8_0")] + Q8_0, + /// 4-bit quantized: about a quarter the size of f16. + #[value(name = "q4_0")] + Q4_0, +} + +impl From for KvCacheType { + fn from(arg: KvCacheTypeArg) -> Self { + match arg { + KvCacheTypeArg::F16 => Self::F16, + KvCacheTypeArg::Q8_0 => Self::Q8_0, + KvCacheTypeArg::Q4_0 => Self::Q4_0, + } + } +} + +/// What `rocm diagnose --model` was asked about. +pub(crate) struct ModelQuery { + pub(crate) model_ref: String, + pub(crate) context: ContextRequest, +} + +impl ModelQuery { + /// The question `--model` asks, with `--context-size` and `--kv-cache-type` + /// as given (clap refuses either without `--model`). + pub(crate) fn new( + model_ref: String, + context_size: Option, + kv_cache_type: Option, + ) -> Self { + Self { + model_ref, + context: ContextRequest { + context_size, + kv_cache_type: kv_cache_type.map(Into::into), + }, + } + } +} + +/// What `--report` says to stderr when it was given a model question: the report +/// is about this machine, so the model flags did nothing, and nothing about the +/// model was fetched. A flag that is accepted and silently has no effect reads as +/// an answer that was never asked for. +pub(crate) fn report_ignores_model_note(query: &ModelQuery) -> String { + let mut ignored = vec!["--model"]; + if query.context.context_size.is_some() { + ignored.push("--context-size"); + } + if query.context.kv_cache_type.is_some() { + ignored.push("--kv-cache-type"); + } + let verb = if ignored.len() == 1 { "was" } else { "were" }; + format!( + "note: --report describes this machine, not a model, so {} {verb} ignored and nothing \ + about the model was fetched", + ignored.join(", ") + ) +} + +/// Add the GGUF estimate to a readiness answer, and the mask caveat that goes +/// with it. +/// +/// `vram` is the `amd-smi` reading the host's memory was classified from, passed +/// in rather than read again: each read is a subprocess. +/// +/// A header read from the hub (a listing, a `HEAD`, then range requests, within +/// a two-minute budget) can take a while with nothing printed, so a spinner +/// covers it. It is drawn on stderr and only when stderr is a terminal, so +/// `--json` and piped output are unchanged. +pub(crate) fn attach_gguf_estimate( + readiness: &mut ModelReadiness, + context: &ContextRequest, + host: &HostFacts, + gguf_source: &dyn GgufSource, + vram: Option<&[GpuVramUsage]>, +) { + let _spinner = reads_from_the_hub(readiness).then(|| { + crate::cli_progress::AnimatedSpinner::start(format!( + "Reading the GGUF header of {} from Hugging Face...", + readiness.model_ref + )) + }); + attach_gguf_estimate_with( + readiness, + context, + host, + gguf_source, + vram, + device_visibility_mask_is_set(), + ); +} + +/// Whether estimating this answer reads a file header from the hub: a catalog +/// miss named as `owner/repo:QUANT` (or `owner/repo:file.gguf`). A curated model, +/// a local path and a bare `owner/repo` make no request. +fn reads_from_the_hub(readiness: &ModelReadiness) -> bool { + readiness.undetermined_reason == Some(UndeterminedReason::ModelNotCurated) + && readiness.canonical_model_id.is_none() + && matches!( + rocm_core::gguf_source::parse_gguf_ref(&readiness.model_ref), + Some(rocm_core::gguf_source::GgufRef::Hub { + variant: Some(_), + .. + }) + ) +} + +/// [`attach_gguf_estimate`] with whether a mask is set passed in, so a test can +/// drive it without touching the process environment. +fn attach_gguf_estimate_with( + readiness: &mut ModelReadiness, + context: &ContextRequest, + host: &HostFacts, + gguf_source: &dyn GgufSource, + vram: Option<&[GpuVramUsage]>, + mask_set: bool, +) { + // The one step that can touch a file or the network, and only for a GGUF + // reference the catalog does not carry; it reads the header, not the weights. + attach_context_estimate(readiness, context, host, gguf_source); + // Which card a device-visibility mask selects, in `amd-smi`'s numbering, has + // not been verified on a multi-GPU machine, so an estimate that compared + // memory says so when it could have compared the wrong card. After the + // estimate, never before: the caveat is attached to the estimate it doubts. + note_mask_for_rows(readiness, vram, mask_set); +} + +/// [`note_unverified_mask`] over the rows `amd-smi` reported. +fn note_mask_for_rows( + readiness: &mut ModelReadiness, + vram: Option<&[GpuVramUsage]>, + mask_set: bool, +) { + let totals: Vec = vram + .iter() + .flat_map(|rows| rows.iter()) + .map(|usage| usage.total_mb) + .collect(); + note_unverified_mask(readiness, mask_set, &totals); +} + +/// Whether any of the variables that hide GPUs from the runtime is set, even to +/// nothing (which hides every one). +/// +/// Wider than the mask [`rocm_core::usable_amd_gpu_indices`] applies, which +/// reads `HIP_VISIBLE_DEVICES` and `ROCR_VISIBLE_DEVICES` only: the HIP runtime +/// also honours `CUDA_VISIBLE_DEVICES`, which the memory classification does +/// not, so a figure compared under it is just as open to doubt. +fn device_visibility_mask_is_set() -> bool { + mask_set_where(|name| std::env::var_os(name).is_some()) +} + +/// The variables that hide GPUs from the runtime. The caveat text +/// ([`rocm_core::model_readiness::MASK_UNVERIFIED_NOTE`]) names each of them; +/// a test holds the two together. +const DEVICE_VISIBILITY_VARIABLES: [&str; 3] = [ + "HIP_VISIBLE_DEVICES", + "ROCR_VISIBLE_DEVICES", + "CUDA_VISIBLE_DEVICES", +]; + +/// Whether `is_set` holds for any of [`DEVICE_VISIBILITY_VARIABLES`]. +fn mask_set_where(is_set: impl Fn(&str) -> bool) -> bool { + DEVICE_VISIBILITY_VARIABLES.iter().any(|name| is_set(name)) +} + +#[cfg(test)] +mod tests { + use super::*; + use rocm_core::gguf_fit::{ContextEstimate, FitRequest}; + use rocm_core::model_readiness::{AcceleratorMemory, ModelCatalogSource}; + + /// The spinner is for a hub read only: a catalog miss named `owner/repo:QUANT`. + /// A local path, a bare repository, a name that is not a GGUF reference and a + /// curated model make no request, so they get none. + #[test] + fn only_a_hub_gguf_outside_the_catalog_is_read_from_the_hub() { + let registry = rocm_core::builtin_model_recipe_registry(); + let host = HostFacts { + accelerator_memory: AcceleratorMemory::Dedicated(24.0), + system_ram_gib: None, + }; + let answer = |model: &str| { + crate::model_fit::assess_model_for_host( + model, + &ModelCatalogSource::Available(®istry), + &host, + None, + None, + ) + }; + assert!(reads_from_the_hub(&answer( + "some-lab/never-curated-7b:Q4_K_M" + ))); + assert!(reads_from_the_hub(&answer( + "some-lab/never-curated-7b:dir/m-Q4_K_M.gguf" + ))); + for model in [ + "models/m.gguf", + "some-lab/never-curated-7b", + "not-a-gguf-at-all", + ] { + assert!(!reads_from_the_hub(&answer(model)), "{model}"); + } + let curated = ®istry.recipes[0].canonical_model_id; + assert!(!reads_from_the_hub(&answer(curated)), "{curated}"); + } + + /// A readiness whose estimate compared memory, built without a network or a file. + fn readiness_with_a_compared_estimate() -> ModelReadiness { + let mut readiness = crate::model_fit::assess_model_for_host( + "some-lab/never-curated-7b:Q4_K_M", + &ModelCatalogSource::Unreachable { + detail: "not needed here".to_owned(), + }, + &HostFacts { + accelerator_memory: AcceleratorMemory::Dedicated(24.0), + system_ram_gib: None, + }, + None, + None, + ); + let mut estimate = ContextEstimate::unreadable( + "x.gguf", + &FitRequest { + context_size: 4096, + context_size_defaulted: true, + kv_cache_type: KvCacheType::F16, + }, + &anyhow::anyhow!("not read"), + ); + estimate.available_gib = Some(24.0); + readiness.estimate = Some(estimate); + readiness + } + + fn gpu_row(index: u32, total_gib: u64) -> GpuVramUsage { + GpuVramUsage { + index, + used_mb: 0, + total_mb: total_gib * 1024, + } + } + + /// `--report` ignores the model flags it was given, and says which, so a flag + /// that did nothing is not taken for one that was answered. + #[test] + fn the_report_note_names_exactly_the_model_flags_that_were_ignored() { + let note = |context_size: Option, kv: Option| { + report_ignores_model_note(&ModelQuery::new("o/r:Q4_K_M".to_owned(), context_size, kv)) + }; + let alone = note(None, None); + assert!(alone.contains("--model was ignored"), "{alone}"); + assert!(!alone.contains("--context-size"), "{alone}"); + assert!(!alone.contains("--kv-cache-type"), "{alone}"); + let both = note(Some(4096), Some(KvCacheTypeArg::Q8_0)); + assert!( + both.contains("--model, --context-size, --kv-cache-type were ignored"), + "{both}" + ); + let size_only = note(Some(4096), None); + assert!( + size_only.contains("--model, --context-size were ignored") + && !size_only.contains("--kv-cache-type"), + "{size_only}" + ); + assert!( + both.contains("nothing about the model was fetched"), + "{both}" + ); + } + + /// The flags reach the request as given. + #[test] + fn the_query_carries_the_flags_as_given() { + let query = ModelQuery::new( + "o/r:Q4_K_M".to_owned(), + Some(8192), + Some(KvCacheTypeArg::Q4_0), + ); + assert_eq!(query.model_ref, "o/r:Q4_K_M"); + assert_eq!( + query.context, + ContextRequest { + context_size: Some(8192), + kv_cache_type: Some(KvCacheType::Q4_0), + } + ); + let unasked = ModelQuery::new("o/r:Q4_K_M".to_owned(), None, None); + assert_eq!(unasked.context, ContextRequest::default()); + } + + /// The glue between `amd-smi`'s rows and the caveat: cards of two sizes and a + /// mask say it, and every other host does not. + #[test] + fn the_unverified_mask_note_reaches_an_estimate_when_the_gpus_differ() { + let noted = |rows: Option<&[GpuVramUsage]>, mask_set: bool| { + let mut readiness = readiness_with_a_compared_estimate(); + note_mask_for_rows(&mut readiness, rows, mask_set); + readiness + .estimate + .unwrap() + .notes + .iter() + .any(|note| note.contains("has not been verified")) + }; + let mixed = [gpu_row(0, 24), gpu_row(1, 8)]; + let same = [gpu_row(0, 24), gpu_row(1, 24)]; + assert!(noted(Some(&mixed), true)); + assert!(!noted(Some(&mixed), false), "no mask, nothing to doubt"); + assert!(!noted(Some(&same), true), "the same card either way"); + assert!(!noted(None, true), "no rows, no figure compared"); + } + + /// Each variable on its own counts as a mask, and the caveat names each. + #[test] + fn each_visibility_variable_alone_is_a_mask_and_the_caveat_names_it() { + // The three the runtime honours, named here so dropping one from the + // constant fails; and every entry of the constant, so adding one that + // the caveat does not name fails too. + for variable in [ + "HIP_VISIBLE_DEVICES", + "ROCR_VISIBLE_DEVICES", + "CUDA_VISIBLE_DEVICES", + ] { + assert!( + DEVICE_VISIBILITY_VARIABLES.contains(&variable), + "{variable}" + ); + } + for variable in DEVICE_VISIBILITY_VARIABLES { + assert!(mask_set_where(|name| name == variable), "{variable}"); + assert!( + rocm_core::model_readiness::MASK_UNVERIFIED_NOTE.contains(variable), + "{variable}" + ); + } + assert!(!mask_set_where(|_| false)); + } + + /// A GGUF of a few kilobytes: 32 layers, 8 KV heads of width 128, so its + /// KV cache is real and an estimate compares it against the card. + fn tiny_gguf() -> Vec { + fn string(out: &mut Vec, value: &str) { + out.extend_from_slice(&(value.len() as u64).to_le_bytes()); + out.extend_from_slice(value.as_bytes()); + } + fn entry_u32(out: &mut Vec, key: &str, value: u32) { + string(out, key); + out.extend_from_slice(&4_u32.to_le_bytes()); + out.extend_from_slice(&value.to_le_bytes()); + } + let mut metadata = Vec::new(); + string(&mut metadata, "general.architecture"); + metadata.extend_from_slice(&8_u32.to_le_bytes()); + string(&mut metadata, "t"); + entry_u32(&mut metadata, "t.block_count", 32); + entry_u32(&mut metadata, "t.embedding_length", 4096); + entry_u32(&mut metadata, "t.feed_forward_length", 14336); + entry_u32(&mut metadata, "t.attention.head_count", 32); + entry_u32(&mut metadata, "t.attention.head_count_kv", 8); + entry_u32(&mut metadata, "t.context_length", 8192); + let mut file = Vec::new(); + file.extend_from_slice(b"GGUF"); + file.extend_from_slice(&3_u32.to_le_bytes()); + file.extend_from_slice(&1_u64.to_le_bytes()); + file.extend_from_slice(&7_u64.to_le_bytes()); + file.extend_from_slice(&metadata); + string(&mut file, "w"); + file.extend_from_slice(&1_u32.to_le_bytes()); + file.extend_from_slice(&1024_u64.to_le_bytes()); + file.extend_from_slice(&0_u32.to_le_bytes()); + file.extend_from_slice(&0_u64.to_le_bytes()); + file.resize(file.len().div_ceil(32) * 32, 0); + file + } + + struct OneFile(Vec); + + impl GgufSource for OneFile { + fn open( + &self, + _reference: &rocm_core::gguf_source::GgufRef, + ) -> anyhow::Result { + Ok(rocm_core::gguf_source::OpenedGguf { + reader: Box::new(std::io::Cursor::new(self.0.clone())), + size: self.0.len() as u64 + 1024 * 1024 * 1024, + description: "t.gguf".to_owned(), + }) + } + } + + /// The glue end to end: an estimate is read through the source and compared + /// with the card, and the mask caveat lands on it -- which it can only do + /// once the estimate exists, so the order of the two steps matters. + #[test] + fn attaching_an_estimate_under_a_mask_on_mixed_gpus_carries_the_caveat() { + let registry = rocm_core::builtin_model_recipe_registry(); + let host = HostFacts { + accelerator_memory: AcceleratorMemory::Dedicated(24.0), + system_ram_gib: Some(64.0), + }; + let attach = |mask_set: bool, rows: &[GpuVramUsage]| { + let mut readiness = crate::model_fit::assess_model_for_host( + "some-lab/never-curated-7b:Q4_K_M", + &ModelCatalogSource::Available(®istry), + &host, + None, + None, + ); + attach_gguf_estimate_with( + &mut readiness, + &ContextRequest::default(), + &host, + &OneFile(tiny_gguf()), + Some(rows), + mask_set, + ); + readiness + }; + let mixed = [gpu_row(0, 24), gpu_row(1, 8)]; + let noted = attach(true, &mixed); + let estimate = noted.estimate.as_ref().expect("an estimate was attached"); + // Sized and compared, not just tagged with the card: `available_gib` is + // set before sizing, so on its own it would hold for an estimate that + // failed. + assert_eq!( + estimate.outcome, + rocm_core::gguf_fit::EstimateOutcome::Fits, + "{estimate:?}" + ); + assert!(estimate.total_gib.is_some(), "{estimate:?}"); + assert!( + estimate + .notes + .iter() + .any(|note| note == rocm_core::model_readiness::MASK_UNVERIFIED_NOTE), + "{:?}", + estimate.notes + ); + let unmasked = attach(false, &mixed); + assert!( + !unmasked + .estimate + .expect("an estimate") + .notes + .iter() + .any(|note| note == rocm_core::model_readiness::MASK_UNVERIFIED_NOTE) + ); + } + + // Every spelling the flag takes maps to the type the estimate uses, so a + // mismatch between the two enums cannot silently estimate another cache. + #[test] + fn every_kv_cache_flag_value_names_the_matching_estimate_type() { + for arg in KvCacheTypeArg::value_variants() { + let name = arg + .to_possible_value() + .expect("named") + .get_name() + .to_owned(); + let kind = KvCacheType::from(*arg); + assert_eq!(kind.name(), name); + } + } +} diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index e879060b5c..61a6f67ff8 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -24,6 +24,7 @@ mod comfyui; mod config_cmd; mod dash; mod dash_seam; +mod diagnose_model; // Unix-only: its generators plant symlinks with `std::os::unix::fs::symlink`. #[cfg(all(test, unix))] mod deletion_properties; @@ -685,9 +686,16 @@ fn dispatch(cli: Cli) -> Result<()> { json, distro, model, + context_size, + kv_cache_type, report, send, - }) => diagnose(symptom, top, json, distro, model, report, send), + }) => { + let model = model.map(|model_ref| { + diagnose_model::ModelQuery::new(model_ref, context_size, kv_cache_type) + }); + diagnose(symptom, top, json, distro, model, report, send) + } // Keep this error chained rather than discarding it into a fresh // `anyhow!(...)` (e.g. via a `.map_err` that restringifies it) -- see // `FixExitCode`'s doc comment for why that would silently break its diff --git a/apps/rocm/src/model_fit.rs b/apps/rocm/src/model_fit.rs index 4ce02b58b6..2d4fadeb27 100644 --- a/apps/rocm/src/model_fit.rs +++ b/apps/rocm/src/model_fit.rs @@ -15,8 +15,11 @@ //! out for the platform (native Windows for vLLM and AIM, an unreachable Docker //! daemon for AIM) and word the reason. //! -//! Mechanically relocated from `main.rs` with no behavior change; see -//! `docs/architecture.md`. It is a module of its own, rather than part of the +//! Relocated from `main.rs` with no behavior change; see +//! `docs/architecture.md`. Since then the memory classification answers "no GPU +//! is visible" when a device mask hides every card (it reported a card of 0 GiB, +//! or an APU's unreadable pool), and the assessment attaches the GGUF estimate. +//! It is a module of its own, rather than part of the //! `rocm diagnose` handler, so `examine_report_text.rs` (which reads //! `PlatformRuleOut` and `engine_ruled_out_by_platform`) and //! `model_catalog_text.rs` (which reads `PlatformRuleOut` and @@ -36,13 +39,16 @@ //! `model_fit::tests::docker_unreachable_subprocess_helper`; that path moved //! with the helper. +use rocm_core::gguf_source::{GgufSource, LiveGgufSource}; use rocm_core::model_readiness::{ - AcceleratorMemory, HostEngineChoice, HostFacts, ModelCatalogSource, ModelReadiness, + AcceleratorMemory, ContextRequest, HostEngineChoice, HostFacts, ModelCatalogSource, + ModelReadiness, }; use rocm_core::{ AppPaths, ModelRecipeRecord, RocmCliConfig, detect_host_gpu_summary, load_model_recipe_registry, }; +use crate::diagnose_model; use crate::gpu_policy::{GpuVramUsage, gpu_vram_usage, vram_capacity_is_meaningful}; use crate::serve_cmd::select_serve_engine; @@ -56,9 +62,16 @@ use crate::serve_cmd::select_serve_engine; pub(crate) fn assess_model_on_this_host( model_ref: &str, examination: &rocm_core::Examination, + context: &ContextRequest, ) -> ModelReadiness { let paths = AppPaths::discover().ok(); - assess_model_with_host_paths(model_ref, examination, paths.as_ref()) + assess_model_with_host_paths( + model_ref, + examination, + paths.as_ref(), + context, + &LiveGgufSource::new(), + ) } /// Testable core of [`assess_model_on_this_host`]. @@ -77,16 +90,25 @@ fn assess_model_with_host_paths( model_ref: &str, examination: &rocm_core::Examination, paths: Option<&AppPaths>, + context: &ContextRequest, + gguf_source: &dyn GgufSource, ) -> ModelReadiness { let host_gpu_summary = detect_host_gpu_summary(paths); + // Read once: the memory classification and the mask caveat below both need + // the rows, and each read is an `amd-smi` subprocess. + let vram = gpu_vram_usage(); let host = HostFacts { - accelerator_memory: host_accelerator_memory(&host_gpu_summary, examination), + accelerator_memory: host_accelerator_memory( + &host_gpu_summary, + examination, + vram.as_deref(), + ), system_ram_gib: rocm_core::detect_system_ram_gib(), }; let configured_engine = paths .and_then(|paths| RocmCliConfig::load(paths).ok()) .and_then(|config| config.default_engine); - match load_model_recipe_registry() { + let mut readiness = match load_model_recipe_registry() { Ok(registry) => assess_model_for_host( model_ref, &ModelCatalogSource::Available(®istry), @@ -103,7 +125,15 @@ fn assess_model_with_host_paths( configured_engine.as_deref(), Some(&host_gpu_summary), ), - } + }; + diagnose_model::attach_gguf_estimate( + &mut readiness, + context, + &host, + gguf_source, + vram.as_deref(), + ); + readiness } /// The memory pool an engine on this host would allocate a model from. @@ -147,9 +177,10 @@ fn examination_reports_amd_gpu(examination: &rocm_core::Examination) -> bool { fn host_accelerator_memory( host_gpu_summary: &rocm_core::HostGpuSummary, examination: &rocm_core::Examination, + vram: Option<&[GpuVramUsage]>, ) -> AcceleratorMemory { classify_accelerator_memory( - gpu_vram_usage().as_deref(), + vram, host_gpu_summary.gfx_target.as_deref(), examination_reports_amd_gpu(examination), rocm_core::usable_amd_gpu_indices().as_deref(), @@ -190,8 +221,11 @@ fn classify_accelerator_memory( .iter() .filter(|usage| usable_indices.is_none_or(|indices| indices.contains(&usage.index))) .map(|usage| usage.total_mb as f64 / 1024.0) - .fold(0.0_f64, f64::max); - return AcceleratorMemory::Dedicated(max_gib); + .reduce(f64::max); + // A mask that leaves no card is "no GPU is visible", the same answer + // `has_usable_amd_gpu` gives -- not a card with 0 GiB, which a verdict + // would then print as a measured figure ("this host offers 0 GiB"). + return max_gib.map_or(AcceleratorMemory::None, AcceleratorMemory::Dedicated); } // This is an APU: the carve-out `amd-smi` reports above is the wrong pool, // per `vram_capacity_is_meaningful`'s own doc. The right pool is the GTT @@ -210,6 +244,14 @@ fn classify_accelerator_memory( // read, this reports `UnifiedMemoryUnreadable`, so the verdict reads // `Undetermined` with no dead-end command attached, rather than a // confident wrong one. + // + // A mask that leaves the APU's one device out is "no GPU is visible", as + // for a discrete card above: that is the answer a command can change. + let masked_out = usable_indices + .is_some_and(|indices| !vram.iter().any(|usage| indices.contains(&usage.index))); + if masked_out { + return AcceleratorMemory::None; + } AcceleratorMemory::UnifiedMemoryUnreadable } @@ -222,7 +264,7 @@ fn classify_accelerator_memory( /// reported as such rather than silently swapped for another: `serve` does not /// fall back either, and a verdict that pretended otherwise would be wrong in /// the user's favour. -fn assess_model_for_host( +pub(crate) fn assess_model_for_host( model_ref: &str, catalog: &ModelCatalogSource<'_>, host: &HostFacts, @@ -571,8 +613,13 @@ mod tests { let summary = detect_host_gpu_summary(Some(&paths)); let expected = select_serve_engine(None, Some("vllm"), Some(recipe), Some(&summary)); - let readiness = - assess_model_with_host_paths(&recipe.canonical_model_id, &examination, Some(&paths)); + let readiness = assess_model_with_host_paths( + &recipe.canonical_model_id, + &examination, + Some(&paths), + &ContextRequest::default(), + &LiveGgufSource::new(), + ); let _ = std::fs::remove_dir_all(&dir); @@ -623,6 +670,16 @@ mod tests { "an APU's memory pool must read UnifiedMemoryUnreadable, not a fabricated figure, \ and not the generic Unknown either: {result:?}" ); + // A mask that leaves the APU visible changes nothing; one that hides it + // is "no GPU is visible", as for a discrete card. + assert_eq!( + classify_accelerator_memory(Some(&apu_carveout), Some("gfx1151"), true, Some(&[0])), + AcceleratorMemory::UnifiedMemoryUnreadable + ); + assert_eq!( + classify_accelerator_memory(Some(&apu_carveout), Some("gfx1151"), true, Some(&[])), + AcceleratorMemory::None + ); } /// No VRAM telemetry at all, but the examination positively saw an AMD @@ -719,6 +776,44 @@ mod tests { ); } + /// A mask that hides every card is "no GPU is visible", not a card with no + /// memory: the fold over the visible rows used to start from 0.0 and report + /// `Dedicated(0.0)`, which a verdict printed as a measured "0 MiB". + #[test] + fn classify_accelerator_memory_reports_none_when_the_mask_leaves_no_gpu() { + let vram = [ + GpuVramUsage { + index: 0, + used_mb: 0, + total_mb: 24 * 1024, + }, + GpuVramUsage { + index: 1, + used_mb: 0, + total_mb: 8 * 1024, + }, + ]; + // `HIP_VISIBLE_DEVICES=""`: no usable device at all. + assert_eq!( + classify_accelerator_memory(Some(&vram), Some("gfx942"), true, Some(&[])), + AcceleratorMemory::None + ); + // A mask naming only a device `amd-smi` has no row for. + assert_eq!( + classify_accelerator_memory(Some(&vram), Some("gfx942"), true, Some(&[5])), + AcceleratorMemory::None + ); + // Unchanged: no mask, and a mask that leaves a card. + assert_eq!( + classify_accelerator_memory(Some(&vram), Some("gfx942"), true, None), + AcceleratorMemory::Dedicated(24.0) + ); + assert_eq!( + classify_accelerator_memory(Some(&vram), Some("gfx942"), true, Some(&[1])), + AcceleratorMemory::Dedicated(8.0) + ); + } + /// The end-to-end proof, one layer above `classify_accelerator_memory`'s own /// unit tests: not just that the function picks the right number, but that /// picking the wrong one would have changed the verdict a user actually diff --git a/apps/rocm/tests/diagnose_model_sends_no_token.rs b/apps/rocm/tests/diagnose_model_sends_no_token.rs new file mode 100644 index 0000000000..d64fa89193 --- /dev/null +++ b/apps/rocm/tests/diagnose_model_sends_no_token.rs @@ -0,0 +1,220 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! `rocm diagnose --model owner/repo:tag` reads a GGUF header from the hub and +//! says it sends no token. That is a promise that something does not happen, so +//! it is checked on the wire: with every Hugging Face token variable set in the +//! command's environment, the listing, the `HEAD` and the range requests that +//! reach a stand-in hub carry no `Authorization` and no `Cookie` header. + +use std::io::{Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::process::{Command, Stdio}; +use std::sync::{Arc, Mutex}; +use std::thread; + +/// A small but complete GGUF header: the shape keys an estimate reads and one +/// tensor record, so the read ends in a sized estimate. 2 layers of 4 KV heads, +/// key and value width 64 each: 1024 cache elements, 2048 bytes at f16, per +/// token of context. +fn gguf_header() -> Vec { + fn string(out: &mut Vec, value: &str) { + out.extend_from_slice(&(value.len() as u64).to_le_bytes()); + out.extend_from_slice(value.as_bytes()); + } + fn uint(out: &mut Vec, key: &str, value: u32) { + string(out, key); + out.extend_from_slice(&4_u32.to_le_bytes()); + out.extend_from_slice(&value.to_le_bytes()); + } + let mut metadata = Vec::new(); + string(&mut metadata, "general.architecture"); + metadata.extend_from_slice(&8_u32.to_le_bytes()); + string(&mut metadata, "llama"); + for (key, value) in [ + ("llama.block_count", 2), + ("llama.embedding_length", 256), + ("llama.feed_forward_length", 512), + ("llama.attention.head_count", 4), + ("llama.attention.head_count_kv", 4), + ("llama.attention.key_length", 64), + ("llama.attention.value_length", 64), + ] { + uint(&mut metadata, key, value); + } + let mut bytes = b"GGUF".to_vec(); + bytes.extend_from_slice(&3_u32.to_le_bytes()); + bytes.extend_from_slice(&1_u64.to_le_bytes()); // tensors + bytes.extend_from_slice(&8_u64.to_le_bytes()); // metadata entries + bytes.extend(metadata); + string(&mut bytes, "blk.0.attn_q.weight"); + bytes.extend_from_slice(&2_u32.to_le_bytes()); + bytes.extend_from_slice(&256_u64.to_le_bytes()); + bytes.extend_from_slice(&256_u64.to_le_bytes()); + bytes.extend_from_slice(&0_u32.to_le_bytes()); // f32 + bytes.extend_from_slice(&0_u64.to_le_bytes()); // offset + bytes +} + +/// The KV cache the header above implies at the default 4096 tokens, f16. +const EXPECTED_KV_CACHE_BYTES: u64 = 2 * 4 * (64 + 64) * 2 * 4096; + +/// What the file is said to weigh: larger than the header, so the reader asks +/// for a range and never the whole file. +const FILE_SIZE: u64 = 1 << 20; + +fn answer(mut stream: TcpStream, requests: &Mutex>, header: &[u8]) { + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + if stream.read(&mut byte).unwrap_or(0) == 0 { + return; + } + request.push(byte[0]); + } + let text = String::from_utf8_lossy(&request).into_owned(); + requests.lock().unwrap().push(text.clone()); + let mut words = text.split_whitespace(); + let method = words.next().unwrap_or(""); + let path = words.next().unwrap_or(""); + let range = text.lines().find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("range") + .then(|| value.trim().strip_prefix("bytes=").map(str::to_owned)) + .flatten() + }); + if path.starts_with("/api/models/o/r") { + // No size in the listing, so the file's size is asked for by `HEAD`. + let body = r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf"}]}"#; + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\ + Connection: close\r\n\r\n{body}", + body.len() + ); + } else if method == "HEAD" { + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Length: {FILE_SIZE}\r\nConnection: close\r\n\r\n" + ); + } else if let Some((start, end)) = range.as_deref().and_then(|r| r.split_once('-')) { + let start: u64 = start.parse().unwrap_or(0); + let end: u64 = end.parse().unwrap_or(0).min(FILE_SIZE - 1); + let body: Vec = (start..=end) + .map(|at| { + usize::try_from(at) + .ok() + .and_then(|at| header.get(at)) + .copied() + .unwrap_or(0) + }) + .collect(); + let _ = write!( + stream, + "HTTP/1.1 206 Partial Content\r\nContent-Range: bytes {start}-{end}/{FILE_SIZE}\r\n\ + Content-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + let _ = stream.write_all(&body); + } else { + let _ = write!( + stream, + "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + } +} + +#[test] +fn a_hub_header_read_sends_no_token_even_when_one_is_set() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let hub = format!("http://{}", listener.local_addr().unwrap()); + let requests = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&requests); + thread::spawn(move || { + let header = gguf_header(); + for stream in listener.incoming() { + let Ok(stream) = stream else { break }; + answer(stream, &seen, &header); + } + }); + + let home = tempfile::tempdir().unwrap(); + let output = Command::new(env!("CARGO_BIN_EXE_rocm")) + .args(["diagnose", "--model", "o/r:Q4_K_M", "--json"]) + .env("HF_ENDPOINT", &hub) + .env("HF_TOKEN", "hf_secret_token") + .env("HUGGING_FACE_HUB_TOKEN", "hf_secret_token") + .env("ROCM_CLI_HUGGINGFACE_TOKEN", "hf_secret_token") + .env("ROCM_CLI_CONFIG_DIR", home.path().join("config")) + .env("ROCM_CLI_DATA_DIR", home.path().join("data")) + .env("ROCM_CLI_CACHE_DIR", home.path().join("cache")) + .stdin(Stdio::null()) + .output() + .expect("run rocm diagnose --model"); + + let requests = requests.lock().unwrap().clone(); + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + let first_line = |request: &String| request.lines().next().unwrap_or("").to_owned(); + let lines: Vec = requests.iter().map(first_line).collect(); + // Each kind of request reached the hub, so the check below is not vacuous. + for (kind, found) in [ + ( + "the listing", + lines.iter().any(|l| l.starts_with("GET /api/models/o/r")), + ), + ( + "the HEAD", + lines.iter().any(|l| l.starts_with("HEAD /o/r/resolve/")), + ), + ( + "a range request", + requests.iter().any(|request| { + request.starts_with("GET /o/r/resolve/") + && request.to_ascii_lowercase().contains("\r\nrange: bytes=") + }), + ), + ] { + assert!( + found, + "{kind} never reached the hub: {lines:?}\nstdout: {stdout}\nstderr: {stderr}" + ); + } + for request in &requests { + let lower = request.to_ascii_lowercase(); + assert!( + !lower.contains("\r\nauthorization:") && !lower.contains("\r\ncookie:"), + "a credential went to the hub:\n{request}" + ); + assert!(!request.contains("hf_secret_token"), "{request}"); + assert!( + request + .to_ascii_lowercase() + .contains("\r\nuser-agent: rocm-cli"), + "{request}" + ); + } + + // And the read is the estimate's: the answer is sized from that header. + let report: serde_json::Value = serde_json::from_str(&stdout) + .unwrap_or_else(|error| panic!("{error}\nstdout: {stdout}\nstderr: {stderr}")); + let estimate = &report["model"]["estimate"]; + assert_eq!( + estimate["source"].as_str(), + Some("o/r: m-Q4_K_M.gguf"), + "{report:#}" + ); + assert_eq!( + estimate["architecture"].as_str(), + Some("llama"), + "{report:#}" + ); + assert_eq!(estimate["layers"].as_u64(), Some(2), "{report:#}"); + assert_eq!( + estimate["kv_cache_bytes"].as_u64(), + Some(EXPECTED_KV_CACHE_BYTES), + "{report:#}" + ); + assert!(estimate["total_bytes"].as_u64().is_some(), "{report:#}"); +} diff --git a/crates/rocm-core/src/download.rs b/crates/rocm-core/src/download.rs index 81f2c7d84d..9dd6cc8c80 100644 --- a/crates/rocm-core/src/download.rs +++ b/crates/rocm-core/src/download.rs @@ -442,7 +442,7 @@ fn header_u64(response: &ureq::Response, name: &str) -> Option { } /// First byte offset from a `Content-Range: bytes -/` header. -fn content_range_start(response: &ureq::Response) -> Option { +pub(crate) fn content_range_start(response: &ureq::Response) -> Option { let value = response.header("Content-Range")?; let range = value.trim().strip_prefix("bytes")?.trim_start(); let start = range.split('-').next()?.trim(); diff --git a/crates/rocm-core/src/gguf.rs b/crates/rocm-core/src/gguf.rs new file mode 100644 index 0000000000..694d22a5ca --- /dev/null +++ b/crates/rocm-core/src/gguf.rs @@ -0,0 +1,1611 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Read the shape of a GGUF model from its header, without its weights. +//! +//! A GGUF file starts with a header -- key/value metadata, then one record per +//! tensor -- and only after that, at an aligned offset, the tensor data. The +//! header is what an estimate of memory needs: layer count, attention head +//! counts, head width, and the parameter count summed from the +//! tensor shapes. It is a few megabytes at most (most of it the tokenizer), where +//! the file it describes can be tens of gigabytes, so [`parse_gguf_header`] takes +//! any `Read + Seek` and a caller that can fetch ranges (see +//! [`crate::gguf_source`]) never has to download the weights. +//! +//! The parser is deliberately a reader of *what the estimate needs* and nothing +//! more: arrays are skipped without being stored (a tokenizer is hundreds of +//! thousands of strings), string values are kept only when short, and a layout it +//! does not understand is reported as missing rather than guessed. See +//! for the format. + +use std::collections::BTreeMap; +use std::fmt; +use std::io::{self, Read, Seek, SeekFrom}; + +const MAGIC: [u8; 4] = *b"GGUF"; +/// Version 1 used 32-bit counts and is long obsolete; 2 and 3 differ only in +/// that 3 added big-endian files, which are not read here. +const SUPPORTED_VERSIONS: [u32; 2] = [2, 3]; +const DEFAULT_ALIGNMENT: u64 = 32; +/// A header claiming more entries than this is not one -- real files carry a few +/// dozen metadata keys and some thousands of tensors -- and the bound keeps a +/// hostile or corrupt count from becoming a loop that runs until memory ends. +const MAX_ENTRIES: u64 = 1 << 20; +const MAX_KEY_LEN: u64 = 4096; + +/// The input-embedding tensor. llama.cpp keeps it in host memory when the file +/// has its own [`OUTPUT_NAME`]; with tied embeddings it is the output layer's +/// weight too, on the GPU. +const TOKEN_EMBD_NAME: &str = "token_embd.weight"; +const OUTPUT_NAME: &str = "output.weight"; +/// String values longer than this are skipped rather than stored; the only +/// strings the estimate reads (an architecture name) are far shorter. +const MAX_KEPT_STRING: u64 = 4096; +/// Integer arrays longer than this are skipped rather than stored. The arrays +/// the estimate reads are per-layer (one entry per block); tokenizer arrays are +/// orders of magnitude longer. +const MAX_KEPT_ARRAY: u64 = 4096; +const MAX_TENSOR_DIMS: u32 = 8; +const MAX_ARRAY_DEPTH: u8 = 2; +/// The most a header may span, in bytes, counting everything read or skipped. +/// Real headers are a few megabytes (mostly the tokenizer); a local file has no +/// other bound, and a hostile one could otherwise make the parse walk all of it. +/// The remote reader enforces the same figure on what it fetches. +pub(crate) const MAX_HEADER_BYTES: u64 = 64 * 1024 * 1024; +/// The most layers a model can plausibly have; the largest real ones have a few +/// hundred. Anything above is a corrupt or hostile count, and the estimate loops +/// once per layer. +pub(crate) const MAX_BLOCK_COUNT: u64 = 10_000; +/// How many metadata values of interest a header may carry. A real one has about +/// ten for its architecture; the bound keeps a hostile one from stacking up +/// arrays under thousands of distinct keys. +const MAX_KEPT_KEYS: usize = 64; +/// How many array elements are skipped between checks of the header's size. +const SIZE_CHECK_EVERY: u64 = 4096; +/// The metadata keys (after the architecture prefix) whose values the estimate +/// reads. Everything else is skipped without being stored. +const KEPT_KEY_SUFFIXES: [&str; 9] = [ + ".block_count", + ".embedding_length", + ".attention.head_count", + ".attention.head_count_kv", + ".attention.key_length", + ".attention.value_length", + ".feed_forward_length", + ".context_length", + ".attention.sliding_window", +]; + +fn is_kept_key(key: &str) -> bool { + KEPT_KEY_SUFFIXES.iter().any(|suffix| key.ends_with(suffix)) +} + +/// Why a GGUF header could not be read. +#[derive(Debug)] +pub enum GgufError { + /// The data does not start with the GGUF magic. + NotGguf, + /// A GGUF version this reader does not handle (this includes a big-endian + /// file, which reads as a byte-swapped version). + UnsupportedVersion(u32), + /// The data ended before the header did. + Truncated, + /// The header is structurally wrong. + Malformed(String), + /// The underlying reader failed. + Io(io::Error), +} + +impl fmt::Display for GgufError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::NotGguf => f.write_str("not a GGUF file (bad magic)"), + Self::UnsupportedVersion(version) => { + write!(f, "GGUF version {version} is not supported (2 and 3 are)") + } + Self::Truncated => f.write_str("the GGUF header ends before it is complete"), + Self::Malformed(why) => write!(f, "malformed GGUF header: {why}"), + Self::Io(error) => write!(f, "could not read the GGUF header: {error}"), + } + } +} + +impl std::error::Error for GgufError {} + +impl From for GgufError { + fn from(error: io::Error) -> Self { + if error.kind() == io::ErrorKind::UnexpectedEof { + Self::Truncated + } else { + Self::Io(error) + } + } +} + +/// A per-layer figure: one value for every layer, or one per layer. +/// +/// Models that mix layer kinds (attention interleaved with state-space or +/// sliding-window blocks) store the head counts as an array. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum PerLayer { + Uniform(u64), + Each(Vec), +} + +impl PerLayer { + /// The value for `layer`, or `None` when a per-layer array is too short. + #[must_use] + pub fn at(&self, layer: usize) -> Option { + match self { + Self::Uniform(value) => Some(*value), + Self::Each(values) => values.get(layer).copied(), + } + } +} + +/// What the header says about the model, as far as a memory estimate needs. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GgufHeader { + pub version: u32, + /// `general.architecture`, the prefix every other key of interest carries. + pub architecture: Option, + pub tensor_count: u64, + /// Sum of the elements of every tensor. + pub param_count: u64, + /// Byte offset at which the tensor data starts: the end of the header + /// rounded up to the file's alignment. + pub data_offset: u64, + pub block_count: Option, + pub embedding_length: Option, + pub head_count: Option, + pub head_count_kv: Option, + pub key_length: Option, + pub value_length: Option, + pub feed_forward_length: Option, + /// The context length the model was trained for. + pub context_length: Option, + /// `split.count`: set when the file is one shard of several. + pub split_count: Option, + /// A sliding-window attention key is present: the KV cache of the windowed + /// layers can be smaller than a full-context figure. + pub sliding_window: bool, + /// A state-space (`*.ssm.*`) key is present: some layers keep no KV cache. + pub ssm: bool, + /// A latent-attention (`kv_lora_rank`) key is present: the KV cache is not + /// laid out as heads times head width. + pub latent_attention: bool, + /// An expert count above one is present: a mixture-of-experts model, whose + /// feed-forward width is per expert. + pub moe: bool, + /// Where `token_embd.weight` sits in the tensor data, when the file has one. + pub token_embd: Option, + /// The file has its own `output.weight`. Without one the embeddings are + /// tied: llama.cpp uses `token_embd.weight` for the output layer too, and + /// places that use of it with the output layer, on the GPU. + pub has_output: bool, +} + +/// A tensor's place in the tensor data. +/// +/// That is enough to size it without knowing its quantization: it runs from +/// `offset` to the start of whichever tensor comes next in the data, or to the +/// end of the file when none does. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TensorSpan { + /// Offset from the start of the tensor data. + pub offset: u64, + /// The smallest offset of any other tensor beyond this one; `None` when this + /// is the last tensor, which then runs to the end of the file. + pub next_offset: Option, +} + +/// What a metadata value was, once read. +enum Value { + Uint(u64), + Str(String), + Ints(Vec), + /// Read past and not kept. + Skipped, +} + +struct Reader<'a, R> { + inner: &'a mut R, +} + +impl Reader<'_, R> { + fn bytes(&mut self) -> Result<[u8; N], GgufError> { + let mut buf = [0_u8; N]; + self.inner.read_exact(&mut buf)?; + Ok(buf) + } + + fn u32(&mut self) -> Result { + Ok(u32::from_le_bytes(self.bytes()?)) + } + + fn u64(&mut self) -> Result { + Ok(u64::from_le_bytes(self.bytes()?)) + } + + /// Move forward `count` bytes. Seeking past the end is allowed by the + /// reader, and the next read then fails as truncated, which is the right + /// error for a length that points outside the data. A single skip longer + /// than any header may be is refused at once, as malformed: on a file the + /// seek itself can fail with an I/O error for a huge offset, which would + /// blame the disk for a defect of the header. + fn skip(&mut self, count: u64) -> Result<(), GgufError> { + if count > MAX_HEADER_BYTES { + return Err(GgufError::Malformed(format!( + "a length of {count} bytes is larger than any header, which is at most {} MiB", + MAX_HEADER_BYTES / (1024 * 1024) + ))); + } + let count = i64::try_from(count) + .map_err(|_| GgufError::Malformed(format!("length {count} is out of range")))?; + self.inner.seek(SeekFrom::Current(count))?; + Ok(()) + } + + /// Refuse a header that has already spanned more than [`MAX_HEADER_BYTES`]. + fn check_size(&mut self) -> Result<(), GgufError> { + if self.inner.stream_position()? > MAX_HEADER_BYTES { + return Err(GgufError::Malformed(format!( + "the header is larger than {} MiB, which no real model's is", + MAX_HEADER_BYTES / (1024 * 1024) + ))); + } + Ok(()) + } + + /// A string: a 64-bit length, then that many bytes. `None` when it is + /// longer than `keep_up_to`, in which case it has been skipped. + fn string(&mut self, keep_up_to: u64) -> Result, GgufError> { + let len = self.u64()?; + if len > keep_up_to { + self.skip(len)?; + return Ok(None); + } + let len = usize::try_from(len) + .map_err(|_| GgufError::Malformed(format!("length {len} is out of range")))?; + let mut buf = vec![0_u8; len]; + self.inner.read_exact(&mut buf)?; + Ok(Some(String::from_utf8_lossy(&buf).into_owned())) + } + + fn value(&mut self, value_type: u32, depth: u8, keep: bool) -> Result { + if let Some((width, num)) = fixed_width(value_type) { + return self.scalar(width, num); + } + match value_type { + 8 => Ok(self + .string(MAX_KEPT_STRING)? + .map_or(Value::Skipped, Value::Str)), + 9 => self.array(depth, keep), + other => Err(GgufError::Malformed(format!( + "unknown metadata value type {other}" + ))), + } + } + + /// A fixed-width scalar. Integers (and booleans) are kept when they are not + /// negative: every figure the estimate reads is one, and the writers in common + /// use (gguf-py among them) store a Python integer as a signed 32-bit one, so + /// refusing signed types would leave most models unreadable. A negative or + /// floating value under a key of interest is not something to interpret, and + /// reads as missing. + fn scalar(&mut self, width: usize, num: Num) -> Result { + let mut buf = [0_u8; 8]; + self.inner.read_exact(&mut buf[..width])?; + Ok(decode_int(buf, width, num).map_or(Value::Skipped, Value::Uint)) + } + + fn array(&mut self, depth: u8, keep: bool) -> Result { + let element_type = self.u32()?; + let len = self.u64()?; + if let Some((width, num)) = fixed_width(element_type) { + if keep && num != Num::Other && len <= MAX_KEPT_ARRAY { + let mut values = Vec::with_capacity(usize::try_from(len).unwrap_or(0)); + let mut usable = true; + for _ in 0..len { + let mut buf = [0_u8; 8]; + self.inner.read_exact(&mut buf[..width])?; + match decode_int(buf, width, num) { + Some(value) => values.push(value), + None => usable = false, + } + } + // One negative entry makes the whole per-layer list meaningless; + // it reads as missing rather than as a list with a hole in it. + return Ok(if usable { + Value::Ints(values) + } else { + Value::Skipped + }); + } + let total = len.checked_mul(width as u64).ok_or_else(|| { + GgufError::Malformed(format!("array of {len} elements is out of range")) + })?; + self.skip(total)?; + return Ok(Value::Skipped); + } + match element_type { + 8 => { + for item in 0..len { + let item_len = self.u64()?; + self.skip(item_len)?; + if item % SIZE_CHECK_EVERY == SIZE_CHECK_EVERY - 1 { + self.check_size()?; + } + } + } + 9 => { + if depth >= MAX_ARRAY_DEPTH { + return Err(GgufError::Malformed("arrays nested too deeply".to_owned())); + } + for item in 0..len { + self.value(9, depth + 1, false)?; + if item % SIZE_CHECK_EVERY == SIZE_CHECK_EVERY - 1 { + self.check_size()?; + } + } + } + other => { + return Err(GgufError::Malformed(format!( + "unknown array element type {other}" + ))); + } + } + Ok(Value::Skipped) + } +} + +/// The width in bytes, and how to read it, of a fixed-width GGUF metadata type +/// (a scalar value, or an array element); `None` for a string, an array, or a +/// type id the format does not define. The one table of fixed-width type ids: a +/// scalar and an array element share it, so a type added here is read in both +/// places. +const fn fixed_width(type_id: u32) -> Option<(usize, Num)> { + // GGUF metadata value types, in the order the format numbers them; 8 is a + // string and 9 an array, neither of fixed width. + match type_id { + // uint8, and bool (one byte, read as its byte value). + 0 | 7 => Some((1, Num::Unsigned)), + 1 => Some((1, Num::Signed)), + 2 => Some((2, Num::Unsigned)), + 3 => Some((2, Num::Signed)), + 4 => Some((4, Num::Unsigned)), + 5 => Some((4, Num::Signed)), + 6 => Some((4, Num::Other)), + 10 => Some((8, Num::Unsigned)), + 11 => Some((8, Num::Signed)), + 12 => Some((8, Num::Other)), + _ => None, + } +} + +/// How the bytes of a fixed-width metadata value are read. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Num { + Unsigned, + Signed, + /// A float, or anything else not read as an integer. + Other, +} + +/// The non-negative integer in the first `width` bytes of `buf`, little-endian; +/// `None` for a negative signed value or a type that is not an integer. +fn decode_int(buf: [u8; 8], width: usize, num: Num) -> Option { + match num { + Num::Other => None, + Num::Unsigned => Some(u64::from_le_bytes(buf)), + Num::Signed => { + // Sign-extend from the value's own width. + let shift = 64 - 8 * u32::try_from(width).unwrap_or(8); + let signed = i64::from_le_bytes(buf) << shift >> shift; + u64::try_from(signed).ok() + } + } +} + +fn per_layer(value: Option<&Value>) -> Option { + match value? { + Value::Uint(value) => Some(PerLayer::Uniform(*value)), + Value::Ints(values) => Some(PerLayer::Each(values.clone())), + Value::Str(_) | Value::Skipped => None, + } +} + +fn uint(value: Option<&Value>) -> Option { + match value? { + Value::Uint(value) => Some(*value), + // A per-layer array where a single figure was wanted (a feed-forward + // width that varies by layer): the widest is the one a buffer is sized + // for. + Value::Ints(values) => values.iter().copied().max(), + Value::Str(_) | Value::Skipped => None, + } +} + +const fn align_up(value: u64, alignment: u64) -> u64 { + value.div_ceil(alignment).saturating_mul(alignment) +} + +/// Read the header of a GGUF file from `source`, positioned at its start. +/// +/// Reads the metadata and the tensor records and stops there: the tensor data is +/// never touched. +/// +/// # Errors +/// +/// [`GgufError`] when the data is not a GGUF this reader handles, ends early, or +/// is structurally wrong. +pub fn parse_gguf_header(source: &mut R) -> Result { + let mut reader = Reader { inner: source }; + let magic: [u8; 4] = reader.bytes()?; + if magic != MAGIC { + return Err(GgufError::NotGguf); + } + let version = reader.u32()?; + if !SUPPORTED_VERSIONS.contains(&version) { + return Err(GgufError::UnsupportedVersion(version)); + } + let tensor_count = reader.u64()?; + let metadata_count = reader.u64()?; + for (what, count) in [("tensor", tensor_count), ("metadata", metadata_count)] { + if count > MAX_ENTRIES { + return Err(GgufError::Malformed(format!( + "{count} {what} entries is not a plausible header" + ))); + } + } + + let mut architecture = None; + let mut alignment = DEFAULT_ALIGNMENT; + let mut split_count = None; + let mut sliding_window = false; + let mut ssm = false; + let mut latent_attention = false; + let mut moe = false; + let mut kept: BTreeMap = BTreeMap::new(); + + for _ in 0..metadata_count { + let key = reader + .string(MAX_KEY_LEN)? + .ok_or_else(|| GgufError::Malformed("a metadata key is implausibly long".to_owned()))?; + let value_type = reader.u32()?; + let keep = is_kept_key(&key); + let value = reader.value(value_type, 0, keep)?; + reader.check_size()?; + match (key.as_str(), value) { + ("general.architecture", Value::Str(name)) => architecture = Some(name), + ("general.alignment", Value::Uint(value)) => alignment = value, + ("split.count", Value::Uint(value)) => split_count = Some(value), + (_, value) => { + ssm |= key.contains(".ssm."); + latent_attention |= key.ends_with(".attention.kv_lora_rank"); + if key.ends_with(".expert_count") { + moe |= uint(Some(&value)).is_some_and(|experts| experts > 1); + } + if key.ends_with(".attention.sliding_window") { + sliding_window |= uint(Some(&value)).is_some_and(|window| window > 0); + } + // The sliding-window key is decoded as kept (so a per-layer array + // is read, not skipped) for the flag above; its value is not + // read again, so it does not take one of the kept slots. + let read_later = !key.ends_with(".attention.sliding_window"); + if keep && read_later && matches!(value, Value::Uint(_) | Value::Ints(_)) { + if kept.len() >= MAX_KEPT_KEYS && !kept.contains_key(&key) { + return Err(GgufError::Malformed(format!( + "more than {MAX_KEPT_KEYS} distinct model-shape keys is not a \ + plausible header" + ))); + } + kept.insert(key, value); + } + } + } + } + if alignment == 0 { + return Err(GgufError::Malformed("general.alignment is zero".to_owned())); + } + + let mut param_count = 0_u64; + let mut offsets: Vec = Vec::new(); + let mut token_embd_offset = None; + let mut has_output = false; + for _ in 0..tensor_count { + // Held to the same bound as a metadata key: a real tensor name is a few + // dozen bytes. + let name = reader + .string(MAX_KEY_LEN)? + .ok_or_else(|| GgufError::Malformed("a tensor name is implausibly long".to_owned()))?; + let dims = reader.u32()?; + if dims > MAX_TENSOR_DIMS { + return Err(GgufError::Malformed(format!( + "a tensor claims {dims} dimensions" + ))); + } + let mut elements = 1_u64; + for _ in 0..dims { + elements = elements.saturating_mul(reader.u64()?); + } + let _ggml_type = reader.u32()?; + let offset = reader.u64()?; + param_count = param_count.saturating_add(elements); + offsets.push(offset); + reader.check_size()?; + if name == TOKEN_EMBD_NAME { + token_embd_offset = Some(offset); + } + has_output |= name == OUTPUT_NAME; + } + let token_embd = token_embd_offset.map(|offset| TensorSpan { + offset, + next_offset: offsets + .iter() + .copied() + .filter(|other| *other > offset) + .min(), + }); + let header_end = reader.inner.stream_position()?; + + let get = |suffix: &str| { + architecture + .as_ref() + .and_then(|arch| kept.get(&format!("{arch}.{suffix}"))) + }; + let block_count = uint(get("block_count")); + if let Some(blocks) = block_count.filter(|blocks| *blocks > MAX_BLOCK_COUNT) { + return Err(GgufError::Malformed(format!( + "{blocks} layers is not a plausible model" + ))); + } + Ok(GgufHeader { + version, + block_count, + embedding_length: uint(get("embedding_length")), + head_count: per_layer(get("attention.head_count")), + head_count_kv: per_layer(get("attention.head_count_kv")), + key_length: uint(get("attention.key_length")), + value_length: uint(get("attention.value_length")), + feed_forward_length: uint(get("feed_forward_length")), + context_length: uint(get("context_length")), + architecture, + tensor_count, + param_count, + data_offset: align_up(header_end, alignment), + split_count, + sliding_window, + ssm, + latent_attention, + moe, + token_embd, + has_output, + }) +} + +#[cfg(test)] +pub(crate) mod test_builder { + //! Byte builders for synthetic GGUF headers, shared with the estimator and + //! source tests so every layer is exercised against the same fixtures. + + pub const U8: u32 = 0; + pub const U32: u32 = 4; + pub const I32: u32 = 5; + pub const F32: u32 = 6; + pub const STRING: u32 = 8; + pub const ARRAY: u32 = 9; + + #[derive(Default)] + pub struct Builder { + version: u32, + metadata: Vec, + metadata_count: u64, + tensors: Vec, + tensor_count: u64, + alignment: u64, + } + + impl Builder { + pub fn new() -> Self { + Self { + version: 3, + alignment: 32, + ..Self::default() + } + } + + pub fn version(mut self, version: u32) -> Self { + self.version = version; + self + } + + fn push_string(buf: &mut Vec, value: &str) { + buf.extend_from_slice(&(value.len() as u64).to_le_bytes()); + buf.extend_from_slice(value.as_bytes()); + } + + fn key(&mut self, key: &str, value_type: u32) { + Self::push_string(&mut self.metadata, key); + self.metadata.extend_from_slice(&value_type.to_le_bytes()); + self.metadata_count += 1; + } + + pub fn string(mut self, key: &str, value: &str) -> Self { + self.key(key, STRING); + Self::push_string(&mut self.metadata, value); + self + } + + pub fn u32(mut self, key: &str, value: u32) -> Self { + self.key(key, U32); + self.metadata.extend_from_slice(&value.to_le_bytes()); + self + } + + /// A signed 32-bit integer, which is how gguf-py stores a Python integer. + pub fn i32(mut self, key: &str, value: i32) -> Self { + self.key(key, I32); + self.metadata.extend_from_slice(&value.to_le_bytes()); + self + } + + pub fn i32_array(mut self, key: &str, values: &[i32]) -> Self { + self.key(key, ARRAY); + self.metadata.extend_from_slice(&I32.to_le_bytes()); + self.metadata + .extend_from_slice(&(values.len() as u64).to_le_bytes()); + for value in values { + self.metadata.extend_from_slice(&value.to_le_bytes()); + } + self + } + + pub fn f32(mut self, key: &str, value: f32) -> Self { + self.key(key, F32); + self.metadata.extend_from_slice(&value.to_le_bytes()); + self + } + + pub fn u32_array(mut self, key: &str, values: &[u32]) -> Self { + self.key(key, ARRAY); + self.metadata.extend_from_slice(&U32.to_le_bytes()); + self.metadata + .extend_from_slice(&(values.len() as u64).to_le_bytes()); + for value in values { + self.metadata.extend_from_slice(&value.to_le_bytes()); + } + self + } + + /// A string array, the shape of a tokenizer's token list. + pub fn string_array(mut self, key: &str, count: usize) -> Self { + self.key(key, ARRAY); + self.metadata.extend_from_slice(&STRING.to_le_bytes()); + self.metadata + .extend_from_slice(&(count as u64).to_le_bytes()); + for index in 0..count { + Self::push_string(&mut self.metadata, &format!("tok{index}")); + } + self + } + + pub fn u8_array_of_len(mut self, key: &str, count: u64) -> Self { + self.key(key, ARRAY); + self.metadata.extend_from_slice(&U8.to_le_bytes()); + self.metadata.extend_from_slice(&count.to_le_bytes()); + self.metadata + .extend(std::iter::repeat_n(0_u8, count as usize)); + self + } + + /// A metadata entry with a value type and bytes of the caller's choosing, + /// for building headers no real writer would produce. + pub fn raw(mut self, key: &str, value_type: u32, value: &[u8]) -> Self { + self.key(key, value_type); + self.metadata.extend_from_slice(value); + self + } + + pub fn alignment(mut self, alignment: u32) -> Self { + self.alignment = u64::from(alignment); + self.u32("general.alignment", alignment) + } + + pub fn tensor(self, name: &str, dims: &[u64]) -> Self { + self.tensor_at(name, dims, 0) + } + + /// A tensor record with an explicit offset into the tensor data. + pub fn tensor_at(mut self, name: &str, dims: &[u64], offset: u64) -> Self { + Self::push_string(&mut self.tensors, name); + self.tensors + .extend_from_slice(&(dims.len() as u32).to_le_bytes()); + for dim in dims { + self.tensors.extend_from_slice(&dim.to_le_bytes()); + } + self.tensors.extend_from_slice(&0_u32.to_le_bytes()); + self.tensors.extend_from_slice(&offset.to_le_bytes()); + self.tensor_count += 1; + self + } + + /// The header bytes alone, ending where the tensor data would begin + /// before alignment padding. + pub fn header_bytes(&self) -> Vec { + let mut out = Vec::new(); + out.extend_from_slice(b"GGUF"); + out.extend_from_slice(&self.version.to_le_bytes()); + out.extend_from_slice(&self.tensor_count.to_le_bytes()); + out.extend_from_slice(&self.metadata_count.to_le_bytes()); + out.extend_from_slice(&self.metadata); + out.extend_from_slice(&self.tensors); + out + } + + /// A whole file: the header, padding to the alignment, then + /// `data_len` bytes standing in for the weights. + pub fn file(&self, data_len: usize) -> Vec { + let mut out = self.header_bytes(); + let aligned = (out.len() as u64).div_ceil(self.alignment) * self.alignment; + out.resize(aligned as usize, 0); + out.extend(std::iter::repeat_n(0_u8, data_len)); + out + } + } + + /// A llama-shaped model: 32 layers, 32 heads of which 8 are KV heads, head + /// width 128, so 4096 embedding. Vocabulary of `vocab` tokens. It has its + /// own `output.weight`, as Llama 3 does, so its embeddings are not tied. + pub fn llama_like(vocab: usize) -> Builder { + Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 32) + .u32("llama.embedding_length", 4096) + .u32("llama.feed_forward_length", 14336) + .u32("llama.attention.head_count", 32) + .u32("llama.attention.head_count_kv", 8) + .u32("llama.context_length", 131_072) + .string_array("tokenizer.ggml.tokens", vocab) + .tensor_at("token_embd.weight", &[4096, vocab as u64], 0) + .tensor_at("blk.0.attn_q.weight", &[4096, 4096], vocab as u64 * 4096) + .tensor_at( + "output.weight", + &[4096, vocab as u64], + vocab as u64 * 4096 + 4096 * 4096, + ) + } +} + +#[cfg(test)] +mod tests { + use super::test_builder::{Builder, llama_like}; + use super::*; + use std::io::Cursor; + + fn parse(bytes: &[u8]) -> Result { + parse_gguf_header(&mut Cursor::new(bytes)) + } + + #[test] + fn reads_the_figures_an_estimate_needs() { + let file = llama_like(10).file(1000); + let header = parse(&file).unwrap(); + assert_eq!(header.version, 3); + assert_eq!(header.architecture.as_deref(), Some("llama")); + assert_eq!(header.block_count, Some(32)); + assert_eq!(header.embedding_length, Some(4096)); + assert_eq!(header.feed_forward_length, Some(14336)); + assert_eq!(header.head_count, Some(PerLayer::Uniform(32))); + assert_eq!(header.head_count_kv, Some(PerLayer::Uniform(8))); + assert_eq!(header.context_length, Some(131_072)); + assert_eq!(header.tensor_count, 3); + assert_eq!(header.param_count, 2 * 4096 * 10 + 4096 * 4096); + assert!(!header.sliding_window && !header.ssm && !header.latent_attention); + } + + #[test] + fn the_input_embedding_runs_to_the_next_tensor_in_the_data() { + // Listed out of data order on purpose: the span ends at the next offset + // beyond it, not at the next record. + let file = Builder::new() + .tensor_at("blk.0.attn_q.weight", &[8, 8], 4096) + .tensor_at("token_embd.weight", &[8, 8], 1024) + .tensor_at("output.weight", &[8, 8], 2048) + .file(8192); + let header = parse(&file).unwrap(); + assert_eq!( + header.token_embd, + Some(TensorSpan { + offset: 1024, + next_offset: Some(2048) + }) + ); + } + + #[test] + fn the_input_embedding_is_the_last_tensor_when_nothing_follows_it() { + let file = Builder::new() + .tensor_at("blk.0.attn_q.weight", &[8, 8], 0) + .tensor_at("token_embd.weight", &[8, 8], 512) + .file(1024); + let header = parse(&file).unwrap(); + assert_eq!( + header.token_embd, + Some(TensorSpan { + offset: 512, + next_offset: None + }) + ); + } + + #[test] + fn an_output_tensor_is_noticed_and_its_absence_means_tied_embeddings() { + // Not the last tensor, so a later one cannot clear what it set. + let untied = Builder::new() + .tensor_at("token_embd.weight", &[8, 8], 0) + .tensor_at("output.weight", &[8, 8], 64) + .tensor_at("output_norm.weight", &[8], 128) + .file(160); + assert!(parse(&untied).unwrap().has_output); + let tied = Builder::new() + .tensor_at("token_embd.weight", &[8, 8], 0) + .tensor_at("output_norm.weight", &[8], 64) + .file(128); + assert!(!parse(&tied).unwrap().has_output); + } + + #[test] + fn a_file_without_an_input_embedding_has_no_span() { + let file = Builder::new() + .tensor_at("blk.0.attn_q.weight", &[8, 8], 0) + .file(64); + assert_eq!(parse(&file).unwrap().token_embd, None); + } + + #[test] + fn the_data_offset_is_the_header_end_rounded_up_to_the_alignment() { + let builder = llama_like(10); + let unpadded = builder.header_bytes().len() as u64; + let header = parse(&builder.file(64)).unwrap(); + assert_eq!(header.data_offset, unpadded.div_ceil(32) * 32); + // Something other than the default must be honoured: a parser that + // ignored `general.alignment` would land on the 32-byte boundary. + let wide = llama_like(10).alignment(4096); + let header = parse(&wide.file(64)).unwrap(); + assert_eq!(header.data_offset, 4096); + } + + #[test] + fn version_two_is_read_and_other_versions_are_refused() { + assert_eq!(parse(&llama_like(3).version(2).file(0)).unwrap().version, 2); + for version in [1, 4, 0x0300_0000] { + let error = parse(&llama_like(3).version(version).file(0)).unwrap_err(); + assert!( + matches!(error, GgufError::UnsupportedVersion(found) if found == version), + "{error:?}" + ); + } + } + + #[test] + fn a_file_without_the_magic_is_not_gguf() { + let mut bytes = llama_like(3).file(0); + bytes[0] = b'X'; + assert!(matches!(parse(&bytes), Err(GgufError::NotGguf))); + assert!(matches!(parse(b"GG"), Err(GgufError::Truncated))); + } + + #[test] + fn a_header_cut_short_is_truncated_wherever_it_is_cut() { + let header = llama_like(50).header_bytes(); + // Every proper prefix must fail, and fail as truncation: a parser that + // returned a header from half the data would report a model it never + // finished reading. + for cut in 0..header.len() { + let error = parse(&header[..cut]).unwrap_err(); + assert!( + matches!(error, GgufError::Truncated | GgufError::NotGguf), + "prefix of {cut} bytes gave {error:?}" + ); + } + parse(&header).unwrap(); + } + + #[test] + fn a_per_layer_head_count_array_is_kept_per_layer() { + let file = Builder::new() + .string("general.architecture", "hybrid") + .u32("hybrid.block_count", 4) + .u32_array("hybrid.attention.head_count_kv", &[0, 8, 0, 8]) + .file(0); + let header = parse(&file).unwrap(); + let heads = header.head_count_kv.unwrap(); + assert_eq!(heads, PerLayer::Each(vec![0, 8, 0, 8])); + assert_eq!(heads.at(1), Some(8)); + assert_eq!(heads.at(2), Some(0)); + assert_eq!(heads.at(4), None); + } + + #[test] + fn parsing_resumes_exactly_after_arrays_it_skips() { + // A kept key placed after two large arrays the parser skips: it is read + // with its value only if each skip moved exactly past its array. + let file = llama_like(1000) + .u8_array_of_len("tokenizer.ggml.token_type", 5000) + .u32_array("tokenizer.ggml.merges_like", &vec![7; 5000]) + .u32("llama.attention.key_length", 99) + .file(0); + let header = parse(&file).unwrap(); + assert_eq!(header.key_length, Some(99)); + assert_eq!(header.tensor_count, 3); + } + + /// A feed-forward width given per layer (an array) is read as the widest, + /// which is the one a compute buffer is sized for, not the first or the + /// narrowest. + #[test] + fn a_per_layer_feed_forward_width_is_read_as_the_widest() { + let file = Builder::new() + .string("general.architecture", "llama") + .u32_array("llama.feed_forward_length", &[11_008, 14_336, 8_192]) + .tensor("w", &[16]) + .file(0); + assert_eq!(parse(&file).unwrap().feed_forward_length, Some(14_336)); + } + + #[test] + fn layouts_the_estimate_cannot_model_are_flagged() { + let base = || { + Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 2) + }; + let header = parse(&base().u32("x.attention.sliding_window", 4096).file(0)).unwrap(); + assert!(header.sliding_window); + // A zero window means no window. + let header = parse(&base().u32("x.attention.sliding_window", 0).file(0)).unwrap(); + assert!(!header.sliding_window); + let header = parse(&base().u32("x.ssm.conv_kernel", 4).file(0)).unwrap(); + assert!(header.ssm); + let header = parse(&base().u32("x.attention.kv_lora_rank", 512).file(0)).unwrap(); + assert!(header.latent_attention); + let header = parse(&base().u32("split.count", 3).file(0)).unwrap(); + assert_eq!(header.split_count, Some(3)); + } + + #[test] + fn implausible_counts_are_refused_before_they_become_a_loop() { + // The tensor count (bytes 8..16) and the metadata count (16..24), each + // made absurd in turn. The message names the count, so the test fails if + // the guard is removed and the parser fails somewhere else instead. + for (range, what) in [(8..16, "tensor entries"), (16..24, "metadata entries")] { + let mut bytes = llama_like(3).header_bytes(); + bytes[range].copy_from_slice(&u64::MAX.to_le_bytes()); + let error = parse(&bytes).unwrap_err(); + assert!( + matches!(&error, GgufError::Malformed(message) if message.contains(what)), + "{what}: {error}" + ); + } + } + + /// A tensor's element count is a product of dimensions the file states; one + /// that overflows saturates rather than panicking. + #[test] + fn tensor_dimensions_that_overflow_saturate() { + let file = Builder::new() + .string("general.architecture", "x") + .tensor("huge", &[u64::MAX, u64::MAX]) + .file(0); + assert_eq!(parse(&file).unwrap().param_count, u64::MAX); + } + + /// A tensor name longer than any real one is refused, as a metadata key is. + #[test] + fn an_implausibly_long_tensor_name_is_refused() { + let file = Builder::new() + .string("general.architecture", "x") + .tensor(&"t".repeat(5000), &[8]) + .file(0); + let error = parse(&file).unwrap_err(); + assert!( + matches!(&error, GgufError::Malformed(message) if message.contains("tensor name")), + "{error}" + ); + } + + #[test] + fn a_zero_alignment_is_malformed_rather_than_a_division_by_zero() { + let file = Builder::new() + .string("general.architecture", "llama") + .u32("general.alignment", 0) + .file(0); + assert!(matches!(parse(&file), Err(GgufError::Malformed(_)))); + } + + #[test] + fn floats_and_other_values_under_keys_of_interest_are_ignored_not_misread() { + let file = Builder::new() + .string("general.architecture", "llama") + .f32("llama.block_count", 32.0) + .file(0); + assert_eq!(parse(&file).unwrap().block_count, None); + } + + // ----- untrusted-input guards --------------------------------------------- + // + // A header can come from a stranger's Hugging Face repository, so each bound + // below is one a hostile or corrupt file would otherwise walk past. Every test + // names the guard it exercises and fails when that guard is removed. + + fn malformed(result: Result) -> String { + match result { + Err(GgufError::Malformed(why)) => why, + other => panic!("expected a malformed header, got {other:?}"), + } + } + + /// The bytes of an array value (after its type marker) nested `depth` deep, + /// innermost an empty byte array. + fn nested_array(depth: usize) -> Vec { + let mut out = Vec::new(); + if depth <= 1 { + out.extend_from_slice(&test_builder::U8.to_le_bytes()); + out.extend_from_slice(&0_u64.to_le_bytes()); + } else { + out.extend_from_slice(&test_builder::ARRAY.to_le_bytes()); + out.extend_from_slice(&1_u64.to_le_bytes()); + out.extend(nested_array(depth - 1)); + } + out + } + + #[test] + fn a_metadata_key_beyond_the_length_bound_is_malformed() { + let long = "k".repeat(usize::try_from(MAX_KEY_LEN).unwrap() + 1); + let file = Builder::new().u32(&long, 1).file(0); + assert!(malformed(parse(&file)).contains("key")); + // Exactly at the bound is still a key. + let ok = "k".repeat(usize::try_from(MAX_KEY_LEN).unwrap()); + parse(&Builder::new().u32(&ok, 1).file(0)).unwrap(); + } + + #[test] + fn a_tensor_count_beyond_the_bound_is_malformed() { + let mut bytes = llama_like(3).header_bytes(); + // The tensor count is bytes 8..16, ahead of the metadata count. + bytes[8..16].copy_from_slice(&(MAX_ENTRIES + 1).to_le_bytes()); + assert!(malformed(parse(&bytes)).contains("tensor")); + } + + #[test] + fn a_tensor_with_too_many_dimensions_is_malformed() { + let file = Builder::new().tensor("t", &[1; 9]).file(0); + assert!(malformed(parse(&file)).contains("dimensions")); + parse(&Builder::new().tensor("t", &[1; 8]).file(0)).unwrap(); + } + + #[test] + fn arrays_nested_beyond_the_depth_bound_are_malformed() { + let at = |depth| { + let value = nested_array(depth); + parse(&Builder::new().raw("k", test_builder::ARRAY, &value).file(0)) + }; + at(MAX_ARRAY_DEPTH as usize + 1).unwrap(); + assert!(malformed(at(MAX_ARRAY_DEPTH as usize + 2)).contains("nested too deeply")); + } + + #[test] + fn an_array_whose_byte_length_overflows_is_malformed_not_truncated() { + // 2^63 four-byte elements: the product does not fit in 64 bits. + let mut value = Vec::new(); + value.extend_from_slice(&test_builder::F32.to_le_bytes()); + value.extend_from_slice(&(1_u64 << 63).to_le_bytes()); + let file = Builder::new().raw("k", test_builder::ARRAY, &value).file(0); + assert!(malformed(parse(&file)).contains("out of range")); + // A length that fits in 64 bits but not in the signed offset a seek takes: + // far past any header, so refused as that before a seek is tried. + let mut value = Vec::new(); + value.extend_from_slice(&test_builder::F32.to_le_bytes()); + value.extend_from_slice(&((u64::MAX / 4) - 1).to_le_bytes()); + let file = Builder::new().raw("k", test_builder::ARRAY, &value).file(0); + assert!(malformed(parse(&file)).contains("larger than any header")); + } + + /// A length far beyond any header is malformed whatever the reader does with + /// a seek that large: an in-memory reader would call it truncated, a file can + /// fail the seek with an I/O error. + #[test] + fn a_single_skip_longer_than_any_header_is_malformed() { + for length in [MAX_HEADER_BYTES + 1, 1_u64 << 44, 1_u64 << 50] { + let file = Builder::new() + .raw("k", test_builder::STRING, &length.to_le_bytes()) + .file(0); + let message = malformed(parse(&file)); + assert!(message.contains("larger than any header"), "{message}"); + } + // At the bound itself the skip is allowed, and the header it leaves + // behind is then past the size bound: refused there, by the size check. + let file = Builder::new() + .raw("k", test_builder::STRING, &MAX_HEADER_BYTES.to_le_bytes()) + .file(0); + let message = malformed(parse(&file)); + assert!(message.contains("header is larger than"), "{message}"); + } + + #[test] + fn an_unknown_value_or_element_type_is_malformed() { + let file = Builder::new().raw("k", 99, &[0; 8]).file(0); + assert!(malformed(parse(&file)).contains("unknown metadata value type 99")); + let mut value = Vec::new(); + value.extend_from_slice(&99_u32.to_le_bytes()); + value.extend_from_slice(&1_u64.to_le_bytes()); + let file = Builder::new().raw("k", test_builder::ARRAY, &value).file(0); + assert!(malformed(parse(&file)).contains("unknown array element type 99")); + } + + #[test] + fn a_layer_count_beyond_the_bound_is_malformed() { + let with = |layers| { + parse( + &Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", layers) + .file(0), + ) + }; + assert_eq!( + with(u32::try_from(MAX_BLOCK_COUNT).unwrap()) + .unwrap() + .block_count, + Some(MAX_BLOCK_COUNT) + ); + assert!(malformed(with(u32::MAX)).contains("not a plausible model")); + assert!( + malformed(with(u32::try_from(MAX_BLOCK_COUNT).unwrap() + 1)) + .contains("not a plausible model") + ); + } + + #[test] + fn too_many_distinct_shape_keys_are_malformed_and_other_keys_are_not_kept() { + // Arrays under keys the estimate never reads are skipped, however many. + let mut builder = Builder::new(); + for index in 0..200 { + builder = builder.u32_array(&format!("junk{index}.table"), &[1; 4000]); + } + parse(&builder.file(0)).unwrap(); + // Keys it does read are kept, so there is a bound on how many. + let mut builder = Builder::new(); + for index in 0..=MAX_KEPT_KEYS { + builder = builder.u32_array(&format!("a{index}.attention.head_count"), &[1; 8]); + } + assert!(malformed(parse(&builder.file(0))).contains("distinct")); + let mut builder = Builder::new(); + for index in 0..MAX_KEPT_KEYS { + builder = builder.u32_array(&format!("a{index}.attention.head_count"), &[1; 8]); + } + parse(&builder.file(0)).unwrap(); + } + + /// The sliding-window key is decoded as a kept key but never read again, so + /// it takes no one of the kept slots: a header at the cap of other shape + /// keys is still readable with it, and the flag is set. + #[test] + fn the_sliding_window_key_does_not_take_a_kept_slot() { + let mut builder = Builder::new(); + for index in 0..MAX_KEPT_KEYS { + builder = builder.u32(&format!("a{index}.attention.head_count"), 1); + } + let header = parse(&builder.u32("x.attention.sliding_window", 4096).file(0)).unwrap(); + assert!(header.sliding_window); + } + + /// A key repeated once the kept slots are full replaces its own value; it is + /// not a new distinct key, so the cap does not refuse it. + #[test] + fn a_repeated_shape_key_at_the_cap_is_not_a_new_distinct_key() { + let mut builder = Builder::new(); + for index in 0..MAX_KEPT_KEYS { + builder = builder.u32(&format!("a{index}.attention.head_count"), 1); + } + builder = builder.u32("a0.attention.head_count", 2); + parse(&builder.file(0)).unwrap(); + } + + #[test] + fn a_header_that_spans_more_than_the_size_bound_is_refused() { + // Two arrays, each under the bound on its own, that together carry the + // header past it: no single length is refused, so it is the running + // check on the header's size that has to catch it. + let half = MAX_HEADER_BYTES / 2 + 1; + let file = Builder::new() + .u8_array_of_len("blob.a", half) + .u8_array_of_len("blob.b", half) + .file(0); + let message = malformed(parse(&file)); + assert!(message.contains("the header is larger than"), "{message}"); + let under = usize::try_from(MAX_HEADER_BYTES).unwrap() / 2; + parse(&Builder::new().u8_array_of_len("blob", under as u64).file(0)).unwrap(); + } + + // ----- signed integers ----------------------------------------------------- + + #[test] + fn signed_integers_are_read_as_the_figures_they_are() { + // gguf-py stores a Python integer as a signed 32-bit value, so this is the + // common shape of a real file, not a corner. + let file = Builder::new() + .string("general.architecture", "x") + .i32("x.block_count", 4) + .i32("x.embedding_length", 256) + .i32_array("x.attention.head_count", &[8, 8, 8, 8]) + .i32_array("x.attention.head_count_kv", &[0, 2, 0, 2]) + .file(0); + let header = parse(&file).unwrap(); + assert_eq!(header.block_count, Some(4)); + assert_eq!(header.embedding_length, Some(256)); + assert_eq!(header.head_count, Some(PerLayer::Each(vec![8; 4]))); + assert_eq!(header.head_count_kv, Some(PerLayer::Each(vec![0, 2, 0, 2]))); + } + + #[test] + fn a_negative_signed_value_reads_as_missing_not_as_a_huge_one() { + let file = Builder::new() + .string("general.architecture", "x") + .i32("x.block_count", -1) + .i32_array("x.attention.head_count_kv", &[2, -2, 2]) + .file(0); + let header = parse(&file).unwrap(); + assert_eq!(header.block_count, None); + // One bad entry spoils the list: it is not a list with a hole in it. + assert_eq!(header.head_count_kv, None); + // The same value as an unsigned one is a large number, not a negative. + let file = Builder::new() + .string("general.architecture", "x") + .u32("x.embedding_length", u32::MAX) + .file(0); + assert_eq!( + parse(&file).unwrap().embedding_length, + Some(u64::from(u32::MAX)) + ); + } + + #[test] + fn signed_values_of_every_width_decode_by_their_own_sign_bit() { + let read = |value_type: u32, bytes: &[u8]| { + parse( + &Builder::new() + .string("general.architecture", "x") + .raw("x.embedding_length", value_type, bytes) + .file(0), + ) + .unwrap() + .embedding_length + }; + assert_eq!(read(1, &[0x7f]), Some(127)); + assert_eq!(read(1, &[0xff]), None); + assert_eq!(read(3, &[0xff, 0x7f]), Some(32767)); + assert_eq!(read(3, &[0x00, 0x80]), None); + assert_eq!(read(5, &[0xff, 0xff, 0xff, 0x7f]), Some(2_147_483_647)); + assert_eq!(read(5, &[0x00, 0x00, 0x00, 0x80]), None); + assert_eq!(read(11, &7_i64.to_le_bytes()), Some(7)); + assert_eq!(read(11, &(-7_i64).to_le_bytes()), None); + // The unsigned types of the same widths read the high bit as a value. + assert_eq!(read(0, &[0xff]), Some(255)); + assert_eq!(read(2, &[0xff, 0xff]), Some(65535)); + } + + /// Every fixed-width value type, as a scalar and as an array element, under a + /// key the parser skips and under one it keeps, each followed by a kept + /// sentinel. The widths are written here from the GGUF specification, not + /// from [`fixed_width`]: a wrong width on any row would desync every later + /// key, so the sentinel would be misread or the parse would fail. + #[test] + fn every_fixed_width_type_is_read_at_its_own_width_as_a_scalar_and_in_an_array() { + // (type id, width in bytes, an integer type the parser keeps a value of) + let types: [(u32, usize, bool); 11] = [ + (0, 1, true), // uint8 + (1, 1, true), // int8 + (2, 2, true), // uint16 + (3, 2, true), // int16 + (4, 4, true), // uint32 + (5, 4, true), // int32 + (6, 4, false), // float32 + (7, 1, true), // bool + (10, 8, true), // uint64 + (11, 8, true), // int64 + (12, 8, false), // float64 + ]; + let le = |value: u64, width: usize| value.to_le_bytes()[..width].to_vec(); + let array = |type_id: u32, width: usize, values: &[u64]| { + let mut bytes = type_id.to_le_bytes().to_vec(); + bytes.extend_from_slice(&(values.len() as u64).to_le_bytes()); + for value in values { + bytes.extend(le(*value, width)); + } + bytes + }; + let parsed = |entry: Builder| { + parse(&entry.u32("x.block_count", 42).file(0)).unwrap_or_else(|error| panic!("{error}")) + }; + let base = || Builder::new().string("general.architecture", "x"); + for (type_id, width, integer) in types { + let skipped_scalar = parsed(base().raw("x.unrelated", type_id, &le(1, width))); + assert_eq!( + skipped_scalar.block_count, + Some(42), + "scalar of type {type_id}" + ); + let skipped_array = parsed(base().raw( + "x.unrelated", + test_builder::ARRAY, + &array(type_id, width, &[1, 0, 1]), + )); + assert_eq!( + skipped_array.block_count, + Some(42), + "array of type {type_id}" + ); + + let kept_scalar = parsed(base().raw("x.feed_forward_length", type_id, &le(1, width))); + assert_eq!( + kept_scalar.block_count, + Some(42), + "kept scalar of type {type_id}" + ); + let kept_array = parsed(base().raw( + "x.feed_forward_length", + test_builder::ARRAY, + &array(type_id, width, &[0, 1, 0]), + )); + assert_eq!( + kept_array.block_count, + Some(42), + "kept array of type {type_id}" + ); + if integer { + assert_eq!(kept_scalar.feed_forward_length, Some(1), "type {type_id}"); + assert_eq!(kept_array.feed_forward_length, Some(1), "type {type_id}"); + } else { + // A float is never read as an integer from its bits. + assert_eq!(kept_scalar.feed_forward_length, None, "type {type_id}"); + assert_eq!(kept_array.feed_forward_length, None, "type {type_id}"); + } + } + } + + #[test] + fn an_expert_count_above_one_marks_a_mixture_of_experts_model() { + let base = |experts| { + parse( + &Builder::new() + .string("general.architecture", "x") + .u32("x.expert_count", experts) + .file(0), + ) + .unwrap() + .moe + }; + assert!(base(8)); + assert!(!base(1)); + assert!(!base(0)); + } + + // ----- guards inside arrays and on what is kept ----------------------------- + + /// A reader that yields `prefix` and then zeros up to `len` bytes, so a + /// header far larger than any test should allocate can be walked by a parser + /// that does not stop itself. + struct ZerosAfter { + prefix: Vec, + len: u64, + pos: u64, + } + + impl Read for ZerosAfter { + fn read(&mut self, out: &mut [u8]) -> io::Result { + let mut written = 0; + while written < out.len() && self.pos < self.len { + out[written] = usize::try_from(self.pos) + .ok() + .and_then(|at| self.prefix.get(at).copied()) + .unwrap_or(0); + written += 1; + self.pos += 1; + } + Ok(written) + } + } + + impl Seek for ZerosAfter { + fn seek(&mut self, to: SeekFrom) -> io::Result { + let next = match to { + SeekFrom::Start(at) => i128::from(at), + SeekFrom::Current(by) => i128::from(self.pos) + i128::from(by), + SeekFrom::End(by) => i128::from(self.len) + i128::from(by), + }; + self.pos = u64::try_from(next) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "seek before start"))?; + Ok(self.pos) + } + } + + /// A header whose only metadata entry is an array of `count` elements of + /// `element_type`, followed by an endless run of zeros that stands for the + /// elements, `total_len` bytes in all. + fn endless_array(element_type: u32, count: u64, total_len: u64) -> ZerosAfter { + let mut prefix = Vec::new(); + prefix.extend_from_slice(b"GGUF"); + prefix.extend_from_slice(&3_u32.to_le_bytes()); + prefix.extend_from_slice(&0_u64.to_le_bytes()); + prefix.extend_from_slice(&1_u64.to_le_bytes()); + prefix.extend_from_slice(&1_u64.to_le_bytes()); + prefix.push(b'k'); + prefix.extend_from_slice(&test_builder::ARRAY.to_le_bytes()); + prefix.extend_from_slice(&element_type.to_le_bytes()); + prefix.extend_from_slice(&count.to_le_bytes()); + ZerosAfter { + prefix, + len: total_len, + pos: 0, + } + } + + #[test] + fn a_string_array_that_walks_past_the_size_bound_is_refused_midway() { + // 20 million empty strings are 160 MiB of eight-byte lengths: well past the + // bound, and a count a real tokenizer never approaches. The check inside + // the loop is what stops it; the one after the value would be too late + // for a file this long. + let mut source = endless_array(test_builder::STRING, 20_000_000, 400 << 20); + let error = parse_gguf_header(&mut source).unwrap_err(); + assert!( + matches!(&error, GgufError::Malformed(why) if why.contains("is larger than")), + "{error:?}" + ); + // The parser stopped near the bound rather than reading to the end. + assert!( + source.pos <= MAX_HEADER_BYTES + 1024 * 1024, + "{}", + source.pos + ); + } + + #[test] + fn an_array_of_arrays_that_walks_past_the_size_bound_is_refused_midway() { + // Each inner array is a 12-byte header of its own: eight million of them + // are 96 MiB. + let mut source = endless_array(test_builder::ARRAY, 8_000_000, 400 << 20); + let error = parse_gguf_header(&mut source).unwrap_err(); + assert!( + matches!(&error, GgufError::Malformed(why) if why.contains("is larger than")), + "{error:?}" + ); + assert!( + source.pos <= MAX_HEADER_BYTES + 1024 * 1024, + "{}", + source.pos + ); + } + + /// A reader that yields `prefix` and then `record` over and over, up to `len` + /// bytes: a tensor table far longer than any test should allocate. + struct RepeatAfter { + prefix: Vec, + record: Vec, + len: u64, + pos: u64, + } + + impl Read for RepeatAfter { + fn read(&mut self, out: &mut [u8]) -> io::Result { + let mut written = 0; + while written < out.len() && self.pos < self.len { + let at = usize::try_from(self.pos).unwrap(); + out[written] = + self.prefix.get(at).copied().unwrap_or_else(|| { + self.record[(at - self.prefix.len()) % self.record.len()] + }); + written += 1; + self.pos += 1; + } + Ok(written) + } + } + + impl Seek for RepeatAfter { + fn seek(&mut self, to: SeekFrom) -> io::Result { + let next = match to { + SeekFrom::Start(at) => i128::from(at), + SeekFrom::Current(by) => i128::from(self.pos) + i128::from(by), + SeekFrom::End(by) => i128::from(self.len) + i128::from(by), + }; + self.pos = u64::try_from(next) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "seek before start"))?; + Ok(self.pos) + } + } + + #[test] + fn a_tensor_table_that_walks_past_the_size_bound_is_refused_midway() { + // The most tensors a header may declare (a count the entry bound allows), + // each a 152-byte record with a long name and eight dimensions: about + // 160 MiB, well past the bound. Only the check inside the tensor loop + // stops it; the metadata loop's checks never see these records. + let mut prefix = Vec::new(); + prefix.extend_from_slice(b"GGUF"); + prefix.extend_from_slice(&3_u32.to_le_bytes()); + prefix.extend_from_slice(&MAX_ENTRIES.to_le_bytes()); + prefix.extend_from_slice(&0_u64.to_le_bytes()); + let mut record = Vec::new(); + record.extend_from_slice(&64_u64.to_le_bytes()); + record.extend_from_slice(&[b'w'; 64]); + record.extend_from_slice(&8_u32.to_le_bytes()); + for _ in 0..8 { + record.extend_from_slice(&1_u64.to_le_bytes()); + } + record.extend_from_slice(&0_u32.to_le_bytes()); + record.extend_from_slice(&0_u64.to_le_bytes()); + assert_eq!(record.len(), 152); + let mut source = RepeatAfter { + prefix, + record, + len: 400 << 20, + pos: 0, + }; + let error = parse_gguf_header(&mut source).unwrap_err(); + assert!( + matches!(&error, GgufError::Malformed(why) if why.contains("is larger than")), + "{error:?}" + ); + assert!( + source.pos <= MAX_HEADER_BYTES + 1024 * 1024, + "{}", + source.pos + ); + } + + #[test] + fn an_integer_array_longer_than_the_kept_bound_is_skipped_not_stored() { + let at = |len: usize| { + parse( + &Builder::new() + .string("general.architecture", "x") + .u32_array("x.attention.head_count_kv", &vec![2; len]) + .file(0), + ) + .unwrap() + .head_count_kv + }; + let kept = usize::try_from(MAX_KEPT_ARRAY).unwrap(); + assert_eq!(at(kept), Some(PerLayer::Each(vec![2; kept]))); + // One more and it is not a per-layer list of any model: the parse goes on + // past it, with nothing stored. + assert_eq!(at(kept + 1), None); + } + + #[test] + fn a_string_longer_than_the_kept_bound_is_skipped_not_stored() { + let at = |len: usize| { + parse( + &Builder::new() + .string("general.architecture", &"a".repeat(len)) + .tensor("w", &[4]) + .file(0), + ) + .unwrap() + }; + let kept = usize::try_from(MAX_KEPT_STRING).unwrap(); + assert_eq!( + at(kept).architecture.as_deref(), + Some("a".repeat(kept).as_str()) + ); + let over = at(kept + 1); + assert_eq!(over.architecture, None); + // What follows the long string was still read: the skip landed on it. + assert_eq!(over.param_count, 4); + } +} diff --git a/crates/rocm-core/src/gguf_fit.rs b/crates/rocm-core/src/gguf_fit.rs new file mode 100644 index 0000000000..1a97dd0e64 --- /dev/null +++ b/crates/rocm-core/src/gguf_fit.rs @@ -0,0 +1,3739 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Will this GGUF model fit in GPU memory at this context size? +//! +//! Weights plus KV cache plus a compute buffer plus a runtime reserve, compared +//! against the dedicated VRAM the readiness check already reads. Pure +//! arithmetic over a [`GgufHeader`]: nothing here reads a file or the network. +//! +//! The KV-cache formula is exact for a plain full-attention model (it is how +//! llama.cpp lays the cache out), and the weights are the file's own size from +//! its tensor offsets, less the input embedding when the file has its own output +//! layer, which is where llama.cpp keeps it: in host memory. A model with tied +//! embeddings (no `output.weight`) uses the same tensor as its output layer's +//! weight, and that layer is on the GPU, so there the embedding is counted. That +//! case is **not measured**: both calibration models have their own output +//! layer, so it is the side that can only overstate. +//! Only two allowances are **calibrated against real loads on one GPU and one +//! engine build**: the compute buffer and the runtime overhead. The loads were on +//! an MI300X (gfx942) with llama.cpp b9752 on ROCm 7.2, flash attention on, two +//! dense full-attention models, each with its own output layer (a 32-layer +//! Llama-3.1-8B-shaped Q4_K_M and a 28-layer Qwen2.5-1.5B-shaped Q8_0), each at +//! several context sizes and KV-cache types (fifteen loads), and one more load of +//! the Llama-shaped model made with the card almost full. In fifteen of the +//! sixteen the total came out *above* what the GPU's used memory grew by: by +//! 0.05% to 2.9% with an f16 KV cache and up to 4.5% with a quantized one, where +//! the dequantization allowance is deliberately generous. In the sixteenth it was +//! 5.5% above, because the process overhead measured there was about 0.4 GiB +//! against 1.05-1.28 GiB in the others, for a reason that is not known. Across +//! all sixteen it was 0.05% to 5.5% above and never below. That is the same data +//! the allowances were fitted to, so it measures the fit, not a guarantee: +//! another engine build, another GPU or a different batch size can miss by more. +//! That is why the answer is labelled an *estimate* everywhere it is shown. Where +//! the figure can be too low -- a mixture-of-experts model, whose expert compute +//! buffers were not measured; a state-space model, whose recurrent state is not +//! counted; a header without the widths the compute buffer is sized from -- the +//! figures are given with no fits / does-not-fit verdict. Layouts the formula +//! does not describe (latent attention, split files, models with no attention +//! layers) get no figures either: all of these are reported as +//! [`EstimateOutcome::CannotEstimate`] rather than guessed at. +//! +//! This is the one place the figures above -- which loads, and how close the +//! estimate came to them -- are written down. The README, `docs/testing.md` and +//! `docs/manual-testing.md` point here for them. The GPU and engine build are +//! also named, without figures, in [`ESTIMATE_CAVEAT`], the README and the +//! calibration tests' comments, so a recalibration on other hardware changes +//! those names as well as this paragraph, the constants below and the tests +//! under "Calibration against measured loads". + +use crate::gguf::{GgufHeader, MAX_BLOCK_COUNT}; +use crate::gguf_source::ProjectorNamed; +use crate::model_readiness::{AcceleratorMemory, format_gib}; +use serde::{Deserialize, Serialize}; +use std::fmt; +#[cfg(test)] +use std::str::FromStr; + +const GIB: f64 = 1024.0 * 1024.0 * 1024.0; + +/// Memory the engine and the GPU runtime hold beyond the weights, the KV cache +/// and the compute buffer (context objects, loaded kernels, graph metadata). +/// Measured at 1050-1101 MiB for the Llama-shaped model and 1228-1277 MiB for the +/// Qwen2-shaped one on the build named in the module docs; 1.25 GiB covers both. +const RUNTIME_RESERVE_BYTES: f64 = 1280.0 * 1024.0 * 1024.0; + +/// llama.cpp's default physical batch size, which sizes its compute buffers. +const COMPUTE_BATCH_TOKENS: f64 = 512.0; + +/// Bytes of the attention mask per token of context and batch token: a 16-bit +/// float each. +const MASK_BYTES: f64 = 2.0; + +/// The largest context size an estimate is made for: 2^24 tokens. +/// +/// Far beyond any model's, and small enough that no figure derived from it can +/// overflow. The CLI refuses a larger request outright, so the report never shows +/// a number a context that size would produce. +pub const MAX_CONTEXT_SIZE: u64 = 1 << 24; + +/// No host has this many bytes of memory, so an estimate that comes to more is +/// not a model any host could load: a corrupt header's head counts or widths. +const MAX_PLAUSIBLE_BYTES: f64 = (1_u64 << 60) as f64; + +const MIB: u64 = 1024 * 1024; + +/// Reported context sizes are rounded down to a multiple of this, so a figure +/// is a number a user would type rather than the exact byte boundary. +const CONTEXT_STEP: u64 = 256; + +/// The range of bits per weight a real GGUF falls in: a 1-bit-ish quantization +/// at the bottom, 32-bit floats at the top. +const MIN_PLAUSIBLE_BITS_PER_WEIGHT: f64 = 1.0; +const MAX_PLAUSIBLE_BITS_PER_WEIGHT: f64 = 33.0; + +/// A smaller quantization is only suggested when it is meaningfully smaller than +/// the current one; two files within this factor of each other are the same +/// quantization for this purpose. +const QUANT_SMALLER_FACTOR: f64 = 0.95; + +/// Typical bits per weight of common llama.cpp quantizations, averaged over a +/// whole model. Approximate: the real figure depends on the model's tensor mix. +/// The K-quant and Q8_0 rows are llama.cpp's `quantize` table sizes for +/// LLaMA-v1-7B (in GiB) converted to bits over its 6 738 415 616 weights; a +/// test holds them to that. +const QUANT_BITS_PER_WEIGHT: [(&str, f64); 7] = [ + ("Q2_K", 3.35), + ("Q3_K_M", 3.91), + ("IQ4_XS", 4.25), + ("Q4_K_M", 4.85), + ("Q5_K_M", 5.69), + ("Q6_K", 6.56), + ("Q8_0", 8.5), +]; + +/// The element type of the KV cache. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum KvCacheType { + #[serde(rename = "f16")] + F16, + #[serde(rename = "q8_0")] + Q8_0, + #[serde(rename = "q4_0")] + Q4_0, +} + +impl KvCacheType { + /// Every type, widest first. + pub const ALL: [Self; 3] = [Self::F16, Self::Q8_0, Self::Q4_0]; + + /// Bytes per cached element. The quantized types store a 16-bit scale per + /// block of 32 values: 34 and 18 bytes per block. + #[must_use] + pub const fn bytes_per_element(self) -> f64 { + match self { + Self::F16 => 2.0, + Self::Q8_0 => 34.0 / 32.0, + Self::Q4_0 => 18.0 / 32.0, + } + } + + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::F16 => "f16", + Self::Q8_0 => "q8_0", + Self::Q4_0 => "q4_0", + } + } +} + +impl fmt::Display for KvCacheType { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.name()) + } +} + +// The CLI parses the flag with clap (`KvCacheTypeArg`); this is for tests +// that read a type back out of the report text. +#[cfg(test)] +impl FromStr for KvCacheType { + type Err = String; + + fn from_str(value: &str) -> Result { + Self::ALL + .into_iter() + .find(|kind| kind.name() == value) + .ok_or_else(|| format!("`{value}` is not a KV cache type (f16, q8_0 or q4_0)")) + } +} + +/// What the estimate concluded. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum EstimateOutcome { + /// The estimated total is within the GPU's dedicated memory. The total + /// compared is the sum of the parts as the report shows them, each in whole + /// MiB rounded up, so it can be a few MiB above `total_bytes`. + Fits, + /// The estimated total exceeds the GPU's dedicated memory. + DoesNotFit, + /// No comparison could be made. See [`ContextEstimate::cannot_estimate_reason`]. + CannotEstimate, +} + +/// Why no comparison could be made. +/// +/// The first group is about the file, the second about the host; the figures +/// the file allows are still reported in the second. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CannotEstimateReason { + /// The GGUF header could not be fetched or read. + HeaderUnreadable, + /// The file named is a multimodal projector (`mmproj`), not a language + /// model, so there is no model to size. + NotAModel, + /// The file is one shard of a split model; the metadata and the weights are + /// spread over several files. + SplitFile, + /// Latent attention (the KV cache is not heads times head width). + LatentAttention, + /// No layer keeps a KV cache. + NoAttentionLayers, + /// State-space (pure or hybrid) or mixture-of-experts layers: the figures + /// are given, but part of the memory is either not counted (a recurrent + /// state) or not measured (expert compute buffers), so the total can be too + /// low and no fits / does-not-fit verdict is given. + UnmeasuredLayout, + /// A figure the formula needs is not in the header, or is there but not + /// plausible (a layer count past the parser's bound, head counts or widths + /// whose total is past any real model). When only the + /// feed-forward or embedding width is missing, the figures are still given + /// (with that width counted as zero, so they can be too low) and no + /// fits / does-not-fit verdict is. + MissingMetadata, + /// No GPU is visible to ROCm. + NoGpu, + /// The GPU's memory could not be measured. + AcceleratorMemoryUnknown, + /// An APU: the pool the engine allocates from is not readable. + UnifiedMemoryUnreadable, +} + +/// What kind of change a [`FitSuggestion`] is. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum SuggestionKind { + SmallerContext, + KvCacheType, + SmallerQuant, + CpuOffload, +} + +/// One thing that would make a model that does not fit, fit. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct FitSuggestion { + pub kind: SuggestionKind, + pub text: String, +} + +/// The estimate for one model at one context size. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ContextEstimate { + /// What the header was read from: a path, or `owner/repo` and file name. + pub source: String, + pub outcome: EstimateOutcome, + pub cannot_estimate_reason: Option, + /// The specific cause, in words, when `cannot_estimate_reason` is set. + pub detail: Option, + pub architecture: Option, + pub layers: Option, + /// The context size the figures below are for. + pub context_size: u64, + /// Set when the user did not ask for a context size and this is the default. + pub context_size_defaulted: bool, + pub kv_cache_type: KvCacheType, + pub weights_gib: Option, + pub kv_cache_gib: Option, + pub compute_buffer_gib: Option, + pub runtime_reserve_gib: Option, + pub total_gib: Option, + /// The same figures in bytes (rounded to the nearest byte), for comparing each + /// component against a measured load without unit conversion. The `*_gib` + /// fields above are these divided by 2^30. + pub weights_bytes: Option, + pub kv_cache_bytes: Option, + pub compute_buffer_bytes: Option, + pub runtime_reserve_bytes: Option, + pub total_bytes: Option, + /// KV cache bytes added by each token of context at `kv_cache_type`. + pub kv_cache_bytes_per_token: Option, + /// Dedicated VRAM, when it was measured. + pub available_gib: Option, + /// The largest context that fits at `kv_cache_type`: a multiple of 256, or + /// the context the model was trained for when that fits and is the limit. + /// `None` when not even one 256-token step fits, or when nothing was + /// compared (the GPU's memory was not measured). It can sit below + /// `context_size` for an outcome of `fits`: a request smaller than one step, + /// or beyond the trained context, fits without being a step the ceiling + /// counts. + pub max_context: Option, + /// The context length the model was trained for. + pub training_context: Option, + /// Set for a sliding-window model, whose KV cache is sized as if every layer + /// attended over the whole context: `max_context` is then a conservative + /// figure, not the exact boundary. + #[serde(default)] + pub conservative_ceiling: bool, + /// What would make it fit; empty unless the outcome is `does_not_fit`. + pub suggestions: Vec, + pub notes: Vec, +} + +/// What was asked. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FitRequest { + pub context_size: u64, + pub context_size_defaulted: bool, + pub kv_cache_type: KvCacheType, +} + +impl ContextEstimate { + /// An estimate that could not be made because opening the file failed. + /// Carries no figure: nothing was learnt about the model. + /// + /// Usually the header could not be reached or read + /// ([`CannotEstimateReason::HeaderUnreadable`]). A name refused as a + /// multimodal projector ([`ProjectorNamed`]) is + /// [`CannotEstimateReason::NotAModel`] instead: nothing failed to be read, the + /// file is not a model. + #[must_use] + pub fn unreadable(source: &str, request: &FitRequest, error: &anyhow::Error) -> Self { + let (reason, detail) = match error.downcast_ref::() { + Some(refusal) => (CannotEstimateReason::NotAModel, refusal.to_string()), + None => ( + CannotEstimateReason::HeaderUnreadable, + format!("the GGUF header could not be read: {error:#}"), + ), + }; + Self { + source: source.to_owned(), + outcome: EstimateOutcome::CannotEstimate, + cannot_estimate_reason: Some(reason), + detail: Some(detail), + architecture: None, + layers: None, + context_size: request.context_size, + context_size_defaulted: request.context_size_defaulted, + kv_cache_type: request.kv_cache_type, + weights_gib: None, + kv_cache_gib: None, + compute_buffer_gib: None, + runtime_reserve_gib: None, + total_gib: None, + weights_bytes: None, + kv_cache_bytes: None, + compute_buffer_bytes: None, + runtime_reserve_bytes: None, + total_bytes: None, + kv_cache_bytes_per_token: None, + available_gib: None, + max_context: None, + training_context: None, + conservative_ceiling: false, + suggestions: Vec::new(), + notes: Vec::new(), + } + } +} + +/// The pieces of the total that do not depend on the context size, plus the +/// KV-cache cost per token. +struct Sizing { + /// Everything after the header: what a bits-per-weight figure is over. + data: f64, + /// The part of `data` that lands in VRAM. llama.cpp keeps the input + /// embedding (`token_embd.weight`) in host memory when the file has its own + /// `output.weight`, and then it is left out; with tied embeddings it is the + /// output layer's weight on the GPU, and `weights` is all of `data`. + weights: f64, + /// The compute buffer's size before anything that grows with the context. + compute_base: f64, + /// KV elements cached per token across all layers. + kv_elements_per_token: f64, + /// KV elements per token in the widest single layer. + widest_layer_kv_elements: f64, + layers: u64, +} + +impl Sizing { + fn kv_bytes(&self, context: u64, kv: KvCacheType) -> f64 { + self.kv_elements_per_token * context as f64 * kv.bytes_per_element() + } + + /// What one more token of context costs in the compute buffer. The attention + /// mask grows with it. A quantized KV cache is also dequantized to 16-bit + /// floats a layer at a time, which the measured loads showed adding up to + /// that layer's K and V width per token; where the fixed part is larger than + /// that it hides it, so this errs high for a small model. + fn compute_bytes_per_token(&self, kv: KvCacheType) -> f64 { + let mask = COMPUTE_BATCH_TOKENS * MASK_BYTES; + if kv == KvCacheType::F16 { + mask + } else { + self.widest_layer_kv_elements.mul_add(2.0, mask) + } + } + + fn compute_bytes(&self, context: u64, kv: KvCacheType) -> f64 { + self.compute_bytes_per_token(kv) + .mul_add(context as f64, self.compute_base) + } + + /// Bytes that do not depend on the context size. + fn fixed(&self) -> f64 { + self.weights + self.compute_base + RUNTIME_RESERVE_BYTES + } + + /// Bytes each token of context adds: its KV cache and its compute buffer. + fn bytes_per_token(&self, kv: KvCacheType) -> f64 { + self.kv_elements_per_token + .mul_add(kv.bytes_per_element(), self.compute_bytes_per_token(kv)) + } + + fn total(&self, context: u64, kv: KvCacheType) -> f64 { + self.bytes_per_token(kv) + .mul_add(context as f64, self.fixed()) + } + + /// The total the report shows for `weights` of weights at `context` and + /// `kv`: its four parts, each as printed (whole MiB, rounded up), summed. + /// + /// This is the one figure a fit is decided on. The verdict, the largest + /// context and every "which fits" suggestion go through [`Self::fits`], so + /// none of them can disagree about whether a given context fits. + fn shown_total_with(&self, weights: f64, context: u64, kv: KvCacheType) -> f64 { + shown_sum([ + weights, + self.kv_bytes(context, kv), + self.compute_bytes(context, kv), + RUNTIME_RESERVE_BYTES, + ]) + } + + fn shown_total(&self, context: u64, kv: KvCacheType) -> f64 { + self.shown_total_with(self.weights, context, kv) + } + + fn fits(&self, context: u64, kv: KvCacheType, available: f64) -> bool { + self.shown_total(context, kv) <= available + } + + /// Whether the GPU holds `on_gpu` of the layers at `context` and `kv`, by the + /// same measure as [`Self::fits`]: the layers' share of the weights and the + /// cache, plus the compute buffer and the reserve, as the report would show + /// them. At every layer this is `fits`. + fn layers_fit(&self, on_gpu: u64, context: u64, kv: KvCacheType, available: f64) -> bool { + let share = on_gpu as f64 / self.layers as f64; + shown_sum([ + self.weights * share, + self.kv_bytes(context, kv) * share, + self.compute_bytes(context, kv), + RUNTIME_RESERVE_BYTES, + ]) <= available + } + + /// The part of a `bits_per_weight` quantization of `params` weights that + /// lands in VRAM: the same share of the file as now, because whether the + /// input embedding stays in host memory (it does unless the embeddings are + /// tied) does not change with the quantization. + fn quant_weights(&self, params: f64, bits_per_weight: f64) -> f64 { + params * bits_per_weight / 8.0 * (self.weights / self.data) + } + + /// The largest context that fits in `available` bytes at `kv`, in steps of + /// [`CONTEXT_STEP`] and no larger than `training` (the trained context, as + /// [`trained_context`] gives it, so never zero): the trained context itself + /// where it fits, else the largest step for which [`Self::fits`] holds. + fn max_context(&self, available: f64, kv: KvCacheType, training: Option) -> Option { + let room = available - self.fixed(); + if room <= 0.0 { + return None; + } + let per_token = self.bytes_per_token(kv); + let raw = (room / per_token).floor(); + // `as` saturates; a context beyond u64 is a model with no KV cost, which + // `sizing` has already refused. + let cap = training; + // Where the estimate reaches the cap, the search starts at the cap + // itself, whole step or not: rounding first would start a cap of 1500 + // at 1280, and the climb below never passes the cap to reach it. + let mut context = cap + .filter(|cap| raw as u64 >= *cap) + .unwrap_or((raw as u64) / CONTEXT_STEP * CONTEXT_STEP); + // The estimate above is on the exact total; what is decided on is the + // shown one, a few MiB higher. Step down until it fits, then up while the + // next step still does, so the answer is exactly the predicate's. + // A cap that is not a whole number of steps (a trained context of 250, or + // of 600) is where this starts, so the first step down lands on a whole + // step and a start below one step goes to zero rather than below it. + while context > 0 && !self.fits(context, kv, available) { + context = if context.is_multiple_of(CONTEXT_STEP) { + context - CONTEXT_STEP + } else { + context / CONTEXT_STEP * CONTEXT_STEP + }; + } + while cap.is_none_or(|cap| context + CONTEXT_STEP <= cap) + && self.fits(context + CONTEXT_STEP, kv, available) + { + context += CONTEXT_STEP; + } + (context > 0).then_some(context) + } +} + +/// The context length the model was trained for, when the header says one. A +/// header that declares zero declares none: no model was trained for no tokens, +/// and a cap of zero would leave no context that fits and a note that every +/// request is "beyond the 0 tokens the model was trained for". +fn trained_context(header: &GgufHeader) -> Option { + header.context_length.filter(|trained| *trained > 0) +} + +fn cannot( + reason: CannotEstimateReason, + detail: impl Into, +) -> (CannotEstimateReason, String) { + (reason, detail.into()) +} + +/// Text from a stranger's header as it may be shown: every character that would +/// change how a terminal renders the text around it (a control, a +/// bidirectional override, a line separator) becomes `?`. The architecture name +/// is the one header string the report prints. +fn shown_text(text: &str) -> String { + text.chars() + .map(|c| { + if crate::terminal::is_control_or_format(c) + || crate::terminal::is_control_or_line_separator(c) + { + '?' + } else { + c + } + }) + .collect() +} + +/// Whether the header lacks the feed-forward or the embedding width, which size +/// the compute buffer's fixed part. A width of zero is no width: no real model +/// has one, and taking it at its word would understate the buffer the same way. +fn widths_missing(header: &GgufHeader) -> bool { + let known = |width: Option| width.is_some_and(|width| width > 0); + !known(header.feed_forward_length) || !known(header.embedding_length) +} + +/// Work out the sizing, or why it cannot be worked out. +fn sizing(header: &GgufHeader, file_size: u64) -> Result { + if header.split_count.is_some_and(|count| count > 1) { + return Err(cannot( + CannotEstimateReason::SplitFile, + "this file is one shard of a split model; its weights and metadata are spread over \ + several files, so one file's size is not the model's", + )); + } + if header.latent_attention { + return Err(cannot( + CannotEstimateReason::LatentAttention, + "this model uses latent attention, whose KV cache is not laid out as heads times \ + head width, so the estimate's formula does not describe it", + )); + } + let missing = |what: &str| { + cannot( + CannotEstimateReason::MissingMetadata, + format!("the GGUF header does not carry {what}"), + ) + }; + let layers = header + .block_count + .filter(|layers| *layers > 0) + .ok_or_else(|| missing("a layer count"))?; + // The sizing below loops once per layer. The parser refuses an implausible + // count; a header built some other way gets the same bound here, so no + // caller can turn a corrupt figure into a loop that runs until memory ends. + if layers > MAX_BLOCK_COUNT { + return Err(cannot( + CannotEstimateReason::MissingMetadata, + format!("the header's layer count ({layers}) is not a plausible model"), + )); + } + let data = file_size + .checked_sub(header.data_offset) + .filter(|w| *w > 0) + .ok_or_else(|| { + cannot( + CannotEstimateReason::MissingMetadata, + "the file is no larger than its header, so there are no weights to size", + ) + })?; + // The input embedding runs from its offset to the next tensor's, or to the + // end of the data when it is the last. Clamped so a header that disagrees + // with the file's size cannot make the weights negative. llama.cpp keeps it + // in host memory only when the file has its own output layer: with tied + // embeddings (no `output.weight`) the same tensor is the output layer's + // weight too, and is placed with that layer on the GPU, so it stays in. + let embedding = header + .token_embd + .filter(|_| header.has_output) + .map_or(0, |span| { + span.next_offset + .unwrap_or(data) + .min(data) + .saturating_sub(span.offset) + }); + let weights = data.saturating_sub(embedding); + + let mut kv_elements = 0_u64; + let mut widest_layer = 0_u64; + for layer in 0..usize::try_from(layers).unwrap_or(usize::MAX) { + let heads = header.head_count.as_ref().and_then(|p| p.at(layer)); + // llama.cpp reads an absent KV head count as "as many as query heads". + let kv_heads = match &header.head_count_kv { + Some(per_layer) => per_layer.at(layer), + None => heads, + } + .ok_or_else(|| missing("a KV head count for every layer"))?; + if kv_heads == 0 { + continue; + } + let width = |explicit: Option| -> Result { + if let Some(width) = explicit { + return Ok(width); + } + match (header.embedding_length, heads) { + (Some(embedding), Some(heads)) if heads > 0 => Ok(embedding / heads), + _ => Err(missing("the attention head width")), + } + }; + let per_head = width(header.key_length)?.saturating_add(width(header.value_length)?); + let layer_elements = kv_heads.saturating_mul(per_head); + widest_layer = widest_layer.max(layer_elements); + kv_elements = kv_elements.saturating_add(layer_elements); + } + if kv_elements == 0 { + return Err(cannot( + CannotEstimateReason::NoAttentionLayers, + "no layer of this model keeps a KV cache, so a context-size estimate has nothing to \ + size", + )); + } + + // The part of llama.cpp's compute buffer that does not grow with the + // context: three activations of the widest intermediate width and three of + // the embedding width per batch token, in 32-bit floats. Flash attention is + // assumed, so no attention-score matrix appears. This reproduced the measured + // buffer exactly for both calibration models. + let activations = 3.0_f64.mul_add( + header.feed_forward_length.unwrap_or(0) as f64, + 3.0 * header.embedding_length.unwrap_or(0) as f64, + ); + let compute_base = COMPUTE_BATCH_TOKENS * activations * 4.0; + Ok(Sizing { + data: data as f64, + weights: weights as f64, + compute_base, + kv_elements_per_token: kv_elements as f64, + widest_layer_kv_elements: widest_layer as f64, + layers, + }) +} + +/// A figure that has to cover a need, so it rounds up. +fn need(bytes: f64) -> String { + format!("{:.1} GiB", (bytes / GIB * 10.0).ceil() / 10.0) +} + +fn gib(bytes: f64) -> f64 { + bytes / GIB +} + +/// "1 token", "4096 tokens". +pub(crate) fn tokens(count: u64) -> String { + if count == 1 { + "1 token".to_owned() + } else { + format!("{count} tokens") + } +} + +/// A byte count in whole MiB, rounded up: a figure that has to cover a need. +const fn mib_up(bytes: u64) -> u64 { + bytes.div_ceil(MIB) +} + +/// The sum of four byte figures as the report shows them: each rounded to a +/// whole byte count as the byte fields are, then up to a whole MiB, and added. +fn shown_sum(parts: [f64; 4]) -> f64 { + let mib: u64 = parts.iter().map(|part| mib_up(part.round() as u64)).sum(); + (mib * MIB) as f64 +} + +/// The parts of an estimate as the report shows them -- each in whole MiB, +/// rounded up -- and their sum. The verdict is decided on that sum, so the +/// figures printed always add up and never contradict it. +fn shown_parts(estimate: &ContextEstimate) -> Option<([u64; 4], u64)> { + let parts = [ + mib_up(estimate.weights_bytes?), + mib_up(estimate.kv_cache_bytes?), + mib_up(estimate.compute_buffer_bytes?), + mib_up(estimate.runtime_reserve_bytes?), + ]; + Some((parts, parts.iter().sum())) +} + +/// Estimate whether the model described by `header` fits at `request`. +/// +/// `file_size` is the size of the whole file in bytes; the weights are what +/// follows the header. `memory` and `system_ram_gib` are the host's, as the +/// readiness check measured them. +#[must_use] +pub fn estimate_fit( + header: &GgufHeader, + file_size: u64, + source: &str, + request: &FitRequest, + memory: AcceleratorMemory, + system_ram_gib: Option, +) -> ContextEstimate { + let mut estimate = ContextEstimate { + source: source.to_owned(), + outcome: EstimateOutcome::CannotEstimate, + cannot_estimate_reason: None, + detail: None, + architecture: header.architecture.as_deref().map(shown_text), + layers: header.block_count, + context_size: request.context_size, + context_size_defaulted: request.context_size_defaulted, + kv_cache_type: request.kv_cache_type, + weights_gib: None, + kv_cache_gib: None, + compute_buffer_gib: None, + runtime_reserve_gib: None, + total_gib: None, + weights_bytes: None, + kv_cache_bytes: None, + compute_buffer_bytes: None, + runtime_reserve_bytes: None, + total_bytes: None, + kv_cache_bytes_per_token: None, + available_gib: memory.measured_gib(), + max_context: None, + training_context: trained_context(header), + conservative_ceiling: header.sliding_window, + suggestions: Vec::new(), + notes: Vec::new(), + }; + let sizing = match sizing(header, file_size) { + Ok(sizing) => sizing, + Err((reason, detail)) => { + estimate.cannot_estimate_reason = Some(reason); + estimate.detail = Some(detail); + return estimate; + } + }; + + let kv = request.kv_cache_type; + let total = sizing.total(request.context_size, kv); + if !total.is_finite() || total >= MAX_PLAUSIBLE_BYTES { + estimate.cannot_estimate_reason = Some(CannotEstimateReason::MissingMetadata); + estimate.detail = Some(format!( + "at {} the figures this header implies come to more than an exabyte, which no model \ + or host has: the header's head counts or widths look corrupt, or the context size \ + is not a real one (the most an estimate is made for is {MAX_CONTEXT_SIZE})", + tokens(request.context_size) + )); + return estimate; + } + estimate.weights_gib = Some(gib(sizing.weights)); + estimate.kv_cache_gib = Some(gib(sizing.kv_bytes(request.context_size, kv))); + let compute = sizing.compute_bytes(request.context_size, kv); + estimate.compute_buffer_gib = Some(gib(compute)); + estimate.runtime_reserve_gib = Some(gib(RUNTIME_RESERVE_BYTES)); + estimate.total_gib = Some(gib(total)); + let bytes = |value: f64| Some(value.round() as u64); + estimate.weights_bytes = bytes(sizing.weights); + estimate.kv_cache_bytes = bytes(sizing.kv_bytes(request.context_size, kv)); + estimate.compute_buffer_bytes = bytes(compute); + estimate.runtime_reserve_bytes = bytes(RUNTIME_RESERVE_BYTES); + estimate.total_bytes = bytes(total); + estimate.kv_cache_bytes_per_token = Some(sizing.kv_elements_per_token * kv.bytes_per_element()); + + if header.sliding_window { + estimate.notes.push( + "this model uses sliding-window attention; its KV cache is sized as if every layer \ + attended over the full context, so the real figure can be lower and the largest \ + context below is conservative" + .to_owned(), + ); + } + if header.ssm { + estimate.notes.push( + "this model has state-space layers: the recurrent state they keep is not counted, so \ + the real figure can be higher; its KV cache is sized for every layer unless the \ + header lists head counts per layer, so that part can be lower" + .to_owned(), + ); + } + if header.moe { + estimate.notes.push( + "this is a mixture-of-experts model: the weights are the file's exact size, but the \ + compute buffer is sized from one feed-forward width and was not measured for \ + experts, so the real figure can be higher" + .to_owned(), + ); + } + if widths_missing(header) { + estimate.notes.push( + "the header does not carry both the feed-forward and the embedding width, so the \ + compute buffer's fixed part is understated and the real figure can be higher" + .to_owned(), + ); + } + if let Some(trained) = trained_context(header) + && request.context_size > trained + { + estimate.notes.push(format!( + "{} is beyond the {} the model was trained for", + tokens(request.context_size), + tokens(trained) + )); + } + + let Some(available_gib) = memory.measured_gib() else { + let (reason, detail) = match memory { + AcceleratorMemory::None => ( + CannotEstimateReason::NoGpu, + "no GPU is visible to ROCm, so there is no memory to compare against", + ), + AcceleratorMemory::UnifiedMemoryUnreadable => ( + CannotEstimateReason::UnifiedMemoryUnreadable, + "this host has no dedicated VRAM and the CLI cannot yet read the memory pool its \ + engine allocates from, so there is nothing to compare the figures against", + ), + AcceleratorMemory::Unknown | AcceleratorMemory::Dedicated(_) => ( + CannotEstimateReason::AcceleratorMemoryUnknown, + "the GPU memory on this host could not be read, so there is nothing to compare \ + the figures against", + ), + }; + estimate.cannot_estimate_reason = Some(reason); + estimate.detail = Some(detail.to_owned()); + return estimate; + }; + // A layout whose figure can be too low gets the figures and no verdict: a + // confident "fits" that is wrong costs a user a failed load, and a margin + // with no measurement behind it would be a guess. The host's memory is + // still reported, so the reader can compare it themselves. + let widths_missing = widths_missing(header); + if header.ssm || header.moe || widths_missing { + let why = match (header.ssm, header.moe) { + (true, true) => { + "it has state-space layers, whose recurrent state is not counted, and \ + mixture-of-experts layers, whose compute buffer was not measured" + } + (true, false) => "it has state-space layers, whose recurrent state is not counted", + (false, true) => { + "it is a mixture-of-experts model, whose compute buffer was not measured" + } + (false, false) => { + "the header does not carry both the feed-forward and the embedding width, \ + which size the compute buffer's fixed part" + } + }; + // A layout the formula does not measure is the stronger statement; a + // missing width only matters on its own, and its note says so either way. + estimate.cannot_estimate_reason = Some(if header.ssm || header.moe { + CannotEstimateReason::UnmeasuredLayout + } else { + CannotEstimateReason::MissingMetadata + }); + estimate.detail = Some(format!( + "no fits / does-not-fit verdict for this model: {why}, so the total above can be too \ + low; compare it with this GPU's {} MiB yourself and leave headroom", + whole_mib_offered(available_gib) + )); + return estimate; + } + let available = available_gib * GIB; + estimate.max_context = sizing.max_context(available, kv, trained_context(header)); + + // Decided on the sum of the parts as the report shows them (each rounded up + // to a whole MiB), which is at most a few MiB above the exact total: what is + // printed then always adds up, and can never sit next to a contradicting + // verdict. `max_context` and the suggestions use the same predicate. + if sizing.fits(request.context_size, kv, available) { + estimate.outcome = EstimateOutcome::Fits; + return estimate; + } + estimate.outcome = EstimateOutcome::DoesNotFit; + estimate.suggestions = suggestions( + header, + &sizing, + request, + available, + estimate.max_context, + system_ram_gib, + ); + estimate +} + +fn suggestions( + header: &GgufHeader, + sizing: &Sizing, + request: &FitRequest, + available: f64, + max_context: Option, + system_ram_gib: Option, +) -> Vec { + let kv = request.kv_cache_type; + let context = request.context_size; + let mut out = Vec::new(); + + if let Some(max) = max_context { + let qualifier = if header.sliding_window { + " (a conservative figure for this model: its sliding-window layers are sized at the \ + full context)" + } else { + "" + }; + out.push(FitSuggestion { + kind: SuggestionKind::SmallerContext, + text: format!( + "use a context of {} or less (llama-server --ctx-size {max}); that is the largest \ + that fits with the KV cache at {kv}{qualifier}", + tokens(max) + ), + }); + } + + // A narrower cache is only worth naming when it lowers the need. A quantized + // cache is also dequantized a layer at a time, which adds to the compute + // buffer; in a model with one or two cached layers that costs more than the + // cache saves, and "a q8_0 cache would cut the need to ..." would be false. + let now = sizing.shown_total(context, kv); + for smaller in KvCacheType::ALL.into_iter().filter(|other| { + other.bytes_per_element() < kv.bytes_per_element() + && sizing.shown_total(context, *other) < now + }) { + let total_mib = sizing.shown_total(context, smaller) as u64 / MIB; + let text = if sizing.fits(context, smaller, available) { + let largest = sizing + .max_context(available, smaller, trained_context(header)) + .map_or(String::new(), |max| { + format!("; the largest context then is {}", tokens(max)) + }); + format!( + "use a {smaller} KV cache (llama-server --cache-type-k {smaller} --cache-type-v \ + {smaller}): about {total_mib} MiB at {}, which fits{largest}", + tokens(context) + ) + } else { + format!( + "a {smaller} KV cache would cut the need to about {total_mib} MiB, which is still \ + more than this GPU offers" + ) + }; + out.push(FitSuggestion { + kind: SuggestionKind::KvCacheType, + text, + }); + } + if out + .iter() + .any(|suggestion| suggestion.kind == SuggestionKind::KvCacheType) + { + out.push(FitSuggestion { + kind: SuggestionKind::KvCacheType, + text: "a quantized V cache needs flash attention enabled in llama.cpp".to_owned(), + }); + } + + let params = header.param_count as f64; + let current_bpw = if header.param_count > 0 { + sizing.data * 8.0 / params + } else { + 0.0 + }; + // Bits per weight outside what any real quantization reaches means the + // header's tensor shapes and the file's size do not describe the same + // model (a truncated or padded file, or a fixture); a "smaller quant" + // computed from that would be noise. + if (MIN_PLAUSIBLE_BITS_PER_WEIGHT..=MAX_PLAUSIBLE_BITS_PER_WEIGHT).contains(¤t_bpw) { + let candidates: Vec<(&str, f64)> = QUANT_BITS_PER_WEIGHT + .iter() + .rev() + .filter(|(_, bpw)| *bpw < current_bpw * QUANT_SMALLER_FACTOR) + .map(|(label, bpw)| (*label, sizing.quant_weights(params, *bpw))) + .collect(); + let fitting = candidates + .iter() + .find(|(_, weights)| sizing.shown_total_with(*weights, context, kv) <= available); + let text = match (fitting, candidates.last()) { + (Some((label, weights)), _) => Some(format!( + "a smaller quantization of the same model, about {label} size (roughly {} of \ + weights against {} now), would fit at {} if the repository \ + publishes one", + need(*weights), + need(sizing.weights), + tokens(context) + )), + (None, Some((label, _))) => Some(format!( + "no smaller quantization in the usual range, down to about {label}, fits at \ + {} with the KV cache at {kv}", + tokens(context) + )), + (None, None) => None, + }; + if let Some(text) = text { + out.push(FitSuggestion { + kind: SuggestionKind::SmallerQuant, + text, + }); + } + } + + out.extend(offload_suggestion( + sizing, + request, + available, + system_ram_gib, + )); + out +} + +/// Partial offload: keep as many layers as fit on the GPU and run the rest from +/// system RAM. Weights are spread evenly over the layers, which ignores the +/// embedding and output tensors; the figure is a rough guide. +fn offload_suggestion( + sizing: &Sizing, + request: &FitRequest, + available: f64, + system_ram_gib: Option, +) -> Option { + let layers = sizing.layers; + let context = request.context_size; + let kv = request.kv_cache_type; + let per_layer = (sizing.weights + sizing.kv_bytes(context, kv)) / layers as f64; + let compute = sizing.compute_bytes(context, kv); + let gpu_fits = |n: u64| sizing.layers_fit(n, context, kv, available); + let room = available - compute - RUNTIME_RESERVE_BYTES; + let mut on_gpu = if room > 0.0 { + ((room / per_layer).floor() as u64).min(layers) + } else { + 0 + }; + while on_gpu > 0 && !gpu_fits(on_gpu) { + on_gpu -= 1; + } + while on_gpu < layers && gpu_fits(on_gpu + 1) { + on_gpu += 1; + } + if on_gpu == 0 || on_gpu >= layers { + return None; + } + let host_bytes = per_layer * (layers - on_gpu) as f64; + if system_ram_gib.is_some_and(|ram| host_bytes > ram * GIB) { + return None; + } + let ram_note = if system_ram_gib.is_some() { + "" + } else { + " (system RAM could not be read, so that is unchecked)" + }; + Some(FitSuggestion { + kind: SuggestionKind::CpuOffload, + text: format!( + "keep about {on_gpu} of {layers} layers on the GPU (llama-server --n-gpu-layers \ + {on_gpu}); the other {} of weights and cache stay in system RAM{ram_note}, and \ + generation is slower", + need(host_bytes) + ), + }) +} + +/// A host's memory in whole MiB, rounded down: a host never reads as offering +/// more than it does, and an integer figure sits beside the integer figures in +/// the report. The `1e-6` keeps floating-point error in the GiB figure from +/// costing a whole MiB. Every line that shows the host's memory in MiB goes +/// through here, so two lines never show it differently. +const fn whole_mib_offered(gib: f64) -> f64 { + gib.mul_add(1024.0, 1e-6).floor() +} + +/// The evidence lines for an estimate, in the report's own voice. +#[must_use] +pub fn evidence_lines(estimate: &ContextEstimate) -> Vec { + let mut lines = Vec::new(); + // Where the figures came from is only worth saying when there was a header + // to read them from; for an unreadable one, or a file that is not a model, + // the detail says what happened. + if !matches!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::HeaderUnreadable | CannotEstimateReason::NotAModel) + ) { + lines.push(format!( + "estimated from the GGUF header of {} ({} architecture, {} layers); the weights were \ + not downloaded", + estimate.source, + estimate.architecture.as_deref().unwrap_or("unknown"), + estimate + .layers + .map_or_else(|| "unknown".to_owned(), |layers| layers.to_string()), + )); + } + if let Some(detail) = &estimate.detail { + lines.push(detail.clone()); + } + if let Some(([weights, kv, compute, reserve], total)) = shown_parts(estimate) { + lines.push(format!( + "at {}{}: weights {weights} MiB + KV cache ({}) {kv} MiB + compute buffer {compute} \ + MiB + runtime reserve {reserve} MiB = about {total} MiB ({})", + tokens(estimate.context_size), + if estimate.context_size_defaulted { + " (the default; pass --context-size to change it)" + } else { + "" + }, + estimate.kv_cache_type, + need((total * MIB) as f64), + )); + } + if let Some(available) = estimate.available_gib { + let offered = whole_mib_offered(available); + lines.push(format!( + "the GPU with the most dedicated VRAM that ROCm can see has {offered} MiB ({}) in \ + total; memory other processes already hold is not subtracted", + format_gib(available) + )); + let conservative = if estimate.conservative_ceiling { + " (conservative for this model: its sliding-window layers are sized at the full \ + context)" + } else { + "" + }; + match (estimate.outcome, estimate.max_context) { + (EstimateOutcome::CannotEstimate, _) => {} + // A request that fits is never followed by a ceiling below it: that + // would read as the request not fitting. When the request is smaller + // than one step or beyond the trained context, the ceiling is left + // out and the trained-context note says what applies. + (EstimateOutcome::Fits, Some(max)) if max < estimate.context_size => {} + (EstimateOutcome::Fits, None) => {} + (_, Some(max)) => lines.push(format!( + "largest context that fits, with the KV cache at {}: {}{conservative}", + estimate.kv_cache_type, + tokens(max) + )), + (_, None) => lines.push(format!( + "not even a {CONTEXT_STEP}-token context fits with the KV cache at {}", + estimate.kv_cache_type + )), + } + } + lines +} + +/// The standing caveat that goes with every estimate that has figures. +/// +/// It names the GPU and engine build the allowances were calibrated on, because +/// that is what a reader of the output needs to judge whether their own setup is +/// like it, and printed output cannot link anywhere. How close the calibration +/// came is in the module documentation, and only there. +pub const ESTIMATE_CAVEAT: &str = "these are estimates from the file's metadata, with the compute buffer and runtime \ + overhead calibrated on one GPU and engine build (an MI300X running llama.cpp b9752) with \ + flash attention on, which is assumed here (without it llama.cpp also allocates an \ + attention-score matrix that is not counted, and the figure is too low); another build or GPU can differ, so leave some headroom \ + and check with a real run"; + +/// The standing note about `rocm serve` when there are suggestions. +/// +/// The options they name are the engine's, and `rocm serve` does not pass them +/// through today. `serve_takes_no_estimate_options_yet` in `apps/rocm/src/cli.rs` fails +/// when `rocm serve` gains one of them, so this text cannot go stale unnoticed. +pub const SERVE_OPTIONS_NOTE: &str = "`rocm serve` does not take a context size, KV cache type or \ + GPU layer count yet, and passes none to the engine, so a model it serves gets the engine's \ + own default context; the options named above are llama.cpp's own llama-server flags"; + +/// The same fact for an estimate with no suggestions: the context size it was +/// sized for is not one `rocm serve` would use. Pinned by the same test as +/// [`SERVE_OPTIONS_NOTE`]. +pub const SERVE_CONTEXT_NOTE: &str = "`rocm serve` does not take a context size yet and passes \ + none to the engine, so a model it serves gets the engine's own default context, which can \ + differ from the size estimated here"; + +#[cfg(test)] +mod tests { + use super::*; + use crate::gguf::test_builder::{Builder, llama_like}; + use crate::gguf::{GgufHeader, parse_gguf_header}; + use std::io::Cursor; + + const MIB: u64 = 1024 * 1024; + const GIB_U: u64 = 1024 * MIB; + + fn header_of(builder: &Builder, data: usize) -> (GgufHeader, u64) { + let file = builder.file(data); + let header = parse_gguf_header(&mut Cursor::new(&file)).unwrap(); + (header, file.len() as u64) + } + + /// 32 layers, 8 KV heads, key and value width 128 each: 32 * 8 * 256 = 65536 + /// elements per token, so 128 KiB per token at f16 and exactly 1 GiB at 8192. + /// + /// `data` is the size of the weights. The file is never built: the header is + /// parsed from a header-only file and the size is declared, so a test of a + /// multi-gigabyte model allocates nothing. + fn llama(data: usize) -> (GgufHeader, u64) { + let header = parse_gguf_header(&mut Cursor::new(llama_like(10).file(0))).unwrap(); + let size = header.data_offset + data as u64; + (header, size) + } + + fn request(context_size: u64, kv: KvCacheType) -> FitRequest { + FitRequest { + context_size, + context_size_defaulted: false, + kv_cache_type: kv, + } + } + + fn estimate( + header: &GgufHeader, + size: u64, + context: u64, + kv: KvCacheType, + memory: AcceleratorMemory, + ) -> ContextEstimate { + estimate_fit( + header, + size, + "test.gguf", + &request(context, kv), + memory, + Some(64.0), + ) + } + + #[test] + fn the_kv_cache_is_layers_times_kv_heads_times_head_width_times_context() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + let kv = result.kv_cache_gib.unwrap(); + assert!( + (kv - 1.0).abs() < 1e-9, + "32 layers x 8 kv heads x 256 x 8192 x 2 B = 1 GiB, got {kv}" + ); + // The quantized types are the block sizes, not a round factor. + let q8 = estimate( + &header, + size, + 8192, + KvCacheType::Q8_0, + AcceleratorMemory::Dedicated(64.0), + ); + assert!((q8.kv_cache_gib.unwrap() - 34.0 / 64.0).abs() < 1e-9); + let q4 = estimate( + &header, + size, + 8192, + KvCacheType::Q4_0, + AcceleratorMemory::Dedicated(64.0), + ); + assert!((q4.kv_cache_gib.unwrap() - 18.0 / 64.0).abs() < 1e-9); + } + + /// The vocabulary `llama_like(10)` declares puts its `token_embd.weight` at + /// the start of the data and the next tensor 10 * 4096 bytes later. + const EMBD_BYTES: u64 = 10 * 4096; + /// 512 batch tokens x (3 x ffn 14336 + 3 x embedding 4096) x 4 bytes: the + /// 108 MiB the Llama-shaped calibration model's compute buffer measured. + const LLAMA_COMPUTE_BASE: u64 = 512 * (3 * 14336 + 3 * 4096) * 4; + + #[test] + fn the_total_is_the_sum_of_its_parts() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + // The file has its own output layer, so the input embedding stays in + // host memory and is not VRAM. + let weights = (size - header.data_offset - EMBD_BYTES) as f64 / GIB; + // The attention mask adds 512 x 2 bytes per token of context. + let compute = (LLAMA_COMPUTE_BASE + 512 * 2 * 8192) as f64 / GIB; + assert!((result.weights_gib.unwrap() - weights).abs() < 1e-12); + assert!((result.compute_buffer_gib.unwrap() - compute).abs() < 1e-12); + assert!((result.runtime_reserve_gib.unwrap() - 1.25).abs() < 1e-12); + let expected = weights + 1.0 + compute + 1.25; + assert!((result.total_gib.unwrap() - expected).abs() < 1e-9); + } + + #[test] + fn the_byte_fields_are_exact_and_add_up_to_the_total() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + // 32 layers x 8 kv heads x (128 + 128) x 2 bytes = 131072 per token. + assert_eq!(result.kv_cache_bytes_per_token, Some(131_072.0)); + assert_eq!(result.kv_cache_bytes, Some(131_072 * 8192)); + assert_eq!( + result.weights_bytes, + Some(size - header.data_offset - EMBD_BYTES) + ); + assert_eq!(result.runtime_reserve_bytes, Some(1280 * MIB)); + assert_eq!( + result.compute_buffer_bytes, + Some(LLAMA_COMPUTE_BASE + 512 * 2 * 8192) + ); + let parts = result.weights_bytes.unwrap() + + result.kv_cache_bytes.unwrap() + + result.compute_buffer_bytes.unwrap() + + result.runtime_reserve_bytes.unwrap(); + assert_eq!(result.total_bytes, Some(parts)); + } + + #[test] + fn a_per_layer_head_array_skips_layers_without_a_cache() { + let builder = Builder::new() + .string("general.architecture", "hybrid") + .u32("hybrid.block_count", 4) + .u32("hybrid.embedding_length", 1024) + .u32("hybrid.attention.head_count", 8) + .u32_array("hybrid.attention.head_count_kv", &[0, 2, 0, 2]) + .tensor("t", &[1000]); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 1000, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + // 2 layers x 2 kv heads x (128 + 128) x 1000 tokens x 2 bytes + let expected = 2.0 * 2.0 * 256.0 * 1000.0 * 2.0 / GIB; + assert!((result.kv_cache_gib.unwrap() - expected).abs() < 1e-12); + } + + #[test] + fn an_absent_kv_head_count_means_as_many_as_query_heads() { + let builder = Builder::new() + .string("general.architecture", "mha") + .u32("mha.block_count", 2) + .u32("mha.embedding_length", 512) + .u32("mha.attention.head_count", 4) + .tensor("t", &[10]); + let (header, size) = header_of(&builder, 64); + let result = estimate( + &header, + size, + 100, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + // 2 layers x 4 heads x (128 + 128) x 100 x 2 bytes + let expected = 2.0 * 4.0 * 256.0 * 100.0 * 2.0 / GIB; + assert!((result.kv_cache_gib.unwrap() - expected).abs() < 1e-12); + } + + #[test] + fn explicit_key_and_value_lengths_override_the_derived_head_width() { + let builder = Builder::new() + .string("general.architecture", "wide") + .u32("wide.block_count", 1) + .u32("wide.embedding_length", 512) + .u32("wide.attention.head_count", 4) + .u32("wide.attention.head_count_kv", 1) + .u32("wide.attention.key_length", 192) + .u32("wide.attention.value_length", 128) + .tensor("t", &[10]); + let (header, size) = header_of(&builder, 64); + let result = estimate( + &header, + size, + 100, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + let expected = (192.0 + 128.0) * 100.0 * 2.0 / GIB; + assert!((result.kv_cache_gib.unwrap() - expected).abs() < 1e-12); + } + + /// The verdict is decided on the total the report shows -- the sum of its + /// parts, each rounded up to a whole MiB -- so a GPU of exactly that many MiB + /// fits and one MiB less does not, and what is printed always agrees. + #[test] + fn fit_is_decided_at_the_total_the_report_shows() { + let (header, size) = llama(GIB_U as usize / 1024); + let reference = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + let (_, shown_mib) = shown_parts(&reference).unwrap(); + let mib_gib = |mib: u64| mib as f64 / 1024.0; + let exact = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(mib_gib(shown_mib)), + ); + assert_eq!(exact.outcome, EstimateOutcome::Fits); + let under = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(mib_gib(shown_mib - 1)), + ); + assert_eq!(under.outcome, EstimateOutcome::DoesNotFit); + // The exact total is never above what is shown, and never more than a few + // MiB below it (one rounding per part). + let exact_bytes = reference.total_bytes.unwrap(); + assert!(exact_bytes <= shown_mib * MIB, "{exact_bytes} {shown_mib}"); + assert!(shown_mib * MIB - exact_bytes < 4 * MIB); + // Memory that is above the exact total and below the shown one: the + // verdict follows the figure that is printed, never the unrounded one. + assert!( + shown_mib * MIB - exact_bytes >= 2, + "the total is a whole number of MiB" + ); + let between = (exact_bytes + shown_mib * MIB) as f64 / 2.0 / GIB; + let tight = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(between), + ); + assert_eq!(tight.outcome, EstimateOutcome::DoesNotFit); + } + + /// The figures printed are the figures decided on: whole MiB, summing to the + /// total, with the host's memory in the same unit -- so no verdict can sit + /// beside a figure that contradicts it. + #[test] + fn the_printed_parts_add_up_and_never_contradict_the_verdict() { + let (header, size) = llama(GIB_U as usize / 1024); + let figures = |lines: &str| -> Vec { + let line = lines.lines().find(|line| line.starts_with("at ")).unwrap(); + regex::Regex::new(r"(\d+) MiB") + .unwrap() + .captures_iter(line) + .map(|found| found[1].parse().unwrap()) + .collect() + }; + // Small and large, fitting and not, f16 and quantized. + for (context, kv, gpu) in [ + (128, KvCacheType::F16, 24.0), + (8192, KvCacheType::Q8_0, 24.0), + (131_072, KvCacheType::F16, 24.0), + (4096, KvCacheType::F16, 1.0), + ] { + let result = estimate( + &header, + size, + context, + kv, + AcceleratorMemory::Dedicated(gpu), + ); + let lines = evidence_lines(&result).join("\n"); + let numbers = figures(&lines); + assert_eq!(numbers.len(), 5, "{lines}"); + assert_eq!(numbers[..4].iter().sum::(), numbers[4], "{lines}"); + // The GiB figure beside the MiB total is that same total, rounded up. + let in_gib = need((numbers[4] * MIB) as f64); + assert!( + lines.contains(&format!("= about {} MiB ({in_gib})", numbers[4])), + "{lines}" + ); + let offered = (gpu * 1024.0) as u64; + assert!(lines.contains(&format!("has {offered} MiB")), "{lines}"); + // The verdict is the comparison of the two printed figures. + assert_eq!( + result.outcome == EstimateOutcome::Fits, + numbers[4] <= offered, + "{lines}" + ); + } + // No part of a small model reads as a rounded-away zero. + let tiny = Builder::new() + .string("general.architecture", "t") + .u32("t.block_count", 2) + .u32("t.embedding_length", 128) + .u32("t.feed_forward_length", 256) + .u32("t.attention.head_count", 2) + .u32("t.attention.head_count_kv", 1) + .tensor("w", &[1024]); + let (header, size) = header_of(&tiny, 4096); + let result = estimate( + &header, + size, + 256, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + let lines = evidence_lines(&result).join("\n"); + assert!( + !lines.contains("0.0 GiB") && lines.contains("weights 1 MiB"), + "{lines}" + ); + } + + /// A GPU of exactly `mib` MiB. + fn gpu_mib(mib: u64) -> AcceleratorMemory { + AcceleratorMemory::Dedicated(mib as f64 / 1024.0) + } + + /// The total the report shows for this model at `context`, in whole MiB. + fn shown_mib(header: &GgufHeader, size: u64, context: u64, kv: KvCacheType) -> u64 { + shown_parts(&estimate( + header, + size, + context, + kv, + AcceleratorMemory::Dedicated(1024.0), + )) + .unwrap() + .1 + } + + #[test] + fn the_largest_context_is_rounded_down_to_a_step_and_capped_at_training() { + let (header, size) = llama(GIB_U as usize / 1024); + let kv = KvCacheType::F16; + let shown = |context| shown_mib(&header, size, context, kv); + let max_with = |mib: u64| estimate(&header, size, 100_000, kv, gpu_mib(mib)).max_context; + // A card with exactly the room the report shows for 24576 tokens. + assert_eq!(max_with(shown(24576)), Some(24576)); + // A step is 256 tokens, 32 MiB here: room for 100 more tokens (12 MiB) + // rounds down to the same step. + assert_eq!(max_with(shown(24576) + 12), Some(24576)); + // One MiB short of the step drops to the one below. + assert_eq!(max_with(shown(24576) - 1), Some(24320)); + // A huge GPU is capped at the context the model was trained for. + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(10_000.0), + ); + assert_eq!(result.max_context, Some(131_072)); + } + + #[test] + fn a_gpu_too_small_for_the_weights_has_no_context_that_fits() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(0.5), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + assert_eq!(result.max_context, None); + assert!( + !result + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::SmallerContext), + "a smaller context was suggested where none can fit: {:?}", + result.suggestions + ); + } + + #[test] + fn a_model_that_fits_has_no_suggestions_and_still_reports_its_ceiling() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(24.0), + ); + assert_eq!(result.outcome, EstimateOutcome::Fits); + assert!(result.suggestions.is_empty()); + assert_eq!(result.max_context, Some(131_072)); + } + + // The three memory states the readiness check distinguishes. In each the + // figures the file allows are still reported, and in none is anything + // compared. + #[test] + fn memory_that_cannot_be_compared_still_reports_what_the_model_needs() { + let (header, size) = llama(GIB_U as usize / 1024); + for (memory, reason) in [ + (AcceleratorMemory::None, CannotEstimateReason::NoGpu), + ( + AcceleratorMemory::Unknown, + CannotEstimateReason::AcceleratorMemoryUnknown, + ), + ( + AcceleratorMemory::UnifiedMemoryUnreadable, + CannotEstimateReason::UnifiedMemoryUnreadable, + ), + ] { + let result = estimate(&header, size, 4096, KvCacheType::F16, memory); + assert_eq!( + result.outcome, + EstimateOutcome::CannotEstimate, + "{memory:?}" + ); + assert_eq!(result.cannot_estimate_reason, Some(reason)); + assert!( + result.total_gib.is_some(), + "{memory:?}: the need was not reported" + ); + assert_eq!(result.available_gib, None); + assert_eq!(result.max_context, None); + assert!(result.suggestions.is_empty()); + } + } + + #[test] + fn suggestions_name_a_quantized_kv_cache_only_when_it_would_fit() { + let (header, size) = llama(GIB_U as usize / 1024); + let fixed = estimate( + &header, + size, + 0, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + .total_gib + .unwrap(); + // 8192 tokens: f16 needs 1 GiB, q8_0 0.53, q4_0 0.28. Room for 0.6. + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(fixed + 0.6), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + let texts: Vec<&str> = result.suggestions.iter().map(|s| s.text.as_str()).collect(); + assert!( + texts + .iter() + .any(|t| t.contains("q8_0 KV cache") && t.contains("which fits")), + "{texts:?}" + ); + assert!( + texts + .iter() + .any(|t| t.contains("q4_0 KV cache") && t.contains("which fits")), + "{texts:?}" + ); + assert!( + texts.iter().any(|t| t.contains("flash attention")), + "{texts:?}" + ); + // Room for 0.4: q8_0 no longer fits and must not be offered as if it did. + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(fixed + 0.4), + ); + let q8 = result + .suggestions + .iter() + .find(|s| s.text.contains("q8_0 KV cache")) + .unwrap(); + assert!(q8.text.contains("still more than"), "{}", q8.text); + let q4 = result + .suggestions + .iter() + .find(|s| s.text.contains("q4_0 KV cache")) + .unwrap(); + assert!(q4.text.contains("which fits"), "{}", q4.text); + } + + #[test] + fn an_already_quantized_cache_is_not_offered_itself() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 131_072, + KvCacheType::Q8_0, + AcceleratorMemory::Dedicated(4.0), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + // Only the KV-type suggestions are about the cache's type: the + // context-size one legitimately names the current type. + for suggestion in result + .suggestions + .iter() + .filter(|s| s.kind == SuggestionKind::KvCacheType) + { + assert!( + !suggestion.text.contains("q8_0 KV cache") + && !suggestion.text.contains("f16 KV cache"), + "{}", + suggestion.text + ); + } + let none_wider = estimate( + &header, + size, + 131_072, + KvCacheType::Q4_0, + AcceleratorMemory::Dedicated(4.0), + ); + assert!( + !none_wider + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::KvCacheType), + "the narrowest cache has no narrower one to suggest" + ); + } + + /// Weights sized like a 7B-parameter model at ~8.5 bits per weight: the + /// header declares one large tensor, and the file carries that many bytes. + fn eight_bit_seven_b() -> (GgufHeader, u64) { + let params: u64 = 7_000_000_000; + let builder = llama_like(10).tensor("big", &[params]); + // A real 7.4 GB buffer is not worth allocating: parse a header-only + // file and size the "file" from the declared data length. + let file = builder.file(0); + let header = parse_gguf_header(&mut Cursor::new(&file)).unwrap(); + let size = header.data_offset + (params as f64 * 8.5 / 8.0) as u64; + (header, size) + } + + #[test] + fn a_smaller_quant_is_suggested_only_when_the_estimate_says_it_fits() { + let (header, size) = eight_bit_seven_b(); + let quant_of = |gpu_gib: f64| { + let result = estimate( + &header, + size, + 2048, + KvCacheType::F16, + AcceleratorMemory::Dedicated(gpu_gib), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit, "{gpu_gib} GiB"); + result + .suggestions + .into_iter() + .find(|s| s.kind == SuggestionKind::SmallerQuant) + .unwrap() + .text + }; + // The 8.5 bpw weights are ~6.9 GiB. The suggestion is the LARGEST quant + // whose weights plus the other ~1.6 GiB (cache, compute buffer, reserve) + // fit, not the smallest that exists: Q5_K_M (~4.6 GiB) is what 6.5 GiB + // leaves room for, Q4_K_M (~3.9 GiB) what 6, and Q3_K_M (~3.2 GiB) what 5. + assert!(quant_of(6.5).contains("Q5_K_M size"), "{}", quant_of(6.5)); + assert!(quant_of(6.0).contains("Q4_K_M size"), "{}", quant_of(6.0)); + assert!(quant_of(5.0).contains("Q3_K_M size"), "{}", quant_of(5.0)); + // A GPU too small for any of them says so instead. + let text = quant_of(1.0); + assert!(text.contains("no smaller quantization"), "{text}"); + } + + #[test] + fn no_quant_is_suggested_when_the_tensor_shapes_and_the_file_size_disagree() { + // 2 GiB of "weights" for ~17 million declared parameters is about a + // thousand bits per weight: not a quantization, a mismatched file. + let (header, size) = llama(2 * GIB_U as usize); + let result = estimate( + &header, + size, + 2048, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1.0), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + assert!( + !result + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::SmallerQuant), + "{:?}", + result.suggestions + ); + } + + #[test] + fn no_smaller_quant_is_suggested_for_a_model_already_at_the_bottom_of_the_table() { + let params: u64 = 7_000_000_000; + let builder = llama_like(10).tensor("big", &[params]); + let file = builder.file(0); + let header = parse_gguf_header(&mut Cursor::new(&file)).unwrap(); + let size = header.data_offset + (params as f64 * 2.5 / 8.0) as u64; + let result = estimate( + &header, + size, + 2048, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1.0), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + assert!( + !result + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::SmallerQuant) + ); + } + + #[test] + fn partial_offload_counts_the_layers_that_fit_and_checks_system_ram() { + let (header, size) = llama(GIB_U as usize); + // ~1 GiB weights over 32 layers + 1 GiB KV at 8192: ~64 MiB per layer. + let fixed = estimate( + &header, + size, + 0, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + .total_gib + .unwrap(); + // Fixed allowances plus room for half the layers (1 GiB per ... 2 GiB total / 2). + let weights = (size - header.data_offset) as f64 / GIB; + let per_layer = (weights + 1.0) / 32.0; + let memory = AcceleratorMemory::Dedicated(fixed - weights + per_layer * 16.5); + let result = estimate(&header, size, 8192, KvCacheType::F16, memory); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + let offload = result + .suggestions + .iter() + .find(|s| s.kind == SuggestionKind::CpuOffload) + .unwrap(); + assert!(offload.text.contains("16 of 32 layers"), "{}", offload.text); + assert!( + offload.text.contains("--n-gpu-layers 16"), + "{}", + offload.text + ); + // With almost no system RAM the offloaded half has nowhere to go. + let starved = estimate_fit( + &header, + size, + "t", + &request(8192, KvCacheType::F16), + memory, + Some(0.01), + ); + assert!( + !starved + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::CpuOffload) + ); + // Unknown RAM: still offered, and the report says it is unchecked. + let unknown = estimate_fit( + &header, + size, + "t", + &request(8192, KvCacheType::F16), + memory, + None, + ); + let offload = unknown + .suggestions + .iter() + .find(|s| s.kind == SuggestionKind::CpuOffload) + .unwrap(); + assert!(offload.text.contains("unchecked"), "{}", offload.text); + } + + #[test] + fn no_offload_is_suggested_when_not_even_one_layer_fits() { + let (header, size) = llama(GIB_U as usize); + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(0.55), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + assert!( + !result + .suggestions + .iter() + .any(|s| s.kind == SuggestionKind::CpuOffload) + ); + } + + fn cannot_estimate(builder: &Builder) -> ContextEstimate { + let (header, size) = header_of(builder, 4096); + estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + } + + #[test] + fn layouts_the_formula_does_not_describe_are_not_guessed_at() { + let base = || { + Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 4) + .u32("x.embedding_length", 512) + .u32("x.attention.head_count", 8) + .tensor("t", &[10]) + }; + let split = cannot_estimate(&base().u32("split.count", 3)); + assert_eq!( + split.cannot_estimate_reason, + Some(CannotEstimateReason::SplitFile) + ); + let mla = cannot_estimate(&base().u32("x.attention.kv_lora_rank", 512)); + assert_eq!( + mla.cannot_estimate_reason, + Some(CannotEstimateReason::LatentAttention) + ); + let ssm = cannot_estimate( + &Builder::new() + .string("general.architecture", "mamba") + .u32("mamba.block_count", 4) + .u32("mamba.ssm.state_size", 16) + .u32_array("mamba.attention.head_count_kv", &[0, 0, 0, 0]) + .tensor("t", &[10]), + ); + assert_eq!( + ssm.cannot_estimate_reason, + Some(CannotEstimateReason::NoAttentionLayers) + ); + let no_layers = cannot_estimate( + &Builder::new() + .string("general.architecture", "x") + .tensor("t", &[10]), + ); + assert_eq!( + no_layers.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + let no_heads = cannot_estimate( + &Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 4) + .tensor("t", &[10]), + ); + assert_eq!( + no_heads.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + let short_array = cannot_estimate(&base().u32_array("x.attention.head_count_kv", &[1, 1])); + assert_eq!( + short_array.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + for result in [&split, &mla, &ssm, &no_layers, &no_heads, &short_array] { + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate); + assert_eq!( + result.total_gib, None, + "a figure was invented for an unsupported layout" + ); + assert!(result.detail.is_some()); + } + } + + #[test] + fn a_file_with_no_weights_after_the_header_is_not_sized() { + let (header, _) = llama(0); + let result = estimate( + &header, + header.data_offset, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + } + + #[test] + fn mixed_attention_and_an_oversized_context_are_noted_not_hidden() { + let builder = llama_like(10).u32("llama.attention.sliding_window", 4096); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 200_000, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1000.0), + ); + assert!( + result.notes.iter().any(|n| n.contains("full context")), + "{:?}", + result.notes + ); + assert!( + result.notes.iter().any(|n| n.contains("beyond the 131072")), + "{:?}", + result.notes + ); + } + + /// The notes say which way each attention kind can move the figure: a + /// sliding-window cache is over-counted, a state-space model's recurrent state + /// is not counted at all, and neither is hidden behind the other's wording. + #[test] + fn each_layout_the_formula_only_approximates_says_which_way_it_can_be_off() { + let notes_of = |builder: &Builder| { + let (header, size) = header_of(builder, 4096); + estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1000.0), + ) + .notes + .join("\n") + }; + let sliding = notes_of(&llama_like(10).u32("llama.attention.sliding_window", 4096)); + assert!(sliding.contains("can be lower"), "{sliding}"); + assert!(!sliding.contains("can be higher"), "{sliding}"); + let ssm = notes_of(&llama_like(10).u32("llama.ssm.state_size", 16)); + assert!(ssm.contains("recurrent state"), "{ssm}"); + assert!(ssm.contains("can be higher"), "{ssm}"); + let moe = notes_of(&llama_like(10).u32("llama.expert_count", 8)); + assert!( + moe.contains("mixture-of-experts") && moe.contains("can be higher"), + "{moe}" + ); + // A dense model with every width present carries none of them. + let plain = notes_of(&llama_like(10)); + assert!(plain.is_empty(), "{plain}"); + } + + /// A layout whose figure can be too low gets its figures and no verdict, on + /// a card where a dense model of the same size fits and on one where it does + /// not: part of its memory is not counted or not measured, so neither "fits" + /// nor "does not fit" can be relied on. A host that could not compare keeps + /// its own reason. + fn assert_figures_and_no_verdict(layout: &str, builder: &Builder, why: &str) { + let (header, size) = header_of(builder, 4096); + for memory in [ + AcceleratorMemory::Dedicated(1000.0), + AcceleratorMemory::Dedicated(0.5), + ] { + let result = estimate(&header, size, 4096, KvCacheType::F16, memory); + assert_eq!( + result.outcome, + EstimateOutcome::CannotEstimate, + "{layout} on {memory:?}" + ); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::UnmeasuredLayout), + "{layout}" + ); + assert!( + result.total_gib.is_some(), + "{layout}: the figures are given" + ); + assert!( + result.available_gib.is_some(), + "{layout}: the card is named" + ); + assert_eq!(result.max_context, None, "{layout}: a ceiling is a verdict"); + assert!( + result.suggestions.is_empty(), + "{layout}: {:?}", + result.suggestions + ); + let detail = result.detail.clone().unwrap_or_default(); + assert!( + detail.contains(why) && detail.contains("can be too low"), + "{layout}: {detail}" + ); + let lines = evidence_lines(&result).join("\n"); + assert!(!lines.contains("largest context"), "{layout}: {lines}"); + } + // A host that could not compare keeps its own reason: there was no + // verdict to withhold. + let unknown = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Unknown, + ); + assert_eq!( + unknown.cannot_estimate_reason, + Some(CannotEstimateReason::AcceleratorMemoryUnknown), + "{layout}" + ); + // And with no GPU visible at all, the answer is that, with the figures. + let none = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::None, + ); + assert_eq!( + none.cannot_estimate_reason, + Some(CannotEstimateReason::NoGpu), + "{layout}" + ); + assert!(none.total_gib.is_some(), "{layout}: the figures are given"); + } + + #[test] + fn a_state_space_model_gets_figures_and_no_verdict() { + assert_figures_and_no_verdict( + "state-space", + &llama_like(10).u32("llama.ssm.state_size", 16), + "recurrent state is not counted", + ); + } + + /// Attention in some layers (a KV head count per layer, zero for the + /// state-space ones) and state-space keys: a hybrid. + #[test] + fn a_hybrid_model_gets_figures_and_no_verdict() { + let hybrid = Builder::new() + .string("general.architecture", "h") + .u32("h.block_count", 4) + .u32("h.embedding_length", 512) + .u32("h.feed_forward_length", 1024) + .u32("h.attention.head_count", 8) + .u32_array("h.attention.head_count_kv", &[0, 8, 0, 8]) + .u32("h.context_length", 8192) + .u32("h.ssm.state_size", 16) + .tensor("w", &[1024]); + assert_figures_and_no_verdict("hybrid", &hybrid, "recurrent state is not counted"); + } + + #[test] + fn a_mixture_of_experts_model_gets_figures_and_no_verdict() { + assert_figures_and_no_verdict( + "mixture-of-experts", + &llama_like(10).u32("llama.expert_count", 8), + "compute buffer was not measured", + ); + } + + /// The no-verdict detail and the evidence line show the same MiB figure, + /// even for a GiB figure a hair under a whole MiB, which a plain floor + /// would show one MiB lower. + #[test] + fn the_no_verdict_detail_shows_the_same_mib_as_the_evidence_line() { + let (header, size) = header_of(&llama_like(10).u32("llama.expert_count", 8), 4096); + let gib = (8192.0 - 1e-9) / 1024.0; + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(gib), + ); + let detail = result.detail.clone().unwrap_or_default(); + assert!(detail.contains("this GPU's 8192 MiB"), "{detail}"); + let lines = evidence_lines(&result).join("\n"); + assert!(lines.contains("has 8192 MiB"), "{lines}"); + } + + /// The control: a dense model on the same card still gets its verdict. + #[test] + fn a_dense_model_still_gets_its_verdict() { + let (header, size) = header_of(&llama_like(10), 4096); + let dense = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1000.0), + ); + assert_eq!(dense.outcome, EstimateOutcome::Fits); + } + + /// The architecture name comes from the file and is printed in the report, + /// so a character that would reorder or break the line is replaced. + #[test] + fn an_architecture_name_cannot_steer_the_terminal() { + let builder = llama_like(10).string("general.architecture", "llama\u{202E}evil\n"); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 1024, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.architecture.as_deref(), Some("llama?evil?")); + assert!(evidence_lines(&result).iter().all(|line| line.is_ascii())); + } + + /// U+2028 and U+2029 are line separators, not controls: a terminal and a + /// log viewer may break the line on them, so they are replaced as well. + #[test] + fn an_architecture_name_cannot_break_the_line_with_a_unicode_separator() { + let builder = llama_like(10).string("general.architecture", "a\u{2028}b\u{2029}c"); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 1024, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.architecture.as_deref(), Some("a?b?c")); + } + + /// A header may declare zero query heads while keeping a KV head count and + /// no explicit head width; the width would be `embedding / 0`. + #[test] + fn a_zero_head_count_is_a_missing_width_not_a_division_by_zero() { + let result = cannot_estimate( + &Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 4) + .u32("x.embedding_length", 512) + .u32("x.attention.head_count", 0) + .u32("x.attention.head_count_kv", 2) + .tensor("t", &[10]), + ); + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + assert_eq!(result.total_gib, None); + } + + #[test] + fn a_header_without_the_widths_the_compute_buffer_needs_gets_figures_and_no_verdict() { + let builder = Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 2) + .u32("x.attention.head_count", 4) + .u32("x.attention.key_length", 64) + .u32("x.attention.value_length", 64) + .tensor("w", &[1024]); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 1024, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + assert!(result.total_bytes.is_some(), "the figures are still given"); + assert!(result.max_context.is_none() && result.suggestions.is_empty()); + assert!( + result.notes.iter().any(|note| note.contains("understated")), + "{:?}", + result.notes + ); + } + + #[test] + fn a_sliding_window_model_is_told_its_largest_context_is_conservative() { + let builder = llama_like(10).u32("llama.attention.sliding_window", 4096); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 131_072, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + assert!(result.conservative_ceiling); + let lines = evidence_lines(&result).join("\n"); + assert!( + lines.contains("largest context that fits") + && lines.contains("conservative for this model"), + "{lines}" + ); + assert!( + result.suggestions.iter().any(|suggestion| suggestion + .text + .contains("conservative figure for this model")), + "{:?}", + result.suggestions + ); + // A dense model's ceiling is exact, and says nothing of the kind. + let (header, size) = llama(GIB_U as usize); + let dense = estimate( + &header, + size, + 131_072, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert!(!dense.conservative_ceiling); + assert!(!evidence_lines(&dense).join("\n").contains("conservative")); + } + + #[test] + fn a_context_size_too_large_to_be_real_is_refused_not_printed_as_an_absurd_figure() { + let (header, size) = llama(GIB_U as usize / 1024); + for context in [u64::MAX, MAX_CONTEXT_SIZE * 1_000_000_000_000] { + let result = estimate( + &header, + size, + context, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate, "{context}"); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + // No figure at all, in any unit: text and JSON cannot disagree about a + // number that is not there. + assert_eq!(result.total_bytes, None); + assert_eq!(result.total_gib, None); + assert_eq!(result.kv_cache_bytes, None); + assert!(result.detail.unwrap().contains("more than an exabyte")); + } + // The cap itself is a real request. + let at_cap = estimate( + &header, + size, + MAX_CONTEXT_SIZE, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + assert!(at_cap.total_bytes.is_some()); + // Head counts that make the cache absurd are refused as well, whatever the + // context size: no figure is derived from a corrupt shape. + let builder = llama_like(10).u32("llama.attention.head_count_kv", u32::MAX); + let (header, size) = header_of(&builder, 4096); + let hostile = estimate( + &header, + size, + MAX_CONTEXT_SIZE, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + assert_eq!(hostile.outcome, EstimateOutcome::CannotEstimate); + assert_eq!(hostile.total_bytes, None); + } + + #[test] + fn signed_integer_headers_are_sized_like_unsigned_ones() { + // gguf-py writes Python integers as signed 32-bit values, so this is the + // shape of most real files: every figure the estimate reads, as type 5. + let signed = Builder::new() + .string("general.architecture", "x") + .i32("x.block_count", 4) + .i32("x.embedding_length", 512) + .i32("x.feed_forward_length", 1024) + .i32_array("x.attention.head_count", &[8, 8, 8, 8]) + .i32_array("x.attention.head_count_kv", &[0, 2, 0, 2]) + .i32("x.context_length", 8192) + .tensor("w", &[4096]); + let unsigned = Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 4) + .u32("x.embedding_length", 512) + .u32("x.feed_forward_length", 1024) + .u32_array("x.attention.head_count", &[8, 8, 8, 8]) + .u32_array("x.attention.head_count_kv", &[0, 2, 0, 2]) + .u32("x.context_length", 8192) + .tensor("w", &[4096]); + let size_up = |builder: &Builder| { + let (header, size) = header_of(builder, 8192); + estimate( + &header, + size, + 2048, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ) + }; + let (a, b) = (size_up(&signed), size_up(&unsigned)); + assert_eq!(a.outcome, EstimateOutcome::Fits); + assert_eq!(a, b); + // Two layers keep a cache of 2 heads x (64 + 64): 256 elements each. + assert_eq!(a.kv_cache_bytes_per_token, Some(2.0 * 256.0 * 2.0)); + } + + #[test] + fn a_single_token_is_not_a_plural() { + assert_eq!(tokens(1), "1 token"); + assert_eq!(tokens(0), "0 tokens"); + assert_eq!(tokens(4096), "4096 tokens"); + let (header, size) = llama(GIB_U as usize / 1024); + let one = estimate( + &header, + size, + 1, + KvCacheType::F16, + AcceleratorMemory::Dedicated(24.0), + ); + let lines = evidence_lines(&one).join("\n"); + assert!(lines.contains("at 1 token:"), "{lines}"); + assert!(!lines.contains("1 tokens"), "{lines}"); + } + + #[test] + fn the_kv_cache_type_round_trips_through_its_name_and_serde() { + for kind in KvCacheType::ALL { + assert_eq!(kind.name().parse::(), Ok(kind)); + assert_eq!( + serde_json::to_string(&kind).unwrap(), + format!("\"{}\"", kind.name()) + ); + } + assert!("q5_0".parse::().is_err()); + } + + #[test] + fn the_evidence_states_where_the_figures_came_from_and_how_big_they_are() { + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 8192, + KvCacheType::F16, + AcceleratorMemory::Dedicated(24.0), + ); + let lines = evidence_lines(&result).join("\n"); + assert!(lines.contains("GGUF header of test.gguf"), "{lines}"); + assert!(lines.contains("llama architecture, 32 layers"), "{lines}"); + assert!(lines.contains("KV cache (f16) 1024 MiB"), "{lines}"); + assert!( + lines.contains( + "has 24576 MiB (24 GiB) in total; memory other processes already hold is not \ + subtracted" + ), + "{lines}" + ); + assert!( + lines.contains("largest context that fits, with the KV cache at f16: 131072 tokens"), + "{lines}" + ); + } + + #[test] + fn a_request_that_fits_is_never_followed_by_a_ceiling_that_says_otherwise() { + // Room for a 128-token request and not for a whole 256-token step: it + // fits, and the report must not go on to say that not even 256 tokens do. + let (header, size) = llama(GIB_U as usize / 1024); + let at_128 = estimate( + &header, + size, + 128, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + let available = (at_128.total_bytes.unwrap() + 4 * MIB) as f64 / GIB; + let tight = estimate( + &header, + size, + 128, + KvCacheType::F16, + AcceleratorMemory::Dedicated(available), + ); + assert_eq!(tight.outcome, EstimateOutcome::Fits); + assert_eq!(tight.max_context, None, "less than one step of room"); + let lines = evidence_lines(&tight).join("\n"); + assert!( + !lines.contains("not even") && !lines.contains("largest context"), + "{lines}" + ); + // And the same request on a card with no room at all is the opposite. + let none = estimate( + &header, + size, + 128, + KvCacheType::F16, + AcceleratorMemory::Dedicated(1.0), + ); + assert_eq!(none.outcome, EstimateOutcome::DoesNotFit); + assert!( + evidence_lines(&none) + .join("\n") + .contains("not even a 256-token context fits"), + ); + } + + #[test] + fn a_request_beyond_the_trained_context_that_fits_is_not_told_a_lower_largest_context() { + // 262144 tokens fits in 192 GiB; the model was trained for 131072, which + // caps the ceiling. Printing "largest context that fits: 131072" right + // after showing 262144 fits would contradict itself. + let (header, size) = llama(GIB_U as usize / 1024); + let result = estimate( + &header, + size, + 262_144, + KvCacheType::F16, + AcceleratorMemory::Dedicated(192.0), + ); + assert_eq!(result.outcome, EstimateOutcome::Fits); + assert_eq!(result.max_context, Some(131_072)); + let lines = evidence_lines(&result).join("\n"); + assert!(!lines.contains("largest context that fits"), "{lines}"); + assert!( + result.notes.iter().any(|note| note + .contains("262144 tokens is beyond the 131072 tokens the model was trained for")), + "{:?}", + result.notes + ); + } + + #[test] + fn a_layer_count_beyond_the_bound_is_refused_not_looped_over() { + let (mut header, size) = llama(GIB_U as usize / 1024); + header.block_count = Some(crate::gguf::MAX_BLOCK_COUNT + 1); + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate); + assert_eq!( + result.cannot_estimate_reason, + Some(CannotEstimateReason::MissingMetadata) + ); + assert!( + result + .detail + .as_deref() + .unwrap_or("") + .contains("not a plausible model"), + "{:?}", + result.detail + ); + assert_eq!(result.total_gib, None); + } + + // ---- Calibration against measured loads ---------------------------------- + // + // The figures below were measured on an MI300X (gfx942) with llama.cpp b9752 + // on ROCm 7.2: `llama-server -fa on -np 1`, the default physical batch of + // 512, every layer on the GPU. The buffer sizes are what llama.cpp logged; + // the "used" figure is how much the GPU's used VRAM grew across the load. + // The headers are synthetic but carry the real shapes, tensor offsets and + // file sizes of the two models. + + /// Llama-3.1-8B Q4_K_M: 32 layers, 8 KV heads, head width 128, embedding 4096, + /// feed-forward 14336. 4_912_898_304 bytes of tensor data, of which the + /// 128256 x 4096 Q4_K `token_embd.weight` is 295_501_824 (llama.cpp logged + /// 281.81 MiB of it host-mapped and 4403.49 MiB on the GPU). + fn measured_llama() -> (GgufHeader, u64) { + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 32) + .u32("llama.embedding_length", 4096) + .u32("llama.feed_forward_length", 14336) + .u32("llama.attention.head_count", 32) + .u32("llama.attention.head_count_kv", 8) + .u32("llama.context_length", 131_072) + .tensor_at("token_embd.weight", &[4096, 128_256], 0) + .tensor_at("blk.0.attn_q.weight", &[4096, 4096], 295_501_824) + .tensor_at("output.weight", &[4096, 128_256], 4_617_396_480); + let (header, size) = header_of(&builder, 0); + (header, size + 4_912_898_304) + } + + /// Qwen2.5-1.5B Q8_0: 28 layers, 2 KV heads, head width 128, embedding 1536, + /// feed-forward 8960. The model's weights are tied, but the GGUF file has its + /// own `output.weight` (a second 247_959_552-byte copy, last in the data), so + /// it is not a tied-embedding file. 1_888_581_632 bytes of tensor data, of + /// which the 151936 x 1536 Q8_0 `token_embd.weight` is 247_959_552 (236.47 MiB + /// logged host-mapped, 1564.63 MiB on the GPU). + fn measured_qwen() -> (GgufHeader, u64) { + let builder = Builder::new() + .string("general.architecture", "qwen2") + .u32("qwen2.block_count", 28) + .u32("qwen2.embedding_length", 1536) + .u32("qwen2.feed_forward_length", 8960) + .u32("qwen2.attention.head_count", 12) + .u32("qwen2.attention.head_count_kv", 2) + .u32("qwen2.context_length", 32_768) + .tensor_at("token_embd.weight", &[1536, 151_936], 0) + .tensor_at("blk.0.attn_q.weight", &[1536, 1536], 247_959_552) + .tensor_at("output.weight", &[1536, 151_936], 1_640_622_080); + let (header, size) = header_of(&builder, 0); + (header, size + 1_888_581_632) + } + + struct Load { + context: u64, + kv: KvCacheType, + /// llama.cpp's logged GPU KV buffer, MiB. + kv_mib: f64, + /// llama.cpp's logged GPU compute buffer, MiB. + compute_mib: f64, + /// Growth of the GPU's used VRAM across the load, MiB. + used_mib: f64, + } + + const fn load( + context: u64, + kv: KvCacheType, + kv_mib: f64, + compute_mib: f64, + used_mib: f64, + ) -> Load { + Load { + context, + kv, + kv_mib, + compute_mib, + used_mib, + } + } + + const LLAMA_LOADS: [Load; 11] = [ + load(4096, KvCacheType::F16, 512.0, 112.01, 6128.0), + load(32768, KvCacheType::F16, 4096.0, 140.01, 9734.0), + load(65536, KvCacheType::F16, 8192.0, 172.01, 13869.0), + load(131_072, KvCacheType::F16, 16384.0, 236.01, 22125.0), + load(4096, KvCacheType::Q8_0, 272.0, 112.09, 5839.0), + load(32768, KvCacheType::Q8_0, 2176.0, 192.09, 7819.0), + load(65536, KvCacheType::Q8_0, 4352.0, 352.09, 10152.0), + load(131_072, KvCacheType::Q8_0, 8704.0, 672.09, 14830.0), + load(4096, KvCacheType::Q4_0, 144.0, 112.09, 5711.0), + load(32768, KvCacheType::Q4_0, 1152.0, 192.09, 6797.0), + load(131_072, KvCacheType::Q4_0, 4608.0, 672.09, 10734.0), + ]; + + const QWEN_LOADS: [Load; 4] = [ + load(4096, KvCacheType::F16, 112.0, 65.51, 3019.0), + load(32768, KvCacheType::F16, 896.0, 93.51, 3829.0), + load(32768, KvCacheType::Q8_0, 476.02, 93.59, 3362.0), + load(131_072, KvCacheType::F16, 3584.0, 189.51, 6615.0), + ]; + + fn mib(bytes: u64) -> f64 { + bytes as f64 / MIB as f64 + } + + /// `above` is how far above the measured use the total may come, as a factor. + fn check_against_measured(header: &GgufHeader, size: u64, loads: &[Load], above: f64) { + for measured in loads { + let result = estimate( + header, + size, + measured.context, + measured.kv, + AcceleratorMemory::Dedicated(192.0), + ); + let label = format!("{} tokens, {}", measured.context, measured.kv); + // The KV cache is exact in every case, quantized ones included. + let kv = mib(result.kv_cache_bytes.unwrap()); + assert!((kv - measured.kv_mib).abs() < 0.03, "{label}: KV {kv}"); + let compute = mib(result.compute_buffer_bytes.unwrap()); + if measured.kv == KvCacheType::F16 { + // The fixed part plus the attention mask reproduces the buffer. + assert!( + (compute - measured.compute_mib).abs() < 0.02, + "{label}: compute {compute} vs {}", + measured.compute_mib + ); + } else { + // The dequantization allowance is an upper bound, never below. + assert!( + compute >= measured.compute_mib, + "{label}: compute {compute}" + ); + assert!( + compute <= measured.compute_mib + 80.0, + "{label}: compute {compute}" + ); + } + // The total is above what the load really used, and by less than + // `above`. + let total = mib(result.total_bytes.unwrap()); + assert!(total >= measured.used_mib, "{label}: total {total}"); + assert!( + total <= measured.used_mib * above, + "{label}: total {total} vs {}", + measured.used_mib + ); + } + } + + #[test] + fn the_llama_shaped_model_reproduces_the_measured_loads() { + let (header, size) = measured_llama(); + check_against_measured(&header, size, &LLAMA_LOADS, 1.05); + } + + #[test] + fn the_qwen_shaped_model_reproduces_the_measured_loads() { + let (header, size) = measured_qwen(); + check_against_measured(&header, size, &QWEN_LOADS, 1.05); + } + + /// The sixteenth load, made with the card almost full: the process overhead + /// measured only about 0.4 GiB there against 1.05-1.28 GiB in the other + /// loads, so the estimate was further above what was used. It is still + /// above, by 5.5% (the module documentation gives the range over all + /// sixteen loads), so the 5% bound the model's other loads meet is not + /// asserted here. + #[test] + fn the_load_made_with_the_card_almost_full_is_estimated_above_by_about_five_and_a_half_percent() + { + let (header, size) = measured_llama(); + let result = estimate( + &header, + size, + 86_016, + KvCacheType::F16, + AcceleratorMemory::Dedicated(192.0), + ); + // llama.cpp logged a 10752 MiB cache and a 192.01 MiB compute buffer. + assert!((mib(result.kv_cache_bytes.unwrap()) - 10752.0).abs() < 0.03); + assert!((mib(result.compute_buffer_bytes.unwrap()) - 192.0).abs() < 0.02); + let total = mib(result.total_bytes.unwrap()); + let used = 15767.0; + assert!(total >= used, "total {total}"); + assert!(total > used * 1.05 && total < used * 1.06, "total {total}"); + } + + #[test] + fn the_components_of_two_measured_loads_are_pinned_to_the_byte() { + let (header, size) = measured_llama(); + let result = estimate( + &header, + size, + 32768, + KvCacheType::F16, + AcceleratorMemory::Dedicated(192.0), + ); + // 4403.49 MiB on the GPU, the embedding's 281.81 MiB left on the host. + assert_eq!(result.weights_bytes, Some(4_617_396_480)); + assert_eq!(result.kv_cache_bytes, Some(4096 * MIB)); + // 108 MiB fixed + 32 MiB of mask. + assert_eq!(result.compute_buffer_bytes, Some(140 * MIB)); + assert_eq!(result.runtime_reserve_bytes, Some(1280 * MIB)); + assert_eq!(result.total_bytes, Some(10_401_341_696)); + + let (header, size) = measured_qwen(); + let result = estimate( + &header, + size, + 32768, + KvCacheType::F16, + AcceleratorMemory::Dedicated(192.0), + ); + // 1564.63 MiB on the GPU, the embedding's 236.47 MiB left on the host. + assert_eq!(result.weights_bytes, Some(1_640_622_080)); + assert_eq!(result.kv_cache_bytes, Some(896 * MIB)); + assert_eq!(result.compute_buffer_bytes, Some(64_487_424 + 32 * MIB)); + assert_eq!(result.total_bytes, Some(4_020_365_312)); + } + + #[test] + fn a_quantized_cache_adds_the_dequantization_buffer_and_f16_does_not() { + let (header, size) = measured_llama(); + let compute = |kv| { + estimate( + &header, + size, + 131_072, + kv, + AcceleratorMemory::Dedicated(192.0), + ) + .compute_buffer_bytes + .unwrap() + }; + let base_and_mask = LLAMA_COMPUTE_BASE + 512 * 2 * 131_072; + assert_eq!(compute(KvCacheType::F16), base_and_mask); + // One layer's K and V as 16-bit floats per token: 8 x 256 x 2 bytes. + let extra = 131_072 * 8 * 256 * 2; + assert_eq!(compute(KvCacheType::Q8_0), base_and_mask + extra); + assert_eq!(compute(KvCacheType::Q4_0), base_and_mask + extra); + } + + #[test] + fn the_largest_context_accounts_for_what_each_token_adds_to_the_compute_buffer() { + let (header, size) = measured_llama(); + for kv in KvCacheType::ALL { + // A card with exactly what the report shows for 32768 tokens holds + // 32768 and not a step more; one MiB less holds a step fewer. + let card = shown_mib(&header, size, 32768, kv); + let result = estimate(&header, size, 1, kv, gpu_mib(card)); + assert_eq!(result.max_context, Some(32768), "{kv}"); + let result = estimate(&header, size, 1, kv, gpu_mib(card - 1)); + assert_eq!(result.max_context, Some(32512), "{kv}"); + } + } + + /// `count` model shapes with awkward sizes: parts that are not whole MiB, so + /// the shown (rounded-up) total differs from the exact one by up to a few MiB + /// -- the case a rounding mismatch needs. Deterministic. + fn odd_shapes(count: usize) -> Vec<(GgufHeader, u64)> { + let mut state = 0x9E37_79B9_7F4A_7C15_u64; + let mut next = |bound: u64| { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state % bound + }; + (0..count) + .map(|_| { + let layers = 4 + next(40) as u32; + let kv_heads = 1 + next(16) as u32; + let head = 32 + next(224) as u32; + let embd = 256 + next(7936) as u32; + let ffn = 512 + next(20000) as u32; + let builder = Builder::new() + .string("general.architecture", "odd") + .u32("odd.block_count", layers) + .u32("odd.embedding_length", embd) + .u32("odd.feed_forward_length", ffn) + .u32("odd.attention.head_count", kv_heads * 4) + .u32("odd.attention.head_count_kv", kv_heads) + .u32("odd.attention.key_length", head) + .u32("odd.attention.value_length", head) + .u32("odd.context_length", 8192 * (1 + next(16) as u32)) + .tensor_at("token_embd.weight", &[u64::from(embd), 1000], 0) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 1_000_003); + let data = 50_000_000 + next(900_000_000); + // One tensor holds the rest of the parameters, at a bits-per-weight + // a real quantization has, so the smaller-quant suggestion is + // reachable (it is withheld when shape and size disagree). + let bits = 4.0 + next(50) as f64 / 10.0; + let params = (data as f64 * 8.0 / bits) as u64; + let builder = builder.tensor_at("blk.1.ffn_up.weight", &[params], 2_000_000); + let (header, size) = header_of(&builder, 0); + (header, size + data) + }) + .collect() + } + + /// The class behind the rounding defect: the verdict, the largest context + /// and every suggestion are one decision. Swept over memory sizes at and + /// within a few MiB of every boundary, for every cache type and several + /// model shapes. + #[test] + fn the_verdict_the_largest_context_and_the_suggestions_are_one_decision() { + let mut shapes = vec![measured_llama(), measured_qwen(), llama(GIB_U as usize / 2)]; + shapes.extend(odd_shapes(24)); + let quant_bits = |label: &str| { + QUANT_BITS_PER_WEIGHT + .iter() + .find(|(name, _)| *name == label) + .map(|(_, bits)| *bits) + .unwrap() + }; + let label_re = regex::Regex::new(r"about (\S+) size").unwrap(); + let layers_re = regex::Regex::new(r"keep about (\d+) of (\d+) layers").unwrap(); + let kv_re = regex::Regex::new(r"use a (q8_0|q4_0) KV cache .*about (\d+) MiB").unwrap(); + let mut checked = 0; + for (header, size) in &shapes { + let sizing = sizing(header, *size).unwrap(); + for kv in KvCacheType::ALL { + // The request every card is judged against, and the boundaries of + // every way to make room for it: each cache type's own total at a + // spread of contexts, and the GPU's share of the layers. + let request_context = 8192; + let mut bases: Vec = Vec::new(); + for other in KvCacheType::ALL { + for context in [256_u64, 4096, 8192, 32768, 65536] { + bases.push(shown_mib(header, *size, context, other)); + } + for on_gpu in [1, sizing.layers / 2, sizing.layers - 1] { + bases.push( + shown_sum([ + sizing.weights * on_gpu as f64 / sizing.layers as f64, + sizing.kv_bytes(request_context, other) * on_gpu as f64 + / sizing.layers as f64, + sizing.compute_bytes(request_context, other), + RUNTIME_RESERVE_BYTES, + ]) as u64 + / MIB, + ); + } + } + { + // Around each boundary, and well away from it. + let mut cards: Vec = Vec::new(); + for base in bases { + cards.extend((-3..=3).map(|d| base.saturating_add_signed(d))); + cards.extend([base / 2 + 1, base + 40]); + } + for card in cards { + let memory = gpu_mib(card); + let result = estimate(header, *size, request_context, kv, memory); + let available = (card * MIB) as f64; + // The verdict is the predicate. + assert_eq!( + result.outcome == EstimateOutcome::Fits, + sizing.fits(request_context, kv, available) + ); + // The largest context fits, and no larger step does. + if let Some(max) = result.max_context { + let at = estimate(header, *size, max, kv, memory); + assert_eq!(at.outcome, EstimateOutcome::Fits, "{kv} {card} MiB: {max}"); + let cap = header.context_length.unwrap_or(u64::MAX); + if max + CONTEXT_STEP <= cap { + let above = estimate(header, *size, max + CONTEXT_STEP, kv, memory); + assert_ne!( + above.outcome, + EstimateOutcome::Fits, + "{kv} {card} MiB: {max} + a step also fits" + ); + } + } else { + let at = estimate(header, *size, CONTEXT_STEP, kv, memory); + assert_ne!(at.outcome, EstimateOutcome::Fits, "{kv} {card} MiB"); + } + if result.outcome == EstimateOutcome::Fits { + assert!(result.suggestions.is_empty()); + continue; + } + for suggestion in &result.suggestions { + checked += 1; + match suggestion.kind { + SuggestionKind::SmallerContext => { + let max = result.max_context.unwrap(); + assert!( + suggestion.text.contains(&format!("--ctx-size {max}")), + "{}", + suggestion.text + ); + } + SuggestionKind::KvCacheType => { + if let Some(found) = kv_re.captures(&suggestion.text) { + let smaller: KvCacheType = found[1].parse().unwrap(); + assert!(sizing.fits(request_context, smaller, available)); + assert_eq!( + found[2].parse::().unwrap(), + shown_mib(header, *size, request_context, smaller) + ); + } else if suggestion.text.contains("is still more than") { + let smaller = KvCacheType::ALL + .into_iter() + .filter(|o| { + o.bytes_per_element() < kv.bytes_per_element() + }) + .any(|o| !sizing.fits(request_context, o, available)); + assert!(smaller, "{}", suggestion.text); + } + } + SuggestionKind::SmallerQuant => { + if suggestion.text.contains("would fit at") { + let label = + &label_re.captures(&suggestion.text).unwrap()[1]; + let weights = sizing.quant_weights( + header.param_count as f64, + quant_bits(label), + ); + assert!( + sizing.shown_total_with(weights, request_context, kv) + <= available, + "{}", + suggestion.text + ); + } + } + SuggestionKind::CpuOffload => { + let found = layers_re.captures(&suggestion.text).unwrap(); + let on_gpu: u64 = found[1].parse().unwrap(); + assert!(sizing.layers_fit( + on_gpu, + request_context, + kv, + available + )); + assert!( + !sizing.layers_fit( + on_gpu + 1, + request_context, + kv, + available + ), + "one more layer also fits: {}", + suggestion.text + ); + } + } + } + } + } + } + } + assert!( + checked > 50, + "the sweep reached too few suggestions: {checked}" + ); + } + + /// The shown total can sit a fraction of a byte *below* the exact one (each + /// part rounds to a byte, then up to a MiB), so the analytic estimate can + /// fall one step short of the largest context the verdict accepts. The + /// largest context is then the verdict's, not the analytic one. + #[test] + fn the_largest_context_is_the_largest_one_the_verdict_accepts() { + let sizing = Sizing { + data: 0.0, + // 0.4 of a byte over a whole MiB: it rounds to the MiB below. + weights: 1000.0_f64.mul_add(MIB as f64, 0.4), + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 1, + }; + let kv = KvCacheType::F16; + // Only the attention mask grows: 512 x 2 bytes a token, 4 MiB at 4096. + let available = (2284 * MIB) as f64; + assert!( + sizing.total(4096, kv) > available, + "the exact total is above" + ); + assert!(sizing.fits(4096, kv, available), "the shown total is not"); + assert_eq!(sizing.max_context(available, kv, None), Some(4096)); + assert!(!sizing.fits(4352, kv, available)); + } + + /// The estimate assumes flash attention is on, and says so where the user reads + /// it: with it off the compute buffer grows with the context and the figure is + /// too low, which only the caveat can tell them. + #[test] + fn the_standing_caveat_says_flash_attention_is_assumed() { + assert!( + ESTIMATE_CAVEAT.contains("flash attention"), + "{ESTIMATE_CAVEAT}" + ); + assert!(ESTIMATE_CAVEAT.contains("assumed"), "{ESTIMATE_CAVEAT}"); + assert!(ESTIMATE_CAVEAT.contains("too low"), "{ESTIMATE_CAVEAT}"); + } + + /// A trained context below one step, on a card whose memory falls between the + /// exact total and the shown one, used to step below zero: plain subtraction + /// took 250 to `250 - CONTEXT_STEP`, a panic in a debug build and a wrap to a + /// huge context, then a loop, in a release one. The first step down now + /// rounds onto a whole step, and the answer is that no step fits. (The + /// `context > 0` guard at zero itself is covered by the next test.) + #[test] + fn a_trained_context_below_one_step_that_does_not_fit_gives_no_context() { + let sizing = Sizing { + data: 0.0, + weights: 1000.0_f64.mul_add(MIB as f64, 0.4), + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 1, + }; + let kv = KvCacheType::F16; + // 250 tokens of mask is under a MiB and shows as one: 2281 MiB in all. + let available = 2280.5 * MIB as f64; + assert!(!sizing.fits(250, kv, available)); + assert_eq!(sizing.max_context(available, kv, Some(250)), None); + } + + /// Where the card's memory falls between the exact total at zero context and + /// the shown one, even zero context does not fit, yet the room left after the + /// exact fixed part is positive, so the search reaches zero (stepping down + /// from 256 with no cap, or rounding 250 down with one) and is stopped there + /// only by its `context > 0` guard. Without that guard it would step to + /// `0 - CONTEXT_STEP`: a panic in a debug build, a wrap in a release one. The + /// test above does not reach this, because there zero context fits. + #[test] + fn when_even_zero_context_does_not_fit_the_search_stops_at_zero() { + let sizing = Sizing { + data: 0.0, + // Half a MiB over 1000 MiB: shown as 1001, so the exact fixed part is + // 2280.5 MiB and the shown one 2281 MiB. + weights: 1000.5 * MIB as f64, + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 1, + }; + let kv = KvCacheType::F16; + let available = 2280.75 * MIB as f64; + assert!(available > sizing.fixed(), "the early return must not fire"); + assert!(!sizing.fits(0, kv, available), "zero context must not fit"); + assert_eq!(sizing.max_context(available, kv, None), None); + assert_eq!(sizing.max_context(available, kv, Some(250)), None); + } + + /// A cap that is not a whole number of steps steps down onto whole steps: + /// 1500 does not fit, so the answer is 1024, not 1244 or 988. + #[test] + fn a_trained_context_that_is_not_a_whole_step_steps_down_to_a_whole_one() { + let sizing = Sizing { + data: 0.0, + weights: 1000.0_f64.mul_add(MIB as f64, 0.4), + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 1, + }; + let kv = KvCacheType::F16; + // The mask at 1500 tokens is 1.46 MiB and shows as 2; at 1024 it is 1. + // The exact total fits 1500 tokens here, so the search starts at the cap + // itself rather than at a whole step below it. + let available = 2281.9 * MIB as f64; + assert!(sizing.total(1500, kv) <= available); + assert!(!sizing.fits(1500, kv, available)); + assert!(sizing.fits(1024, kv, available)); + assert_eq!(sizing.max_context(available, kv, Some(1500)), Some(1024)); + } + + /// A trained context that is not a whole step and fits is the answer, even + /// where the estimate does not reach the next whole step: rounding to a step + /// before capping gave 1280 here. + #[test] + fn a_trained_context_that_fits_is_the_largest_even_when_not_a_whole_step() { + let sizing = Sizing { + data: 0.0, + weights: 1000.0 * MIB as f64, + compute_base: 0.0, + // One MiB of f16 cache per token, so a MiB of rounding is a token. + kv_elements_per_token: 512.0 * 1024.0, + widest_layer_kv_elements: 512.0 * 1024.0, + layers: 1, + }; + let kv = KvCacheType::F16; + // 1500 tokens show as 1000 + 1500 + 2 + 1280 = 3782 MiB; the exact + // estimate reaches about 1502 tokens, short of the step at 1536. + let available = 3784.0 * MIB as f64; + assert!(sizing.fits(1500, kv, available)); + assert!(!sizing.fits(1536, kv, available)); + assert_eq!(sizing.max_context(available, kv, Some(1500)), Some(1500)); + // Without the cap the answer is a whole step. + assert_eq!(sizing.max_context(available, kv, None), Some(1280)); + } + + /// The same for partial offload: a layer count that the verdict's own + /// measure accepts is offered, even where the analytic share says none. + #[test] + fn partial_offload_offers_the_layers_the_verdict_accepts() { + let sizing = Sizing { + data: 0.0, + // A quarter of it, 1000 MiB and 0.4 of a byte, rounds to 1000 MiB. + weights: 4000.0_f64.mul_add(MIB as f64, 1.6), + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 4, + }; + let request = request(4096, KvCacheType::F16); + // 1000 MiB of weights, 4 MiB of mask, 1280 MiB of reserve. + let available = (2284 * MIB) as f64; + assert!(sizing.layers_fit(1, 4096, KvCacheType::F16, available)); + assert!(!sizing.layers_fit(2, 4096, KvCacheType::F16, available)); + let suggestion = offload_suggestion(&sizing, &request, available, Some(64.0)) + .expect("one layer fits by the verdict's measure"); + assert!( + suggestion.text.contains("keep about 1 of 4 layers"), + "{}", + suggestion.text + ); + } + + #[test] + fn the_input_embedding_is_only_left_out_of_the_weights_when_the_header_has_one() { + // Last in the data: it runs to the end of the tensor data. + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 2) + .u32("llama.embedding_length", 64) + .u32("llama.attention.head_count", 4) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 0) + .tensor_at("output.weight", &[64, 100], 500) + .tensor_at("token_embd.weight", &[64, 100], 1000); + let (header, size) = header_of(&builder, 0); + let size = size + 5000; + let result = estimate( + &header, + size, + 512, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.weights_bytes, Some(1000)); + // The same file without its own output layer has tied embeddings: the + // output layer uses the embedding on the GPU, so it stays in. + let tied = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 2) + .u32("llama.embedding_length", 64) + .u32("llama.attention.head_count", 4) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 0) + .tensor_at("token_embd.weight", &[64, 100], 1000); + let (header, size) = header_of(&tied, 0); + let result = estimate( + &header, + size + 5000, + 512, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.weights_bytes, Some(5000)); + // No input embedding at all: every byte of the data is weights. + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 2) + .u32("llama.embedding_length", 64) + .u32("llama.attention.head_count", 4) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 0); + let (header, size) = header_of(&builder, 0); + let result = estimate( + &header, + size + 5000, + 512, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.weights_bytes, Some(5000)); + // A header that puts the embedding past the end of the data cannot + // make the weights negative. + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 2) + .u32("llama.embedding_length", 64) + .u32("llama.attention.head_count", 4) + .tensor_at("output.weight", &[64, 100], 0) + .tensor_at("token_embd.weight", &[64, 100], 9_000); + let (header, size) = header_of(&builder, 0); + let result = estimate( + &header, + size + 5000, + 512, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.weights_bytes, Some(5000)); + } + + // The tests below pin one rule each, with figures a mutation of that rule + // changes: a case where the right and the wrong expression happen to agree + // proves nothing, so every one is asymmetric. + + /// A model that costs only its weights, the attention mask and the reserve. + fn weights_only(weights: u64, layers: u64) -> Sizing { + Sizing { + data: weights as f64, + weights: weights as f64, + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers, + } + } + + #[test] + fn partial_offload_names_the_share_left_in_system_ram_not_the_share_on_the_gpu() { + // Eight layers of 1 GiB. The card holds two: 2048 MiB of them, the 4 MiB + // mask at 4096 tokens and the 1280 MiB reserve make 3332 MiB; a third + // layer would need 4356. + let sizing = weights_only(8 * GIB_U, 8); + let request = request(4096, KvCacheType::F16); + let available = f64::from(3332 + 100) * MIB as f64; + assert!(sizing.layers_fit(2, 4096, KvCacheType::F16, available)); + assert!(!sizing.layers_fit(3, 4096, KvCacheType::F16, available)); + let text = |ram: Option| { + offload_suggestion(&sizing, &request, available, ram).map(|suggestion| suggestion.text) + }; + let offered = text(Some(64.0)).unwrap(); + assert!(offered.contains("keep about 2 of 8 layers"), "{offered}"); + // The six layers that do not fit stay in RAM: 6 GiB, not the 2 GiB on the GPU. + assert!( + offered.contains("the other 6.0 GiB of weights and cache"), + "{offered}" + ); + // So 4 GiB of RAM cannot hold them (the 2 GiB on the GPU would fit in + // it), and 7 GiB can. + assert!(text(Some(4.0)).is_none()); + assert!(text(Some(7.0)).is_some()); + } + + #[test] + fn a_smaller_quantization_keeps_the_input_embedding_out_of_vram() { + let sizing = Sizing { + data: 1000.0, + weights: 400.0, + compute_base: 0.0, + kv_elements_per_token: 0.0, + widest_layer_kv_elements: 0.0, + layers: 1, + }; + // 1000 weights at 8 bits is 1000 bytes of file, of which 60% is the + // embedding that is not VRAM, leaving 400 bytes on the GPU. + assert!((sizing.quant_weights(1000.0, 8.0) - 400.0).abs() < 1e-9); + assert!((sizing.quant_weights(1000.0, 4.0) - 200.0).abs() < 1e-9); + } + + /// Each row is a bits-per-weight figure, not a file size: llama.cpp's + /// `quantize` table gives LLaMA-v1-7B sizes in GiB, and the row is that size + /// in bits over the model's weights. A size entered as bits (Q2_K's 2.63 GiB + /// read as 2.63 bits) prices the quantization about 20% light, and would + /// name it as fitting where it does not. + #[test] + fn the_bits_per_weight_table_is_the_published_sizes_in_bits() { + const LLAMA_V1_7B_WEIGHTS: f64 = 6_738_415_616.0; + let published_gib = [ + ("Q2_K", 2.63), + ("Q3_K_M", 3.07), + ("Q4_K_M", 3.80), + ("Q5_K_M", 4.45), + ("Q6_K", 5.15), + ("Q8_0", 6.70), + ]; + for (label, gib) in published_gib { + let bits = gib * GIB * 8.0 / LLAMA_V1_7B_WEIGHTS; + let row = QUANT_BITS_PER_WEIGHT + .iter() + .find(|(name, _)| *name == label) + .unwrap_or_else(|| panic!("{label} is not in the table")) + .1; + assert!((row - bits).abs() < 0.05, "{label}: {row} vs {bits:.3}"); + } + // The table runs from smallest to largest, which the suggestion relies on. + assert!( + QUANT_BITS_PER_WEIGHT + .windows(2) + .all(|pair| pair[0].1 < pair[1].1) + ); + } + + /// A table entry within [`QUANT_SMALLER_FACTOR`] of the current figure is the + /// same quantization for this purpose: at 4.4 bits per weight, IQ4_XS (4.25) + /// is not "smaller", so the suggestion skips it for Q3_K_M even though + /// IQ4_XS would fit. At a factor of 1.0 it would be named. + #[test] + fn a_quantization_within_the_smaller_factor_of_the_current_one_is_not_offered() { + let params: u64 = 7_000_000_000; + let builder = llama_like(10).tensor("big", &[params]); + let file = builder.file(0); + let header = parse_gguf_header(&mut Cursor::new(&file)).unwrap(); + let size = header.data_offset + (params as f64 * 4.4 / 8.0) as u64; + // 7e9 weights at 4.25 bits are about 3547 MiB, at 4.4 bits about 3672; + // with the cache (256 MiB), compute buffer (about 110) and the 1280 MiB + // reserve, IQ4_XS needs about 5193 MiB and the file as it is about 5318. + let result = estimate(&header, size, 2048, KvCacheType::F16, gpu_mib(5250)); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + let text = &result + .suggestions + .iter() + .find(|s| s.kind == SuggestionKind::SmallerQuant) + .unwrap() + .text; + assert!(text.contains("Q3_K_M size"), "{text}"); + assert!(!text.contains("IQ4_XS"), "{text}"); + } + + #[test] + fn a_large_embedding_changes_which_smaller_quantization_is_named() { + // 7e9 parameters at 8.5 bits, 40% of the data being the embedding. + let params: u64 = 7_000_000_000; + let data = (params as f64 * 8.5 / 8.0) as u64; + let embedding = data / 10 * 4; + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 32) + .u32("llama.embedding_length", 4096) + .u32("llama.feed_forward_length", 14336) + .u32("llama.attention.head_count", 32) + .u32("llama.attention.head_count_kv", 8) + .u32("llama.context_length", 131_072) + .tensor_at("token_embd.weight", &[4096, 10], 0) + .tensor_at("big", &[params], embedding) + .tensor_at("output.weight", &[4096, 10], data); + let (header, size) = header_of(&builder, 0); + let size = size + data; + // What Q5_K_M (5.69 bits per weight) needs in VRAM, worked out here: the + // 60% of it that is not the embedding, then the cache, compute buffer and + // reserve at 2048 tokens. + let weights_q5 = params as f64 * 5.69 / 8.0 * 0.6; + let other = (128.0_f64 * 1024.0) + .mul_add(2048.0, (LLAMA_COMPUTE_BASE + 512 * 2 * 2048) as f64) + + (1280 * MIB) as f64; + let card_mib = ((weights_q5 + other) / MIB as f64).ceil() as u64 + 8; + let result = estimate(&header, size, 2048, KvCacheType::F16, gpu_mib(card_mib)); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + let text = &result + .suggestions + .iter() + .find(|s| s.kind == SuggestionKind::SmallerQuant) + .unwrap() + .text; + // Counting the embedding as VRAM would price Q5_K_M at 4.6 GiB, past this + // card, and the suggestion would be a much smaller quantization. + assert!(text.contains("Q5_K_M size"), "{text}"); + } + + #[test] + fn the_largest_context_after_a_smaller_cache_is_the_one_for_that_cache() { + let (header, size) = llama(GIB_U as usize / 1024); + let fixed = estimate( + &header, + size, + 0, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + .total_gib + .unwrap(); + let memory = AcceleratorMemory::Dedicated(fixed + 0.6); + let result = estimate(&header, size, 8192, KvCacheType::F16, memory); + let for_cache = |kv| { + estimate(&header, size, 8192, kv, memory) + .max_context + .unwrap() + }; + assert_ne!( + for_cache(KvCacheType::F16), + for_cache(KvCacheType::Q8_0), + "the two caches must give different ceilings for this to mean anything" + ); + let q8 = result + .suggestions + .iter() + .find(|s| s.text.contains("q8_0 KV cache")) + .unwrap(); + assert!( + q8.text.contains(&format!( + "the largest context then is {}", + tokens(for_cache(KvCacheType::Q8_0)) + )), + "{}", + q8.text + ); + } + + /// A smaller cache's "largest context then" is capped at the trained context, + /// as the main ceiling is: a q4_0 cache with room for ~29k tokens on a model + /// trained for 8192 offers 8192, not a context the model was never trained + /// for. + #[test] + fn a_smaller_cache_suggestion_caps_its_largest_context_at_the_trained_one() { + let (header, size) = llama_trained_for(8192); + let fixed = estimate( + &header, + size, + 0, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + .total_gib + .unwrap(); + // 16384 tokens at f16 is 2 GiB of cache, at q4_0 about 0.56: room for 1. + let memory = AcceleratorMemory::Dedicated(fixed + 1.0); + let result = estimate(&header, size, 16384, KvCacheType::F16, memory); + assert_eq!(result.outcome, EstimateOutcome::DoesNotFit); + let uncapped = sizing(&header, size) + .unwrap() + .max_context((fixed + 1.0) * GIB, KvCacheType::Q4_0, None) + .unwrap(); + assert!( + uncapped > 8192, + "the cap must bind for this to mean anything" + ); + let q4 = result + .suggestions + .iter() + .find(|s| s.text.contains("q4_0 KV cache")) + .expect("a q4_0 suggestion"); + assert!( + q4.text + .contains(&format!("the largest context then is {}", tokens(8192))), + "{}", + q4.text + ); + } + + /// A request exactly at the ceiling fits, and the ceiling is still printed: + /// it is only a ceiling *below* a request that fits which is left out. + #[test] + fn a_request_at_exactly_the_ceiling_still_prints_the_ceiling() { + let (header, size) = llama(GIB_U as usize / 1024); + let kv = KvCacheType::F16; + let card = gpu_mib(shown_mib(&header, size, 24576, kv)); + let result = estimate(&header, size, 24576, kv, card); + assert_eq!(result.outcome, EstimateOutcome::Fits); + assert_eq!(result.max_context, Some(24576)); + let lines = evidence_lines(&result).join("\n"); + assert!( + lines.contains(&format!( + "largest context that fits, with the KV cache at f16: {}", + tokens(24576) + )), + "{lines}" + ); + } + + /// Either width missing understates the compute buffer, not only both, and + /// either one withholds the verdict. + #[test] + fn a_header_missing_either_width_says_the_compute_buffer_is_understated() { + let base = || { + Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 2) + .u32("x.attention.head_count", 4) + .u32("x.attention.key_length", 64) + .u32("x.attention.value_length", 64) + }; + for (which, builder) in [ + ( + "no feed-forward width", + base().u32("x.embedding_length", 256), + ), + ( + "no embedding width", + base().u32("x.feed_forward_length", 512), + ), + ( + "a feed-forward width of zero", + base() + .u32("x.embedding_length", 256) + .u32("x.feed_forward_length", 0), + ), + ( + "an embedding width of zero", + base() + .u32("x.embedding_length", 0) + .u32("x.feed_forward_length", 512), + ), + ] { + let (header, size) = header_of(&builder.tensor("w", &[1024]), 4096); + let result = estimate( + &header, + size, + 1024, + KvCacheType::F16, + AcceleratorMemory::Dedicated(8.0), + ); + assert!( + result.notes.iter().any(|note| note.contains("understated")), + "{which}: {:?}", + result.notes + ); + assert_eq!(result.outcome, EstimateOutcome::CannotEstimate, "{which}"); + } + } + + fn split_header(count: u32) -> Builder { + Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 4) + .u32("x.embedding_length", 512) + .u32("x.feed_forward_length", 1024) + .u32("x.attention.head_count", 8) + .u32("split.count", count) + .tensor("t", &[10]) + } + + #[test] + fn a_two_shard_model_is_refused_and_a_one_file_split_is_sized() { + assert_eq!( + cannot_estimate(&split_header(2)).cannot_estimate_reason, + Some(CannotEstimateReason::SplitFile), + "half the weights is not the model's" + ); + // `split.count` of 1 is a whole model that happens to carry the key. + let whole = cannot_estimate(&split_header(1)); + assert_ne!(whole.outcome, EstimateOutcome::CannotEstimate, "{whole:?}"); + assert!(whole.total_gib.is_some()); + } + + /// The measured Llama-shaped header, with `llama.context_length` set to `trained`. + fn llama_trained_for(trained: u32) -> (GgufHeader, u64) { + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", 32) + .u32("llama.embedding_length", 4096) + .u32("llama.feed_forward_length", 14336) + .u32("llama.attention.head_count", 32) + .u32("llama.attention.head_count_kv", 8) + .u32("llama.context_length", trained) + .tensor_at("token_embd.weight", &[4096, 10], 0) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 40_960); + let (header, size) = header_of(&builder, 0); + (header, size + GIB_U) + } + + #[test] + fn a_trained_context_of_zero_is_no_trained_context() { + let (header, size) = llama_trained_for(0); + let result = estimate( + &header, + size, + 4096, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ); + // No cap: 64 GiB holds far more than the 131072 a real model would trap it at. + assert!(result.max_context.unwrap() > 131_072, "{result:?}"); + assert_eq!(result.training_context, None); + assert!( + !result.notes.iter().any(|note| note.contains("trained for")), + "a request was called beyond zero trained tokens: {:?}", + result.notes + ); + } + + #[test] + fn only_a_request_past_the_trained_context_is_told_so() { + let (header, size) = llama_trained_for(8192); + let notes = |context| { + estimate( + &header, + size, + context, + KvCacheType::F16, + AcceleratorMemory::Dedicated(64.0), + ) + .notes + }; + let beyond = |notes: &[String]| notes.iter().any(|n| n.contains("beyond the")); + assert!(!beyond(¬es(8192)), "the trained context itself is fine"); + assert!(beyond(¬es(8193))); + } + + #[test] + fn each_part_of_the_shown_total_rounds_to_a_byte_and_up_to_a_whole_mib() { + let mib = MIB as f64; + // A part 0.6 of a byte over a whole MiB is a byte over it: it rounds up + // to the next MiB. A truncating conversion would show the MiB below. + assert_eq!( + shown_sum([mib.mul_add(1000.0, 0.6), 0.0, 0.0, 0.0]) as u64, + 1001 * MIB + ); + // 0.4 of a byte over rounds down to the MiB itself. + assert_eq!( + shown_sum([mib.mul_add(1000.0, 0.4), 0.0, 0.0, 0.0]) as u64, + 1000 * MIB + ); + } + + #[test] + fn the_byte_fields_round_a_fractional_cache_size_to_the_nearest_byte() { + // One layer, one KV head, key and value 4 and 6 wide: 10 elements a + // token, 10.625 bytes at q8_0 for one token. + let builder = Builder::new() + .string("general.architecture", "x") + .u32("x.block_count", 1) + .u32("x.embedding_length", 64) + .u32("x.attention.head_count", 1) + .u32("x.attention.head_count_kv", 1) + .u32("x.attention.key_length", 4) + .u32("x.attention.value_length", 6) + .tensor("t", &[10]); + let (header, size) = header_of(&builder, 4096); + let result = estimate( + &header, + size, + 1, + KvCacheType::Q8_0, + AcceleratorMemory::Dedicated(8.0), + ); + assert_eq!(result.kv_cache_bytes, Some(11)); + let parts = result.weights_bytes.unwrap() + + 11 + + result.compute_buffer_bytes.unwrap() + + result.runtime_reserve_bytes.unwrap(); + assert_eq!(result.total_bytes, Some(parts)); + } + + /// A model with `layers` cached layers of 8 KV heads and 128-wide keys and values. + fn few_layered(layers: u32) -> (GgufHeader, u64) { + let builder = Builder::new() + .string("general.architecture", "llama") + .u32("llama.block_count", layers) + .u32("llama.embedding_length", 4096) + .u32("llama.feed_forward_length", 14336) + .u32("llama.attention.head_count", 32) + .u32("llama.attention.head_count_kv", 8) + .u32("llama.context_length", 131_072) + .tensor_at("token_embd.weight", &[4096, 10], 0) + .tensor_at("blk.0.attn_q.weight", &[64, 64], 40_960); + let (header, size) = header_of(&builder, 0); + (header, size + GIB_U) + } + + #[test] + fn a_quantized_cache_is_not_offered_where_it_would_need_more_than_the_wider_one() { + // With two cached layers a q8_0 cache (0.53 GiB at this context) plus the + // f16 copy of one layer it is dequantized through (0.5 GiB) comes to more + // than the f16 cache itself (1 GiB), mask aside: it would not cut the need. + let (header, size) = few_layered(2); + let suggestions = |memory_gib| { + estimate( + &header, + size, + 131_072, + KvCacheType::F16, + AcceleratorMemory::Dedicated(memory_gib), + ) + .suggestions + }; + let texts: Vec = suggestions(2.5).into_iter().map(|s| s.text).collect(); + assert!( + !texts.iter().any(|t| t.contains("q8_0 KV cache")), + "{texts:?}" + ); + assert!( + texts.iter().any(|t| t.contains("q4_0 KV cache")), + "{texts:?}" + ); + // With four layers the same cache does save, and is offered. + let (header, size) = few_layered(4); + let texts: Vec = estimate( + &header, + size, + 131_072, + KvCacheType::F16, + AcceleratorMemory::Dedicated(2.5), + ) + .suggestions + .into_iter() + .map(|s| s.text) + .collect(); + assert!( + texts.iter().any(|t| t.contains("q8_0 KV cache")), + "{texts:?}" + ); + } +} diff --git a/crates/rocm-core/src/gguf_source.rs b/crates/rocm-core/src/gguf_source.rs new file mode 100644 index 0000000000..74b0b12da0 --- /dev/null +++ b/crates/rocm-core/src/gguf_source.rs @@ -0,0 +1,2508 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Where a GGUF header comes from: a local file, or a Hugging Face repository +//! read by HTTP range requests. +//! +//! A remote file is found by one repository-listing request (plus a `HEAD` +//! request for a file the listing does not size), then read in windows. +//! +//! The point of the range reader is that a model's *shape* is in the first few +//! megabytes of a file that may be tens of gigabytes. [`RangedReader`] presents +//! the remote file as `Read + Seek` and fetches it in 4 MiB windows on demand, so +//! [`crate::gguf::parse_gguf_header`] stops reading where the header ends and +//! the weights are never requested. A server that ignores `Range` is refused +//! rather than streamed, because streaming it is exactly the download this +//! exists to avoid. + +use anyhow::{Context, Result, anyhow, bail}; +use serde_json::Value; +use std::fs::File; +use std::io::{self, BufReader, Read, Seek, SeekFrom}; +use std::path::{Path, PathBuf}; +use std::time::{Duration, Instant}; + +/// How much of the remote file one request asks for. +const WINDOW_BYTES: u64 = 4 * 1024 * 1024; +/// The most a header read may fetch in total. Real headers are a few megabytes +/// (mostly the tokenizer); a read that goes beyond this is not reading a header. +const MAX_FETCH_BYTES: u64 = crate::gguf::MAX_HEADER_BYTES; +/// A repository listing larger than this is not read. +const MAX_LISTING_BYTES: u64 = 16 * 1024 * 1024; +const CONNECT_TIMEOUT: Duration = Duration::from_secs(10); +const REQUEST_TIMEOUT: Duration = Duration::from_mins(1); +/// How long listing a repository and reading a header may take in all. The +/// per-request timeout alone would let sixteen slow windows hold the command for +/// sixteen minutes. +const TOTAL_BUDGET: Duration = Duration::from_mins(2); +const DEFAULT_HUB: &str = "https://huggingface.co"; + +/// A model reference that names a GGUF file. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GgufRef { + /// A path on this machine. + Local(PathBuf), + /// `owner/repo` on Hugging Face, with the quantization label or file name + /// after a colon. The variant is optional here so that a reference without + /// one can be refused with a message, not mistaken for something else. + Hub { + owner: String, + repo: String, + variant: Option, + }, +} + +/// Whether a string is a safe Hugging Face path component: the charset of +/// `is_safe_hf_component` in the lemonade engine, which is what `rocm serve` +/// applies to `owner/repo:variant`, so a reference serve would refuse is not sent +/// to the hub either. +fn is_safe_component(component: &str) -> bool { + !component.is_empty() + && component != "." + && component != ".." + && component + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-')) +} + +/// Whether a string is a valid `:variant`: a quantization label, or a file name +/// ending in `.gguf` that may carry a repository sub-directory +/// (`dir/model-Q4_K_M.gguf`), which is the only way to name one of two files that +/// share a name in different directories. `rocm serve` takes no `/` in a variant; +/// here it is safe because every segment is checked and the whole is sent to the +/// hub as percent-encoded path segments, never to the filesystem. +/// +/// A file name may also carry `+`, which hub file names do (`m+x-Q4_K_M.gguf`): +/// it reaches the hub percent-encoded, so it is as safe as the other characters, +/// and refusing it would leave such a file impossible to name exactly. +fn is_safe_variant(variant: &str) -> bool { + if is_safe_component(variant) { + return true; + } + let file_component = |component: &str| is_safe_component(&component.replace('+', "_")); + variant.to_ascii_lowercase().ends_with(".gguf") + && variant.split('/').count() <= 8 + && variant.split('/').all(file_component) +} + +/// Read a model reference as a GGUF file, if it is shaped like one. +/// +/// A file that exists on this machine is that file, whatever its name looks +/// like. Otherwise `owner/repo:variant` is a Hugging Face repository, a name +/// ending in `.gguf` is a local path (that does not exist, so it is reported as +/// not openable rather than sent to the hub), and a bare `owner/repo` is a +/// repository with no variant, recognised so the caller can say a quantization is +/// needed. Returns `None` for names, aliases and anything else, so a catalog model +/// never reaches this module. +#[must_use] +pub fn parse_gguf_ref(model_ref: &str) -> Option { + parse_gguf_ref_with(model_ref, &|path| path.is_file()) +} + +/// [`parse_gguf_ref`] with the "does this file exist" question answered by the +/// caller, so a test can say a path exists without a working directory to put it +/// in. +fn parse_gguf_ref_with(model_ref: &str, is_file: &dyn Fn(&Path) -> bool) -> Option { + let trimmed = model_ref.trim(); + if trimmed.is_empty() { + return None; + } + if is_file(Path::new(trimmed)) { + return Some(GgufRef::Local(PathBuf::from(trimmed))); + } + let is_gguf_name = |name: &str| name.to_ascii_lowercase().ends_with(".gguf"); + if let Some((repo_path, variant)) = trimmed.split_once(':') + && let Some((owner, repo)) = repo_path.split_once('/') + && is_safe_component(owner) + && is_safe_component(repo) + && is_safe_variant(variant) + { + return Some(GgufRef::Hub { + owner: owner.to_owned(), + repo: repo.to_owned(), + variant: Some(variant.to_owned()), + }); + } + if is_gguf_name(trimmed) { + return Some(GgufRef::Local(PathBuf::from(trimmed))); + } + let (owner, repo) = trimmed.split_once('/')?; + (is_safe_component(owner) && is_safe_component(repo)).then(|| GgufRef::Hub { + owner: owner.to_owned(), + repo: repo.to_owned(), + variant: None, + }) +} + +/// A file of a Hugging Face repository. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HubFile { + pub name: String, + /// The size the listing reported; `None` when it did not. + pub size: Option, +} + +fn is_boundary(c: Option, extra: &[char]) -> bool { + c.is_none_or(|c| matches!(c, '-' | '.' | '/') || extra.contains(&c)) +} + +/// Whether `label` appears in `name` as a whole quantization label: bounded by +/// separators, so `Q4_0` does not match `Q4_0_4_4`. +fn has_label(name: &str, label: &str) -> bool { + let name = name.to_ascii_lowercase(); + let label = label.to_ascii_lowercase(); + name.match_indices(&label).any(|(start, _)| { + let before = name[..start].chars().next_back(); + let after = name[start + label.len()..].chars().next(); + is_boundary(before, &['_']) && is_boundary(after, &[]) + }) +} + +/// A name that matches only multimodal projectors, which are not models. +/// +/// Raised by [`select_gguf_file`] as a typed error so a caller can tell it from +/// a file that could not be found or read, and offer one of `models` instead. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ProjectorNamed { + /// What the user named, as they wrote it. + pub variant: String, + /// The projector file or files that name matched, by full name in the + /// repository: what the user actually pointed at. + pub projectors: Vec, + /// The repository's model files (every GGUF that is not a projector), by + /// full name. + pub models: Vec, +} + +impl std::fmt::Display for ProjectorNamed { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + // Capped as `names_for_error` caps a listing; `models` itself stays + // complete. + const SHOWN: usize = 8; + let models = if self.models.is_empty() { + "none: every GGUF file in it is a projector".to_owned() + } else { + let mut shown: Vec<&str> = self.models.iter().take(SHOWN).map(String::as_str).collect(); + if self.models.len() > SHOWN { + shown.push("..."); + } + shown.join(", ") + }; + let mut named: Vec = self + .projectors + .iter() + .take(SHOWN) + .map(|name| format!("`{name}`")) + .collect(); + if self.projectors.len() > SHOWN { + named.push("...".to_owned()); + } + let named = named.join(", "); + let is = if self.projectors.len() == 1 { + "is a multimodal projector" + } else { + "are multimodal projectors" + }; + write!( + f, + "{named} {is} (mmproj), not a language model, so `{}` cannot be estimated; name \ + the model's own GGUF file or its quantization (the repository's models: {models})", + self.variant + ) + } +} + +impl std::error::Error for ProjectorNamed {} + +/// Whether a GGUF file name is a multimodal projector (`mmproj`, in any case), +/// which ships beside a vision model and is not a language model itself. +#[must_use] +pub fn names_projector(name: &str) -> bool { + name.to_ascii_lowercase().contains("mmproj") +} + +/// Pick the GGUF file a `variant` names out of a repository's files. +/// +/// An exact file name, or a quantization label as `rocm serve` takes it. Never +/// picks between two quantizations: a label that matches several files is an +/// error listing them, except that the shards of one split model resolve to +/// their first shard. A multimodal projector (`mmproj` in its name, in any +/// case) is not a model: it is never picked, and a file name that matches only +/// projectors is refused as naming one ([`ProjectorNamed`]) rather than +/// reported as missing. A quantization label that matches only a projector is +/// reported as matching nothing. +/// +/// # Errors +/// +/// When nothing matches, the match is ambiguous, or the file named is a +/// projector. +pub fn select_gguf_file<'a>(files: &'a [HubFile], variant: &str) -> Result<&'a HubFile> { + let is_projector = |file: &HubFile| names_projector(&file.name); + let all_ggufs: Vec<&HubFile> = files + .iter() + .filter(|file| file.name.to_ascii_lowercase().ends_with(".gguf")) + .collect(); + let ggufs: Vec<&HubFile> = all_ggufs + .iter() + .copied() + .filter(|file| !is_projector(file)) + .collect(); + let matches: Vec<&HubFile> = if variant.to_ascii_lowercase().ends_with(".gguf") { + // A name is unique in a repository, so one that matches in full is the + // file asked for, even where another file in a directory has the same + // last component (`m.gguf` at the root, `dir/m.gguf` below it). Searched + // over every GGUF, projectors included, so that naming one is answered + // for what it is. + let exact: Vec<&HubFile> = all_ggufs + .iter() + .copied() + .filter(|file| file.name == variant) + .collect(); + let named = if exact.is_empty() { + all_ggufs + .iter() + .copied() + .filter(|file| file.name.rsplit('/').next() == Some(variant)) + .collect() + } else { + exact + }; + // A projector named this way is not the model asked for. Where the name + // also matches a model (a bare name both share), the model is the + // answer; where it matches only projectors, the user is told what they + // named rather than that it is missing. + let models: Vec<&HubFile> = named + .iter() + .copied() + .filter(|file| !is_projector(file)) + .collect(); + if models.is_empty() && !named.is_empty() { + return Err(anyhow::Error::new(ProjectorNamed { + variant: variant.to_owned(), + projectors: named.iter().map(|file| file.name.clone()).collect(), + models: ggufs.iter().map(|file| file.name.clone()).collect(), + })); + } + models + } else { + ggufs + .iter() + .copied() + .filter(|file| has_label(&file.name, variant)) + .collect() + }; + match matches.as_slice() { + [only] => Ok(only), + // The repository's GGUF files are listed with the projectors among them, + // marked as such, rather than left out (up to eight, then `...`). + [] => bail!( + "no GGUF file matching `{variant}` in the repository (it has: {})", + names_for_error_marking_projectors(&all_ggufs) + ), + several => first_shard_of_one_set(several).ok_or_else(|| { + anyhow!( + "`{variant}` matches several files ({}); name one exactly, as \ + `owner/repo:file.gguf` (or `owner/repo:dir/file.gguf` when the name is not \ + unique)", + names_for_error(several) + ) + }), + } +} + +/// Split a shard's name `-00002-of-00004.gguf` into what the shards of one +/// model share (the stem and the total) and the shard's own index. +fn shard_parts(name: &str) -> Option<(String, u32)> { + let of = name.rfind("-of-")?; + let digits = |text: &str| -> Option { + (text.len() == 5 && text.bytes().all(|b| b.is_ascii_digit())) + .then(|| text.parse().ok()) + .flatten() + }; + let index_start = of.checked_sub(5)?; + let index = digits(name.get(index_start..of)?)?; + let stem = name.get(..index_start)?.strip_suffix('-')?; + let rest = name.get(of + 4..)?; + let total_end = rest.find('.').unwrap_or(rest.len()); + let total = digits(rest.get(..total_end)?)?; + Some((format!("{stem}|{total}|{}", &rest[total_end..]), index)) +} + +/// The first shard, when every candidate is a shard of the same split model. +/// Shards that belong to different models (two sets under one label) are +/// ambiguous, not one model: taking "the" first shard would size the wrong one. +fn first_shard_of_one_set<'a>(candidates: &[&'a HubFile]) -> Option<&'a HubFile> { + let parts: Vec<(String, u32)> = candidates + .iter() + .map(|file| shard_parts(&file.name)) + .collect::>()?; + let set = &parts.first()?.0; + if parts.iter().any(|(other, _)| other != set) { + return None; + } + candidates + .iter() + .zip(&parts) + .find(|(_, (_, index))| *index == 1) + .map(|(file, _)| *file) +} + +/// [`names_for_error`], with each multimodal projector marked as one. +fn names_for_error_marking_projectors(files: &[&HubFile]) -> String { + const SHOWN: usize = 8; + let mut names: Vec = files + .iter() + .take(SHOWN) + .map(|file| { + if names_projector(&file.name) { + format!("{} (multimodal projector)", file.name) + } else { + file.name.clone() + } + }) + .collect(); + if files.len() > SHOWN { + names.push("...".to_owned()); + } + if names.is_empty() { + "no GGUF files".to_owned() + } else { + names.join(", ") + } +} + +fn names_for_error(files: &[&HubFile]) -> String { + const SHOWN: usize = 8; + let mut names: Vec<&str> = files.iter().take(SHOWN).map(|f| f.name.as_str()).collect(); + if files.len() > SHOWN { + names.push("..."); + } + if names.is_empty() { + "no GGUF files".to_owned() + } else { + names.join(", ") + } +} + +/// `Read + Seek`, as one object-safe trait. +pub trait ReadSeek: Read + Seek {} +impl ReadSeek for T {} + +/// A GGUF that has been opened for header reading. +pub struct OpenedGguf { + pub reader: Box, + /// The size of the whole file, in bytes. + pub size: u64, + /// What to call the file in a report. + pub description: String, +} + +/// A buffered reader whose relative seeks stay inside its buffer where they can. +/// +/// The header parser skips every string it does not keep with a relative seek, +/// and a vocabulary is 128k to 262k of them. `BufReader::seek` drops the buffer +/// every time, so each skip became a syscall and the next read a refill: a +/// 300k-token header took seconds to read from disk. `seek_relative` keeps the +/// buffer when the target is inside it. The position is tracked here, because +/// a relative seek has to know where it lands to report it, and `BufReader`'s +/// own `stream_position` would ask the file, which is a seek too. +struct RelativeSeekBuf { + inner: BufReader, + /// Where the next read starts, from the start of the stream. + position: u64, +} + +impl RelativeSeekBuf { + /// Over a stream positioned at its start. + const fn new(inner: BufReader) -> Self { + Self { inner, position: 0 } + } +} + +impl Read for RelativeSeekBuf { + fn read(&mut self, buf: &mut [u8]) -> io::Result { + let read = self.inner.read(buf)?; + self.position += read as u64; + Ok(read) + } +} + +impl Seek for RelativeSeekBuf { + fn seek(&mut self, pos: SeekFrom) -> io::Result { + match pos { + SeekFrom::Current(offset) => { + let target = self.position.checked_add_signed(offset).ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidInput, "seek before the start") + })?; + self.inner.seek_relative(offset)?; + self.position = target; + } + other => self.position = self.inner.seek(other)?, + } + Ok(self.position) + } + + fn stream_position(&mut self) -> io::Result { + Ok(self.position) + } +} + +/// Opens a GGUF named by a [`GgufRef`]. A trait so the estimate can be tested +/// without a network, and so the one real implementation is the only place that +/// reaches for one. +pub trait GgufSource { + /// # Errors + /// + /// When the file cannot be found or reached, or (for a hub file named by + /// its file name) is a multimodal projector: then the error is a + /// [`ProjectorNamed`], which a caller can downcast to. + fn open(&self, reference: &GgufRef) -> Result; +} + +/// The real source: the local filesystem and Hugging Face. +#[derive(Debug, Clone)] +pub struct LiveGgufSource { + hub: String, + /// How long listing a repository and reading a header may take in all. + budget: Duration, +} + +impl Default for LiveGgufSource { + fn default() -> Self { + Self::new() + } +} + +impl LiveGgufSource { + /// Uses `HF_ENDPOINT` as the Hugging Face base URL when it is set (the + /// variable the Hugging Face tooling reads), else `https://huggingface.co`. + #[must_use] + pub fn new() -> Self { + let hub = std::env::var("HF_ENDPOINT") + .ok() + .map(|value| value.trim().trim_end_matches('/').to_owned()) + .filter(|value| !value.is_empty()) + .unwrap_or_else(|| DEFAULT_HUB.to_owned()); + Self { + hub, + budget: TOTAL_BUDGET, + } + } + + /// A source against an explicit hub base URL. + #[cfg(test)] + #[must_use] + pub fn with_hub(hub: &str) -> Self { + Self { + hub: hub.trim_end_matches('/').to_owned(), + budget: TOTAL_BUDGET, + } + } + + /// The same source with a shorter total budget, so a test can show the + /// reader is bound by the deadline `open` sets rather than by a fresh one. + #[cfg(test)] + #[must_use] + const fn with_budget(mut self, budget: Duration) -> Self { + self.budget = budget; + self + } + + fn agent() -> ureq::Agent { + // The same User-Agent the CLI's other downloads send. The agent follows + // no redirect itself: [`send`] does, so that it can refuse one that + // leaves https for plain http. + ureq::AgentBuilder::new() + .user_agent("rocm-cli") + .redirects(0) + .timeout_connect(CONNECT_TIMEOUT) + .timeout(REQUEST_TIMEOUT) + .build() + } + + fn list_files( + &self, + agent: &ureq::Agent, + owner: &str, + repo: &str, + deadline: Instant, + ) -> Result> { + let url = format!("{}/api/models/{owner}/{repo}?blobs=true", self.hub); + let response = send( + agent, + "GET", + &url, + deadline, + None, + &format!("`{owner}/{repo}`"), + )?; + let mut text = Vec::new(); + // One byte past the cap, so a listing that is exactly too long is told + // apart from one that fits instead of being cut short and misread as bad + // JSON. Read as bytes: the cut can fall inside a multi-byte character, + // and reading it as text would report that, not the size. + response + .into_reader() + .take(MAX_LISTING_BYTES + 1) + .read_to_end(&mut text) + .with_context(|| format!("could not read the file list of `{owner}/{repo}`"))?; + if text.len() as u64 > MAX_LISTING_BYTES { + bail!( + "the file list of `{owner}/{repo}` is larger than {} MiB, which is not read", + MAX_LISTING_BYTES / (1024 * 1024) + ); + } + let body: Value = serde_json::from_slice(&text) + .with_context(|| format!("the file list of `{owner}/{repo}` was not JSON"))?; + let siblings = body + .get("siblings") + .and_then(Value::as_array) + .ok_or_else(|| anyhow!("the file list of `{owner}/{repo}` has no `siblings`"))?; + // A name the hub lists is put into a request path, so a name that could + // climb out of the repository (`..`), or hide in a terminal (a control, + // bidirectional or zero-width character), is dropped here rather than + // offered, requested or printed. + // A real repository has none; a hostile mirror is what could. + Ok(siblings + .iter() + .filter_map(|sibling| { + let name = sibling.get("rfilename")?.as_str()?; + is_safe_listed_name(name).then(|| HubFile { + name: name.to_owned(), + size: sibling.get("size").and_then(Value::as_u64), + }) + }) + .collect()) + } +} + +/// Whether a file name from a repository listing may be requested and shown: no +/// empty, `.` or `..` path segment, and no character that changes how a terminal +/// shows the text around it: the predicates in `terminal` decide which, so this +/// and the other untrusted text the CLI prints are held to one set. +fn is_safe_listed_name(name: &str) -> bool { + !name.is_empty() + && !name.chars().any(|c| { + crate::terminal::is_control_or_format(c) + || crate::terminal::is_control_or_line_separator(c) + }) + && name + .split('/') + .all(|segment| !segment.is_empty() && segment != "." && segment != "..") +} + +fn describe_http_error(error: &ureq::Error, what: &str) -> anyhow::Error { + match error { + // The hub answers 401 for a repository that does not exist as well as + // for one that is gated or private, so the two cannot be told apart. No + // token is ever sent, so authentication is not something to offer. + ureq::Error::Status(401 | 403 | 404, _) => anyhow!( + "{what} was not found, or its repository is gated or private; no token is sent, so \ + only public repositories can be read" + ), + ureq::Error::Status(code, _) => anyhow!("the hub answered {code} for {what}"), + ureq::Error::Transport(transport) => { + anyhow!("could not reach the hub for {what}: {transport}") + } + } +} + +/// What to say when a path that does not exist is shaped like a file in a Hugging +/// Face repository (`owner/repo/dir/file.gguf`): the hub form of the same name. +/// Empty for anything else. +fn hub_hint(path: &Path) -> String { + let Some(text) = path.to_str() else { + return String::new(); + }; + let parts: Vec<&str> = text.split('/').collect(); + if path.is_absolute() + || parts.len() < 3 + || !text.to_ascii_lowercase().ends_with(".gguf") + || !parts.iter().all(|part| is_safe_component(part)) + { + return String::new(); + } + format!( + " (if you meant a file in a Hugging Face repository, write it `{}/{}:{}`, with a colon \ + after the repository)", + parts[0], + parts[1], + parts[2..].join("/") + ) +} + +impl GgufSource for LiveGgufSource { + fn open(&self, reference: &GgufRef) -> Result { + match reference { + GgufRef::Local(path) => { + // Checked before opening: opening a named pipe blocks until + // something writes to it, and a directory opens and then fails + // on its first read. + let metadata = std::fs::metadata(path).with_context(|| { + format!("could not open {}{}", path.display(), hub_hint(path)) + })?; + if !metadata.is_file() { + bail!("{} is not a regular file", path.display()); + } + let file = File::open(path) + .with_context(|| format!("could not open {}", path.display()))?; + let size = metadata.len(); + Ok(OpenedGguf { + reader: Box::new(RelativeSeekBuf::new(BufReader::new(file))), + size, + description: path.display().to_string(), + }) + } + GgufRef::Hub { + owner, + repo, + variant, + } => { + let Some(variant) = variant else { + bail!( + "`{owner}/{repo}` needs a quantization to read: use \ + `{owner}/{repo}:QUANT` (for example `{owner}/{repo}:Q4_K_M`) or a file \ + name ending in .gguf" + ); + }; + let agent = Self::agent(); + let deadline = Instant::now() + self.budget; + let files = self.list_files(&agent, owner, repo, deadline)?; + let file = select_gguf_file(&files, variant)?; + let path: Vec = file + .name + .split('/') + .map(crate::report_delivery::percent_encode) + .collect(); + let url = format!( + "{}/{owner}/{repo}/resolve/main/{}", + self.hub, + path.join("/") + ); + // The listing carries a size for a file the hub stores whole; if + // it left one out, the file's own headers say. + let size = match file.size { + Some(size) => size, + None => head_size(&agent, &url, deadline)?, + }; + Ok(OpenedGguf { + reader: Box::new(RangedReader::new(agent, url, size).with_deadline(deadline)), + size, + description: format!("{owner}/{repo}: {}", file.name), + }) + } + } + } +} + +/// How long is left before `deadline`, as the timeout for the next request. +fn time_left(deadline: Instant) -> io::Result { + let left = deadline.saturating_duration_since(Instant::now()); + if left.is_zero() { + return Err(io::Error::other(format!( + "reading the header took longer than {} seconds", + TOTAL_BUDGET.as_secs() + ))); + } + Ok(left.min(REQUEST_TIMEOUT)) +} + +/// The most redirects one request follows: the hub sends `resolve/main` to its +/// CDN, and a chain longer than this is a loop or a mirror that is not one. +const MAX_REDIRECTS: usize = 5; + +/// Send one request and follow its redirects by hand, each hop under what is +/// left of `deadline`, keeping the `Range` header on every hop. +/// +/// By hand because the agent cannot say where a redirect is going until it has +/// gone there, and [`redirect_target`] must refuse one that leaves https for +/// plain http: `HF_ENDPOINT` is an environment variable, and a mirror that is +/// not trusted must not be able to send the request out in the clear. +fn send( + agent: &ureq::Agent, + method: &str, + url: &str, + deadline: Instant, + range: Option<&str>, + what: &str, +) -> Result { + let mut current = url.to_owned(); + for _ in 0..=MAX_REDIRECTS { + let mut request = agent + .request(method, ¤t) + .timeout(time_left(deadline).map_err(|error| anyhow!("{error}"))?); + if let Some(range) = range { + request = request.set("Range", range); + } + let response = request + .call() + .map_err(|error| describe_http_error(&error, what))?; + if !matches!(response.status(), 301 | 302 | 303 | 307 | 308) { + return Ok(response); + } + let location = response + .header("Location") + .ok_or_else(|| anyhow!("the hub redirected {what} without saying where"))?; + current = redirect_target(¤t, location, what)?; + } + bail!("the hub redirected {what} more than {MAX_REDIRECTS} times") +} + +/// Where a redirect from `current` to `location` goes, or why it is refused: it +/// must stay on http(s), and a request that began on https never ends on http. +fn redirect_target(current: &str, location: &str, what: &str) -> Result { + let location = location.trim(); + let (scheme, rest) = current + .split_once("://") + .ok_or_else(|| anyhow!("{what} is at `{current}`, which is not a URL"))?; + let authority = rest.split(['/', '?', '#']).next().unwrap_or(rest); + let lower = location.to_ascii_lowercase(); + let target = if lower.starts_with("http://") || lower.starts_with("https://") { + location.to_owned() + } else if location.starts_with("//") { + format!("{scheme}:{location}") + } else if location.starts_with('/') { + format!("{scheme}://{authority}{location}") + } else if location.contains("://") { + bail!("the hub redirected {what} to something that is not http(s)"); + } else { + let path = rest[authority.len()..] + .split(['?', '#']) + .next() + .unwrap_or(""); + let directory = path.rsplit_once('/').map_or("", |(directory, _)| directory); + format!("{scheme}://{authority}{directory}/{location}") + }; + if scheme.eq_ignore_ascii_case("https") && target.to_ascii_lowercase().starts_with("http://") { + bail!( + "the hub redirected {what} from https to plain http, which is refused: no token is \ + sent, but the request must not leave an encrypted connection" + ); + } + Ok(target) +} + +/// The size of a remote file from its `Content-Length`, without fetching it. +fn head_size(agent: &ureq::Agent, url: &str, deadline: Instant) -> Result { + let response = send(agent, "HEAD", url, deadline, None, "the model file")?; + response + .header("Content-Length") + .and_then(|value| value.trim().parse().ok()) + .ok_or_else(|| anyhow!("the hub did not report how large the model file is")) +} + +/// A remote file as `Read + Seek`, fetched in windows by `Range` request. +pub struct RangedReader { + agent: ureq::Agent, + url: String, + len: u64, + position: u64, + window: Vec, + window_start: u64, + fetched: u64, + deadline: Instant, +} + +impl RangedReader { + #[must_use] + pub fn new(agent: ureq::Agent, url: String, len: u64) -> Self { + Self { + agent, + url, + len, + position: 0, + window: Vec::new(), + window_start: 0, + fetched: 0, + deadline: Instant::now() + TOTAL_BUDGET, + } + } + + /// Stop reading at `deadline`, however many windows are left. + #[must_use] + pub const fn with_deadline(mut self, deadline: Instant) -> Self { + self.deadline = deadline; + self + } + + /// Bytes requested from the server so far. + #[cfg(test)] + #[must_use] + pub const fn fetched(&self) -> u64 { + self.fetched + } + + fn fill(&mut self) -> io::Result<()> { + let end = self.position.saturating_add(WINDOW_BYTES).min(self.len); + let wanted = end - self.position; + if self.fetched.saturating_add(wanted) > MAX_FETCH_BYTES { + return Err(io::Error::other(format!( + "the header is larger than {} MiB; this is not a GGUF header read", + MAX_FETCH_BYTES / (1024 * 1024) + ))); + } + let response = send( + &self.agent, + "GET", + &self.url, + self.deadline, + Some(&format!("bytes={}-{}", self.position, end - 1)), + "the model file", + ) + .map_err(|error| io::Error::other(format!("{error:#}")))?; + // 200 means the server ignored the range and would send the whole file. + if response.status() != 206 { + return Err(io::Error::other(format!( + "the server answered {} to a range request; it does not support reading part of \ + a file, and the whole file is not downloaded to read its header", + response.status() + ))); + } + // A 206 says which bytes it carries. One that starts somewhere else would + // be parsed as if it began at `position` -- a header read from the wrong + // offset -- so it is refused, as the download path refuses it. + let start = crate::download::content_range_start(&response); + if start != Some(self.position) { + return Err(io::Error::other(format!( + "the server answered a range that starts at {} when {} was asked for", + start.map_or_else(|| "an unstated offset".to_owned(), |at| at.to_string()), + self.position + ))); + } + // `wanted` is at most WINDOW_BYTES already; the clamp states that bound + // where the allocation is made. A comparison rather than `min`, so a + // static analyser sees the bound as well as a reader does. + let capacity = if wanted <= WINDOW_BYTES { + wanted + } else { + WINDOW_BYTES + }; + let mut window = Vec::with_capacity(usize::try_from(capacity).unwrap_or(0)); + // A body that ends early surfaces as `UnexpectedEof`, which would read as + // "the header ends here". It is the connection that ended. + if let Err(error) = response.into_reader().take(wanted).read_to_end(&mut window) { + return Err(io::Error::other(format!( + "the connection ended after {} of the {wanted} bytes asked for: {error}", + window.len() + ))); + } + if window.len() as u64 != wanted { + // Not `UnexpectedEof`: that reads as "the header ends here", which + // would blame the file for a connection that dropped. + return Err(io::Error::other(format!( + "the connection ended after {} of the {wanted} bytes asked for", + window.len() + ))); + } + self.fetched += wanted; + self.window = window; + self.window_start = self.position; + Ok(()) + } +} + +impl Read for RangedReader { + fn read(&mut self, buf: &mut [u8]) -> io::Result { + if self.position >= self.len || buf.is_empty() { + return Ok(0); + } + let window_end = self.window_start + self.window.len() as u64; + if self.position < self.window_start || self.position >= window_end { + self.fill()?; + } + let offset = usize::try_from(self.position - self.window_start) + .map_err(|_| io::Error::other("offset out of range"))?; + let count = buf.len().min(self.window.len() - offset); + buf[..count].copy_from_slice(&self.window[offset..offset + count]); + self.position += count as u64; + Ok(count) + } +} + +impl Seek for RangedReader { + fn seek(&mut self, target: SeekFrom) -> io::Result { + let next = match target { + SeekFrom::Start(offset) => Some(offset), + SeekFrom::Current(delta) => self.position.checked_add_signed(delta), + SeekFrom::End(delta) => self.len.checked_add_signed(delta), + }; + let next = next.ok_or_else(|| { + io::Error::new( + io::ErrorKind::InvalidInput, + "seek before the start of the file", + ) + })?; + self.position = next; + Ok(next) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::gguf::test_builder::llama_like; + use crate::gguf::{GgufError, parse_gguf_header}; + use std::io::{Cursor, Write as _}; + use std::net::TcpListener; + use std::sync::{Arc, Mutex}; + use std::thread; + + fn file(name: &str, size: u64) -> HubFile { + HubFile { + name: name.to_owned(), + size: Some(size), + } + } + + #[test] + fn a_reference_is_read_as_a_hub_repository_a_path_or_neither() { + assert_eq!( + parse_gguf_ref("owner/repo:Q4_K_M"), + Some(GgufRef::Hub { + owner: "owner".into(), + repo: "repo".into(), + variant: Some("Q4_K_M".into()) + }) + ); + assert_eq!( + parse_gguf_ref("owner/repo:model-Q4_K_M.gguf"), + Some(GgufRef::Hub { + owner: "owner".into(), + repo: "repo".into(), + variant: Some("model-Q4_K_M.gguf".into()) + }) + ); + // A bare repository is recognised so it can be refused with a message. + assert_eq!( + parse_gguf_ref("owner/repo"), + Some(GgufRef::Hub { + owner: "owner".into(), + repo: "repo".into(), + variant: None + }) + ); + // A path ending in .gguf is a file even though it looks like owner/repo. + assert_eq!( + parse_gguf_ref("models/thing.gguf"), + Some(GgufRef::Local(PathBuf::from("models/thing.gguf"))) + ); + assert_eq!( + parse_gguf_ref("/abs/path/Thing.GGUF"), + Some(GgufRef::Local(PathBuf::from("/abs/path/Thing.GGUF"))) + ); + for not_a_gguf in [ + "qwen", + "", + " ", + "a/b/c", + "../x/y:Q4", + "owner/..:Q4", + "o/r:a/b", + ] { + assert_eq!(parse_gguf_ref(not_a_gguf), None, "{not_a_gguf:?}"); + } + } + + /// `m.gguf` at the root and `dir/m.gguf` share a last component. Naming the + /// full path of either picks that one; the bare name picks the root file, which + /// matches in full, instead of reporting the two as ambiguous. + #[test] + fn a_full_path_match_beats_a_match_on_the_last_component() { + let files = [file("dir/m.gguf", 4), file("m.gguf", 4)]; + assert_eq!(select_gguf_file(&files, "m.gguf").unwrap().name, "m.gguf"); + assert_eq!( + select_gguf_file(&files, "dir/m.gguf").unwrap().name, + "dir/m.gguf" + ); + // With no root file the bare name finds the one nested file it names... + let nested = [file("dir/m.gguf", 4), file("dir/other.gguf", 4)]; + assert_eq!( + select_gguf_file(&nested, "m.gguf").unwrap().name, + "dir/m.gguf" + ); + // ...and is ambiguous when it is the last component of two. + let two = [file("a/m.gguf", 4), file("b/m.gguf", 4)]; + assert!(select_gguf_file(&two, "m.gguf").is_err()); + } + + #[test] + fn listed_names_that_could_climb_out_of_the_repository_or_hide_in_a_terminal_are_dropped() { + for name in [ + "../m-Q4_K_M.gguf", + "a/../m-Q4_K_M.gguf", + "./m-Q4_K_M.gguf", + "a//m-Q4_K_M.gguf", + "/m-Q4_K_M.gguf", + "m-Q4_K_M\n.gguf", + "m-Q4_K_M\u{1b}[2J.gguf", + "m-\u{202E}fuggQ4_K_M.gguf", + "m-Q4_K_M\u{200B}.gguf", + "m-Q4_K_M\u{2066}.gguf", + // Both ends of each range, and the byte order mark. + "m-Q4_K_M\u{200F}.gguf", + "m-Q4_K_M\u{202A}.gguf", + "m-Q4_K_M\u{2069}.gguf", + "m-Q4_K_M\u{FEFF}.gguf", + // What the shared terminal predicates add: line and paragraph + // separators, invisible operators, the soft hyphen, tag characters. + "m-Q4_K_M\u{2028}.gguf", + "m-Q4_K_M\u{2029}.gguf", + "m-Q4_K_M\u{2060}.gguf", + "m-Q4_K_M\u{2064}.gguf", + "m-Q4\u{00AD}_K_M.gguf", + "m-Q4_K_M\u{E0041}.gguf", + ] { + assert!(!is_safe_listed_name(name), "{name:?}"); + } + for name in ["m-Q4_K_M.gguf", "dir/m-Q4_K_M.gguf", "a.b/c_d-e.gguf"] { + assert!(is_safe_listed_name(name), "{name:?}"); + } + } + + /// The same through a listing a hub sent: the hostile names are not offered, + /// so the one safe file is the only match and nothing but it is requested. + #[test] + fn a_hostile_listing_is_not_followed_out_of_the_repository() { + let listing = r#"{"siblings":[ + {"rfilename":"../m-Q4_K_M.gguf","size":1048576}, + {"rfilename":"a/../m-Q4_K_M.gguf","size":1048576}, + {"rfilename":"m-Q4_K_M\u001b[2J.gguf","size":1048576}, + {"rfilename":"ok-Q4_K_M.gguf","size":1048576}]}"#; + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Honour, + listing.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + assert_eq!(opened.description, "o/r: ok-Q4_K_M.gguf"); + parse_gguf_header(&mut opened.reader).unwrap(); + let requests = hub.requests.lock().unwrap().clone(); + assert!( + !requests.is_empty() + && requests + .iter() + .all(|request| request.ends_with("/o/r/resolve/main/ok-Q4_K_M.gguf")), + "{requests:?}" + ); + } + + /// A projector is never picked as the model, but a name that matches only + /// projectors says what it named, rather than that the repository has no + /// such file (which is what filtering projectors out before the match used to + /// say, with the file missing from the list too). + #[test] + fn a_name_that_matches_only_projectors_is_refused_as_naming_a_projector() { + let files = [ + file("m-Q4_K_M.gguf", 5), + file("mmproj-m-f16.gguf", 1), + file("vision/MMPROJ-Q8_0.gguf", 1), + file("x/mmproj.gguf", 1), + file("y/mmproj.gguf", 1), + ]; + // What was typed, and the real file (or files) it points at, which the + // refusal names. + for (named, real) in [ + ( + "mmproj-m-f16.gguf", + "`mmproj-m-f16.gguf` is a multimodal projector", + ), + ( + "vision/MMPROJ-Q8_0.gguf", + "`vision/MMPROJ-Q8_0.gguf` is a multimodal projector", + ), + ( + "MMPROJ-Q8_0.gguf", + "`vision/MMPROJ-Q8_0.gguf` is a multimodal projector", + ), + ( + "mmproj.gguf", + "`x/mmproj.gguf`, `y/mmproj.gguf` are multimodal projectors", + ), + ] { + let error = select_gguf_file(&files, named).unwrap_err().to_string(); + assert!(error.starts_with(real), "{named}: {error}"); + assert!( + error.contains(&format!("so `{named}` cannot be estimated")), + "{named}: {error}" + ); + assert!(!error.contains("no GGUF file matching"), "{named}: {error}"); + assert!(!error.contains("matches several files"), "{named}: {error}"); + // The model is listed as the alternative, and no projector is. + let models = error.split("the repository's models:").nth(1).unwrap_or(""); + assert!(models.contains("m-Q4_K_M.gguf"), "{named}: {error}"); + assert!( + !models.to_ascii_lowercase().contains("mmproj"), + "{named}: {error}" + ); + } + // The model itself is still picked by name and by label. + assert_eq!( + select_gguf_file(&files, "m-Q4_K_M.gguf").unwrap().name, + "m-Q4_K_M.gguf" + ); + assert_eq!( + select_gguf_file(&files, "Q4_K_M").unwrap().name, + "m-Q4_K_M.gguf" + ); + // A name that matches nothing at all is still "no such file", and the + // listing shows every real file, the projectors marked as such. + for absent in ["other.gguf", "F16"] { + let missing = select_gguf_file(&files, absent).unwrap_err().to_string(); + assert!(missing.contains("no GGUF file matching"), "{missing}"); + assert!( + missing.contains("m-Q4_K_M.gguf, mmproj-m-f16.gguf (multimodal projector)"), + "{missing}" + ); + assert!( + missing.contains("vision/MMPROJ-Q8_0.gguf (multimodal projector)"), + "{missing}" + ); + } + } + + /// A bare name shared by a model and a projector picks the model, as it did + /// before projectors were searched by name at all. + #[test] + fn a_bare_name_shared_with_a_projector_picks_the_model() { + let files = [file("a/m.gguf", 5), file("mmproj/m.gguf", 1)]; + assert_eq!(select_gguf_file(&files, "m.gguf").unwrap().name, "a/m.gguf"); + } + + /// A repository with nothing but projectors says so, not that it has no GGUF. + #[test] + fn a_repository_of_only_projectors_says_it_has_no_model() { + let files = [file("mmproj-a.gguf", 1)]; + let error = select_gguf_file(&files, "mmproj-a.gguf") + .unwrap_err() + .to_string(); + assert!( + error.contains("every GGUF file in it is a projector"), + "{error}" + ); + // Asked for something else, it lists the real file rather than claiming + // the repository has no GGUF. + let other = select_gguf_file(&files, "Q4_K_M").unwrap_err().to_string(); + assert!( + other.contains("it has: mmproj-a.gguf (multimodal projector)"), + "{other}" + ); + } + + /// A letter before a label is not a boundary: `F16` does not match `BF16`, + /// in either order of listing. + #[test] + fn a_label_inside_a_longer_label_is_not_matched() { + for files in [ + [file("m-BF16.gguf", 16), file("m-F16.gguf", 16)], + [file("m-F16.gguf", 16), file("m-BF16.gguf", 16)], + ] { + assert_eq!(select_gguf_file(&files, "F16").unwrap().name, "m-F16.gguf"); + assert_eq!( + select_gguf_file(&files, "BF16").unwrap().name, + "m-BF16.gguf" + ); + } + let only_bf16 = [file("m-BF16.gguf", 16)]; + assert!(select_gguf_file(&only_bf16, "F16").is_err()); + } + + /// A shard's numbering is the last `-of-` in its name, so a model whose own + /// name contains `-of-` is still recognised as one split model. + #[test] + fn the_shards_of_a_model_with_of_in_its_name_are_one_model() { + let shards = [ + file("Chain-of-Thought-Q4_K_M-00002-of-00003.gguf", 1), + file("Chain-of-Thought-Q4_K_M-00001-of-00003.gguf", 1), + file("Chain-of-Thought-Q4_K_M-00003-of-00003.gguf", 1), + ]; + assert_eq!( + select_gguf_file(&shards, "Q4_K_M").unwrap().name, + "Chain-of-Thought-Q4_K_M-00001-of-00003.gguf" + ); + } + + /// `_` before a label is a boundary as `-` and `.` are: llama.cpp's own + /// conversions name files `model_Q4_K_M.gguf` as often as `model-Q4_K_M.gguf`. + #[test] + fn a_label_after_an_underscore_is_matched() { + let files = [file("model_Q4_K_M.gguf", 5), file("model_Q8_0.gguf", 8)]; + assert_eq!( + select_gguf_file(&files, "Q4_K_M").unwrap().name, + "model_Q4_K_M.gguf" + ); + } + + /// A listed name is sent percent-encoded, one segment at a time: a `+` sent + /// raw is read as a space by some servers, and a space breaks the request. + #[test] + fn a_listed_file_name_is_requested_percent_encoded() { + let listing = r#"{"siblings":[{"rfilename":"dir/m+x-Q4_K_M.gguf","size":1048576}]}"#; + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Honour, + listing.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + parse_gguf_header(&mut opened.reader).unwrap(); + let requests = hub.requests.lock().unwrap().clone(); + assert!( + !requests.is_empty() + && requests + .iter() + .all(|request| request.ends_with("/o/r/resolve/main/dir/m%2Bx-Q4_K_M.gguf")), + "{requests:?}" + ); + } + + /// A loopback server that accepts every connection, counts it, and answers + /// nothing for `stall`: the shape of a hub that has stopped responding. + fn stalling_server(stall: Duration) -> (String, Arc>) { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let connections = Arc::new(Mutex::new(0)); + let counted = Arc::clone(&connections); + thread::spawn(move || { + for stream in listener.incoming() { + let Ok(stream) = stream else { break }; + *counted.lock().unwrap() += 1; + thread::spawn(move || { + thread::sleep(stall); + drop(stream); + }); + } + }); + (base, connections) + } + + /// The listing request is bound by the read's deadline, not only by the + /// agent's one-minute timeout: a spent deadline makes no request at all, and + /// a hub that stops answering is cut off when the deadline passes. + #[test] + fn the_listing_request_is_bound_by_the_deadline() { + let (base, connections) = stalling_server(Duration::from_secs(3)); + let source = LiveGgufSource::with_hub(&base); + let agent = LiveGgufSource::agent(); + let error = source + .list_files(&agent, "o", "r", Instant::now()) + .unwrap_err() + .to_string(); + assert!(error.contains("took longer than"), "{error}"); + assert_eq!( + *connections.lock().unwrap(), + 0, + "a request after the deadline" + ); + let began = Instant::now(); + assert!( + source + .list_files( + &agent, + "o", + "r", + Instant::now() + Duration::from_millis(300) + ) + .is_err() + ); + assert!( + began.elapsed() < Duration::from_secs(2), + "waited {:?} for a hub that never answered", + began.elapsed() + ); + } + + /// The same bound on the `HEAD` request for a file the listing did not size. + #[test] + fn the_head_request_is_bound_by_the_deadline() { + let (base, connections) = stalling_server(Duration::from_secs(3)); + let url = format!("{base}/o/r/resolve/main/m.gguf"); + let agent = LiveGgufSource::agent(); + let error = head_size(&agent, &url, Instant::now()) + .unwrap_err() + .to_string(); + assert!(error.contains("took longer than"), "{error}"); + assert_eq!( + *connections.lock().unwrap(), + 0, + "a request after the deadline" + ); + let began = Instant::now(); + assert!(head_size(&agent, &url, Instant::now() + Duration::from_millis(300)).is_err()); + assert!( + began.elapsed() < Duration::from_secs(2), + "waited {:?} for a hub that never answered", + began.elapsed() + ); + } + + /// Counts the seeks that reach the file under the buffer. + struct CountingSeeks { + inner: R, + seeks: Arc>, + } + + impl Read for CountingSeeks { + fn read(&mut self, buf: &mut [u8]) -> io::Result { + self.inner.read(buf) + } + } + + impl Seek for CountingSeeks { + fn seek(&mut self, pos: SeekFrom) -> io::Result { + *self.seeks.lock().unwrap() += 1; + self.inner.seek(pos) + } + } + + /// Skipping a large vocabulary stays inside the read buffer: the file under + /// it is sought a handful of times, not once per token, which is what made a + /// real header take seconds to read from disk. + #[test] + fn a_local_header_is_skipped_through_without_seeking_the_file_per_token() { + let tokens = 50_000; + let file = llama_like(tokens).header_bytes(); + let seeks = Arc::new(Mutex::new(0)); + let mut reader = RelativeSeekBuf::new(BufReader::new(CountingSeeks { + inner: Cursor::new(file), + seeks: Arc::clone(&seeks), + })); + let header = parse_gguf_header(&mut reader).unwrap(); + assert_eq!(header.tensor_count, 3); + // The same header read straight from memory: the wrapper changes how + // the bytes are reached, never what is read. + let direct = + parse_gguf_header(&mut Cursor::new(llama_like(tokens).header_bytes())).unwrap(); + assert_eq!(header.data_offset, direct.data_offset); + assert_eq!(header.token_embd, direct.token_embd); + let seeks = *seeks.lock().unwrap(); + assert!( + seeks < tokens / 100, + "{seeks} seeks reached the file for {tokens} tokens" + ); + } + + /// The position reported is where the reads and skips have reached, and + /// asking for it never reaches the file under the buffer. + #[test] + fn the_buffered_reader_tracks_its_position_without_asking_the_file() { + let seeks = Arc::new(Mutex::new(0)); + let mut reader = RelativeSeekBuf::new(BufReader::new(CountingSeeks { + inner: Cursor::new(vec![7_u8; 100_000]), + seeks: Arc::clone(&seeks), + })); + let mut byte = [0_u8; 10]; + reader.read_exact(&mut byte).unwrap(); + assert_eq!(reader.seek(SeekFrom::Current(90)).unwrap(), 100); + reader.read_exact(&mut byte).unwrap(); + assert_eq!(reader.seek(SeekFrom::Current(-5)).unwrap(), 105); + let before = *seeks.lock().unwrap(); + for _ in 0..1000 { + assert_eq!(reader.stream_position().unwrap(), 105); + } + assert_eq!( + *seeks.lock().unwrap(), + before, + "asking the position sought the file" + ); + // A skip past the buffer still lands where it was asked to. + assert_eq!(reader.seek(SeekFrom::Current(50_000)).unwrap(), 50_105); + reader.read_exact(&mut byte).unwrap(); + assert_eq!(reader.stream_position().unwrap(), 50_115); + } + + /// A `+` is allowed in an exact file name, which is how a hub file such as + /// `m+x-Q4_K_M.gguf` is named; it is sent percent-encoded. + #[test] + fn an_exact_file_name_with_a_plus_can_be_named() { + assert!(matches!( + parse_gguf_ref_with("o/r:dir/m+x-Q4_K_M.gguf", &|_| false), + Some(GgufRef::Hub { variant: Some(ref v), .. }) if v == "dir/m+x-Q4_K_M.gguf" + )); + // Not in a bare quantization label, which `rocm serve` also reads. + assert!(parse_gguf_ref_with("o/r:Q4+K", &|_| false).is_none()); + } + + #[test] + fn a_quantization_label_picks_one_file_and_never_a_lookalike() { + let files = [ + file("m-Q4_0.gguf", 4), + file("m-Q4_0_4_4.gguf", 4), + file("m-Q4_K_M.gguf", 5), + file("m-Q8_0.gguf", 8), + file("README.md", 1), + file("mmproj-Q8_0.gguf", 1), + ]; + assert_eq!( + select_gguf_file(&files, "Q4_K_M").unwrap().name, + "m-Q4_K_M.gguf" + ); + assert_eq!( + select_gguf_file(&files, "q4_k_m").unwrap().name, + "m-Q4_K_M.gguf" + ); + assert_eq!( + select_gguf_file(&files, "Q4_0").unwrap().name, + "m-Q4_0.gguf" + ); + // The vision projector shares the label and is not the model. + assert_eq!( + select_gguf_file(&files, "Q8_0").unwrap().name, + "m-Q8_0.gguf" + ); + assert_eq!( + select_gguf_file(&files, "m-Q8_0.gguf").unwrap().name, + "m-Q8_0.gguf" + ); + let missing = select_gguf_file(&files, "IQ2_XS").unwrap_err().to_string(); + assert!( + missing.contains("no GGUF file matching `IQ2_XS`") && missing.contains("m-Q4_K_M.gguf"), + "{missing}" + ); + } + + #[test] + fn two_files_for_one_label_are_refused_but_the_shards_of_one_model_are_not() { + let twins = [file("a/m-Q4_K_M.gguf", 1), file("b/m-Q4_K_M.gguf", 1)]; + let error = select_gguf_file(&twins, "Q4_K_M").unwrap_err().to_string(); + assert!(error.contains("matches several files"), "{error}"); + // The exact name still resolves one of them. + assert_eq!( + select_gguf_file(&twins, "a/m-Q4_K_M.gguf").unwrap().name, + "a/m-Q4_K_M.gguf" + ); + let shards = [ + file("Q4_K_M/m-Q4_K_M-00002-of-00002.gguf", 1), + file("Q4_K_M/m-Q4_K_M-00001-of-00002.gguf", 1), + ]; + assert_eq!( + select_gguf_file(&shards, "Q4_K_M").unwrap().name, + "Q4_K_M/m-Q4_K_M-00001-of-00002.gguf" + ); + } + + #[test] + fn a_bare_repository_is_refused_without_touching_the_network() { + // No server is listening on this port: a refusal that came from a + // connection error rather than from the missing variant would show it. + let source = LiveGgufSource::with_hub("http://127.0.0.1:1"); + let error = source + .open(&GgufRef::Hub { + owner: "o".into(), + repo: "r".into(), + variant: None, + }) + .err() + .unwrap() + .to_string(); + assert!(error.contains("needs a quantization"), "{error}"); + } + + #[test] + fn a_local_file_is_opened_with_its_size() { + let dir = std::env::temp_dir().join(format!("rocm-gguf-local-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("m.gguf"); + let bytes = llama_like(5).file(100); + std::fs::write(&path, &bytes).unwrap(); + let mut opened = LiveGgufSource::with_hub("http://127.0.0.1:1") + .open(&GgufRef::Local(path)) + .unwrap(); + assert_eq!(opened.size, bytes.len() as u64); + assert_eq!( + parse_gguf_header(&mut opened.reader).unwrap().block_count, + Some(32) + ); + let missing = LiveGgufSource::new() + .open(&GgufRef::Local(dir.join("absent.gguf"))) + .err() + .unwrap() + .to_string(); + assert!(missing.contains("could not open"), "{missing}"); + // An ordinary missing file is not told it might be a hub reference... + assert!(!missing.contains("Hugging Face"), "{missing}"); + // ...but one shaped like a file in a repository is shown its hub form. + let hubby = LiveGgufSource::new() + .open(&GgufRef::Local(PathBuf::from("owner/repo/dir/file.gguf"))) + .err() + .unwrap() + .to_string(); + assert!(hubby.contains("`owner/repo:dir/file.gguf`"), "{hubby}"); + let two = LiveGgufSource::new() + .open(&GgufRef::Local(PathBuf::from("owner/file.gguf"))) + .err() + .unwrap() + .to_string(); + assert!(!two.contains("Hugging Face"), "{two}"); + std::fs::remove_dir_all(&dir).ok(); + } + + /// How the fake hub answers a request for the model file. + #[derive(Clone, Copy, PartialEq, Eq)] + enum Serve { + /// A correct `206` for the range asked. + Honour, + /// A `200` with the whole (here tiny) body, ignoring `Range`. + IgnoreRange, + /// A `206` whose `Content-Range` starts one byte after the asked offset. + WrongRange, + /// A `206` that promises the whole range and drops the connection partway. + ShortBody, + /// A `206` whose `Content-Range` names the whole range but whose body, with + /// a `Content-Length` that matches it, is only half of it: nothing breaks + /// at the HTTP level, the body is simply short. + ShortConsistent, + /// A `401`, as for a gated repository. + Unauthorized, + /// A `403`, as for a repository that refuses the request. + Forbidden, + /// A `206` that arrives after a pause. + Slow, + /// A `206` with no `Content-Range`, so which bytes it carries is unstated. + NoContentRange, + /// A `HEAD` answered with no `Content-Length`; ranges are honoured. + HeadWithoutLength, + /// A `206` whose `Content-Range` is right but whose body, with no + /// `Content-Length`, goes on past the range: the bytes asked for, then + /// 8 MiB of `0xFF`, then the connection closes. + Overlong, + } + + /// A hub on loopback. The "model" is a real header followed by zeros out to + /// `virtual_size`, served only as far as a range asks for it, so a client that + /// tried to read the weights would show in `ranges`. + struct FakeHub { + base: String, + ranges: Arc>>, + /// Every request line that reached the model file, with its method. + requests: Arc>>, + } + + fn fake_hub(header: Vec, virtual_size: u64, honour_range: bool) -> FakeHub { + let serve = if honour_range { + Serve::Honour + } else { + Serve::IgnoreRange + }; + let listing = format!( + r#"{{"siblings":[{{"rfilename":"m-Q4_K_M.gguf","size":{virtual_size}}},{{"rfilename":"README.md","size":10}}]}}"# + ); + fake_hub_with(header, virtual_size, serve, listing) + } + + fn fake_hub_with(header: Vec, virtual_size: u64, serve: Serve, listing: String) -> FakeHub { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let ranges = Arc::new(Mutex::new(Vec::new())); + let requests = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::clone(&ranges); + let logged = Arc::clone(&requests); + thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + if stream.read(&mut byte).unwrap_or(0) == 0 { + break; + } + request.push(byte[0]); + } + let text = String::from_utf8_lossy(&request).into_owned(); + let method = text.split_whitespace().next().unwrap_or("").to_owned(); + let path = text.split_whitespace().nth(1).unwrap_or("").to_owned(); + let range = text + .lines() + .find_map(|line| { + line.strip_prefix("Range: bytes=") + .or_else(|| line.strip_prefix("range: bytes=")) + }) + .map(str::to_owned); + if path.starts_with("/api/models/o/r") { + let body = listing.clone(); + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + } else if path.starts_with("/o/r/resolve/main/") { + logged.lock().unwrap().push(format!("{method} {path}")); + if method == "HEAD" && serve == Serve::HeadWithoutLength { + let _ = write!(stream, "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n"); + continue; + } + if method == "HEAD" { + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Length: {virtual_size}\r\nConnection: close\r\n\r\n" + ); + continue; + } + seen.lock() + .unwrap() + .push(range.clone().unwrap_or_else(|| "none".into())); + match (range, serve) { + (_, Serve::Unauthorized) => { + let _ = write!( + stream, + "HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + } + (_, Serve::Forbidden) => { + let _ = write!( + stream, + "HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + } + ( + Some(range), + Serve::Honour + | Serve::WrongRange + | Serve::ShortBody + | Serve::ShortConsistent + | Serve::Slow + | Serve::NoContentRange + | Serve::HeadWithoutLength + | Serve::Overlong, + ) => { + let (start, end) = range.split_once('-').unwrap(); + let (start, end): (u64, u64) = + (start.parse().unwrap(), end.parse().unwrap()); + let mut body: Vec = (start..=end) + .map(|at| header.get(at as usize).copied().unwrap_or(0)) + .collect(); + if serve == Serve::ShortConsistent { + body.truncate(body.len() / 2); + } + let claimed_start = if serve == Serve::WrongRange { + start + 1 + } else { + start + }; + if serve == Serve::Slow { + thread::sleep(Duration::from_millis(1500)); + } + let content_range = if serve == Serve::NoContentRange { + String::new() + } else { + format!( + "Content-Range: bytes {claimed_start}-{end}/{virtual_size}\r\n" + ) + }; + if serve == Serve::Overlong { + let _ = write!( + stream, + "HTTP/1.1 206 Partial Content\r\n{content_range}Connection: close\r\n\r\n" + ); + let _ = stream.write_all(&body); + let _ = stream.write_all(&vec![0xFF_u8; 8 << 20]); + continue; + } + let _ = write!( + stream, + "HTTP/1.1 206 Partial Content\r\n{content_range}Content-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + if serve == Serve::ShortBody { + let _ = stream.write_all(&body[..body.len() / 2]); + } else { + let _ = stream.write_all(&body); + } + } + _ => { + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Length: 4\r\nConnection: close\r\n\r\nGGUF" + ); + } + } + } else { + let _ = write!( + stream, + "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + } + } + }); + FakeHub { + base, + ranges, + requests, + } + } + + /// A loopback server that answers every request with a `302` to the same + /// path on `target`, as the hub's `resolve/main` answers with a CDN host. + /// Counts the requests it redirected. + fn redirecting_server(target: String) -> (String, Arc>) { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let redirected = Arc::new(Mutex::new(0)); + let counted = Arc::clone(&redirected); + thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + if stream.read(&mut byte).unwrap_or(0) == 0 { + break; + } + request.push(byte[0]); + } + let text = String::from_utf8_lossy(&request).into_owned(); + let path = text.split_whitespace().nth(1).unwrap_or("/").to_owned(); + *counted.lock().unwrap() += 1; + let _ = write!( + stream, + "HTTP/1.1 302 Found\r\nLocation: {target}{path}\r\nContent-Length: \ + 0\r\nConnection: close\r\n\r\n" + ); + } + }); + (base, redirected) + } + + /// The hub answers `resolve/main` with a redirect to its CDN. The listing, + /// the `HEAD` and every range request follow it to the other host, and the + /// range requests still carry their `Range` there. + #[test] + fn a_redirect_to_another_host_is_followed_with_the_range_kept() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Honour, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf"}]}"#.to_owned(), + ); + let (front, redirected) = redirecting_server(hub.base.clone()); + let mut opened = LiveGgufSource::with_hub(&front) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + assert_eq!( + opened.size, + 1 << 20, + "the HEAD's size, read after the redirect" + ); + let header = parse_gguf_header(&mut opened.reader).unwrap(); + assert_eq!(header.block_count, Some(32)); + let ranges = hub.ranges.lock().unwrap().clone(); + assert!( + !ranges.is_empty() && ranges.iter().all(|range| range != "none"), + "{ranges:?}" + ); + // The listing, the HEAD and each range request went through the front. + assert!(*redirected.lock().unwrap() >= 2 + ranges.len()); + } + + #[test] + fn only_the_header_is_requested_from_a_file_far_larger_than_it() { + // 40 000 tokens makes the header a few hundred KB: bigger than nothing, + // far smaller than the 8 GiB the server claims the file to be. + let header = llama_like(40_000).header_bytes(); + let virtual_size: u64 = 8 * 1024 * 1024 * 1024; + let hub = fake_hub(header.clone(), virtual_size, true); + let source = LiveGgufSource::with_hub(&hub.base); + let mut opened = source + .open(&GgufRef::Hub { + owner: "o".into(), + repo: "r".into(), + variant: Some("Q4_K_M".into()), + }) + .unwrap(); + assert_eq!(opened.size, virtual_size); + assert_eq!(opened.description, "o/r: m-Q4_K_M.gguf"); + let parsed = parse_gguf_header(&mut opened.reader).unwrap(); + assert_eq!(parsed.block_count, Some(32)); + let ranges = hub.ranges.lock().unwrap().clone(); + assert!( + !ranges.is_empty() && ranges.iter().all(|r| r != "none"), + "{ranges:?}" + ); + let fetched: u64 = ranges + .iter() + .map(|r| { + let (start, end) = r.split_once('-').unwrap(); + end.parse::().unwrap() - start.parse::().unwrap() + 1 + }) + .sum(); + assert!( + fetched <= 2 * WINDOW_BYTES && fetched < virtual_size / 1000, + "{fetched} bytes were requested for a header of {} bytes: {ranges:?}", + header.len() + ); + } + + #[test] + fn a_server_that_ignores_range_is_refused_rather_than_streamed() { + let hub = fake_hub(llama_like(10).header_bytes(), 1 << 30, false); + let source = LiveGgufSource::with_hub(&hub.base); + let mut opened = source + .open(&GgufRef::Hub { + owner: "o".into(), + repo: "r".into(), + variant: Some("Q4_K_M".into()), + }) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader) + .unwrap_err() + .to_string(); + assert!( + error.contains("does not support reading part of a file"), + "{error}" + ); + } + + #[test] + fn a_missing_repository_is_reported_as_not_found() { + let hub = fake_hub(Vec::new(), 1, true); + let source = LiveGgufSource::with_hub(&hub.base); + let error = source + .open(&GgufRef::Hub { + owner: "nobody".into(), + repo: "nothing".into(), + variant: Some("Q4_K_M".into()), + }) + .err() + .unwrap() + .to_string(); + assert!( + error.contains("was not found, or its repository is gated or private"), + "{error}" + ); + } + + #[test] + fn the_reader_never_fetches_more_than_its_budget() { + let mut reader = RangedReader::new( + ureq::AgentBuilder::new().build(), + "http://127.0.0.1:1/unused".into(), + 1 << 40, + ); + reader.fetched = MAX_FETCH_BYTES; + let mut buf = [0_u8; 1]; + let error = reader.read(&mut buf).unwrap_err().to_string(); + assert!(error.contains("not a GGUF header read"), "{error}"); + } + + #[test] + fn seeking_within_the_window_costs_no_request_and_past_the_end_reads_nothing() { + let header = llama_like(10).header_bytes(); + let hub = fake_hub(header, 1 << 20, true); + let mut reader = RangedReader::new( + ureq::AgentBuilder::new().build(), + format!("{}/o/r/resolve/main/m-Q4_K_M.gguf", hub.base), + 1 << 20, + ); + let mut four = [0_u8; 4]; + reader.read_exact(&mut four).unwrap(); + assert_eq!(&four, b"GGUF"); + reader.seek(SeekFrom::Current(100)).unwrap(); + reader.read_exact(&mut four).unwrap(); + reader.seek(SeekFrom::Start(0)).unwrap(); + reader.read_exact(&mut four).unwrap(); + assert_eq!(hub.ranges.lock().unwrap().len(), 1); + assert_eq!(reader.fetched(), 1 << 20); + reader.seek(SeekFrom::Start(1 << 21)).unwrap(); + assert_eq!(reader.read(&mut four).unwrap(), 0); + assert!(reader.seek(SeekFrom::Current(-(1 << 30))).is_err()); + } + + #[test] + fn a_file_that_exists_wins_over_the_hub_reading_of_its_name() { + // `m/x:y.gguf` has the shape of `owner/repo:variant`. When such a file is + // there it is the file; when it is not, the same string is the hub's. + let exists = |path: &Path| path == Path::new("m/x:y.gguf"); + assert_eq!( + parse_gguf_ref_with("m/x:y.gguf", &exists), + Some(GgufRef::Local(PathBuf::from("m/x:y.gguf"))) + ); + assert_eq!( + parse_gguf_ref_with("m/x:y.gguf", &|_| false), + Some(GgufRef::Hub { + owner: "m".into(), + repo: "x".into(), + variant: Some("y.gguf".into()) + }) + ); + // A `.gguf` name that does not exist is a path that cannot be opened, not + // a repository to ask the hub about. + assert_eq!( + parse_gguf_ref_with("nosuch.gguf", &|_| false), + Some(GgufRef::Local(PathBuf::from("nosuch.gguf"))) + ); + } + + #[test] + fn a_variant_may_carry_a_directory_only_when_it_names_a_gguf_file() { + assert_eq!( + parse_gguf_ref("o/r:b/m-Q4_K_M.gguf"), + Some(GgufRef::Hub { + owner: "o".into(), + repo: "r".into(), + variant: Some("b/m-Q4_K_M.gguf".into()) + }) + ); + // None of these is a hub name: a directory without a file name, a + // traversal, an empty segment, a backslash, an escape. The ones ending in + // `.gguf` are read as local paths (which then cannot be opened), never + // sent to the hub. + for rejected in [ + "o/r:b/Q4_K_M", + "o/r:../m.gguf", + "o/r:b/../m.gguf", + "o/r:/m.gguf", + "o/r:b//m.gguf", + "o/r:b\\m.gguf", + "o/r:b/m%2F.gguf", + ] { + assert!( + !matches!(parse_gguf_ref(rejected), Some(GgufRef::Hub { .. })), + "{rejected:?}" + ); + } + } + + #[test] + fn shards_of_two_different_models_are_not_one_model() { + let two_sets = [ + file("a/m-Q4_K_M-00001-of-00002.gguf", 1), + file("a/m-Q4_K_M-00002-of-00002.gguf", 1), + file("b/m-Q4_K_M-00001-of-00003.gguf", 1), + file("b/m-Q4_K_M-00002-of-00003.gguf", 1), + file("b/m-Q4_K_M-00003-of-00003.gguf", 1), + ]; + let error = select_gguf_file(&two_sets, "Q4_K_M") + .unwrap_err() + .to_string(); + assert!(error.contains("matches several files"), "{error}"); + // Naming one shard exactly still selects it. + assert_eq!( + select_gguf_file(&two_sets, "b/m-Q4_K_M-00001-of-00003.gguf") + .unwrap() + .name, + "b/m-Q4_K_M-00001-of-00003.gguf" + ); + // The same stem with a different shard count is a different split. + let recut = [ + file("m-Q4_K_M-00001-of-00002.gguf", 1), + file("m-Q4_K_M-00001-of-00003.gguf", 1), + ]; + assert!(select_gguf_file(&recut, "Q4_K_M").is_err()); + // A match that is not shards at all is never "the first shard". + let mixed = [ + file("m-Q4_K_M-00001-of-00002.gguf", 1), + file("other-Q4_K_M.gguf", 1), + ]; + assert!(select_gguf_file(&mixed, "Q4_K_M").is_err()); + } + + fn hub_ref(variant: &str) -> GgufRef { + parse_gguf_ref(&format!("o/r:{variant}")).unwrap() + } + + #[test] + fn an_exact_file_in_a_directory_is_reachable_from_the_reference_string() { + // Two files share a name in different directories. The label is + // ambiguous, and the error says how to be exact; doing what it says, + // as the user would type it, reads that file and no other. + let listing = r#"{"siblings":[{"rfilename":"a/m-Q4_K_M.gguf","size":1048576},{"rfilename":"b/m-Q4_K_M.gguf","size":1048576}]}"#; + let header = llama_like(10).header_bytes(); + let hub = fake_hub_with(header, 1 << 20, Serve::Honour, listing.to_owned()); + let source = LiveGgufSource::with_hub(&hub.base); + let error = source.open(&hub_ref("Q4_K_M")).err().unwrap().to_string(); + assert!( + error.contains("matches several files") && error.contains("owner/repo:dir/file.gguf"), + "{error}" + ); + let mut opened = source.open(&hub_ref("b/m-Q4_K_M.gguf")).unwrap(); + assert_eq!(opened.description, "o/r: b/m-Q4_K_M.gguf"); + parse_gguf_header(&mut opened.reader).unwrap(); + let requests = hub.requests.lock().unwrap().clone(); + assert!( + !requests.is_empty() + && requests + .iter() + .all(|request| request.ends_with("/o/r/resolve/main/b/m-Q4_K_M.gguf")), + "{requests:?}" + ); + } + + #[test] + fn a_range_that_starts_elsewhere_is_refused_rather_than_read_from_the_wrong_offset() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::WrongRange, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader).unwrap_err(); + assert!( + matches!(&error, GgufError::Io(_)) && error.to_string().contains("starts at"), + "{error}" + ); + } + + /// A `206` that does not say which bytes it carries is refused like one that + /// names the wrong ones: the reader cannot know it starts where it asked. + #[test] + fn a_partial_answer_with_no_content_range_is_refused() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::NoContentRange, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader).unwrap_err(); + assert!( + matches!(&error, GgufError::Io(_)) && error.to_string().contains("an unstated offset"), + "{error}" + ); + } + + /// A file the listing gave no size for, whose `HEAD` gives none either, is + /// refused with a reason rather than read as zero bytes long. + #[test] + fn a_head_with_no_content_length_is_refused() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::HeadWithoutLength, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf"}]}"#.to_owned(), + ); + let error = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .err() + .expect("a file of unknown size is not opened") + .to_string(); + assert!(error.contains("did not report how large"), "{error}"); + } + + #[test] + fn a_connection_that_drops_is_not_reported_as_a_header_that_ends() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::ShortBody, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader).unwrap_err(); + // `Truncated` is "the GGUF header ends before it is complete": a claim + // about the file that a dropped connection does not support. + assert!( + matches!(&error, GgufError::Io(_)) && error.to_string().contains("connection ended"), + "{error}" + ); + } + + #[test] + fn a_short_but_well_formed_body_is_reported_as_the_connection_ending() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::ShortConsistent, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader).unwrap_err(); + // The body ended cleanly, so the HTTP layer reported nothing: it is the + // length check that says fewer bytes came than were asked for, and it must + // not say the header ends (`Truncated`), which blames the file. + assert!( + matches!(&error, GgufError::Io(_)) + && error.to_string().contains("connection ended after") + && !error.to_string().contains("bytes asked for:"), + "{error}" + ); + } + + #[test] + fn a_gated_file_is_said_to_be_unreadable_without_offering_a_token() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Unauthorized, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader) + .unwrap_err() + .to_string(); + assert!( + error.contains("gated or private") && error.contains("no token is sent"), + "{error}" + ); + assert!( + !error.contains("authentication"), + "no token is sent, so authentication is not on offer: {error}" + ); + } + + /// The hub answers 403 as well as 401 for a repository it will not serve to + /// an anonymous client; both read as "gated or private", and neither as a + /// generic status. + #[test] + fn a_forbidden_file_is_said_to_be_gated_or_private_like_an_unauthorized_one() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Forbidden, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let error = parse_gguf_header(&mut opened.reader) + .unwrap_err() + .to_string(); + assert!( + error.contains("gated or private") && !error.contains("answered 403"), + "{error}" + ); + } + + /// `open` sets one deadline for the listing and the reads that follow; the + /// reader it returns is bound by it, not by a fresh budget of its own. + #[test] + fn the_reader_open_returns_is_bound_by_the_budget_open_was_given() { + let slow = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Slow, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&slow.base) + .with_budget(Duration::from_millis(500)) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let began = Instant::now(); + assert!(parse_gguf_header(&mut opened.reader).is_err()); + assert!( + began.elapsed() < Duration::from_millis(1400), + "waited for the slow reply" + ); + } + + #[test] + fn a_variant_directory_may_be_at_most_eight_segments_deep() { + let deep = |segments: usize| { + let dirs = vec!["d"; segments - 1].join("/"); + format!("o/r:{dirs}/m-Q4_K_M.gguf") + }; + assert!(matches!( + parse_gguf_ref(&deep(8)), + Some(GgufRef::Hub { .. }) + )); + assert!(!matches!( + parse_gguf_ref(&deep(9)), + Some(GgufRef::Hub { .. }) + )); + } + + /// The hub's redirect to its CDN may be relative, absolute or + /// scheme-relative; none may carry the request from https to plain http. + #[test] + fn a_redirect_may_not_leave_https_for_plain_http() { + let from = "https://hub.example/o/r/resolve/main/m.gguf?download=1"; + let refused = |location: &str| { + redirect_target(from, location, "the model file") + .unwrap_err() + .to_string() + }; + assert!( + refused("http://cdn.example/x").contains("from https to plain http"), + "an absolute downgrade" + ); + assert!( + refused("HTTP://cdn.example/x").contains("from https to plain http"), + "the scheme is not case-sensitive" + ); + assert!(refused("ftp://cdn.example/x").contains("not http(s)")); + let allowed = |current: &str, location: &str| { + redirect_target(current, location, "the model file").unwrap() + }; + assert_eq!( + allowed(from, "https://cdn.example/x"), + "https://cdn.example/x" + ); + assert_eq!(allowed(from, "//cdn.example/x"), "https://cdn.example/x"); + assert_eq!( + allowed(from, "/api/x?y=http://z"), + "https://hub.example/api/x?y=http://z" + ); + assert_eq!( + allowed(from, "other.gguf"), + "https://hub.example/o/r/resolve/main/other.gguf" + ); + // A plain-http mirror may stay on http, and may move up to https. + assert_eq!( + allowed("http://127.0.0.1:9/a/b", "http://127.0.0.1:8/c"), + "http://127.0.0.1:8/c" + ); + assert_eq!( + allowed("http://127.0.0.1:9/a/b", "https://cdn.example/c"), + "https://cdn.example/c" + ); + } + + /// A server that redirects to itself is followed `MAX_REDIRECTS` times and + /// then refused, relative `Location` and all. + #[test] + fn a_redirect_loop_is_stopped_after_the_limit() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let hits = Arc::new(Mutex::new(0_usize)); + let counted = Arc::clone(&hits); + thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + if stream.read(&mut byte).unwrap_or(0) == 0 { + break; + } + request.push(byte[0]); + } + *counted.lock().unwrap() += 1; + let _ = write!( + stream, + "HTTP/1.1 302 Found\r\nLocation: /again\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ); + } + }); + let error = send( + &LiveGgufSource::agent(), + "GET", + &format!("{base}/start"), + Instant::now() + Duration::from_secs(20), + None, + "the model file", + ) + .err() + .unwrap() + .to_string(); + assert!(error.contains("more than 5 times"), "{error}"); + assert_eq!(*hits.lock().unwrap(), MAX_REDIRECTS + 1); + } + + #[test] + fn a_listing_without_a_size_falls_back_to_the_files_own_length() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Honour, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf"}]}"#.to_owned(), + ); + let opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + assert_eq!(opened.size, 1 << 20); + let requests = hub.requests.lock().unwrap().clone(); + assert!( + requests.iter().any(|request| request.starts_with("HEAD ")), + "{requests:?}" + ); + } + + #[test] + fn a_file_the_listing_gave_no_size_for_is_still_named_in_a_refusal() { + // A sibling the listing gave no size for is still a file the repository + // has, so the "it has: ..." list names it. + let files = [ + HubFile { + name: "sized-Q8_0.gguf".into(), + size: Some(1), + }, + HubFile { + name: "unsized-Q6_K.gguf".into(), + size: None, + }, + ]; + let error = select_gguf_file(&files, "Q2_K").unwrap_err().to_string(); + assert!( + error.contains("sized-Q8_0.gguf") && error.contains("unsized-Q6_K.gguf"), + "{error}" + ); + } + + #[test] + fn a_listing_beyond_the_cap_is_refused_not_cut_short() { + let cap = usize::try_from(MAX_LISTING_BYTES).unwrap(); + // ASCII padding, and two-byte characters placed so that the cut falls + // inside one: the size is what is reported either way, not bad text. + for padding in ["x".repeat(cap), "é".repeat(cap / 2 + 1)] { + let listing = format!(r#"{{"siblings":[],"pad":"{padding}"}}"#); + let hub = fake_hub_with(Vec::new(), 1, Serve::Honour, listing); + let error = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .err() + .unwrap() + .to_string(); + assert!(error.contains("is larger than 16 MiB"), "{error}"); + } + } + + /// A range answer that runs on past the bytes asked for is read only as far + /// as asked. The header spans two windows, so a window that kept the extra + /// bytes would hand the parser `0xFF` where the second window should begin. + #[test] + fn a_range_answer_longer_than_asked_is_read_only_as_far_as_asked() { + let header = llama_like(400_000).header_bytes(); + assert!( + header.len() as u64 > WINDOW_BYTES, + "the header must span two windows" + ); + let hub = fake_hub_with( + header.clone(), + 1 << 30, + Serve::Overlong, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1073741824}]}"#.to_owned(), + ); + let mut opened = LiveGgufSource::with_hub(&hub.base) + .open(&hub_ref("Q4_K_M")) + .unwrap(); + let parsed = parse_gguf_header(&mut opened.reader).unwrap(); + let direct = parse_gguf_header(&mut Cursor::new(header)).unwrap(); + assert_eq!(parsed.data_offset, direct.data_offset); + assert!(hub.ranges.lock().unwrap().len() >= 2); + } + + /// A listing that never ends is cut at the cap and refused as too large, + /// promptly: the read stops one byte past the cap rather than waiting for a + /// body that does not end. + #[test] + fn an_endless_listing_is_cut_at_the_cap() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + thread::spawn(move || { + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + if stream.read(&mut byte).unwrap_or(0) == 0 { + return; + } + request.push(byte[0]); + } + let _ = write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nConnection: \ + close\r\n\r\n{{\"siblings\":[],\"pad\":\"" + ); + // Twice the cap, then the connection is held open: a reader + // with no cap would wait here until its timeout. + let chunk = vec![b'x'; 1 << 20]; + for _ in 0..((2 * MAX_LISTING_BYTES) >> 20) { + if stream.write_all(&chunk).is_err() { + return; + } + } + thread::sleep(Duration::from_secs(90)); + }); + } + }); + let began = Instant::now(); + let error = LiveGgufSource::with_hub(&base) + .open(&hub_ref("Q4_K_M")) + .err() + .expect("an endless listing is refused") + .to_string(); + assert!(error.contains("is larger than 16 MiB"), "{error}"); + assert!( + began.elapsed() < Duration::from_secs(20), + "took {:?}", + began.elapsed() + ); + } + + #[test] + fn the_whole_read_has_a_deadline_not_only_each_request() { + let hub = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Honour, + r#"{"siblings":[{"rfilename":"m-Q4_K_M.gguf","size":1048576}]}"#.to_owned(), + ); + let url = format!("{}/o/r/resolve/main/m-Q4_K_M.gguf", hub.base); + let mut spent = RangedReader::new(ureq::AgentBuilder::new().build(), url, 1 << 20) + .with_deadline(Instant::now()); + let mut buf = [0_u8; 4]; + let error = spent.read(&mut buf).unwrap_err().to_string(); + assert!(error.contains("took longer than"), "{error}"); + assert!( + hub.requests.lock().unwrap().is_empty(), + "a request was made after the deadline" + ); + // A reply slower than what is left is cut off, not waited for. + let slow = fake_hub_with( + llama_like(10).header_bytes(), + 1 << 20, + Serve::Slow, + String::new(), + ); + let url = format!("{}/o/r/resolve/main/m-Q4_K_M.gguf", slow.base); + let mut reader = RangedReader::new(ureq::AgentBuilder::new().build(), url, 1 << 20) + .with_deadline(Instant::now() + Duration::from_millis(300)); + let began = Instant::now(); + assert!(reader.read(&mut buf).is_err()); + assert!( + began.elapsed() < Duration::from_millis(1400), + "waited for the slow reply" + ); + } + + #[test] + fn a_path_that_is_not_a_regular_file_is_refused_without_being_opened() { + let dir = std::env::temp_dir().join(format!("rocm-gguf-kind-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let source = LiveGgufSource::with_hub("http://127.0.0.1:1"); + let error = source + .open(&GgufRef::Local(dir.clone())) + .err() + .unwrap() + .to_string(); + assert!(error.contains("is not a regular file"), "{error}"); + // Opening a named pipe would block until something wrote to it. + #[cfg(unix)] + { + let fifo = dir.join("pipe.gguf"); + let made = std::process::Command::new("mkfifo").arg(&fifo).status(); + if made.is_ok_and(|status| status.success()) { + let error = source + .open(&GgufRef::Local(fifo)) + .err() + .unwrap() + .to_string(); + assert!(error.contains("is not a regular file"), "{error}"); + } + } + std::fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn path_segments_are_percent_encoded() { + let encode = crate::report_delivery::percent_encode; + assert_eq!(encode("m Q4+K.gguf"), "m%20Q4%2BK.gguf"); + assert_eq!(encode("a-b_c.d~e"), "a-b_c.d~e"); + } +} diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index c142c3332f..aad2dda627 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -27,6 +27,9 @@ pub mod examine; pub mod fake_executable; pub mod fix; mod gfx_target; +pub mod gguf; +pub mod gguf_fit; +pub mod gguf_source; pub mod hardware_root; pub mod host_gpu; mod jsonl_store; diff --git a/crates/rocm-core/src/model_readiness.rs b/crates/rocm-core/src/model_readiness.rs index ceeead4e71..27d4832dbe 100644 --- a/crates/rocm-core/src/model_readiness.rs +++ b/crates/rocm-core/src/model_readiness.rs @@ -8,10 +8,14 @@ //! after a failure. This answers the question that comes before both, from data //! the CLI already has: a curated [`ModelRecipeRecord`] (dtype, quantization, //! minimum GPU memory, device policy, preferred engines, declared alternatives) -//! composed with what the host can offer. **It fetches nothing** — no weights, -//! no metadata probe, no network call of any kind — which is the whole point: -//! today a user learns a model will not run by starting a download and waiting -//! for it to fail. +//! composed with what the host can offer. **It never fetches weights** — which +//! is the whole point: a user should not learn a model will not run by starting +//! a download and waiting for it to fail. The curated path reads nothing beyond +//! what the caller hands over. A GGUF reference *outside* the catalog is the one +//! exception, and a small one: [`attach_context_estimate`] reads the file's +//! header (for a remote file: one repository listing, then a few megabytes by +//! range request) and sizes the +//! model from it. See [`crate::gguf_fit`]. //! //! Three distinctions carry the design, and collapsing any of them produces a //! confident wrong answer: @@ -27,14 +31,22 @@ //! today. Conflating the two hands an APU user a remediation that can never //! work. See [`AcceleratorMemory::UnifiedMemoryUnreadable`]. //! -//! Curated recipes only. Answering for an arbitrary hub model needs a metadata -//! probe and a table mapping quantization schemes to available kernels, neither -//! of which has a source of truth here; such a model is reported -//! [`UndeterminedReason::ModelNotCurated`], never blocked. +//! A model outside the catalog has no recipe, so it is reported +//! [`UndeterminedReason::ModelNotCurated`], never blocked. When it is a GGUF the +//! header allows an *estimate* (weights, KV cache at a context size, compute +//! buffer) against dedicated VRAM, carried in [`ModelReadiness::estimate`] under +//! its own label rather than as a verdict: an estimate from a header is not a +//! recipe's tested minimum. use crate::ModelRecipeRecord; use crate::ModelRecipeRegistry; use crate::diagnose::{Fix, Route, upstream_route}; +use crate::gguf::parse_gguf_header; +use crate::gguf_fit::{ + CannotEstimateReason, ContextEstimate, ESTIMATE_CAVEAT, EstimateOutcome, FitRequest, + KvCacheType, SERVE_CONTEXT_NOTE, SERVE_OPTIONS_NOTE, estimate_fit, evidence_lines, tokens, +}; +use crate::gguf_source::{GgufRef, GgufSource, ProjectorNamed, parse_gguf_ref}; use serde::{Deserialize, Serialize}; /// How many fallback alternatives to offer when a recipe declares none that fit. @@ -113,7 +125,7 @@ pub enum AcceleratorMemory { impl AcceleratorMemory { /// The figure to compare a recipe minimum against, when there is one. - const fn measured_gib(self) -> Option { + pub(crate) const fn measured_gib(self) -> Option { match self { Self::Dedicated(gib) => Some(gib), Self::None | Self::Unknown | Self::UnifiedMemoryUnreadable => None, @@ -196,6 +208,14 @@ pub struct ModelReadiness { /// Informational — every curated recipe carries at least one — so they do /// not move the verdict. Degradation is decided from measurements. pub warnings: Vec, + /// A memory estimate from a GGUF file's header, for a model outside the + /// catalog. Absent for curated models and for references that are not GGUF. + /// + /// Deliberately not a [`ModelVerdict`]: `verdict` stays `undetermined`, and + /// this carries its own outcome, so no reader mistakes an estimate for a + /// recipe's tested minimum. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub estimate: Option, } impl ModelReadiness { @@ -222,6 +242,7 @@ impl ModelReadiness { route: None, undetermined_reason: Some(reason), warnings: Vec::new(), + estimate: None, } } } @@ -303,7 +324,7 @@ fn requires_gpu(recipe: &ModelRecipeRecord) -> bool { /// values come from `u32` catalog fields, so they are always exact integers /// and never reach this branch; flooring only ever shades a measured figure /// down, never a minimum. -fn format_gib(value: f64) -> String { +pub(crate) fn format_gib(value: f64) -> String { if value.fract().abs() < f64::EPSILON { format!("{value:.0} GiB") } else { @@ -392,8 +413,9 @@ fn assess( UndeterminedReason::ModelNotCurated, vec![ format!("`{model_ref}` is not in the curated recipe catalog"), - "`rocm diagnose --model` answers for curated recipes only: judging an arbitrary \ - model needs metadata this CLI does not fetch" + "there is no recipe for it, so no tested GPU memory minimum to compare; a GGUF \ + reference (a local .gguf path, or `owner/repo:QUANT` on Hugging Face) is \ + estimated from its file header instead" .to_owned(), ], ); @@ -408,8 +430,8 @@ fn assess( commands: vec!["rocm model".to_owned()], verify: format!("rocm serve {model_ref}"), notes: vec![ - "`rocm serve` accepts models outside this catalog; it just cannot say in advance \ - whether they will fit" + "`rocm serve` accepts models outside this catalog; for one that is not a GGUF \ + reference this cannot say in advance whether it will fit" .to_owned(), ], ..Fix::default() @@ -448,6 +470,7 @@ fn assess( route: None, undetermined_reason: None, warnings: recipe.warnings.clone(), + estimate: None, }; // Ordered by what pre-empts what. The engine gate comes first because a @@ -476,17 +499,7 @@ fn assess( fallback", recipe.device_policy )); - readiness.fix = Some(Fix { - summary: "make a GPU visible to ROCm, then ask again".to_owned(), - commands: vec!["rocm diagnose".to_owned()], - verify: format!("rocm diagnose --model {model_ref}"), - notes: vec![ - "`rocm diagnose` matches this machine against the known reasons a GPU does not \ - show up" - .to_owned(), - ], - ..Fix::default() - }); + readiness.fix = Some(no_gpu_fix(format!("rocm diagnose --model {model_ref}"))); return readiness; } @@ -522,17 +535,7 @@ fn assess( is a gap in what the CLI can see, not a verdict about the model", format_gib(required) )); - readiness.fix = Some(Fix { - summary: "let the CLI read this host's GPU memory, then ask again".to_owned(), - commands: vec!["amd-smi metric --json".to_owned()], - verify: format!("rocm diagnose --model {model_ref}"), - notes: vec![ - "the reading comes from `amd-smi`; if it is missing or failing, the ROCm \ - install is what needs attention first" - .to_owned(), - ], - ..Fix::default() - }); + readiness.fix = Some(read_gpu_memory_fix(model_ref)); } (Some(required), Some(available)) if available < required => { readiness.verdict = ModelVerdict::Blocked; @@ -675,6 +678,480 @@ fn alternatives_for( .collect() } +/// The context size an estimate is made for when none was asked for. +pub const DEFAULT_CONTEXT_SIZE: u64 = 4096; + +/// What the user asked of a context-size estimate. +/// +/// Each field is `None` when the flag was not passed, which is not the same as +/// passing the default: it decides whether a curated model, which has no +/// context estimate, should say so. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ContextRequest { + pub context_size: Option, + pub kv_cache_type: Option, +} + +/// Size a GGUF model outside the catalog from its file header. +/// +/// Acts only on an answer of [`UndeterminedReason::ModelNotCurated`] whose +/// reference names a GGUF. Every other answer keeps its verdict, plan and +/// alternatives; the only thing added to it is, when `--context-size` or +/// `--kv-cache-type` was passed, an evidence line saying why the flag did not +/// apply. The verdict stays `Undetermined`: what is added is [`ModelReadiness::estimate`] +/// and the evidence that goes with it. `source` is the only thing that reads a +/// file or the network, and it reads the header, never the weights. +/// +/// Three steps, each its own function: [`plan_context_estimate`] decides whether +/// there is a file to read (and, when there is not, what to say about flags that +/// were passed), [`read_context_estimate`] reads the header and sizes it, and +/// [`apply_context_estimate`] rewrites the readiness for what came back. +pub fn attach_context_estimate( + readiness: &mut ModelReadiness, + request: &ContextRequest, + host: &HostFacts, + source: &dyn GgufSource, +) { + match plan_context_estimate(readiness, request) { + ContextPlan::LeaveAsIs { note } => readiness.evidence.extend(note), + ContextPlan::Estimate { + reference, + fit_request, + } => { + let (estimate, refused_models) = + read_context_estimate(&reference, &readiness.model_ref, &fit_request, host, source); + apply_context_estimate( + readiness, + &reference, + estimate, + refused_models.as_deref(), + request, + ); + } + } +} + +/// What [`plan_context_estimate`] decided for an answer. +#[derive(Debug, PartialEq)] +enum ContextPlan { + /// Nothing is read and the answer keeps its shape. `note` is the evidence + /// line that says why a context flag the user passed was not applied; `None` + /// when no flag was passed, or for an answer that owes no explanation. + LeaveAsIs { note: Option }, + /// Read this GGUF's header and size it for this request. + Estimate { + reference: GgufRef, + fit_request: FitRequest, + }, +} + +/// Decide whether `readiness` is owed an estimate, without reading anything. +fn plan_context_estimate(readiness: &ModelReadiness, request: &ContextRequest) -> ContextPlan { + let asked = request.context_size.is_some() || request.kv_cache_type.is_some(); + let leave = |note: String| ContextPlan::LeaveAsIs { + note: asked.then_some(note), + }; + if readiness.canonical_model_id.is_some() { + return leave( + "a context size and KV cache type do not apply to a curated recipe, whose GPU memory \ + minimum is fixed; estimates for them cover GGUF models outside the catalog" + .to_owned(), + ); + } + if readiness.undetermined_reason == Some(UndeterminedReason::CatalogUnreachable) { + // Whether the reference names a curated model is exactly what could not be + // found out, so there is no telling which answer it is owed: say that the + // flags were not applied rather than dropping them without a word. + return leave( + "the context size and KV cache type were not applied: the recipe catalog could \ + not be read, so it is not known whether this is a curated model or a GGUF \ + outside the catalog" + .to_owned(), + ); + } + // Every answer that reaches here today is a catalog miss: the other + // undetermined answers carry the recipe they were about, and are left above. + // This keeps a future recipe-less answer from being estimated by accident. + if readiness.undetermined_reason != Some(UndeterminedReason::ModelNotCurated) { + return ContextPlan::LeaveAsIs { note: None }; + } + let Some(reference) = parse_gguf_ref(&readiness.model_ref) else { + return leave( + "a context size and KV cache type are estimated for GGUF models only: a local \ + .gguf path, or `owner/repo:QUANT` on Hugging Face" + .to_owned(), + ); + }; + // `owner/repo` with no quantization names a repository, not a file, and a + // repository is not read (every file in it is a different model). Without a + // context flag the answer is the plain "not in the catalog" it always was, + // reached without touching the network; with one, the user is told what + // would make it answerable. + if matches!(reference, GgufRef::Hub { variant: None, .. }) { + return leave(format!( + "`{}` names a repository, not a model file: add `:QUANT` (for example \ + `:Q4_K_M`) to estimate one of its GGUF files", + readiness.model_ref + )); + } + ContextPlan::Estimate { + reference, + fit_request: FitRequest { + context_size: request.context_size.unwrap_or(DEFAULT_CONTEXT_SIZE), + context_size_defaulted: request.context_size.is_none(), + kv_cache_type: request.kv_cache_type.unwrap_or(KvCacheType::F16), + }, + } +} + +/// Read `reference`'s header through `source` and size it against `host`. A +/// header that cannot be reached or read is an estimate with no figures, named +/// after `model_ref` as the user wrote it. When the name was refused as a +/// multimodal projector, the repository's model files come back as well, for +/// the plan to offer one. +fn read_context_estimate( + reference: &GgufRef, + model_ref: &str, + fit_request: &FitRequest, + host: &HostFacts, + source: &dyn GgufSource, +) -> (ContextEstimate, Option>) { + let read = source.open(reference).and_then(|mut opened| { + let header = + parse_gguf_header(&mut opened.reader).map_err(|error| anyhow::anyhow!("{error}"))?; + Ok((header, opened.size, opened.description)) + }); + match read { + Ok((header, size, description)) => ( + estimate_fit( + &header, + size, + &description, + fit_request, + host.accelerator_memory, + host.system_ram_gib, + ), + None, + ), + Err(error) => { + let refused_models = error + .downcast_ref::() + .map(|refusal| refusal.models.clone()); + ( + ContextEstimate::unreadable(model_ref, fit_request, &error), + refused_models, + ) + } + } +} + +/// Put `estimate` on `readiness`, and rewrite the parts of the answer that were +/// written for a model nobody had looked at: the route, the fix, the +/// alternatives and the warnings, each according to what the estimate found. +fn apply_context_estimate( + readiness: &mut ModelReadiness, + reference: &GgufRef, + estimate: ContextEstimate, + refused_models: Option<&[String]>, + request: &ContextRequest, +) { + readiness.evidence.extend(evidence_lines(&estimate)); + let ask_again = |model_ref: &str| rediagnose(model_ref, request); + // The route sends a user to the upstream tracker because the catalog lacks + // the model. A GGUF was looked at directly, so that is no longer the + // question -- and for a path that does not exist or a hub that did not + // answer, the tracker is the wrong place to take it. + readiness.route = None; + if estimate.total_gib.is_some() { + // The header was read and sized, so the "serve it directly and read the + // engine's refusal" plan answered a question that no longer applies -- + // whether or not this host could be compared against. A header that could + // not be read gets a plan of its own below (check the path, or the name + // and the network), as does a hub name refused as a projector; one that + // was read but could not be sized keeps the "serve it" plan, pointed at a + // command that accepts the reference. + // The host gaps with a next step get the plan a curated model gets: + // memory that could not be read is looked into with `amd-smi`, and no + // visible GPU with `rocm diagnose`. An APU's unreadable pool has no + // command that changes the answer, and gets none. + readiness.fix = match estimate.cannot_estimate_reason { + Some(CannotEstimateReason::AcceleratorMemoryUnknown) => { + let mut fix = read_gpu_memory_fix(&readiness.model_ref); + fix.verify = ask_again(&readiness.model_ref); + Some(fix) + } + Some(CannotEstimateReason::NoGpu) => Some(no_gpu_fix(ask_again(&readiness.model_ref))), + // A layout whose figure can be too low (state-space, hybrid, + // mixture-of-experts, or a header missing a width) was sized and + // compared, and its detail tells the reader to compare the total + // with the GPU's memory and leave headroom. No command would turn it + // into a verdict, so there is no plan to offer. + Some( + CannotEstimateReason::UnmeasuredLayout | CannotEstimateReason::MissingMetadata, + ) => None, + // An APU's pool cannot be read and no command changes that; a fit or + // a misfit is the answer itself, with its suggestions. + _ => None, + }; + if estimate.outcome == EstimateOutcome::Fits { + readiness.alternatives.clear(); + } + readiness.warnings.push(ESTIMATE_CAVEAT.to_owned()); + } else if estimate.cannot_estimate_reason == Some(CannotEstimateReason::HeaderUnreadable) + && matches!(reference, GgufRef::Local(_)) + { + // A path that could not be read is a path to check. "Serve it directly and + // read the engine's refusal" is advice for a model nobody has looked at, + // and `rocm serve ` on a file that is not there only repeats this. + // A URL is read as a path, and the "could not open" it gets does not say + // why; the form that is read is `owner/repo:QUANT`. + let lower = readiness.model_ref.to_ascii_lowercase(); + let notes = if lower.starts_with("http://") || lower.starts_with("https://") { + vec![ + "a URL is not read: name a Hugging Face file as `owner/repo:QUANT` (or \ + `owner/repo:file.gguf`), or download it and pass the local path" + .to_owned(), + ] + } else { + Vec::new() + }; + readiness.fix = Some(Fix { + summary: "check the path, and that the file is a GGUF model, then ask again".to_owned(), + verify: ask_again(&readiness.model_ref), + notes, + ..Fix::default() + }); + } else if estimate.cannot_estimate_reason == Some(CannotEstimateReason::HeaderUnreadable) + && matches!(reference, GgufRef::Hub { .. }) + { + // The header could not be fetched or read: a name that does not exist, a + // gated or private repository, no network, a busy hub, or a file that is + // not a GGUF. The estimate's detail says which. Serving it would start a + // download of the whole file to learn the same thing, so the plan is to + // act on that detail and ask again. + readiness.fix = Some(Fix { + summary: "act on the reason above (a misspelt name or quantization, no route to \ + Hugging Face, a busy hub to wait for, or a file that is not a GGUF), \ + then ask again" + .to_owned(), + verify: ask_again(&readiness.model_ref), + notes: vec![ + "a gated or private repository cannot be read, because no token is sent: \ + download the file yourself and ask about its local path" + .to_owned(), + ], + ..Fix::default() + }); + } else if estimate.cannot_estimate_reason == Some(CannotEstimateReason::NotAModel) { + readiness.fix = Some(projector_fix(reference, refused_models, request)); + } else if let Some(fix) = &mut readiness.fix { + point_fix_at_unsized_gguf(fix, reference, &ask_again(&readiness.model_ref)); + } + // A compared estimate is sized for a context `rocm serve` would not pass, + // so the answer says so, once: with the options a suggestion names, or on + // its own. + if matches!( + estimate.outcome, + EstimateOutcome::Fits | EstimateOutcome::DoesNotFit + ) { + let note = if estimate.suggestions.is_empty() { + SERVE_CONTEXT_NOTE + } else { + SERVE_OPTIONS_NOTE + }; + readiness.warnings.push(note.to_owned()); + } + readiness.estimate = Some(estimate); +} + +/// The command that asks the same GGUF question again: the same reference, +/// quoted for this platform's shell when it needs to be, and the same +/// `--context-size` and `--kv-cache-type` when they were passed, so following +/// the plan does not quietly answer a different question. +fn rediagnose(model_ref: &str, request: &ContextRequest) -> String { + rediagnose_for(model_ref, request, cfg!(windows)) +} + +/// [`rediagnose`] for a chosen platform, so both are tested on either. +/// +/// A reference made only of characters no shell treats specially is printed +/// as it is, so a plain Windows path (`C:\models\m.gguf`) still works in both +/// `cmd.exe` and PowerShell; anything else is quoted for PowerShell on Windows +/// and for a POSIX shell elsewhere. A reference that starts with `-` is joined +/// to the flag (`--model=-x.gguf`), because the argument parser would read it +/// as another flag. +fn rediagnose_for(model_ref: &str, request: &ContextRequest, windows: bool) -> String { + use std::fmt::Write as _; + let quoted = shell_word(model_ref, windows); + let mut command = if model_ref.starts_with('-') { + format!("rocm diagnose --model={quoted}") + } else { + format!("rocm diagnose --model {quoted}") + }; + if let Some(size) = request.context_size { + let _ = write!(command, " --context-size {size}"); + } + if let Some(kind) = request.kv_cache_type { + let _ = write!(command, " --kv-cache-type {kind}"); + } + command +} + +/// `model_ref` as one word of a command for this platform's shell: as it is +/// when no shell would split or expand it, quoted for PowerShell on Windows and +/// for a POSIX shell elsewhere otherwise. +fn shell_word(model_ref: &str, windows: bool) -> String { + // `~` is plain after the first character, where neither shell expands it; + // Windows short names (`ADMINI~1`) carry one in the middle of a path. + let plain = !model_ref.is_empty() + && !model_ref.starts_with('~') + && model_ref.chars().all(|c| { + c.is_ascii_alphanumeric() + || matches!(c, '/' | ':' | '.' | '_' | '-' | '=' | '~') + || (windows && c == '\\') + }); + if plain { + model_ref.to_owned() + } else if windows { + crate::quoting::powershell_quote(model_ref) + } else { + crate::quoting::posix_shell_quote(model_ref) + } +} + +/// The plan when no GPU is visible to ROCm: find out why, then ask again with +/// `verify`. Shared by a curated model that needs a GPU and a GGUF estimate. +fn no_gpu_fix(verify: String) -> Fix { + Fix { + summary: "make a GPU visible to ROCm, then ask again".to_owned(), + commands: vec!["rocm diagnose".to_owned()], + verify, + notes: vec![ + "`rocm diagnose` matches this machine against the known reasons a GPU does not show \ + up" + .to_owned(), + ], + ..Fix::default() + } +} + +/// The plan when this host's GPU memory could not be read: look at what +/// `amd-smi` reports, then ask again. Shared by a curated model and a GGUF +/// estimate, which hit the same gap. +fn read_gpu_memory_fix(model_ref: &str) -> Fix { + Fix { + summary: "let the CLI read this host's GPU memory, then ask again".to_owned(), + commands: vec!["amd-smi metric --json".to_owned()], + verify: format!("rocm diagnose --model {model_ref}"), + notes: vec![ + "the reading comes from `amd-smi`; if it is missing or failing, the ROCm install is \ + what needs attention first" + .to_owned(), + ], + ..Fix::default() + } +} + +/// The plan for a name refused as a multimodal projector. "Serve it and read +/// the engine's refusal" would point back at the file just refused, so the plan +/// is to ask about the model instead: by its exact file name when the +/// repository holds one model, and with no command when it holds several (the +/// estimate's detail lists them) or none. +fn projector_fix(reference: &GgufRef, models: Option<&[String]>, request: &ContextRequest) -> Fix { + let summary = "name the model's own GGUF file or quantization, not its multimodal projector, \ + then ask again" + .to_owned(); + // The model's name comes from the hub's listing, not from the user, so it is + // offered only when it reads back as that same hub file: a name the parser + // would take for a local path, or that a shell would expand, is not printed + // as a command. + let verify = match (reference, models) { + (GgufRef::Hub { owner, repo, .. }, Some([only])) => { + let candidate = format!("{owner}/{repo}:{only}"); + let reads_back = parse_gguf_ref(&candidate) + == Some(GgufRef::Hub { + owner: owner.clone(), + repo: repo.clone(), + variant: Some(only.clone()), + }); + if reads_back { + rediagnose(&candidate, request) + } else { + String::new() + } + } + _ => String::new(), + }; + Fix { + summary, + verify, + ..Fix::default() + } +} + +/// Rewrite the "serve it and see" plan for a GGUF reference whose header was +/// reached for but not sized. +fn point_fix_at_unsized_gguf(fix: &mut Fix, reference: &GgufRef, verify: &str) { + // The plan was written for a model that is not a GGUF reference, and its + // note says so. This one is, and the estimate says why it could not be + // sized, so the note says that instead of contradicting it. + fix.notes = vec![ + "`rocm serve` accepts models outside this catalog; the estimate above could not size \ + this GGUF, so this cannot say in advance whether it will fit" + .to_owned(), + ]; + // `rocm serve` takes `owner/repo:QUANT`, whose label has no `/`; the + // `owner/repo:dir/file.gguf` form selects an exact file for this estimate + // only, and serve would not read it as a checkpoint. Naming it in a + // `rocm serve` command would hand the user a line that fails. + if matches!(reference, GgufRef::Hub { variant: Some(variant), .. } if variant.contains('/')) { + verify.clone_into(&mut fix.verify); + fix.notes.push( + "`rocm serve` does not accept an exact file name after the colon; it takes \ + `owner/repo:QUANT`" + .to_owned(), + ); + } + // `rocm serve` does take a local file, and serves it as it is; the path is + // quoted for the shell. One that starts with `-` would be read as a flag, + // so that one is asked about again instead. + if let GgufRef::Local(path) = reference { + let path = path.to_string_lossy(); + if path.starts_with('-') { + verify.clone_into(&mut fix.verify); + } else { + fix.verify = format!("rocm serve {}", shell_word(&path, cfg!(windows))); + } + } +} + +/// What an estimate says when a device-visibility variable is set on a host +/// whose GPUs differ in memory. +pub const MASK_UNVERIFIED_NOTE: &str = "a device-visibility variable (HIP_VISIBLE_DEVICES, \ + ROCR_VISIBLE_DEVICES or CUDA_VISIBLE_DEVICES) is set and this host's GPUs do not all \ + have the same memory; how the mask's numbering maps onto amd-smi's has not been verified \ + on such a host, so the memory compared with may be that of a different card (or the host \ + may be mistaken for an integrated GPU)"; + +/// Add [`MASK_UNVERIFIED_NOTE`] to a compared estimate when `mask_set` and the +/// GPUs' totals (`gpu_totals_mb`, one per amd-smi row) are not all equal. +/// +/// Which row a mask selects is not something this binary has verified on a +/// multi-GPU machine, so the visible rows are not guessed at: if any two cards +/// differ, whichever one the mask picked might not be the one compared. An +/// estimate that compared nothing has no figure to doubt and is left alone. +pub fn note_unverified_mask(readiness: &mut ModelReadiness, mask_set: bool, gpu_totals_mb: &[u64]) { + let differ = gpu_totals_mb.windows(2).any(|pair| pair[0] != pair[1]); + if !(mask_set && differ) { + return; + } + if let Some(estimate) = readiness.estimate.as_mut() + && estimate.available_gib.is_some() + { + estimate.notes.push(MASK_UNVERIFIED_NOTE.to_owned()); + } +} + /// The word this verdict is reported under. const fn verdict_label(verdict: ModelVerdict) -> &'static str { match verdict { @@ -685,6 +1162,23 @@ const fn verdict_label(verdict: ModelVerdict) -> &'static str { } } +/// The word the report opens with. An estimate that compared its figures with +/// the GPU is labelled as one, never as a verdict; every other answer -- an +/// estimate that could not compare included -- keeps the verdict's own word. +fn header_label(readiness: &ModelReadiness) -> String { + match &readiness.estimate { + Some(estimate) if estimate.outcome == EstimateOutcome::Fits => format!( + "ESTIMATED: fits at {} (calibrated on one MI300X, not a tested minimum)", + tokens(estimate.context_size) + ), + Some(estimate) if estimate.outcome == EstimateOutcome::DoesNotFit => format!( + "ESTIMATED: does not fit at {} (calibrated on one MI300X, not a tested minimum)", + tokens(estimate.context_size) + ), + _ => verdict_label(readiness.verdict).to_owned(), + } +} + /// Render the human-facing model verdict. /// /// Deliberately laid out like `diagnose::render_report_text`: the same ` - ` @@ -700,7 +1194,7 @@ pub fn render_model_readiness_text(readiness: &ModelReadiness) -> String { out, "rocm diagnose --model {}: {}", readiness.model_ref, - verdict_label(readiness.verdict) + header_label(readiness) ); if let Some(canonical) = &readiness.canonical_model_id { let _ = writeln!(out, " recipe: {canonical}"); @@ -708,6 +1202,14 @@ pub fn render_model_readiness_text(readiness: &ModelReadiness) -> String { for line in &readiness.evidence { let _ = writeln!(out, " - {line}"); } + if let Some(estimate) = &readiness.estimate + && !estimate.suggestions.is_empty() + { + let _ = writeln!(out, " what would make it fit:"); + for suggestion in &estimate.suggestions { + let _ = writeln!(out, " - {}", suggestion.text); + } + } if !readiness.alternatives.is_empty() { let _ = writeln!(out, " what would run here instead:"); for alternative in &readiness.alternatives { @@ -738,9 +1240,20 @@ pub fn render_model_readiness_text(readiness: &ModelReadiness) -> String { let _ = writeln!(out, " report it: {:>12}: {}", route.target, route.url); } // The recipe's own warnings come last and carry no verdict weight. Putting - // them above the plan would read as reasons for it. + // them above the plan would read as reasons for it. The estimate's notes + // are about the figures, not a recipe, and are labelled as such. + if let Some(estimate) = &readiness.estimate { + for note in &estimate.notes { + let _ = writeln!(out, " note: {note}"); + } + } for warning in &readiness.warnings { - let _ = writeln!(out, " recipe note: {warning}"); + let label = if readiness.estimate.is_some() { + "note" + } else { + "recipe note" + }; + let _ = writeln!(out, " {label}: {warning}"); } out } @@ -1215,4 +1728,1134 @@ mod tests { fn format_gib_renders_a_whole_number_with_no_decimal() { assert_eq!(format_gib(8.0), "8 GiB"); } + + // ----- GGUF estimate (models outside the catalog) ----------------------- + + use crate::gguf::test_builder::llama_like; + use crate::gguf_fit::SuggestionKind; + use crate::gguf_source::{GgufRef, OpenedGguf}; + use std::io::Cursor; + use std::sync::Mutex; + + /// Serves one in-memory GGUF for every reference and records what it was + /// asked to open, so a test can show the source was (or was not) reached. + struct FakeSource { + file: Option>, + /// The size the source reports for the file: the header plus the + /// weights it pretends to carry, so no test allocates a real model. + size: u64, + opened: Mutex>, + /// When set, opening refuses the name as a multimodal projector, as + /// file selection does, listing these model files. + refusal: Option>, + } + + impl FakeSource { + fn with(file: Vec) -> Self { + Self { + size: file.len() as u64, + file: Some(file), + opened: Mutex::new(Vec::new()), + refusal: None, + } + } + + /// A llama-shaped file whose weights are `weights_mib` MiB: 8192 + /// tokens of f16 cache is exactly 1 GiB for it. + fn weights(weights_mib: u64) -> Self { + let header = llama_like(10).file(0); + Self { + size: header.len() as u64 + weights_mib * 1024 * 1024, + file: Some(header), + opened: Mutex::new(Vec::new()), + refusal: None, + } + } + + fn failing() -> Self { + Self { + file: None, + size: 0, + opened: Mutex::new(Vec::new()), + refusal: None, + } + } + + /// A hub that refuses the name as a projector, with `models` beside it. + fn refusing_a_projector(models: &[&str]) -> Self { + Self { + refusal: Some(models.iter().map(|&model| model.to_owned()).collect()), + ..Self::failing() + } + } + + fn calls(&self) -> usize { + self.opened.lock().unwrap().len() + } + } + + impl GgufSource for FakeSource { + fn open(&self, reference: &GgufRef) -> anyhow::Result { + self.opened.lock().unwrap().push(reference.clone()); + if let (Some(models), GgufRef::Hub { variant, .. }) = (&self.refusal, reference) { + return Err(anyhow::Error::new(ProjectorNamed { + variant: variant.clone().unwrap_or_default(), + projectors: vec![variant.clone().unwrap_or_default()], + models: models.clone(), + })); + } + let Some(file) = &self.file else { + anyhow::bail!("the hub could not be reached"); + }; + Ok(OpenedGguf { + size: self.size, + reader: Box::new(Cursor::new(file.clone())), + description: "fake.gguf".to_owned(), + }) + } + } + + fn not_curated(model_ref: &str) -> ModelReadiness { + let registry = builtin_model_recipe_registry(); + assess_model_readiness( + model_ref, + &ModelCatalogSource::Available(®istry), + &host_with(24.0, 64.0), + &engine_choice, + ) + } + + fn asked(context: u64) -> ContextRequest { + ContextRequest { + context_size: Some(context), + kv_cache_type: None, + } + } + + /// The decide step on its own: a GGUF outside the catalog is planned for a + /// read with the defaults filled in, and an answer that is not one is left as + /// it is, with an explanation only when a context flag was passed. + #[test] + fn the_plan_reads_only_a_gguf_outside_the_catalog_and_fills_in_the_defaults() { + let gguf = not_curated("some-lab/never-curated-7b:Q4_K_M"); + assert_eq!( + plan_context_estimate(&gguf, &ContextRequest::default()), + ContextPlan::Estimate { + reference: GgufRef::Hub { + owner: "some-lab".to_owned(), + repo: "never-curated-7b".to_owned(), + variant: Some("Q4_K_M".to_owned()), + }, + fit_request: FitRequest { + // 4096, written out: `--help` and the README state it + // independently of the constant. + context_size: 4096, + context_size_defaulted: true, + kv_cache_type: KvCacheType::F16, + }, + } + ); + let not_gguf = not_curated("some-lab/never-curated-7b"); + assert_eq!( + plan_context_estimate(¬_gguf, &ContextRequest::default()), + ContextPlan::LeaveAsIs { note: None } + ); + assert!(matches!( + plan_context_estimate(¬_gguf, &asked(8192)), + ContextPlan::LeaveAsIs { note: Some(note) } if note.contains("add `:QUANT`") + )); + // Flags that were passed are carried as given, not replaced by defaults. + let both = ContextRequest { + context_size: Some(32768), + kv_cache_type: Some(KvCacheType::Q8_0), + }; + assert!(matches!( + plan_context_estimate(&gguf, &both), + ContextPlan::Estimate { fit_request, .. } if fit_request == FitRequest { + context_size: 32768, + context_size_defaulted: false, + kv_cache_type: KvCacheType::Q8_0, + } + )); + // A curated model is never read, and is told why only when it asked. + let curated = not_curated(SMALL_MODEL_REF); + assert_eq!( + plan_context_estimate(&curated, &ContextRequest::default()), + ContextPlan::LeaveAsIs { note: None } + ); + assert!(matches!( + plan_context_estimate(&curated, &both), + ContextPlan::LeaveAsIs { note: Some(note) } if note.contains("curated recipe") + )); + } + + /// A name refused as a projector is `not_a_model`, and the plan points at + /// the model rather than at serving the projector it refused: by its exact + /// name when there is one model, and with no command to run otherwise. + #[test] + fn a_refused_projector_is_not_offered_back_as_a_serve_command() { + let refused = |models: &[&str]| { + let mut readiness = not_curated("some-lab/vision-7b:mmproj-vision-7b-f16.gguf"); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::refusing_a_projector(models), + ); + readiness + }; + let one = refused(&["vision-7b-Q4_K_M.gguf"]); + let estimate = one.estimate.as_ref().expect("an estimate"); + assert_eq!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::NotAModel) + ); + assert!( + estimate.detail.as_deref().is_some_and(|detail| detail + .starts_with("`mmproj-vision-7b-f16.gguf` is a multimodal projector")), + "{:?}", + estimate.detail + ); + let fix = one.fix.as_ref().expect("a plan for the user"); + assert!( + fix.summary.contains("not its multimodal projector"), + "{fix:?}" + ); + assert_eq!( + fix.verify, + "rocm diagnose --model some-lab/vision-7b:vision-7b-Q4_K_M.gguf" + ); + // Asked with a context size, the command for the model asks with it too. + let mut with_flags = not_curated("some-lab/vision-7b:mmproj-vision-7b-f16.gguf"); + attach_context_estimate( + &mut with_flags, + &ContextRequest { + context_size: Some(16_384), + kv_cache_type: Some(KvCacheType::Q4_0), + }, + &host_with(24.0, 64.0), + &FakeSource::refusing_a_projector(&["vision-7b-Q4_K_M.gguf"]), + ); + assert_eq!( + with_flags.fix.as_ref().expect("a plan").verify, + "rocm diagnose --model some-lab/vision-7b:vision-7b-Q4_K_M.gguf --context-size \ + 16384 --kv-cache-type q4_0" + ); + let text = render_model_readiness_text(&one); + assert!(!text.contains("header could not be read"), "{text}"); + // No header was read, so the report does not say the figures came from one. + assert!(!text.contains("estimated from the GGUF header"), "{text}"); + assert!(!text.contains("rocm serve"), "{text}"); + // A listed name the parser would not read back as that hub file (a shell + // expansion, a space) is not printed as a command. + for unsafe_name in ["x$(id).gguf", "my model.gguf"] { + let odd = refused(&[unsafe_name]); + assert_eq!( + odd.fix.as_ref().map(|fix| fix.verify.as_str()), + Some(""), + "{unsafe_name}" + ); + } + // Several models: no single command to name, so none is given. + let several = refused(&["vision-7b-Q4_K_M.gguf", "vision-7b-Q8_0.gguf"]); + assert_eq!( + several.fix.as_ref().map(|fix| fix.verify.as_str()), + Some("") + ); + // A projector-like name that failed for another reason (an unreachable + // hub) is not blamed on the name: it keeps the unreadable answer. + let mut unreachable = not_curated("some-lab/vision-7b:mmproj-vision-7b-f16.gguf"); + attach_context_estimate( + &mut unreachable, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + assert_eq!( + unreachable + .estimate + .and_then(|estimate| estimate.cannot_estimate_reason), + Some(CannotEstimateReason::HeaderUnreadable) + ); + assert!( + unreachable + .fix + .is_some_and(|fix| !fix.summary.contains("multimodal projector")), + ); + } + + /// A sized estimate that could not be compared keeps a next step only where + /// one exists, the one a curated model gets: memory that could not be read + /// is looked into with `amd-smi`, no visible GPU with `rocm diagnose`; an + /// APU has no command that would change the answer. + #[test] + fn a_sized_estimate_that_could_not_compare_keeps_a_plan_only_when_one_exists() { + for (memory, expected) in [ + (AcceleratorMemory::Unknown, Some("amd-smi metric --json")), + (AcceleratorMemory::None, Some("rocm diagnose")), + (AcceleratorMemory::UnifiedMemoryUnreadable, None), + ] { + let host = HostFacts { + accelerator_memory: memory, + system_ram_gib: Some(64.0), + }; + let registry = builtin_model_recipe_registry(); + let mut readiness = assess_model_readiness( + "some-lab/never-curated-7b:Q4_K_M", + &ModelCatalogSource::Available(®istry), + &host, + &engine_choice, + ); + attach_context_estimate( + &mut readiness, + &ContextRequest { + context_size: Some(32_768), + kv_cache_type: None, + }, + &host, + &FakeSource::weights(64), + ); + let estimate = readiness.estimate.as_ref().expect("an estimate"); + assert!( + estimate.total_gib.is_some(), + "{memory:?}: the header was sized" + ); + let commands = readiness + .fix + .as_ref() + .map(|fix| fix.commands.clone()) + .unwrap_or_default(); + match expected { + Some(command) => { + assert_eq!(commands, vec![command.to_owned()], "{memory:?}"); + assert_eq!( + readiness.fix.as_ref().map(|fix| fix.verify.as_str()), + Some( + "rocm diagnose --model some-lab/never-curated-7b:Q4_K_M \ + --context-size 32768" + ), + "{memory:?}" + ); + } + None => assert!(readiness.fix.is_none(), "{memory:?}: {:?}", readiness.fix), + } + } + } + + #[test] + fn a_gguf_outside_the_catalog_gets_an_estimate_and_stays_undetermined() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + // The catalog-miss answer offers curated models that would run here; an + // estimate that says the model fits has no use for them. + assert!( + !readiness.alternatives.is_empty(), + "the catalog-miss answer carries curated alternatives to clear" + ); + let source = FakeSource::weights(2048); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &source, + ); + let estimate = readiness.estimate.as_ref().expect("no estimate attached"); + assert_eq!(estimate.outcome, EstimateOutcome::Fits); + assert!( + readiness.alternatives.is_empty(), + "ESTIMATED: fits offered alternatives: {:?}", + readiness.alternatives + ); + assert!(!render_model_readiness_text(&readiness).contains("what would run here instead"),); + assert_eq!(estimate.context_size, 8192); + // The closed-set verdict is not widened: an estimate is not a recipe. + assert_eq!(readiness.verdict, ModelVerdict::Undetermined); + assert_eq!( + readiness.undetermined_reason, + Some(UndeterminedReason::ModelNotCurated) + ); + assert!( + readiness.canonical_model_id.is_none() && readiness.required_gpu_memory_gib.is_none() + ); + assert!(readiness.fix.is_none() && readiness.route.is_none()); + assert_eq!(source.calls(), 1); + let text = render_model_readiness_text(&readiness); + assert!( + text.starts_with("rocm diagnose --model some-lab/never-curated-7b:Q4_K_M: ESTIMATED: fits at 8192 tokens"), + "{text}" + ); + assert!( + !text.contains("READY") && !text.contains("BLOCKED"), + "{text}" + ); + // Labelled `note:`, as a figure's caveat, not `recipe note:`: there is no + // recipe behind an estimate. + assert!( + text.lines() + .any(|line| line.starts_with(" note: these are estimates")), + "{text}" + ); + assert!(!text.contains("recipe note"), "{text}"); + } + + #[test] + fn a_gguf_that_does_not_fit_says_what_would_make_it_fit() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + // 2 GiB of weights, 1 GiB of cache at 8192, ~1.4 GiB for the compute + // buffer and runtime: 4.4 GiB against 4, with room for ~5000 tokens. + let host = host_with(4.0, 64.0); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host, + &FakeSource::weights(2048), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!(estimate.outcome, EstimateOutcome::DoesNotFit); + // A model that does not fit keeps the curated models that would run here: + // they answer the question the user is left with. + assert!( + !readiness.alternatives.is_empty(), + "ESTIMATED: does not fit dropped the alternatives" + ); + let kinds: Vec<_> = estimate.suggestions.iter().map(|s| s.kind).collect(); + assert!(kinds.contains(&SuggestionKind::SmallerContext), "{kinds:?}"); + assert!(kinds.contains(&SuggestionKind::KvCacheType), "{kinds:?}"); + let text = render_model_readiness_text(&readiness); + assert!( + text.contains("ESTIMATED: does not fit at 8192 tokens"), + "{text}" + ); + assert!(text.contains("what would make it fit:"), "{text}"); + // Nothing printed as a command: `rocm serve` takes none of these options, + // and a runnable-looking line for them would be false. + assert!(!text.contains(" $ "), "{text}"); + assert!(!text.contains("rocm serve some-lab"), "{text}"); + assert!(!text.contains("rocm serve --"), "{text}"); + assert!( + text.contains("`rocm serve` does not take a context size"), + "{text}" + ); + } + + #[test] + fn a_bare_repository_is_answered_as_before_and_read_nowhere() { + // `owner/repo` with no quantization names a repository, not a file. With + // no context flag the answer is exactly the plain "not in the catalog" + // one, reached without touching the network or the disk. + let plain = not_curated("some-lab/never-curated-7b"); + let mut readiness = plain.clone(); + let source = FakeSource::weights(64); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &source, + ); + assert_eq!(readiness, plain); + assert_eq!(source.calls(), 0); + assert!(readiness.estimate.is_none() && readiness.route.is_some()); + // With a flag, the user is told what would make it answerable -- still + // without reading anything. + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &source, + ); + assert_eq!(source.calls(), 0); + assert!(readiness.estimate.is_none()); + let text = render_model_readiness_text(&readiness); + assert!( + text.contains("names a repository, not a model file: add `:QUANT`"), + "{text}" + ); + } + + /// The command a GGUF plan offers asks the same question again: the flags + /// that were passed come with it, and a path a shell would split is quoted. + #[test] + fn the_command_to_ask_again_keeps_the_flags_and_quotes_the_path() { + let mut readiness = not_curated("my models/nosuch.gguf"); + attach_context_estimate( + &mut readiness, + &ContextRequest { + context_size: Some(32_768), + kv_cache_type: Some(KvCacheType::Q8_0), + }, + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + assert_eq!( + readiness.fix.as_ref().expect("a plan").verify, + "rocm diagnose --model 'my models/nosuch.gguf' --context-size 32768 \ + --kv-cache-type q8_0" + ); + let plain = ContextRequest::default(); + assert_eq!( + rediagnose("o/r:Q4_K_M", &plain), + "rocm diagnose --model o/r:Q4_K_M" + ); + assert_eq!( + rediagnose_for("it's here.gguf", &plain, false), + r"rocm diagnose --model 'it'\''s here.gguf'" + ); + // Windows: a plain path is left as it is, so cmd.exe reads it too; + // one with a space is quoted for PowerShell, which doubles a `'`. + assert_eq!( + rediagnose_for(r"C:\models\m.gguf", &plain, true), + r"rocm diagnose --model C:\models\m.gguf" + ); + assert_eq!( + rediagnose_for(r"C:\my models\it's.gguf", &plain, true), + r"rocm diagnose --model 'C:\my models\it''s.gguf'" + ); + // A Windows short name keeps a path plain on either shell; a leading + // `~` is quoted, since a shell would expand it. + assert_eq!( + rediagnose_for(r"C:\Users\ADMINI~1\Temp\m.gguf", &plain, true), + r"rocm diagnose --model C:\Users\ADMINI~1\Temp\m.gguf" + ); + assert_eq!( + rediagnose_for("~/m.gguf", &plain, false), + "rocm diagnose --model '~/m.gguf'" + ); + // A backslash is not plain to a POSIX shell. + assert_eq!( + rediagnose_for(r"a\b.gguf", &plain, false), + r"rocm diagnose --model 'a\b.gguf'" + ); + // A reference that starts with `-` is joined to the flag. + assert_eq!( + rediagnose_for("-x.gguf", &plain, false), + "rocm diagnose --model=-x.gguf" + ); + assert_eq!( + rediagnose("$HOME/m.gguf", &plain), + "rocm diagnose --model '$HOME/m.gguf'" + ); + } + + /// A local file that was read but cannot be sized (here one shard of a + /// split model) is offered to `rocm serve`, which takes a local path, with + /// the path quoted for the shell where it needs to be. + #[test] + fn an_unsized_local_gguf_is_offered_to_rocm_serve_quoted() { + let mut readiness = not_curated("models/split.gguf"); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &FakeSource::with(llama_like(10).u32("split.count", 3).file(0)), + ); + assert_eq!( + readiness.estimate.as_ref().unwrap().cannot_estimate_reason, + Some(CannotEstimateReason::SplitFile) + ); + let fix = readiness.fix.as_ref().expect("a plan"); + assert_eq!(fix.verify, "rocm serve models/split.gguf"); + let mut spaced = not_curated("my models/split.gguf"); + attach_context_estimate( + &mut spaced, + &asked(8192), + &host_with(24.0, 64.0), + &FakeSource::with(llama_like(10).u32("split.count", 3).file(0)), + ); + assert_eq!( + spaced.fix.as_ref().expect("a plan").verify, + "rocm serve 'my models/split.gguf'" + ); + } + + /// A context flag with a reference that is neither curated nor a GGUF says + /// the estimate is for GGUF models only, and reads nothing. + #[test] + fn a_context_flag_on_a_reference_that_is_not_a_gguf_says_why_it_was_not_applied() { + let mut readiness = not_curated("some-bare-model-name"); + let source = FakeSource::weights(64); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &source, + ); + assert_eq!(source.calls(), 0); + assert!(readiness.estimate.is_none()); + assert!( + readiness + .evidence + .iter() + .any(|line| line.contains("estimated for GGUF models only")), + "{:?}", + readiness.evidence + ); + } + + /// A hub header that could not be fetched is not answered with `rocm serve`, + /// which would download the whole file to fail the same way: the plan is to + /// check the name and the network, and ask again. + #[test] + fn an_unreadable_hub_header_is_not_answered_with_a_download() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + let fix = readiness.fix.as_ref().expect("a plan"); + assert!(fix.summary.contains("no route to Hugging Face"), "{fix:?}"); + assert_eq!( + fix.verify, + "rocm diagnose --model some-lab/never-curated-7b:Q4_K_M --context-size 8192" + ); + assert!( + fix.notes.iter().any(|note| note.contains("local path")), + "{fix:?}" + ); + assert!(!render_model_readiness_text(&readiness).contains("rocm serve some-lab")); + } + + /// A pasted URL is read as a path and cannot be opened; the plan says which + /// form is read instead. + #[test] + fn a_url_is_told_which_form_is_read() { + let mut readiness = not_curated("https://huggingface.co/o/r/resolve/main/m.gguf"); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + let fix = readiness.fix.as_ref().expect("a plan"); + assert!( + fix.notes + .iter() + .any(|note| note.contains("`owner/repo:QUANT`")), + "{fix:?}" + ); + } + + /// A layout that gets figures and no verdict is sized and compared, so the + /// "serve it and see" plan and the tracker route no longer apply, and no + /// command would turn it into a verdict: there is no plan, and the detail + /// says to compare the total with the GPU's memory. + #[test] + fn a_layout_without_a_verdict_gets_its_figures_and_no_plan() { + for (layout, source) in [ + ( + "mixture-of-experts", + FakeSource::with(llama_like(10).u32("llama.expert_count", 8).file(64)), + ), + ( + "state-space", + FakeSource::with(llama_like(10).u32("llama.ssm.state_size", 16).file(64)), + ), + ] { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(4096), + &host_with(24.0, 64.0), + &source, + ); + let estimate = readiness.estimate.as_ref().expect("an estimate"); + assert_eq!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::UnmeasuredLayout), + "{layout}" + ); + assert!(estimate.total_gib.is_some(), "{layout}"); + assert!(readiness.fix.is_none(), "{layout}: {:?}", readiness.fix); + assert!(readiness.route.is_none(), "{layout}"); + let text = render_model_readiness_text(&readiness); + assert!( + text.contains("compare it with this GPU's"), + "{layout}: {text}" + ); + } + } + + #[test] + fn a_gguf_that_cannot_be_read_is_not_sent_to_the_upstream_tracker() { + // A typo'd path: the tracker cannot help with it, and the plan is to + // check the path. + let mut readiness = not_curated("models/nosuch.gguf"); + assert!( + readiness.route.is_some(), + "the catalog-miss route is the default" + ); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::HeaderUnreadable) + ); + assert!(readiness.route.is_none()); + // A typo'd local path is a path to check, not a model to "serve directly". + let fix = readiness.fix.as_ref().expect("a plan is still given"); + assert!(fix.summary.contains("check the path"), "{}", fix.summary); + assert!(fix.commands.is_empty()); + assert_eq!(fix.verify, "rocm diagnose --model models/nosuch.gguf"); + let text = render_model_readiness_text(&readiness); + assert!(!text.contains("report it:"), "{text}"); + assert!( + !text.contains("serve this model directly") && !text.contains("rocm serve"), + "{text}" + ); + } + + fn compared_estimate() -> ModelReadiness { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(24.0, 64.0), + &FakeSource::weights(2048), + ); + assert!(readiness.estimate.as_ref().unwrap().available_gib.is_some()); + readiness + } + + #[test] + fn a_mask_on_a_host_whose_gpus_differ_is_said_to_be_unverified() { + let notes = |readiness: &ModelReadiness| readiness.estimate.as_ref().unwrap().notes.clone(); + // Mask set, cards of two sizes: the figure may be for the other card. + let mut readiness = compared_estimate(); + note_unverified_mask(&mut readiness, true, &[24_576, 8192]); + assert!( + notes(&readiness).iter().any(|n| n == MASK_UNVERIFIED_NOTE), + "{:?}", + notes(&readiness) + ); + let text = render_model_readiness_text(&readiness); + assert!(text.contains("has not been verified"), "{text}"); + // Whichever order the rows come in. + let mut readiness = compared_estimate(); + note_unverified_mask(&mut readiness, true, &[8192, 24_576]); + assert!(notes(&readiness).iter().any(|n| n == MASK_UNVERIFIED_NOTE)); + // No mask: nothing to doubt. Same cards: whichever is picked, the figure is the same. + for (mask_set, totals) in [ + (false, vec![24_576_u64, 8192]), + (true, vec![24_576, 24_576]), + (true, vec![24_576]), + (true, vec![]), + ] { + let mut readiness = compared_estimate(); + note_unverified_mask(&mut readiness, mask_set, &totals); + assert!( + !notes(&readiness).iter().any(|n| n == MASK_UNVERIFIED_NOTE), + "{mask_set} {totals:?}" + ); + } + // An estimate that compared nothing has no figure to doubt. + let mut unmeasured = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut unmeasured, + &asked(8192), + &HostFacts { + accelerator_memory: AcceleratorMemory::Unknown, + system_ram_gib: Some(64.0), + }, + &FakeSource::weights(2048), + ); + note_unverified_mask(&mut unmeasured, true, &[24_576, 8192]); + assert!( + !notes(&unmeasured).iter().any(|n| n == MASK_UNVERIFIED_NOTE), + "{:?}", + notes(&unmeasured) + ); + } + + /// `owner/repo:dir/file.gguf` selects an exact file for the estimate; `rocm + /// serve` reads only `owner/repo:QUANT`, so a verify line naming it would fail. + /// Shown on a file that was read but could not be sized (one shard of a + /// split model), where serving it is still the plan. + #[test] + fn the_verify_command_is_one_rocm_serve_accepts() { + let verify = |model_ref: &str| { + let mut readiness = not_curated(model_ref); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::with(llama_like(10).u32("split.count", 3).file(64)), + ); + readiness + .fix + .expect("the plan stays for a file that could not be sized") + .verify + }; + assert_eq!( + verify("some-lab/never-curated-7b:Q4_K_M"), + "rocm serve some-lab/never-curated-7b:Q4_K_M" + ); + assert_eq!( + verify("some-lab/never-curated-7b:sub/model-Q4_K_M.gguf"), + "rocm diagnose --model some-lab/never-curated-7b:sub/model-Q4_K_M.gguf" + ); + } + + #[test] + fn the_default_context_is_labelled_as_a_default() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::weights(64), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!(estimate.context_size, 4096, "the default --help documents"); + assert!(estimate.context_size_defaulted); + assert_eq!(estimate.kv_cache_type, KvCacheType::F16); + let text = render_model_readiness_text(&readiness); + assert!( + text.contains("(the default; pass --context-size to change it)"), + "{text}" + ); + } + + /// The three states of host memory, for a model that would otherwise fit: + /// none produces a comparison, each says its own reason, and the figures + /// the file allows are still shown. + #[test] + fn host_memory_that_cannot_be_compared_never_yields_a_fits_or_blocked_answer() { + for (memory, reason) in [ + (AcceleratorMemory::None, CannotEstimateReason::NoGpu), + ( + AcceleratorMemory::Unknown, + CannotEstimateReason::AcceleratorMemoryUnknown, + ), + ( + AcceleratorMemory::UnifiedMemoryUnreadable, + CannotEstimateReason::UnifiedMemoryUnreadable, + ), + ] { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + let host = HostFacts { + accelerator_memory: memory, + system_ram_gib: Some(64.0), + }; + attach_context_estimate( + &mut readiness, + &asked(4096), + &host, + &FakeSource::weights(64), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!( + estimate.outcome, + EstimateOutcome::CannotEstimate, + "{memory:?}" + ); + assert_eq!(estimate.cannot_estimate_reason, Some(reason), "{memory:?}"); + assert!( + estimate.total_gib.is_some(), + "{memory:?}: the need was not shown" + ); + assert_eq!(readiness.verdict, ModelVerdict::Undetermined); + // Sized, so the "serve it directly" plan no longer applies; any plan + // left is the host's own (see the test after this one). + assert!(readiness.route.is_none(), "{memory:?}"); + assert!( + readiness + .fix + .as_ref() + .is_none_or(|fix| !fix.summary.contains("serve this model directly")), + "{memory:?}: {:?}", + readiness.fix + ); + let text = render_model_readiness_text(&readiness); + assert!(text.contains(": UNDETERMINED"), "{memory:?}: {text}"); + assert!(!text.contains("ESTIMATED:"), "{memory:?}: {text}"); + assert!(text.contains("KV cache (f16)"), "{memory:?}: {text}"); + } + } + + #[test] + fn an_unreachable_or_unreadable_file_is_a_gap_in_the_estimate_not_a_verdict_on_the_model() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(4096), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::HeaderUnreadable) + ); + assert!( + estimate + .detail + .as_deref() + .unwrap() + .contains("could not be reached") + ); + assert_eq!(estimate.total_gib, None); + // The plan is to act on that reason and ask again, not to serve a file + // whose header could not even be read. + let fix = readiness.fix.as_ref().expect("a plan"); + assert!(fix.summary.contains("act on the reason above"), "{fix:?}"); + assert!(fix.verify.starts_with("rocm diagnose --model "), "{fix:?}"); + + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(4096), + &host_with(24.0, 64.0), + &FakeSource::with(b"not a gguf file at all".to_vec()), + ); + let estimate = readiness.estimate.as_ref().unwrap(); + assert_eq!( + estimate.cannot_estimate_reason, + Some(CannotEstimateReason::HeaderUnreadable) + ); + assert!( + estimate + .detail + .as_deref() + .unwrap() + .contains("not a GGUF file") + ); + } + + /// The plan's note was written for a model that is not a GGUF reference. For + /// one that is, and whose header could not be sized, it must not say so. + #[test] + fn the_plan_note_never_claims_a_gguf_is_not_one() { + let note_of = |source: &FakeSource| { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate(&mut readiness, &asked(4096), &host_with(24.0, 64.0), source); + let text = render_model_readiness_text(&readiness); + assert!(!text.contains("not a GGUF reference"), "{text}"); + readiness.fix.expect("the plan stays").notes.join("\n") + }; + // Unreadable (its own plan, about the name and the network), and read + // but not sizeable (a shard of a split model). + let unreadable = note_of(&FakeSource::failing()); + assert!(unreadable.contains("no token is sent"), "{unreadable}"); + let shard = note_of(&FakeSource::with( + llama_like(10).u32("split.count", 3).file(64), + )); + assert!(shard.contains("could not size this GGUF"), "{shard}"); + // Sized: there is no plan left to carry a note. + let mut sized = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut sized, + &asked(4096), + &host_with(24.0, 64.0), + &FakeSource::weights(64), + ); + assert!(sized.fix.is_none()); + // Without a context flag nothing is read and the original note stands: + // that reference is not asked about as a GGUF at all. + let mut plain = not_curated("lab/plain-name"); + attach_context_estimate( + &mut plain, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &FakeSource::failing(), + ); + assert!(plain.estimate.is_none()); + assert!( + plain + .fix + .unwrap() + .notes + .join("\n") + .contains("for one that is not a GGUF reference") + ); + } + + #[test] + fn flags_are_never_dropped_in_silence_when_the_catalog_could_not_be_read() { + let unreachable = || { + assess_model_readiness( + "/models/local.gguf", + &ModelCatalogSource::Unreachable { + detail: "no index".to_owned(), + }, + &host_with(24.0, 64.0), + &engine_choice, + ) + }; + let source = FakeSource::weights(64); + let mut readiness = unreachable(); + let before = readiness.evidence.clone(); + attach_context_estimate( + &mut readiness, + &asked(4096), + &host_with(24.0, 64.0), + &source, + ); + assert!(readiness.estimate.is_none()); + assert_eq!(source.calls(), 0, "which answer it is owed is not known"); + assert_eq!(readiness.evidence.len(), before.len() + 1); + assert!( + readiness + .evidence + .last() + .unwrap() + .contains("were not applied"), + "{:?}", + readiness.evidence + ); + assert_eq!( + readiness.undetermined_reason, + Some(UndeterminedReason::CatalogUnreachable) + ); + // No flag, no line: the answer is exactly what it was. + let mut readiness = unreachable(); + let before = readiness.clone(); + attach_context_estimate( + &mut readiness, + &ContextRequest::default(), + &host_with(24.0, 64.0), + &source, + ); + assert_eq!(readiness, before); + } + + /// A compared estimate says once that `rocm serve` would not use the size it + /// was sized for: with the suggestions' options when it does not fit, on its + /// own when it fits; an estimate that compared nothing says neither. + #[test] + fn a_compared_estimate_says_rocm_serve_passes_no_context() { + let warnings = |memory_gib: f64| { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(8192), + &host_with(memory_gib, 64.0), + &FakeSource::weights(2048), + ); + readiness.warnings + }; + let fits = warnings(24.0); + assert!(fits.iter().any(|w| w == SERVE_CONTEXT_NOTE), "{fits:?}"); + assert!(!fits.iter().any(|w| w == SERVE_OPTIONS_NOTE), "{fits:?}"); + let misfit = warnings(2.0); + assert!(misfit.iter().any(|w| w == SERVE_OPTIONS_NOTE), "{misfit:?}"); + assert!( + !misfit.iter().any(|w| w == SERVE_CONTEXT_NOTE), + "{misfit:?}" + ); + let mut uncompared = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut uncompared, + &asked(8192), + &HostFacts { + accelerator_memory: AcceleratorMemory::Unknown, + system_ram_gib: None, + }, + &FakeSource::weights(2048), + ); + assert!( + !uncompared + .warnings + .iter() + .any(|w| w == SERVE_CONTEXT_NOTE || w == SERVE_OPTIONS_NOTE) + ); + } + + #[test] + fn a_one_token_context_reads_as_one_token() { + let mut readiness = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut readiness, + &asked(1), + &host_with(24.0, 64.0), + &FakeSource::weights(64), + ); + let text = render_model_readiness_text(&readiness); + assert!( + text.contains(": ESTIMATED: fits at 1 token (calibrated"), + "{text}" + ); + assert!(!text.contains("1 tokens"), "{text}"); + } + + #[test] + fn only_a_model_outside_the_catalog_that_names_a_gguf_reaches_the_source() { + let source = FakeSource::weights(64); + let host = host_with(24.0, 64.0); + // A curated model: no network, no estimate; and an explicit context + // flag is answered with why it does not apply, not silently ignored. + let mut curated = not_curated(SMALL_MODEL_REF); + attach_context_estimate(&mut curated, &asked(8192), &host, &source); + assert!(curated.estimate.is_none()); + assert!( + curated + .evidence + .iter() + .any(|line| line.contains("do not apply to a curated recipe")), + "{:?}", + curated.evidence + ); + let mut curated = not_curated(SMALL_MODEL_REF); + let before = curated.evidence.clone(); + attach_context_estimate(&mut curated, &ContextRequest::default(), &host, &source); + assert_eq!( + curated.evidence, before, + "an unasked curated answer must not change" + ); + // A name that is not a GGUF reference. + let mut other = not_curated("never-curated"); + let before = other.evidence.len(); + attach_context_estimate(&mut other, &ContextRequest::default(), &host, &source); + assert!(other.estimate.is_none() && other.evidence.len() == before); + attach_context_estimate(&mut other, &asked(4096), &host, &source); + assert!(other.estimate.is_none()); + assert!( + other + .evidence + .iter() + .any(|line| line.contains("GGUF models only")) + ); + // A reference of the right shape, but the catalog could not be read: + // the answer is about the catalog, and an estimate would contradict it. + let mut unreachable = assess_model_readiness( + "some-lab/never-curated-7b:Q4_K_M", + &ModelCatalogSource::Unreachable { + detail: "gone".to_owned(), + }, + &host, + &engine_choice, + ); + attach_context_estimate(&mut unreachable, &asked(4096), &host, &source); + assert!(unreachable.estimate.is_none()); + assert_eq!( + source.calls(), + 0, + "the source was reached for a model that did not qualify" + ); + } + + #[test] + fn an_answer_without_an_estimate_serializes_exactly_as_it_did_before() { + let readiness = not_curated("some-lab/never-curated-7b"); + let json = serde_json::to_value(&readiness).unwrap(); + assert!(json.get("estimate").is_none(), "{json}"); + // And a JSON answer from before the field existed still reads. + let round_trip: ModelReadiness = serde_json::from_value(json).unwrap(); + assert_eq!(round_trip, readiness); + let mut with = not_curated("some-lab/never-curated-7b:Q4_K_M"); + attach_context_estimate( + &mut with, + &asked(4096), + &host_with(24.0, 64.0), + &FakeSource::weights(64), + ); + let json = serde_json::to_value(&with).unwrap(); + assert_eq!(json["estimate"]["outcome"], "fits"); + assert_eq!(json["estimate"]["kv_cache_type"], "f16"); + assert_eq!(json["verdict"], "undetermined"); + } } diff --git a/crates/rocm-core/src/report_delivery.rs b/crates/rocm-core/src/report_delivery.rs index c835fd3f34..5980def262 100644 --- a/crates/rocm-core/src/report_delivery.rs +++ b/crates/rocm-core/src/report_delivery.rs @@ -171,13 +171,15 @@ fn choose(url: String, sending: bool, env: &dyn Fn(&str) -> Option) -> D } } -/// Percent-encode for a query-string value. +/// Percent-encode for a query-string value or one URL path segment. /// /// Written out rather than taken from a crate: this is the only encoding this /// binary needs, and a signed artifact is not worth a dependency for fifteen /// lines. Everything outside the unreserved set of RFC 3986 is escaped, which -/// is stricter than necessary and wrong in no case. -fn percent_encode(value: &str) -> String { +/// is stricter than necessary and wrong in no case; that also makes it right +/// for a path segment, where `/` must be escaped too. The GGUF header reader +/// sends hub file names through it, one segment at a time. +pub(crate) fn percent_encode(value: &str) -> String { const HEX: &[u8; 16] = b"0123456789ABCDEF"; let mut out = String::with_capacity(value.len()); for byte in value.bytes() { diff --git a/docs/architecture.md b/docs/architecture.md index 9508db8415..ad51f2aaf6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -53,6 +53,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`dash.rs`](../apps/rocm/src/dash.rs) — full domain extraction. - [`dash_seam.rs`](../apps/rocm/src/dash_seam.rs) — full domain extraction. - [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: the command handlers for `rocm diagnose` and `rocm fix` (`diagnose()`, `fix()`) and the report step behind `rocm diagnose --report` (`show_prepared_report`, `perform_delivery`, `established_entry`). `diagnose()` and `fix()` are `pub(crate)` and re-imported in [`main.rs`](../apps/rocm/src/main.rs) for `dispatch()`; the other three functions are private. `diagnose()` gets its `--model` verdict from [`model_fit.rs`](../apps/rocm/src/model_fit.rs), and `fix()` returns a failing exit code through `FixExitCode`, which stays in `main.rs` with the other exit-code plumbing. The report and delivery regression tests live in this module's own `mod tests`; the exit-code tests stay in `main.rs`. +- [`diagnose_model.rs`](../apps/rocm/src/diagnose_model.rs) — full domain extraction: the binary's side of `rocm diagnose --model`'s GGUF context-size estimate — the `--kv-cache-type` value enum, the model question the flags ask (`ModelQuery`), the note `--report` prints when it ignores them, and `attach_gguf_estimate`, which adds the estimate and the device-visibility-mask caveat over `amd-smi`'s rows to a readiness answer. Reached by qualified path. The estimate itself is in [`model_readiness.rs`](../crates/rocm-core/src/model_readiness.rs) and the `gguf*` modules; the `Diagnose` variant stays in [`main.rs`](../apps/rocm/src/main.rs), `diagnose()` is in [`diagnose_cmd.rs`](../apps/rocm/src/diagnose_cmd.rs), and the curated-model host assessment it predates (`assess_model_on_this_host` and its helpers) is in [`model_fit.rs`](../apps/rocm/src/model_fit.rs). - [`driver_install.rs`](../apps/rocm/src/driver_install.rs) — mechanical relocation, owns types (part of [`main.rs`](../apps/rocm/src/main.rs)'s Phase 6a split, ROCMAI-82; see the Phase 6a variant note above): `install_driver()`/`reconcile_driver_install()`, running an approved plan's shell commands, and the install state persisted to `state.json` (`DriverInstallState`, `DriverReconciliationState` and the passive-check types). The plan it runs lives in [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs), which this module imports from and which does not import from it; the os-release the plan is built from comes from `real_os_release` in [`hardware_root.rs`](../crates/rocm-core/src/hardware_root.rs), the one reader of the real host's os-release. Its `pub(crate)` surface is `install_driver` and `reconcile_driver_install`, `DriverInstallResult` and `DriverInstallError` (read from `install_cmd.rs`'s `install()`), `driver_install_state_path`, and a re-export of `host_install_driver_only_reports` for [`chat.rs`](../apps/rocm/src/chat.rs) and [`freeform_planner.rs`](../apps/rocm/src/freeform_planner.rs). - [`driver_plan.rs`](../apps/rocm/src/driver_plan.rs) — full domain extraction, crate-level (split out of [`driver_install.rs`](../apps/rocm/src/driver_install.rs)): the plan `rocm install driver` builds and prints. It owns the model (`DriverInstallPlan` with `execution_commands`, `DriverPlanCommand`, `DriverCommandPhase`, and `PrivilegeEscalation`, which decides whether a generated shell line gets a `sudo ` prefix), `render_driver_install_plan` with its approval label, the helpers the builders share (`driver_command`, `driver_root_preflight_checks`, `resolve_shell_default_template`, `is_safe_shell_token`), and the builders: `build_driver_install_plan_read`, which `install_driver` calls with the real host's os-release, and `build_driver_install_plan_at` beneath it, with the apt, dnf and SLES plans, the `ID_LIKE` fallback, the repository URL helpers, `resolve_repo_version_expr` (the `ROCM_CLI_AMDGPU_DRIVER_VERSION` override and the retired version variables it refuses), and the refusal plans for an unreadable os-release or an invalid override. It also owns `install_driver_only_reports_on`/`host_install_driver_only_reports` (true on native Windows, where the plan only reports); `driver_install.rs` re-exports the latter. It declares the child module [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs) and re-exports its `wsl_rocdxg_driver_plan`. Production code in this file uses only `std` and `rocm_core`, and nothing here depends on `driver_install.rs`. The model types and their fields, `execution_commands`, `PrivilegeEscalation::detect`, `driver_command`, `render_driver_install_plan`, `build_driver_install_plan_read`, `install_driver_only_reports_on` and `host_install_driver_only_reports` are `pub(crate)`, as is `invalid_driver_version_override_plan`, which a test that stays in `driver_install.rs` builds a plan with; the rest is private. Its test module is `pub(crate)` so `driver_install.rs`'s tests can reuse its `test_examine` and `os_release_found` helpers. It holds the plan-builder tests, including the one that keeps the supported-distro matrix in step with `dkms_driver_plan_is_supported` in `rocm-core`, and the tests of the template resolver and the shell-token filter. - [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs) — full domain extraction, a child of `driver_plan.rs` that depends on the plan model, `driver_command`, `driver_root_preflight_checks` and `resolve_shell_default_template` in its parent, and on `truthy_env` in [`therock.rs`](../apps/rocm/src/therock.rs) (split out of [`driver_install.rs`](../apps/rocm/src/driver_install.rs)): the WSL2 rocDXG plan. `wsl_rocdxg_driver_plan` builds the plan that installs the `rocdxg-roct` package, `resolve_rocdxg_verification` decides how that download is authenticated (a supplied digest, the digest pinned for the version in `ROCDXG_PINNED_DIGESTS`, or the explicit opt-out), and `wsl_rocdxg_refusal_plan` is the plan that installs nothing and says why when it cannot be authenticated or the version is not a well-formed package version. `ROCDXG_PINNED_DIGESTS` is the trust anchor an unattended root install is checked against, so a change to it is a security-relevant diff; it moved byte for byte. `wsl_rocdxg_driver_plan` is `pub(crate)` and re-exported by `driver_plan.rs` (its builders call it, and tests in `driver_install.rs` use it as a plan fixture); everything else in production code is private. Its tests are the `wsl_rocdxg_*` plan tests, including the one that runs the generated digest step against a real file on Unix. The test-only `scoped_rocdxg_env` helper that clears the rocDXG environment lives here too, `pub(super)` for the plan-builder tests in `driver_plan.rs`. @@ -183,12 +184,15 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`endpoint_readiness.rs`](../crates/rocm-core/src/endpoint_readiness.rs) — how far along a managed service's OpenAI-compatible endpoint is: `managed_service_endpoint_readiness` and `EndpointReadiness` (unreachable, listing the model, or serving a real single-token chat request), the lower-level `openai_chat_completion_probe`/`openai_chat_completion_status`/`openai_models_endpoint_has_model`, the shared inference-verified latch the engine adapters call (`engine_state_inference_verified`, `merge_json_state_file`) and the model-reference matching behind the model-listing check. It depends on [`net.rs`](../crates/rocm-core/src/net.rs), [`service_record.rs`](../crates/rocm-core/src/service_record.rs) and [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs); [`net.rs`](../crates/rocm-core/src/net.rs) and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) do not depend on it. The module is private and [`lib.rs`](../crates/rocm-core/src/lib.rs) re-exports its public names, so callers keep using `rocm_core::` paths. - [`examine.rs`](../crates/rocm-core/src/examine.rs) — the host examination probe: the `Examination` document and its types, and `Examination::probe`, which runs the probes in order. The probes live in child modules under `examine/`, with `examine.rs` kept as the module root so the [`examine_driver_proptests.rs`](../crates/rocm-core/src/examine_driver_proptests.rs) `#[path]` attribute and the links to it stay valid: [`linux_gpus.rs`](../crates/rocm-core/src/examine/linux_gpus.rs) (Linux GPU enumeration from `lspci`, `rocminfo` and the kernel topology, with the marketing-name and APU classification they share), [`linux_driver.rs`](../crates/rocm-core/src/examine/linux_driver.rs) (kernel modules, device-node access and Secure Boot), [`rocm_install_probe.rs`](../crates/rocm-core/src/examine/rocm_install_probe.rs) (how ROCm was installed and which repository files configure it), [`library_copies.rs`](../crates/rocm-core/src/examine/library_copies.rs) (the loader library-copy search for comgr and the HIP runtime, and `comgr_matches_runtime`, which `diagnose/engine.rs` reads), [`framework.rs`](../crates/rocm-core/src/examine/framework.rs) (the PyTorch and llama.cpp probes), [`host_env.rs`](../crates/rocm-core/src/examine/host_env.rs) (the OS and distro, CPU, tracked environment variables, shared memory, container check and kernel log), [`wsl.rs`](../crates/rocm-core/src/examine/wsl.rs) (the WSL probe, including the remote probe that reads a named distribution from the Windows host, and the distro-floor check) and [`windows.rs`](../crates/rocm-core/src/examine/windows.rs) (the native Windows probes: the GPU query, the HIP SDK, the Adrenalin driver and the MSVC redistributable, plus `parse_dotted_version`, which `diagnose/windows.rs` reads). - [`gfx_target.rs`](../crates/rocm-core/src/gfx_target.rs) — gfx-target identification and the TheRock package-family tables, split out of [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) so [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) can read the family mapping without importing the module that probes the host: `extract_first_gfx_token`, `normalize_therock_family`/`known_therock_families`, `gfx_target_has_no_rocm_build`, the AMD PCI-device-id and marketing-name tables behind `gfx_target_from_pnp_device_id`, and the parsers for the Windows display text and the kernel's gfx-target version. The module is private; `lib.rs` re-exports four of the public names (all but `gfx_target_from_pnp_device_id`) and `host_gpu.rs`, as before, re-exports all five. +- [`gguf.rs`](../crates/rocm-core/src/gguf.rs) — reads the shape of a GGUF model (layers, KV heads, head width, parameter count, data offset) from its header over any `Read + Seek`, never touching the weights. +- [`gguf_fit.rs`](../crates/rocm-core/src/gguf_fit.rs) — pure arithmetic over that header: weights + KV cache at a context size and KV type + compute buffer + runtime reserve against dedicated VRAM, the largest context that fits, and what would make a model that does not fit, fit. +- [`gguf_source.rs`](../crates/rocm-core/src/gguf_source.rs) — where a header comes from: a local file, or a Hugging Face file, found by one repository-listing request (and a `HEAD` request for a file the listing does not size) and read by HTTP range requests (`HF_ENDPOINT` selects a mirror; redirects are followed by hand, up to five, and never from https to plain http). The only code in the estimate that reads a file or the network. - [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) — (ROCMAI-85, Phase 7a of ROCMAI-27's sequencing) GPU/host hardware detection — see the module's own doc comment for what it owns. Its leaf concerns are child modules under `host_gpu/`, with `host_gpu.rs` kept as the module root: [`capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs) (running an optional tool under a timeout and the tool-on-`PATH` lookup), [`os_facts.rs`](../crates/rocm-core/src/host_gpu/os_facts.rs) (kernel version, distro name, CPU model and installed RAM), [`linux.rs`](../crates/rocm-core/src/host_gpu/linux.rs) (the sysfs, KFD and DRM probes), [`gpu_visibility.rs`](../crates/rocm-core/src/host_gpu/gpu_visibility.rs) (which AMD GPUs this process may use, from the KFD/DRM counts and the `HIP_VISIBLE_DEVICES`/`ROCR_VISIBLE_DEVICES` masks, plus `windows_amd_gpu_listing`, the native-Windows adapter bound `rocm serve --gpu ` is checked against), [`windows_inventory.rs`](../crates/rocm-core/src/host_gpu/windows_inventory.rs) (the Windows display-adapter inventory and its parsers, including the AMD-adapter count behind `windows_amd_gpu_listing`), [`wsl.rs`](../crates/rocm-core/src/host_gpu/wsl.rs) (WSL detection, the librocdxg chain and the Windows host driver as seen from the guest), [`render.rs`](../crates/rocm-core/src/host_gpu/render.rs) (the `rocm examine` text of an `ExamineSummary`, as a second `impl` block), [`diagnostics.rs`](../crates/rocm-core/src/host_gpu/diagnostics.rs) (the GPU detection diagnostics text) and [`engine_default.rs`](../crates/rocm-core/src/host_gpu/engine_default.rs) (the default serve engine, and the vLLM preference for the TheRock families that serve through it). - [`host-probing.md`](host-probing.md) — a design note, not a module: [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) and [`examine.rs`](../crates/rocm-core/src/examine.rs) are two separate host-probing stacks that still read some of the same facts twice (kernel version, CPU model, the Windows GPU query, the marketing-name table, the `PATH` lookup and the subprocess runners). It proposes which of each pair should survive, how the two differ today and the order in which they could converge; it changes no code. It is a snapshot, to be retired when the convergence tickets close. - [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) — the JSONL store engine behind the automation stores: the sibling `.lock` file (`jsonl_lock_path`, re-exported at the crate root), single-line appends, torn-tail repair, size-based rotation (`JsonlRotation`) and the tail-first readers that span rotated generations. It is generic over the record type; [`automation.rs`](../crates/rocm-core/src/automation.rs) supplies the record types and each store's policy, and nothing in this module depends on it, in production or in its own tests: those drive the engine with a small record type of their own. The module is private; its engine items are `pub(crate)`. - [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) — (ROCMAI-85, Phase 7a) the TheRock managed-runtime-environment surface — see the module's own doc comment for what it owns. - [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs) — (ROCMAI-152, Phase 7c) the curated model catalog's types, built-in catalog JSON loading, signed external recipe indexes, and artifact cache-marker bookkeeping — see the module's own doc comment for what it owns. -- [`model_readiness.rs`](../crates/rocm-core/src/model_readiness.rs) — the `rocm diagnose --model` fit assessment: curated-recipe lookup against a host's measured GPU/RAM. `rocm model --verbose` renders its own fit labels in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), which uses exactly two items from this module: `curated_alternatives`, called directly, and `recipe_display_ref`, reached through a private one-line wrapper of the same name in that file. +- [`model_readiness.rs`](../crates/rocm-core/src/model_readiness.rs) — the `rocm diagnose --model` fit assessment: curated-recipe lookup against a host's measured GPU/RAM. `rocm model --verbose` renders its own fit labels in [`model_catalog_text.rs`](../apps/rocm/src/model_catalog_text.rs), which uses exactly two items from this module: `curated_alternatives`, called directly, and `recipe_display_ref`, reached through a private one-line wrapper of the same name in that file. For a GGUF outside the catalog it also attaches a context-size estimate (the three `gguf*` modules above). - [`net.rs`](../crates/rocm-core/src/net.rs) — (ROCMAI-154) networking primitives: the default local host and port, URL/endpoint formatting and parsing, and the hand-rolled loopback HTTP/TCP client (`http_get_text*`, `http_post_json_with_auth`, `read_http_response_bounded`). It depends on no other service module: the managed-service readiness probes built on it live in [`endpoint_readiness.rs`](../crates/rocm-core/src/endpoint_readiness.rs) and the streaming downloads in [`download.rs`](../crates/rocm-core/src/download.rs). - [`paths.rs`](../crates/rocm-core/src/paths.rs) — (ROCMAI-154, Phase 7b) `AppPaths` and its filesystem-location primitives. - [`placeholder.rs`](../crates/rocm-core/src/placeholder.rs) — the `` placeholder rendered reports print for an empty value: `empty_as_unknown` (returns the value untouched) and `trimmed_or_unknown` (trims first), shared by `apps/rocm`'s driver-install plan and chat tool summaries and by [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs)'s Windows GPU diagnostics, replacing the copy those two `apps/rocm` callers shared in [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and `host_gpu.rs`'s own trimming copy. diff --git a/docs/llm-tool-use.md b/docs/llm-tool-use.md index 46fffe82ab..9626d47799 100644 --- a/docs/llm-tool-use.md +++ b/docs/llm-tool-use.md @@ -29,7 +29,10 @@ rocm-cli local assistants use structured tools, not shell commands. The provider-assisted planner's argv is held to the same rule, and a planner note carrying such a character is left out and counted. - CPU fallback is not a supported path. GPU-required ROCm commands must fail - loudly when the ROCm GPU path is not ready. + loudly when the ROCm GPU path is not ready. Keeping some of a model's layers + in system RAM on a GPU that is ready (llama.cpp's `--n-gpu-layers`), which + `rocm diagnose --model` suggests when a GGUF model does not fit in VRAM, is + not CPU fallback, and the assistant may relay it. - A `rocm serve` the model proposes through `rocm_command`, or the provider-assisted planner proposes in its tool call, is refused unless it asks for no public bind (`--allow-public-bind`), no CPU device, a loopback diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 2171fdd3a5..c8528d7118 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1250,3 +1250,66 @@ Expected result: - Re-enabling a watcher right after disabling the last one (within the few seconds before the helper exits) still ends with a helper running: enable waits for an exiting helper to finish, then starts a new one. + +## 12. Context-Size Estimate for a GGUF Model + +Needs a host with a Radeon GPU (a discrete card: an APU answers `undetermined` +by design) and network access to Hugging Face. The point is to check the +estimate against what a real load uses: the compute buffer and runtime reserve +were calibrated on one GPU and one engine build (an MI300X with llama.cpp b9752), +so a different GPU or build can differ. How close the estimate came on them is +recorded in the module documentation of +[gguf_fit.rs](../crates/rocm-core/src/gguf_fit.rs), not repeated here. + +```bash +rocm diagnose --model /:Q4_K_M --context-size 8192 +``` + +Expected result: + +- The first lines say the header was read and the weights were not downloaded, + and the header read finishes in seconds. +- The report opens `ESTIMATED: fits at 8192 tokens` or `ESTIMATED: does not fit + at 8192 tokens`, and lists weights, KV cache, compute buffer, runtime reserve + and the total, with this GPU's dedicated VRAM and the largest context that + fits. +- Repeat with `--kv-cache-type q8_0` and `q4_0`: the KV cache figure shrinks to + about 53% and 28% of the `f16` one. +- Serve the same file with llama.cpp's `llama-server` at the same `--ctx-size` + (with `-fa on` and `--n-gpu-layers 99`, and `--cache-type-k`/`--cache-type-v` if changed) and compare the total in + the report with the GPU memory it uses once loaded (`amd-smi metric`). The + estimate should be at or a little above the real figure; if it is below it, or + far above, report the model and both figures. +- `llama-server` at the reported largest context should load. The one check of + this kind made so far was on a single MI300X, with a second process holding + most of the card's memory so that about 15.1-15.5 GiB could still be allocated + (the process's `amd-smi` baselines were 181092 and 180738 MiB used of 196592), + for a Llama-3.1-8B Q4_K_M with an f16 cache. The card was described to + `rocm diagnose` as one with 15854 MiB, for which the estimate was 79872 tokens + (77056 tokens for 15500 MiB). `llama-server` loaded at 80896, 83968, 84992 and + 86016 tokens (used memory grew by 15449, 15837, 15788 and 15767 MiB) and failed + to allocate at 87040, 90112 and 93184, so the real limit was between 86016 and + 87040 tokens and the estimate was about 7-8% under it (79872 is 7.1% under + 86016 and 8.2% under 87040). The load at 86016 tokens is the calibration load + "made with the card almost full" in + [gguf_fit.rs](../crates/rocm-core/src/gguf_fit.rs)'s module documentation, + which says how the estimate compared with it. The baseline moved + by 354 MiB between runs, so those used-memory figures are good to a few + hundred MiB at best. That is one model on one card, not a bound; a different + model, GPU or build can land elsewhere. At a size well above what loads, the + report should say `does not fit` and list what would make it fit. +- `rocm serve` is unchanged: it takes no context-size option, and the report + names llama.cpp's options only as llama.cpp's. + +Also check the refusals: `owner/repo` with no `:QUANT` reads nothing and, with a +`--context-size`, says a quantization is needed; a repository that does not +exist says it was not found; a `.gguf` path that does not exist says it could +not be opened and does not point at the upstream tracker; and naming a vision +model's `mmproj-*.gguf` file on Hugging Face (`owner/repo:mmproj-....gguf`) says +it is a multimodal projector, not a model, names the model file instead, and +does not suggest `rocm serve` for the projector. None downloads +anything. + +Finally, `rocm diagnose --report --model /:Q4_K_M --context-size +8192` prints the report with no model section, and stderr says that `--model, +--context-size were ignored` and nothing about the model was fetched. diff --git a/docs/testing.md b/docs/testing.md index 1cbf599274..b3369520b6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -3339,12 +3339,15 @@ cargo xtask e2e -- -n '(diagnose-48)' ## Model Fit Preflight -`rocm diagnose --model ` answers whether a curated model will run on this -host before anything downloads. It reports one of four verdicts: `ready`, +`rocm diagnose --model ` answers whether a model will run on this host +before anything downloads: a curated model from its recipe (this section), a +GGUF outside the catalog from its file header (the next one). For a curated +model it reports one of four verdicts: `ready`, `degraded` (runs, but under the recipe's recommended system RAM), `blocked` (will not run here, with alternatives that would), or `undetermined` (the CLI could not judge it — an unreachable catalog, an -unmeasured GPU, or a ref outside the curated set). +unmeasured GPU, or a ref outside the curated set; for a GGUF reference the +estimate below is added to that answer). ```bash rocm diagnose --model qwen-smoke --json # smallest curated recipe @@ -3364,6 +3367,101 @@ including the two that need a synthetic signed catalog to trigger deterministically (`ModelNotCurated`, `Degraded`) since no built-in recipe can produce them on an arbitrary real host. +### GGUF estimate (models outside the catalog) + +A GGUF reference outside the catalog (a local `.gguf` path, or +`owner/repo:QUANT`) is estimated from its header instead: +`rocm diagnose --model --context-size N --kv-cache-type f16|q8_0|q4_0`. +The verdict stays `undetermined` (`model_not_curated`); the answer is +`model.estimate`, with outcome `fits`, `does_not_fit` or `cannot_estimate`, and +the text report opens `ESTIMATED:` for the first two. The parts and their tests: + +- [gguf.rs](../crates/rocm-core/src/gguf.rs): the header parser, tested against + synthetic headers built byte by byte (versions, truncation at every offset, + per-layer head arrays, signed 32-bit integers as gguf-py writes them, + every fixed-width value type read at its own width as a scalar and as an + array element, skipped arrays, alignment, the span of the input embedding) and one test per + bound a hostile header would walk past (key length, entry counts, tensor + dimensions, array nesting, an overflowing array length, an unknown value type, + the layer count, how many shape keys are kept, how large a header may be -- + including the checks inside a string array and an array of arrays, which are + walked over an endless stream and must stop near the bound -- a single length + larger than any header, and the length at which an array or string is no + longer kept). +- [gguf_fit.rs](../crates/rocm-core/src/gguf_fit.rs): the arithmetic (exact KV + figures, the boundary between fits and does not, the largest context -- a + trained context that is not a whole step included, on both sides -- the + suggestions, each layout it refuses to estimate, the state-space, hybrid and + mixture-of-experts layouts that get their figures with no verdict, and the + measured llama.cpp loads the allowances were calibrated on, which the synthetic Llama- and + Qwen-shaped headers must reproduce: buffer sizes, and a total never below the + measured use and within a bound of it per model. The module documentation + describes the loads and records the actual margins). +- [gguf_source.rs](../crates/rocm-core/src/gguf_source.rs): reference parsing + (an existing file wins over `owner/repo:QUANT`; `owner/repo:dir/file.gguf` + names one of two same-named files), file selection (shards of one model versus + two models; a full path beats a bare name that matches a nested file; a + multimodal projector is never picked, and a name that matches only projectors + is refused as naming one), names + in a listing that could leave the repository or hide in a terminal (dropped, + and never requested), and the range reader, tested against a loopback server that + records the ranges it is asked for, so a read of the weights would show, and + that also answers with a wrong or missing `Content-Range`, a dropped body, a + missing size in the listing and in the `HEAD`, a gated repository and a slow + reply. [diagnose_model_sends_no_token.rs](../apps/rocm/tests/diagnose_model_sends_no_token.rs) + runs `rocm diagnose --model owner/repo:QUANT` against such a server with every + Hugging Face token variable set, and checks that the listing, `HEAD` and + range requests carry no `Authorization` or `Cookie` header. +- [model_readiness.rs](../crates/rocm-core/src/model_readiness.rs): the + host-memory states (dedicated, none, unknown, APU), what is attached to a + curated or non-GGUF answer (nothing beyond a note on a flag that did not + apply), the `rocm diagnose` command a GGUF plan offers (the same flags, the + reference quoted for a shell), the plan a sized estimate keeps when it + could not compare (`amd-smi metric --json` where the memory was unreadable, + `rocm diagnose` where no GPU is visible, as for a curated model, and none on + an APU; unit-tested only, since no lane has a GPU whose memory cannot be + read), and the text and JSON shapes. +- [diagnose_model.rs](../apps/rocm/src/diagnose_model.rs) and + `classify_accelerator_memory` in [model_fit.rs](../apps/rocm/src/model_fit.rs): a + device mask that hides every GPU ("no GPU is visible", not a card of 0 GiB, on + a discrete card and on an APU alike), + and the note for a mask on a host whose GPUs differ in memory. Both are unit + tests only: the hosted e2e runners have no GPU for a mask to hide, and no CI + lane has GPUs of different sizes. + +Scenario `diagnose-51` points `HF_ENDPOINT` at a loopback stand-in that counts +requests, and shows that `rocm diagnose --report --model owner/repo:QUANT` makes +none and says on stderr that `--model` was ignored, while the same question +without `--report` does reach the endpoint and prints no such note. Scenario +`diagnose-52` has the stand-in answer the repository listing with a model and +its multimodal projector, and shows that naming the projector exactly is +answered as a projector, with the model file named instead, after the listing +request alone. + +Scenarios `diagnose-49` and `diagnose-50` write a small GGUF into the scenario's +isolated directory and assert the shape and the sizes (weights and KV cache are +exact), and that a host that measured its GPU gets a comparison while one that +could not is told why. Which branch runs depends on the lane: the GPU-less +hosted runners take the "could not compare" one (the text report must then read +UNDETERMINED), and the MI300X self-hosted lane (`E2E tests (MI300X, ...)` in +`e2e-selfhosted.yml`) takes the compared one, where the text report must read +ESTIMATED. The "does not fit" half depends on the host's VRAM, so it is covered +by the unit tests, not by a scenario. Nothing in either runs against a real GPU +load; see [manual-testing.md](manual-testing.md) for the check that does. + +Scenarios `diagnose-53` to `diagnose-56` cover the refusals, all host-independent: +the context options without `--model` (a usage error naming it), a curated model +asked about a context size (told the options do not apply, no estimate), a +`.gguf` path that does not exist (named as not opened, no upstream route, and a +plan to ask again about that path, the path quoted where the platform's shell +needs it, so the step accepts it bare or quoted), and a bare `owner/repo` with a context size +(told to add `:QUANT`, with a counting stand-in hub showing nothing was fetched). Scenario `diagnose-57` asks the stand-in hub for a quantization that matches +nothing, and the answer lists the model and the projector, marked as one. + +```bash +cargo xtask e2e -- -n '(diagnose-49|diagnose-5[0-7])' +``` + ## Doctor Report Preflight Preview the content a problem report would carry, without sending anything: diff --git a/skills/rocm-cli-assistant/SKILL.md b/skills/rocm-cli-assistant/SKILL.md index a386d6fa11..799ddfc92f 100644 --- a/skills/rocm-cli-assistant/SKILL.md +++ b/skills/rocm-cli-assistant/SKILL.md @@ -29,7 +29,7 @@ Use this skill when answering ROCm CLI local assistant questions. - Installing an engine and running a model server are different states. Answer each separately when the user asks. - `rocm serve` accepts `--gpu auto|` to pick the AMD GPU. `auto` (default) prefers a GPU that looks idle from VRAM telemetry (`amd-smi`, or the amdgpu DRM sysfs counters when `amd-smi` is absent) and is not already used by another rocm-cli server (managed or foreground), falling back to the GPU with the most free memory; a single index pins one GPU. Serving one model across multiple GPUs is not supported. Do not suggest CPU fallback when a GPU is busy or out of range. - Automatic port selection only works on host `127.0.0.1` exactly. `localhost` and `::1` are loopback for auth purposes but still need an explicit `--port` when launching a server, or the call is refused. -- Which engines actually run on the user's machine is stated in the host facts the CLI appends to this prompt. Answer from those facts; do not assume a platform. Never suggest CPU fallback. +- Which engines actually run on the user's machine is stated in the host facts the CLI appends to this prompt. Answer from those facts; do not assume a platform. Never suggest CPU fallback. Keeping some of a model's layers in system RAM on a GPU that is ready (llama.cpp's `--n-gpu-layers`) is not CPU fallback: when `rocm diagnose --model` suggests it for a GGUF that does not fit in VRAM, relay that suggestion. - On native Windows, vLLM serving/install live checks are skipped; tell the user to use WSL/Linux for that ROCm GPU engine and do not suggest CPU fallback. ## ComfyUI diff --git a/skills/rocm-doctor/reference.md b/skills/rocm-doctor/reference.md index fe111b4581..18ec363ab6 100644 --- a/skills/rocm-doctor/reference.md +++ b/skills/rocm-doctor/reference.md @@ -16,7 +16,7 @@ tooling. - It's a general system inspector, so it **always exits 0**. The verdict is the `status` field: `ok` · `no-amd-gpu` · `wsl` · `unsupported-os` · `degraded`. -### `rocm diagnose [--symptom ""] [--top N] [--json] [--distro [NAME]] [--model REF]` +### `rocm diagnose [--symptom ""] [--top N] [--json] [--distro [NAME]] [--model REF [--context-size N] [--kv-cache-type f16|q8_0|q4_0]]` Match the host + symptom against the closed catalog. **Always exits 0**; read the result from `--json`: @@ -48,14 +48,48 @@ the result from `--json`: nothing was checked — not a clean bill of health. - `route_when_no_match` — `{ target, url }` upstream tracker to use when `has_match` is false. - `model` — set only when `--model ` was passed; otherwise not set. Answers - "will this curated model run here" from the recipe catalog and this host's - GPU/RAM, without downloading anything: `{ verdict, evidence[], engine, - required_gpu_memory_gib, available_gpu_memory_gib, alternatives[], - undetermined_reason, fix }`. `verdict` is one of `ready`, `degraded`, + "will this model run here", without downloading the weights: for a curated + model from the recipe catalog and this host's GPU/RAM, and for a GGUF outside + the catalog with an `estimate` from its file header (below). Its fields: + `{ verdict, evidence[], engine, required_gpu_memory_gib, + available_gpu_memory_gib, alternatives[], undetermined_reason, fix }`. `verdict` is one of `ready`, `degraded`, `blocked`, `undetermined` — treat `undetermined` as "the CLI could not find out", never as evidence the model is incompatible. `--model` is refused together with `--distro`: the verdict is about the machine running the command, not the one `--distro` names. + For a GGUF outside the catalog (a local `.gguf` path, or `owner/repo:QUANT`) + the verdict stays `undetermined` and an `estimate` object is added, sized from + the file's header (the weights are never downloaded). Its fields, by group: + where it came from (`source`, `architecture`, `layers`); the outcome + (`outcome` is `fits`, `does_not_fit` or `cannot_estimate`, and with + `cannot_estimate` the `cannot_estimate_reason` and `detail` say why; the + figures are still given when the reason is about the host (`no_gpu`, + `accelerator_memory_unknown`, `unified_memory_unreadable`: what the model would + need, with nothing to compare it to) and are null when it is about the file + (`header_unreadable`, `not_a_model` for a multimodal projector named in place + of a model, `split_file`, `latent_attention`, `no_attention_layers`, + `missing_metadata`), and are given with no verdict for `unmeasured_layout` (a + state-space, hybrid or mixture-of-experts model, part of whose memory is not + counted or not measured, so the total can be too low) and for a + `missing_metadata` header that lacks only the feed-forward or embedding width + (the compute buffer is then understated)); what was asked for (`context_size`, `context_size_defaulted`, + `kv_cache_type`); the memory figures (`weights_gib`, `kv_cache_gib`, + `compute_buffer_gib`, `runtime_reserve_gib`, `total_gib`, the same in bytes as + `weights_bytes`, `kv_cache_bytes`, `compute_buffer_bytes`, + `runtime_reserve_bytes` and `total_bytes`, and `kv_cache_bytes_per_token`, the + cache's cost of one token); the comparison (`available_gib` is the total + dedicated VRAM of the GPU with the most of it that ROCm can see, not what is + free; `max_context` is the largest context that fits, capped at + `training_context`, and is null when GPU memory was not measured, when no + verdict is given (`unmeasured_layout`, or `missing_metadata` with figures), + and when not even one 256-token step fits); and what the figures leave out + (`conservative_ceiling` says the ceiling is known to be conservative, + `suggestions[]` what would make it fit, `notes[]` the caveats). + It is an estimate, not a tested minimum: report it as one. With `--report` + the model is not assessed at all and nothing is fetched; a note on stderr names + the model flags that were ignored. + `suggestions[]` name llama.cpp options; `rocm serve` does not take a context + size yet, so do not propose a `rocm serve` command for them. ### `rocm fix [] [--yes] [--dry-run] [--device-index N] [--json]` diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index 0a690d3439..98eef53e17 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -304,7 +304,8 @@ Feature: Diagnosing failures and listing fixes And the preview states that the CLI can run it automatically # `--model` answers the question that comes before the other two: given this # machine and that model, will it run. The point is that it answers in seconds - # and fetches nothing, so the user is not told by a download that failed. + # and downloads no weights (a curated model fetches nothing at all), so the user + # is not told by a download that failed. # # Host-agnostic in the same way diagnose-10 is, and for the same reason. The # verdict depends on what this machine can measure of its own GPU, which @@ -748,3 +749,103 @@ Feature: Diagnosing failures and listing fixes Given a machine with an AMD GPU When the user diagnoses this machine in machine-readable form, with no symptom Then the diagnosis lists the HIP SDK cause only when the inspection finds something that needs it + + # A GGUF outside the catalog has no recipe, so there is no tested minimum to + # compare. Its header still says how many layers it has, how many KV heads and + # how wide, and the file's size is the weights: enough to estimate memory at a + # chosen context size. (That a remote file is read for its header alone is + # asserted by the gguf_source unit test that serves an 8 GiB file and records + # every range; a local file is only opened, so no step here could see a + # download.) The answer must stay + # visibly an estimate -- a different outcome from the closed verdict, and a + # different word -- because a figure derived from metadata and allowances + # calibrated on one GPU is not a model that was loaded. The file is a few kilobytes of header and a + # tiny weight region, so the scenario is deterministic on every lane; the + # comparison with GPU memory is host-specific and asserted per half, as + # diagnose-21 and diagnose-22 do. + @id:diagnose-model-gguf-is-estimated-from-its-header-not-judged + Scenario: diagnose-49 - A GGUF outside the catalog is sized from its header and reported as an estimate, not a verdict + Given a user asking about a small GGUF file on this machine + When the user asks the CLI whether that file would run at 2048 tokens with a q8_0 KV cache, in machine-readable form + Then the answer is an estimate under its own outcome, and the verdict stays undetermined + And the estimate sizes the file's weights and its KV cache at that context + And a machine that measured its GPU is told whether the estimate fits, and one that could not is told why + + # The human-readable half of diagnose-49: the label is what a reader sees + # first, and READY or BLOCKED over a figure derived from metadata would claim + # a certainty the estimate does not have. + @id:diagnose-model-gguf-estimate-is-labelled-in-the-text-report + Scenario: diagnose-50 - The text report labels a GGUF estimate as an estimate and never calls the file ready or blocked + Given a user asking about a small GGUF file on this machine + When the user asks the CLI whether that file would run at 2048 tokens with a q8_0 KV cache, in human-readable form + Then the report is labelled an estimate, or undetermined with the figures, and is never ready or blocked + + # `--report` promises that nothing leaves the machine. A GGUF reference such as + # owner/repo:Q4_K_M is read from Hugging Face, which would put the model's name + # on the wire, so a prepared report must not assess the model at all. The + # second half proves the endpoint is wired in: without `--report` the same + # question does reach it, and the note about ignoring the model is not shown. + @id:diagnose-report-never-reaches-the-hub-for-a-model + Scenario: diagnose-51 - A prepared problem report never asks the hub about a model + Given a Hugging Face endpoint that records every request made to it + When the user asks the CLI to prepare a problem report that names a model on that endpoint + Then no request reached the endpoint + And the CLI says the model was ignored rather than assessed + When the user asks the CLI the same model question without preparing a report + Then the endpoint was asked about the model + And the CLI does not say the model was ignored + + # A llama.cpp vision model ships its multimodal projector (mmproj) beside the + # model. Naming the projector exactly is answered for what it is, with the + # model file named instead, rather than as a file the repository lacks; and + # it is refused from the listing alone, before any header is read. + @id:diagnose-a-projector-named-exactly-is-answered-as-a-projector + Scenario: diagnose-52 - Naming a model's multimodal projector says what it is and names the model + Given a Hugging Face repository that holds a model and its multimodal projector + When the user asks the CLI whether the projector file would run, naming it exactly + Then the CLI says the file is a multimodal projector and names the model file instead + And the endpoint was asked only for the repository's file list + + # The context options describe the model being asked about. Without --model + # they would be accepted and do nothing, so the CLI refuses them as a usage + # error before anything runs. + @id:diagnose-context-options-need-a-model + Scenario: diagnose-53 - The context options are refused without a model to apply them to + When the user asks the CLI for a context size or a KV cache type without naming a model + Then the CLI refuses each as a usage error that names --model + + # A curated model's memory minimum comes from its recipe and is fixed, so a + # context size cannot change the answer. The report says so instead of + # silently ignoring the option, and makes no estimate. + @id:diagnose-context-options-do-not-apply-to-a-curated-model + Scenario: diagnose-54 - A curated model is told the context options do not apply to it + When the user asks the CLI whether the smallest curated model would run at 8192 tokens, in machine-readable form + Then the answer says the context options do not apply to a curated recipe, and carries no estimate + + # A .gguf path that does not exist is a path to check, not a model to report + # upstream: the answer names the path, gives no upstream route, and its plan + # is to ask again once the path is right. + @id:diagnose-a-missing-gguf-path-is-a-path-to-check + Scenario: diagnose-55 - A GGUF path that does not exist is answered as a path to check, not a model to report + When the user asks the CLI whether a GGUF file that does not exist would run, in machine-readable form + Then the answer says the file could not be opened and offers no upstream route + And the plan is to ask again about that path + + # owner/repo with no :QUANT names a repository, and every file in it is a + # different model, so nothing is read; with a context size the user is told + # what would make it answerable. The stand-in hub shows no request was made. + @id:diagnose-a-bare-repository-with-a-context-size-asks-for-a-quantization + Scenario: diagnose-56 - A bare repository with a context size is told to name a quantization, and nothing is fetched + Given a Hugging Face endpoint that records every request made to it + When the user asks the CLI about a repository with no quantization at 8192 tokens, in machine-readable form + Then the answer says the reference names a repository and asks for a quantization + And no request reached the endpoint + + # A quantization that matches nothing lists what the repository does hold, + # and a multimodal projector is among the files listed, marked as one, rather + # than left out: a user looking at the hub sees it there too. + @id:diagnose-a-quantization-that-matches-nothing-lists-every-gguf + Scenario: diagnose-57 - A quantization that matches nothing lists the repository's files, projectors marked + Given a Hugging Face repository that holds a model and its multimodal projector + When the user asks the CLI about a quantization the repository does not hold + Then the CLI says nothing matches and lists the model and the projector, marked as a projector diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index a9f6a391e6..dee0ee502f 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -70,6 +70,10 @@ pub struct E2eWorld { /// Listener that deliberately occupies the default serve port while a real /// `rocm serve` invocation proves automatic allocation advances past it. pub automatic_port_guard: Option, + /// How many requests a stand-in Hugging Face endpoint has answered, for + /// scenarios that must show the CLI did or did not reach out. + /// The serving thread stops once the World drops its handle. + pub hub_requests: Option>, /// Cache-marker destination discovered from `rocmd`'s own JSON report. pub artifact_marker_path: Option, pub endpoint: Option, @@ -327,6 +331,7 @@ impl Default for E2eWorld { paced_download_server: None, comfyui_port_listener: None, automatic_port_guard: None, + hub_requests: None, artifact_marker_path: None, endpoint: None, model_name: None, diff --git a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs index a95ddf04c6..35c47321df 100644 --- a/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/diagnose_steps.rs @@ -3331,3 +3331,682 @@ async fn assert_hip_sdk_advice_follows_the_inspection(world: &mut E2eWorld) { field("hipinfo_present"), ); } + +const TINY_GGUF_LAYERS: u64 = 2; +const TINY_GGUF_KV_HEADS: u64 = 1; +const TINY_GGUF_HEAD_WIDTH: u64 = 64; +const TINY_GGUF_WEIGHT_BYTES: u64 = 1024 * 1024; +const TINY_GGUF_REQUESTED_CONTEXT: u64 = 2048; + +/// A minimal, valid GGUF v3 file: the metadata an estimate reads, one tensor +/// record, and a zeroed weight region of [`TINY_GGUF_WEIGHT_BYTES`]. +fn tiny_gguf_bytes() -> Vec { + fn string(out: &mut Vec, value: &str) { + out.extend_from_slice(&(value.len() as u64).to_le_bytes()); + out.extend_from_slice(value.as_bytes()); + } + fn entry_u32(out: &mut Vec, key: &str, value: u32) { + string(out, key); + out.extend_from_slice(&4_u32.to_le_bytes()); + out.extend_from_slice(&value.to_le_bytes()); + } + let mut metadata = Vec::new(); + string(&mut metadata, "general.architecture"); + metadata.extend_from_slice(&8_u32.to_le_bytes()); + string(&mut metadata, "e2e"); + entry_u32(&mut metadata, "e2e.block_count", TINY_GGUF_LAYERS as u32); + entry_u32(&mut metadata, "e2e.embedding_length", 128); + entry_u32(&mut metadata, "e2e.feed_forward_length", 256); + entry_u32(&mut metadata, "e2e.attention.head_count", 2); + entry_u32( + &mut metadata, + "e2e.attention.head_count_kv", + TINY_GGUF_KV_HEADS as u32, + ); + entry_u32(&mut metadata, "e2e.context_length", 8192); + + let mut tensor = Vec::new(); + string(&mut tensor, "w"); + tensor.extend_from_slice(&1_u32.to_le_bytes()); + tensor.extend_from_slice(&1024_u64.to_le_bytes()); + tensor.extend_from_slice(&0_u32.to_le_bytes()); + tensor.extend_from_slice(&0_u64.to_le_bytes()); + + let mut file = Vec::new(); + file.extend_from_slice(b"GGUF"); + file.extend_from_slice(&3_u32.to_le_bytes()); + file.extend_from_slice(&1_u64.to_le_bytes()); + file.extend_from_slice(&7_u64.to_le_bytes()); + file.extend_from_slice(&metadata); + file.extend_from_slice(&tensor); + // Tensor data starts at the next multiple of the default 32-byte alignment. + file.resize(file.len().div_ceil(32) * 32, 0); + file.resize(file.len() + TINY_GGUF_WEIGHT_BYTES as usize, 0); + file +} + +#[given("a user asking about a small GGUF file on this machine")] +async fn user_asks_about_a_small_gguf(world: &mut E2eWorld) { + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .to_path_buf(); + let path = root.join("tiny-e2e-model.gguf"); + std::fs::write(&path, tiny_gguf_bytes()).expect("write the GGUF fixture"); + world.model_name = Some(path.to_string_lossy().into_owned()); +} + +#[when( + "the user asks the CLI whether that file would run at 2048 tokens with a q8_0 KV cache, in machine-readable form" +)] +async fn user_asks_whether_a_gguf_would_run_json(world: &mut E2eWorld) { + let model = world.model_name.clone().expect("no model named"); + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &[ + "diagnose", + "--model", + &model, + "--context-size", + "2048", + "--kv-cache-type", + "q8_0", + "--json", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[when( + "the user asks the CLI whether that file would run at 2048 tokens with a q8_0 KV cache, in human-readable form" +)] +async fn user_asks_whether_a_gguf_would_run_text(world: &mut E2eWorld) { + let model = world.model_name.clone().expect("no model named"); + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &[ + "diagnose", + "--model", + &model, + "--context-size", + "2048", + "--kv-cache-type", + "q8_0", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +fn estimate_section(model: &serde_json::Value) -> &serde_json::Value { + model + .get("estimate") + .filter(|value| value.is_object()) + .unwrap_or_else(|| panic!("the model section carries no estimate: {model:#}")) +} + +#[then("the answer is an estimate under its own outcome, and the verdict stays undetermined")] +async fn assert_gguf_answer_is_an_estimate(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "diagnose is a query and exits 0"); + let model = model_section(world); + assert_eq!( + model_verdict(&model), + "undetermined", + "an estimate from a header is not a recipe's verdict: {model:#}" + ); + assert_eq!( + model_field(&model, "undetermined_reason").as_str(), + Some("model_not_curated"), + "{model:#}" + ); + let estimate = estimate_section(&model); + assert_eq!(estimate["context_size"], TINY_GGUF_REQUESTED_CONTEXT); + assert_eq!(estimate["context_size_defaulted"], false); + assert_eq!(estimate["kv_cache_type"], "q8_0"); + assert_eq!(estimate["architecture"], "e2e"); + assert_eq!(estimate["layers"], TINY_GGUF_LAYERS); +} + +#[then("the estimate sizes the file's weights and its KV cache at that context")] +async fn assert_gguf_estimate_sizes_weights_and_cache(world: &mut E2eWorld) { + let model = model_section(world); + let estimate = estimate_section(&model); + let gib = f64::from(1024 * 1024 * 1024_u32); + let weights = estimate["weights_gib"] + .as_f64() + .unwrap_or_else(|| panic!("no weights figure: {estimate:#}")); + assert!( + (weights - TINY_GGUF_WEIGHT_BYTES as f64 / gib).abs() < 1e-9, + "the weights are the file's size after its header: {estimate:#}" + ); + // layers x kv heads x (key + value width) x tokens x 34/32 bytes (q8_0). + let kv_bytes = (TINY_GGUF_LAYERS * TINY_GGUF_KV_HEADS * 2 * TINY_GGUF_HEAD_WIDTH) as f64 + * TINY_GGUF_REQUESTED_CONTEXT as f64 + * 34.0 + / 32.0; + let kv = estimate["kv_cache_gib"] + .as_f64() + .unwrap_or_else(|| panic!("no KV cache figure: {estimate:#}")); + assert!( + (kv - kv_bytes / gib).abs() < 1e-9, + "expected {} GiB of KV cache: {estimate:#}", + kv_bytes / gib + ); +} + +#[then( + "a machine that measured its GPU is told whether the estimate fits, and one that could not is told why" +)] +async fn assert_gguf_estimate_compares_only_a_measured_gpu(world: &mut E2eWorld) { + let model = model_section(world); + let estimate = estimate_section(&model); + let outcome = estimate["outcome"].as_str().unwrap_or_default(); + if estimate["available_gib"].is_number() { + // The outcome the reported figures imply: the verdict is decided on the + // parts as shown (each rounded up to a whole MiB) against the card's + // memory, so the same arithmetic over the JSON says which answer is + // owed on this lane, and a predicate turned the wrong way fails here. + const MIB: f64 = 1024.0 * 1024.0; + let shown_mib: f64 = [ + "weights_bytes", + "kv_cache_bytes", + "compute_buffer_bytes", + "runtime_reserve_bytes", + ] + .iter() + .map(|field| { + let bytes = estimate[field] + .as_u64() + .unwrap_or_else(|| panic!("no {field}: {estimate:#}")); + (bytes as f64 / MIB).ceil() + }) + .sum(); + let available_mib = estimate["available_gib"].as_f64().unwrap_or_default() * 1024.0; + let expected = if shown_mib <= available_mib { + "fits" + } else { + "does_not_fit" + }; + assert_eq!( + outcome, expected, + "{shown_mib} MiB needed against {available_mib} MiB: {estimate:#}" + ); + if outcome == "fits" { + assert!( + estimate["max_context"].as_u64().unwrap_or(0) >= TINY_GGUF_REQUESTED_CONTEXT, + "fits at 2048 tokens but reports a smaller ceiling: {estimate:#}" + ); + } + } else { + assert_eq!(outcome, "cannot_estimate", "{estimate:#}"); + let reason = estimate["cannot_estimate_reason"] + .as_str() + .unwrap_or_default(); + assert!( + matches!( + reason, + "no_gpu" | "accelerator_memory_unknown" | "unified_memory_unreadable" + ), + "no memory was measured, so the reason must say so: {estimate:#}" + ); + assert!( + estimate["total_gib"].is_number(), + "what the file needs is still worth reporting: {estimate:#}" + ); + } +} + +#[then( + "the report is labelled an estimate, or undetermined with the figures, and is never ready or blocked" +)] +async fn assert_gguf_text_report_is_labelled(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "diagnose is a query and exits 0"); + let output = world.cli_output.clone().expect("no diagnose output"); + let model = world.model_name.clone().expect("no model named"); + let header = output + .lines() + .find(|line| line.starts_with(&format!("rocm diagnose --model {model}: "))) + .unwrap_or_else(|| panic!("no model header in:\n{output}")); + for verdict in ["READY", "BLOCKED", "DEGRADED"] { + assert!( + !header.ends_with(verdict) && !header.contains(&format!(": {verdict}")), + "an estimate was reported with the verdict word {verdict}: {header}" + ); + } + // Which label is owed depends on whether this host's memory was compared, + // and the report says which: a compared estimate names the card it was + // compared with. A compared estimate must be labelled ESTIMATED with the + // outcome its own figures imply, and one that was not, UNDETERMINED -- so + // the label is checked on whichever lane runs this, not only accepted. + // A GPU-less hosted runner takes the second branch; a lane that measures + // dedicated VRAM (the MI300X self-hosted lane) takes the first. + let compared = output.contains("the GPU with the most dedicated VRAM that ROCm can see has"); + if compared { + // The label owed follows from the two figures the report prints: the + // total it needs ("= about N MiB") and the card's ("has M MiB"). + let number_after = |marker: &str| -> u64 { + let at = output + .find(marker) + .unwrap_or_else(|| panic!("no `{marker}` in:\n{output}")) + + marker.len(); + output[at..] + .split(|c: char| !c.is_ascii_digit()) + .next() + .and_then(|digits| digits.parse().ok()) + .unwrap_or_else(|| panic!("no figure after `{marker}` in:\n{output}")) + }; + let needed = number_after("= about "); + let offered = number_after("can see has "); + let label = if needed <= offered { + ": ESTIMATED: fits at 2048 tokens" + } else { + ": ESTIMATED: does not fit at 2048 tokens" + }; + assert!( + header.contains(label), + "{needed} MiB needed against {offered} MiB, so `{label}` is owed: {header}" + ); + } else { + assert!( + header.ends_with(": UNDETERMINED"), + "nothing was compared, so the answer is undetermined: {header}" + ); + } + assert!( + output.contains("KV cache (q8_0)"), + "the figures the estimate rests on were not shown:\n{output}" + ); +} + +/// Start a stand-in Hugging Face endpoint on a loopback port and point the CLI at +/// it. It counts every connection. A repository listing request +/// (`/api/models/...`) is answered with `listing` when one is given; every other +/// request, and a listing when none is given, gets an empty 404. It stops when +/// the World lets go of its handle, so a scenario leaves no thread behind. +fn start_stand_in_hub(world: &mut E2eWorld, listing: Option<&'static str>) { + use std::io::{Read as _, Write as _}; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind the stand-in hub"); + listener + .set_nonblocking(true) + .expect("make the stand-in hub non-blocking"); + let port = listener.local_addr().expect("hub address").port(); + let requests = Arc::new(AtomicUsize::new(0)); + let seen = Arc::clone(&requests); + std::thread::spawn(move || { + while Arc::strong_count(&seen) > 1 { + match listener.accept() { + Ok((mut stream, _)) => { + seen.fetch_add(1, Ordering::SeqCst); + let _ = stream.set_nonblocking(false); + let _ = stream.set_read_timeout(Some(std::time::Duration::from_secs(5))); + // The request line is all that is needed, and it arrives first. + let mut buf = [0_u8; 2048]; + let read = stream.read(&mut buf).unwrap_or(0); + let request = String::from_utf8_lossy(&buf[..read]); + let is_listing = request + .split_whitespace() + .nth(1) + .is_some_and(|path| path.starts_with("/api/models/")); + let response = match listing { + Some(body) if is_listing => format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\ + Content-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ), + _ => "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + .to_owned(), + }; + let _ = stream.write_all(response.as_bytes()); + } + Err(_) => std::thread::sleep(std::time::Duration::from_millis(10)), + } + } + }); + world.hub_requests = Some(requests); + world + .command_env + .push(("HF_ENDPOINT", format!("http://127.0.0.1:{port}").into())); +} + +#[given("a Hugging Face endpoint that records every request made to it")] +async fn a_hub_endpoint_that_counts_requests(world: &mut E2eWorld) { + start_stand_in_hub(world, None); +} + +/// A repository holding one model and its multimodal projector, as llama.cpp +/// vision models ship. Sizes are given so no `HEAD` request is needed. +const MODEL_AND_PROJECTOR_LISTING: &str = r#"{"siblings":[{"rfilename":"e2e-model-Q4_K_M.gguf","size":1048576},{"rfilename":"mmproj-e2e-model-f16.gguf","size":1048576}]}"#; + +#[given("a Hugging Face repository that holds a model and its multimodal projector")] +async fn a_hub_repository_with_a_model_and_a_projector(world: &mut E2eWorld) { + start_stand_in_hub(world, Some(MODEL_AND_PROJECTOR_LISTING)); +} + +#[when("the user asks the CLI whether the projector file would run, naming it exactly")] +async fn user_asks_about_the_projector_file(world: &mut E2eWorld) { + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env( + world, + &[ + "diagnose", + "--model", + "e2e-owner/e2e-repo:mmproj-e2e-model-f16.gguf", + ], + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the CLI says the file is a multimodal projector and names the model file instead")] +async fn assert_the_cli_named_the_projector_and_the_model(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let output = world.cli_output.clone().unwrap_or_default(); + assert!( + output.contains( + "`mmproj-e2e-model-f16.gguf` is a multimodal projector (mmproj), not a language model" + ), + "{output}" + ); + assert!(output.contains("e2e-model-Q4_K_M.gguf"), "{output}"); + assert!(!output.contains("no GGUF file matching"), "{output}"); + // Nothing points the user back at the file it refused. + assert!( + !output.contains("rocm serve e2e-owner/e2e-repo:mmproj"), + "the refused projector was offered as a serve command:\n{output}" + ); + // The one model in the repository is offered by its exact name. + assert!( + output.contains( + "verify after fix: rocm diagnose --model e2e-owner/e2e-repo:e2e-model-Q4_K_M.gguf" + ), + "{output}" + ); + assert!(!output.contains("header could not be read"), "{output}"); +} + +#[then("the endpoint was asked only for the repository's file list")] +async fn assert_only_the_listing_was_requested(world: &mut E2eWorld) { + let seen = world + .hub_requests + .as_ref() + .expect("no stand-in hub") + .load(std::sync::atomic::Ordering::SeqCst); + // One listing request; the projector was refused before any header read. + assert_eq!(seen, 1, "the hub was asked {seen} time(s)"); +} + +const HUB_MODEL_REF: &str = "e2e-owner/e2e-repo:Q4_K_M"; + +async fn run_hub_model_question(world: &mut E2eWorld, report: bool) { + // The environment the Given recorded is consumed by one run; keep it for the next. + let env = world.command_env.clone(); + let mut args = vec!["diagnose", "--model", HUB_MODEL_REF, "--json"]; + if report { + args.insert(1, "--report"); + } + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env(world, &args); + world.command_env = env; + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[when("the user asks the CLI to prepare a problem report that names a model on that endpoint")] +async fn user_prepares_a_report_naming_a_hub_model(world: &mut E2eWorld) { + run_hub_model_question(world, true).await; +} + +#[when("the user asks the CLI the same model question without preparing a report")] +async fn user_asks_the_hub_model_question_without_a_report(world: &mut E2eWorld) { + run_hub_model_question(world, false).await; +} + +#[then("no request reached the endpoint")] +async fn assert_no_request_reached_the_hub(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + // A moment for a request that was sent to be counted before the check. + std::thread::sleep(std::time::Duration::from_millis(300)); + let seen = world + .hub_requests + .as_ref() + .expect("no stand-in hub") + .load(std::sync::atomic::Ordering::SeqCst); + assert_eq!(seen, 0, "a prepared report reached the hub {seen} time(s)"); +} + +#[then("the CLI says the model was ignored rather than assessed")] +async fn assert_the_report_said_it_ignored_the_model(world: &mut E2eWorld) { + let stderr = world.cli_stderr.clone().unwrap_or_default(); + assert!( + stderr.contains("--model was ignored") && stderr.contains("nothing about the model"), + "stderr was: {stderr}" + ); +} + +#[then("the CLI does not say the model was ignored")] +async fn assert_the_cli_did_not_say_it_ignored_the_model(world: &mut E2eWorld) { + let stderr = world.cli_stderr.clone().unwrap_or_default(); + assert!(!stderr.contains("was ignored"), "stderr was: {stderr}"); +} + +#[then("the endpoint was asked about the model")] +async fn assert_the_hub_was_asked_about_the_model(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let seen = world + .hub_requests + .as_ref() + .expect("no stand-in hub") + .load(std::sync::atomic::Ordering::SeqCst); + assert!( + seen > 0, + "the same question without --report never reached the endpoint, so the check above \ + proved nothing" + ); +} + +fn run_model_question(world: &mut E2eWorld, args: &[&str]) { + let (stdout, stderr, rc) = crate::run_rocm_with_scenario_env(world, args); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +fn evidence_text(model: &serde_json::Value) -> String { + model_field(model, "evidence") + .as_array() + .map(|lines| { + lines + .iter() + .filter_map(serde_json::Value::as_str) + .collect::>() + .join("\n") + }) + .unwrap_or_default() +} + +#[when("the user asks the CLI for a context size or a KV cache type without naming a model")] +async fn user_asks_for_context_options_without_a_model(world: &mut E2eWorld) { + let mut outcomes = Vec::new(); + for option in [["--context-size", "8192"], ["--kv-cache-type", "q8_0"]] { + let (_, stderr, rc) = + crate::run_rocm_with_scenario_env(world, &["diagnose", option[0], option[1]]); + outcomes.push(format!("{}\t{rc}\t{stderr}", option[0])); + } + world.cli_output = Some(outcomes.join("\n\u{0}\n")); +} + +#[then("the CLI refuses each as a usage error that names --model")] +async fn assert_context_options_refused_without_a_model(world: &mut E2eWorld) { + let output = world.cli_output.clone().unwrap_or_default(); + let outcomes: Vec<&str> = output.split("\n\u{0}\n").collect(); + assert_eq!(outcomes.len(), 2, "{output}"); + for outcome in outcomes { + let mut parts = outcome.splitn(3, '\t'); + let (option, rc, stderr) = ( + parts.next().unwrap_or(""), + parts.next().unwrap_or(""), + parts.next().unwrap_or(""), + ); + // clap's usage-error exit code. + assert_eq!( + rc, "2", + "{option} was not refused as a usage error: {stderr}" + ); + assert!( + stderr.contains("--model"), + "{option}: the refusal does not name what is missing: {stderr}" + ); + } +} + +#[when( + "the user asks the CLI whether the smallest curated model would run at 8192 tokens, in machine-readable form" +)] +async fn user_asks_about_a_curated_model_with_a_context_size(world: &mut E2eWorld) { + run_model_question( + world, + &[ + "diagnose", + "--model", + SMALLEST_MODEL_REF, + "--context-size", + "8192", + "--json", + ], + ); +} + +#[then( + "the answer says the context options do not apply to a curated recipe, and carries no estimate" +)] +async fn assert_curated_model_says_context_options_do_not_apply(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let model = model_section(world); + let evidence = evidence_text(&model); + assert!( + evidence.contains("do not apply to a curated recipe"), + "the option was accepted and nothing said it was not applied: {model:#}" + ); + assert!( + model.get("estimate").is_none_or(serde_json::Value::is_null), + "a curated model was estimated from a header it has none of: {model:#}" + ); +} + +fn missing_gguf_path(world: &E2eWorld) -> String { + world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("no-such-model.gguf") + .to_string_lossy() + .into_owned() +} + +#[when( + "the user asks the CLI whether a GGUF file that does not exist would run, in machine-readable form" +)] +async fn user_asks_about_a_missing_gguf(world: &mut E2eWorld) { + let path = missing_gguf_path(world); + assert!( + !std::path::Path::new(&path).exists(), + "{path} must not exist" + ); + run_model_question(world, &["diagnose", "--model", &path, "--json"]); +} + +#[then("the answer says the file could not be opened and offers no upstream route")] +async fn assert_missing_gguf_could_not_be_opened(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let model = model_section(world); + let path = missing_gguf_path(world); + let evidence = evidence_text(&model); + assert!( + evidence.contains(&format!("could not open {path}")), + "the answer does not say which file could not be opened: {model:#}" + ); + assert_eq!( + estimate_section(&model)["cannot_estimate_reason"], + "header_unreadable", + "{model:#}" + ); + assert!( + model.get("route").is_none_or(serde_json::Value::is_null), + "a path that does not exist was routed to the upstream tracker: {model:#}" + ); +} + +#[then("the plan is to ask again about that path")] +async fn assert_missing_gguf_plan_is_to_check_the_path(world: &mut E2eWorld) { + let model = model_section(world); + let path = missing_gguf_path(world); + let fix = model_field(&model, "fix"); + // The command names the path as given, quoted when the platform's shell + // would split or expand it (the scenario's temp directory decides which). + let verify = fix["verify"].as_str().unwrap_or(""); + let bare = format!("rocm diagnose --model {path}"); + let quoted = format!("rocm diagnose --model '{path}'"); + // Either form asks again with `rocm diagnose`; serving a file that is not + // there would only repeat the error. + assert!(verify == bare || verify == quoted, "{model:#}"); +} + +#[when( + "the user asks the CLI about a repository with no quantization at 8192 tokens, in machine-readable form" +)] +async fn user_asks_about_a_bare_repository_with_a_context_size(world: &mut E2eWorld) { + run_model_question( + world, + &[ + "diagnose", + "--model", + UNCURATED_MODEL_REF, + "--context-size", + "8192", + "--json", + ], + ); +} + +#[then("the answer says the reference names a repository and asks for a quantization")] +async fn assert_bare_repository_asks_for_a_quantization(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let model = model_section(world); + let evidence = evidence_text(&model); + assert!( + evidence.contains(&format!( + "`{UNCURATED_MODEL_REF}` names a repository, not a model file: add `:QUANT`" + )), + "{model:#}" + ); + assert!( + model.get("estimate").is_none_or(serde_json::Value::is_null), + "a repository was estimated as if it were one file: {model:#}" + ); +} + +#[when("the user asks the CLI about a quantization the repository does not hold")] +async fn user_asks_about_a_missing_quantization(world: &mut E2eWorld) { + run_model_question(world, &["diagnose", "--model", "e2e-owner/e2e-repo:Q8_0"]); +} + +#[then("the CLI says nothing matches and lists the model and the projector, marked as a projector")] +async fn assert_missing_quantization_lists_every_gguf(world: &mut E2eWorld) { + assert_eq!(world.cli_rc, Some(0), "{:?}", world.cli_stderr); + let output = world.cli_output.clone().unwrap_or_default(); + assert!( + output.contains( + "no GGUF file matching `Q8_0` in the repository (it has: e2e-model-Q4_K_M.gguf, \ + mmproj-e2e-model-f16.gguf (multimodal projector))" + ), + "{output}" + ); +} From a94103d80782ba156008be769e6b513a31cb581e Mon Sep 17 00:00:00 2001 From: Roman Date: Mon, 9 Nov 2026 09:30:39 +0000 Subject: [PATCH 007/102] ROCMAI-860: fix(dash): stop local Qwen3 chat answers being lost to unbounded reasoning (#910) * fix(dash): stop local Qwen3 chat answers being lost to unbounded reasoning Asked a question in the dashboard's chat, a locally served Qwen3 model could spend the entire request generating reasoning tokens and return no visible answer, while the same model answered normally in the CLI. Qwen3 chat templates enable hybrid thinking by default with an effectively unbounded reasoning budget, and only the CLI's local provider was turning it off. The predicate deciding that was private to the CLI path, so the dashboard's chat client had no way to agree with it. Move it into the leaf crate both front ends already depend on, with its own tests, so there is one copy of the rule rather than two that drift. Sending the flag from the dashboard had to not cost anything else. rig-core's `AgentBuilder::additional_params` replaces that field rather than merging into it, so adding the chat-template object as a second call would have silently dropped a configured `top_p` off the wire. Both now go into one object set by a single call, and a request with neither knob set is byte-for-byte unchanged. A model with no hybrid thinking mode is left alone: the key is absent from the request body entirely rather than sent as true, so a backend that rejects an unrecognised field is never handed one. Signed-off-by: Roman Sirokov * refactor(dash): drop a dead disjunct and pin what the Qwen3 match covers `local_model_should_disable_thinking` lowercased its argument and then asked whether it contained `qwen3` or `qwen/qwen3`. The second disjunct can only be true where the first already is, so it never decided anything. Dropping it leaves the predicate matching exactly the ids it matched before. What it does match is worth stating outright, because the name does not give it away: the decision is made on the model id alone, so every id that spells `qwen3` anywhere matches, whether or not what is served behind that name actually has a hybrid-thinking chat template. The doc comment now says so. Coverage pins both of the awkward ends. `qwen32b` is a catalog alias whose parameter count runs straight on from `qwen3` with no separator; `qwen3.5` and `qwen3.6-27b` are later families that match on the strength of their names. Narrowing the rule now means changing those tests and that doc comment together, deliberately, rather than finding out afterwards which real identifier stopped matching. Signed-off-by: Roman Sirokov * docs(architecture): point the local-model thinking rule at rocm-dash-core The `providers/wire.rs` inventory still listed `local_model_should_disable_thinking` among the helpers behind its request-body builders, and declared the module to depend on `providers.rs` and `providers/tool_schema.rs` and nothing else. Both statements stopped holding when the rule moved into `rocm-dash-core`'s `chat_template` module, so that the CLI's local chat provider and the dashboard's chat client cannot decide it differently. Name the rule where it now lives, record the edge `wire.rs` gained, and add the module to the `rocm-dash-core` entry, whose crate-level summary of pure types, traits, schemas and the reducer does not otherwise account for it. Signed-off-by: Roman Sirokov * docs(providers): record wire.rs's dependency on chat_template The module doc enumerates what the wire formats depend on, but was not updated when the local-model request defaults started reading the chat-template rule from rocm-dash-core. docs/architecture.md already records that edge; this brings the in-file comment in line with it. Signed-off-by: Roman Sirokov --------- Signed-off-by: Roman Sirokov (cherry picked from commit 17f655d3f94f965f99e3cc1ed23b5609bd7f1a9b) Signed-off-by: juhovainio --- apps/rocm/src/providers/wire.rs | 16 +- crates/rocm-dash-core/src/chat_template.rs | 98 +++++++++ crates/rocm-dash-core/src/lib.rs | 1 + crates/rocm-dash-tui/src/agent/clients.rs | 233 ++++++++++++++++----- docs/architecture.md | 4 +- 5 files changed, 295 insertions(+), 57 deletions(-) create mode 100644 crates/rocm-dash-core/src/chat_template.rs diff --git a/apps/rocm/src/providers/wire.rs b/apps/rocm/src/providers/wire.rs index e30a6ae433..4892bf2866 100644 --- a/apps/rocm/src/providers/wire.rs +++ b/apps/rocm/src/providers/wire.rs @@ -7,8 +7,9 @@ //! non-streaming responses, and the SSE emitters that turn a streamed response //! into `ProviderStreamEvent`s. //! -//! Depends on the shared request/response types in `providers` and on the ROCm -//! tool schema in `tool_schema`; the local HTTP transport in `local_http` +//! Depends on the shared request/response types in `providers`, on the ROCm +//! tool schema in `tool_schema` and on the chat-template rule in +//! [`rocm_dash_core::chat_template`]; the local HTTP transport in `local_http` //! reads [`OpenAiSseEmitter`] from here, so the dependency is one-way. use super::tool_schema::{rocm_anthropic_tool_definitions, rocm_openai_tool_definitions}; @@ -46,19 +47,18 @@ pub(super) fn openai_local_stream_chat_request_body( body } +/// The chat-template rule itself lives in +/// [`rocm_dash_core::chat_template`], not here: the dashboard's chat client +/// sends the same request to the same locally served model and has to decide +/// this identically, and a rule kept in two places drifts. fn apply_local_model_request_defaults(model: &str, body: &mut serde_json::Value) { - if local_model_should_disable_thinking(model) { + if rocm_dash_core::chat_template::local_model_should_disable_thinking(model) { body["chat_template_kwargs"] = serde_json::json!({ "enable_thinking": false }); } } -fn local_model_should_disable_thinking(model: &str) -> bool { - let normalized = model.to_ascii_lowercase(); - normalized.contains("qwen3") || normalized.contains("qwen/qwen3") -} - fn local_openai_compatible_request(request: &ChatRequest) -> ChatRequest { let system = request .messages diff --git a/crates/rocm-dash-core/src/chat_template.rs b/crates/rocm-dash-core/src/chat_template.rs new file mode 100644 index 0000000000..d9757b8db8 --- /dev/null +++ b/crates/rocm-dash-core/src/chat_template.rs @@ -0,0 +1,98 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Chat-template defaults a locally served model needs on an +//! OpenAI-compatible chat request. Pure: a decision over a model id, no I/O. +//! +//! Two front ends send that request — the `rocm` CLI's local chat provider and +//! the dashboard's chat client — and both must decide this the same way. They +//! did not: the CLI turned hybrid thinking off while the dashboard left it on, +//! so the same locally served model answered in one and reasoned past the +//! request deadline in the other. The rule therefore lives here, in the leaf +//! crate both already depend on, as one copy with one set of tests. + +/// Whether a locally served `model` should have its chat template's hybrid +/// "thinking" mode turned off — i.e. whether the request body needs +/// `chat_template_kwargs.enable_thinking = false`. +/// +/// Qwen3 chat templates enable thinking by default with an effectively +/// unbounded reasoning budget, so a small quantized build on modest hardware +/// can spend an entire request generating reasoning tokens and return no +/// visible answer at all. Matching is case-insensitive and by substring, so a +/// repository-qualified and quantization-suffixed id such as +/// `Qwen/Qwen3-1.7B-GGUF:Q4_K_M` matches too. +/// +/// The decision is made on the name alone and is therefore approximate: every +/// id that spells `qwen3` anywhere matches, whether or not what is actually +/// served behind that name has a hybrid-thinking chat template. +pub fn local_model_should_disable_thinking(model: &str) -> bool { + model.to_ascii_lowercase().contains("qwen3") +} + +#[cfg(test)] +mod tests { + use super::local_model_should_disable_thinking; + + #[test] + fn qwen3_ids_match_in_every_spelling_a_serving_endpoint_reports() { + // A served model id reaches this helper in whatever shape the engine + // reports it: bare, repository-qualified, quantization-suffixed, and + // in any case. All of them name the same chat template. `qwen32b` is + // the awkward one: a catalog alias where the parameter count runs + // straight on from `qwen3` with no separator at all. + for model in [ + "qwen3", + "Qwen3-0.6B-GGUF", + "Qwen/Qwen3-8B", + "QWEN3-32B-AWQ", + "unsloth/Qwen3-1.7B-GGUF:Q4_K_M", + "qwen32b", + ] { + assert!( + local_model_should_disable_thinking(model), + "`{model}` names a hybrid-thinking chat template and must match" + ); + } + } + + #[test] + fn later_qwen3_families_match_on_the_strength_of_the_name_alone() { + // Pins the approximation the doc comment above describes, rather than + // endorsing it. `qwen3.5` and `qwen3.6-27b` are real catalog aliases + // (`crates/rocm-core/src/model_catalog.json`) that spell `qwen3` as a + // prefix, so they match on their names alone — nothing here has + // looked at the chat template actually served behind them. Narrowing + // the rule to exclude them would change behaviour for a model that + // matches today, so it has to be a deliberate decision taken against + // this test and that doc comment together. + for model in ["qwen3.5", "qwen3.6-27b"] { + assert!( + local_model_should_disable_thinking(model), + "`{model}` matches by name today; excluding it is a behaviour \ + change, not a refactor" + ); + } + } + + #[test] + fn other_models_are_left_alone() { + // Everything else keeps the server's own chat-template defaults: the + // caller sends no `chat_template_kwargs` at all, so a backend that + // rejects or misreads the key is never handed one. `qwen2.5` is the + // near miss that matters — same family, no hybrid thinking mode. + for model in [ + "local-model", + "qwen2.5-7b-instruct", + "Qwen/Qwen2.5-Coder-32B", + "llama-3.1-8b-instruct", + "gpt-4o-mini", + "", + ] { + assert!( + !local_model_should_disable_thinking(model), + "`{model}` must keep the server's own chat-template defaults" + ); + } + } +} diff --git a/crates/rocm-dash-core/src/lib.rs b/crates/rocm-dash-core/src/lib.rs index 40b13a970d..03bd4fb6e7 100644 --- a/crates/rocm-dash-core/src/lib.rs +++ b/crates/rocm-dash-core/src/lib.rs @@ -11,6 +11,7 @@ pub mod bench_rollup; pub mod bench_schema; +pub mod chat_template; pub mod config; pub mod efficiency; pub mod metrics; diff --git a/crates/rocm-dash-tui/src/agent/clients.rs b/crates/rocm-dash-tui/src/agent/clients.rs index ae42afd3e1..4cb007aec6 100644 --- a/crates/rocm-dash-tui/src/agent/clients.rs +++ b/crates/rocm-dash-tui/src/agent/clients.rs @@ -14,6 +14,7 @@ use std::sync::{Arc, Mutex}; use std::time::Instant; use async_trait::async_trait; +use rocm_dash_core::chat_template::local_model_should_disable_thinking; use tracing::{error, info, warn}; use crate::app::{ChatRole, ChatTurn}; @@ -37,17 +38,31 @@ const MAX_TOOL_TURNS: usize = 5; /// Used as the fallback when no explicit `--max-tokens` is configured. const MAX_AGENT_TOKENS: u64 = 1024; -/// Apply the optional sampling controls to a fresh Rig `AgentBuilder`. +/// Apply the optional sampling controls, plus any chat-template defaults the +/// named model needs, to a fresh Rig `AgentBuilder`. /// /// `max_tokens` always resolves to a concrete value (the CLI override or the /// shared [`MAX_AGENT_TOKENS`] default). `temperature` maps to the builder's -/// native `.temperature()`; `top_p` has no dedicated builder method in -/// rig-core, so it rides in `.additional_params({"top_p": ..})`, which the -/// provider merges into the request body. Both are set only when supplied, so -/// an unset knob leaves the request untouched. +/// native `.temperature()`. The rest have no dedicated builder method in +/// rig-core and ride in `.additional_params(..)`, which the provider flattens +/// into the request body: +/// +/// - `top_p`, when supplied; +/// - `chat_template_kwargs.enable_thinking = false`, when `model` names a +/// model whose chat template would otherwise reason without a budget (see +/// [`local_model_should_disable_thinking`]). `model` is `Some` only for the +/// OpenAI-compatible backend, which is the one that can be pointed at a +/// locally served model. +/// +/// Both are collected into ONE object and set with a single +/// `.additional_params(..)` call, because rig-core 0.38.1's builder *replaces* +/// that field rather than merging into it — calling it twice silently drops +/// whatever the first call set. An empty object is not sent at all, so a +/// request with neither knob set stays byte-for-byte what it was. fn apply_inference_params( builder: rig::agent::AgentBuilder, params: &InferenceParams, + model: Option<&str>, ) -> rig::agent::AgentBuilder where M: rig::completion::CompletionModel, @@ -57,8 +72,18 @@ where if let Some(temperature) = params.temperature { builder = builder.temperature(f64::from(temperature)); } + let mut additional = serde_json::Map::new(); if let Some(top_p) = params.top_p { - builder = builder.additional_params(serde_json::json!({ "top_p": top_p })); + additional.insert("top_p".to_string(), serde_json::json!(top_p)); + } + if model.is_some_and(local_model_should_disable_thinking) { + additional.insert( + "chat_template_kwargs".to_string(), + serde_json::json!({ "enable_thinking": false }), + ); + } + if !additional.is_empty() { + builder = builder.additional_params(serde_json::Value::Object(additional)); } builder } @@ -240,9 +265,17 @@ where /// shared site. A backend with ordering constraints (ChatGPT's auth must not /// run on an empty history) does its own cheap `history.is_empty()` guard /// first — see `ChatGptAgentClient::complete` — without needing the split. +/// +/// `model` carries the model id for the OpenAI-compatible backend, the one +/// that can be pointed at a locally served model, so +/// [`apply_inference_params`] can add the chat-template defaults such a model +/// needs. The hosted backends pass `None`: their models are never served +/// locally, and an unrecognised key on a hosted request is a needless risk. +#[allow(clippy::too_many_arguments)] async fn run_agent_request( agent: rig::agent::AgentBuilder, params: &InferenceParams, + model: Option<&str>, executor: Option<&SharedRocmToolExecutor>, approval_tx: Option<&UnboundedSender>, history: &[ChatTurn], @@ -262,7 +295,7 @@ where let snap = Arc::new(snapshot); let fired: FiredLog = Arc::new(Mutex::new(Vec::new())); - let agent = apply_inference_params(agent, params); + let agent = apply_inference_params(agent, params, model); // Telemetry tools (shared registration site). let agent = register_telemetry_tools(agent, &snap, &fired); // Read-only ROCm machine-inspection tools (forward across the seam). @@ -373,6 +406,7 @@ where run_agent_request( agent, &self.params, + Some(self.model.as_str()), self.executor.as_ref(), self.approval_tx.as_ref(), history, @@ -627,6 +661,7 @@ where run_agent_request( agent, &self.params, + None, self.executor.as_ref(), self.approval_tx.as_ref(), history, @@ -722,6 +757,7 @@ where run_agent_request( agent, &self.params, + None, self.executor.as_ref(), self.approval_tx.as_ref(), history, @@ -1782,6 +1818,7 @@ mod tests { ¶ms, None, None, + None, &history, fixture_snapshot(), "mock-test", @@ -1879,6 +1916,7 @@ mod tests { ¶ms, None, None, + None, &history, fixture_snapshot(), "mock-test", @@ -1972,6 +2010,7 @@ mod tests { &InferenceParams::default(), None, None, + None, &[], fixture_snapshot(), "mock-test", @@ -2017,6 +2056,7 @@ mod tests { let reply = run_agent_request( agent, &InferenceParams::default(), + None, Some(&exec), Some(&tx), &history, @@ -2249,6 +2289,56 @@ mod tests { /// if the test builds its client with `DEFAULT_PREAMBLE` itself. const TEST_PREAMBLE: &str = "test-only preamble, distinct from the production default"; + /// Build a `RigAgentClient` whose HTTP client is a + /// [`rig::test_utils::RecordingHttpClient`] scripted with one successful + /// OpenAI chat completion, returning both so the caller can drive a real + /// `complete()` and then read the request that reached the wire. + /// + /// `RigAgentClient`'s `H` (HTTP client) type parameter — defaulted to + /// the live `reqwest::Client` for production — exists for exactly this. + fn rig_client_over_recorded_response( + model: &str, + params: InferenceParams, + ) -> ( + RigAgentClient, + rig::test_utils::RecordingHttpClient, + ) { + use rig::test_utils::RecordingHttpClient; + + let response_body = serde_json::to_vec(&json!({ + "id": "chatcmpl-test", + "object": "chat.completion", + "created": 0, + "model": model, + "system_fingerprint": null, + "choices": [{ + "index": 0, + "message": {"role": "assistant", "content": "GPU 0 is healthy."}, + "logprobs": null, + "finish_reason": "stop", + }], + "usage": {"prompt_tokens": 1, "total_tokens": 2}, + })) + .expect("serialize stub OpenAI completion response"); + let http_client = RecordingHttpClient::new(response_body); + let rig_client = rig::providers::openai::CompletionsClient::builder() + .api_key("sk-test") + .base_url("http://rocm-dash-tui-test.invalid") + .http_client(http_client.clone()) + .build() + .expect("build client over the recording HTTP double"); + + let client = RigAgentClient { + client: rig_client, + model: model.to_string(), + preamble: TEST_PREAMBLE.to_string(), + params, + executor: None, + approval_tx: None, + }; + (client, http_client) + } + /// Asserts the `model`/`temperature`/`max_tokens` fields a captured wire /// request carries, shared by the two forwards-params-to-the-wire tests /// below. Each caller separately asserts where its provider places the @@ -2280,52 +2370,19 @@ mod tests { /// actually passes it `&self.params` rather than, say, /// `&InferenceParams::default()`, or the right tag — a swap at that one /// call site stays green under every other test in this file; only a - /// real `complete()` round-trip catches it. - /// - /// `RigAgentClient`'s `H` (HTTP client) type parameter — defaulted to - /// the live `reqwest::Client` for production — exists for exactly this: - /// swap in `rig::test_utils::RecordingHttpClient` and drive the actual - /// `complete()` method over a scripted response instead of a live - /// endpoint. + /// real `complete()` round-trip catches it — over + /// [`rig_client_over_recorded_response`]'s scripted HTTP double rather + /// than a live endpoint. #[tokio::test] async fn rig_agent_client_complete_forwards_params_to_the_wire() { - use rig::test_utils::RecordingHttpClient; - - let response_body = serde_json::to_vec(&json!({ - "id": "chatcmpl-test", - "object": "chat.completion", - "created": 0, - "model": "test-model", - "system_fingerprint": null, - "choices": [{ - "index": 0, - "message": {"role": "assistant", "content": "GPU 0 is healthy."}, - "logprobs": null, - "finish_reason": "stop", - }], - "usage": {"prompt_tokens": 1, "total_tokens": 2}, - })) - .expect("serialize stub OpenAI completion response"); - let http_client = RecordingHttpClient::new(response_body); - let rig_client = rig::providers::openai::CompletionsClient::builder() - .api_key("sk-test") - .base_url("http://rocm-dash-tui-test.invalid") - .http_client(http_client.clone()) - .build() - .expect("build client over the recording HTTP double"); - - let client = RigAgentClient { - client: rig_client, - model: "test-model".to_string(), - preamble: TEST_PREAMBLE.to_string(), - params: InferenceParams { + let (client, http_client) = rig_client_over_recorded_response( + "test-model", + InferenceParams { temperature: Some(0.4), top_p: None, max_tokens: Some(256), }, - executor: None, - approval_tx: None, - }; + ); let history = [ChatTurn::user("check gpu 0")]; let (reply, tags) = complete_capturing_backend_tags(&client, &history).await; @@ -2357,6 +2414,88 @@ mod tests { ); } + /// Drive one `RigAgentClient::complete` round-trip over a scripted + /// OpenAI-compatible response and hand back the JSON body that reached + /// the wire. The chat-template tests below care only about the body, so + /// they skip the reply/backend-tag assertions + /// `rig_agent_client_complete_forwards_params_to_the_wire` makes over the + /// same seam. + async fn rig_wire_request_body(model: &str, params: InferenceParams) -> serde_json::Value { + let (client, http_client) = rig_client_over_recorded_response(model, params); + let history = [ChatTurn::user("check gpu 0")]; + complete_capturing_backend_tags(&client, &history).await; + + let requests = http_client.requests(); + assert_eq!( + requests.len(), + 1, + "exactly one request should have been sent" + ); + serde_json::from_slice(&requests[0].body).expect("request body is JSON") + } + + /// A model whose chat template reasons without a budget must be told to + /// stop: the request body carries + /// `chat_template_kwargs.enable_thinking = false`, so the answer the + /// operator asked for is generated instead of an unbounded reasoning + /// trace that can consume the whole request deadline. + /// + /// The `top_p` half of this is not incidental. rig-core 0.38.1's + /// `AgentBuilder::additional_params` **replaces** rather than merges, so + /// adding the chat-template object as a second call would silently drop + /// `top_p` — a configured sampling knob vanishing off the wire, with + /// nothing else in this file to notice. Both must arrive together. + #[tokio::test] + async fn rig_agent_client_disables_unbounded_thinking_without_dropping_top_p() { + let body = rig_wire_request_body( + "unsloth/Qwen3-1.7B-GGUF:Q4_K_M", + InferenceParams { + temperature: Some(0.4), + top_p: Some(0.2), + max_tokens: Some(256), + }, + ) + .await; + + assert_eq!( + body["chat_template_kwargs"]["enable_thinking"].as_bool(), + Some(false), + "a hybrid-thinking model must be sent enable_thinking=false: {body}" + ); + assert_eq!( + body["top_p"].as_f64(), + Some(f64::from(0.2_f32)), + "top_p must survive alongside chat_template_kwargs, not be replaced by it: {body}" + ); + } + + /// The other half of the contract: every other model keeps the serving + /// endpoint's own chat-template defaults. The key is absent from the body + /// entirely — not sent as `true` — so a hosted backend that rejects an + /// unrecognised field is never handed one. + #[tokio::test] + async fn rig_agent_client_sends_no_chat_template_kwargs_for_other_models() { + let body = rig_wire_request_body( + "test-model", + InferenceParams { + temperature: Some(0.4), + top_p: Some(0.2), + max_tokens: Some(256), + }, + ) + .await; + + assert!( + body.get("chat_template_kwargs").is_none(), + "a model with no hybrid thinking mode must get no chat_template_kwargs: {body}" + ); + assert_eq!( + body["top_p"].as_f64(), + Some(f64::from(0.2_f32)), + "top_p must still reach the wire on its own: {body}" + ); + } + /// As `rig_agent_client_complete_forwards_params_to_the_wire`, for /// `AnthropicAgentClient::complete`'s call site. #[tokio::test] diff --git a/docs/architecture.md b/docs/architecture.md index ad51f2aaf6..ac2ba48a37 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -78,7 +78,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`providers.rs`](../apps/rocm/src/providers.rs) — full domain extraction (no dedicated command enum; invoked from the existing chat/config command flows). Keeps the request/response types, the `ProviderAdapter` trait and the `provider_*` dispatch, the `LocalProvider` and `RemoteProvider` implementations, local-service selection (`select_local_chat_service`, `ready_local_services`), the remote model and endpoint resolution, the `ureq` remote transport and the `SseBytesEmitter` trait. The ROCm tool schema, the local HTTP/SSE transport and the provider wire formats live in child modules under `providers/`, below; `providers.rs` imports the request-body builders, response parsers and SSE emitters it calls from `providers/wire.rs`. - [`providers/tool_schema.rs`](../apps/rocm/src/providers/tool_schema.rs) — full domain extraction, a leaf child of `providers.rs`: the ROCm tool catalog a provider advertises to a model, as OpenAI function definitions (`rocm_openai_tool_definitions`) and reshaped for Anthropic's `tools` field (`rocm_anthropic_tool_definitions`). Both are `pub(super)`, read by the request-body builders in `providers/wire.rs`; `rocm_openai_tool`, the one-tool constructor behind them, is private. Depends on nothing else in `providers`. - [`providers/local_http.rs`](../apps/rocm/src/providers/local_http.rs) — full domain extraction, a child of `providers.rs`: the local provider's hand-rolled HTTP/1.1 transport. `post_json_to_local_endpoint` (with `post_json_to_local_endpoint_body` and `local_bearer_header`) is the blocking JSON POST; `stream_json_from_local_endpoint` is the streamed POST, which reads the response in whichever body framing the engine answers with (`read_sse_body` picks between `read_chunked_sse_body`, `read_fixed_sse_body` and `read_close_delimited_sse_body` from the headers, via `header_value`) and feeds an `OpenAiSseEmitter`; `body_truncated_at_eof` and `LOCAL_STREAM_TRUNCATED` name the failure when an engine closes the connection mid-answer; `parse_http_endpoint` reads the `http://` endpoint URL. `post_json_to_local_endpoint` and `stream_json_from_local_endpoint` are `pub(super)`, called from `LocalProvider`, and `LOCAL_STREAM_TRUNCATED` is also `pub(super)` (imported by `providers.rs` for its test module only); the rest is private. Its edges back into `providers` are `OpenAiSseEmitter` from `providers/wire.rs`, the `SseBytesEmitter` trait the emitter is pushed bytes through (which stays in `providers.rs`), and the shared `ProviderStreamEvent` type; nothing in `wire.rs` reads `local_http.rs`, so the edge is one-way. [`providers/local_http/proptests.rs`](../apps/rocm/src/providers/local_http/proptests.rs) holds the property tests of the SSE body readers and is declared from this module so they sit with the readers they test; the example-based test of `parse_http_endpoint` is in this module's own `mod tests`, and the tests that drive `LocalProvider` end to end against a mock server stay in `providers.rs`'s `mod tests` with their server helpers. -- [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) — full domain extraction, a child of `providers.rs`: the OpenAI-compatible and Anthropic wire formats. The request-body builders (`openai_chat_request_body`, `openai_stream_chat_request_body`, `openai_local_chat_request_body`, `openai_local_stream_chat_request_body`, `anthropic_chat_request_body`, `anthropic_stream_chat_request_body`, and the helpers behind them, including `local_openai_compatible_request`, `apply_local_model_request_defaults` and `local_model_should_disable_thinking`), the parsers for non-streaming responses (`parse_openai_chat_message`, `parse_anthropic_chat_message`), and the SSE emitters with their line parsers (`OpenAiSseEmitter` and `AnthropicSseEmitter`, which implement the `SseBytesEmitter` trait that stays in `providers.rs`; `parse_openai_sse_line`, `parse_anthropic_sse_line`). The builders, the two chat-message parsers and the two emitters (with their `new` and `finish`) are `pub(super)`, called from `LocalProvider` and `RemoteProvider`; `OpenAiSseEmitter` is also read by `providers/local_http.rs`. Depends on the shared request/response types in `providers.rs` and on `providers/tool_schema.rs`, and on nothing else. The tests of the builders and parsers live in this module's own `mod tests`, with the tool-catalog assertions that read through the request bodies, and [`providers/wire/sse_emitter_proptests.rs`](../apps/rocm/src/providers/wire/sse_emitter_proptests.rs), the split-invariance and panic-guard properties of the emitters and line parsers, is declared from this module; the stream tests that run an emitter against a mock server stay in `providers.rs`'s `mod tests`. +- [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) — full domain extraction, a child of `providers.rs`: the OpenAI-compatible and Anthropic wire formats. The request-body builders (`openai_chat_request_body`, `openai_stream_chat_request_body`, `openai_local_chat_request_body`, `openai_local_stream_chat_request_body`, `anthropic_chat_request_body`, `anthropic_stream_chat_request_body`, and the helpers behind them, including `local_openai_compatible_request` and `apply_local_model_request_defaults`; the rule that one applies, `local_model_should_disable_thinking`, lives in [`rocm-dash-core`](../crates/rocm-dash-core/src/chat_template.rs)'s `chat_template` module, because the dashboard's chat client has to decide it identically), the parsers for non-streaming responses (`parse_openai_chat_message`, `parse_anthropic_chat_message`), and the SSE emitters with their line parsers (`OpenAiSseEmitter` and `AnthropicSseEmitter`, which implement the `SseBytesEmitter` trait that stays in `providers.rs`; `parse_openai_sse_line`, `parse_anthropic_sse_line`). The builders, the two chat-message parsers and the two emitters (with their `new` and `finish`) are `pub(super)`, called from `LocalProvider` and `RemoteProvider`; `OpenAiSseEmitter` is also read by `providers/local_http.rs`. Depends on the shared request/response types in `providers.rs`, on `providers/tool_schema.rs` and on `rocm-dash-core`'s `chat_template`, and on nothing else. The tests of the builders and parsers live in this module's own `mod tests`, with the tool-catalog assertions that read through the request bodies, and [`providers/wire/sse_emitter_proptests.rs`](../apps/rocm/src/providers/wire/sse_emitter_proptests.rs), the split-invariance and panic-guard properties of the emitters and line parsers, is declared from this module; the stream tests that run an emitter against a mock server stay in `providers.rs`'s `mod tests`. - [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs) — the argv helpers that `chat.rs`, `tool_policy.rs` and the freeform planner share, in a leaf module whose only `crate::` dependency is the leaf [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs): `chat_cli_arg_value`, `chat_cli_arg_value_checked`, `chat_cli_has_flag` and `ensure_flag` (from `chat.rs`), and `format_structured_tool_call` (which quotes each argument with `shell_quote`), `serve_args_request_cpu_device` and `device_policy_is_cpu_mode` (from `freeform_planner.rs`). They moved so that `freeform_planner.rs` no longer imports `chat` for them; `chat.rs` re-exports `ensure_flag` (read by `freeform_execution.rs`) and `freeform_planner.rs` re-exports `format_structured_tool_call`, so every other caller keeps its path. - [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): registering a runtime that `rocm install sdk` did not install. `import_runtime_manifest` records a manifest written elsewhere, `adopt_runtime_from_python_options` (with `AdoptRuntimeOptions`, `AdoptRuntimeRequest`, `infer_adopt_runtime_request` and `adopt_runtime_from_probe`) probes an existing Python environment and registers it, and `recover_setup_runtime_registration` registers the runtime `rocm setup` created, with `newest_installed_runtime_manifest` and `setup_runtime_paths` beside it. `select_single_ready_runtime` is the one rule for picking "the one ready runtime" when nothing else says which to use: it recovers the setup runtime first, then returns a `SingleReadyRuntime` (`One`, `Zero` with every registered runtime, or `Many` with the ready ones), where ready is `validate_runtime_manifest_for_activation`; its pure half is `classify_single_ready_runtime`. A setup runtime that cannot be used is not a ready runtime and is not counted: it does not stop a registered ready runtime from being picked (or refused as ambiguous), and its own error is returned only when the registry holds no ready runtime, while a failure to write a usable setup runtime's entry back is always returned. Its `SetupRecovery` argument says whether recovering the setup runtime may write its lost registry entry back (`Persist`, what the engine paths and a real ComfyUI install use) or only count the runtime from its own manifest and write nothing (`PreviewOnly`, for a `--dry-run` preview, which must name the runtime the real run will use without changing a file); `SetupRecovery::for_dry_run` maps the flag. `registered_runtimes_with_setup` is the list that rule counts from (the registry plus the setup runtime, written back or not as `SetupRecovery` says), which `comfyui/runtime_env.rs` also searches when a choice that names a runtime is not answered by the registry alone. Both [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs) (which treats `Zero` and `Many` as no selection and leaves the advice to its callers) and [`comfyui/runtime_env.rs`](../apps/rocm/src/comfyui/runtime_env.rs) (which words them as refusals) call it directly rather than through `runtimes_cmd.rs`, and neither keeps its own copy. It depends on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (manifest paths, activation validation, `paths_equivalent`) and `therock.rs`, not on `runtimes_cmd.rs`, which calls into it and re-exports `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths`. `import_runtime_manifest`, `AdoptRuntimeOptions`, `adopt_runtime_from_python_options` (with the fields of `AdoptRuntimeOptions` and `AdoptRuntimeRequest`) and `adopt_runtime_from_probe` are `pub(crate)` for `runtimes()` and for the `runtimes_cmd.rs` tests that adopt or import a runtime and then list or activate it; `newest_installed_runtime_manifest`, `recover_setup_runtime_registration` and `setup_runtime_paths` are `pub(crate)` too, for the modules that import them through `runtimes_cmd.rs`'s re-exports; the rest is private. Its tests cover the import and adopt rules that do not involve activation or listing. - [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) — mechanical relocation, owns types (part of [`main.rs`](../apps/rocm/src/main.rs)'s Phase 6c split, ROCMAI-89, alongside [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) below): settling an engine onto a runtime after an SDK install: the orchestration (`finish_sdk_install`, `settle_engine_install`, `finalize_successful_sdk_install`), the engine-dependency check, the runtime device check and the post-install verdict built on it, and `SdkInstallConsents`. The torch alignment and retention it runs live in the child module [`runtime_install_finalize/torch_alignment.rs`](../apps/rocm/src/runtime_install_finalize/torch_alignment.rs). `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`; `settle_engine_install` is `pub(crate)` and imported by [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), so the import edge between the two runs one way, from `engines_cmd.rs` to this module. This module (and its torch-alignment child) does not import `engines_cmd.rs`: the engine environment and runtime-selection helpers it needs (`engine_manages_own_runtime`, `env_root_for_engine_install`, `runtime_key_for_python`, and `runtime_manifest_for_selector` from the torch-alignment child) are imported from [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs). The host system-package installs live in [`system_packages.rs`](../apps/rocm/src/system_packages.rs); this module re-exports `ensure_libatomic_for_torch`/`ensure_libnuma_for_torch` (called from both `install_cmd.rs`'s `install()` and `engines_cmd.rs`) and `ensure_openmpi_for_vllm` (called from `engines_cmd.rs` and, for the engine auto-install after an SDK install, from here) so those callers keep their imports. @@ -296,7 +296,7 @@ The `ui/` modules the split moved code out of or into, beyond the shared compone The other three dashboard crates are described at crate level here, not module by module: - [`rocm-dash-collectors`](../crates/rocm-dash-collectors/src/lib.rs) — the IO-heavy collector implementations behind `rocm-dash-core`'s traits (amd-smi, Docker, host metrics, Lemonade and vLLM scrapers, and on Windows the [`winperf.rs`](../crates/rocm-dash-collectors/src/winperf.rs) perf-counter/ADLX GPU collector) plus the benchmark load generator; used by `rocm-dash-daemon` and, on Windows, by `rocm-dash-tui` in-process. -- [`rocm-dash-core`](../crates/rocm-dash-core/src/lib.rs) — pure types, traits, schemas and the reducer, with no rendering dependencies. +- [`rocm-dash-core`](../crates/rocm-dash-core/src/lib.rs) — pure types, traits, schemas and the reducer, with no rendering dependencies. [`chat_template.rs`](../crates/rocm-dash-core/src/chat_template.rs) also holds the chat-template defaults a locally served model needs on a chat request (`local_model_should_disable_thinking`), kept here as the leaf crate both front ends already depend on so that `apps/rocm`'s [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) and the TUI's [`agent/clients.rs`](../crates/rocm-dash-tui/src/agent/clients.rs) cannot drift apart on it. - [`rocm-dash-daemon`](../crates/rocm-dash-daemon/src/lib.rs) — the daemon library: [`runner.rs`](../crates/rocm-dash-daemon/src/runner.rs) drives the collectors on tick cadences and broadcasts snapshots, and [`server.rs`](../crates/rocm-dash-daemon/src/server.rs) streams them to TUI clients as NDJSON. [`registry.rs`](../crates/rocm-dash-daemon/src/registry.rs) reads the managed-service records `rocm serve` writes and tells a registry that is confirmed empty (no directory, or no records) from one that could not be read: the run loop retires a managed service that an empty registry lacks, but when a pass cannot read the registry in full (a directory that cannot be listed, or a record file that cannot be opened) it retires nothing: the known services and their data stay, as when Docker discovery errors, the records that did read still add and refresh their services, and the header warns until a read succeeds. `apps/rocm`'s [`dash.rs`](../apps/rocm/src/dash.rs) drives it. ### `crates/rocm-engine-protocol` — engine IPC protocol From 92dc541b8719dd032b09053c795ec8ca7094eb6f Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 9 Nov 2026 10:44:12 +0000 Subject: [PATCH 008/102] refactor(core): share one atomic writer between rocm and rocmd (ROCMAI-681) (#610) * refactor(core): share one atomic file writer between rocm and rocmd `apps/rocm/src/therock.rs` and `apps/rocmd/src/sandbox.rs` each carried their own copy of `write_file_atomically` and its staging/publish helpers. Move the `therock` copy, the more complete of the two, into `rocm_core::atomic_write` and call it from both binaries, so the next caller (the active-runtime marker) gets the same implementation rather than a third copy. The move is mechanical apart from the error text. rocmd gets two extra pieces of context it was missing: the full-disk mapping through `disk_space::map_write_error`, and "failed to publish " when the rename fails. The `failed to create ` context that only rocmd had is kept. The helper tests move with the code. The five rocmd tests that duplicated therock tests are dropped. The one rocmd test that drove the temp-name check through a real write is kept. Count: 15 tests before (two of them ignored), 10 now (one ignored), with no assertion lost. Signed-off-by: Roman Inflianskas * test(runtimes): pin the active-runtime marker's atomic replacement `write_active_runtime_marker` used to stage `runtimes/active.json` at `active.json.tmp-` with a truncating write, remove the marker, then rename the temp file into place (#605). Two activations against one data dir in the same millisecond therefore wrote through one shared temp file. One of them failed with "failed to move active runtime marker". The other could publish its payload followed by the tail of a longer one, which later readers such as `rocm runtimes uninstall` could not parse. Between the remove and the rename, the marker did not exist at all. write itself does not change here beyond going through `rocm_core::atomic_write::write_file_atomically` (the one writer the previous commit moved into rocm-core) for both the forward write and the failed-activation restore. Each writer stages into its own `create_new` temp file named with pid, millis and attempt, and the marker is replaced in one step rather than removed first. The doc comment now says what that does and does not guarantee for two concurrent activations. Two regression tests pin it: - a deterministic one pre-creates a file at every name the old scheme could pick during the test and asserts none is written through or moved; - a Unix-only stress test runs two writers in lock-step plus a reader, and asserts every write succeeds and the reader always finds a complete marker. Against the pre-#424 body, 190 of 400 writes failed. Re-adding only the remove before the publish made the reader miss the marker thousands of times. Signed-off-by: Roman Inflianskas * docs(architecture): point at rocm-core's shared atomic writer The rocmd sandbox module no longer carries its own atomic-write helpers; record where the shared one lives so the next caller reuses it. Signed-off-by: Roman Inflianskas * style(core): give the atomic writer's test seams short doc summaries clippy's too_long_first_doc_paragraph rejects the two seam doc comments carried over from therock, where the private functions were not linted for it. Signed-off-by: Roman Inflianskas * fix(rocmd): gate std::fs import to the targets that still use it With the atomic writer moved to rocm-core, sandbox.rs uses std::fs only in its Linux bubblewrap path and its tests, so the Windows build warned about an unused import. Signed-off-by: Roman Inflianskas * ci(coverage): lower rocmd's floor for the tests that moved to rocm-core Moving the atomic writer out of rocmd's sandbox.rs took its helper code and six tests with it. Those lines were close to fully covered, and the line count includes test modules, so rocmd's percentage drops although nothing it still owns lost coverage. The code and its tests are now measured under rocm-core. Only rocm's tests reach rocmd, so `cargo llvm-cov --no-cfg-coverage -p rocm -p rocmd` measures rocmd exactly as the workspace run does. Measured locally that way, rocmd went from 75.30% (7776 lines) on the base to 75.02% (7549 lines) here. That is a 0.28pp drop, past the 0.2pp tolerance. The floor is set to the branch measurement. Only this one line is changed. A full `--bless` from a local run would also rewrite every other crate's floor from a different host's numbers. rocm moved by -0.04pp, inside tolerance, and is left as is. Signed-off-by: Roman Inflianskas * fix(rocmd): drop the windows-sys dependency the move left unused `ReplaceFileW` was rocmd's only use of `windows-sys`; it moved into `rocm_core::atomic_write` with the rest of the atomic writer, and rocm-core declares its own dependency. Also narrow the marker regression test's doc comment: it guards against a return to the millis-only temp names, while collision safety under the current names is covered by the concurrent tests. Signed-off-by: Roman Inflianskas * build(rocm): drop the windows-sys dependency the move left unused ReplaceFileW was apps/rocm's only use of windows-sys, and it moved into rocm_core::atomic_write with the rest of the writer; rocm-core declares its own dependency. Same leftover as the rocmd one removed earlier. Signed-off-by: Roman Inflianskas * fix(core): state what the atomic writer guarantees, and test the full disk The module doc now gives the real reason for ReplaceFileW (a sharing violation while another process holds the destination open; std rename does replace), says Windows concurrent publishes are untested, and says atomic but not durable, since nothing is fsynced. The test seams no external production code calls are narrowed: write_file_atomically_with is private, and the two that apps/rocm's cross-crate tests drive stay pub but #[doc(hidden)] and say they are seams. The staging doc no longer implies an outside caller can publish and clean up on its own. The full-disk test called disk_space::map_write_error directly, so it stayed green if staging stopped mapping errors. It now drives the real staging path with the byte write swapped for one that fails like a full disk, and checks the message and that no temp file is left. Signed-off-by: Roman Inflianskas * test(runtimes): describe the marker tests as what they pin after #424 #424 already fixed the temp-name collision and the missing-marker window on main. The two tests now say they pin that the marker keeps those properties through the shared writer, not that they are the regression for #605. The collision test no longer fails when the write lands after its pre-created names: it retries with a window sized from how long the pre-creation and write took, and where the filesystem is too slow for any window to cover the write it reports itself inconclusive and passes. The marker writer's doc says the Windows replace is untested under concurrency. Signed-off-by: Roman Inflianskas * docs(architecture): call atomic_write the shared writer and name exceptions It is not the only temp-then-rename writer: remote/session.rs keeps its own, and RocmCliConfig::save keeps its own until #541 moves it, tracked in #618. Also say it is atomic, not durable. Signed-off-by: Roman Inflianskas * fix(core): keep the new contents when a Windows replace fails part-way ReplaceFileW, called without a backup name, can fail after it has moved the old file off the destination (ERROR_UNABLE_TO_MOVE_REPLACEMENT and _2). The writer then deleted its temp file, the only remaining copy. On either code, retry once as a plain rename; if the destination that existed is still gone, keep the temp file and name it in the error. The Windows arm is type-checked by CI only. Also, from review: the module doc no longer claims the destination is never absent, and says which concurrency tests run on Windows; the multi-dot temp-name test captures the name at publish instead of after cleanup; the concurrent staging test forces the shared temp name so the collision happens every run; the restore path drops a create_dir_all the writer already does; a stale test-helper doc and the writer list in docs/architecture.md are corrected; the millis-only marker test is renamed to what it pins. Signed-off-by: Roman Inflianskas * fix(core): decide a kept replacement from the error, not a stale look A publish kept its temp file only when the destination had existed before the publish and was gone after it. A second writer that created the destination in between made that pre-publish look stale, and the temp file holding the only copy was deleted. The Windows partial-replace path now reports itself: when ReplaceFileW fails with ERROR_UNABLE_TO_MOVE_REPLACEMENT or _2 and the retried rename fails too, the publish returns a typed error, and the writer keeps the temp file on that error alone, returning a KeptReplacement that names it. The code check and retry are a plain function over the error code and an injected rename, so they run as unit tests on every platform; a Windows-only test ties the codes to windows-sys. A failed activation rolls the marker back, so the kept file then holds the marker of the activation that failed, and the advice to rename it into place would install that marker over the restored one. The rollback now removes it and rebuilds the error without that advice, on both the restored and the failed-restore path; a file it cannot remove is named with advice to delete it instead. The marker writer's doc no longer says a reader always finds a complete marker without naming the Windows exception. Signed-off-by: Roman Inflianskas * test(core): cover the writer giving up when every temp name is taken After 128 collisions the stager fails with an error that says so. No test reached that branch; this one makes every name collide and checks the bounded number of tries, the error, and that the other writer's file is left as it was. Signed-off-by: Roman Inflianskas * fix(core): name the destination, not the temp file, on a full disk A write that ran out of space named the staged temp file, which the writer has already removed by the time the user reads the message. The destination is the file they asked for and the one on the filesystem they need to free space on. Signed-off-by: Roman Inflianskas * test(rocm): drop the millis-only temp-name marker test The test pre-created files at every name the retired millis-only scheme could pick, then checked the marker write left them alone. When the write missed that window it reported "inconclusive" and passed, after creating up to thousands of files. It also guarded a naming scheme no longer in the tree, since the marker now goes through rocm_core::atomic_write. concurrent_active_runtime_marker_writes_keep_a_complete_marker_published and the writer's own collision tests carry the guarantee. Signed-off-by: Roman Inflianskas * test(runtimes): gate the marker test helpers to Unix Dropping the millis-only temp-name test left the concurrent marker test, which is Unix-only, as the sole user of test_active_runtime_marker and leftover_marker_temp_files. The Windows clippy job failed on both as dead code. Signed-off-by: Roman Inflianskas --------- Signed-off-by: Roman Inflianskas (cherry picked from commit b0224ecbf42a1b87cda1f6e18f040a46c5473b96) Signed-off-by: juhovainio --- Cargo.lock | 2 - apps/rocm/Cargo.toml | 1 - apps/rocm/src/runtimes_cmd.rs | 417 ++++++++++- apps/rocm/src/therock.rs | 12 - apps/rocm/src/therock/fetch.rs | 10 +- apps/rocm/src/therock/io.rs | 467 +----------- apps/rocm/src/therock/update.rs | 2 +- apps/rocmd/Cargo.toml | 3 - apps/rocmd/src/sandbox.rs | 352 +-------- coverage-floors.toml | 2 +- crates/rocm-core/src/atomic_write.rs | 1020 ++++++++++++++++++++++++++ crates/rocm-core/src/config.rs | 2 +- crates/rocm-core/src/lib.rs | 1 + docs/architecture.md | 11 +- 14 files changed, 1436 insertions(+), 866 deletions(-) create mode 100644 crates/rocm-core/src/atomic_write.rs diff --git a/Cargo.lock b/Cargo.lock index 7071a703eb..2151dc3607 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3741,7 +3741,6 @@ dependencies = [ "unicode-width", "ureq", "windows-native-keyring-store", - "windows-sys 0.61.2", "zbus-secret-service-keyring-store", ] @@ -3937,7 +3936,6 @@ dependencies = [ "sha2", "tokio", "ureq", - "windows-sys 0.61.2", ] [[package]] diff --git a/apps/rocm/Cargo.toml b/apps/rocm/Cargo.toml index e48eb7cf8a..7d53b8c7d4 100644 --- a/apps/rocm/Cargo.toml +++ b/apps/rocm/Cargo.toml @@ -65,7 +65,6 @@ libc.workspace = true [target.'cfg(windows)'.dependencies] self-replace = "1.5" windows-native-keyring-store = "1.1" -windows-sys = { version = "0.61", features = ["Win32_Storage_FileSystem"] } [target.'cfg(target_os = "macos")'.dependencies] apple-native-keyring-store = { version = "1.0", features = ["keychain"] } diff --git a/apps/rocm/src/runtimes_cmd.rs b/apps/rocm/src/runtimes_cmd.rs index 885e0bade4..731c1afee1 100644 --- a/apps/rocm/src/runtimes_cmd.rs +++ b/apps/rocm/src/runtimes_cmd.rs @@ -473,13 +473,9 @@ impl ActivationSnapshot { // failed, so it is the write least able to afford leaving a // truncated marker behind — which runtime resolution and the // storage retention holds would both read as "nothing is active". + // The helper creates the `runtimes` folder when it is missing. MarkerSnapshot::Contents(bytes) => { - let parent = path - .parent() - .context("active runtime marker path has no parent directory")?; - fs::create_dir_all(parent) - .with_context(|| format!("failed to create {}", parent.display()))?; - therock::write_file_atomically(&path, &bytes) + rocm_core::atomic_write::write_file_atomically(&path, &bytes) .with_context(|| format!("failed to restore {}", path.display()))?; } // Only a regular file is removed: whatever else may sit at that @@ -527,11 +523,11 @@ fn restore_after_failed_activation( error: anyhow::Error, ) -> anyhow::Error { match snapshot.restore(paths, config) { - Ok(()) => error.context( + Ok(()) => discard_failed_marker_replacement(error).context( "failed to record the active ROCm runtime; the previously active runtime is still \ the active one", ), - Err(restore_error) => error.context(format!( + Err(restore_error) => discard_failed_marker_replacement(error).context(format!( "failed to record the active ROCm runtime, and restoring the previous state failed \ too ({restore_error:#}); the ROCm CLI config and the active runtime marker may now \ disagree. Run `rocm runtimes list` to see both, then re-run `rocm runtimes activate \ @@ -540,6 +536,38 @@ fn restore_after_failed_activation( } } +/// Drop the marker a failed activation left behind, and the advice to use it. +/// +/// On Windows a marker publish can fail after taking the old marker away, and +/// the shared writer then keeps the new marker in its temp file and advises +/// renaming it into place (`rocm_core::atomic_write::KeptReplacement`). The +/// activation is being rolled back, so that file holds the marker of an +/// activation that failed, and renaming it would install that marker over +/// whatever the rollback restored. So the file is removed, and the error is +/// rebuilt without the advice. If the file cannot be removed, the error names +/// it and says to delete it instead. +fn discard_failed_marker_replacement(error: anyhow::Error) -> anyhow::Error { + let Some(kept) = error.downcast_ref::() else { + return error; + }; + let failure = format!( + "failed to write active runtime marker {}: {}", + kept.destination().display(), + kept.cause() + ); + match fs::remove_file(kept.kept()) { + Ok(()) => anyhow::anyhow!(failure), + Err(remove_error) if remove_error.kind() == std::io::ErrorKind::NotFound => { + anyhow::anyhow!(failure) + } + Err(remove_error) => anyhow::anyhow!( + "{failure}; {} holds the marker of the activation that failed and could not be \ + removed ({remove_error}): delete it, do not rename it into place", + kept.kept().display() + ), + } +} + /// Marker shown beside the active runtime in `rocm runtimes list`. Every /// place that renders this glyph MUST use this constant so the rendered /// character and the legend text stay in sync. @@ -843,12 +871,20 @@ pub(crate) const fn toolchain_state_text(includes_devel: bool) -> &'static str { } } +/// Replace the active-runtime marker in one step. +/// +/// Two `rocm` processes can activate a runtime against the same data +/// directory at once (two terminals, a script), so the write goes through the +/// shared atomic writer: each writer stages into its own exclusively created +/// temp file, and the marker is replaced rather than removed first. On Unix, +/// and on Windows whenever `ReplaceFileW` succeeds, a reader therefore always +/// finds a complete marker. A `ReplaceFileW` that fails part-way is the +/// exception: it can leave the marker missing, which the writer repairs with a +/// plain rename when that rename succeeds (see `rocm_core::atomic_write`); two +/// replacements in flight at once on Windows are not tested. It does not order +/// the two activations: the last one to publish wins, as before. fn write_active_runtime_marker(paths: &AppPaths, marker: ActiveRuntimeMarker) -> Result<()> { let path = active_runtime_marker_path(paths); - fs::create_dir_all( - path.parent() - .context("active runtime marker path has no parent directory")?, - )?; // The shared helper rather than a local temp-file-and-rename: it is the same // write the failed-activation restore uses, and it is strictly stronger than // a bare `fs::rename`, which on Windows is `MoveFileExW` and fails with a @@ -858,13 +894,16 @@ fn write_active_runtime_marker(paths: &AppPaths, marker: ActiveRuntimeMarker) -> // window with no marker at all, which runtime resolution and the storage // retention holds both read as "nothing is active" — and a failed publish // takes its temp file with it, so a retried activation cannot strand one in - // the folder `rocm runtimes` reads. - therock::write_file_atomically( - &path, - &serde_json::to_vec_pretty(&marker).context("failed to serialize active runtime marker")?, - ) - .with_context(|| format!("failed to write {}", path.display()))?; - Ok(()) + // the folder `rocm runtimes` reads. The one exception is a Windows publish + // that took the old marker away and could not put the new one in place: + // the temp file is then the only copy, so it is kept and the error names + // it; a failed activation removes it again once the restore has put the + // previous marker back (`discard_failed_marker_replacement`). It also + // creates the `runtimes` folder when it is missing. + let bytes = + serde_json::to_vec_pretty(&marker).context("failed to serialize active runtime marker")?; + rocm_core::atomic_write::write_file_atomically(&path, &bytes) + .with_context(|| format!("failed to write active runtime marker {}", path.display())) } #[cfg(test)] @@ -1194,6 +1233,146 @@ pub(crate) mod tests { Ok(()) } + #[cfg(unix)] + fn test_active_runtime_marker(paths: &AppPaths, runtime_key: &str) -> ActiveRuntimeMarker { + ActiveRuntimeMarker { + runtime_id: "therock-release:gfx120X-all".to_owned(), + runtime_key: runtime_key.to_owned(), + manifest_path: runtime_manifest_path(paths, runtime_key), + install_root: paths.data_dir.join("runtimes").join(runtime_key), + previous_runtime_id: None, + previous_runtime_key: None, + activated_at_unix_ms: rocm_core::unix_time_millis(), + } + } + + #[cfg(unix)] + fn leftover_marker_temp_files(paths: &AppPaths) -> Result> { + let dir = active_runtime_marker_path(paths) + .parent() + .context("marker has a parent")? + .to_path_buf(); + let mut leftovers = Vec::new(); + for entry in fs::read_dir(dir)? { + let name = entry?.file_name().to_string_lossy().into_owned(); + if name.starts_with("active.json.tmp-") { + leftovers.push(name); + } + } + Ok(leftovers) + } + + /// Two activations writing the marker at once must both succeed, and a + /// reader must find a complete marker throughout. #605 reported a writer + /// that removed `active.json` before renaming the replacement in, and two + /// writers in the same millisecond sharing one temp file, so one failed + /// with "failed to move active runtime marker" and the other could publish + /// the shorter payload followed by the longer one's tail. #424 fixed that; + /// this change routes the marker through `rocm_core::atomic_write`, the + /// writer `rocm` and `rocmd` now share, and this test pins that the marker + /// keeps the property through it. + /// + /// The bug is a race, so against the old code this fails with high + /// probability per round rather than deterministically; against the + /// current writer nothing in it is timing-dependent. Unix only: the + /// property under test is that `rename` replaces in one step. The Windows + /// publish path goes through `ReplaceFileW` instead, whose behaviour with + /// two replacements of one file in flight this test has not been run + /// against. + #[cfg(unix)] + #[test] + fn concurrent_active_runtime_marker_writes_keep_a_complete_marker_published() -> Result<()> { + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::{Arc, Barrier}; + + const ROUNDS: usize = 200; + let (root, paths) = test_paths("marker-concurrent"); + write_active_runtime_marker(&paths, test_active_runtime_marker(&paths, "initial"))?; + let marker_path = active_runtime_marker_path(&paths); + + let done = Arc::new(AtomicBool::new(false)); + let reader = { + let done = Arc::clone(&done); + let marker_path = marker_path.clone(); + std::thread::spawn(move || { + let mut problems = Vec::new(); + while !done.load(Ordering::Acquire) { + match fs::read(&marker_path) { + Ok(bytes) => { + if let Err(error) = + serde_json::from_slice::(&bytes) + { + problems.push(format!("unparsable marker: {error}")); + } + } + Err(error) => problems.push(format!("marker unreadable: {error}")), + } + } + problems + }) + }; + + let barrier = Arc::new(Barrier::new(2)); + let writers: Vec<_> = [ + "short", + "a-runtime-key-long-enough-to-leave-a-tail-behind-a-shorter-payload", + ] + .into_iter() + .map(|runtime_key| { + let barrier = Arc::clone(&barrier); + let paths = paths.clone(); + std::thread::spawn(move || { + let mut errors = Vec::new(); + for _ in 0..ROUNDS { + barrier.wait(); + if let Err(error) = write_active_runtime_marker( + &paths, + test_active_runtime_marker(&paths, runtime_key), + ) { + errors.push(format!("{error:#}")); + } + } + errors + }) + }) + .collect(); + + let mut writer_errors = Vec::new(); + for writer in writers { + writer_errors.extend(writer.join().expect("writer thread panicked")); + } + done.store(true, Ordering::Release); + let reader_problems = reader.join().expect("reader thread panicked"); + + let published: ActiveRuntimeMarker = serde_json::from_slice(&fs::read(&marker_path)?)?; + let leftovers = leftover_marker_temp_files(&paths)?; + let _ = fs::remove_dir_all(root); + + assert!( + writer_errors.is_empty(), + "{} of {} marker writes failed; first: {:?}", + writer_errors.len(), + 2 * ROUNDS, + writer_errors.first() + ); + assert!( + reader_problems.is_empty(), + "a reader found no complete marker {} times; first: {:?}", + reader_problems.len(), + reader_problems.first() + ); + assert!( + [ + "short", + "a-runtime-key-long-enough-to-leave-a-tail-behind-a-shorter-payload" + ] + .contains(&published.runtime_key.as_str()), + "unexpected marker: {published:?}" + ); + assert_eq!(leftovers, Vec::::new()); + Ok(()) + } + #[test] fn runtime_activation_records_exact_key_and_rollback() -> Result<()> { let (root, paths) = test_paths("runtime-activation"); @@ -1299,10 +1478,10 @@ pub(crate) mod tests { } /// Replace the active-runtime marker with a NON-EMPTY directory, so the - /// next `write_active_runtime_marker` fails at a predictable point: its - /// `create_dir_all` of the parent succeeds, and the final `fs::rename` of - /// the temp file onto a non-empty directory fails on both supported - /// platforms without needing privileges or a real full disk. + /// next `write_active_runtime_marker` fails at a predictable point: the + /// temp file is staged beside it, and publishing that file onto a + /// non-empty directory fails on both supported platforms without needing + /// privileges or a real full disk. fn break_active_runtime_marker(paths: &AppPaths) -> Result { let marker = active_runtime_marker_path(paths); let _ = fs::remove_file(&marker); @@ -1994,6 +2173,198 @@ pub(crate) mod tests { Ok(()) } + /// The Windows partial-replace failure during activation: the marker publish + /// took the old marker away and kept the new one in its temp file. Once the + /// restore has put the previous marker back, the kept file holds the failed + /// activation's marker, so it is removed, and the error no longer tells the + /// user to rename it into place. Driven through the shared writer's publish + /// seam, so it runs on every platform. + #[test] + fn a_restored_activation_drops_the_marker_a_partial_replace_kept() -> Result<()> { + let (root, paths) = test_paths("activation-partial-replace"); + paths.ensure()?; + let mut config = activate_old_runtime(&paths)?; + let snapshot = ActivationSnapshot::capture(&paths, &config); + let previous_marker = fs::read(active_runtime_marker_path(&paths))?; + config.active_runtime_key = Some(NEW_RUNTIME_KEY.to_owned()); + config.previous_runtime_key = Some(OLD_RUNTIME_KEY.to_owned()); + config.save(&paths)?; + + let marker_path = active_runtime_marker_path(&paths); + let write_error = partial_replace_marker_error(&paths, |_| Ok(())); + assert!( + !marker_temp_file_names(&paths)?.is_empty(), + "the shared writer keeps the new marker when the old one is gone" + ); + + let error = restore_after_failed_activation(&paths, &mut config, snapshot, write_error); + let rendered = format!("{error:#}"); + let restored_marker = fs::read(&marker_path)?; + let leftovers = marker_temp_file_names(&paths)?; + let persisted = RocmCliConfig::load(&paths)?; + let _ = fs::remove_dir_all(&root); + + assert_eq!(restored_marker, previous_marker); + assert_eq!( + persisted.active_runtime_key.as_deref(), + Some(OLD_RUNTIME_KEY) + ); + assert!( + leftovers.is_empty(), + "the failed activation's marker was left beside the restored one: {leftovers:?}" + ); + assert!( + rendered.contains("the previously active runtime is still the active one"), + "{rendered}" + ); + assert!( + !rendered.contains("rename that file"), + "renaming would install the failed activation's marker:\n{rendered}" + ); + Ok(()) + } + + /// Produce the error the shared writer returns when a marker publish took + /// the old marker away and kept the new one, through its publish seam. + /// `prepare_kept` runs on the kept temp file before the error is returned. + fn partial_replace_marker_error( + paths: &AppPaths, + prepare_kept: impl FnOnce(&std::path::Path) -> std::io::Result<()>, + ) -> anyhow::Error { + let marker_path = active_runtime_marker_path(paths); + rocm_core::atomic_write::write_file_atomically_with_publish( + &marker_path, + b"{\"runtime_key\":\"new\"}", + |attempt| std::ffi::OsString::from(format!("partial-{attempt}")), + || {}, + |tmp, path| { + fs::remove_file(path)?; + prepare_kept(tmp)?; + Err(rocm_core::atomic_write::destination_removed_error( + std::io::Error::from_raw_os_error(1176), + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + )) + }, + ) + .expect_err("the simulated partial replace fails the write") + .context(format!( + "failed to write active runtime marker {}", + marker_path.display() + )) + } + + /// When the kept marker cannot be removed, the error names it and says to + /// delete it, and nothing in it still says to rename it into place. The + /// kept path is made a non-empty directory, which `remove_file` refuses on + /// every platform. + #[test] + fn a_kept_marker_that_cannot_be_removed_is_named_with_delete_advice_only() -> Result<()> { + let (root, paths) = test_paths("activation-partial-replace-stuck"); + paths.ensure()?; + let mut config = activate_old_runtime(&paths)?; + let snapshot = ActivationSnapshot::capture(&paths, &config); + let previous_marker = fs::read(active_runtime_marker_path(&paths))?; + config.active_runtime_key = Some(NEW_RUNTIME_KEY.to_owned()); + config.save(&paths)?; + let write_error = partial_replace_marker_error(&paths, |tmp| { + fs::remove_file(tmp)?; + fs::create_dir(tmp)?; + fs::write(tmp.join("occupied.txt"), "stuck") + }); + let kept = marker_temp_file_names(&paths)?; + + let error = restore_after_failed_activation(&paths, &mut config, snapshot, write_error); + let rendered = format!("{error:#}"); + let restored_marker = fs::read(active_runtime_marker_path(&paths))?; + let still_there = marker_temp_file_names(&paths)?; + let _ = fs::remove_dir_all(&root); + + assert_eq!(kept.len(), 1, "{kept:?}"); + assert_eq!(still_there, kept, "the stuck file is left where it was"); + assert_eq!(restored_marker, previous_marker); + assert!( + rendered.contains(&kept[0]), + "the stuck file is named:\n{rendered}" + ); + assert!( + rendered.contains("delete it, do not rename it"), + "{rendered}" + ); + assert!( + !rendered.contains("rename that file"), + "renaming would install the failed activation's marker:\n{rendered}" + ); + Ok(()) + } + + /// A kept marker that is already gone by the time the rollback looks for it + /// is as good as removed: the error neither names it nor tells the user to + /// delete or rename it. + #[test] + fn a_kept_marker_already_gone_is_treated_as_removed() -> Result<()> { + let (root, paths) = test_paths("activation-partial-replace-gone"); + paths.ensure()?; + let mut config = activate_old_runtime(&paths)?; + let snapshot = ActivationSnapshot::capture(&paths, &config); + config.active_runtime_key = Some(NEW_RUNTIME_KEY.to_owned()); + let write_error = partial_replace_marker_error(&paths, |_| Ok(())); + let kept = marker_temp_file_names(&paths)?; + let runtimes_dir = active_runtime_marker_path(&paths) + .parent() + .context("active runtime marker path has no parent directory")? + .to_path_buf(); + for name in &kept { + fs::remove_file(runtimes_dir.join(name))?; + } + + let error = restore_after_failed_activation(&paths, &mut config, snapshot, write_error); + let rendered = format!("{error:#}"); + let _ = fs::remove_dir_all(&root); + + assert_eq!(kept.len(), 1, "{kept:?}"); + assert!( + rendered.contains("the previously active runtime is still the active one"), + "{rendered}" + ); + assert!(!rendered.contains(&kept[0]), "{rendered}"); + assert!(!rendered.contains("rename that file"), "{rendered}"); + assert!(!rendered.contains("delete it"), "{rendered}"); + Ok(()) + } + + /// When the rollback itself fails as well, the kept marker is still the + /// failed activation's, so it goes too, and the error keeps the rollback's + /// own recovery advice without the advice to rename it. The config path is + /// made a non-empty directory so the restore's config save fails. + #[test] + fn a_failed_rollback_still_drops_the_marker_a_partial_replace_kept() -> Result<()> { + let (root, paths) = test_paths("activation-partial-replace-double-fault"); + paths.ensure()?; + let mut config = activate_old_runtime(&paths)?; + let snapshot = ActivationSnapshot::capture(&paths, &config); + config.active_runtime_key = Some(NEW_RUNTIME_KEY.to_owned()); + let write_error = partial_replace_marker_error(&paths, |_| Ok(())); + assert!(!marker_temp_file_names(&paths)?.is_empty()); + let config_path = paths.config_path(); + let _ = fs::remove_file(&config_path); + fs::create_dir_all(&config_path)?; + fs::write(config_path.join("occupied.txt"), "not a config")?; + + let error = restore_after_failed_activation(&paths, &mut config, snapshot, write_error); + let rendered = format!("{error:#}"); + let leftovers = marker_temp_file_names(&paths)?; + let _ = fs::remove_dir_all(&root); + + assert!( + rendered.contains("restoring the previous state failed too"), + "{rendered}" + ); + assert!(rendered.contains("rocm runtimes list"), "{rendered}"); + assert!(!rendered.contains("rename that file"), "{rendered}"); + assert!(leftovers.is_empty(), "{leftovers:?}"); + Ok(()) + } + // A restore that fails too must say so and name the way out, rather than // claim the previous runtime is still the active one. The marker's parent is // made a regular file, so writing the captured bytes back fails at diff --git a/apps/rocm/src/therock.rs b/apps/rocm/src/therock.rs index 1cfe879e86..3371bbdb52 100644 --- a/apps/rocm/src/therock.rs +++ b/apps/rocm/src/therock.rs @@ -33,7 +33,6 @@ pub(crate) use self::consent::{ use self::family::{family_override_or_recovered_arch, resolve_family}; pub(crate) use self::fetch::truthy_env; use self::fetch::{download_file, preflight_tarball_space}; -pub(crate) use self::io::write_file_atomically; use self::probe::probe_rocm_sdk_runtime_for_target; pub(crate) use self::probe::{ RuntimeDeviceProbe, TorchAlignmentProbe, install_pinned_package, probe_rocm_sdk_runtime, @@ -1161,17 +1160,6 @@ mod tests { assert_eq!(estimate, archive * disk_space::EXTRACTED_SIZE_MULTIPLIER); } - #[test] - fn write_file_atomically_reports_a_full_disk_clearly() { - // Exercise the mapping the write path uses, without filling a disk. - let error = disk_space::map_write_error( - std::io::Error::from(std::io::ErrorKind::StorageFull), - Path::new("/cache/rocm.tar.gz.tmp"), - ); - let text = format!("{error:#}"); - assert!(text.contains("ran out of disk space"), "{text}"); - } - #[test] fn normalize_therock_family_maps_gfx1103_to_gfx110x_all() { assert_eq!( diff --git a/apps/rocm/src/therock/fetch.rs b/apps/rocm/src/therock/fetch.rs index 1257042bc0..6829709e25 100644 --- a/apps/rocm/src/therock/fetch.rs +++ b/apps/rocm/src/therock/fetch.rs @@ -8,14 +8,14 @@ //! //! Owns the pinned metadata trust root, //! which `cargo xtask verify-pinned-keys` checks against the canonical -//! published key. Depends on the sibling `io` module for atomic writes and on -//! `slugify` in the parent for cache file names. +//! published key. Depends on `rocm_core::atomic_write` for atomic writes and +//! on `slugify` in the parent for cache file names. -#[cfg(test)] -use super::io::{publish_temp_file, write_file_atomically_with_publish}; -use super::io::{stage_file_for_atomic_publish, write_file_atomically}; use super::slugify; use anyhow::{Context, Result, bail}; +#[cfg(test)] +use rocm_core::atomic_write::{publish_temp_file, write_file_atomically_with_publish}; +use rocm_core::atomic_write::{stage_file_for_atomic_publish, write_file_atomically}; use rocm_core::{AppPaths, disk_space, unix_time_millis, verify_rsa_pkcs1_sha256_signature}; use serde::{Deserialize, Serialize}; #[cfg(test)] diff --git a/apps/rocm/src/therock/io.rs b/apps/rocm/src/therock/io.rs index b4d3d06f72..9347220aee 100644 --- a/apps/rocm/src/therock/io.rs +++ b/apps/rocm/src/therock/io.rs @@ -2,21 +2,15 @@ // // SPDX-License-Identifier: MIT -//! Filesystem helpers behind TheRock installs: unique scratch directories and -//! the stage-then-publish atomic file writes. The writes serve the metadata -//! cache entries and the staged metadata public key, the startup update-check -//! record, and (through `therock`'s re-export of `write_file_atomically`) the -//! active-runtime marker that `runtimes_cmd` maintains. Runtime registry -//! manifests do not go through these writers. +//! Unique scratch directories behind TheRock installs. The atomic file writes +//! that used to live here are `rocm_core::atomic_write`, shared with `rocmd`. //! //! A leaf of the `therock` module: nothing here depends on the rest of it. use anyhow::{Context, Result, bail}; -use rocm_core::{disk_space, unix_time_millis}; -use std::ffi::{OsStr, OsString}; +use rocm_core::unix_time_millis; use std::fs; -use std::io::{self, Write}; -use std::path::{Path, PathBuf}; +use std::path::PathBuf; pub(super) fn linux_temp_dir(prefix: &str) -> Result { let root = std::env::temp_dir(); @@ -69,456 +63,3 @@ fn windows_runtime_temp_root() -> Option { } None } - -/// A unique temp path next to `path`, preserving the full file name so a -/// multi-extension artifact keeps its extensions (`sdk.tar.gz` becomes -/// `sdk.tar.gz.tmp-`, where `with_extension` would drop `.gz`). -const ATOMIC_WRITE_TEMP_ATTEMPTS: u32 = 128; - -fn temp_sibling_path(path: &Path, suffix: &OsStr) -> Result { - let parent = path.parent().context("file path has no parent directory")?; - let mut file_name = path - .file_name() - .context("file path has no file name")? - .to_os_string(); - file_name.push(".tmp-"); - file_name.push(suffix); - Ok(parent.join(file_name)) -} - -pub(crate) fn write_file_atomically(path: &Path, bytes: &[u8]) -> Result<()> { - let temp_id = format!("{}-{}", std::process::id(), unix_time_millis()); - write_file_atomically_with( - path, - bytes, - |attempt| OsString::from(format!("{temp_id}-{attempt}")), - || {}, - ) -} - -fn write_file_atomically_with( - path: &Path, - bytes: &[u8], - suffix_for_attempt: S, - before_publish: P, -) -> Result<()> -where - S: FnMut(u32) -> OsString, - P: FnOnce(), -{ - write_file_atomically_with_publish( - path, - bytes, - suffix_for_attempt, - before_publish, - publish_temp_file, - ) -} - -pub(super) fn write_file_atomically_with_publish( - path: &Path, - bytes: &[u8], - suffix_for_attempt: S, - before_publish: P, - publish: F, -) -> Result<()> -where - S: FnMut(u32) -> OsString, - P: FnOnce(), - F: FnOnce(&Path, &Path) -> io::Result<()>, -{ - let tmp = stage_file_for_atomic_publish_with(path, bytes, suffix_for_attempt)?; - before_publish(); - publish_staged_file_with(&tmp, path, publish) -} - -pub(super) fn stage_file_for_atomic_publish(path: &Path, bytes: &[u8]) -> Result { - let temp_id = format!("{}-{}", std::process::id(), unix_time_millis()); - stage_file_for_atomic_publish_with(path, bytes, |attempt| { - OsString::from(format!("{temp_id}-{attempt}")) - }) -} - -fn stage_file_for_atomic_publish_with( - path: &Path, - bytes: &[u8], - mut suffix_for_attempt: S, -) -> Result -where - S: FnMut(u32) -> OsString, -{ - let parent = path.parent().context("file path has no parent directory")?; - fs::create_dir_all(parent)?; - - let mut reserved = None; - for attempt in 0..ATOMIC_WRITE_TEMP_ATTEMPTS { - let suffix = suffix_for_attempt(attempt); - let tmp = temp_sibling_path(path, &suffix)?; - match fs::OpenOptions::new() - .write(true) - .create_new(true) - .open(&tmp) - { - Ok(file) => { - reserved = Some((tmp, file)); - break; - } - Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} - Err(error) => { - return Err(error).with_context(|| format!("failed to create {}", tmp.display())); - } - } - } - let Some((tmp, mut file)) = reserved else { - bail!( - "failed to reserve a temporary file next to {} after {} attempts", - path.display(), - ATOMIC_WRITE_TEMP_ATTEMPTS - ); - }; - - if let Err(error) = file.write_all(bytes) { - drop(file); - let _ = fs::remove_file(&tmp); - return Err(disk_space::map_write_error(error, &tmp)); - } - drop(file); - Ok(tmp) -} - -#[cfg(test)] -fn publish_staged_file(tmp: &Path, path: &Path) -> Result<()> { - publish_staged_file_with(tmp, path, publish_temp_file) -} - -fn publish_staged_file_with(tmp: &Path, path: &Path, publish: F) -> Result<()> -where - F: FnOnce(&Path, &Path) -> io::Result<()>, -{ - publish(tmp, path) - .inspect_err(|_| { - let _ = fs::remove_file(tmp); - }) - .with_context(|| format!("failed to publish {}", path.display())) -} - -#[cfg(not(windows))] -pub(super) fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { - fs::rename(tmp, path) -} - -#[cfg(windows)] -pub(super) fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { - if path.try_exists()? { - return replace_file_windows(path, tmp); - } - - match fs::rename(tmp, path) { - Ok(()) => Ok(()), - Err(rename_error) => { - if path.try_exists()? { - replace_file_windows(path, tmp) - } else { - Err(rename_error) - } - } - } -} - -#[cfg(windows)] -#[allow(unsafe_code)] -fn replace_file_windows(path: &Path, replacement: &Path) -> io::Result<()> { - use std::os::windows::ffi::OsStrExt; - use windows_sys::Win32::Storage::FileSystem::ReplaceFileW; - - let path_wide: Vec = path.as_os_str().encode_wide().chain(Some(0)).collect(); - let replacement_wide: Vec = replacement - .as_os_str() - .encode_wide() - .chain(Some(0)) - .collect(); - - // SAFETY: both path buffers are valid, NUL-terminated UTF-16 strings and - // remain alive for the duration of the synchronous Windows API call. The - // optional backup, exclude, and reserved pointers are intentionally null. - let replaced = unsafe { - ReplaceFileW( - path_wide.as_ptr(), - replacement_wide.as_ptr(), - std::ptr::null(), - 0, - std::ptr::null(), - std::ptr::null(), - ) - }; - if replaced == 0 { - Err(io::Error::last_os_error()) - } else { - Ok(()) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - /// The temp name keeps every extension, so a cleanup sweep over a cache - /// directory can still tell what a leftover was going to be. - #[test] - fn temp_sibling_path_preserves_multi_dot_file_names() { - let temp = temp_sibling_path( - Path::new("/tmp/cache/sdk.tar.gz"), - std::ffi::OsStr::new("test"), - ) - .unwrap(); - let name = temp.file_name().unwrap().to_string_lossy().into_owned(); - assert_eq!(name, "sdk.tar.gz.tmp-test"); - assert_eq!(temp.parent().unwrap(), Path::new("/tmp/cache")); - } - - #[test] - fn concurrent_atomic_writes_do_not_remove_a_published_destination() { - let root = std::env::temp_dir().join(format!( - "rocm-atomic-collision-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("sdk.tar.gz"); - let before_publish = std::sync::Arc::new(std::sync::Barrier::new(2)); - - let writers: Vec<_> = [b"first".as_slice(), b"second".as_slice()] - .into_iter() - .enumerate() - .map(|(writer, bytes)| { - let destination = destination.clone(); - let before_publish = std::sync::Arc::clone(&before_publish); - std::thread::spawn(move || { - write_file_atomically_with( - &destination, - bytes, - |attempt| { - if attempt == 0 { - std::ffi::OsString::from("same-millisecond") - } else { - std::ffi::OsString::from(format!( - "same-millisecond-{writer}-{attempt}" - )) - } - }, - || { - before_publish.wait(); - }, - ) - }) - }) - .collect(); - - for writer in writers { - writer.join().unwrap().unwrap(); - } - let published = fs::read(&destination).expect("a writer must remain published"); - let _ = fs::remove_dir_all(&root); - assert!(published == b"first" || published == b"second"); - } - - #[test] - fn concurrent_cached_publications_use_distinct_staging_files() { - let root = std::env::temp_dir().join(format!( - "rocm-cache-publish-collision-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("index.body"); - let before_publish = std::sync::Arc::new(std::sync::Barrier::new(2)); - - let writers: Vec<_> = [b"first".as_slice(), b"second".as_slice()] - .into_iter() - .map(|bytes| { - let destination = destination.clone(); - let before_publish = std::sync::Arc::clone(&before_publish); - std::thread::spawn(move || { - let staged = stage_file_for_atomic_publish(&destination, bytes)?; - before_publish.wait(); - publish_staged_file(&staged, &destination) - }) - }) - .collect(); - - for writer in writers { - writer.join().unwrap().unwrap(); - } - let published = fs::read(&destination).expect("a cache writer must remain published"); - let leftovers: Vec<_> = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name()) - .filter(|name| name.to_string_lossy().contains(".tmp-")) - .collect(); - let _ = fs::remove_dir_all(&root); - assert!(published == b"first" || published == b"second"); - assert!( - leftovers.is_empty(), - "staged cache files leaked: {leftovers:?}" - ); - } - - #[test] - fn failed_cached_publication_preserves_destination_and_cleans_staging_file() { - let root = std::env::temp_dir().join(format!( - "rocm-cache-publish-failure-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("index.body"); - fs::write(&destination, b"published").unwrap(); - let staged = stage_file_for_atomic_publish(&destination, b"replacement").unwrap(); - - publish_staged_file_with(&staged, &destination, |_, _| { - Err(io::Error::new( - io::ErrorKind::PermissionDenied, - "simulated cache publication failure", - )) - }) - .expect_err("simulated cache publication failure must be returned"); - - assert_eq!(fs::read(&destination).unwrap(), b"published"); - assert!( - !staged.exists(), - "failed publication leaked its staging file" - ); - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn failed_atomic_replace_preserves_destination_and_cleans_temp() { - let root = std::env::temp_dir().join(format!( - "rocm-atomic-replace-failure-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("sdk.tar.gz"); - fs::write(&destination, b"published").unwrap(); - - write_file_atomically_with_publish( - &destination, - b"replacement", - |attempt| OsString::from(format!("replace-failure-{attempt}")), - || {}, - |tmp, path| { - assert_eq!(fs::read(tmp).unwrap(), b"replacement"); - assert_eq!(path, destination); - Err(std::io::Error::new( - std::io::ErrorKind::PermissionDenied, - "simulated atomic replacement failure", - )) - }, - ) - .expect_err("simulated replacement failure must be returned"); - - assert_eq!(fs::read(&destination).unwrap(), b"published"); - let leftovers: Vec<_> = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name()) - .collect(); - let _ = fs::remove_dir_all(&root); - assert_eq!(leftovers, vec![OsString::from("sdk.tar.gz")]); - } - - #[cfg(unix)] - #[test] - fn atomic_temp_name_preserves_non_unicode_file_name_bytes() { - use std::os::unix::ffi::{OsStrExt, OsStringExt}; - - let file_name = std::ffi::OsString::from_vec(b"sdk-\xff.tar.gz".to_vec()); - let destination = Path::new("/tmp").join(&file_name); - let temp = temp_sibling_path(&destination, std::ffi::OsStr::new("collision")).unwrap(); - - let mut expected = file_name.into_vec(); - expected.extend_from_slice(b".tmp-collision"); - assert_eq!(temp.file_name().unwrap().as_bytes(), expected); - } - - /// Regression: a failed write must not leave a `.tmp-*` scratch file - /// behind. The name is unique per attempt, so before this an orphan - /// accumulated per retry — and when the failure is a full disk, those - /// orphans are exactly what keeps it full. - /// - /// Provokes the failure by pointing the destination at a non-empty - /// directory: the temp file is written, then neither the rename nor the - /// replace fallback can succeed. Portable, unlike an out-of-space test. - #[test] - fn write_file_atomically_cleans_up_temp_when_the_rename_fails() { - let root = std::env::temp_dir().join(format!( - "rocm-atomic-{}-{}", - std::process::id(), - unix_time_millis() - )); - let occupied = root.join("sdk.tar.gz"); - fs::create_dir_all(occupied.join("nested")).unwrap(); - fs::write(occupied.join("nested").join("keep"), b"x").unwrap(); - - write_file_atomically(&occupied, b"payload") - .expect_err("renaming onto a non-empty directory should fail"); - - let leftovers: Vec = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) - .filter(|name| name.contains(".tmp-")) - .collect(); - let _ = fs::remove_dir_all(&root); - assert!( - leftovers.is_empty(), - "failed write left temp files behind: {leftovers:?}" - ); - } - - /// Mirrors the `/dev/shm` reproduction from the original report: a genuine - /// ENOSPC, not a rename failure standing in for one. - /// - /// Ignored by default because it fills `/dev/shm`, which is shared with - /// anything else on the host, so it is not safe to run concurrently. Run - /// with `cargo test -p rocm -- --ignored write_file_atomically_cleans_up`. - #[test] - #[ignore = "fills /dev/shm to provoke ENOSPC; not safe to run concurrently"] - fn write_file_atomically_cleans_up_temp_on_write_failure() { - let shm = Path::new("/dev/shm"); - if !shm.is_dir() { - eprintln!("skipping: /dev/shm unavailable"); - return; - } - let dir = shm.join(format!("rocm-enospc-{}", std::process::id())); - let _ = fs::remove_dir_all(&dir); - fs::create_dir_all(&dir).unwrap(); - let dest = dir.join("artifact.tar.gz"); - // Larger than the tmpfs, so the write is guaranteed to hit ENOSPC. - let payload = vec![0u8; 256 * 1024 * 1024]; - - let mut failures = Vec::new(); - for _ in 0..2 { - write_file_atomically(&dest, &payload) - .expect_err("writing past the end of the filesystem should fail"); - failures.push( - fs::read_dir(&dir) - .unwrap() - .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) - .collect::>(), - ); - } - let destination_exists = dest.exists(); - let _ = fs::remove_dir_all(&dir); - - for leftovers in &failures { - assert!( - leftovers.is_empty(), - "failed write left files behind: {leftovers:?}" - ); - } - assert!( - !destination_exists, - "destination must not exist after failure" - ); - } -} diff --git a/apps/rocm/src/therock/update.rs b/apps/rocm/src/therock/update.rs index 5d49245313..08060f6deb 100644 --- a/apps/rocm/src/therock/update.rs +++ b/apps/rocm/src/therock/update.rs @@ -23,7 +23,6 @@ //! install and activation paths share. use super::family::family_override_or_recovered_arch; -use super::io::write_file_atomically; use super::process::SuppressProgressOutput; use super::python::{ PythonLauncher, PythonSource, python_launcher_install_ready, python_requirement, @@ -41,6 +40,7 @@ use crate::runtime_registry::{ wheel_composition_device_target, }; use anyhow::{Context, Result, bail}; +use rocm_core::atomic_write::write_file_atomically; use rocm_core::{AppPaths, unix_time_millis}; use serde::{Deserialize, Serialize}; use std::cmp::Ordering; diff --git a/apps/rocmd/Cargo.toml b/apps/rocmd/Cargo.toml index 2f0064e18a..945a47e903 100644 --- a/apps/rocmd/Cargo.toml +++ b/apps/rocmd/Cargo.toml @@ -24,6 +24,3 @@ serde_json.workspace = true sha2 = "0.10" tokio.workspace = true ureq = { version = "2.12", features = ["native-certs"] } - -[target.'cfg(windows)'.dependencies] -windows-sys = { version = "0.61", features = ["Win32_Storage_FileSystem"] } diff --git a/apps/rocmd/src/sandbox.rs b/apps/rocmd/src/sandbox.rs index 01b6d0ab71..9eaf6c8c4a 100644 --- a/apps/rocmd/src/sandbox.rs +++ b/apps/rocmd/src/sandbox.rs @@ -8,6 +8,7 @@ use crate::sandbox_tool::{SandboxToolArg, SandboxToolPolicy}; use crate::service::{stop_managed_service, stop_report_confirmed}; use crate::supervise::restart_managed_service; use anyhow::{Context, Result, bail}; +use rocm_core::atomic_write::write_file_atomically; use rocm_core::{ AppPaths, AuditEventRecord, CommandCapture, ExamineSummary, ModelRecipeArtifactRecord, append_audit_event, model_artifact_cache_status, resolve_model_recipe_artifact, @@ -17,9 +18,8 @@ use serde_json::Value; use serde_json::json; #[cfg(test)] use sha2::Digest; -use std::ffi::{OsStr, OsString}; +#[cfg(any(test, target_os = "linux"))] use std::fs; -use std::io::{self, Write}; use std::path::{Path, PathBuf}; #[cfg(target_os = "linux")] use std::process::{Command as ProcessCommand, Stdio}; @@ -667,160 +667,6 @@ fn sha256_hex(bytes: &[u8]) -> String { format!("{:x}", sha2::Sha256::digest(bytes)) } -const ATOMIC_WRITE_TEMP_ATTEMPTS: u32 = 128; - -fn temp_sibling_path(path: &Path, suffix: &OsStr) -> Result { - let parent = path.parent().context("file path has no parent directory")?; - let mut file_name = path - .file_name() - .context("file path has no file name")? - .to_os_string(); - file_name.push(".tmp-"); - file_name.push(suffix); - Ok(parent.join(file_name)) -} - -fn write_file_atomically(path: &Path, bytes: &[u8]) -> Result<()> { - let temp_id = format!("{}-{}", std::process::id(), unix_time_millis()); - write_file_atomically_with( - path, - bytes, - |attempt| OsString::from(format!("{temp_id}-{attempt}")), - || {}, - ) -} - -fn write_file_atomically_with( - path: &Path, - bytes: &[u8], - suffix_for_attempt: S, - before_publish: P, -) -> Result<()> -where - S: FnMut(u32) -> OsString, - P: FnOnce(), -{ - write_file_atomically_with_publish( - path, - bytes, - suffix_for_attempt, - before_publish, - publish_temp_file, - ) -} - -fn write_file_atomically_with_publish( - path: &Path, - bytes: &[u8], - mut suffix_for_attempt: S, - before_publish: P, - publish: F, -) -> Result<()> -where - S: FnMut(u32) -> OsString, - P: FnOnce(), - F: FnOnce(&Path, &Path) -> io::Result<()>, -{ - let parent = path.parent().context("file path has no parent directory")?; - fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; - - let mut reserved = None; - for attempt in 0..ATOMIC_WRITE_TEMP_ATTEMPTS { - let suffix = suffix_for_attempt(attempt); - let tmp = temp_sibling_path(path, &suffix)?; - match fs::OpenOptions::new() - .write(true) - .create_new(true) - .open(&tmp) - { - Ok(file) => { - reserved = Some((tmp, file)); - break; - } - Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {} - Err(error) => { - return Err(error).with_context(|| format!("failed to create {}", tmp.display())); - } - } - } - let Some((tmp, mut file)) = reserved else { - bail!( - "failed to reserve a temporary file next to {} after {} attempts", - path.display(), - ATOMIC_WRITE_TEMP_ATTEMPTS - ); - }; - - if let Err(error) = file.write_all(bytes) { - drop(file); - let _ = fs::remove_file(&tmp); - return Err(error).with_context(|| format!("failed to write {}", tmp.display())); - } - drop(file); - before_publish(); - - publish(&tmp, path).inspect_err(|_| { - let _ = fs::remove_file(&tmp); - })?; - Ok(()) -} - -#[cfg(not(windows))] -fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { - fs::rename(tmp, path) -} - -#[cfg(windows)] -fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { - if path.try_exists()? { - return replace_file_windows(path, tmp); - } - - match fs::rename(tmp, path) { - Ok(()) => Ok(()), - Err(rename_error) => { - if path.try_exists()? { - replace_file_windows(path, tmp) - } else { - Err(rename_error) - } - } - } -} - -#[cfg(windows)] -#[allow(unsafe_code)] -fn replace_file_windows(path: &Path, replacement: &Path) -> io::Result<()> { - use std::os::windows::ffi::OsStrExt; - use windows_sys::Win32::Storage::FileSystem::ReplaceFileW; - - let path_wide: Vec = path.as_os_str().encode_wide().chain(Some(0)).collect(); - let replacement_wide: Vec = replacement - .as_os_str() - .encode_wide() - .chain(Some(0)) - .collect(); - - // SAFETY: both path buffers are valid, NUL-terminated UTF-16 strings and - // remain alive for the duration of the synchronous Windows API call. The - // optional backup, exclude, and reserved pointers are intentionally null. - let replaced = unsafe { - ReplaceFileW( - path_wide.as_ptr(), - replacement_wide.as_ptr(), - std::ptr::null(), - 0, - std::ptr::null(), - std::ptr::null(), - ) - }; - if replaced == 0 { - Err(io::Error::last_os_error()) - } else { - Ok(()) - } -} - pub(crate) fn sandbox_check_updates_value(output: CommandCapture) -> Value { let status = update_check_status(&output); let update_available = @@ -997,7 +843,7 @@ mod tests { temp_app_paths, }; use rocm_core::ManagedServiceRecord; - use std::io::Read; + use std::io::{Read, Write}; fn serve_one_http_response(bytes: Vec) -> Result { let listener = std::net::TcpListener::bind(("127.0.0.1", 0))?; @@ -1017,198 +863,6 @@ mod tests { Ok(format!("http://{addr}/artifact.bin")) } - /// Regression: a failed write must not leave a `.tmp-*` scratch file - /// behind. The name is unique per attempt, so before this an orphan - /// accumulated per retry — and when the failure is a full disk, those - /// orphans are exactly what keeps it full. - /// - /// Provokes the failure by pointing the destination at a non-empty - /// directory: the temp file is written, then neither the rename nor the - /// replace fallback can succeed. Portable, unlike an out-of-space test. - #[test] - fn write_file_atomically_cleans_up_temp_when_the_rename_fails() { - let root = std::env::temp_dir().join(format!( - "rocmd-atomic-{}-{}", - std::process::id(), - unix_time_millis() - )); - let occupied = root.join("manifest.json"); - fs::create_dir_all(occupied.join("nested")).unwrap(); - fs::write(occupied.join("nested").join("keep"), b"x").unwrap(); - - write_file_atomically(&occupied, b"payload") - .expect_err("renaming onto a non-empty directory should fail"); - - let leftovers: Vec = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) - .filter(|name| name.contains(".tmp-")) - .collect(); - let _ = fs::remove_dir_all(&root); - assert!( - leftovers.is_empty(), - "failed write left temp files behind: {leftovers:?}" - ); - } - - /// The temp name keeps every extension, so a cleanup sweep over a cache - /// directory can still tell what a leftover was going to be. - #[test] - fn write_file_atomically_temp_name_preserves_multi_dot_file_names() { - let root = std::env::temp_dir().join(format!( - "rocmd-atomic-name-{}-{}", - std::process::id(), - unix_time_millis() - )); - let occupied = root.join("artifact.tar.gz"); - fs::create_dir_all(occupied.join("nested")).unwrap(); - fs::write(occupied.join("nested").join("keep"), b"x").unwrap(); - - // Fails after the temp file exists, so the observed name is the real one. - write_file_atomically(&occupied, b"payload").expect_err("rename should fail"); - - let names: Vec = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) - .collect(); - let _ = fs::remove_dir_all(&root); - assert_eq!( - names, - vec!["artifact.tar.gz".to_owned()], - "only the occupied destination should remain" - ); - } - - #[test] - fn concurrent_atomic_writes_do_not_remove_a_published_destination() { - let root = std::env::temp_dir().join(format!( - "rocmd-atomic-collision-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("manifest.json"); - let before_publish = std::sync::Arc::new(std::sync::Barrier::new(2)); - - let writers: Vec<_> = [b"first".as_slice(), b"second".as_slice()] - .into_iter() - .enumerate() - .map(|(writer, bytes)| { - let destination = destination.clone(); - let before_publish = std::sync::Arc::clone(&before_publish); - std::thread::spawn(move || { - write_file_atomically_with( - &destination, - bytes, - |attempt| { - if attempt == 0 { - OsString::from("same-millisecond") - } else { - OsString::from(format!("same-millisecond-{writer}-{attempt}")) - } - }, - || { - before_publish.wait(); - }, - ) - }) - }) - .collect(); - - for writer in writers { - writer.join().unwrap().unwrap(); - } - let published = fs::read(&destination).expect("a writer must remain published"); - let _ = fs::remove_dir_all(&root); - assert!(published == b"first" || published == b"second"); - } - - #[test] - fn failed_atomic_replace_preserves_destination_and_cleans_temp() { - let root = std::env::temp_dir().join(format!( - "rocmd-atomic-replace-failure-{}-{}", - std::process::id(), - unix_time_millis() - )); - fs::create_dir_all(&root).unwrap(); - let destination = root.join("manifest.json"); - fs::write(&destination, b"published").unwrap(); - - write_file_atomically_with_publish( - &destination, - b"replacement", - |attempt| OsString::from(format!("replace-failure-{attempt}")), - || {}, - |tmp, path| { - assert_eq!(fs::read(tmp).unwrap(), b"replacement"); - assert_eq!(path, destination); - Err(io::Error::new( - io::ErrorKind::PermissionDenied, - "simulated atomic replacement failure", - )) - }, - ) - .expect_err("simulated replacement failure must be returned"); - - assert_eq!(fs::read(&destination).unwrap(), b"published"); - let leftovers: Vec<_> = fs::read_dir(&root) - .unwrap() - .map(|entry| entry.unwrap().file_name()) - .collect(); - let _ = fs::remove_dir_all(&root); - assert_eq!(leftovers, vec![OsString::from("manifest.json")]); - } - - #[cfg(unix)] - #[test] - fn atomic_temp_name_preserves_non_unicode_file_name_bytes() { - use std::os::unix::ffi::{OsStrExt, OsStringExt}; - - let file_name = OsString::from_vec(b"manifest-\xff.json".to_vec()); - let destination = Path::new("/tmp").join(&file_name); - let temp = temp_sibling_path(&destination, std::ffi::OsStr::new("collision")).unwrap(); - - let mut expected = file_name.into_vec(); - expected.extend_from_slice(b".tmp-collision"); - assert_eq!(temp.file_name().unwrap().as_bytes(), expected); - } - - /// Mirrors the `/dev/shm` reproduction from the original report: a genuine - /// ENOSPC, not a rename failure standing in for one. - /// - /// Ignored by default because it fills `/dev/shm`, which is shared with - /// anything else on the host, so it is not safe to run concurrently. - #[test] - #[ignore = "fills /dev/shm to provoke ENOSPC; not safe to run concurrently"] - fn write_file_atomically_cleans_up_temp_on_write_failure() { - let shm = Path::new("/dev/shm"); - if !shm.is_dir() { - eprintln!("skipping: /dev/shm unavailable"); - return; - } - let dir = shm.join(format!("rocmd-enospc-{}", std::process::id())); - let _ = fs::remove_dir_all(&dir); - fs::create_dir_all(&dir).unwrap(); - let dest = dir.join("manifest.json"); - // Larger than the tmpfs, so the write is guaranteed to hit ENOSPC. - let payload = vec![b'x'; 256 * 1024 * 1024]; - - write_file_atomically(&dest, &payload) - .expect_err("writing past the end of the filesystem should fail"); - let leftovers: Vec = fs::read_dir(&dir) - .unwrap() - .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) - .collect(); - let destination_exists = dest.exists(); - let _ = fs::remove_dir_all(&dir); - - assert!( - leftovers.is_empty(), - "failed write left files behind: {leftovers:?}" - ); - assert!(!destination_exists); - } - #[test] fn sandbox_check_updates_value_is_read_only_and_preserves_output() { let value = sandbox_check_updates_value(CommandCapture { diff --git a/coverage-floors.toml b/coverage-floors.toml index aa2ece4edd..3cc2e5514e 100644 --- a/coverage-floors.toml +++ b/coverage-floors.toml @@ -23,5 +23,5 @@ "rocm-engine-lemonade" = 63.82 # 5554 lines "rocm-engine-protocol" = 98.37 # 491 lines "rocm-engine-vllm" = 66.76 # 3030 lines -"rocmd" = 75.36 # 7714 lines +"rocmd" = 75.02 # 7549 lines "xtask" = 76.18 # 5447 lines diff --git a/crates/rocm-core/src/atomic_write.rs b/crates/rocm-core/src/atomic_write.rs new file mode 100644 index 0000000000..a3899fde64 --- /dev/null +++ b/crates/rocm-core/src/atomic_write.rs @@ -0,0 +1,1020 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Atomic file replacement, shared by `rocm` and `rocmd`. +//! +//! [`write_file_atomically`] stages the bytes in a fresh sibling temp file and +//! then publishes it over the destination in one step, so a reader sees either +//! the previous complete file or the new complete file, never a partial one, +//! and the destination is not removed before the new file takes its place. +//! +//! The temp file is named `{file name}.tmp-{pid}-{millis}-{attempt}` and +//! reserved with `create_new`, so two writers — threads of one process or +//! separate processes — can never write through the same temp file, even when +//! their pid and millisecond collide; a collision just moves on to the next +//! attempt. A failed write or publish removes its temp file, with the one +//! exception described below. +//! +//! On Windows, when the destination already exists, the publish uses +//! `ReplaceFileW` rather than `rename`. `std::fs::rename` does replace an +//! existing file there, but as `MoveFileExW` it fails with a sharing violation +//! while another process holds the destination open, which is the case +//! `ReplaceFileW` is used for. The concurrency tests below run on every +//! platform, but each starts with no destination, so on Windows at most one of +//! their publishes goes through `ReplaceFileW`: two `ReplaceFileW` publishes of +//! one file in flight at once are not exercised by any test. +//! +//! One failure leaves the destination absent: Microsoft documents that +//! `ReplaceFileW`, called without a backup name, can fail after it has already +//! moved the old file off the destination name +//! (`ERROR_UNABLE_TO_MOVE_REPLACEMENT` and `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2`). +//! The publish then retries once as a plain rename. If that fails too, the +//! temp file is kept rather than removed, since it holds the only copy of the +//! new contents, and the error is a [`KeptReplacement`] naming it. The +//! decision rests on the error code the replace returned, not on whether the +//! destination existed beforehand, so a second writer that created the +//! destination in between cannot turn it into a deletion. +//! +//! Atomic, not durable: neither the temp file nor its directory is fsynced +//! before the publish. That survives a crashed process and a concurrent +//! writer, but after a power loss a filesystem that does not order data before +//! the rename can still show an empty or stale destination. +//! +//! This makes each file individually atomic. It does not serialize a +//! read-modify-write across processes: two writers that each read, change and +//! write the same file are still last-writer-wins. + +use crate::disk_space; +use crate::unix_time_millis; +use anyhow::{Context, Result, bail}; +use std::ffi::{OsStr, OsString}; +use std::fmt; +use std::fs; +use std::io::{self, Write}; +use std::path::{Path, PathBuf}; + +const ATOMIC_WRITE_TEMP_ATTEMPTS: u32 = 128; + +/// The raw values of Windows' `ERROR_UNABLE_TO_MOVE_REPLACEMENT` (1176) and +/// `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2` (1177), the two codes with which +/// `ReplaceFileW` reports that it moved the old file off the destination name +/// and then failed. Plain numbers so the decision built on them can be tested +/// on every platform; a Windows-only test ties them to `windows-sys`. +#[cfg(any(windows, test))] +const PARTIAL_REPLACE_ERROR_CODES: [i32; 2] = [1176, 1177]; + +/// Whether a failed `ReplaceFileW` left the destination name empty. +#[cfg(any(windows, test))] +fn is_partial_replace(error: &io::Error) -> bool { + error + .raw_os_error() + .is_some_and(|code| PARTIAL_REPLACE_ERROR_CODES.contains(&code)) +} + +/// A publish that took the old destination away and could not put the new +/// file in its place: the replace failed part-way, and so did the rename +/// retried after it. +#[derive(Debug)] +struct DestinationRemoved { + replace: io::Error, + retry: io::Error, +} + +impl fmt::Display for DestinationRemoved { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{}; retrying as a plain rename also failed: {}", + self.replace, self.retry + ) + } +} + +impl std::error::Error for DestinationRemoved {} + +/// Test seam: the error a publish returns when it took the old destination +/// away and could not put the new file in its place, which is what Windows' +/// partial `ReplaceFileW` followed by a failed rename produces. +/// +/// `pub` only so tests in other crates can drive that path through +/// [`write_file_atomically_with_publish`] on any platform; production code +/// never builds one. +#[doc(hidden)] +pub fn destination_removed_error(replace: io::Error, retry: io::Error) -> io::Error { + io::Error::new(retry.kind(), DestinationRemoved { replace, retry }) +} + +fn removed_the_destination(error: &io::Error) -> bool { + error + .get_ref() + .is_some_and(::is::) +} + +/// The error of a publish that took the old file away and kept the new +/// contents in the temp file, which then holds the only copy. +/// +/// A caller that knows better than to want those contents back — one that has +/// already put the previous file back, say — can downcast to this to find the +/// kept file and remove it. +#[derive(Debug)] +pub struct KeptReplacement { + destination: PathBuf, + kept: PathBuf, + cause: io::Error, +} + +impl KeptReplacement { + /// The path the publish was meant to replace, which is now missing. + pub fn destination(&self) -> &Path { + &self.destination + } + + /// The temp file holding the new contents. + pub fn kept(&self) -> &Path { + &self.kept + } + + /// Why the publish failed. + pub const fn cause(&self) -> &io::Error { + &self.cause + } +} + +impl fmt::Display for KeptReplacement { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "failed to publish {}, and the replace took the old file away; the new contents \ + are kept in {} - rename that file to {} to restore it", + self.destination.display(), + self.kept.display(), + self.destination.display() + ) + } +} + +impl std::error::Error for KeptReplacement { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.cause) + } +} + +/// What to do after `ReplaceFileW` failed with `error`: give up when the +/// destination is untouched, and otherwise put the replacement in place with +/// `rename`, which is a plain rename in production and injected by tests. +#[cfg(any(windows, test))] +fn recover_partial_replace( + error: io::Error, + replacement: &Path, + path: &Path, + rename: R, +) -> io::Result<()> +where + R: FnOnce(&Path, &Path) -> io::Result<()>, +{ + if !is_partial_replace(&error) { + return Err(error); + } + rename(replacement, path).map_err(|retry| destination_removed_error(error, retry)) +} + +/// A temp path next to `path`, preserving the full file name so a +/// multi-extension artifact keeps its extensions (`sdk.tar.gz` becomes +/// `sdk.tar.gz.tmp-`, where `with_extension` would drop `.gz`). +fn temp_sibling_path(path: &Path, suffix: &OsStr) -> Result { + let parent = path.parent().context("file path has no parent directory")?; + let mut file_name = path + .file_name() + .context("file path has no file name")? + .to_os_string(); + file_name.push(".tmp-"); + file_name.push(suffix); + Ok(parent.join(file_name)) +} + +fn process_temp_id() -> String { + format!("{}-{}", std::process::id(), unix_time_millis()) +} + +/// Replace `path` with `bytes` atomically, creating missing parent directories. +pub fn write_file_atomically(path: &Path, bytes: &[u8]) -> Result<()> { + let temp_id = process_temp_id(); + write_file_atomically_with( + path, + bytes, + |attempt| OsString::from(format!("{temp_id}-{attempt}")), + || {}, + ) +} + +/// [`write_file_atomically`] with test seams for the temp name and publish point. +/// +/// `suffix_for_attempt` names the temp file for each attempt, and +/// `before_publish` runs after the temp file is fully written and before it is +/// published. Both exist so tests can force a temp-name collision and hold +/// writers at the publish point; production code calls +/// [`write_file_atomically`]. +fn write_file_atomically_with( + path: &Path, + bytes: &[u8], + suffix_for_attempt: S, + before_publish: P, +) -> Result<()> +where + S: FnMut(u32) -> OsString, + P: FnOnce(), +{ + write_file_atomically_with_publish( + path, + bytes, + suffix_for_attempt, + before_publish, + publish_temp_file, + ) +} + +/// Test seam: [`write_file_atomically`] with the temp name, a hook before the +/// publish, and the publish step itself injected. +/// +/// Tests use it to force a temp-name collision, to hold writers at the publish +/// point, to observe the staged file and the destination at the moment of +/// publication, or to make the publication fail. Pass [`publish_temp_file`] +/// for the real publish. Production code calls [`write_file_atomically`]. +/// +/// `pub` only because `apps/rocm`'s metadata-cache tests drive it across the +/// crate boundary; hidden from the docs because it is not an API to build on. +#[doc(hidden)] +pub fn write_file_atomically_with_publish( + path: &Path, + bytes: &[u8], + suffix_for_attempt: S, + before_publish: P, + publish: F, +) -> Result<()> +where + S: FnMut(u32) -> OsString, + P: FnOnce(), + F: FnOnce(&Path, &Path) -> io::Result<()>, +{ + let tmp = stage_file_for_atomic_publish_with(path, bytes, suffix_for_attempt)?; + before_publish(); + publish_staged_file_with(&tmp, path, publish) +} + +/// Write `bytes` to a fresh, uniquely named sibling of `path` and return the +/// staged file's path without publishing it. +/// +/// The caller owns the staged file and must remove it once done with it; on a +/// staging failure nothing is left behind. To *replace* `path` with the bytes, +/// call [`write_file_atomically`] instead, which also removes the staged file +/// when the publish fails. +pub fn stage_file_for_atomic_publish(path: &Path, bytes: &[u8]) -> Result { + let temp_id = process_temp_id(); + stage_file_for_atomic_publish_with(path, bytes, |attempt| { + OsString::from(format!("{temp_id}-{attempt}")) + }) +} + +fn stage_file_for_atomic_publish_with( + path: &Path, + bytes: &[u8], + suffix_for_attempt: S, +) -> Result +where + S: FnMut(u32) -> OsString, +{ + stage_file_for_atomic_publish_with_writer(path, bytes, suffix_for_attempt, |file, bytes| { + file.write_all(bytes) + }) +} + +/// [`stage_file_for_atomic_publish_with`] with the write into the reserved +/// temp file injected, so a test can make it fail the way a full disk does +/// without filling one. +fn stage_file_for_atomic_publish_with_writer( + path: &Path, + bytes: &[u8], + mut suffix_for_attempt: S, + write: W, +) -> Result +where + S: FnMut(u32) -> OsString, + W: FnOnce(&mut fs::File, &[u8]) -> io::Result<()>, +{ + let parent = path.parent().context("file path has no parent directory")?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + + let mut reserved = None; + for attempt in 0..ATOMIC_WRITE_TEMP_ATTEMPTS { + let suffix = suffix_for_attempt(attempt); + let tmp = temp_sibling_path(path, &suffix)?; + match fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&tmp) + { + Ok(file) => { + reserved = Some((tmp, file)); + break; + } + Err(error) if error.kind() == io::ErrorKind::AlreadyExists => {} + Err(error) => { + return Err(error).with_context(|| format!("failed to create {}", tmp.display())); + } + } + } + let Some((tmp, mut file)) = reserved else { + bail!( + "failed to reserve a temporary file next to {} after {} attempts", + path.display(), + ATOMIC_WRITE_TEMP_ATTEMPTS + ); + }; + + if let Err(error) = write(&mut file, bytes) { + drop(file); + let _ = fs::remove_file(&tmp); + return Err(disk_space::map_write_error(error, path)); + } + drop(file); + Ok(tmp) +} + +#[cfg(test)] +fn publish_staged_file(tmp: &Path, path: &Path) -> Result<()> { + publish_staged_file_with(tmp, path, publish_temp_file) +} + +/// Publish a staged file over `path`, removing the temp file if publishing +/// fails — unless the failed publish took the existing file away. +/// +/// A rename either replaces the destination or leaves it alone, but Windows' +/// `ReplaceFileW` can fail after moving the old file off the destination name +/// (see `recover_partial_replace`). Removing the temp file then would leave no +/// copy of the contents under any name the user can find, so it is kept and +/// the error is a [`KeptReplacement`] that says where. The publish step itself +/// reports that case, so it is told apart by the error rather than by a look +/// at the destination taken before the publish, which another writer can make +/// stale. +fn publish_staged_file_with(tmp: &Path, path: &Path, publish: F) -> Result<()> +where + F: FnOnce(&Path, &Path) -> io::Result<()>, +{ + let Err(error) = publish(tmp, path) else { + return Ok(()); + }; + if removed_the_destination(&error) && tmp.exists() { + return Err(anyhow::Error::new(KeptReplacement { + destination: path.to_path_buf(), + kept: tmp.to_path_buf(), + cause: error, + })); + } + let _ = fs::remove_file(tmp); + Err(error).with_context(|| format!("failed to publish {}", path.display())) +} + +/// Test seam: move the staged file `tmp` over `path` in one step, replacing +/// any existing file without first removing it. +/// +/// The real publish step to pass to [`write_file_atomically_with_publish`]. +/// `pub` only for that cross-crate test use; it does not remove `tmp` on +/// failure, so production code calls [`write_file_atomically`] instead. +#[cfg(not(windows))] +#[doc(hidden)] +pub fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { + fs::rename(tmp, path) +} + +/// Test seam: move the staged file `tmp` over `path` in one step, replacing +/// any existing file without first removing it. +/// +/// The real publish step to pass to [`write_file_atomically_with_publish`]. +/// `pub` only for that cross-crate test use; it does not remove `tmp` on +/// failure, so production code calls [`write_file_atomically`] instead. +#[cfg(windows)] +#[doc(hidden)] +pub fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { + if path.try_exists()? { + return replace_file_windows(path, tmp); + } + + match fs::rename(tmp, path) { + Ok(()) => Ok(()), + Err(rename_error) => { + if path.try_exists()? { + replace_file_windows(path, tmp) + } else { + Err(rename_error) + } + } + } +} + +#[cfg(windows)] +#[allow(unsafe_code)] // Win32 FFI +fn replace_file_windows(path: &Path, replacement: &Path) -> io::Result<()> { + use std::os::windows::ffi::OsStrExt; + use windows_sys::Win32::Storage::FileSystem::ReplaceFileW; + + let path_wide: Vec = path.as_os_str().encode_wide().chain(Some(0)).collect(); + let replacement_wide: Vec = replacement + .as_os_str() + .encode_wide() + .chain(Some(0)) + .collect(); + + // SAFETY: both path buffers are valid, NUL-terminated UTF-16 strings and + // remain alive for the duration of the synchronous Windows API call. The + // optional backup, exclude, and reserved pointers are intentionally null. + let replaced = unsafe { + ReplaceFileW( + path_wide.as_ptr(), + replacement_wide.as_ptr(), + std::ptr::null(), + 0, + std::ptr::null(), + std::ptr::null(), + ) + }; + if replaced != 0 { + return Ok(()); + } + // Microsoft documents two partial outcomes for a call without a backup + // name, and in both the replacement is still under its own name: + // `ERROR_UNABLE_TO_MOVE_REPLACEMENT` ("the replaced file no longer exists + // and the replacement file exists under its original name") and + // `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2` ("the file to be replaced still + // exists with a different name"). Either way nothing is at `path` any + // more. One plain rename puts the new contents there; if that fails too, + // `publish_staged_file_with` keeps the replacement file. + recover_partial_replace(io::Error::last_os_error(), replacement, path, rename_file) +} + +/// `fs::rename` with both paths borrowed, the shape `recover_partial_replace` +/// takes. +#[cfg(windows)] +fn rename_file(from: &Path, to: &Path) -> io::Result<()> { + fs::rename(from, to) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn scratch_dir(label: &str) -> PathBuf { + std::env::temp_dir().join(format!( + "rocm-core-{label}-{}-{}", + std::process::id(), + unix_time_millis() + )) + } + + /// A write that runs out of space while staging says so in words, and + /// leaves no temp file behind to keep the disk full. Drives the real + /// staging path — the one both [`write_file_atomically`] and + /// [`stage_file_for_atomic_publish`] go through — with only the byte write + /// into the reserved temp file swapped for one that fails like a full + /// disk, so the mapping is checked where it is applied rather than on its + /// own. + #[test] + fn staging_into_a_full_disk_reports_it_clearly_and_removes_the_temp_file() { + let root = scratch_dir("atomic-full-disk"); + let destination = root.join("rocm.tar.gz"); + + let error = stage_file_for_atomic_publish_with_writer( + &destination, + b"payload", + |attempt| OsString::from(format!("full-disk-{attempt}")), + |_, _| Err(io::Error::from(io::ErrorKind::StorageFull)), + ) + .expect_err("a write that hits a full disk must fail"); + + let leftovers: Vec = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + let _ = fs::remove_dir_all(&root); + let text = format!("{error:#}"); + assert!(text.contains("ran out of disk space"), "{text}"); + // The user knows the file they asked for, not the temp file that was + // already removed, so the message names the destination. + assert!( + text.contains(&destination.display().to_string()) && !text.contains(".tmp-"), + "{text}" + ); + assert!( + leftovers.is_empty(), + "the failed staging left files behind: {leftovers:?}" + ); + } + + /// When every temp name the writer tries is taken, it gives up after a + /// bounded number of tries with an error that says so, rather than + /// looping or writing through a file another writer reserved. + #[test] + fn staging_gives_up_when_every_temp_name_is_taken() { + let root = scratch_dir("atomic-names-exhausted"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("active.json"); + let taken = temp_sibling_path(&destination, OsStr::new("taken")).unwrap(); + fs::write(&taken, b"another writer's staged file").unwrap(); + let mut tries = 0; + + let error = stage_file_for_atomic_publish_with(&destination, b"payload", |_| { + tries += 1; + OsString::from("taken") + }) + .expect_err("a writer that finds every name taken must fail"); + + let foreign = fs::read(&taken); + let leftovers: Vec<_> = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .collect(); + let _ = fs::remove_dir_all(&root); + let text = format!("{error:#}"); + assert_eq!(tries, ATOMIC_WRITE_TEMP_ATTEMPTS); + assert!( + text.contains(&format!("after {ATOMIC_WRITE_TEMP_ATTEMPTS} attempts")), + "{text}" + ); + assert_eq!(foreign.unwrap(), b"another writer's staged file"); + assert_eq!(leftovers, vec![OsString::from("active.json.tmp-taken")]); + } + + /// The temp name keeps every extension, so a cleanup sweep over a cache + /// directory can still tell what a leftover was going to be. + #[test] + fn temp_sibling_path_preserves_multi_dot_file_names() { + let temp = + temp_sibling_path(Path::new("/tmp/cache/sdk.tar.gz"), OsStr::new("test")).unwrap(); + let name = temp.file_name().unwrap().to_string_lossy().into_owned(); + assert_eq!(name, "sdk.tar.gz.tmp-test"); + assert_eq!(temp.parent().unwrap(), Path::new("/tmp/cache")); + } + + /// Same property end to end: the temp file a write stages, observed at the + /// moment it is published, keeps the destination's full name, and the + /// publish leaves only the destination behind. + #[test] + fn write_file_atomically_temp_name_preserves_multi_dot_file_names() { + let root = scratch_dir("atomic-name"); + let destination = root.join("artifact.tar.gz"); + let temp_id = process_temp_id(); + let mut staged_name = None; + + write_file_atomically_with_publish( + &destination, + b"payload", + |attempt| OsString::from(format!("{temp_id}-{attempt}")), + || {}, + |tmp, path| { + staged_name = tmp.file_name().map(OsStr::to_os_string); + publish_temp_file(tmp, path) + }, + ) + .unwrap(); + + let names: Vec = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) + .collect(); + let published = fs::read(&destination); + let _ = fs::remove_dir_all(&root); + assert_eq!( + staged_name, + Some(OsString::from(format!("artifact.tar.gz.tmp-{temp_id}-0"))) + ); + assert_eq!(published.unwrap(), b"payload"); + assert_eq!(names, vec!["artifact.tar.gz".to_owned()]); + } + + #[test] + fn concurrent_atomic_writes_do_not_remove_a_published_destination() { + let root = scratch_dir("atomic-collision"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("sdk.tar.gz"); + let before_publish = std::sync::Arc::new(std::sync::Barrier::new(2)); + + let writers: Vec<_> = [b"first".as_slice(), b"second".as_slice()] + .into_iter() + .enumerate() + .map(|(writer, bytes)| { + let destination = destination.clone(); + let before_publish = std::sync::Arc::clone(&before_publish); + std::thread::spawn(move || { + write_file_atomically_with( + &destination, + bytes, + |attempt| { + if attempt == 0 { + OsString::from("same-millisecond") + } else { + OsString::from(format!("same-millisecond-{writer}-{attempt}")) + } + }, + || { + before_publish.wait(); + }, + ) + }) + }) + .collect(); + + for writer in writers { + writer.join().unwrap().unwrap(); + } + let published = fs::read(&destination).expect("a writer must remain published"); + let _ = fs::remove_dir_all(&root); + assert!(published == b"first" || published == b"second"); + } + + #[test] + fn concurrent_cached_publications_use_distinct_staging_files() { + let root = scratch_dir("cache-publish-collision"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("index.body"); + let before_publish = std::sync::Arc::new(std::sync::Barrier::new(2)); + + let writers: Vec<_> = [b"first".as_slice(), b"second".as_slice()] + .into_iter() + .enumerate() + .map(|(writer, bytes)| { + let destination = destination.clone(); + let before_publish = std::sync::Arc::clone(&before_publish); + std::thread::spawn(move || { + // Both threads ask for the same first name, as two writers + // sharing a pid and a millisecond would, so the collision + // happens every run rather than when the clock lines up. + let staged = + stage_file_for_atomic_publish_with(&destination, bytes, |attempt| { + if attempt == 0 { + OsString::from("same-millisecond") + } else { + OsString::from(format!("same-millisecond-{writer}-{attempt}")) + } + })?; + before_publish.wait(); + publish_staged_file(&staged, &destination) + }) + }) + .collect(); + + for writer in writers { + writer.join().unwrap().unwrap(); + } + let published = fs::read(&destination).expect("a cache writer must remain published"); + let leftovers: Vec<_> = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .filter(|name| name.to_string_lossy().contains(".tmp-")) + .collect(); + let _ = fs::remove_dir_all(&root); + assert!(published == b"first" || published == b"second"); + assert!( + leftovers.is_empty(), + "staged cache files leaked: {leftovers:?}" + ); + } + + #[test] + fn failed_cached_publication_preserves_destination_and_cleans_staging_file() { + let root = scratch_dir("cache-publish-failure"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("index.body"); + fs::write(&destination, b"published").unwrap(); + let staged = stage_file_for_atomic_publish(&destination, b"replacement").unwrap(); + + publish_staged_file_with(&staged, &destination, |_, _| { + Err(io::Error::new( + io::ErrorKind::PermissionDenied, + "simulated cache publication failure", + )) + }) + .expect_err("simulated cache publication failure must be returned"); + + assert_eq!(fs::read(&destination).unwrap(), b"published"); + assert!( + !staged.exists(), + "failed publication leaked its staging file" + ); + let _ = fs::remove_dir_all(&root); + } + + #[test] + fn failed_atomic_replace_preserves_destination_and_cleans_temp() { + let root = scratch_dir("atomic-replace-failure"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("sdk.tar.gz"); + fs::write(&destination, b"published").unwrap(); + + write_file_atomically_with_publish( + &destination, + b"replacement", + |attempt| OsString::from(format!("replace-failure-{attempt}")), + || {}, + |tmp, path| { + assert_eq!(fs::read(tmp).unwrap(), b"replacement"); + assert_eq!(path, destination); + Err(io::Error::new( + io::ErrorKind::PermissionDenied, + "simulated atomic replacement failure", + )) + }, + ) + .expect_err("simulated replacement failure must be returned"); + + assert_eq!(fs::read(&destination).unwrap(), b"published"); + let leftovers: Vec<_> = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .collect(); + let _ = fs::remove_dir_all(&root); + assert_eq!(leftovers, vec![OsString::from("sdk.tar.gz")]); + } + + fn simulated_partial_replace() -> io::Error { + destination_removed_error( + io::Error::from_raw_os_error(PARTIAL_REPLACE_ERROR_CODES[0]), + io::Error::from(io::ErrorKind::PermissionDenied), + ) + } + + /// A publish that fails after taking the old file away — what Windows' + /// `ReplaceFileW` documents for `ERROR_UNABLE_TO_MOVE_REPLACEMENT` without + /// a backup name, when the rename retried after it fails too — must not + /// also delete the temp file, which then holds the only copy. The error + /// names it so the user can put it back, and a caller can find it. + #[test] + fn a_publish_that_removes_the_destination_keeps_the_new_contents() { + let root = scratch_dir("replace-partial"); + let destination = root.join("active.json"); + fs::create_dir_all(&root).unwrap(); + fs::write(&destination, b"old").unwrap(); + + let error = write_file_atomically_with_publish( + &destination, + b"new", + |attempt| OsString::from(format!("replace-partial-{attempt}")), + || {}, + |_tmp, path| { + fs::remove_file(path).unwrap(); + Err(simulated_partial_replace()) + }, + ) + .expect_err("the failed publish must be reported"); + let message = format!("{error:#}"); + let reported = error + .downcast_ref::() + .map(|kept| (kept.kept().to_path_buf(), kept.destination().to_path_buf())); + + let kept = temp_sibling_path(&destination, OsStr::new("replace-partial-0")).unwrap(); + let kept_bytes = fs::read(&kept); + let destination_exists = destination.exists(); + let _ = fs::remove_dir_all(&root); + assert!(!destination_exists); + assert_eq!(kept_bytes.unwrap(), b"new", "the new contents were deleted"); + assert!( + message.contains(&kept.display().to_string()), + "the error does not say where the contents are: {message}" + ); + assert_eq!(reported, Some((kept, destination))); + } + + /// The race a pre-publish look at the destination gets wrong: no + /// destination when this writer starts, another writer publishes one, and + /// then this writer's replace takes it away and fails. The temp file still + /// holds the only copy, so it is still kept. + #[test] + fn a_publish_that_removes_a_destination_another_writer_created_keeps_the_new_contents() { + let root = scratch_dir("replace-partial-race"); + let destination = root.join("active.json"); + fs::create_dir_all(&root).unwrap(); + + let error = write_file_atomically_with_publish( + &destination, + b"second", + |attempt| OsString::from(format!("replace-partial-race-{attempt}")), + || {}, + |_tmp, path| { + fs::write(path, b"first").unwrap(); + fs::remove_file(path).unwrap(); + Err(simulated_partial_replace()) + }, + ) + .expect_err("the failed publish must be reported"); + + let kept = temp_sibling_path(&destination, OsStr::new("replace-partial-race-0")).unwrap(); + let kept_bytes = fs::read(&kept); + let _ = fs::remove_dir_all(&root); + assert_eq!( + kept_bytes.unwrap(), + b"second", + "the new contents were deleted" + ); + assert!( + error.downcast_ref::().is_some(), + "{error:#}" + ); + } + + /// When the temp file is gone too there is nothing to keep, so the error + /// must not point the user at it. + #[test] + fn a_partial_replace_that_lost_the_temp_file_does_not_claim_to_keep_it() { + let root = scratch_dir("replace-partial-no-temp"); + let destination = root.join("active.json"); + fs::create_dir_all(&root).unwrap(); + fs::write(&destination, b"old").unwrap(); + + let error = write_file_atomically_with_publish( + &destination, + b"new", + |attempt| OsString::from(format!("replace-partial-no-temp-{attempt}")), + || {}, + |tmp, path| { + fs::remove_file(path).unwrap(); + fs::remove_file(tmp).unwrap(); + Err(simulated_partial_replace()) + }, + ) + .expect_err("the failed publish must be reported"); + let _ = fs::remove_dir_all(&root); + let message = format!("{error:#}"); + assert!( + error.downcast_ref::().is_none(), + "{message}" + ); + assert!(!message.contains("kept in"), "{message}"); + } + + /// Any other failed replace leaves the destination alone, so nothing is + /// retried and the error comes back as it was. + #[test] + fn a_replace_that_left_the_destination_alone_is_not_retried() { + let mut renamed = false; + let error = recover_partial_replace( + io::Error::from_raw_os_error(5), + Path::new("staged"), + Path::new("destination"), + |_, _| { + renamed = true; + Ok(()) + }, + ) + .expect_err("an ordinary failure is returned"); + assert!(!renamed); + assert_eq!(error.raw_os_error(), Some(5)); + assert!(!removed_the_destination(&error)); + } + + /// Both documented partial codes retry once as a rename of the staged + /// file onto the destination, and a successful retry is a success. + #[test] + fn a_partial_replace_is_retried_as_a_rename() { + for code in PARTIAL_REPLACE_ERROR_CODES { + let mut renamed = None; + recover_partial_replace( + io::Error::from_raw_os_error(code), + Path::new("staged"), + Path::new("destination"), + |from, to| { + renamed = Some((from.to_path_buf(), to.to_path_buf())); + Ok(()) + }, + ) + .unwrap_or_else(|error| panic!("code {code}: {error}")); + assert_eq!( + renamed, + Some((PathBuf::from("staged"), PathBuf::from("destination"))), + "code {code}" + ); + } + } + + /// A retry that fails too is reported as the case that keeps the temp file. + #[test] + fn a_partial_replace_whose_retry_fails_reports_the_destination_removed() { + for code in PARTIAL_REPLACE_ERROR_CODES { + let error = recover_partial_replace( + io::Error::from_raw_os_error(code), + Path::new("staged"), + Path::new("destination"), + |_, _| Err(io::Error::from(io::ErrorKind::PermissionDenied)), + ) + .expect_err("a failed retry is returned"); + assert!(removed_the_destination(&error), "code {code}: {error}"); + assert!(error.to_string().contains("retrying as a plain rename")); + } + } + + /// The plain numbers the partial-replace decision uses are the codes + /// `ReplaceFileW` actually returns. + #[cfg(windows)] + #[test] + fn partial_replace_codes_match_the_windows_constants() { + use windows_sys::Win32::Foundation::{ + ERROR_UNABLE_TO_MOVE_REPLACEMENT, ERROR_UNABLE_TO_MOVE_REPLACEMENT_2, + }; + assert_eq!( + PARTIAL_REPLACE_ERROR_CODES.map(|code| u32::try_from(code).unwrap()), + [ + ERROR_UNABLE_TO_MOVE_REPLACEMENT, + ERROR_UNABLE_TO_MOVE_REPLACEMENT_2 + ] + ); + } + + #[cfg(unix)] + #[test] + fn atomic_temp_name_preserves_non_unicode_file_name_bytes() { + use std::os::unix::ffi::{OsStrExt, OsStringExt}; + + let file_name = OsString::from_vec(b"sdk-\xff.tar.gz".to_vec()); + let destination = Path::new("/tmp").join(&file_name); + let temp = temp_sibling_path(&destination, OsStr::new("collision")).unwrap(); + + let mut expected = file_name.into_vec(); + expected.extend_from_slice(b".tmp-collision"); + assert_eq!(temp.file_name().unwrap().as_bytes(), expected); + } + + /// Regression: a failed write must not leave a `.tmp-*` scratch file + /// behind. The name is unique per attempt, so before this an orphan + /// accumulated per retry — and when the failure is a full disk, those + /// orphans are exactly what keeps it full. + /// + /// Provokes the failure by pointing the destination at a non-empty + /// directory: the temp file is written, then neither the rename nor the + /// replace fallback can succeed. Portable, unlike an out-of-space test. + #[test] + fn write_file_atomically_cleans_up_temp_when_the_rename_fails() { + let root = scratch_dir("atomic"); + let occupied = root.join("sdk.tar.gz"); + fs::create_dir_all(occupied.join("nested")).unwrap(); + fs::write(occupied.join("nested").join("keep"), b"x").unwrap(); + + write_file_atomically(&occupied, b"payload") + .expect_err("renaming onto a non-empty directory should fail"); + + let leftovers: Vec = fs::read_dir(&root) + .unwrap() + .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) + .filter(|name| name.contains(".tmp-")) + .collect(); + let _ = fs::remove_dir_all(&root); + assert!( + leftovers.is_empty(), + "failed write left temp files behind: {leftovers:?}" + ); + } + + /// Mirrors the `/dev/shm` reproduction from the original report: a genuine + /// ENOSPC, not a rename failure standing in for one. + /// + /// Ignored by default because it fills `/dev/shm`, which is shared with + /// anything else on the host, so it is not safe to run concurrently. Run + /// with `cargo test -p rocm-core -- --ignored write_file_atomically_cleans_up`. + #[test] + #[ignore = "fills /dev/shm to provoke ENOSPC; not safe to run concurrently"] + fn write_file_atomically_cleans_up_temp_on_write_failure() { + let shm = Path::new("/dev/shm"); + if !shm.is_dir() { + eprintln!("skipping: /dev/shm unavailable"); + return; + } + let dir = shm.join(format!("rocm-core-enospc-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let dest = dir.join("artifact.tar.gz"); + // Larger than the tmpfs, so the write is guaranteed to hit ENOSPC. + let payload = vec![0u8; 256 * 1024 * 1024]; + + let mut failures = Vec::new(); + for _ in 0..2 { + write_file_atomically(&dest, &payload) + .expect_err("writing past the end of the filesystem should fail"); + failures.push( + fs::read_dir(&dir) + .unwrap() + .map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()) + .collect::>(), + ); + } + let destination_exists = dest.exists(); + let _ = fs::remove_dir_all(&dir); + + for leftovers in &failures { + assert!( + leftovers.is_empty(), + "failed write left files behind: {leftovers:?}" + ); + } + assert!( + !destination_exists, + "destination must not exist after failure" + ); + } +} diff --git a/crates/rocm-core/src/config.rs b/crates/rocm-core/src/config.rs index 58e59acfe4..62eaf05fad 100644 --- a/crates/rocm-core/src/config.rs +++ b/crates/rocm-core/src/config.rs @@ -1151,7 +1151,7 @@ last_installed_runtime_id = "therock-release" /// A directory at the destination is what makes the rename fail on both /// supported platforms without needing permissions a root CI lane ignores — /// the same device `write_file_atomically_cleans_up_temp_when_the_rename_fails` - /// uses in `apps/rocm`. + /// uses in `rocm_core::atomic_write`. #[test] fn a_config_save_that_cannot_publish_cleans_up_its_temp_file() -> Result<()> { let (root, paths) = temp_app_paths("config-save-rename-fails"); diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index aad2dda627..c8075f47a2 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -10,6 +10,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; pub mod amd_smi; #[cfg(all(test, unix))] mod archive_props; +pub mod atomic_write; pub mod automation; pub mod binary_paths; pub mod browser; diff --git a/docs/architecture.md b/docs/architecture.md index ac2ba48a37..c1e2af33af 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -103,9 +103,9 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and reads the os-release every one of these plans is built from through `rocm_core::hardware_root::real_os_release`, which has no copy here. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs). `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `rocm_core::hardware_root::real_os_release` and the three OpenMPI/libatomic/libnuma install hints ignore a simulated host root, and a scan that no source file of the binary reads os-release by hand. - [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token` and `slugify`, plus `ensure_tarball_install_target_is_clean`, the refusal to extract a tarball into a root that already holds a payload. What "holds a payload" means is not defined here: it is `rocm_core::install_root_has_payload`, the one rule shared with `runtime_registry.rs` and `therock/update.rs`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read five of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key` and `wheel_runtime_key` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). - [`therock/version.rs`](../apps/rocm/src/therock/version.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): the version grammar behind runtime selection and update checks. `parse_version` is the strict gate an install is routed on; `parse_version_for_ordering`, `compare_version_strings` and `version_relation` implement the PEP 440 ordering and the "is this a different version" question; `parse_host_version` reads a host-reported ROCm version through the same parser; `runtime_version_display`, `runtime_version_build_date`, `normalize_requested_build_date` and the calendar helpers behind them handle build dates; `is_stable_runtime_version` and the `ParsedVersion`/`VersionOrderKey`/`VersionStage` types back them. The items `therock.rs` and its property-test module use are `pub(super)` (including the `ParsedVersion` and `VersionOrderKey` fields they read); the parsing and calendar helpers behind them are private; and `runtime_version_display` is `pub(crate)`, re-exported by `therock.rs` for the other modules that render versions. Depends on nothing else in `therock`. `therock_version_proptests.rs` stays declared from `therock.rs` because it also reaches update, wheel and registry code. -- [`therock/io.rs`](../apps/rocm/src/therock/io.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): unique scratch directories (`linux_temp_dir`, `windows_temp_dir`) and the stage-then-publish atomic file writes (`write_file_atomically`, `stage_file_for_atomic_publish`, `temp_sibling_path`, and the platform `publish_temp_file`/`replace_file_windows` pair) for the metadata cache entries and the staged metadata public key, the startup update-check record and the active-runtime marker; runtime registry manifests do not go through them. `write_file_atomically` is `pub(crate)` and re-exported by `therock.rs` for [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs), which writes the active-runtime marker with it; the rest is `pub(super)` or private. The `cfg(windows)` publish and replace paths compile only on the Windows lane. +- [`therock/io.rs`](../apps/rocm/src/therock/io.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): unique scratch directories (`linux_temp_dir`, `windows_temp_dir`), both `pub(super)`. The atomic file writes that used to live here are rocm-core's shared [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs), which [`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs), [`therock/update.rs`](../apps/rocm/src/therock/update.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) call directly. - [`therock/process.rs`](../apps/rocm/src/therock/process.rs) — full domain extraction, a child of `therock.rs` that depends on `therock/io.rs` for scratch directories (split out of `therock.rs`): running and capturing child commands (`run_command*`, `capture_command_output*`, `capture_python_stdout*`, `run_uv_progress_command`, and the Windows temp-file variants with `windows_child_path`), the bounded wait (`wait_with_output_bounded`), the progress-line output with its `--json` suppression guard (`SuppressProgressOutput`, `progress_line`), and unpacking the SDK tarball (`extract_tarball`, `extract_tarball_and_discard_archive`, which is why it sits here and not in `io.rs`: it spawns `tar` through `run_command`). Its `archive_properties` property tests run on Unix only. The process-wide environment lock these tests take is the single `PROCESS_ENV_TEST_LOCK` in [`test_support.rs`](../apps/rocm/src/test_support.rs), shared with the tests left in `therock.rs`. -- [`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) — full domain extraction, a child of `therock.rs` that depends on `therock/io.rs` for atomic writes and on `slugify` in the parent for cache file names (split out of `therock.rs`): HTTP fetching for installs. `download_text_cached` is the metadata GET with its on-disk etag cache (`CachedHttp*`) and detached-signature verification (`MetadataSignaturePolicy`, `PINNED_METADATA_PUBLIC_KEY_PEM`, the unset trust root that `cargo xtask verify-pinned-keys` checks against the canonical published key, so its path is pinned in `xtask`); `download_file` is the streaming artifact download; `head_content_length` and `preflight_tarball_space` size the download and check free space first; `http_get` is the shared request. `truthy_env` is `pub(crate)` and re-exported by `therock.rs` for [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs); `download_text_cached`, `download_file`, `preflight_tarball_space` and the `CachedHttpText` struct (with its `text` field) that `download_text_cached` returns are `pub(super)` for the install and resolution code that stays in `therock.rs`; everything else is private. +- [`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) — full domain extraction, a child of `therock.rs` that depends on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for atomic writes and on `slugify` in the parent for cache file names (split out of `therock.rs`): HTTP fetching for installs. `download_text_cached` is the metadata GET with its on-disk etag cache (`CachedHttp*`) and detached-signature verification (`MetadataSignaturePolicy`, `PINNED_METADATA_PUBLIC_KEY_PEM`, the unset trust root that `cargo xtask verify-pinned-keys` checks against the canonical published key, so its path is pinned in `xtask`); `download_file` is the streaming artifact download; `head_content_length` and `preflight_tarball_space` size the download and check free space first; `http_get` is the shared request. `truthy_env` is `pub(crate)` and re-exported by `therock.rs` for [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs); `download_text_cached`, `download_file`, `preflight_tarball_space` and the `CachedHttpText` struct (with its `text` field) that `download_text_cached` returns are `pub(super)` for the install and resolution code that stays in `therock.rs`; everything else is private. - [`therock/source.rs`](../apps/rocm/src/therock/source.rs) — full domain extraction, a child of `therock.rs` that depends only on `therock/version.rs` (split out of `therock.rs`): where an install reads from. `TheRockChannel` (release or nightly) and `SourceLayout` (the canonical stream, or ROCm 10's separately hosted next layout) are the two axes; `resolve_source` maps a pair to the `ResolvedSource` wheel-index and tarball-catalog bases, honouring a `ROCM_CLI_THEROCK_*_BASE` override only behind the explicit opt-in (`therock_base_override_allowed`, `select_base_override`, `env_override_base`); `select_source_layout`, `requested_source_layout` and `next_layout_requested` decide the layout from a pin; `RuntimeVersionSelector` is the version or build date a user pins (`pub(crate)`, re-exported by `therock.rs`). `FamilyResolution` lives here rather than in `therock/family.rs` because choosing the layout reads it; that keeps the dependency one-way, with `family.rs` building on this file. `TheRockChannel`, `SourceLayout`, `ResolvedSource` and `FamilyResolution` (with their methods and fields), `RuntimeVersionSelector`'s `describe` and `matches_version`, `resolve_source`, the three layout functions, and the `THEROCK_SOURCE_LAYOUT_GENERATION` and `THEROCK_NEXT_MIN_MAJOR` constants are `pub(super)` for `therock.rs` and its siblings; the base-override helpers and the other constants (the per-channel and next-layout base URLs, and the next layout's generation tag) are private to this file. - [`therock/family.rs`](../apps/rocm/src/therock/family.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` for `TheRockChannel` and `FamilyResolution` (split out of `therock.rs`): which package family a request targets. `resolve_family` picks it from `--family`, `ROCM_CLI_THEROCK_FAMILY`, a managed runtime or the host GPU, in that order, and refuses a GPU with no ROCm build (`unsupported_host_gpu_message`, `unsupported_requested_target_message`) instead of substituting another family; `family_override_or_recovered_arch` and `raw_arch_agreeing_with_family` recover the exact arch behind a grouped family; `family_resolution_hint` is the recovery text appended when no runtime is found. `resolve_family`, `family_override_or_recovered_arch` and `family_resolution_hint` are `pub(super)`; the rest is private. - [`therock/python.rs`](../apps/rocm/src/therock/python.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/io.rs`, `therock/process.rs` and `therock/source.rs` (split out of `therock.rs`): the Python an install runs on. `python_requirement` maps a `SourceLayout` to the interpreter the layout needs (`PythonRequirement`: wheel tag and `uv` version), which is why `source.rs` had to move first; `resolve_python_launcher` finds one in order of an explicit `ROCM_CLI_PYTHON`, `PATH`, the saved managed Python, then `ensure_managed_python`, which installs one through `uv` and records it in the managed-Python manifest; the resolver reads its environment through `PythonResolverEnv`, so tests steer it with a thread-local override (`OverridePythonResolverEnv`) rather than rewriting `PATH`; `wheel_compatibility_for_python` and `current_platform_wheel_tags` report the wheel tags an interpreter and platform accept (`WheelCompatibility`); `ensure_uv_venv` creates, or recreates when the interpreter tag no longer matches, the runtime's virtual environment. `venv_python_path` stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The items `therock.rs` reaches, and `split_runtime_path`, which `therock/probe.rs` reads, are `pub(super)`; the rest is private, including `ensure_managed_python` and `resolve_python_launcher_in` (the resolver's entry point that takes its environment as an argument). Its Unix-only tests drive fake interpreters and `uv` scripts. @@ -114,7 +114,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/fetch.rs`, `therock/family.rs`, `therock/source.rs` and `therock/version.rs` (split out of `therock.rs`): resolving the SDK tarball a request installs. `resolve_tarball_artifact` (and `resolve_tarball_artifact_with_timeout`, which takes the metadata download timeout) resolves the family and source layout, downloads the catalog through the cached metadata fetch, and returns the `TarballArtifact` it chose; `parse_tarball_index_html` reads the catalog's embedded file list into `TarballIndexFile`s and `validate_tarball_file_name` refuses an entry that is not a plain file name; `select_tarball_candidate` picks the newest archive the request can install, with `tarball_family_token` spelling the family the way the next layout's catalog does. `platform_tarball_token`, which names the archives' platform, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. Only `resolve_tarball_artifact*` and `TarballArtifact` are visible beyond this file. - [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/family.rs`, `therock/fetch.rs`, `therock/process.rs`, `therock/python.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for `WheelRuntimeComposition` (split out of `therock.rs`): resolving the wheel stack a request installs. `resolve_pip_runtime` (and `resolve_pip_runtime_with_timeout`, which takes the metadata download timeout and an optional source layout) resolves the family and the layout, reads the aggregate root, decides the device payload and returns a `PipRuntimeResolution`; on the canonical layout `select_matching_pip_package_versions` then takes the newest `rocm` whose `torch`, `torchvision` and `torchaudio` listings agree, and on the next layout `torchaudio_pin` pins a torchaudio first (the hardware-verified release when it is published, read from the vLLM engine's table) and `resolve_published_pip_package_versions` has `uv pip compile` choose the rest. `wheel_runtime_composition` turns a resolution into the exact package specs (`therock_pip_package_specs`) an install gives `uv` and records in the manifest, `published_pip_requirements` is the matching requirement set for `uv pip compile`, and `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` listing so the interpreter can be picked before resolution. Beneath that, `parse_aggregate_package_links` and `parse_aggregate_device_targets` read the package links of the canonical aggregate root and the exact device targets it publishes, and `validate_aggregate_index_layout` refuses a root that lacks the expected packages or any device payload; `AggregateDeviceTarget::resolve` picks the one exact target a host gets, or records why it could not (`Undetermined`); `load_simple_index_versions` fetches a package's simple-index listing through the cached metadata download and `parse_simple_index_versions` reads the versions out of it, dropping wheels the interpreter's tags do not accept when given them (`wheel_stem_matches_compatibility` and the `wheel_*_tag_matches` helpers); `package_release_key` orders the PEP 440 versions found there (`PackageReleaseKey`). `slugify`, which keys the cached listing, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The functions and types `therock.rs` and `therock/update.rs` name are `pub(in crate::therock)` and reach them through `therock/resolve.rs`; `PipRuntimeResolution` and `TheRockPipPackageVersions` are `pub(in crate::therock)` with the fields they read, and `TorchaudioPin` with its `describe`, so they can use what `resolve_pip_runtime` returns without naming the types; the rest is private. - [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` and `therock/version.rs` and reaches the registry through [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (split out of `therock.rs`): the consent gate for an install that would displace the active default runtime. `active_default_runtime_relation` says what the install would replace (an upgrade, downgrade or reinstall when it is the same family and channel, otherwise the different runtime it displaces) or `None` when nothing is the active default, and it fails closed, with a relation naming what could not be resolved, when the config names an active default the registry cannot resolve; `SdkInstallConsent` and `SdkInstallApprovalSource` say how consent is obtained (asked for, or already granted by `--yes`, `--approve-replacing-active-default` or `rocm update --apply`); `sdk_install_approval` turns those and whether a terminal is attached into proceed, prompt or refuse (`SdkInstallApproval`); `preapproved_install_line`, `fresh_install_line`, `refuse_non_interactive_message` and `confirm_overwrite_existing_sdk` are the wording of each outcome, and the prompt reads its answer through `prompt_yes_no` and the shared [`confirm.rs`](../crates/rocm-core/src/confirm.rs). `SdkInstallConsent`, `SdkInstallApprovalSource` and `preapproved_install_line` are re-exported by `therock.rs`; the other items `therock.rs` calls are `pub(super)`, as is `active_default_relation_text`, which the version property tests call, and the rest is private. -- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/io.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. +- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests and on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for the startup-check record (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. - [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. - [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_serve_command_safety`, the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), and `validate_chat_rocm_command_safety`, which applies them to a `serve` argv and also refuses a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_serve_command_safety` and `validate_chat_rocm_command_safety` here instead of through `chat`, and keeps no copy of the serve rules. `chat.rs`'s child modules call into it and `chat.rs` re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). - [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. @@ -150,7 +150,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`mcp.rs`](../apps/rocmd/src/mcp.rs) — the MCP stdio server, tool schema table, tool dispatch, and the result-shaping helpers behind the MCP tools (the `rocm` subprocess itself is spawned through [`common.rs`](../apps/rocmd/src/common.rs), via `run_rocm_capture` here). The argument handling the dispatch calls is in [`mcp_argv.rs`](../apps/rocmd/src/mcp_argv.rs). - [`mcp_argv.rs`](../apps/rocmd/src/mcp_argv.rs) — the argument handling for the MCP tools that spawn `rocm`: `normalized_rocm_command_args` and the read-only allowlist `ensure_rocm_command_is_read_only` behind the `rocm_command` tool, and the argv builders for `install_sdk`, `install_engine`, `launch_server` and `watcher_enable` (`build_install_sdk_args`, `build_install_engine_args`, `build_launch_server_args`, `build_watcher_enable_args`, with the public-bind and system-prefix acknowledgement checks they apply). Functions over the tool's JSON arguments: it uses no other module of the crate, spawns nothing and writes nothing, and [`mcp.rs`](../apps/rocmd/src/mcp.rs) is its only caller. Its own `mod tests` holds the allowlist and argv-builder tests. - [`persistence.rs`](../apps/rocmd/src/persistence.rs) — `record_event`/`load_managed_services`/`load_service_record`, the automation-event/audit-log and managed-service-registry I/O shared across the daemon's sandbox, MCP, service-lifecycle, and watcher code. -- [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) — bubblewrap/native sandbox execution, atomic-write helpers, artifact prefetch policy gating, and the sandbox-tool result shaping for `check_updates`/`driver_plan`/`stop_server`. +- [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) — bubblewrap/native sandbox execution, artifact prefetch policy gating, and the sandbox-tool result shaping for `check_updates`/`driver_plan`/`stop_server`. - [`sandbox_tool.rs`](../apps/rocmd/src/sandbox_tool.rs) — the sandbox-tool vocabulary: the `SandboxToolArg` value enum naming each sandbox tool, and `SandboxToolPolicy`, the artifact-download policy built from the `--allow-*` flags (including the Hugging Face token read from the environment when authenticated downloads are allowed). It uses no other module of the crate, so modules that need the vocabulary do not import [`cli.rs`](../apps/rocmd/src/cli.rs). - [`service.rs`](../apps/rocmd/src/service.rs) — managed-service stop: terminating the recorded processes of a service after verifying their identity, and the stop report (`stop_managed_service`, `stop_report_confirmed`) that the sandbox and MCP tools read. - [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses, and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. @@ -171,6 +171,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): Already-extracted subsystem modules, following the same full-domain-extraction pattern (and several further siblings beyond these): - [`amd_smi.rs`](../crates/rocm-core/src/amd_smi.rs) — (ROCMAI-723) locating the `amd-smi` binary: `resolve_amd_smi_binary` (re-exported at the crate root) prefers the newest managed ROCm SDK runtime recorded in the registry, then the home-directory venv and legacy locations, then the bare name for a `PATH` lookup. It only reads the registry records [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) exposes and [`runtime.rs`](../crates/rocm-core/src/runtime.rs)'s data/home directory lookups. As of EAI-8457 it also owns the *launch*: `amd_smi_json` (also re-exported at the crate root) is the one supported way for a synchronous caller to run `amd-smi`, because it is the only place the mandatory `/dev/kfd` pre-flight runs (`KFD_DEVICE`, `preflight_kfd_device`, `kfd_readable`/`kfd_readable_with`, and the thread-bounded, budget-capped `device_readable`/`device_readable_within_budget` probe behind `KFD_OPEN_TIMEOUT`). A device that cannot be opened declines the spawn rather than letting `amd-smi` park in uninterruptible sleep, where no timeout can reach it. The spawn itself still goes through [`process.rs`](../crates/rocm-core/src/process.rs)'s `run_with_timeout`, and the non-zero-exit quote is bounded with that module's `MAX_QUOTED_OUTPUT_CHARS` rather than a second copy of the constant. Callers: [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s `gpu_vram_usage_amd_smi`/`detect_gpu_count`, both through that module's own `amd_smi_probe_json` (which logs the discarded decline at `info`, since the default filter is `warn,rocm=info` and a decline on a healthy host is otherwise indistinguishable from "no GPU"), and `rocmd`'s [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs)`::capture_amd_smi_json`. That the list stays complete is enforced rather than asserted: [`amd_smi_launch_contract.rs`](../xtask/src/amd_smi_launch_contract.rs) is a static contract test failing any file outside its two-entry allowlist that calls `resolve_amd_smi_binary`, so a new probe cannot be spawned directly without a reviewed exemption. The `rocm dash` collectors keep their own pre-flight *decision* (`preflight_passes`), since they gate an async launch, but share this module's bounded device probe through the `pub fn device_readable_bounded` re-export — an allowlisted `rocm-dash-collectors -> rocm-core` edge recorded in [`crate_edges.rs`](../xtask/src/crate_edges.rs), so the one launch site outside this chokepoint cannot drift away from the bound and the budget. +- [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) — the shared atomic file writer `rocm` and `rocmd` both use (unique exclusive temp file, then rename, or `ReplaceFileW` on Windows); atomic, not durable, since nothing is fsynced. Other hand-rolled temp-then-rename writers remain outside it, for example [`remote/session.rs`](../apps/rocm/src/remote/session.rs)'s session-file write, [`uninstall.rs`](../apps/rocm/src/uninstall.rs)'s profile replacement, `write_atomically` in [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs), and rocm-core's `write_atomic` in [`lib.rs`](../crates/rocm-core/src/lib.rs), which `RocmCliConfig::save` in [`config.rs`](../crates/rocm-core/src/config.rs), the automation event and proposal stores in [`automation.rs`](../crates/rocm-core/src/automation.rs), the JSONL stores in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and the service-record manifests in [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use; it is `pub` because `rocm-engine-lemonade` also writes through it, in [`install.rs`](../engines/lemonade/src/install.rs) (see the `write_atomic` entry below). #541, which builds on this writer, moves the config save onto it; #618 tracks that move. A new atomic write should call this module rather than add another copy. - [`automation.rs`](../crates/rocm-core/src/automation.rs) — (ROCMAI-151) automation-daemon runtime state and its on-disk event/proposal/audit records, and the policy applied to each store (which are rotated, and the proposal compaction) — see the module's own doc comment for what it owns. The generic JSONL machinery under those stores lives in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs). The dependency runs one way: this module reads `WatcherMode` from [`config.rs`](../crates/rocm-core/src/config.rs), and [`config.rs`](../crates/rocm-core/src/config.rs) imports nothing from here, because the built-in watcher catalog (whose entries carry a default `WatcherMode`, and which `RocmCliConfig::watcher_enabled` and `effective_watcher_mode` take as an argument) lives in [`config.rs`](../crates/rocm-core/src/config.rs) beside `WatcherMode`. - [`binary_paths.rs`](../crates/rocm-core/src/binary_paths.rs) — (ROCMAI-151, Phase 7d) locating `rocm-engine-` siblings, and the CLI binary itself (a sibling `rocm` only when running from a cargo `deps/` directory). - [`codex_bridge.rs`](../crates/rocm-core/src/codex_bridge.rs) — (ROCMAI-662) the `CodexBridgeSnapshot`/`CodexBridgeGpuSnapshot`/`CodexBridgeEngine` host-snapshot DTOs. Plain serde data: `rocm` and `rocmd` each build their own snapshot of this shape, so the types live here rather than in either binary. @@ -216,7 +217,7 @@ What else stays in [`lib.rs`](../crates/rocm-core/src/lib.rs), beside module dec - `sanitize_component` — the lowercase-alphanumeric slugger shared by [`service_identity.rs`](../crates/rocm-core/src/service_identity.rs)'s `generate_service_id` and [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs)'s artifact cache-marker names. It is not model-catalog-specific, so it does not move there. - `TORCH_ALIGNMENT_DISABLED_ENV`/`torch_alignment_disabled` — the one read of the torch-alignment opt-out, shared by the CLI and the vLLM engine so the two cannot disagree. - `VLLM_GPU_MEMORY_UTILIZATION_HINT` — a single wording contract between the CLI, the vLLM engine, and this crate's own `fix-16-vllm-oom` diagnosis; `rocm-core` is the only crate all of them can reach. -- **`write_atomic`** — not permanent. It is the temp-then-rename writer [`config.rs`](../crates/rocm-core/src/config.rs), [`automation.rs`](../crates/rocm-core/src/automation.rs), [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use for their atomic replaces. The open [#610](https://github.com/ROCm/rocm-cli/pull/610) adds a shared, stronger atomic writer to this crate for `rocm` and `rocmd`, and the open [#541](https://github.com/ROCm/rocm-cli/pull/541) moves `RocmCliConfig::save` onto it. It is `pub`, visible to other crates, because `rocm-engine-lemonade` also writes Lemonade's `backend_versions.json` through it, in [`install.rs`](../engines/lemonade/src/install.rs). Once #610 lands, the automation and service-record callers follow; `write_atomic` is deleted only after that Lemonade caller has moved to the shared writer too (ROCMAI-724). That is a behavior change, not a pure move, so it is not done here. +- **`write_atomic`** — not permanent. It is the temp-then-rename writer [`config.rs`](../crates/rocm-core/src/config.rs), [`automation.rs`](../crates/rocm-core/src/automation.rs), [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use for their atomic replaces. The shared, stronger writer `rocm` and `rocmd` use is [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs); [#541](https://github.com/ROCm/rocm-cli/pull/541), which builds on it, moves `RocmCliConfig::save` onto it. It is `pub`, visible to other crates, because `rocm-engine-lemonade` also writes Lemonade's `backend_versions.json` through it, in [`install.rs`](../engines/lemonade/src/install.rs). The automation, JSONL-store and service-record callers follow; `write_atomic` is deleted only after that Lemonade caller has moved to the shared writer too (ROCMAI-724). That is a behavior change, not a pure move, so it is not done here. Once the `write_atomic` move lands, [`lib.rs`](../crates/rocm-core/src/lib.rs) holds only module declarations, re-exports, the crate-root primitives above, the private `use` shims that let sibling modules reach some `host_gpu`, `managed_runtime`, `rocm_install` and `runtime` items as `crate::` paths, and its test-only modules — the end state of the modularization effort for this crate (ROCMAI-27, planned last: highest fan-in, since every app and engine crate depends on it, but lowest novelty, since the sibling modules already prove the pattern). From d45d37b4e4064cf3f66325f9c290f4a209d77d83 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 12:11:04 +0000 Subject: [PATCH 009/102] Capture backend identity for Lemonade managed-serve failures (#912) * test(e2e): capture Lemonade backend identity on serve failures Strix Halo Ubuntu managed serves can return success without exposing the model, with logs showing argument rejection, backend-ready/router-unavailable, or download failure. The existing xfails attribute all three to an immediate shutdown. Capture the runtime version, configured pins, candidate backend paths, SHA-256 fingerprints and version/help output alongside the archived service logs before cleanup. Probe duration and retained output are bounded, hashing is capped and runs off the async executor, and timed-out children are killed and reaped. The engine is read from both `--engine X` and `--engine=X`, falls back to the printed serve plan, and the failure report says so when no engine can be determined. Probe capture and kill/reap results are printed only when they fail. Replace the shutdown-only xfail attribution with the observed failure signatures and cover serve-40's shared serve precondition. Intermittent-failure handling and platform scope are unchanged. This is diagnostic coverage, not a confirmed backend fix: configured pins and candidate binaries are evidence of what could have run, not proof of what did. Refs #897 Signed-off-by: Jussi Elo * test(e2e): treat a flag after --engine as no engine in serve failure reports A serve invocation such as 'serve m --engine --managed' made the engine lookup return '--managed'. The failure report then skipped the backend inventory without printing the 'engine unknown' line that makes a skipped inventory visible. A value that starts with '--' is another flag, not an engine, so it now counts as absent, for both the separate and the '--engine=' forms. Signed-off-by: Jussi Elo * test(e2e): keep the per-row flaky rule in the expectations header The header comment replaced in this PR also said that flaky is set per row and is not derived from the bug reference, so a new row for the same bug has to set it explicitly. Restore that sentence next to the rule that keeps flaky=true for the #897 rows. Signed-off-by: Jussi Elo * test(e2e): cover Linux hosts without a usable GPU in the #897 expectation rows The existing regression test resolves the real-GPU rows against the mock capability, whose os_family is 'other', so the os=linux condition excluded it before the therock_family and has_amd_gpu guards were consulted. Removing either guard left the test green even though Linux MockServer-backed bench and chat scenarios would then be expected to fail. Add Linux hosts with no GPU, with a gfx target but no usable GPU, and with a GPU but no gfx target, and assert that a full real-GPU host is still expected to fail. Signed-off-by: Jussi Elo * test(e2e): assert the report says why an oversized backend has no fingerprint The size-cap tests only covered the fingerprint helper. Cover the report itself: a llama-server larger than the hash cap is still listed and probed, and the report prints the reason it has no sha256 line, so the missing fingerprint is visible in the failure output. Signed-off-by: Jussi Elo * docs(testing): state the flag-after---engine, oversized-binary and probe-timing behaviour Document that a value starting with -- after --engine counts as no engine named, that a llama-server over the hash cap is listed and probed with an explicit 'sha256 unavailable' reason, and that the version/help probes run before the failed service is stopped, with the GPU contention question marked as unchecked. Signed-off-by: Jussi Elo * test(e2e): compare backend candidate paths as paths, not printed text The oversized-executable test asserted that the report contained the display form of a path it built with join("bin/llamacpp/rocm-stable"), while the inventory joins "bin/llamacpp" first and the backend directory second. On Windows the two spell the same path with different separators, so the substring check failed even though the report was right. Parse the candidate lines back into paths and compare those; Path equality is component-wise, so both spellings agree. The same helper replaces a negative check in the candidate-cap test that could never fail on Windows: asserting the report did not contain the display form of the skipped candidate passed vacuously there. It now asserts the listed candidates exactly (count, the skipped one absent, the first one present). Signed-off-by: Jussi Elo * fix(e2e): read the Lemonade runtime manifest through the path the engine writes it to The serve failure report reads the installed runtime's manifest to find the backend directory, and built the path with its own "runtime.json" literal next to the engine's. A rename on either side would have degraded the report to "unavailable:" on the one lane it exists for, with nothing failing before then. AppPaths::engine_runtime_manifest_path is now the one definition. The engine's manifest_path and the harness both call it, and an engine test writes a manifest and asserts it lands at that path with the field names the report reads. Signed-off-by: Jussi Elo * fix(e2e): run the llama-server probes after the failed service is stopped The version and help probes executed a backend binary while the failed serve's router could still be running. llama.cpp builds commonly initialise a GPU backend even for --version, so on a shared-memory device a probe could contend with the router and change the state the report exists to describe. docs/testing.md said this had not been checked. The report now reads files first and executes second: manifest, pins, candidate paths, fingerprints and version.txt are taken before the stop, and the probes run after it, with their own 20 second budget so a slow stop cannot use it up. The order lives in one function, backend_evidence_around_stop, which also runs the stop exactly once for every engine, so the serve step cannot reorder or skip it. Its tests make the stop delete the candidate, which shows the fingerprint came before the stop and, through a recording probe, that the probes came after it. docs/testing.md and the expectations header now state what the code does. Signed-off-by: Jussi Elo * fix(e2e): search deeper for llama-server candidates and say where an empty search looked The inventory looked only at depth 0 and 1 under four backend directories, while the router chooses how a llama.cpp build is unpacked, so a build one level deeper would have produced no candidates and no probe. When nothing was found, the report also suggested the backend download may not have completed, which an unsearched layout or a never-installed build would equally explain. The search now goes three directory levels below each backend directory, bounded by an entry cap per directory. An empty result names every location searched, the depth, that the cpu backend is deliberately left out, and what bin/llamacpp actually contains, without naming a cause. docs/testing.md states the search. Signed-off-by: Jussi Elo * test(e2e): expect the depth-limit candidates in the order the search sorts them The previous commit's depth test listed its two expected candidates in the wrong order: the search sorts sibling directories by path, so a/b comes before a/build. The test therefore failed even with the depth limit in place. With the order corrected it passes, and with the limit lowered to two it fails because the three-level candidate is no longer found. Signed-off-by: Jussi Elo * fix(e2e): fingerprint llama-server builds up to 512 MiB instead of 32 MiB A server above the old 32 MiB cap got no fingerprint, so the report would be missing its identity evidence on exactly the kind of build it is meant to describe. A dynamically linked llama-server is a few MiB, but one that links the HIP kernels for several GPU targets statically can be hundreds, and neither the repository nor this environment has the release asset sizes to settle it. The cap is now 512 MiB. Hashing already streams 64 KiB at a time and checks its deadline between reads, so memory stays bounded and time is bounded by the inventory budget; nothing about the cap changes either. The over-cap line is worded from the constant so it cannot drift from it, the boundary test uses a small limit instead of hashing half a gibibyte, and a separate test shows a file above the old cap is fingerprinted. docs/testing.md states the cap and that it is an assumption about the builds. Signed-off-by: Jussi Elo * test(e2e): run the probe timeout and kill path on every platform The probe's timeout, kill and reap handling was tested only through shell-script llama-server stand-ins, which exist on unix, so the Windows lane never ran it and the change did not say so. The probe is split into building the llama-server command and running a command with bounded time and output, so the second part can run any child. Two tests re-run the unit-test binary itself as that child, through an ignored helper test selected by an environment variable: one lets the probe time out and requires the child to be gone afterwards (killed and reaped, not merely abandoned), the other covers a child that exits and its status and output. Replacing the kill with a leak turns the first red. The kill/reap failed branch stays untested, because it needs an operating system that refuses to kill a child it just started, and docs/testing.md says so. Signed-off-by: Jussi Elo * test(e2e): keep the revisit-before-deleting caveat in the expectations header The header rewrite for the #897 rows dropped the sentence telling the next person to revisit before deleting any of these rows, because an XPASS streak is not evidence of a fix while the lane cannot say which Lemonade build a run used. That is still true until a failed run there has carried the backend evidence this branch adds, so it is back, reworded for that. The sentence that flaky is set per row and not derived from bug was restored earlier and is unchanged. Signed-off-by: Jussi Elo * fix(e2e): make the stop-ordering helper and its tests pass the CI clippy lints The CI clippy runs found what the unit tests could not: the async helper and its Probe trait were not Send, two helpers could be const, one assertion was on a constant, and a probe test spelled a minute in seconds. A Send helper cannot borrow the scenario world, which holds PTY handles and is not Sync, so the serve step now hands it the isolated root and the app paths by value, and the stop is split into one function on the world and one on the root path. The ordering tests use a mutex and an atomic instead of cells, and the size-cap assertion is a const block. Signed-off-by: Jussi Elo * docs(testing): record that a current ROCm llama-server fits under the old hash cap The Strix Halo Ubuntu lanes on this branch fingerprinted the rocm-stable b12109 llama-server under the former 32 MiB cap, so the cap change is margin for larger static builds and not a repair for current ones. The cap's note said the size was an assumption; it now says what was observed and what remains an assumption. Signed-off-by: Jussi Elo * fix(e2e): bound the candidate search by time and size, and keep it off the async executor The three-level candidate search added earlier ran before the inventory's deadline existed, applied the eight-candidate cap only after walking everything, read directories on an async worker, followed symlinks, and listed an unbounded bin/llamacpp when it found nothing. A large or linked cache could therefore delay the stop and the failure report well past the intended budget. The deadline now starts at inventory entry and is read at every directory and entry during the walk, through an injected clock so a test drives it without sleeping. Directory and entry limits (128 and 2048) apply to the whole search, not per directory, and the walk ends as soon as it has eight candidates, in sorted order. Symlinks are never entered or taken as a candidate, so a cycle ends and a link out of the tree is not read; the report counts them. A search cut short by the clock or a limit says so instead of claiming an empty result, and the listing of what bin/llamacpp holds reads at most 64 entries, shows 16 and says how many it left out. The manifest and pin reads, the search and each candidate's fingerprint run on blocking threads with the in-traversal deadline, since a timeout around a blocking task only drops its handle. docs/testing.md states the limits. Signed-off-by: Jussi Elo * fix(e2e): do not say the search found nothing when a limit ended it early The empty-search explanation, which lists every backend directory as searched, was suppressed only when the clock ended the search. A search stopped by the directory or entry limit still produced it, so with a real build under rocm-stable/b and the cap reached before b was entered, the report said all four backend trees had been searched and held nothing. The explanation is now emitted only when the search finished, and every early stop reports its own reason instead. The regression test uses that exact tree and asserts the empty-result claim is absent for the directory limit, the entry limit and the clock, and present for a finished search that found nothing. The candidate cap needs no row: it only stops a search that already has candidates. Signed-off-by: Jussi Elo * fix(e2e): find the helper child's pid when libtest runs it on one test thread The probe timeout test reads the pid its helper child prints. With one test thread (RUST_TEST_THREADS=1, or a one-CPU host) libtest writes the test-name label first and the child's output continues that line, so the line-start match for the marker found nothing and the test failed even though the child had been killed and reaped. The child now starts a fresh line before the marker, and the test runs the child both on one test thread and on several (overlapping, so it costs one timeout), so the single-thread case is exercised wherever the suite runs. With the newline removed the single-thread run fails; it failed the same way before this change under RUST_TEST_THREADS=1. Signed-off-by: Jussi Elo * fix(e2e): name the engine install manifest path so the registry writer guard does not match it The registry guard in runtime_registry.rs reads each production file as text and flags one that both writes a file and spells runtime_manifest_path(. The shared helper added for the Lemonade install manifest was called engine_runtime_manifest_path, which contains that spelling, and the Lemonade installer both calls it and writes its manifest, so the guard reported a second registry writer. It only showed once the guard reached main. The helper is now AppPaths::engine_install_manifest_path, which also says what it is: the engine's own install manifest, not an entry in the runtime registry. The doc comment records why the names must stay apart. The path it returns is unchanged, and so is the guard. Signed-off-by: Jussi Elo * fix(e2e): refuse symlinked files the report reads, not only symlinked directories The directory walk counted a symlinked version.txt as a link it did not follow, but the report then read version.txt through read_clamped, which follows it, so a real llama-server beside a marker linking outside the backend directory put the outside file's content into the report while the report said the link was skipped. Every file the report reads from the tree now goes through the same rule: the manifest, the pins file, version.txt and a candidate's fingerprint target are refused when the last path component is a symlink, and the report says "symlink not followed" in place of the content. The symlink regression fixture gains the real server with a version marker, pins file and manifest that link outside the tree; the test asserts the outside content is absent and each refusal is named. Letting the link be followed again turns it red. Signed-off-by: Jussi Elo * fix(e2e): keep the stop and the failure report when the Lemonade manifest lookup fails The serve step looked the runtime manifest up with an expect that runs before the service is stopped. For a Lemonade serve without an isolated scenario root it panicked, so the stop never ran and the whole failure report was lost, where the code before the evidence change reported "not attempted: scenario has no isolated root". The lookup now returns Err(reason) when there is nothing to look for, and a panic inside it is caught and treated the same way. The report says "backend evidence unavailable: ", the stop still runs exactly once, and nothing is probed. The only other expect on that path is on a child pipe that is configured to be piped and runs after the stop. The test covers both the Err and the panic and goes red when the panic guard is removed. Signed-off-by: Jussi Elo * fix(e2e): say so when a log grew past the cap after its size was read read_clamped read a file whose size fit the cap with take(max), so a log that grew after the size was taken was cut at the cap with no note, for the archived service log and for the pins and version.txt in the Lemonade report alike. The note existed only on the branch for a file that already exceeded the cap. The read now asks for one byte past the cap. If it is there, the file grew, and it is clamped the way an oversized file is, head and tail with the gap marked, and the note says the file grew while it was read. A file that fits is still returned verbatim with no note. The size is a parameter of an inner function so a test can hand it a stale one; with the extra byte not requested the test is red. All callers consume the note (the manifest read discards it, but a cut JSON manifest fails to parse and the report says so). Signed-off-by: Jussi Elo * test(e2e): point the Lemonade serve xfail rows at the open ticket GitHub issue #897 is closed, but the xfail rows, their header comments, the harness tests that pin them and the docs still named it as the live tracker. The open ticket is ROCMAI-898, so every row, comment, test and doc line that named #897 as the tracker now names ROCMAI-898, with one note in the header that it was first tracked as #897 and that issue is closed. The issue stays closed. The stale-xfail hint reads the ticket ids out of the matrix and hard-codes none, so it follows the new id on its own; its self-test now covers that (a PR that mentions ROCMAI-898 is told about the rows, one that mentions #897 is not). The rows' conditions, timeouts and flaky handling are unchanged. Signed-off-by: Jussi Elo * fix(e2e): refuse a symlink in the open itself, not in a check before it The report refused symlinked files by reading the path's metadata and then opening it, which left a window in which the path could become a link while the docs said links are refused. On unix the open now passes O_NOFOLLOW (through the libc the crate already depends on, so no new dependency), so a link is refused whenever it is there at the moment of the open, with the same "symlink not followed" message. Manifest, pins, version.txt and fingerprint reads all go through that one helper. On Windows the behaviour is unchanged: a metadata check followed by an open. A swap between the two is not caught there, and docs/testing.md says so, and that only the last path component is checked. A unix test calls the open helper directly on a link, which is what a swap after the check looks like to it, and goes red when O_NOFOLLOW is removed. Signed-off-by: Jussi Elo * fix(e2e): report a truncated candidate list only when it is truncated, and say the budget is out once Two cosmetic lies in the Lemonade report. The "truncated after 8 candidates" note was printed when exactly eight candidates existed, because the search stopped on reaching the cap and could not tell whether a ninth existed. And "remaining evidence skipped: evidence budget exhausted" was printed twice when the search ended on the clock and the candidate loop then found the budget gone too. The search now keeps eight and stops on finding a ninth, so the note appears only when there is something not listed, and exactly eight is a complete list. The budget line is said once. Both have tests, and each turns red when the old behaviour returns. docs/testing.md states the ninth-candidate rule. Signed-off-by: Jussi Elo * test(e2e): do not fail the probe kill test when the helper child starts slowly The kill and reap test lets a helper child run for a fixed 5 seconds and reads its pid from the probe's report. On a loaded runner a cold test binary can take longer than that to start and print the pid, which failed the test for a slow start and not for anything the probe did. An attempt whose child printed no pid before the timeout is now retried with a longer timeout (20 seconds, then 60), and only an attempt that did report a pid is asserted on, so the generous waits matter only when the start is slow and a healthy run still takes 5 seconds. With a child that takes 7 seconds to start the test passes on the second attempt, and with the retries removed it fails. Signed-off-by: Jussi Elo * test(e2e): keep the helper child alive past the longest probe attempt The kill and reap test retries with timeouts of 5, 20 and 60 seconds, but the helper child exited by itself after 30. If the first two attempts timed out before the child printed its pid and the last one started it, the child exited normally before the 60 second timeout, so the probe saw a normal exit, not a timeout, and the test failed for a correct probe. Reproduced with a child that takes 25 seconds to start: the third attempt reports "exit status: 0". The helper's lifetime is now derived from the attempt timeouts (the longest plus a 60 second margin), so they cannot drift apart, and a test pins that the child outlives every attempt by at least 30 seconds. With the same 25 second start the test now passes on the third attempt; with a literal lifetime or no margin the pinning test fails. A child that takes 7 seconds to start still passes on the second attempt and fails with the retries removed. The other helper lifetimes in the file (30 second sleeps against 1 second timeouts, an immediate exit against a 60 second timeout) have no such pair. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit a3b40141efea4f41a250cad6f844468019d7d3bc) Signed-off-by: juhovainio --- Cargo.lock | 1 + crates/rocm-core/src/paths.rs | 24 + docs/testing.md | 107 +- engines/lemonade/src/install.rs | 39 +- scripts/xfail_expectations_hint.py | 14 + tests/e2e-cucumber/Cargo.toml | 3 +- tests/e2e-cucumber/expectations.toml | 109 +- .../features/model_serving.feature | 2 +- tests/e2e-cucumber/src/expectation.rs | 102 +- .../e2e-cucumber/src/lemonade_diagnostics.rs | 2116 +++++++++++++++++ tests/e2e-cucumber/src/lib.rs | 1 + tests/e2e-cucumber/src/serve_log.rs | 187 +- tests/e2e-cucumber/tests/e2e/serving_steps.rs | 84 +- 13 files changed, 2683 insertions(+), 106 deletions(-) create mode 100644 tests/e2e-cucumber/src/lemonade_diagnostics.rs diff --git a/Cargo.lock b/Cargo.lock index 2151dc3607..dc61a39a04 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1284,6 +1284,7 @@ dependencies = [ "rocm-core", "serde", "serde_json", + "sha2", "shlex", "tempfile", "tokio", diff --git a/crates/rocm-core/src/paths.rs b/crates/rocm-core/src/paths.rs index 15840e56e9..7d526244e4 100644 --- a/crates/rocm-core/src/paths.rs +++ b/crates/rocm-core/src/paths.rs @@ -194,6 +194,21 @@ impl AppPaths { self.engine_dir(engine).join("manifests") } + /// The manifest an engine's installer writes for its installed runtime. + /// + /// One definition for the writer (the engine) and for every reader of the + /// file, so a reader cannot drift to a different name: a reader that + /// misses it degrades to "no manifest", which is indistinguishable from an + /// engine that was never installed. + /// + /// This is the engine's own install manifest, not an entry in the runtime + /// registry (that is `apps/rocm`'s `runtime_manifest_path`), and the name keeps + /// the two apart: the registry has a text guard that treats any file spelling + /// that helper's name and writing a file as a second registry writer. + pub fn engine_install_manifest_path(&self, engine: &str) -> PathBuf { + self.engine_manifests_dir(engine).join("runtime.json") + } + pub fn engine_state_dir(&self, engine: &str) -> PathBuf { self.engine_dir(engine).join("state") } @@ -355,6 +370,15 @@ mod tests { paths.data_dir.join("automations").join("proposals.jsonl") ); } + #[test] + fn engine_install_manifest_lives_in_the_engine_manifests_dir() { + let (_root, paths) = temp_app_paths("engine-runtime-manifest-path"); + assert_eq!( + paths.engine_install_manifest_path("lemonade"), + paths.engine_manifests_dir("lemonade").join("runtime.json") + ); + } + #[test] fn with_managed_root_keeps_reprovisioning_flat_from_runtime_leaf() { let data_root = PathBuf::from("/tmp/rocm-cli-reprovision"); diff --git a/docs/testing.md b/docs/testing.md index b3369520b6..d984e61639 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1139,6 +1139,111 @@ Docker exercises AIM serve. The `e2e-gpu-nightly-strix` job logs whether Docker is reachable ("Report whether Docker is reachable"); check that line before relying on this coverage. +### Lemonade managed-serve failure evidence + +On a failed serve, the shared serve steps preserve the full service log under +`service-logs/` in the E2E results artifact before stopping the service. The engine +is taken from `--engine X` or `--engine=X`, or else from the printed serve plan; if +neither names one, the report says `engine unknown: backend evidence skipped`. A +value that starts with `--` (as in `serve m --engine --managed`) is another flag, not +an engine, and counts as no engine named. +Lemonade failures also include the installed runtime manifest's version/backend, configured +`backend_versions.json`, candidate GPU `llama-server` paths, SHA-256 fingerprints +and `version.txt`, +and each candidate's `--version` and `--help` output. Candidates are searched +under the `rocm-stable`, `rocm-nightly`, `rocm` and `vulkan` directories of +`bin/llamacpp`, down to three directory levels below each; the cpu backend is not +searched. The search is bounded as a whole, not per directory: it reads at most +128 directories and 2048 directory entries across all four backend directories, +stops as soon as it finds a ninth candidate (it lists the first eight and says the +list is cut; exactly eight is not reported as cut), and stops when the +inventory's time is up, checking the clock at every directory and entry. Siblings are visited in +sorted order, so a search that finishes inside its limits is deterministic. +Symlinks are never followed: only real directories are entered and only a regular +file named `llama-server` counts, so a link cycle ends and a link out of the +backend directory is not read; the report says how many symlinked entries it +declined to follow. The same rule covers every file the report reads: the +manifest, the pins file, `version.txt` and a candidate's fingerprint target are +refused when the last path component is a symlink, and the report says `symlink +not followed` in place of the content. On unix the refusal is made by the open +itself (`O_NOFOLLOW`), so a path swapped for a link after an earlier check is still +refused; on Windows it is a metadata check followed by an open, so a swap between +the two is not caught (read-only evidence, and the swap needs write access to the +backend directory). Only the last component is checked: a link in a parent +directory is followed. When a limit or the clock ends the search early the report +says which, and does not claim an empty result. When nothing is found, the report +lists the searched locations and what `bin/llamacpp` holds (the first 16 names +out of at most 64 entries read, with a count of those omitted), and does not +guess at the cause: an interrupted download, a layout outside the search and a +build that was never installed look the same. The manifest and pin reads, the +search and each candidate's fingerprint run on blocking threads, never on an +async worker, and the search carries its own deadline because a timeout around a +blocking task would only drop its handle. +At most eight candidates +are hashed and probed, with two read-only probes per candidate. Each probe retains +up to 32 KiB of stdout and 32 KiB of stderr; only streams that discard bytes are +marked as truncated. Across sixteen probes this is up to 1 MiB of retained raw +output, plus metadata and formatting (invalid UTF-8 bytes may expand when rendered). +Each probe has a five-second timeout. The inventory has a 20-second budget that +starts when it is entered and covers the manifest and pin reads, the search and +the hashing; the probes have their own 20-second budget, which starts +when the service stop returns; hashing runs off the async executor, rejects files +over 512 MiB, and checks its deadline between 64 KiB reads, so memory stays at +one buffer whatever the size. The cap sits well above a dynamically linked +`llama-server`: the `rocm-stable` build b12109 on the Strix Halo Ubuntu lanes +was fingerprinted under the former 32 MiB cap, so current builds fit with room to +spare. The larger cap is margin for a statically linked multi-target build, +which is an assumption about future builds, not a measured size, and a larger +file is a finding in its own right. A candidate over the size cap gets no fingerprint, and +its entry says so with `sha256 unavailable: executable exceeds 512 MiB hash +limit`; it is still listed and probed. Filesystem +operations and process cleanup are not preemptible, so this is not a hard +wall-clock limit on the entire report. +A candidate or configured build pin is **not proof** of which binary the router +launched; a failed probe is retained as evidence, not treated as an identity. +Probes inherit the harness environment rather than reconstructing the engine's +managed library paths, so a probe loader error alone does not establish why the +serve failed. The report reads files first and executes a backend second. The +manifest, pins, candidate paths, fingerprints and `version.txt` are read once the +serve has exited non-zero or its readiness wait has timed out and the log has +been read and archived, but before the service is stopped, so they describe the state being +diagnosed. The `--version` and `--help` probes run after the stop returns, in the +report's "probes, run after the service stop was attempted" section: llama.cpp +builds commonly initialise a GPU backend even for `--version`, and doing that next +to a still-running router on a shared-memory device could change the state being +diagnosed. A stop that fails or finds no record can leave the router running; the +report's `stop` section says so, and the probes still run. No lane has shown +whether a probe initialises a GPU backend, which is why they are kept off the +router's side of the stop. The order is fixed in one function, +`backend_evidence_around_stop`, whose unit tests make the stop delete the candidate +to show the fingerprint was read before it and the probes came after it. Nothing +before the stop may keep it from running: a serve with no isolated scenario root +has no manifest to look for, so the report says `backend evidence unavailable: +the scenario has no isolated root` (and the stop's own status says it was not +attempted), and a panic while looking the manifest up is caught and reported the +same way, so the stop runs exactly once and the failure report is not lost. + +The probe's timeout, kill and reap path is tested on every platform, Windows +included: the unit tests re-run the test binary itself as the child (an ignored +helper test selected by an environment variable), let the probe time out, and +require that the child is gone afterwards. The shell-script `llama-server` +stand-ins used by the other probe tests exist on unix only. Not covered anywhere +is the `kill/reap failed` branch, which needs an operating system that refuses to +kill a child it just started. + +Use the evidence to distinguish backend startup, download, and router readiness +failures; ROCMAI-898 tracks the Strix Halo Ubuntu investigation (first tracked +as [issue #897](https://github.com/ROCm/rocm-cli/issues/897), which is closed). A +backend's ready line does not establish that the router serves the requested +model, and a readiness timeout alone does not establish a shutdown. Diagnostics do not change the CLI's asynchronous +managed-launch contract or imply a backend fix. + +Exercise the diagnostics and expectation scoping without a GPU: + +```bash +cargo test -p e2e-cucumber --lib lemonade +``` + ## Runtime Selection And Activation List registered managed or read-only runtimes and their exact side-by-side keys: @@ -2146,7 +2251,7 @@ exit) and `serve-40` in (GPU lane: serving with no watcher enabled starts no helper; on a Linux host that runs it with lemonade it is an expected failure in [expectations.toml](../tests/e2e-cucumber/expectations.toml) while lemonade -serving on the Strix Halo Ubuntu lanes is broken (#897), so the helper check is +serving on the Strix Halo Ubuntu lanes is broken (ROCMAI-898), so the helper check is not reached on those lanes. The Strix Halo Windows lane runs it and passes, and the row also matches other Linux lemonade hosts, where a pass is tolerated). diff --git a/engines/lemonade/src/install.rs b/engines/lemonade/src/install.rs index e0fa931d11..5c67cbcad4 100644 --- a/engines/lemonade/src/install.rs +++ b/engines/lemonade/src/install.rs @@ -196,7 +196,7 @@ where } fn manifest_path(paths: &AppPaths) -> PathBuf { - paths.engine_manifests_dir(ENGINE_NAME).join("runtime.json") + paths.engine_install_manifest_path(ENGINE_NAME) } fn lemonade_root(paths: &AppPaths, env_root: Option<&Path>) -> PathBuf { @@ -573,6 +573,43 @@ mod tests { #[cfg(windows)] use std::process::{Command as ProcessCommand, Stdio}; + /// The e2e harness reads this manifest back to describe a failed serve + /// (`tests/e2e-cucumber/src/lemonade_diagnostics.rs`), by the same path + /// helper and the field names asserted here. A rename on either side would + /// otherwise show up only as `unavailable:` in a report from the real lane. + #[test] + fn written_manifest_is_where_and_how_the_e2e_failure_report_reads_it() { + let dir = scratch_dir("manifest-path-and-fields"); + let paths = AppPaths { + config_dir: dir.join("config"), + data_dir: dir.join("data"), + cache_dir: dir.join("cache"), + }; + let manifest = LemonadeInstallManifest { + env_id: "lemonade-embeddable-fixture".to_owned(), + version: "fixture".to_owned(), + runtime_dir: dir.join("runtime"), + lemond: dir.join("runtime/lemond"), + lemonade: dir.join("runtime/lemonade"), + backend_recipe: LLAMACPP_RECIPE.to_owned(), + backend_name: ROCM_BACKEND_NAME.to_owned(), + installed_at_unix_ms: 1, + }; + write_manifest(&paths, &manifest).unwrap(); + + let path = paths.engine_install_manifest_path("lemonade"); + assert_eq!(manifest_path(&paths), path); + let written: serde_json::Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + assert_eq!(written["version"], "fixture"); + assert_eq!(written["backend_recipe"], LLAMACPP_RECIPE); + assert_eq!(written["backend_name"], ROCM_BACKEND_NAME); + assert_eq!( + written["runtime_dir"], + serde_json::json!(manifest.runtime_dir) + ); + fs::remove_dir_all(&dir).ok(); + } + #[test] fn embeddable_package_matches_runtime_os() { let version = rocm_deps::LEMONADE_VERSION; diff --git a/scripts/xfail_expectations_hint.py b/scripts/xfail_expectations_hint.py index 7a8439f6ce..ff38838d13 100644 --- a/scripts/xfail_expectations_hint.py +++ b/scripts/xfail_expectations_hint.py @@ -143,6 +143,11 @@ def emit(hits: dict[str, list[str]]) -> None: when = {} bug = "#123" reason = "example" + +[["serve-lemonade-inference"]] +when = { effective_engine = "lemonade", os = "linux" } +bug = "ROCMAI-898" +reason = "example" """ @@ -188,6 +193,15 @@ def check(title="", body="", commits=""): assert hits == {"EAI-7333": ["serve-vllm-inference"]}, hits print("self-test: refs read from commit messages ok") + # A row keyed to an open tracker id follows that id, and an id that is no + # longer in any row (a closed issue the rows were repointed away from) does + # not match: the hint reads the ids out of the matrix, it hard-codes none. + hits = check(body="Refs ROCMAI-898") + assert hits == {"ROCMAI-898": ["serve-lemonade-inference"]}, hits + hits = check(body="Refs #897") + assert hits == {}, hits + print("self-test: hint follows the id in the matrix (ROCMAI-898 yes, #897 no) ok") + print("xfail hint self-test: ok") diff --git a/tests/e2e-cucumber/Cargo.toml b/tests/e2e-cucumber/Cargo.toml index 48fe7bbeac..8a46d60c7c 100644 --- a/tests/e2e-cucumber/Cargo.toml +++ b/tests/e2e-cucumber/Cargo.toml @@ -61,8 +61,9 @@ rand.workspace = true reqwest = { version = "0.13", features = ["json"] } serde.workspace = true serde_json.workspace = true +sha2.workspace = true tempfile = "3" -tokio.workspace = true +tokio = { workspace = true, features = ["process", "io-util"] } toml = "1.1" # `ServeDir` backs the install scenario's loopback download server (see # `src/loopback_http.rs`), so the E2E suite does not hand-roll HTTP framing, diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index 340bc546d9..fcb0b52a4e 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -58,60 +58,53 @@ when = {} bug = "EAI-7219" reason = "A vLLM serve of a short name does not resolve to lemonade's full model name: it either stops before printing a serve plan or prints the alias unexpanded." -# EAI-7423 was INTERMITTENT on Strix Halo Ubuntu as of 2026-08-05, not fixed: -# which of its scenarios pass varied run to run (run 30995473578: 2 XPASS; run -# 30993213188: 3 XPASS; run 30939967076: all 6). A deterministic-XPASS row fails -# that lane on unrelated PRs, so the EAI-7423 lemonade-linux rows below carry -# `flaky = true` — either outcome is tolerated while the bug stays open. The flag -# is per-row and is NOT derived from `bug`, so a new EAI-7423 row needs it added -# explicitly. Revisit under EAI-7853 before deleting any of them: which lemonade -# build a given run used cannot currently be read back from this lane, so an -# XPASS streak here is not by itself evidence that the bug is fixed. +# ROCMAI-898 tracks multiple Strix Halo Ubuntu Lemonade serve signatures (first +# tracked as GitHub issue ROCm/rocm-cli#897, since closed; ROCMAI-898 is the open +# ticket and what the rows below name): +# the child rejects --no-mmap (41984921053), backend readiness on :8001 without +# the model appearing on :11435 (41984503055), and backend download failure +# (41986772728). A backend ready line does not prove a later shutdown. +# EAI-7423 described only the earlier ready-then-unavailable symptom. +# Retain flaky=true because historical runs also passed intermittently; do not +# infer a fix from a single XPASS. The flag is per-row and is NOT derived from +# `bug`, so a new ROCMAI-898 row needs `flaky = true` added explicitly. +# Failed serves now capture configured pins, candidate executable paths and +# fingerprints before stopping the service, and each candidate's --version/--help +# after it. +# Revisit under EAI-7853 before deleting any of these rows: which lemonade build +# a given run used could not be read back from this lane, and until a failed run +# there has carried that evidence, an XPASS streak is not by itself evidence that +# the bug is fixed. # -# On a lemonade-native LINUX host (Strix Halo Ubuntu) the lemonade managed serve -# loads the model and reaches ready on :8001, then is shut down ~0.08s later, so -# the endpoint never becomes reachable (EAI-7423, root-caused on the box -# 2026-07-15 — NOT the earlier "backend install hides the plan line" theory, which -# was disproven: the `engine:` line IS emitted). os=linux: on native Windows this -# scenario PASSES (run 29338475133 Strix-Windows), so it must NOT be xfail'd there. +# os=linux: native Windows passed (29338475133 Strix-Windows). [["serve-default-engine-working-endpoint"]] when = { effective_engine = "lemonade", os = "linux" } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so the endpoint never becomes reachable." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true -# On a lemonade-native LINUX host the served model's endpoint drops before -# inference: the lemonade managed serve reaches ready then is shut down ~0.08s -# later (EAI-7423, root-caused on the box 2026-07-15 — not a Vulkan hang). -# os=linux: passes on native Windows (run 29338475133). +# The same serve precondition blocks inference; native Windows passed. [["serve-default-engine-inference"]] when = { effective_engine = "lemonade", os = "linux" } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so inference never succeeds." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true -# On a lemonade LINUX host this scenario does a real lemonade serve, which reaches -# ready then shuts down immediately (EAI-7423, see serve-lemonade-inference). -# os=linux: on native Windows lemonade serve works, so this PASSES there (verified -# run 29338475133 Strix-Windows XPASS) and must not be xfail'd. +# The readiness contract depends on the same real Lemonade serve. [["serve-readiness-contract"]] when = { effective_engine = "lemonade", os = "linux" } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so the ready-to-serve contract fails." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true -# --- EAI-7423: on a lemonade LINUX host the managed serve reaches ready then is -# shut down immediately, so inference never happens (root-caused on the box -# 2026-07-15; previously mis-attributed to the EAI-7052 Vulkan hang). os=linux: -# native Windows lemonade serve works (run 29338475133 Strix-Windows XPASS'd -# this), so scope the xfail to Linux. --- +# Explicit Lemonade serving shares the same failure on the Ubuntu lane. [["serve-lemonade-inference"]] when = { effective_engine = "lemonade", os = "linux" } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so inference never succeeds." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true @@ -124,16 +117,18 @@ flaky = true # the scenario with lemonade (the MI300X and W7900 lanes had not reported in that # run); whether lemonade serves there is not verified, and `flaky = true` tolerates # a pass on such a host. The cause on the Strix Halo Ubuntu lanes is not settled -# and varies between runs (#897), so this row names that issue rather than EAI-7423 +# and varies between runs (ROCMAI-898), so this row names that ticket rather than +# EAI-7423 # and carries `flaky = true` like its siblings. While it is in place the # scenario's own check (serve starts no background helper when no watcher is # enabled) is not exercised on the hosts it matches, and a genuine failure of that # check there would also read as a failure that was expected. `flaky` also means a -# fixed #897 is not flagged as a stale row, so delete the row by hand when #897 is +# fixed ROCMAI-898 is not flagged as a stale row, so delete the row by hand when +# ROCMAI-898 is # closed. [["serve-stop-leaves-no-background-helper"]] when = { effective_engine = "lemonade", os = "linux" } -bug = "#897" +bug = "ROCMAI-898" reason = "Lemonade serving fails on the Strix Halo Ubuntu lanes, so the model never appears on /v1/models and the helper check is never reached." serve_timeout_secs = 90 flaky = true @@ -145,16 +140,11 @@ bug = "EAI-7223" reason = "Chat requests carrying tool definitions are not accepted by the served vLLM model." serve_timeout_secs = 90 -# On a lemonade LINUX host this needs a live lemonade serve, which reaches ready -# then shuts down immediately (EAI-7423). os=linux: passes on native Windows -# lemonade (run 29354305288 XPASS'd it), consistent with the other lemonade -# serve scenarios. therock_family=gfx* scopes this to a REAL GPU host: the -# scenario now also runs on the no-GPU mock lane (backed by MockServer, not a -# real lemonade serve), where EAI-7423 does not apply and it must expect-pass. +# Chat's mock lane must remain expect-pass: only a real GPU serve is affected. [["chat-tool-definitions-accepted"]] when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true @@ -163,27 +153,16 @@ flaky = true # scenario exercises the CLI `rocm chat --prompt` path (which correctly discovers the # model via /v1/models). Verified expect-pass on MI300X vLLM (XPASS on run 29322186691, # then 3/3 green re-runs), so there is NO vLLM xfail for this scenario — the CLI chat -# path works. Only the lemonade LINUX path stays xfail: it needs a live lemonade -# serve, which reaches ready then shuts down immediately (EAI-7423); native Windows -# lemonade serve works (run 29338475133 XPASS). therock_family=gfx* scopes this to -# a REAL GPU host: the scenario also runs on the no-GPU mock lane (MockServer, not -# a real lemonade serve), where EAI-7423 does not apply and it must expect-pass. +# path works. Only the real Lemonade Linux serve stays xfail under ROCMAI-898; +# the no-GPU MockServer path and native Windows remain expect-pass. [["chat-end-to-end-local-model"]] when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true -# The GPU-backed bench scenario benchmarks a really served model, so on a -# lemonade LINUX host it inherits the same managed-serve bug as the sibling -# serve/chat rows below `serve-default-engine-working-endpoint`: the serve -# reaches ready then shuts down immediately, so this scenario's `a model is -# being served on GPU` precondition never yields a live endpoint. Scoped -# identically — os=linux only, because native Windows lemonade serve works, and -# therock_family=gfx* so it applies only on a real GPU host. `flaky = true` -# matches those rows too: which of them pass varies run to run on that lane, and -# a deterministic-XPASS row would fail the lane on unrelated PRs. +# Bench's real-serve precondition is blocked by ROCMAI-898 on the same Linux hosts. # # This does NOT weaken the coverage the bench fix relies on: the five # MockServer-backed bench scenarios (bench-01 to bench-05) run unxfail'd on @@ -191,7 +170,7 @@ flaky = true # failure reporting, the CSV columns, and the rollup-split warning. [["bench-load-real-serve"]] when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*", has_amd_gpu = true } -bug = "EAI-7423" -reason = "Lemonade managed serve reaches ready then shuts down immediately, so the benchmark never reaches a live model." +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index ffbb2fd692..64dd0d9028 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -75,7 +75,7 @@ Feature: Model serving # (`serve_hf_checkpoint`), which bails explicitly if no backend binary is found # under bin/llamacpp// — unlike the short-recipe-name path above, whose # managed-lemonade fallback would mask the identical failure behind the - # unrelated EAI-7423 xfail. Reuses the same small Qwen3-0.6B-GGUF checkpoint as + # ROCMAI-898 xfail. Reuses the same small Qwen3-0.6B-GGUF checkpoint as # `serve-lemonade-inference` (cache-shared, no extra download) so this stays a # fast per-PR canary rather than needing the @nightly large-checkpoint path. @id:serve-hf-checkpoint-inference @requires-gpu @requires-engine:lemonade diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index 01ed93ca03..3e7a5c13c8 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -838,6 +838,102 @@ mod tests { } } + #[test] + fn lemonade_serve_failures_track_rocmai_898_without_xfailing_mock_or_windows() { + let matrix = Expectations::parse(include_str!("../expectations.toml")).unwrap(); + for id in [ + "bench-load-real-serve", + "chat-tool-definitions-accepted", + "chat-end-to-end-local-model", + "serve-lemonade-inference", + "serve-stop-leaves-no-background-helper", + ] { + let scenario = decl(&[&format!("id:{id}")]); + assert!( + matches!( + resolve(&scenario, &cap("strix-ubuntu"), &matrix, Included::default()), + Expectation::ExpectXfail { bug, flaky: true, .. } if bug == "ROCMAI-898" + ), + "{id}" + ); + assert_eq!( + matrix.serve_timeout_for(id, &cap("strix-ubuntu"), "lemonade"), + Some(90), + "{id}", + ); + for platform in ["mock", "strix-windows"] { + assert_eq!( + resolve(&scenario, &cap(platform), &matrix, Included::default()), + Expectation::ExpectPass, + "{id} on {platform}", + ); + } + } + } + + /// The rows that carry `therock_family` and `has_amd_gpu` exist so that the + /// no-GPU lane, where the same scenarios run against MockServer instead of a + /// real Lemonade serve, stays expect-pass. `cap("mock")` cannot prove that: + /// its `os_family` is `other`, so the `os = "linux"` condition excludes it + /// before either GPU guard is consulted. These hosts are Linux, so each guard + /// is the only thing standing between them and an xfail. The two partial + /// hosts isolate the guards from each other: dropping `has_amd_gpu` from a row + /// fails the host that reports only a gfx target, and dropping `therock_family` + /// fails the host that reports only a GPU. + #[test] + fn real_gpu_lemonade_rows_do_not_xfail_linux_hosts_without_a_usable_gpu() { + let matrix = Expectations::parse(include_str!("../expectations.toml")).unwrap(); + let linux_lemonade = cap("strix-ubuntu"); + let hosts = [ + ( + "linux, no GPU, no gfx target", + HostCapability { + gfx_target: None, + has_amd_gpu: false, + amd_gpu_count: Some(0), + ..linux_lemonade.clone() + }, + ), + ( + "linux, gfx target reported but no usable GPU", + HostCapability { + has_amd_gpu: false, + amd_gpu_count: Some(0), + ..linux_lemonade.clone() + }, + ), + ( + "linux, usable GPU but no gfx target reported", + HostCapability { + gfx_target: None, + ..linux_lemonade.clone() + }, + ), + ]; + for id in [ + "bench-load-real-serve", + "chat-tool-definitions-accepted", + "chat-end-to-end-local-model", + ] { + let scenario = decl(&[&format!("id:{id}")]); + for (label, host) in &hosts { + assert_eq!( + resolve(&scenario, host, &matrix, Included::default()), + Expectation::ExpectPass, + "{id} on {label}", + ); + } + // The full real-GPU host is the one that must still be xfailed. + assert!( + matches!( + resolve(&scenario, &linux_lemonade, &matrix, Included::default()), + Expectation::ExpectXfail { bug, .. } if bug == "ROCMAI-898" + ), + "{id} on a real GPU host", + ); + } + } + fn decl(tags: &[&str]) -> ScenarioDecl { ScenarioDecl::from_tags(tags) } @@ -2260,8 +2356,8 @@ flaky = true /// serve-40 serves through the same lemonade managed serve as the other /// lemonade serve rows, so its row is scoped the same way: xfail on a /// lemonade Linux host, with a 90s serve wait, and not an expected failure - /// on native Windows. The row must stay `flaky` (a fixed #897 would - /// otherwise turn the Ubuntu lanes red as a stale XPASS) and must name #897. + /// on native Windows. The row must stay `flaky` (a fixed ROCMAI-898 would + /// otherwise turn the Ubuntu lanes red as a stale XPASS) and must name ROCMAI-898. #[test] fn serve_40_row_is_scoped_like_the_other_lemonade_serve_rows() { let m = Expectations::parse(include_str!("../expectations.toml")).unwrap(); @@ -2275,7 +2371,7 @@ flaky = true let entries = m.entries_for(id); assert_eq!(entries.len(), 1); assert!(entries[0].flaky); - assert_eq!(entries[0].bug, "#897"); + assert_eq!(entries[0].bug, "ROCMAI-898"); } /// The row's 90s only matters if the harness hands it to the serve steps: diff --git a/tests/e2e-cucumber/src/lemonade_diagnostics.rs b/tests/e2e-cucumber/src/lemonade_diagnostics.rs new file mode 100644 index 0000000000..5d4eed66ab --- /dev/null +++ b/tests/e2e-cucumber/src/lemonade_diagnostics.rs @@ -0,0 +1,2116 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Read-only backend evidence for a failed Lemonade serve. +//! +//! The evidence is taken in two steps with the service stop between them (see +//! [`backend_evidence_around_stop`]): what is only read from disk (manifest, +//! pins, candidate paths, fingerprints) is collected while the failed service is +//! still in the state being diagnosed, and the `llama-server` probes, which +//! execute a backend binary, run after it has been stopped. + +use sha2::{Digest as _, Sha256}; +use std::fmt::Write as _; +use std::io::Read as _; +use std::path::{Path, PathBuf}; +use std::process::Stdio; +use std::time::{Duration, Instant}; +use tokio::io::{AsyncRead, AsyncReadExt as _}; + +use crate::serve_log::{ENGINE_UNKNOWN_EVIDENCE, read_clamped_file}; + +const MAX_EVIDENCE_BYTES: u64 = 64 * 1024; +const PROBE_TIMEOUT: Duration = Duration::from_secs(5); +const MAX_CANDIDATES: usize = 8; +/// Time for the read-only inventory (pins, candidate paths, fingerprints), +/// taken before the service is stopped. +const INVENTORY_BUDGET: Duration = Duration::from_secs(20); +/// Time for the probes, started after the stop returns so a slow stop does not +/// use it up. +const PROBE_BUDGET: Duration = Duration::from_secs(20); +/// Largest executable that is fingerprinted. +/// +/// A fingerprint that is absent on the lane the report is for is a defect, so the +/// cap is set well above a dynamically linked `llama-server` (a few MiB; the HIP +/// kernels live in separate shared libraries). It is still finite: a build that +/// statically linked kernels for several GPU targets can reach hundreds of MiB, and +/// a corrupt cache entry should not be hashed at all. Hashing streams 64 KiB at a +/// time, so memory stays bounded whatever this is; the time is bounded by the +/// inventory budget, which the hashing loop checks between reads. +const MAX_HASH_BYTES: u64 = 512 * 1024 * 1024; + +#[derive(Clone, Copy)] +struct Budgets { + inventory: Duration, + probes: Duration, +} + +const DEFAULT_BUDGETS: Budgets = Budgets { + inventory: INVENTORY_BUDGET, + probes: PROBE_BUDGET, +}; + +/// Runs one `llama-server` flag and renders what came back. A trait so the order +/// of probes against the service stop is testable without executing a binary. +trait Probe: Sync { + fn run( + &self, + server: &Path, + flag: &str, + timeout: Duration, + ) -> impl std::future::Future + Send; +} + +struct RealProbe; + +impl Probe for RealProbe { + async fn run(&self, server: &Path, flag: &str, timeout: Duration) -> String { + probe(server, flag, timeout).await + } +} + +/// Collect the backend evidence for a failed serve of `engine` around the call +/// that stops the service, and return the stop's own status with it. +/// +/// `stop` always runs exactly once, whatever the engine, so a caller cannot +/// forget it or run it twice. For Lemonade the order is fixed: the read-only +/// inventory first, then `stop`, then the probes. A probe executes a backend +/// binary, and llama.cpp builds commonly initialise a GPU backend even for +/// `--version`, so running one next to a still-live router on a shared-memory +/// device could change the very state being diagnosed. Reading files cannot. +/// +/// `manifest_path` is only called for Lemonade, and nothing it does may keep the +/// stop from running: it returns `Err(reason)` when there is no manifest to look +/// for (the report then says `unavailable: `), and a panic inside it is +/// caught and reported the same way, so the stop still runs exactly once and the +/// failure report survives. The report is the one thing a failed serve has left. +/// +/// The report says `engine unknown` when no engine was determined, so a skipped +/// inventory stays visible, and returns no evidence for another engine. +pub async fn backend_evidence_around_stop( + manifest_path: impl FnOnce() -> Result + Send, + engine: Option<&str>, + stop: impl FnOnce() -> String + Send, +) -> (String, Option) { + match engine { + Some("lemonade") => { + let manifest = std::panic::catch_unwind(std::panic::AssertUnwindSafe(manifest_path)) + .unwrap_or_else(|_| Err("finding the runtime manifest failed".to_owned())); + match manifest { + Ok(manifest) => { + let (stop_status, evidence) = + evidence_around_stop(&manifest, DEFAULT_BUDGETS, &RealProbe, stop).await; + (stop_status, Some(evidence)) + } + Err(reason) => ( + stop(), + Some(format!("backend evidence unavailable: {reason}")), + ), + } + } + Some(_) => (stop(), None), + None => (stop(), Some(ENGINE_UNKNOWN_EVIDENCE.to_owned())), + } +} + +async fn evidence_around_stop( + manifest_path: &Path, + budgets: Budgets, + probe: &impl Probe, + stop: impl FnOnce() -> String + Send, +) -> (String, String) { + let inventory = take_inventory(manifest_path, budgets.inventory).await; + let stop_status = stop(); + let probes = inventory.probe_all(probe, budgets.probes).await; + (stop_status, format!("{}{probes}", inventory.report)) +} + +/// What was read from disk before the stop: the report so far, and the +/// candidates the probes will run against once the service is stopped. +struct Inventory { + report: String, + servers: Vec, +} + +impl Inventory { + const fn without_probes(report: String) -> Self { + Self { + report, + servers: Vec::new(), + } + } + + async fn probe_all(&self, probe: &impl Probe, budget: Duration) -> String { + if self.servers.is_empty() { + return String::new(); + } + let mut report = String::new(); + let _ = writeln!( + report, + "probes, run after the service stop was attempted; they inherit the harness \ + environment, not the engine's managed library environment, so a probe loader \ + failure does not establish a serving failure:" + ); + let deadline = Instant::now() + budget; + for server in &self.servers { + let _ = writeln!(report, "probe target: {}", server.display()); + for flag in ["--version", "--help"] { + let remaining = deadline.saturating_duration_since(Instant::now()); + if remaining.is_zero() { + let _ = writeln!( + report, + "remaining evidence skipped: evidence budget exhausted" + ); + return report; + } + let output = probe.run(server, flag, remaining.min(PROBE_TIMEOUT)).await; + let _ = writeln!(report, "{flag}:\n{output}"); + } + } + report + } +} + +/// What the report says about a file it did not read because it is a symlink. +const SYMLINK_NOT_FOLLOWED: &str = "symlink not followed"; + +/// Whether the last component of `path` is itself a symlink (not followed). Only +/// the non-unix [`open_no_follow`] needs it; unix refuses in the open itself. +#[cfg(not(unix))] +fn is_symlink(path: &Path) -> bool { + std::fs::symlink_metadata(path).is_ok_and(|meta| meta.file_type().is_symlink()) +} + +/// Whether `error` is the refusal [`open_no_follow`] makes for a symlink. +fn is_symlink_refusal(error: &std::io::Error) -> bool { + error.kind() == std::io::ErrorKind::Other && error.to_string() == SYMLINK_NOT_FOLLOWED +} + +/// Open `path` for reading, refusing it when its last component is a symlink. +/// +/// On unix the refusal is made by the open itself (`O_NOFOLLOW`), so there is no +/// window between a check and the open in which the path could be swapped for a +/// link: a link is refused whenever it is there at the moment of the open. On +/// Windows the check is a metadata read followed by an open, so a path swapped for +/// a link between the two is not caught; the evidence is read-only and the swap +/// needs write access to the backend directory, so that is accepted rather than +/// reached for with platform calls this harness does not otherwise make. Only +/// the last component is checked on either platform: a link in a parent directory +/// is followed. +fn open_no_follow(path: &Path) -> std::io::Result { + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + std::fs::OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW) + .open(path) + .map_err(|error| { + if error.raw_os_error() == Some(libc::ELOOP) { + std::io::Error::other(SYMLINK_NOT_FOLLOWED) + } else { + error + } + }) + } + #[cfg(not(unix))] + { + if is_symlink(path) { + return Err(std::io::Error::other(SYMLINK_NOT_FOLLOWED)); + } + std::fs::File::open(path) + } +} + +/// [`read_clamped`] on a file opened with [`open_no_follow`]: a link inside the +/// backend tree that points outside it must not put the outside file into the +/// report, the same rule the directory walk applies. +fn read_regular_clamped(path: &Path, max: u64) -> std::io::Result<(Vec, String)> { + let file = open_no_follow(path)?; + let len = file.metadata()?.len(); + read_clamped_file(file, len, max) +} + +/// Answers "has the inventory's time run out?". Injected so a test can drive the +/// traversal's deadline without sleeping; the real one compares against an +/// [`Instant`]. `Arc` because the scan runs on a blocking thread. +type Expired = std::sync::Arc bool + Send + Sync>; + +/// Record the installed version and pins separately from executable build output: +/// a pin describes what should be installed, not what a runner actually cached. +/// +/// Every candidate is labelled as such; finding a binary does not prove that the +/// router launched it. Nothing here executes anything. No install, download, +/// model load, or cache repair is done. +/// +/// The budget starts here, at entry, and covers the manifest and pin reads, the +/// directory search and the hashing alike. +async fn take_inventory(manifest_path: &Path, budget: Duration) -> Inventory { + let deadline = Instant::now() + budget; + take_inventory_with( + manifest_path, + deadline, + SEARCH_LIMITS, + std::sync::Arc::new(move || Instant::now() >= deadline), + ) + .await +} + +/// What [`scan_runtime`] read from disk. +enum Scan { + /// The manifest could not be used; the report says why. + Unavailable(String), + Found { + /// Manifest fields and pins. + report: String, + discovery: Discovery, + /// Where an empty search looked, when it found nothing and was not cut short + /// by the clock. + empty_note: Option, + }, +} + +/// Everything the inventory reads before it hashes anything: the manifest, the +/// pins and the directory search. Blocking filesystem work, so it runs on a +/// blocking thread; it carries its own clock because a `timeout` around +/// `spawn_blocking` would only drop the join handle and leave this running. +fn scan_runtime(manifest_path: &Path, limits: SearchLimits, expired: &dyn Fn() -> bool) -> Scan { + let mut report = String::new(); + let _ = writeln!(report, "manifest: {}", manifest_path.display()); + let manifest = + match read_regular_clamped(manifest_path, MAX_EVIDENCE_BYTES).and_then(|(bytes, _)| { + serde_json::from_slice::(&bytes).map_err(std::io::Error::other) + }) { + Ok(manifest) => manifest, + Err(error) => return Scan::Unavailable(format!("{report}unavailable: {error}")), + }; + for key in ["version", "backend_recipe", "backend_name", "runtime_dir"] { + let _ = writeln!( + report, + "{key}: {}", + manifest.get(key).unwrap_or(&serde_json::Value::Null) + ); + } + let Some(runtime) = manifest + .get("runtime_dir") + .and_then(serde_json::Value::as_str) + else { + return Scan::Unavailable(format!( + "{report}backend inventory unavailable: no runtime_dir" + )); + }; + let runtime = Path::new(runtime); + let pins = runtime.join("resources/backend_versions.json"); + let _ = writeln!(report, "configured pins (not executable identity):"); + match read_regular_clamped(&pins, MAX_EVIDENCE_BYTES) { + Ok((bytes, note)) => { + let _ = writeln!(report, "{}{note}", String::from_utf8_lossy(&bytes)); + } + Err(error) => { + let _ = writeln!(report, "{}: {error}", pins.display()); + } + } + let discovery = discover_candidates(runtime, limits, expired); + // "Nothing found" is only a finding when the search looked everywhere; one cut + // short by the clock or a limit reports that instead (see `take_inventory_with`). + // A candidate-cap stop has candidates by definition, so it never gets here. + let empty_note = (discovery.candidates.is_empty() && discovery.stop.is_none()) + .then(|| empty_search_note(runtime)); + Scan::Found { + report, + discovery, + empty_note, + } +} + +/// One candidate's fingerprint and `version.txt`, rendered. Blocking reads, run on +/// a blocking thread with the inventory's deadline checked between hash reads. +fn inspect_candidate(server: &Path, deadline: Instant) -> String { + let mut report = String::new(); + match fingerprint(server, deadline) { + Ok(hash) => { + let _ = writeln!(report, "sha256: {hash}"); + } + Err(error) => { + let _ = writeln!(report, "sha256 unavailable: {error}"); + } + } + if let Some(parent) = server.parent() { + let version = parent.join("version.txt"); + match read_regular_clamped(&version, MAX_EVIDENCE_BYTES) { + Ok((bytes, note)) => { + let _ = writeln!( + report, + "{}: {}{note}", + version.display(), + String::from_utf8_lossy(&bytes) + ); + } + Err(error) if is_symlink_refusal(&error) => { + let _ = writeln!(report, "{}: {SYMLINK_NOT_FOLLOWED}", version.display()); + } + Err(_) => {} + } + } + report +} + +async fn take_inventory_with( + manifest_path: &Path, + deadline: Instant, + limits: SearchLimits, + expired: Expired, +) -> Inventory { + let scan_path = manifest_path.to_owned(); + let scan_clock = expired.clone(); + let scan = + tokio::task::spawn_blocking(move || scan_runtime(&scan_path, limits, scan_clock.as_ref())) + .await; + let (mut report, discovery, empty_note) = match scan { + Ok(Scan::Found { + report, + discovery, + empty_note, + }) => (report, discovery, empty_note), + Ok(Scan::Unavailable(report)) => return Inventory::without_probes(report), + Err(error) => { + return Inventory::without_probes(format!( + "manifest: {}\nunavailable: inventory scan failed: {error}", + manifest_path.display() + )); + } + }; + if let Some(note) = empty_note { + let _ = writeln!(report, "{note}"); + } + match discovery.stop { + Some(Stop::Deadline) => { + let _ = writeln!( + report, + "remaining evidence skipped: evidence budget exhausted" + ); + } + Some(Stop::Directories) => { + let _ = writeln!( + report, + "candidate search stopped: {} directories visited (limit)", + discovery.dirs_visited + ); + } + Some(Stop::Entries) => { + let _ = writeln!( + report, + "candidate search stopped: {} directory entries read (limit)", + discovery.entries_read + ); + } + Some(Stop::Candidates) | None => {} + } + if discovery.symlinks_skipped > 0 { + let _ = writeln!( + report, + "{} symlinked entries under the backend directories were not followed", + discovery.symlinks_skipped + ); + } + let mut servers = Vec::new(); + // Said once: when the search itself ended on the clock the line is already in. + let budget_noted = matches!(discovery.stop, Some(Stop::Deadline)); + for server in &discovery.candidates { + if Instant::now() >= deadline { + if !budget_noted { + let _ = writeln!( + report, + "remaining evidence skipped: evidence budget exhausted" + ); + } + return Inventory { report, servers }; + } + let _ = writeln!( + report, + "candidate (not proof of selection): {}", + server.display() + ); + servers.push(server.clone()); + let path = server.clone(); + match tokio::task::spawn_blocking(move || inspect_candidate(&path, deadline)).await { + Ok(lines) => report.push_str(&lines), + Err(error) => { + let _ = writeln!(report, "sha256 unavailable: {error}"); + } + } + } + if matches!(discovery.stop, Some(Stop::Candidates)) { + let _ = writeln!( + report, + "backend inventory truncated after {MAX_CANDIDATES} candidates \ + (a further one was found and is not listed)" + ); + } + Inventory { report, servers } +} + +/// The GPU backend directories under `bin/llamacpp` that are searched. The cpu +/// backend is left out on purpose: a CPU build is never what a GPU serve should +/// have launched, and listing it would read as one. +const BACKEND_DIRS: [&str; 4] = ["rocm-stable", "rocm-nightly", "rocm", "vulkan"]; +/// How many directory levels below a backend directory are searched. Lemonade +/// unpacks some llama.cpp builds into a build-numbered directory, and the layout +/// is the router's to choose, so the search is not pinned to one level. +const MAX_SEARCH_DEPTH: usize = 3; +/// Names listed when explaining an empty search. +const MAX_LISTED_NAMES: usize = 16; +/// Directory entries read to build that listing. Bounded so a huge `bin/llamacpp` +/// cannot make the explanation of an empty search expensive; the report says how +/// many names it left out. +const MAX_LISTING_SCAN: usize = 64; + +/// Global bounds on one candidate search. They apply to the whole search, across +/// every backend directory, not per directory: a tree of many shallow directories +/// is as expensive as a deep one. +#[derive(Clone, Copy)] +struct SearchLimits { + /// Directories whose entries are read. + dirs: usize, + /// Directory entries read, in total. + entries: usize, + /// Candidates kept. The search stops as soon as it finds one more than this, + /// which is how it knows the list is cut; the extra one is dropped. + candidates: usize, +} + +const SEARCH_LIMITS: SearchLimits = SearchLimits { + dirs: 128, + entries: 2048, + candidates: MAX_CANDIDATES, +}; + +/// Why a candidate search ended before it had looked everywhere. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Stop { + Deadline, + Directories, + Entries, + Candidates, +} + +#[derive(Debug, Default)] +struct Discovery { + /// In a fixed order: backend directories as listed, then sibling directories + /// sorted by path. + candidates: Vec, + stop: Option, + /// Symlinks met and not followed. A cache that links out of its backend + /// directory, or back into it, is neither searched nor allowed to loop the + /// search. + symlinks_skipped: usize, + dirs_visited: usize, + entries_read: usize, +} + +const fn server_binary() -> &'static str { + if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + } +} + +/// Look for `llama-server` under each backend directory, within `limits` and until +/// `expired` says time is up. Symlinks are never followed: only real directories +/// are entered and only a regular file counts as a candidate, so a link cycle +/// ends and a link out of the tree is not read. +fn discover_candidates( + runtime: &Path, + limits: SearchLimits, + expired: &dyn Fn() -> bool, +) -> Discovery { + let mut search = Search { + limits, + expired, + out: Discovery::default(), + }; + for backend in BACKEND_DIRS { + let dir = runtime.join("bin/llamacpp").join(backend); + match std::fs::symlink_metadata(&dir) { + Ok(meta) if meta.is_dir() => search.visit(&dir, 0), + Ok(meta) if meta.file_type().is_symlink() => search.out.symlinks_skipped += 1, + _ => {} + } + if search.out.stop.is_some() { + break; + } + } + search.out.candidates.truncate(limits.candidates); + search.out +} + +struct Search<'a> { + limits: SearchLimits, + expired: &'a dyn Fn() -> bool, + out: Discovery, +} + +impl Search<'_> { + fn stop(&mut self, why: Stop) { + self.out.stop.get_or_insert(why); + } + + fn visit(&mut self, dir: &Path, depth: usize) { + if self.out.stop.is_some() { + return; + } + if (self.expired)() { + self.stop(Stop::Deadline); + return; + } + if self.out.dirs_visited >= self.limits.dirs { + self.stop(Stop::Directories); + return; + } + self.out.dirs_visited += 1; + let direct = dir.join(server_binary()); + // A symlink named llama-server is not a candidate (and is counted below, + // as a directory entry, not here). + if std::fs::symlink_metadata(&direct).is_ok_and(|meta| meta.is_file()) { + self.out.candidates.push(direct); + } + // One more than the cap: finding it is what shows the list is cut, so + // exactly `candidates` of them is a complete list and not a truncated one. + if self.out.candidates.len() > self.limits.candidates { + self.stop(Stop::Candidates); + return; + } + if depth >= MAX_SEARCH_DEPTH { + return; + } + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + let mut subdirs = Vec::new(); + for entry in entries { + if (self.expired)() { + self.stop(Stop::Deadline); + break; + } + if self.out.entries_read >= self.limits.entries { + self.stop(Stop::Entries); + break; + } + self.out.entries_read += 1; + let Ok(entry) = entry else { + continue; + }; + match entry.file_type() { + Ok(kind) if kind.is_dir() => subdirs.push(entry.path()), + Ok(kind) if kind.is_symlink() => self.out.symlinks_skipped += 1, + _ => {} + } + } + subdirs.sort(); + for subdir in subdirs { + self.visit(&subdir, depth + 1); + } + } +} + +/// Why a search found nothing, in terms of where it looked. An absent binary can +/// mean a backend download that did not finish, a layout the search does not +/// cover, or a build that was never installed, and the report cannot tell which, +/// so it says what was searched and what `bin/llamacpp` actually holds. +fn empty_search_note(runtime: &Path) -> String { + let llamacpp = runtime.join("bin/llamacpp"); + let searched = BACKEND_DIRS + .iter() + .map(|backend| llamacpp.join(backend).display().to_string()) + .collect::>() + .join(", "); + let present = match std::fs::read_dir(&llamacpp) { + Ok(entries) => { + let mut names: Vec = entries + .take(MAX_LISTING_SCAN) + .filter_map(Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .collect(); + let hit_scan_cap = names.len() >= MAX_LISTING_SCAN; + names.sort(); + if names.is_empty() { + format!("{} is empty", llamacpp.display()) + } else { + let omitted = names.len().saturating_sub(MAX_LISTED_NAMES); + names.truncate(MAX_LISTED_NAMES); + let more = match (omitted, hit_scan_cap) { + (0, false) => String::new(), + (_, false) => format!(" and {omitted} more"), + (_, true) => format!( + " and at least {omitted} more (only the first {MAX_LISTING_SCAN} \ + entries were read)" + ), + }; + format!( + "{} contains: {}{more}", + llamacpp.display(), + names.join(", ") + ) + } + } + Err(error) => format!("{} cannot be read: {error}", llamacpp.display()), + }; + format!( + "no llama-server candidates found; searched {searched}, down to {MAX_SEARCH_DEPTH} \ + directory levels below each (the cpu backend is not searched); {present}" + ) +} + +/// `limit` as the report words it: whole MiB when it is one, else bytes. +fn describe_hash_limit(limit: u64) -> String { + const MIB: u64 = 1024 * 1024; + if limit.is_multiple_of(MIB) { + format!("{} MiB", limit / MIB) + } else { + format!("{limit} bytes") + } +} + +fn fingerprint(server: &Path, deadline: Instant) -> std::io::Result { + fingerprint_with_limit(server, deadline, MAX_HASH_BYTES) +} + +fn fingerprint_with_limit(server: &Path, deadline: Instant, limit: u64) -> std::io::Result { + let too_large = || { + std::io::Error::other(format!( + "executable exceeds {} hash limit", + describe_hash_limit(limit) + )) + }; + let mut file = open_no_follow(server)?; + // Do not hash an arbitrarily large corrupt cache entry while reporting a + // failure. Never label a partial hash as SHA256. + if file.metadata()?.len() > limit { + return Err(too_large()); + } + let mut hasher = Sha256::new(); + let mut count = 0; + let mut buffer = vec![0; 64 * 1024]; + loop { + // Check inside the worker, not just around its JoinHandle: timing out a + // spawn_blocking await alone would leave the hashing thread running. + if Instant::now() >= deadline { + return Err(std::io::Error::new( + std::io::ErrorKind::TimedOut, + "hash evidence budget exhausted", + )); + } + let read = file.read(&mut buffer)?; + count += read as u64; + if count > limit { + return Err(too_large()); + } + if read == 0 { + break; + } + hasher.update(&buffer[..read]); + } + Ok(format!("{:x}", hasher.finalize())) +} + +#[derive(Default)] +struct CapturedOutput { + bytes: Vec, + truncated: bool, +} + +impl CapturedOutput { + fn truncation_note(&self, stream: &str) -> String { + if self.truncated { + format!("\n<{stream} truncated at 32 KiB>") + } else { + String::new() + } + } +} + +async fn probe(server: &Path, flag: &str, timeout: Duration) -> String { + let mut command = tokio::process::Command::new(server); + command.arg(flag); + run_probe(command, timeout).await +} + +/// Run `command` with bounded time and output, and render what came back. +/// +/// Split from [`probe`] so the timeout, kill and reap path can be driven with any +/// child, not only a `llama-server` stand-in: shell-script stand-ins exist on unix +/// only, and this path is what must also hold on Windows. +async fn run_probe(mut command: tokio::process::Command, timeout: Duration) -> String { + command + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .kill_on_drop(true); + let mut child = match command.spawn() { + Ok(child) => child, + Err(error) => return format!("probe could not start: {error}"), + }; + let stdout = child.stdout.take().expect("piped stdout"); + let stderr = child.stderr.take().expect("piped stderr"); + let mut out = CapturedOutput::default(); + let mut err = CapturedOutput::default(); + let status = if let Ok((status, stdout_result, stderr_result)) = + tokio::time::timeout(timeout, async { + tokio::join!( + child.wait(), + drain_output(stdout, &mut out), + drain_output(stderr, &mut err), + ) + }) + .await + { + let mut status = status.map_or_else(|error| error.to_string(), |status| status.to_string()); + for (stream, result) in [("stdout", stdout_result), ("stderr", stderr_result)] { + if let Err(error) = result { + let _ = write!(status, "\n{stream} capture failed: {error}"); + } + } + status + } else { + // kill_on_drop also covers cancellation of the whole reporter. + // Explicitly wait here so a normal timeout leaves no zombie child. + let mut status = format!("probe timed out after {}s", timeout.as_secs_f64()); + if let Err(error) = child.kill().await { + let _ = write!(status, "; kill/reap failed: {error}"); + } + status + }; + format!( + "{status}\nstdout:\n{}{}\nstderr:\n{}{}", + String::from_utf8_lossy(&out.bytes), + out.truncation_note("stdout"), + String::from_utf8_lossy(&err.bytes), + err.truncation_note("stderr"), + ) +} + +async fn drain_output( + mut reader: impl AsyncRead + Unpin, + retained: &mut CapturedOutput, +) -> std::io::Result<()> { + let limit = MAX_EVIDENCE_BYTES as usize / 2; + let mut buffer = [0; 4096]; + loop { + let count = reader.read(&mut buffer).await?; + if count == 0 { + return Ok(()); + } + // Continue draining both pipes after the cap, without retaining more + // bytes or writing them to disk. A noisy child must not block on output. + let keep = count.min(limit.saturating_sub(retained.bytes.len())); + retained.bytes.extend_from_slice(&buffer[..keep]); + retained.truncated |= keep < count; + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn candidates(runtime: &Path) -> Vec { + discover_candidates(runtime, SEARCH_LIMITS, &|| false).candidates + } + + fn touch_server(dir: &Path) -> PathBuf { + let server = dir.join(server_binary()); + std::fs::create_dir_all(dir).unwrap(); + std::fs::write(&server, "").unwrap(); + server + } + + async fn evidence(manifest: &Path) -> String { + evidence_with(manifest, DEFAULT_BUDGETS).await + } + + async fn evidence_with(manifest: &Path, budgets: Budgets) -> String { + evidence_around_stop(manifest, budgets, &RealProbe, String::new) + .await + .1 + } + + fn write_manifest(runtime: &Path) -> PathBuf { + let manifest = runtime.join("runtime.json"); + std::fs::write( + &manifest, + serde_json::json!({"runtime_dir": runtime}).to_string(), + ) + .unwrap(); + manifest + } + + /// The candidate paths a report lists, parsed back into paths. + /// + /// Compared as paths, never as printed text: the inventory joins + /// `bin/llamacpp` as one string, so on Windows a report line mixes `/` and + /// `\` while a test that builds the expected path another way prints it + /// differently. `Path` equality compares components, so both spellings agree, + /// and a negative check ("this candidate was skipped") cannot pass vacuously + /// just because the two spellings differ. + fn candidate_paths(report: &str) -> Vec { + report + .lines() + .filter_map(|line| line.strip_prefix("candidate (not proof of selection): ")) + .map(PathBuf::from) + .collect() + } + + #[test] + fn executable_identity_does_not_depend_on_the_dynamic_loader() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("not-executable"); + std::fs::write(&server, b"abc").unwrap(); + assert_eq!( + fingerprint(&server, Instant::now() + INVENTORY_BUDGET).unwrap(), + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" + ); + } + + #[test] + fn hash_limit_accepts_the_boundary_but_never_returns_a_partial_hash() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + let limit = 3 * 64 * 1024 + 5; + let file = std::fs::File::create(&server).unwrap(); + file.set_len(limit).unwrap(); + assert_eq!( + fingerprint_with_limit(&server, Instant::now() + INVENTORY_BUDGET, limit) + .unwrap() + .len(), + 64 + ); + file.set_len(limit + 1).unwrap(); + assert_eq!( + fingerprint_with_limit(&server, Instant::now() + INVENTORY_BUDGET, limit) + .unwrap_err() + .to_string(), + format!("executable exceeds {limit} bytes hash limit") + ); + } + + /// The cap is not the old 32 MiB: a server above it, such as a static HIP build, + /// must still get a fingerprint on the lane this report is for. The file is + /// sparse, so the test hashes zeros rather than holding real data. + #[test] + fn a_server_above_the_old_32_mib_cap_is_still_fingerprinted() { + const OLD_CAP: u64 = 32 * 1024 * 1024; + const { assert!(MAX_HASH_BYTES >= 256 * 1024 * 1024) }; + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + std::fs::File::create(&server) + .unwrap() + .set_len(OLD_CAP + 1) + .unwrap(); + assert_eq!( + fingerprint(&server, Instant::now() + INVENTORY_BUDGET) + .unwrap() + .len(), + 64 + ); + } + + #[test] + fn hash_limit_is_worded_in_the_unit_it_was_set_in() { + assert_eq!(describe_hash_limit(512 * 1024 * 1024), "512 MiB"); + assert_eq!(describe_hash_limit(1000), "1000 bytes"); + } + + #[test] + fn hash_worker_obeys_its_own_deadline() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + std::fs::write(&server, b"abc").unwrap(); + assert_eq!( + fingerprint(&server, Instant::now()).unwrap_err().kind(), + std::io::ErrorKind::TimedOut + ); + } + + #[test] + fn inventory_covers_flat_and_nested_gpu_backends_not_cpu() { + let root = tempfile::tempdir().unwrap(); + let binary = if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + }; + let mut wanted = Vec::new(); + for relative in ["rocm-stable", "rocm-nightly/llama-b9752", "vulkan/build"] { + let server = root.path().join("bin/llamacpp").join(relative).join(binary); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::write(&server, "").unwrap(); + wanted.push(server); + } + let cpu = root.path().join("bin/llamacpp/cpu").join(binary); + std::fs::create_dir_all(cpu.parent().unwrap()).unwrap(); + std::fs::write(cpu, "").unwrap(); + assert_eq!(candidates(root.path()), wanted); + } + + #[test] + fn candidate_search_reaches_three_levels_below_a_backend_and_no_further() { + let root = tempfile::tempdir().unwrap(); + let backend = root.path().join("bin/llamacpp/rocm-nightly"); + // Literal layouts, not derived from MAX_SEARCH_DEPTH: the depth is the + // behaviour under test, so changing the constant must turn this red. + let found_at_three = backend.join("a/b/llama-b9752").join(server_binary()); + let found_at_two = backend.join("a/build").join(server_binary()); + let beyond = backend.join("a/b/c/llama-b9752").join(server_binary()); + for server in [&found_at_three, &found_at_two, &beyond] { + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::write(server, "").unwrap(); + } + assert_eq!( + candidates(root.path()), + // Sorted by path, so `a/b/...` precedes `a/build/...`. + vec![found_at_three, found_at_two] + ); + } + + /// An empty search says where it looked and what `bin/llamacpp` holds, and does + /// not guess that a download was interrupted: the same report follows from a + /// layout the search does not cover or a build that was never installed. + #[tokio::test] + async fn empty_search_lists_the_searched_locations_and_blames_nothing() { + let root = tempfile::tempdir().unwrap(); + let cpu = root.path().join("bin/llamacpp/cpu"); + std::fs::create_dir_all(&cpu).unwrap(); + std::fs::write(cpu.join(server_binary()), "").unwrap(); + let report = evidence(&write_manifest(root.path())).await; + let llamacpp = root.path().join("bin/llamacpp"); + for backend in BACKEND_DIRS { + assert!( + report.contains(&llamacpp.join(backend).display().to_string()), + "{report}" + ); + } + assert!( + report.contains("no llama-server candidates found"), + "{report}" + ); + assert!( + report.contains(&format!("{} contains: cpu", llamacpp.display())), + "{report}" + ); + assert!( + !report.contains("download") && !report.contains("complete"), + "{report}" + ); + assert!(!report.contains("candidate (not proof"), "{report}"); + + let bare = tempfile::tempdir().unwrap(); + let report = evidence(&write_manifest(bare.path())).await; + assert!(report.contains("cannot be read"), "{report}"); + } + + fn limits(dirs: usize, entries: usize) -> SearchLimits { + SearchLimits { + dirs, + entries, + candidates: MAX_CANDIDATES, + } + } + + /// The clock is read while the tree is walked, not once before it: it reports + /// time up from its third reading on, so the walk must end partway through, + /// with what it had already found and the reason it stopped. + #[test] + fn the_clock_is_checked_during_the_walk() { + let root = tempfile::tempdir().unwrap(); + let wanted: Vec = BACKEND_DIRS + .iter() + .map(|backend| touch_server(&root.path().join("bin/llamacpp").join(backend))) + .collect(); + let calls = std::sync::atomic::AtomicUsize::new(0); + let found = discover_candidates(root.path(), SEARCH_LIMITS, &|| { + calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst) >= 2 + }); + assert_eq!(found.stop, Some(Stop::Deadline)); + assert_eq!(found.candidates, wanted[..1], "{found:?}"); + let all = discover_candidates(root.path(), SEARCH_LIMITS, &|| false); + assert_eq!(all.candidates, wanted); + assert_eq!(all.stop, None); + } + + /// The same, through the inventory: a clock that is already up stops the search + /// even though the hashing deadline is far off, so the walk consults the + /// inventory's clock and not only the hash loop's. + #[tokio::test] + async fn an_inventory_whose_time_is_up_skips_the_search() { + let root = tempfile::tempdir().unwrap(); + touch_server(&root.path().join("bin/llamacpp/rocm-stable")); + let inventory = take_inventory_with( + &write_manifest(root.path()), + Instant::now() + INVENTORY_BUDGET, + SEARCH_LIMITS, + std::sync::Arc::new(|| true), + ) + .await; + let report = &inventory.report; + assert!( + report.contains("remaining evidence skipped: evidence budget exhausted"), + "{report}" + ); + assert!(!report.contains("candidate (not proof"), "{report}"); + assert!( + !report.contains("no llama-server candidates found"), + "an unfinished search must not claim to have found nothing: {report}" + ); + assert!(inventory.servers.is_empty()); + } + + /// The scan is blocking filesystem work and must not run on the async executor. + /// On a current-thread runtime the test's own thread is the executor, so a clock + /// read from any other thread shows the walk ran on a blocking one. + #[tokio::test] + async fn the_directory_scan_runs_off_the_async_executor() { + let root = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(root.path().join("bin/llamacpp/rocm-stable/build")).unwrap(); + let threads = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let recorded = threads.clone(); + take_inventory_with( + &write_manifest(root.path()), + Instant::now() + INVENTORY_BUDGET, + SEARCH_LIMITS, + std::sync::Arc::new(move || { + recorded.lock().unwrap().push(std::thread::current().id()); + false + }), + ) + .await; + let threads = threads.lock().unwrap(); + assert!(!threads.is_empty(), "the walk never read the clock"); + let executor = std::thread::current().id(); + assert!( + threads.iter().all(|thread| *thread != executor), + "the scan ran on the async executor" + ); + } + + /// The limits cover the whole search: two backend directories with three + /// subdirectories each read six entries between them, so a limit of four stops + /// the second directory, which a per-directory limit of four would not. + #[test] + fn entry_and_directory_limits_apply_to_the_whole_search() { + let root = tempfile::tempdir().unwrap(); + for backend in ["rocm-stable", "rocm-nightly"] { + for build in ["a", "b", "c"] { + std::fs::create_dir_all(root.path().join("bin/llamacpp").join(backend).join(build)) + .unwrap(); + } + } + let found = discover_candidates(root.path(), limits(100, 4), &|| false); + assert_eq!(found.stop, Some(Stop::Entries), "{found:?}"); + assert_eq!(found.entries_read, 4, "{found:?}"); + + let found = discover_candidates(root.path(), limits(2, 100), &|| false); + assert_eq!(found.stop, Some(Stop::Directories), "{found:?}"); + assert_eq!(found.dirs_visited, 2, "{found:?}"); + + let found = discover_candidates(root.path(), SEARCH_LIMITS, &|| false); + assert_eq!( + found.stop, None, + "the default limits are not the small ones" + ); + } + + /// A search that did not finish must not say it found nothing. The tree is the + /// one a reviewer described: an empty `a`, and a real build under `b`, with the + /// cap reached before `b` is entered. Each way a search can stop early reports + /// its own reason and never the completed-empty-search text, which is asserted + /// absent; a search that did finish and found nothing still gets it. (The + /// candidate cap has no row: it stops a search that already has candidates.) + #[tokio::test] + async fn a_search_cut_short_says_why_and_never_claims_an_empty_result() { + const EMPTY_CLAIM: &str = "no llama-server candidates found"; + let root = tempfile::tempdir().unwrap(); + let backend = root.path().join("bin/llamacpp/rocm-stable"); + std::fs::create_dir_all(backend.join("a")).unwrap(); + touch_server(&backend.join("b")); + let manifest = write_manifest(root.path()); + let far = Instant::now() + INVENTORY_BUDGET; + let never: Expired = std::sync::Arc::new(|| false); + let always: Expired = std::sync::Arc::new(|| true); + + for (stop, search_limits, clock, stop_line) in [ + ( + Stop::Directories, + limits(2, 100), + never.clone(), + "candidate search stopped: 2 directories visited (limit)", + ), + ( + Stop::Entries, + limits(100, 1), + never.clone(), + "candidate search stopped: 1 directory entries read (limit)", + ), + ( + Stop::Deadline, + SEARCH_LIMITS, + always, + "remaining evidence skipped: evidence budget exhausted", + ), + ] { + let inventory = take_inventory_with(&manifest, far, search_limits, clock).await; + let report = &inventory.report; + assert!(report.contains(stop_line), "{stop:?}: {report}"); + assert!(!report.contains(EMPTY_CLAIM), "{stop:?}: {report}"); + assert!( + !report.contains("candidate (not proof"), + "{stop:?}: {report}" + ); + } + + let finished = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(finished.path().join("bin/llamacpp/rocm-stable/a")).unwrap(); + let inventory = + take_inventory_with(&write_manifest(finished.path()), far, SEARCH_LIMITS, never).await; + assert!( + inventory.report.contains(EMPTY_CLAIM), + "a search that finished and found nothing says so: {}", + inventory.report + ); + } + + /// The candidate cap ends the walk as soon as it is certain the list is cut, + /// instead of the walk finishing and the list being cut afterwards: with twenty + /// candidate directories only the root and the first nine (eight kept, one more + /// to show there are further ones) are entered, in sorted order. + #[test] + fn the_candidate_cap_stops_the_walk_when_it_is_reached() { + let root = tempfile::tempdir().unwrap(); + let backend = root.path().join("bin/llamacpp/rocm-stable"); + let wanted: Vec = (0..20) + .map(|i| touch_server(&backend.join(format!("{i:02}")))) + .collect(); + let found = discover_candidates(root.path(), SEARCH_LIMITS, &|| false); + assert_eq!(found.stop, Some(Stop::Candidates)); + assert_eq!(found.candidates, wanted[..MAX_CANDIDATES]); + assert_eq!(found.dirs_visited, 1 + MAX_CANDIDATES + 1, "{found:?}"); + } + + /// Exactly the cap is a complete list, not a truncated one: no stop, and the + /// report prints no truncation note. A ninth candidate makes it truncated, and + /// then the note is printed. Counting the cap itself as "truncated" turns the + /// first half red. + #[tokio::test] + async fn exactly_the_cap_is_not_reported_as_truncated() { + let root = tempfile::tempdir().unwrap(); + let backend = root.path().join("bin/llamacpp/rocm-stable"); + for i in 0..MAX_CANDIDATES { + touch_server(&backend.join(format!("{i:02}"))); + } + let found = discover_candidates(root.path(), SEARCH_LIMITS, &|| false); + assert_eq!(found.stop, None, "{found:?}"); + assert_eq!(found.candidates.len(), MAX_CANDIDATES); + let report = evidence(&write_manifest(root.path())).await; + assert!(!report.contains("truncated"), "{report}"); + assert_eq!( + report + .matches("candidate (not proof of selection):") + .count(), + MAX_CANDIDATES, + "{report}" + ); + + touch_server(&backend.join(format!("{MAX_CANDIDATES:02}"))); + let report = evidence(&write_manifest(root.path())).await; + assert!( + report.contains("backend inventory truncated after 8 candidates"), + "{report}" + ); + } + + /// When the search ends on the clock after finding candidates, and the hashing + /// deadline has passed too, "remaining evidence skipped" is said once, not once + /// by the search and again by the candidate loop. + #[tokio::test] + async fn the_budget_exhausted_line_is_printed_once() { + let root = tempfile::tempdir().unwrap(); + for backend in BACKEND_DIRS { + touch_server(&root.path().join("bin/llamacpp").join(backend)); + } + let calls = std::sync::atomic::AtomicUsize::new(0); + let calls = std::sync::Arc::new(calls); + let counter = calls.clone(); + let inventory = take_inventory_with( + &write_manifest(root.path()), + Instant::now() + .checked_sub(Duration::from_secs(1)) + .expect("a second before now"), + SEARCH_LIMITS, + std::sync::Arc::new(move || { + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst) >= 2 + }), + ) + .await; + let report = &inventory.report; + assert_eq!( + report + .matches("remaining evidence skipped: evidence budget exhausted") + .count(), + 1, + "{report}" + ); + assert!(!report.contains("candidate (not proof"), "{report}"); + } + + /// A symlink is neither entered nor read as a candidate. The tree holds a link + /// back to its own backend directory (a cycle), a link out to a directory with a + /// real `llama-server`, a `llama-server` that is a link to a file outside, and a + /// whole backend directory that is a link. The walk must end, find only the real + /// file, and count what it declined to follow. + #[cfg(unix)] + #[test] + fn symlinks_are_not_followed_and_a_cycle_ends() { + use std::os::unix::fs::symlink; + let root = tempfile::tempdir().unwrap(); + let outside = tempfile::tempdir().unwrap(); + let outside_server = touch_server(outside.path()); + let backends = root.path().join("bin/llamacpp"); + let real = touch_server(&backends.join("rocm-stable/real")); + symlink( + backends.join("rocm-stable"), + backends.join("rocm-stable/loop"), + ) + .unwrap(); + symlink(outside.path(), backends.join("rocm-stable/out")).unwrap(); + symlink( + &outside_server, + backends.join("rocm-stable").join(server_binary()), + ) + .unwrap(); + symlink(outside.path(), backends.join("rocm-nightly")).unwrap(); + // A version marker beside the real server that links out of the tree. + std::fs::write(outside.path().join("version.txt"), "OUTSIDE-VERSION").unwrap(); + symlink( + outside.path().join("version.txt"), + backends.join("rocm-stable/real/version.txt"), + ) + .unwrap(); + + let found = discover_candidates(root.path(), SEARCH_LIMITS, &|| false); + assert_eq!(found.candidates, vec![real], "{found:?}"); + assert_eq!(found.stop, None, "{found:?}"); + assert_eq!(found.symlinks_skipped, 5, "{found:?}"); + assert_eq!(found.dirs_visited, 2, "{found:?}"); + } + + /// The refusal is made by the open itself, not by a check before it, so a path + /// that becomes a link after any check is still refused. The helper is called + /// directly on a link, which is what a swap after the metadata check looks like + /// to it; removing `O_NOFOLLOW` makes the open succeed and turns this red. A + /// regular file still opens, and a missing one reports its own error, not a + /// refusal. (Unix only: on Windows the check and the open are separate steps.) + #[cfg(unix)] + #[test] + fn the_open_itself_refuses_a_symlink() { + use std::io::Read as _; + use std::os::unix::fs::symlink; + let dir = tempfile::tempdir().unwrap(); + let real = dir.path().join("real"); + std::fs::write(&real, "OUTSIDE").unwrap(); + let link = dir.path().join("link"); + symlink(&real, &link).unwrap(); + + let error = open_no_follow(&link).unwrap_err(); + assert!(is_symlink_refusal(&error), "{error}"); + assert_eq!(error.to_string(), SYMLINK_NOT_FOLLOWED); + + let mut text = String::new(); + open_no_follow(&real) + .unwrap() + .read_to_string(&mut text) + .unwrap(); + assert_eq!(text, "OUTSIDE"); + let missing = open_no_follow(&dir.path().join("missing")).unwrap_err(); + assert!(!is_symlink_refusal(&missing), "{missing}"); + assert_eq!(missing.kind(), std::io::ErrorKind::NotFound); + } + + /// Every file the report reads from the backend tree is refused when it is a + /// symlink, and the report says so, the way the walk reports a link it did not + /// enter. A real `llama-server` sits beside a `version.txt` that links to a file + /// outside the tree, the pins file is a link out too, and so is a manifest: none + /// of the outside content may appear, and each refusal is named. A fingerprint + /// of a path that has become a link is refused as well. + #[cfg(unix)] + #[tokio::test] + async fn linked_files_are_refused_and_named_not_embedded() { + use std::os::unix::fs::symlink; + let root = tempfile::tempdir().unwrap(); + let outside = tempfile::tempdir().unwrap(); + for (name, text) in [ + ("version.txt", "OUTSIDE-VERSION"), + ("pins.json", "OUTSIDE-PINS"), + ("manifest.json", r#"{"version": "OUTSIDE-MANIFEST"}"#), + ("binary", "OUTSIDE-BINARY"), + ] { + std::fs::write(outside.path().join(name), text).unwrap(); + } + let server = touch_server(&root.path().join("bin/llamacpp/rocm-stable")); + std::fs::write(&server, "abc").unwrap(); + symlink( + outside.path().join("version.txt"), + server.parent().unwrap().join("version.txt"), + ) + .unwrap(); + std::fs::create_dir_all(root.path().join("resources")).unwrap(); + symlink( + outside.path().join("pins.json"), + root.path().join("resources/backend_versions.json"), + ) + .unwrap(); + + let inventory = take_inventory_with( + &write_manifest(root.path()), + Instant::now() + INVENTORY_BUDGET, + SEARCH_LIMITS, + std::sync::Arc::new(|| false), + ) + .await; + let report = &inventory.report; + assert!(!report.contains("OUTSIDE"), "{report}"); + assert!( + report.contains(&format!( + "{}: {SYMLINK_NOT_FOLLOWED}", + server.parent().unwrap().join("version.txt").display() + )), + "{report}" + ); + assert!( + report.contains(&format!( + "{}: {SYMLINK_NOT_FOLLOWED}", + root.path() + .join("resources/backend_versions.json") + .display() + )), + "{report}" + ); + // The real server beside the links is still fingerprinted, and the walk + // counts the one link it saw among the entries. + assert!( + report.contains(&format!("sha256: {ABC_SHA256}")), + "{report}" + ); + assert!( + report.contains("1 symlinked entries under the backend directories were not followed"), + "{report}" + ); + + let linked_manifest = root.path().join("linked-manifest.json"); + symlink(outside.path().join("manifest.json"), &linked_manifest).unwrap(); + let inventory = take_inventory_with( + &linked_manifest, + Instant::now() + INVENTORY_BUDGET, + SEARCH_LIMITS, + std::sync::Arc::new(|| false), + ) + .await; + assert!( + !inventory.report.contains("OUTSIDE"), + "{}", + inventory.report + ); + assert!( + inventory + .report + .contains(&format!("unavailable: {SYMLINK_NOT_FOLLOWED}")), + "{}", + inventory.report + ); + + let linked_binary = root.path().join("linked-binary"); + symlink(outside.path().join("binary"), &linked_binary).unwrap(); + assert_eq!( + fingerprint(&linked_binary, Instant::now() + INVENTORY_BUDGET) + .unwrap_err() + .to_string(), + SYMLINK_NOT_FOLLOWED + ); + } + + #[tokio::test] + async fn the_empty_search_listing_is_bounded_and_counts_what_it_omits() { + let root = tempfile::tempdir().unwrap(); + let llamacpp = root.path().join("bin/llamacpp"); + for i in 0..(MAX_LISTED_NAMES + 4) { + std::fs::create_dir_all(llamacpp.join(format!("other-{i:03}"))).unwrap(); + } + let report = evidence(&write_manifest(root.path())).await; + assert!(report.contains(" and 4 more"), "{report}"); + assert!(!report.contains("at least"), "{report}"); + + for i in 0..(2 * MAX_LISTING_SCAN) { + std::fs::create_dir_all(llamacpp.join(format!("other-{i:03}"))).unwrap(); + } + let report = evidence(&write_manifest(root.path())).await; + let listing = report + .split("contains: ") + .nth(1) + .and_then(|rest| rest.lines().next()) + .unwrap(); + let (names, omitted) = listing.split_once(" and at least ").unwrap(); + assert_eq!(names.split(", ").count(), MAX_LISTED_NAMES, "{report}"); + assert!( + omitted.starts_with(&format!("{} more", MAX_LISTING_SCAN - MAX_LISTED_NAMES)), + "{report}" + ); + assert!( + omitted.contains(&format!("first {MAX_LISTING_SCAN} entries")), + "{report}" + ); + } + + #[tokio::test] + async fn evidence_truncates_candidates_without_hashing_or_probing_the_rest() { + let root = tempfile::tempdir().unwrap(); + let binary = if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + }; + for i in 0..=MAX_CANDIDATES { + let server = root + .path() + .join(format!("bin/llamacpp/rocm-stable/{i:02}/{binary}")); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::write(server, "").unwrap(); + } + let report = evidence(&write_manifest(root.path())).await; + assert_eq!( + report + .matches("candidate (not proof of selection):") + .count(), + MAX_CANDIDATES, + "{report}" + ); + assert_eq!(report.matches("sha256:").count(), MAX_CANDIDATES); + assert_eq!(report.matches("--help:").count(), MAX_CANDIDATES); + assert!(report.contains("backend inventory truncated after 8 candidates")); + let skipped = root + .path() + .join(format!("bin/llamacpp/rocm-stable/08/{binary}")); + let listed = candidate_paths(&report); + assert_eq!(listed.len(), MAX_CANDIDATES, "{report}"); + assert!(!listed.contains(&skipped), "{report}"); + let first = root + .path() + .join(format!("bin/llamacpp/rocm-stable/00/{binary}")); + assert!( + listed.contains(&first), + "the comparison must be able to match a listed candidate: {report}" + ); + } + + /// An executable past the hash cap is not fingerprinted, but the report says + /// so instead of omitting the line, and still lists the candidate and probes it. + #[tokio::test] + async fn oversized_executable_reports_why_it_has_no_fingerprint() { + let root = tempfile::tempdir().unwrap(); + let binary = if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + }; + let server = root.path().join("bin/llamacpp/rocm-stable").join(binary); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::File::create(&server) + .unwrap() + .set_len(MAX_HASH_BYTES + 1) + .unwrap(); + let report = evidence(&write_manifest(root.path())).await; + assert!( + report.contains(&format!( + "sha256 unavailable: executable exceeds {} hash limit", + describe_hash_limit(MAX_HASH_BYTES) + )), + "{report}" + ); + assert!(!report.contains("sha256: "), "{report}"); + assert_eq!(candidate_paths(&report), vec![server], "{report}"); + assert!(report.contains("--version:"), "{report}"); + } + + #[tokio::test] + async fn exhausted_budget_skips_hashing_and_probes() { + let root = tempfile::tempdir().unwrap(); + let binary = if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + }; + let server = root.path().join("bin/llamacpp/rocm-stable").join(binary); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::write(server, "").unwrap(); + let report = evidence_with( + &write_manifest(root.path()), + Budgets { + inventory: Duration::ZERO, + probes: PROBE_BUDGET, + }, + ) + .await; + assert!(report.contains("remaining evidence skipped: evidence budget exhausted")); + for skipped in ["sha256", "--version:", "--help:", "candidate ("] { + assert!(!report.contains(skipped), "{report}"); + } + } + + type Events = std::sync::Mutex>; + + struct RecordingProbe<'a> { + events: &'a Events, + } + + impl Probe for RecordingProbe<'_> { + async fn run(&self, _server: &Path, flag: &str, _timeout: Duration) -> String { + self.events.lock().unwrap().push(format!("probe {flag}")); + "recorded, not executed".to_owned() + } + } + + /// A lemonade data dir whose manifest sits where the engine writes it, with one + /// small candidate whose SHA-256 is a known value. + fn lemonade_data_dir_with_candidate() -> (tempfile::TempDir, PathBuf, PathBuf) { + let root = tempfile::tempdir().unwrap(); + let paths = rocm_core::AppPaths { + config_dir: root.path().join("config"), + data_dir: root.path().join("data"), + cache_dir: root.path().join("cache"), + }; + let runtime = root.path().join("runtime"); + let binary = if cfg!(windows) { + "llama-server.exe" + } else { + "llama-server" + }; + let server = runtime.join("bin/llamacpp/rocm-stable").join(binary); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + std::fs::write(&server, b"abc").unwrap(); + let manifest = paths.engine_install_manifest_path("lemonade"); + std::fs::create_dir_all(manifest.parent().unwrap()).unwrap(); + std::fs::write( + &manifest, + serde_json::json!({"runtime_dir": runtime}).to_string(), + ) + .unwrap(); + (root, manifest, server) + } + + const ABC_SHA256: &str = "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"; + + /// The inventory is read while the failed service is still in place and the + /// probes, which execute a backend binary, run only after the stop. + /// + /// The stop deletes the candidate. A fingerprint in the report therefore proves + /// the file was read before the stop, and the recorded events prove the probes + /// came after it; running either step on the wrong side turns this red. + #[tokio::test] + async fn inventory_precedes_the_stop_and_probes_follow_it() { + let (_root, manifest, server) = lemonade_data_dir_with_candidate(); + let events = Events::default(); + let (stop_status, report) = evidence_around_stop( + &manifest, + DEFAULT_BUDGETS, + &RecordingProbe { events: &events }, + || { + events.lock().unwrap().push("stop".to_owned()); + std::fs::remove_file(&server).unwrap(); + "stopped".to_owned() + }, + ) + .await; + assert_eq!(stop_status, "stopped"); + assert_eq!( + *events.lock().unwrap(), + ["stop", "probe --version", "probe --help"], + "{report}" + ); + assert!( + report.contains(&format!("sha256: {ABC_SHA256}")), + "{report}" + ); + assert!(!report.contains("sha256 unavailable"), "{report}"); + let inventory = report.find("candidate (not proof of selection):").unwrap(); + let probes = report.find("probe target:").unwrap(); + assert!(inventory < probes, "{report}"); + assert!(report.contains("recorded, not executed"), "{report}"); + } + + #[tokio::test] + async fn stop_runs_once_for_every_engine_and_an_unknown_engine_stays_visible() { + let stops = std::sync::atomic::AtomicUsize::new(0); + let stop = || { + stops.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + "stopped".to_owned() + }; + let no_manifest = + || -> Result { panic!("the manifest is only read for lemonade") }; + + assert_eq!( + backend_evidence_around_stop(no_manifest, Some("vllm"), stop).await, + ("stopped".to_owned(), None) + ); + assert_eq!( + backend_evidence_around_stop(no_manifest, None, stop).await, + ( + "stopped".to_owned(), + Some(ENGINE_UNKNOWN_EVIDENCE.to_owned()) + ) + ); + assert_eq!(stops.load(std::sync::atomic::Ordering::SeqCst), 2); + + let (_root, manifest, _server) = lemonade_data_dir_with_candidate(); + let (status, evidence) = + backend_evidence_around_stop(|| Ok(manifest), Some("lemonade"), stop).await; + assert_eq!(status, "stopped"); + assert!(evidence.unwrap().contains("manifest: ")); + assert_eq!(stops.load(std::sync::atomic::Ordering::SeqCst), 3); + } + + /// A Lemonade serve with nothing to look the manifest up in (the scenario has no + /// isolated root) used to panic before the stop, so the stop never ran and the + /// whole failure report was lost. The lookup now fails soft, whether it says so + /// (`Err`) or panics, and the stop still runs exactly once. Removing the panic + /// guard turns the second case red. + #[tokio::test] + async fn a_failing_manifest_lookup_still_runs_the_stop_once_and_reports_why() { + let stops = std::sync::atomic::AtomicUsize::new(0); + let stop = || { + stops.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + "stopped".to_owned() + }; + let (status, evidence) = backend_evidence_around_stop( + || Err("the scenario has no isolated root".to_owned()), + Some("lemonade"), + stop, + ) + .await; + assert_eq!(status, "stopped"); + assert_eq!( + evidence.as_deref(), + Some("backend evidence unavailable: the scenario has no isolated root") + ); + assert_eq!(stops.load(std::sync::atomic::Ordering::SeqCst), 1); + + let (status, evidence) = backend_evidence_around_stop( + || -> Result { panic!("no isolated root") }, + Some("lemonade"), + stop, + ) + .await; + assert_eq!(status, "stopped"); + assert_eq!( + evidence.as_deref(), + Some("backend evidence unavailable: finding the runtime manifest failed") + ); + assert_eq!(stops.load(std::sync::atomic::Ordering::SeqCst), 2); + } + + #[tokio::test] + async fn an_unreadable_manifest_is_reported_and_the_service_is_still_stopped() { + let root = tempfile::tempdir().unwrap(); + let events = Events::default(); + let (stop_status, report) = evidence_around_stop( + &root.path().join("missing.json"), + DEFAULT_BUDGETS, + &RecordingProbe { events: &events }, + || "stopped".to_owned(), + ) + .await; + assert_eq!(stop_status, "stopped"); + assert!(report.contains("unavailable:"), "{report}"); + assert!(events.lock().unwrap().is_empty(), "no candidate, no probe"); + assert!(!report.contains("probe target:"), "{report}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn timed_out_probe_leaves_no_budget_for_the_next_probe() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("bin/llamacpp/rocm-stable/llama-server"); + std::fs::create_dir_all(server.parent().unwrap()).unwrap(); + rocm_core::fake_executable::write_fake_executable( + &server, + "#!/bin/sh\nexec /bin/sleep 30\n", + ) + .unwrap(); + let report = evidence_with( + &write_manifest(root.path()), + Budgets { + inventory: INVENTORY_BUDGET, + probes: Duration::from_secs(1), + }, + ) + .await; + assert!(report.contains("--version:\nprobe timed out"), "{report}"); + assert!(report.contains("remaining evidence skipped: evidence budget exhausted")); + assert!(!report.contains("--help:"), "{report}"); + } + + #[tokio::test] + async fn stream_capture_marks_only_bytes_actually_discarded() { + let limit = MAX_EVIDENCE_BYTES as usize / 2; + for length in [0, limit - 1, limit, limit + 1] { + let bytes = vec![b'x'; length]; + let mut captured = CapturedOutput::default(); + drain_output(bytes.as_slice(), &mut captured).await.unwrap(); + assert_eq!(captured.bytes, vec![b'x'; length.min(limit)]); + assert_eq!(captured.truncated, length > limit); + assert_eq!( + captured.truncation_note("stdout"), + if length > limit { + "\n" + } else { + "" + } + ); + } + } + + #[tokio::test] + async fn stream_truncation_survives_timeout_before_eof() { + use tokio::io::AsyncWriteExt as _; + let length = MAX_EVIDENCE_BYTES as usize / 2 + 1; + let (reader, mut writer) = tokio::io::duplex(length); + writer.write_all(&vec![b'x'; length]).await.unwrap(); + let mut captured = CapturedOutput::default(); + assert!( + tokio::time::timeout( + Duration::from_millis(50), + drain_output(reader, &mut captured) + ) + .await + .is_err() + ); + assert_eq!(captured.bytes.len(), length - 1); + assert_eq!( + captured.truncation_note("stderr"), + "\n" + ); + } + + #[tokio::test] + async fn missing_manifest_is_reported_without_panicking() { + let root = tempfile::tempdir().unwrap(); + let report = evidence(&root.path().join("missing.json")).await; + assert!(report.contains("unavailable:"), "{report}"); + } + + #[tokio::test] + async fn failed_download_keeps_pins_but_does_not_invent_an_installed_build() { + let root = tempfile::tempdir().unwrap(); + let runtime = root.path().join("runtime"); + std::fs::create_dir_all(runtime.join("resources")).unwrap(); + std::fs::write( + runtime.join("resources/backend_versions.json"), + r#"{"llamacpp":{"rocm-stable":"b9752"}}"#, + ) + .unwrap(); + let manifest = root.path().join("runtime.json"); + std::fs::write( + &manifest, + serde_json::json!({ + "version": "fixture", + "runtime_dir": runtime, + "backend_name": "rocm" + }) + .to_string(), + ) + .unwrap(); + let report = evidence(&manifest).await; + assert!( + report.contains("configured pins (not executable identity):"), + "{report}" + ); + assert!(report.contains("\"rocm-stable\":\"b9752\""), "{report}"); + assert!(report.contains("no llama-server candidates"), "{report}"); + } + + #[tokio::test] + async fn probe_start_error_is_evidence_not_a_second_failure() { + let root = tempfile::tempdir().unwrap(); + let report = probe(&root.path().join("missing"), "--version", PROBE_TIMEOUT).await; + assert!(report.contains("probe could not start:"), "{report}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn inventory_probes_the_cached_executable_not_a_path_lookup() { + let root = tempfile::tempdir().unwrap(); + let runtime = root.path().join("runtime"); + let dir = runtime.join("bin/llamacpp/rocm-stable"); + std::fs::create_dir_all(&dir).unwrap(); + let server = dir.join("llama-server"); + rocm_core::fake_executable::write_fake_executable( + &server, + "\ +#!/bin/sh +case \"$1\" in + --version) echo 'version: 9752 (fixture-build)' ;; + --help) echo '--mmap, --no-mmap'; echo 'fixture help stderr' >&2; exit 2 ;; + *) exit 99 ;; +esac +", + ) + .unwrap(); + std::fs::write(dir.join("version.txt"), "cached-build").unwrap(); + let manifest = root.path().join("runtime.json"); + std::fs::write( + &manifest, + serde_json::json!({ + "version": "fixture", + "runtime_dir": runtime, + "backend_name": "rocm" + }) + .to_string(), + ) + .unwrap(); + let report = evidence(&manifest).await; + assert!(report.contains(&server.display().to_string()), "{report}"); + assert!(report.contains("version: 9752 (fixture-build)"), "{report}"); + assert!(report.contains("version.txt: cached-build"), "{report}"); + assert!( + report.contains("--help:\nexit status: 2\nstdout:"), + "{report}" + ); + assert!(!report.contains("capture"), "{report}"); + assert!(!report.contains("Ok("), "{report}"); + assert!(report.contains("--mmap, --no-mmap"), "{report}"); + assert!(report.contains("fixture help stderr"), "{report}"); + assert!( + server.is_file(), + "diagnostics must not repair or remove the cache" + ); + } + + /// Selects what [`probe_child`] does when a probe test re-runs this test binary + /// as its child process. + const PROBE_CHILD_ENV: &str = "ROCM_E2E_PROBE_CHILD"; + + /// Not a test: the child process the probe tests below start. + /// + /// The probe's timeout, kill and reap path has to hold on Windows too, and a + /// shell-script stand-in does not exist there. This test binary does, on every + /// platform, so the tests re-run it with `--ignored --exact` and this function + /// plays the stand-in. It is `#[ignore]`d so a normal run skips it, and does + /// nothing unless [`PROBE_CHILD_ENV`] selects a behaviour. + #[test] + #[ignore = "child process of the probe tests, not a test of its own"] + fn probe_child() { + match std::env::var(PROBE_CHILD_ENV).as_deref() { + Ok("sleep") => { + // With one test thread (RUST_TEST_THREADS=1, or one CPU) libtest + // writes "test ... " first and our output continues that + // line, so start a fresh one: the parent reads the first line that + // begins with the marker. + println!("\npid:{}", std::process::id()); + std::thread::sleep(Duration::from_secs(CHILD_LIFETIME_SECS)); + } + Ok("exit") => println!("child-ran"), + _ => {} + } + } + + fn probe_child_command(behaviour: &str) -> tokio::process::Command { + let module = module_path!().split_once("::").unwrap().1; + let mut command = tokio::process::Command::new(std::env::current_exe().unwrap()); + command + .args([ + "--exact", + &format!("{module}::probe_child"), + "--ignored", + "--nocapture", + ]) + .env(PROBE_CHILD_ENV, behaviour); + command + } + + /// How long each attempt at the timeout test lets the helper child run before the + /// probe's timeout fires. The first is short, so a healthy run is quick; the + /// later ones are generous and matter only when the child was too slow to start + /// and print its pid inside the earlier one (a loaded runner, a cold test + /// binary), which is a slow start and not a failure of the probe. + const CHILD_START_ATTEMPTS: [u64; 3] = [5, 20, 60]; + + /// How much longer than the longest attempt the helper child stays alive. The + /// child must still be running when the probe's timeout fires on any attempt: + /// one that had exited by itself would be a normal exit, not a timeout, and the + /// test would fail for a correct probe. + const CHILD_LIFETIME_MARGIN_SECS: u64 = 60; + + /// The helper child's lifetime, derived from the attempt timeouts so the two + /// cannot drift apart. + const CHILD_LIFETIME_SECS: u64 = { + let mut longest = 0; + let mut at = 0; + while at < CHILD_START_ATTEMPTS.len() { + if CHILD_START_ATTEMPTS[at] > longest { + longest = CHILD_START_ATTEMPTS[at]; + } + at += 1; + } + longest + CHILD_LIFETIME_MARGIN_SECS + }; + + /// The helper child outlives every attempt's timeout by at least half a minute + /// (the margin is asserted as a literal so shrinking the constant fails), so the + /// timeout, kill and reap path is what ends it on each retry. Giving the child a + /// literal lifetime that a longer attempt can overtake (it was 30 seconds against + /// a 60-second attempt) turns this red. + #[test] + fn the_helper_child_outlives_every_attempt_deadline() { + for attempt in CHILD_START_ATTEMPTS { + assert!( + CHILD_LIFETIME_SECS >= attempt + 30, + "the child lives {CHILD_LIFETIME_SECS}s but an attempt waits {attempt}s" + ); + } + } + + /// One run of the timeout path against a helper child that libtest runs with + /// `test_threads` test threads: the child is still running when the timeout + /// fires, so the report must say so, return promptly, and leave no process + /// behind. `process_is_running` is false only once the child has been killed + /// and reaped, so a probe that merely abandoned it turns this red. + /// + /// The pid is read back from the report, so an attempt whose child had not + /// started before the timeout (no pid line) is retried with a longer timeout + /// rather than failed; only an attempt that did report a pid is asserted on. + async fn assert_timeout_kills_and_reaps(test_threads: &str) { + let mut last = String::new(); + for seconds in CHILD_START_ATTEMPTS { + let start = Instant::now(); + let mut command = probe_child_command("sleep"); + command.env("RUST_TEST_THREADS", test_threads); + let report = run_probe(command, Duration::from_secs(seconds)).await; + assert!( + report.starts_with(&format!("probe timed out after {seconds}s")), + "{report}" + ); + assert!(!report.contains("kill/reap failed"), "{report}"); + assert!( + start.elapsed() < Duration::from_secs(seconds + 15), + "{report}" + ); + let Some(pid) = report + .lines() + .find_map(|line| line.strip_prefix("pid:")) + .map(|pid| pid.parse::().unwrap()) + else { + last = report; + continue; + }; + assert!(!rocm_core::process_is_running(pid), "{report}"); + return; + } + panic!( + "the child never reported its pid, even with {CHILD_START_ATTEMPTS:?} seconds \ + (test threads {test_threads}): {last}" + ); + } + + /// The timeout path on every platform, with the child run both on one test + /// thread and on several. With one thread (also what a one-CPU host gets) libtest + /// prefixes the child's output with its test-name label, so a pid printed on + /// that same line is not found by a line-start match. The two runs overlap, so + /// this takes one timeout, not two. + #[tokio::test] + async fn a_probe_that_outlives_its_timeout_is_killed_and_reaped() { + tokio::join!( + assert_timeout_kills_and_reaps("1"), + assert_timeout_kills_and_reaps("8"), + ); + } + + /// The normal path on every platform: status and output come back, and the + /// timeout branch is not taken. + #[tokio::test] + async fn a_probe_that_exits_reports_its_status_and_output() { + let report = run_probe(probe_child_command("exit"), Duration::from_mins(1)).await; + let status = if cfg!(windows) { + "exit code: 0\n" + } else { + "exit status: 0\n" + }; + assert!(report.starts_with(status), "{report}"); + assert!(report.contains("child-ran"), "{report}"); + assert!(!report.contains("timed out"), "{report}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn probe_timeout_is_bounded() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + // exec keeps the sleeper in the child's PID, so cancellation kills it + // rather than leaving a grandchild behind. + rocm_core::fake_executable::write_fake_executable( + &server, + "#!/bin/sh\necho $$\nexec /bin/sleep 30\n", + ) + .unwrap(); + let start = std::time::Instant::now(); + let report = probe(&server, "--help", Duration::from_secs(1)).await; + assert!( + report.contains("probe timed out after 1s\nstdout:"), + "{report}" + ); + assert!(start.elapsed() < Duration::from_secs(5)); + let pid = report + .split("stdout:\n") + .nth(1) + .unwrap() + .lines() + .next() + .unwrap() + .parse::() + .unwrap(); + assert!(!rocm_core::process_is_running(pid), "{report}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn exact_stream_limit_is_not_reported_as_truncated() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + rocm_core::fake_executable::write_fake_executable( + &server, + "#!/bin/sh\nhead -c 32768 /dev/zero\n", + ) + .unwrap(); + let report = probe(&server, "--help", PROBE_TIMEOUT).await; + assert!(report.contains("exit status: 0"), "{report}"); + assert_eq!(report.bytes().filter(|byte| *byte == 0).count(), 32768); + assert!(!report.contains("capped"), "{report}"); + assert!(!report.contains("truncated"), "{report}"); + } + + #[cfg(unix)] + #[tokio::test] + async fn noisy_probe_output_is_capped() { + let root = tempfile::tempdir().unwrap(); + let server = root.path().join("llama-server"); + rocm_core::fake_executable::write_fake_executable( + &server, + "#!/bin/sh\nhead -c 100000 /dev/zero\nhead -c 100000 /dev/zero >&2\n", + ) + .unwrap(); + let report = probe(&server, "--help", PROBE_TIMEOUT).await; + assert!(report.contains(""), "{report}"); + assert!(report.ends_with(""), "{report}"); + assert_eq!( + report.bytes().filter(|byte| *byte == 0).count(), + MAX_EVIDENCE_BYTES as usize + ); + assert!(report.len() < MAX_EVIDENCE_BYTES as usize + 300); + } +} diff --git a/tests/e2e-cucumber/src/lib.rs b/tests/e2e-cucumber/src/lib.rs index acd458f969..520ceb1574 100644 --- a/tests/e2e-cucumber/src/lib.rs +++ b/tests/e2e-cucumber/src/lib.rs @@ -9,6 +9,7 @@ pub mod fixture_binary; pub mod harness_diagnostics; pub mod http_server; pub mod installer_fixture; +pub mod lemonade_diagnostics; pub mod loopback_http; pub mod mock_server; pub mod model_id; diff --git a/tests/e2e-cucumber/src/serve_log.rs b/tests/e2e-cucumber/src/serve_log.rs index bf0ff9d8c1..90d7573ecc 100644 --- a/tests/e2e-cucumber/src/serve_log.rs +++ b/tests/e2e-cucumber/src/serve_log.rs @@ -55,6 +55,44 @@ fn parse_log_path(serve_stdout: &str) -> Option<&str> { .filter(|path| !path.is_empty()) } +/// The selected engine, if the serve reached the point of printing its plan. +#[must_use] +pub fn reported_engine(serve_stdout: &str) -> Option<&str> { + serve_stdout + .lines() + .find_map(|line| line.trim().strip_prefix("engine:")) + .map(str::trim) + .filter(|engine| !engine.is_empty()) +} + +/// The engine named on a `rocm serve` command line. +/// +/// Accepts `--engine X` and `--engine=X`. `None` when no engine was requested: +/// the CLI then picks a default, which only the printed plan reveals (see +/// [`reported_engine`]). A value that starts with `--` is another flag, not an +/// engine (`serve m --engine --managed`), so it counts as absent too: reporting +/// `--managed` as the engine would skip the backend inventory without the +/// [`ENGINE_UNKNOWN_EVIDENCE`] line that makes the skip visible. +#[must_use] +pub fn engine_from_serve_args<'a>(args: &[&'a str]) -> Option<&'a str> { + let mut args = args.iter().copied(); + while let Some(arg) = args.next() { + let value = if let Some(inline) = arg.strip_prefix("--engine=") { + Some(inline) + } else if arg == "--engine" { + args.next() + } else { + continue; + }; + return value.filter(|engine| !engine.is_empty() && !engine.starts_with("--")); + } + None +} + +/// What a failed serve's report says when no engine could be determined, so a +/// skipped backend inventory is visible rather than silently absent. +pub const ENGINE_UNKNOWN_EVIDENCE: &str = "engine unknown: backend evidence skipped"; + /// The last `max` lines of `text`. fn tail_lines(text: &str, max: usize) -> String { let lines: Vec<&str> = text.lines().collect(); @@ -119,16 +157,55 @@ fn slugify(name: &str) -> String { /// this is collecting. Seeking keeps the cost fixed however big the log grows. /// /// The log may still be being written, so `len` is a snapshot: the head and tail -/// are always real, and only the elided count can be slightly stale. -fn read_clamped(path: &Path, max: u64) -> std::io::Result<(Vec, String)> { +/// are always real, and only the elided count can be slightly stale. A log that +/// grows past the cap after the size was taken is clamped the same way and the +/// note says it grew (see [`read_clamped_file`]); no caller gets a silent +/// truncation. Callers: `archive_service_log` (the note lands on the archived-path +/// line) and the Lemonade report's pins and `version.txt` reads (the note follows +/// the quoted text); its manifest read discards the note, but a truncated JSON +/// manifest fails to parse and the report says so. +pub(crate) fn read_clamped(path: &Path, max: u64) -> std::io::Result<(Vec, String)> { + let file = std::fs::File::open(path)?; + let len = file.metadata()?.len(); + read_clamped_file(file, len, max) +} + +/// [`read_clamped`] on an open file whose size was read as `len`. Split out so a +/// test can hand it a `len` that has gone stale, which is what a log that grows +/// between the size check and the read looks like. +/// +/// A file that fits is read in full, but one byte past `max` is asked for: if it +/// is there, the log grew after `len` was taken. That is not allowed to pass as +/// "fits" with the surplus dropped without a word, so the read falls back to the +/// head and tail of the file as it is now and the note says it grew. A file that +/// shrank instead is returned as read, with a note that it changed. +pub(crate) fn read_clamped_file( + mut file: std::fs::File, + len: u64, + max: u64, +) -> std::io::Result<(Vec, String)> { use std::io::{Read as _, Seek as _, SeekFrom}; - let mut file = std::fs::File::open(path)?; - let len = file.metadata()?.len(); + let mut len = len; + let mut grown = false; if len <= max { let mut body = Vec::new(); - file.read_to_end(&mut body)?; - return Ok((body, String::new())); + file.by_ref() + .take(max.saturating_add(1)) + .read_to_end(&mut body)?; + if body.len() as u64 <= max { + return Ok((body, String::new())); + } + len = file.metadata()?.len(); + if len <= max { + body.truncate(usize::try_from(max).unwrap_or(usize::MAX)); + return Ok(( + body, + format!(" (truncated at {max} bytes: the file changed while it was read)"), + )); + } + file.seek(SeekFrom::Start(0))?; + grown = true; } // `len > max >= 2 * half`, so the two halves can never overlap. let half = max / 2; @@ -143,7 +220,15 @@ fn read_clamped(path: &Path, max: u64) -> std::io::Result<(Vec, String)> { let mut body = head; body.extend_from_slice(marker.as_bytes()); body.extend_from_slice(&tail); - Ok((body, format!(" ({elided} bytes elided from the middle)"))) + let grown_note = if grown { + "; the file grew while it was read" + } else { + "" + }; + Ok(( + body, + format!(" ({elided} bytes elided from the middle{grown_note})"), + )) } /// Copy the log written by the managed service `serve_stdout` launched into @@ -230,6 +315,8 @@ pub struct ServeAttempt<'a> { /// failed, means the engine may still hold the port and the device — which /// changes how every later failure in the run should be read. pub stop_status: &'a str, + /// Only Lemonade attempts carry the optional backend inventory. + pub backend_evidence: Option<&'a str>, } /// Render one serve attempt's evidence as a failure message. @@ -241,7 +328,7 @@ pub struct ServeAttempt<'a> { #[must_use] pub fn serve_attempt_report(attempt: &ServeAttempt<'_>) -> String { use crate::section; - format!( + let mut report = format!( "{}\n{}\n{}\n{}\n{}\n{}\n{}", attempt.headline, section("device state", attempt.device_state), @@ -250,14 +337,20 @@ pub fn serve_attempt_report(attempt: &ServeAttempt<'_>) -> String { section("service log (tail)", attempt.log_tail), section("service log (full)", attempt.archived_log), section("stop", attempt.stop_status), - ) + ); + if let Some(evidence) = attempt.backend_evidence { + report.push_str("\n--- Lemonade backend evidence ---\n"); + report.push_str(evidence); + } + report } #[cfg(test)] mod tests { use super::{ - MAX_ARCHIVED_LOG_BYTES, ServeAttempt, archive_service_log, parse_log_path, read_clamped, - serve_attempt_report, service_log_tail, slugify, tail_lines, + MAX_ARCHIVED_LOG_BYTES, ServeAttempt, archive_service_log, engine_from_serve_args, + parse_log_path, read_clamped, read_clamped_file, reported_engine, serve_attempt_report, + service_log_tail, slugify, tail_lines, }; const SERVE_PLAN: &str = "\ @@ -270,6 +363,38 @@ managed service launched readiness: starting "; + #[test] + fn engine_lookup_is_safe_for_missing_or_incomplete_serve_plans() { + assert_eq!(reported_engine(SERVE_PLAN), Some("vllm")); + assert_eq!(reported_engine(" engine: lemonade\n"), Some("lemonade")); + for output in ["", "serve failed", " engine: \n"] { + assert_eq!(reported_engine(output), None); + } + } + + #[test] + fn engine_is_read_from_both_flag_forms_and_absent_otherwise() { + assert_eq!( + engine_from_serve_args(&["serve", "m", "--engine", "lemonade", "--managed"]), + Some("lemonade") + ); + assert_eq!( + engine_from_serve_args(&["serve", "m", "--engine=lemonade", "--managed"]), + Some("lemonade") + ); + for args in [ + &["serve", "m", "--managed"][..], + &["serve", "m", "--engine"], + &["serve", "m", "--engine="], + &["serve", "m", "--engine", ""], + &["serve", "m", "--engine", "--managed"], + &["serve", "m", "--engine=--managed"], + &["serve", "--engine-version=1"], + ] { + assert_eq!(engine_from_serve_args(args), None, "{args:?}"); + } + } + #[test] fn log_path_is_read_from_the_indented_plan_line() { assert_eq!( @@ -395,6 +520,34 @@ managed service launched read_clamped(&path, max).expect("read log") } + /// A log that grows after its size was read must not be cut at the cap without + /// saying so. `len` is handed in stale (10 bytes under a 20-byte cap) for a file + /// that now holds 60: the read keeps the head and tail of what is there and the + /// note says it grew. Cutting at the cap silently turns this red, and a file + /// that really fits (stale or not) stays verbatim with no note. + #[test] + fn a_log_that_grew_after_its_size_was_read_is_clamped_and_says_so() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("service.log"); + let body: Vec = (0..60u8).map(|i| b'a' + i % 26).collect(); + std::fs::write(&path, &body).expect("write log"); + + let (clamped, note) = + read_clamped_file(std::fs::File::open(&path).expect("open"), 10, 20).expect("read"); + assert!(note.contains("the file grew while it was read"), "{note}"); + assert!(note.contains("40 bytes elided from the middle"), "{note}"); + let text = String::from_utf8_lossy(&clamped); + assert!(text.starts_with(&String::from_utf8_lossy(&body[..10]).to_string())); + assert!(text.ends_with(&String::from_utf8_lossy(&body[50..]).to_string())); + + let (whole, note) = + read_clamped_file(std::fs::File::open(&path).expect("open"), 60, 100).expect("read"); + assert_eq!((whole, note), (body.clone(), String::new())); + let (exact, note) = + read_clamped_file(std::fs::File::open(&path).expect("open"), 60, 60).expect("read"); + assert_eq!((exact, note), (body, String::new())); + } + /// A runaway log is bounded, but never at the cost of the startup banner: /// both ends survive and the gap says so. #[test] @@ -502,7 +655,7 @@ managed service launched /// full log went — none of which the plain readiness timeout reported. #[test] fn a_stalled_serve_report_carries_every_piece_of_evidence() { - let report = serve_attempt_report(&ServeAttempt { + let mut attempt = ServeAttempt { headline: "`rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0 --engine lemonade --managed` \ exited 0, but the endpoint never served `Qwen3-0.6B` within 600s", device_state: "device state: drained (48000 MiB free of 49000 MiB, floor 44100 MiB)", @@ -511,7 +664,15 @@ managed service launched log_tail: "load: failed to load model", archived_log: "archived to service-logs/scenario--svc.log", stop_status: "lemonade-qwen: stopped", - }); + backend_evidence: None, + }; + let report = serve_attempt_report(&attempt); + assert!(!report.contains("Lemonade backend evidence"), "{report}"); + attempt.backend_evidence = Some("fixture backend identity"); + assert_eq!( + serve_attempt_report(&attempt), + format!("{report}\n--- Lemonade backend evidence ---\nfixture backend identity") + ); for expected in [ "exited 0", diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 5e126d1f01..d3e236b973 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -13,7 +13,8 @@ use crate::E2eWorld; use crate::e2e::tui_driver::{TuiSession, default_timeout}; use e2e_cucumber::mock_server::{MockServer, ServiceRecordOptions, write_service_record_with}; use e2e_cucumber::serve_log::{ - ServeAttempt, archive_service_log, serve_attempt_report, service_log_tail, + ServeAttempt, archive_service_log, engine_from_serve_args, reported_engine, + serve_attempt_report, service_log_tail, }; use e2e_cucumber::serve_plan::{ServeRun, resolved_model}; @@ -136,11 +137,15 @@ fn stall_headline(invocation: &str, ready_substr: &str, timeout_secs: u64) -> St /// rather than three calls at each site: the log must be read and copied out /// while the service still owns it, because stopping the service both changes /// what the log's last lines say and is the step most likely to fail outright. +/// For Lemonade the backend inventory is read before the stop too, and the +/// `llama-server` probes, which execute a backend binary, run only after it so +/// they cannot contend with the still-running router. /// /// Mirrors what [`setup_gpu_model`] has always collected, so a stalled serve /// reports the same evidence wherever it happens. -fn serve_failure_evidence( - world: &E2eWorld, +async fn serve_failure_evidence( + world: &mut E2eWorld, + engine: Option<&str>, headline: &str, device_state: &str, stdout: &str, @@ -155,7 +160,24 @@ fn serve_failure_evidence( &crate::results_path(), world.current_scenario.as_deref().unwrap_or("scenario"), ); - let stop_status = stop_scenario_services(world); + // The stop sits inside this call because its position relative to the + // backend probes is the point: see `backend_evidence_around_stop`. + let app_paths = world.isolated_root.is_some().then(|| world.app_paths()); + let isolated_root = world + .isolated_root + .as_ref() + .map(|root| root.path().to_owned()); + let (stop_status, backend_evidence) = + e2e_cucumber::lemonade_diagnostics::backend_evidence_around_stop( + move || { + app_paths + .map(|paths| paths.engine_install_manifest_path("lemonade")) + .ok_or_else(|| "the scenario has no isolated root".to_owned()) + }, + engine, + move || stop_services_in(isolated_root.as_deref()), + ) + .await; serve_attempt_report(&ServeAttempt { headline, device_state, @@ -164,6 +186,7 @@ fn serve_failure_evidence( log_tail: &log_tail, archived_log: &archived_log, stop_status: &stop_status, + backend_evidence: backend_evidence.as_deref(), }) } @@ -202,7 +225,15 @@ async fn serve_and_wait( }; panic!( "{}", - serve_failure_evidence(world, &headline, &device_state, &stdout, &stderr) + serve_failure_evidence( + world, + engine_from_serve_args(args).or_else(|| reported_engine(&stdout)), + &headline, + &device_state, + &stdout, + &stderr, + ) + .await ); } @@ -543,9 +574,16 @@ fn default_engine_serve_target() -> &'static str { /// attempt shares the device with this one, and that must be visible in the /// failure rather than inferred from a serve that looks broken. fn stop_scenario_services(world: &E2eWorld) -> String { - world.isolated_root.as_ref().map_or_else( + stop_services_in(world.isolated_root.as_ref().map(tempfile::TempDir::path)) +} + +/// [`stop_scenario_services`] on the scenario's isolated root rather than the +/// world, so a caller that must hand the stop to a `Send` closure (the world holds +/// PTY handles and is not `Sync`) can capture the path alone. +fn stop_services_in(isolated_root: Option<&std::path::Path>) -> String { + isolated_root.map_or_else( || "not attempted: scenario has no isolated root".to_owned(), - |root| crate::stop_managed_services(root.path()), + crate::stop_managed_services, ) } @@ -658,13 +696,17 @@ async fn serve_gpu_model(world: &mut E2eWorld) { // outcome is quoted: a stop that found no record (the launch had not yet // written one) or failed means the next attempt did NOT get a clean // device, and the failure says so instead of looking like a broken serve. - diagnostics.push(serve_failure_evidence( - world, - &format!("attempt {attempt} of {attempts} (rc={rc})"), - &device_state, - &stdout, - &stderr, - )); + diagnostics.push( + serve_failure_evidence( + world, + Some(engine), + &format!("attempt {attempt} of {attempts} (rc={rc})"), + &device_state, + &stdout, + &stderr, + ) + .await, + ); if attempt == attempts { break; } @@ -1041,15 +1083,19 @@ async fn user_serves_default_engine(world: &mut E2eWorld) { &ready_substr, timeout_secs, ); + let stdout = world.cli_output.clone().unwrap_or_default(); + let stderr = world.cli_stderr.clone().unwrap_or_default(); panic!( "{}", serve_failure_evidence( world, + reported_engine(&stdout), &headline, &device_state, - world.cli_output.as_deref().unwrap_or_default(), - world.cli_stderr.as_deref().unwrap_or_default(), + &stdout, + &stderr, ) + .await ); } } @@ -2652,11 +2698,7 @@ async fn assert_next_automatic_port(world: &mut E2eWorld) { /// Extract the engine name from a serve plan's `engine: ` line. fn selected_engine(output: &str) -> &str { - let Some(engine) = output - .lines() - .find_map(|l| l.trim().strip_prefix("engine:")) - .map(str::trim) - else { + let Some(engine) = reported_engine(output) else { panic!("no 'engine:' line in serve output:\n{output}"); }; engine From 1dd034046a3395e0b5f67f605d8255b5fbe36468 Mon Sep 17 00:00:00 2001 From: Juho Date: Mon, 9 Nov 2026 12:31:17 +0000 Subject: [PATCH 010/102] ROCMAI-898: resolve registered Lemonade model names when serving directly on Linux (#963) On Linux, once a packaged llama-server backend is installed, `rocm serve --engine lemonade` loads the model through a short-lived lemond, then looks for its GGUF in the Hugging Face cache to launch llama-server directly. That lookup only knew the built-in default model and `owner/repo:variant` refs, so for any other name Lemonade registers (`Qwen3-0.6B-GGUF`, or an alias such as `qwen-smoke`) it killed the loaded server and bailed with "did not download", even with the GGUF on disk. The command exited 0 with `readiness: starting` and the endpoint never served the model. Resolve a registered name to its checkpoint through the `resources/server_models.json` registry shipped in the Lemonade embeddable, validated like any other checkpoint, and look that up. A checkpoint matching more than one cached file is not guessed at. The runtime directory is now a required argument of the lookup, so no caller can fall back to one that cannot see registered names, and the error text is derived from the same lookup instead of claiming nothing was downloaded. Verified on a Strix Halo (gfx1151) box in an isolated root: on main the e2e scenarios serve-07, serve-40, chat-07 and bench-06 fail; with this change they pass, and a chat request returns a reply. Signed-off-by: juhovainio (cherry picked from commit 8b8fdfc0ac8c10cdd48052738b9ad85a2f12a209) Signed-off-by: juhovainio --- engines/lemonade/src/direct_llama.rs | 443 +++++++++++++++++++++++++-- 1 file changed, 419 insertions(+), 24 deletions(-) diff --git a/engines/lemonade/src/direct_llama.rs b/engines/lemonade/src/direct_llama.rs index 0c3ba14e22..de4a96a2e6 100644 --- a/engines/lemonade/src/direct_llama.rs +++ b/engines/lemonade/src/direct_llama.rs @@ -103,7 +103,8 @@ fn ensure_hf_checkpoint_downloaded( log_path: Option<&Path>, ) -> Result<()> { let paths = AppPaths::discover()?; - if direct_llama_model_path(&paths, &request.model_ref).is_some() { + if direct_llama_model_path(&paths, &runtime.manifest.runtime_dir, &request.model_ref).is_some() + { return Ok(()); } let download_port = free_local_port()?; @@ -131,7 +132,9 @@ fn ensure_hf_checkpoint_downloaded( log_path, process_env, )?; - if direct_llama_model_path(&paths, &request.model_ref).is_some() { + if direct_llama_model_path(&paths, &runtime.manifest.runtime_dir, &request.model_ref) + .is_some() + { Ok(()) } else { bail!( @@ -153,7 +156,8 @@ pub(crate) fn ensure_direct_llama_model_available( log_path: Option<&Path>, ) -> Result<()> { let paths = AppPaths::discover()?; - if direct_llama_model_path(&paths, &request.model_ref).is_some() { + let runtime_dir = runtime.manifest.runtime_dir.as_path(); + if direct_llama_model_path(&paths, runtime_dir, &request.model_ref).is_some() { return Ok(()); } @@ -184,13 +188,12 @@ pub(crate) fn ensure_direct_llama_model_available( log_path, process_env, ); - if direct_llama_model_path(&paths, &request.model_ref).is_some() { + if direct_llama_model_path(&paths, runtime_dir, &request.model_ref).is_some() { Ok(()) } else { bail!( - "Lemonade did not download `{}` for direct ROCm serving (searched {})", - request.model_ref, - describe_hf_cache_roots(&paths) + "{}", + unresolved_model_message(&hf_cache_roots(&paths), runtime_dir, &request.model_ref) ) } })(); @@ -212,12 +215,14 @@ pub(crate) fn serve_direct_llama_server( reason: &anyhow::Error, ) -> Result<()> { let paths = AppPaths::discover()?; - let model_path = direct_llama_model_path(&paths, &request.model_ref).with_context(|| { - format!( - "Lemonade downloaded model `{}` was not found for direct serving", - request.model_ref - ) - })?; + let model_path = + direct_llama_model_path(&paths, &runtime.manifest.runtime_dir, &request.model_ref) + .with_context(|| { + format!( + "Lemonade downloaded model `{}` was not found for direct serving", + request.model_ref + ) + })?; if !server.is_file() { bail!("Lemonade llama-server is missing at {}", server.display()); } @@ -518,22 +523,103 @@ pub(crate) fn llama_server_backend_label(server: &Path) -> String { .to_owned() } -pub(crate) fn direct_llama_model_path(paths: &AppPaths, model_ref: &str) -> Option { +/// Where the GGUF for `model_ref` is cached, if it is. +/// +/// `model_ref` may be a file path, an `owner/repo[:variant]` checkpoint, the built-in +/// default model, or a name from the Lemonade registry shipped in `runtime_dir` (e.g. +/// `Qwen3-0.6B-GGUF`). The runtime directory is required rather than optional so that +/// no caller can fall back to a lookup that cannot see registered names, which is the +/// state in which every non-default model failed to serve on Linux. +pub(crate) fn direct_llama_model_path( + paths: &AppPaths, + runtime_dir: &Path, + model_ref: &str, +) -> Option { + direct_llama_model_path_in(&hf_cache_roots(paths), runtime_dir, model_ref) +} + +/// The lookup over explicit cache `roots`, so a test is not at the mercy of whichever +/// models the host it runs on has cached. +fn direct_llama_model_path_in( + roots: &[PathBuf], + runtime_dir: &Path, + model_ref: &str, +) -> Option { let as_path = PathBuf::from(model_ref); if as_path.is_file() { return Some(as_path); } if let Some(checkpoint) = parse_hf_checkpoint(model_ref) { - return hf_cache_roots(paths) - .into_iter() - .find_map(|root| find_hf_checkpoint_gguf(&root, &checkpoint)); + return roots + .iter() + .find_map(|root| find_hf_checkpoint_gguf(root, &checkpoint)); } - if resolve_lemonade_model_ref(model_ref) != DEFAULT_MODEL { - return None; + if resolve_lemonade_model_ref(model_ref) == DEFAULT_MODEL { + return roots.iter().cloned().find_map(find_default_qwen_gguf); } - hf_cache_roots(paths) - .into_iter() - .find_map(find_default_qwen_gguf) + // A name Lemonade registers, other than the default: its registry says which + // checkpoint it downloads, and that checkpoint's GGUF is what ends up in the cache. + // Aliases (`qwen-smoke`, `tiny`, ...) name a registered model too. + let checkpoint = registry_checkpoint(runtime_dir, &resolve_lemonade_model_ref(model_ref))?; + let mut matches = hf_checkpoint_gguf_matches_in(roots, &checkpoint); + // Lemonade fetches the one file its registry names, so a single match is the + // expected case. Several (a `Q4_0` label also naming an `IQ4_0` file, say) cannot + // be told apart from here, and serving a different quantization than the one + // registered is worse than not finding the file. + if matches.len() == 1 { + matches.pop() + } else { + None + } +} + +/// Why direct ROCm serving cannot find a GGUF for `model_ref` after Lemonade loaded it, +/// worded from the same lookup that failed so the text cannot disagree with it: several +/// cached files matching one registered checkpoint is a different problem, with a +/// different remedy, from none being found. +fn unresolved_model_message(roots: &[PathBuf], runtime_dir: &Path, model_ref: &str) -> String { + if let Some(checkpoint) = + registry_checkpoint(runtime_dir, &resolve_lemonade_model_ref(model_ref)) + { + let matches = hf_checkpoint_gguf_matches_in(roots, &checkpoint); + if matches.len() > 1 { + let names = matches + .iter() + .filter_map(|path| path.file_name().and_then(|name| name.to_str())) + .collect::>() + .join(", "); + return format!( + "`{model_ref}` is registered as `{}/{}` and {} cached files match it ({names}), so direct ROCm serving cannot tell which to use; serve the one you want by its exact file name as `{}/{}:.gguf`", + checkpoint.owner, + checkpoint.repo, + matches.len(), + checkpoint.owner, + checkpoint.repo, + ); + } + } + format!( + "no cached GGUF was found for `{model_ref}`, so direct ROCm serving cannot start (searched {})", + render_hf_cache_roots(roots) + ) +} + +/// Lemonade's model registry inside the extracted embeddable, mapping each registered +/// model name to the Hugging Face checkpoint it downloads. +const SERVER_MODELS_RESOURCE: &str = "resources/server_models.json"; + +/// The Hugging Face checkpoint Lemonade's registry in `runtime_dir` records for the +/// registered model `name` (e.g. `Qwen3-0.6B-GGUF` -> `unsloth/Qwen3-0.6B-GGUF:Q4_0`), +/// or `None` when the registry is absent or unreadable, does not list `name`, or +/// records something that is not a safe `owner/repo[:variant]` checkpoint. +/// +/// The checkpoint is validated by [`parse_hf_checkpoint`], which is what keeps a +/// hand-edited registry from steering the cache lookup outside the cache. +fn registry_checkpoint(runtime_dir: &Path, name: &str) -> Option { + let registry = fs::read_to_string(runtime_dir.join(SERVER_MODELS_RESOURCE)).ok()?; + let registry = serde_json::from_str::(®istry).ok()?; + let checkpoint = registry.get(name)?.get("checkpoint")?.as_str()?; + parse_hf_checkpoint(checkpoint) } fn hf_cache_roots(paths: &AppPaths) -> Vec { @@ -632,10 +718,15 @@ fn collect_hf_checkpoint_ggufs(cache_root: &Path, checkpoint: &HfCheckpoint) -> /// deduplicated by file name. Used to detect an ambiguous variant (more than one match) /// so the caller can refuse to pick arbitrarily. fn hf_checkpoint_gguf_matches(paths: &AppPaths, checkpoint: &HfCheckpoint) -> Vec { + hf_checkpoint_gguf_matches_in(&hf_cache_roots(paths), checkpoint) +} + +/// [`hf_checkpoint_gguf_matches`] over explicit cache `roots`. +fn hf_checkpoint_gguf_matches_in(roots: &[PathBuf], checkpoint: &HfCheckpoint) -> Vec { let mut matches = Vec::new(); let mut seen_names = std::collections::HashSet::new(); - for root in hf_cache_roots(paths) { - let ggufs = collect_hf_checkpoint_ggufs(&root, checkpoint); + for root in roots { + let ggufs = collect_hf_checkpoint_ggufs(root, checkpoint); for gguf in filter_ggufs_by_variant(ggufs, checkpoint.variant.as_deref()) { if let Some(name) = gguf.file_name().and_then(|name| name.to_str()) && seen_names.insert(name.to_owned()) @@ -892,6 +983,310 @@ mod tests { fs::remove_dir_all(&dir).ok(); } + /// A runtime tree holding a Lemonade registry, and a hub cache holding the GGUFs + /// named in `cached` (`(repo_dir, file)` pairs), under one scratch directory. + fn registry_fixture( + tag: &str, + registry: &str, + cached: &[(&str, &str)], + ) -> (PathBuf, PathBuf, PathBuf) { + let dir = scratch_dir(tag); + let runtime_dir = dir.join("runtime"); + let resources = runtime_dir.join("resources"); + fs::create_dir_all(&resources).unwrap(); + fs::write(resources.join("server_models.json"), registry).unwrap(); + let hub = dir.join("hub"); + for (repo_dir, file) in cached { + let snapshot = hub.join(repo_dir).join("snapshots").join("rev0"); + fs::create_dir_all(&snapshot).unwrap(); + fs::write(snapshot.join(file), b"gguf").unwrap(); + } + (dir, runtime_dir, hub) + } + + /// Two entries trimmed from `resources/server_models.json` as shipped in the + /// Lemonade 11.5.1 embeddable (checked against the real file: top-level object keyed + /// by model name, each with a string `checkpoint`). + const SMALL_MODEL_REGISTRY: &str = r#"{ + "Qwen3-0.6B-GGUF": {"checkpoint": "unsloth/Qwen3-0.6B-GGUF:Q4_0", "recipe": "llamacpp"}, + "Qwen3-4B-Instruct-2507-GGUF": { + "checkpoint": "unsloth/Qwen3-4B-Instruct-2507-GGUF:Qwen3-4B-Instruct-2507-Q4_K_M.gguf", + "recipe": "llamacpp" + } + }"#; + + /// The failure behind the Linux Lemonade serve lanes: `rocm serve Qwen3-0.6B-GGUF + /// --engine lemonade` loads the model through a short-lived `lemond`, which leaves + /// the GGUF in the hub cache, and the serve then bailed "did not download" because + /// only the built-in default model and `owner/repo:variant` refs could be found. + #[test] + fn a_registered_model_name_resolves_through_its_registry_checkpoint() { + let (dir, runtime_dir, hub) = registry_fixture( + "registered-model", + SMALL_MODEL_REGISTRY, + &[("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf")], + ); + let found = + direct_llama_model_path_in(std::slice::from_ref(&hub), &runtime_dir, "Qwen3-0.6B-GGUF"); + assert_eq!( + found, + Some( + hub.join("models--unsloth--Qwen3-0.6B-GGUF") + .join("snapshots") + .join("rev0") + .join("Qwen3-0.6B-Q4_0.gguf") + ) + ); + fs::remove_dir_all(&dir).ok(); + } + + #[test] + fn a_registered_model_that_is_not_cached_resolves_to_nothing() { + let (dir, runtime_dir, hub) = + registry_fixture("registered-uncached", SMALL_MODEL_REGISTRY, &[]); + assert_eq!( + direct_llama_model_path_in(std::slice::from_ref(&hub), &runtime_dir, "Qwen3-0.6B-GGUF"), + None + ); + fs::remove_dir_all(&dir).ok(); + } + + /// A name the registry does not list is not guessed at, and without a runtime to + /// read a registry from nothing but the old lookups apply. + #[test] + fn an_unregistered_name_or_a_missing_registry_resolves_to_nothing() { + let (dir, runtime_dir, hub) = registry_fixture( + "unregistered", + SMALL_MODEL_REGISTRY, + &[("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf")], + ); + let roots = std::slice::from_ref(&hub); + assert_eq!( + direct_llama_model_path_in(roots, &runtime_dir, "Not-A-Registered-Model"), + None + ); + let no_registry = dir.join("no-registry"); + fs::create_dir_all(&no_registry).unwrap(); + assert_eq!( + direct_llama_model_path_in(roots, &no_registry, "Qwen3-0.6B-GGUF"), + None + ); + fs::remove_dir_all(&dir).ok(); + } + + /// A malformed registry degrades to "not found", never a panic or a guess. + #[test] + fn a_malformed_registry_resolves_to_nothing() { + for (tag, registry) in [ + ("registry-not-json", "not json"), + ("registry-not-object", "[]"), + ( + "registry-no-checkpoint", + r#"{"Qwen3-0.6B-GGUF": {"recipe": "llamacpp"}}"#, + ), + ( + "registry-checkpoint-not-string", + r#"{"Qwen3-0.6B-GGUF": {"checkpoint": 7}}"#, + ), + ] { + let (dir, runtime_dir, hub) = registry_fixture( + tag, + registry, + &[("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf")], + ); + assert_eq!( + direct_llama_model_path_in( + std::slice::from_ref(&hub), + &runtime_dir, + "Qwen3-0.6B-GGUF" + ), + None, + "{tag}" + ); + fs::remove_dir_all(&dir).ok(); + } + } + + /// The registry is a file inside the runtime tree, so a hand-edited one must not be + /// able to point the cache lookup outside the cache: the checkpoint's components + /// are joined into a path, and `b/../../x` would climb out of the hub through a + /// directory that exists. The target here sits outside the hub and is a real file, + /// so only the validation in `parse_hf_checkpoint` stops it being served. + #[test] + fn a_registry_checkpoint_cannot_point_outside_the_cache() { + let (dir, runtime_dir, hub) = registry_fixture( + "registered-escape", + r#"{"Evil-Model": {"checkpoint": "a/b/../../x:Q4_0"}}"#, + &[], + ); + fs::create_dir_all(hub.join("models--a--b")).unwrap(); + let outside = dir.join("x").join("snapshots").join("rev0"); + fs::create_dir_all(&outside).unwrap(); + fs::write(outside.join("escape-Q4_0.gguf"), b"gguf").unwrap(); + assert_eq!( + direct_llama_model_path_in(std::slice::from_ref(&hub), &runtime_dir, "Evil-Model"), + None + ); + fs::remove_dir_all(&dir).ok(); + } + + /// The registry's variant is a substring match, so a cache holding two files it + /// matches (`Q4_0` also names `IQ4_0`) must not have one picked arbitrarily. + #[test] + fn a_registered_model_matching_two_cached_files_is_not_guessed() { + let (dir, runtime_dir, hub) = registry_fixture( + "registered-ambiguous", + SMALL_MODEL_REGISTRY, + &[ + ("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf"), + ("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-IQ4_0.gguf"), + ], + ); + assert_eq!( + direct_llama_model_path_in(std::slice::from_ref(&hub), &runtime_dir, "Qwen3-0.6B-GGUF"), + None + ); + fs::remove_dir_all(&dir).ok(); + } + + /// A registry entry whose variant is an exact filename resolves to that file, which + /// is how the built-in default model is registered. + #[test] + fn a_registry_checkpoint_naming_an_exact_file_resolves_to_that_file() { + let (dir, runtime_dir, hub) = registry_fixture( + "registered-exact-file", + SMALL_MODEL_REGISTRY, + &[ + ( + "models--unsloth--Qwen3-4B-Instruct-2507-GGUF", + "Qwen3-4B-Instruct-2507-Q4_K_M.gguf", + ), + ( + "models--unsloth--Qwen3-4B-Instruct-2507-GGUF", + "Qwen3-4B-Instruct-2507-Q8_0.gguf", + ), + ], + ); + // Registered under a name that is not the built-in default's, so only the + // registry lookup can find it. + let registry = + SMALL_MODEL_REGISTRY.replace("Qwen3-4B-Instruct-2507-GGUF\": {", "Other-Name\": {"); + fs::write( + runtime_dir.join("resources").join("server_models.json"), + registry, + ) + .unwrap(); + assert_eq!( + direct_llama_model_path_in(std::slice::from_ref(&hub), &runtime_dir, "Other-Name") + .and_then(|path| path.file_name().map(std::borrow::ToOwned::to_owned)), + Some("Qwen3-4B-Instruct-2507-Q4_K_M.gguf".into()) + ); + fs::remove_dir_all(&dir).ok(); + } + + /// The message is derived from the lookup that failed: with two cached files + /// matching one registered checkpoint it names both and how to pick one, and with + /// none it says nothing was found rather than claiming nothing was downloaded. + #[test] + fn the_unresolved_message_follows_the_reason_the_lookup_failed() { + let (dir, runtime_dir, hub) = registry_fixture( + "unresolved-ambiguous", + SMALL_MODEL_REGISTRY, + &[ + ("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf"), + ("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-IQ4_0.gguf"), + ], + ); + let roots = std::slice::from_ref(&hub); + // The state: the lookup refuses to pick. + assert_eq!( + direct_llama_model_path_in(roots, &runtime_dir, "Qwen3-0.6B-GGUF"), + None + ); + // The message: it says why, and what to do. + let ambiguous = unresolved_model_message(roots, &runtime_dir, "Qwen3-0.6B-GGUF"); + assert!(ambiguous.contains("2 cached files match"), "{ambiguous}"); + assert!( + ambiguous.contains("Qwen3-0.6B-Q4_0.gguf") + && ambiguous.contains("Qwen3-0.6B-IQ4_0.gguf"), + "{ambiguous}" + ); + assert!( + ambiguous.contains("unsloth/Qwen3-0.6B-GGUF:.gguf"), + "{ambiguous}" + ); + assert!(!ambiguous.contains("did not download"), "{ambiguous}"); + + // Nothing cached: the lookup finds nothing, and the text does not claim a + // download outcome it cannot know. + let (empty_dir, empty_runtime, empty_hub) = + registry_fixture("unresolved-empty", SMALL_MODEL_REGISTRY, &[]); + let empty_roots = std::slice::from_ref(&empty_hub); + assert_eq!( + direct_llama_model_path_in(empty_roots, &empty_runtime, "Qwen3-0.6B-GGUF"), + None + ); + let none = unresolved_model_message(empty_roots, &empty_runtime, "Qwen3-0.6B-GGUF"); + assert!(none.contains("no cached GGUF was found"), "{none}"); + assert!( + none.contains(&empty_hub.display().to_string()), + "names the searched root: {none}" + ); + assert!(!none.contains("did not download"), "{none}"); + fs::remove_dir_all(&dir).ok(); + fs::remove_dir_all(&empty_dir).ok(); + } + + /// `rocm serve qwen-smoke` reaches the lookup under the alias, which Lemonade + /// registers as `Qwen3-0.6B-GGUF`: the registry is keyed by the registered name, so + /// the alias has to be resolved first or only the exact name would ever serve. + #[test] + fn a_model_alias_resolves_through_the_registry_like_its_registered_name() { + let (dir, runtime_dir, hub) = registry_fixture( + "registered-alias", + SMALL_MODEL_REGISTRY, + &[("models--unsloth--Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q4_0.gguf")], + ); + let roots = std::slice::from_ref(&hub); + let exact = direct_llama_model_path_in(roots, &runtime_dir, "Qwen3-0.6B-GGUF"); + assert!(exact.is_some()); + for alias in ["qwen-smoke", "tiny", "lemonade-tiny", "qwen3-0.6b-gguf"] { + assert_eq!( + direct_llama_model_path_in(roots, &runtime_dir, alias), + exact, + "{alias}" + ); + } + fs::remove_dir_all(&dir).ok(); + } + + /// `direct_llama_model_path` is the entry point every serve path goes through, so it + /// must pass the runtime's registry on to the lookup: found here only through the + /// registry, in the cache directory the CLI itself owns. + #[test] + fn the_entry_point_threads_the_runtime_registry_through() { + let (dir, runtime_dir, _hub) = registry_fixture("entry-point", SMALL_MODEL_REGISTRY, &[]); + let cache_dir = dir.join("cli-cache"); + let snapshot = cache_dir + .join("huggingface") + .join("hub") + .join("models--unsloth--Qwen3-0.6B-GGUF") + .join("snapshots") + .join("rev0"); + fs::create_dir_all(&snapshot).unwrap(); + fs::write(snapshot.join("Qwen3-0.6B-Q4_0.gguf"), b"gguf").unwrap(); + let paths = AppPaths { + config_dir: dir.join("config"), + data_dir: dir.join("data"), + cache_dir, + }; + assert_eq!( + direct_llama_model_path(&paths, &runtime_dir, "Qwen3-0.6B-GGUF"), + Some(snapshot.join("Qwen3-0.6B-Q4_0.gguf")) + ); + fs::remove_dir_all(&dir).ok(); + } + fn test_manifest(runtime_dir: PathBuf) -> LemonadeInstallManifest { LemonadeInstallManifest { env_id: "test".to_owned(), From 14f97400eb44774e806078b27f571d11f0130194 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 12:34:39 +0000 Subject: [PATCH 011/102] test(e2e): resize the emulated terminal before the PTY so a redraw is never parsed at the old size (#955) * test(e2e): resize the emulated terminal before the PTY so a redraw is never parsed at the old size The TUI driver resized the real PTY and then the emulated screen. The application redraws for the new size the moment it is signalled, and the reader thread parses that frame as soon as it arrives, so whenever the harness thread was descheduled between the two calls the frame, laid out for 120 columns, was parsed by a screen still 80 wide. Its lines wrapped and scrolled the top rows, the header with its warnings badge among them, off the screen, and because the dashboard only repaints the cells that changed the damage was never repaired. The dash-09 step that waits for the header badge to grow then timed out on a frozen, garbled screen, which is the intermittent failure seen on loaded runners; dash-08 and every other scenario that resizes shared the exposure. Resize the emulator first and the PTY second, under one lock, through a new helper in the library crate so a plain unit test can cover the ordering (the driver itself only builds into the cucumber binary). A failed PTY resize puts the emulated screen back, as before. Signed-off-by: Jussi Elo * test(e2e): leave the emulated screen unlocked while the PTY resize runs, and say what the helper does not guarantee The reader thread drains the PTY under the emulator lock, so holding that lock across the PTY resize meant the reader could not drain while the resize ran. A Linux ioctl does not care, but nothing rules out a Windows pseudoconsole resize waiting on undrained output, and then the two would deadlock. The emulator is now resized first and unlocked before the PTY call, which keeps the ordering that fixes the corrupted-redraw race and removes the question. A failed PTY resize still restores the old size. The ordering test now makes the PTY call wait until the reader has parsed the child's redraw, so it also fails if the emulator is kept locked across the call, not only if the PTY is resized first. The doc comment says what the rollback does not restore (cells a shrink truncated), why shrinking is not tested (it would pass with either order, since the child clears and redraws), and the helper no longer returns a generic value that only a test used. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 6b9c17ee8894a2e5e710ae1b730d1de3925a3219) Signed-off-by: juhovainio --- tests/e2e-cucumber/src/terminal_size.rs | 174 ++++++++++++++++++++- tests/e2e-cucumber/tests/e2e/tui_driver.rs | 40 +++-- 2 files changed, 190 insertions(+), 24 deletions(-) diff --git a/tests/e2e-cucumber/src/terminal_size.rs b/tests/e2e-cucumber/src/terminal_size.rs index 4481cdf456..b861672487 100644 --- a/tests/e2e-cucumber/src/terminal_size.rs +++ b/tests/e2e-cucumber/src/terminal_size.rs @@ -2,9 +2,11 @@ // // SPDX-License-Identifier: MIT -//! Validation of a requested terminal size before it reaches the emulator. +//! Validation of a requested terminal size, and the order in which the emulated +//! screen and the real PTY take it. use std::num::NonZeroU16; +use std::sync::Mutex; /// `rows` and `cols` as the `NonZeroU16`s the `vt100` emulator takes, or an /// error naming the requested size if either is zero. @@ -21,9 +23,75 @@ pub fn non_zero_dimensions(rows: u16, cols: u16) -> Result<(NonZeroU16, NonZeroU } } +/// Resize the emulated screen first, then the real PTY. +/// +/// `resize_pty` is the call that resizes the real PTY and so signals the child. +/// It runs after the emulated screen has taken the new size and with the +/// emulator unlocked, so the reader thread keeps draining the child's output +/// while it runs. If it fails, the emulated screen goes back to its old size so +/// the two never disagree; that restores the size only, not any cells a shrink +/// truncated, which is acceptable because a failed resize fails the step. +/// +/// The order is the point. The child redraws the moment it learns the new size, +/// and the reader thread feeds whatever it writes to the emulator. Resizing the +/// PTY first and the emulator second leaves a window between the two calls in +/// which a frame laid out for the new width is parsed by the old, narrower +/// screen: its lines wrap and scroll the top rows (the header among them) off, +/// and because a terminal UI only repaints the cells that changed, nothing ever +/// repairs the damage. Any scheduling stall of the harness thread between the +/// two calls (a loaded CI runner) opens that window. With the emulator resized +/// first, whatever the child writes in answer to the resize is parsed at the new +/// size. +/// +/// The same argument is not made, and not tested, for shrinking: a frame the +/// child drew for the old, larger size can reach the already smaller emulator +/// and wrap there, but the child clears and redraws on a resize, so that residue +/// is repainted over. Shrinking relies on the child doing so; a test of it would +/// pass with either order and so would not guard anything. +/// +/// The emulator is deliberately not held locked across `resize_pty`: the reader +/// drains the PTY under that lock, and a PTY resize that waits for the child's +/// output to drain (which a Windows pseudoconsole is not documented to rule out) +/// would then deadlock against it. +/// +/// Pure of any PTY on purpose, like [`non_zero_dimensions`]: the driver lives in +/// the cucumber binary, where plain `#[test]`s never run, so the ordering is +/// covered here by `cargo test -p e2e-cucumber --lib`. +pub fn resize_screen_then_pty( + parser: &Mutex, + rows: u16, + cols: u16, + resize_pty: impl FnOnce() -> Result<(), String>, +) -> Result<(), String> { + // Validate before touching either side, so a zero dimension fails cleanly + // instead of leaving the PTY and the emulated screen at different sizes. + let (vt_rows, vt_cols) = non_zero_dimensions(rows, cols)?; + // Recover rather than fail on a poisoned lock: resizing is still meaningful + // even if some earlier operation panicked while holding it. + let lock = || { + parser + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + }; + let (old_rows, old_cols) = { + let mut parser = lock(); + let old = parser.screen().size(); + parser.screen_mut().set_size(vt_rows, vt_cols); + old + }; + let resized = resize_pty(); + if resized.is_err() { + lock().screen_mut().set_size(old_rows, old_cols); + } + resized +} + #[cfg(test)] mod tests { - use super::non_zero_dimensions; + use super::{non_zero_dimensions, resize_screen_then_pty}; + use std::fmt::Write as _; + use std::sync::{Arc, Mutex, mpsc}; + use std::time::Duration; #[test] fn accepts_non_zero_dimensions_and_keeps_their_order() { @@ -54,4 +122,106 @@ mod tests { Err("cannot resize to rows=0 cols=0: zero dimension".to_owned()) ); } + + /// Cursor-addressed frame of `rows` full-width lines, as a terminal UI + /// writes it after a resize: clear, then every row at an absolute position. + fn frame(rows: u16, cols: u16) -> Vec { + let mut out = String::from("\x1b[2J"); + for row in 0..rows { + let label = format!("row{row:02}"); + let fill = "x".repeat(usize::from(cols) - label.len()); + let _ = write!(out, "\x1b[{};1H{label}{fill}", row + 1); + } + out.into_bytes() + } + + fn parser(rows: u16, cols: u16) -> Arc> { + let (rows, cols) = non_zero_dimensions(rows, cols).expect("a valid test size"); + Arc::new(Mutex::new(vt100::Parser::new(rows, cols, 0))) + } + + fn size(parser: &Mutex) -> (u16, u16) { + let (rows, cols) = parser.lock().unwrap().screen().size(); + (rows.get(), cols.get()) + } + + #[test] + fn a_frame_the_child_draws_in_answer_to_the_resize_lands_on_the_new_size() { + let parser = parser(24, 80); + // The reader thread parses the child's answer the moment it is released, + // under the parser lock, exactly as the driver's reader does. + let (go, released) = mpsc::channel::<()>(); + let (parsed, wait_parsed) = mpsc::channel::<()>(); + let reader = { + let parser = Arc::clone(&parser); + std::thread::spawn(move || { + released + .recv() + .expect("the resize never signalled the child"); + parser.lock().unwrap().process(&frame(40, 120)); + parsed.send(()).unwrap(); + }) + }; + + let resized = resize_screen_then_pty(&parser, 40, 120, || { + // The child redraws as soon as it is signalled, and the harness + // thread does not return from the PTY call until the reader has + // parsed that redraw: the window a PTY-first order leaves open (a + // frame parsed before the emulator has its new size), and, if the + // emulator were kept locked across this call, one the reader could + // never get through, so the wait below would time out. + go.send(()).unwrap(); + wait_parsed + .recv_timeout(Duration::from_secs(5)) + .map_err(|_| { + "the reader could not parse the child's redraw while the PTY resize ran" + .to_owned() + }) + }); + assert_eq!(resized, Ok(())); + reader.join().unwrap(); + + assert_eq!(size(&parser), (40, 120)); + let guard = parser.lock().unwrap(); + let screen = guard.screen(); + for (row, text) in screen.rows(0, 120).enumerate() { + let label = format!("row{row:02}"); + assert_eq!( + text, + format!("{label}{}", "x".repeat(120 - label.len())), + "row {row} was not drawn intact: a frame laid out for 120 columns \ + was parsed by a screen still sized for the old geometry" + ); + } + } + + #[test] + fn a_failed_pty_resize_puts_the_emulated_screen_back_and_reports_the_error() { + let parser = parser(24, 80); + let resized = + resize_screen_then_pty( + &parser, + 40, + 120, + || Err("failed to resize pty: boom".into()), + ); + assert_eq!(resized, Err("failed to resize pty: boom".to_owned())); + assert_eq!(size(&parser), (24, 80)); + } + + #[test] + fn a_zero_dimension_resizes_neither_side() { + let parser = parser(24, 80); + let mut pty_resized = false; + let resized = resize_screen_then_pty(&parser, 0, 120, || { + pty_resized = true; + Ok(()) + }); + assert_eq!( + resized, + Err("cannot resize to rows=0 cols=120: zero dimension".to_owned()) + ); + assert!(!pty_resized, "the PTY was resized for an invalid size"); + assert_eq!(size(&parser), (24, 80)); + } } diff --git a/tests/e2e-cucumber/tests/e2e/tui_driver.rs b/tests/e2e-cucumber/tests/e2e/tui_driver.rs index f818136f80..61d1757d3c 100644 --- a/tests/e2e-cucumber/tests/e2e/tui_driver.rs +++ b/tests/e2e-cucumber/tests/e2e/tui_driver.rs @@ -36,7 +36,7 @@ use e2e_cucumber::panic_capture::panic_message; use e2e_cucumber::quit_and_wait::{ExitPoll, QuitError, quit_and_wait as retry_quit_and_wait}; use e2e_cucumber::reader_failure::{ReaderFailure, ReaderFailureObservation}; use e2e_cucumber::send_until::{RetryTiming, TerminalState, send_until as retry_send_until}; -use e2e_cucumber::terminal_size::non_zero_dimensions; +use e2e_cucumber::terminal_size::resize_screen_then_pty; use crate::E2eWorld; @@ -645,28 +645,24 @@ impl TuiSession { /// Resize both the real PTY and the emulated screen to `rows`x`cols`. The /// application receives the normal terminal resize event; assertions /// continue to inspect exactly what a user would see at the new geometry. + /// + /// The emulated screen is resized first and the PTY second, through + /// [`resize_screen_then_pty`]: the other order lets the application's + /// redraw for the new size be parsed by the old, narrower screen whenever + /// this thread is descheduled between the two calls, which corrupts the + /// screen for good (see that function). fn resize_to(&mut self, rows: u16, cols: u16) -> Result<(), String> { - // Validate before resizing the PTY so a zero dimension fails cleanly - // instead of leaving the PTY and the emulated screen at different sizes. - let (vt_rows, vt_cols) = non_zero_dimensions(rows, cols)?; - let size = PtySize { - rows, - cols, - pixel_width: 0, - pixel_height: 0, - }; - self.master - .resize(size) - .map_err(|e| format!("failed to resize pty: {e}"))?; - // As in `screen_snapshot`, recover rather than fail on a poisoned lock — - // resizing is still meaningful even if some earlier operation panicked - // while holding it. - self.parser - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .screen_mut() - .set_size(vt_rows, vt_cols); - Ok(()) + let master = &self.master; + resize_screen_then_pty(&self.parser, rows, cols, || { + master + .resize(PtySize { + rows, + cols, + pixel_width: 0, + pixel_height: 0, + }) + .map_err(|e| format!("failed to resize pty: {e}")) + }) } /// Enlarge to [`DETAIL_ROWS`]x[`DETAIL_COLS`], for journeys that assert From 4a6f0d06cc86aacaf8d7755d22ed5b94123544fd Mon Sep 17 00:00:00 2001 From: Juho Date: Mon, 9 Nov 2026 12:35:29 +0000 Subject: [PATCH 012/102] ROCMAI-899, ROCMAI-900: expect the two Windows Lemonade scenarios to fail on a 7.14.x SDK (#964) `rocm engines install lemonade` aligns Lemonade's llama.cpp backend with the active ROCm SDK. Lemonade 11.5.1, the pinned version, downloads the TheRock tarball it pairs with the backend only from repo.amd.com/rocm/tarball/, which answers HTTP 403 for 7.14.x (those tarballs are published under tarball-multi-arch/), and the pinned llama.cpp build has no ROCm 7.14 asset (HTTP 404). Every alignment tier fails and the install reverts to the pinned 7.13 backend. This also reproduces off CI, on a Linux box. Two Windows lane scenarios fail as a result, on every run examined (2026-10-28, 2026-11-05, 2026-11-06): runtime-16 asserts the alignment succeeded, and serve-06 serves a 35B model whose server crashes in the SDK's Tensile library on the first inference request. The backend/SDK mismatch is the suspected cause of the crash and has not been reproduced off Windows, which the row text says. Add deterministic xfail rows for both, scoped to native Windows with lemonade, so an XPASS flags them as stale once the cause is fixed. Neither row sets a serve timeout: serve-06's own tag takes precedence over a row. A test pins the scope, including that Linux hosts and a non-lemonade engine on Windows do not inherit the rows, and note the runtime-16 scenario comment that its premise no longer holds for 7.14.x. Signed-off-by: juhovainio (cherry picked from commit 4aabdb2f94aa2b4eec36995ef39d4b601d7a2723) Signed-off-by: juhovainio --- tests/e2e-cucumber/expectations.toml | 36 +++++++++++++++++++ .../features/runtime_setup.feature | 5 ++- tests/e2e-cucumber/src/expectation.rs | 36 +++++++++++++++++++ 3 files changed, 76 insertions(+), 1 deletion(-) diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index fcb0b52a4e..2307516266 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -174,3 +174,39 @@ bug = "ROCMAI-898" reason = "Lemonade managed serve never exposes the model: observed backend argument rejection, backend download failure, or backend-ready/router-unavailable." serve_timeout_secs = 90 flaky = true + +# --- Lemonade 11.5.1 cannot align its llama.cpp backend to a ROCm 7.14.x SDK --- +# `rocm engines install lemonade` tries to align Lemonade's `llamacpp:rocm` +# backend with the active ROCm SDK. Lemonade 11.5.1, the pin in +# `runtime-deps.toml`, downloads the TheRock tarball it pairs with the backend +# only from https://repo.amd.com/rocm/tarball/, a path the 7.14.x tarballs are not +# published to (AMD's 7.14 install guide uses /rocm/tarball-multi-arch/), so the +# download answers HTTP 403 for `therock-dist---7.14.1.tar.gz`. The +# pinned llama.cpp build has no ROCm 7.14 asset either (HTTP 404), so every tier +# of the alignment fails and the CLI reverts to Lemonade's pinned ROCm 7.13 +# backend. Reproduced on a Strix Halo Linux box with an isolated root, not only on +# the CI runners, so the failed alignment is not specific to Windows; the scenarios +# below only fail there (runtime-16 is `@requires-os:windows`, and serve-06 passed on +# the W7900 and MI300X lanes of the 2026-11-05 run). +# +# runtime-16 asserts the alignment succeeded, which it cannot while the 7.14.x +# tarball is unreachable. serve-06 serves a 35B model through the Windows lane's +# 7.13 backend against the SDK's 7.14.1 libraries and the server crashes in the +# SDK's Tensile library on the first inference request (access violation in the +# lane's service log). The mismatch between the backend and the SDK is the +# suspected cause; it has NOT been reproduced off Windows, so the row names both +# facts. Both rows are deterministic (both scenarios failed on each Windows lane +# run examined: 2026-10-28, 2026-11-05 and 2026-11-06), so neither is `flaky`: an +# XPASS means the cause is gone and the row should be deleted. Neither row sets +# `serve_timeout_secs`: serve-06's own `@serve-timeout:2400` tag takes precedence +# over a row, so a shorter wait here would have no effect, and the failing +# serve-06 still holds the Windows lane for the full 2400s wait. +[["runtime-lemonade-backend-alignment-reported"]] +when = { effective_engine = "lemonade", os = "windows" } +bug = "ROCMAI-899" +reason = "Lemonade 11.5.1 fetches TheRock only from repo.amd.com/rocm/tarball/, where 7.14.x is not published (HTTP 403), so its backend cannot be aligned to a 7.14.x SDK and the install reverts to the pinned 7.13 backend." + +[["serve-large-model-inference"]] +when = { effective_engine = "lemonade", os = "windows" } +bug = "ROCMAI-900" +reason = "The Windows large-model serve crashes in the SDK's Tensile library on the first inference request; the pinned ROCm 7.13 backend runs against the 7.14.1 SDK because Lemonade 11.5.1 cannot align it (suspected cause, not reproduced off Windows)." diff --git a/tests/e2e-cucumber/features/runtime_setup.feature b/tests/e2e-cucumber/features/runtime_setup.feature index bc1b24ef42..4f820a1099 100644 --- a/tests/e2e-cucumber/features/runtime_setup.feature +++ b/tests/e2e-cucumber/features/runtime_setup.feature @@ -327,7 +327,10 @@ Feature: Runtime configuration # against real hardware (Strix Halo, gfx1151): a fresh SDK's version does not # match Lemonade's packaged pin, Tier 1's install 404s (the pinned build # predates a matching ROCm-version asset), and Tier 2's newest build - # succeeds -- producing exactly the line this scenario asserts. + # succeeds -- producing exactly the line this scenario asserts. That held for + # an SDK whose TheRock tarball Lemonade could download; for a 7.14.x SDK Tier 2 + # fails too (the tarball is not published where Lemonade 11.5.1 looks), which is + # why `expectations.toml` carries a Windows row for this scenario (ROCMAI-899). # # `@requires-engine:lemonade` because vLLM shares the SDK's own runtime # environment and has no llama.cpp backend to align; `@nightly` for the same diff --git a/tests/e2e-cucumber/src/expectation.rs b/tests/e2e-cucumber/src/expectation.rs index 3e7a5c13c8..52736ff4ca 100644 --- a/tests/e2e-cucumber/src/expectation.rs +++ b/tests/e2e-cucumber/src/expectation.rs @@ -2353,6 +2353,42 @@ flaky = true )); } + /// The two rows for Lemonade 11.5.1's failed backend alignment apply to native + /// Windows with lemonade and to nothing else: not to Linux hosts (including one whose + /// default engine is vLLM, where the engine named is what decides), and they are not + /// `flaky` (an XPASS means the cause is gone and the row is stale). Neither sets + /// `serve_timeout_secs`: serve-06's own `@serve-timeout` tag takes precedence over a + /// row, so one would change nothing. + #[test] + fn lemonade_alignment_rows_apply_only_to_native_windows_with_lemonade() { + let m = Expectations::parse(include_str!("../expectations.toml")).unwrap(); + for id in [ + "runtime-lemonade-backend-alignment-reported", + "serve-large-model-inference", + ] { + assert!(m.is_xfail(id, &cap("strix-windows"), "lemonade"), "{id}"); + for linux in ["strix-ubuntu", "mi300x", "wsl2"] { + assert!(!m.is_xfail(id, &cap(linux), "lemonade"), "{id} on {linux}"); + } + // Same Windows host, other engine: the row is about lemonade. + assert!(!m.is_xfail(id, &cap("strix-windows"), "vllm"), "{id} vllm"); + let entries = m.entries_for(id); + assert_eq!(entries.len(), 1, "{id}"); + assert!(!entries[0].flaky, "{id} is deterministic"); + assert_eq!(entries[0].serve_timeout_secs, None, "{id}"); + } + // The row cannot shorten serve-06's wait: its own tag wins, so the harness + // still hands the 2400s the feature file asks for. + let tags = feature_tags( + include_str!("../features/model_serving.feature"), + "serve-large-model-inference", + ); + assert_eq!( + m.serve_timeout_override(&ScenarioDecl::from_tags(&tags), &cap("strix-windows")), + Some(2400) + ); + } + /// serve-40 serves through the same lemonade managed serve as the other /// lemonade serve rows, so its row is scoped the same way: xfail on a /// lemonade Linux host, with a 90s serve wait, and not an expected failure From f8e89aa793bd158295e2b8e8ca5fdf76d44ba093 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 12:54:23 +0000 Subject: [PATCH 013/102] docs: map every production source file and empty EXEMPT (#965) * refactor(dash): remove five dashboard modules that nothing uses Four of these were placeholders from the dashboard merge and the fifth was never called: - collectors proc_scan.rs and sysfs.rs: stub collectors whose every method returned CollectorError::Unsupported. Nothing constructed them. - tui ui/bench.rs and ui/instance_list.rs: empty modules, a doc comment saying "Stub" and nothing else. The real bench screen is ui/tabs/bench.rs, so the empty ui::bench only invited confusion. - collectors vllm_log.rs: VllmLogSlicer and VllmPeaks are referenced only by their own tests. The collectors crate doc no longer says stubs return Unsupported. The variant stays, because the Docker and vLLM-Prometheus implementations still return it from their sync trait methods. regex was used only by vllm_log.rs, so the dependency goes with it. Each file was listed in EXEMPT in check-architecture-doc as uncited; deleting them removes their entries and lowers EXEMPT_CEILING from 76 to 71. Signed-off-by: Jussi Elo * docs: give every production source file an entry in the module map and empty EXEMPT check-architecture-doc exempted the 71 files that were uncited when it landed, mostly in the dashboard crates. The exemption was a leftover of how the doc was written, not a judgment about the files: it inventoried only what the earlier modularization touched and described rocm-dash-core, -collectors and -daemon at crate level. Most of the files carry real constraints a contributor needs, such as which crate may not depend on rocm-core and the seams that stand in for it (tool_exec, registry_probe), the transport framing duplicated on purpose in the daemon and the TUI, and the approval path every overlay shares. This adds an entry for each of them, written from the code, and empties EXEMPT and EXEMPT_CEILING to zero. The two sentences that said the dashboard crates are not inventoried module by module are replaced, as are the four places that described EXEMPT as a shrinking backlog (architecture.md, CONTRIBUTING.md, AGENTS.md, testing.md). The ordering rule gains the ui/tabs list, which keeps mod.rs first. The check and its tests are unchanged. The exemption mechanism stays, with a ceiling of zero, so an exemption is still a visible, reviewed edit. Signed-off-by: Jussi Elo * docs(architecture): correct the ownership and ordering claims found in review Copilot found two wrong statements in the new module-map entries: the replay entry promised a two-second wall-clock cap when the code caps the recorded gap first and then divides by the speed, and the state.rs entry said each operational screen holds its own job State when there is one, AppState::jobs, which event_loop.rs lends to every handler. Both were instances of a class, claims about what is applied before what and who owns which state, so the new text was audited for that class against the code, and the same pass corrected what it found: - replay: the start sequence (Connecting, Connected, then events, ending in Disconnected), who starts it, and the gap cap bounding the recorded gap, so at 0.25x a long gap sleeps 8 s. - jobs: a spawn failure posts JobErr at once, a cancelled job posts nothing, the exit code is -1 without one, and cancel is polled after each line. - overlays: also opened by slash commands, the quit-confirm key and the startup focus; the launcher has its own blocking loop; the Chat tab writes its scroll bounds back while drawing, so tabs are not all read-only. - daemon persist: there is no --persist-dir flag; rocm dash always sets the directory, and a failed open disables persistence with a warning. - engine_registry: only label and from_label are used outside tests. - bench_load, bench_tail, docker, amd_smi, observation, rings and a few smaller entries: the stop, lock, reset, fallback and tracker-lifetime details that the first wording left out or got backwards. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit f126f417d515a9a7edcf902308d2c8cf99d20a5c) Signed-off-by: juhovainio --- AGENTS.md | 2 +- CONTRIBUTING.md | 2 +- Cargo.lock | 1 - crates/rocm-dash-collectors/Cargo.toml | 1 - crates/rocm-dash-collectors/src/lib.rs | 4 - crates/rocm-dash-collectors/src/proc_scan.rs | 27 ----- crates/rocm-dash-collectors/src/sysfs.rs | 39 ------- crates/rocm-dash-collectors/src/vllm_log.rs | 106 ----------------- crates/rocm-dash-tui/src/ui/bench.rs | 5 - crates/rocm-dash-tui/src/ui/instance_list.rs | 5 - crates/rocm-dash-tui/src/ui/mod.rs | 2 - docs/architecture.md | 96 ++++++++++++++-- docs/testing.md | 2 +- xtask/src/architecture_doc.rs | 114 +++---------------- 14 files changed, 107 insertions(+), 299 deletions(-) delete mode 100644 crates/rocm-dash-collectors/src/proc_scan.rs delete mode 100644 crates/rocm-dash-collectors/src/sysfs.rs delete mode 100644 crates/rocm-dash-collectors/src/vllm_log.rs delete mode 100644 crates/rocm-dash-tui/src/ui/bench.rs delete mode 100644 crates/rocm-dash-tui/src/ui/instance_list.rs diff --git a/AGENTS.md b/AGENTS.md index c02e9f866e..155b474510 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -278,7 +278,7 @@ Guardrails: - `crates/rocm-engine-protocol` is a contract surface; verify all impacted engines after protocol changes - first-party crate-layering invariants (e.g. `rocmd` must never depend on `rocm`) are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`); a new first-party dependency edge failing that check means the edge needs review, not a bypass - source file size is checked by `cargo xtask check-module-size` ([`module_size.rs`](xtask/src/module_size.rs)): no source file may exceed 1,500 production lines (test-gated items excluded), with one limit for every file and no exemption mechanism. A PR that would push a file past it splits the file in the same change; do not raise the limit or opt a file out. Merely editing a file that is already over the limit does not by itself require a split: those files are split under their own tickets, and the check cannot become blocking until none is left. Today the CI step and pre-commit hook only report; see `docs/architecture.md` -- every production source file (a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`) has an entry in `docs/architecture.md`, enforced by `cargo xtask check-architecture-doc` ([architecture_doc.rs](xtask/src/architecture_doc.rs)) in the `clippy` CI job and in the `architecture-doc` prek hook at commit and push. Add the entry in the same change as the file. Do not add the file to the check's `EXEMPT` list to make it pass: that list holds only what was uncited when the check was added and is meant to shrink: its size is capped by `EXEMPT_CEILING`, so growing it is a visible, reviewed exception. Remove an entry from it when you document that file. The check reads no prose, so keeping an entry's description true is still on you +- every production source file (a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`) has an entry in `docs/architecture.md`, enforced by `cargo xtask check-architecture-doc` ([architecture_doc.rs](xtask/src/architecture_doc.rs)) in the `clippy` CI job and in the `architecture-doc` prek hook at commit and push. Add the entry in the same change as the file. Do not add the file to the check's `EXEMPT` list to make it pass: the list is empty and `EXEMPT_CEILING` caps it at zero, so adding to it is a visible, reviewed exception. The check reads no prose, so keeping an entry's description true is still on you - preserve strict GPU-required behavior; do not introduce silent CPU fallback - respect platform gates (for example, native Windows handling for vLLM) - pin third-party GitHub Actions to a full commit SHA with a trailing `# vX.Y.Z` comment, never a moving tag (`@v2`, `@main`); a retagged or compromised action otherwise enters CI silently. Bump the SHA and comment together when upgrading diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 35c78f3e19..c7540bb9c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,7 +110,7 @@ The `docs-links` job below checks the file links in that document, including eve Crate-layering invariants, such as `rocmd` never depending on `rocm`, are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`). File size is checked by `cargo xtask check-module-size` (`xtask/src/module_size.rs`): no source file may exceed 1,500 production lines, test code excluded, and a change that would cross the limit splits the file instead. Editing a file that is already over the limit does not by itself require a split; those files are split under their own tickets. For now the CI step and the `cargo-module-size` pre-commit hook report offenders without failing. -Every production source file, meaning a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`, must have an entry in `docs/architecture.md`: a relative markdown link to it, which has to render as a link (a path inside an HTML comment, a code block or a code span does not count, and neither does a malformed link such as one missing its `)`, or an image). Use an inline `[text](path)` link with its text on one line; reference-style links, a `?query` suffix and root-relative paths are not recognised. `build.rs` files and the `tests/` and `examples/` trees are out of scope. `cargo xtask check-architecture-doc` (`xtask/src/architecture_doc.rs`) enforces that in the `clippy` CI job and in the `architecture-doc` prek hook, which runs on every commit and every push, over the whole tree. A new file with no entry fails it. This is the opposite direction to the `lychee` check below, which only confirms that links already in the doc still resolve. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. The files that had none when the check was added are listed in `EXEMPT` in `xtask/src/architecture_doc.rs`; that list is meant to shrink, and its size is capped by `EXEMPT_CEILING` in the same file, so growing it is a visible, reviewed exception rather than a way to make the check pass. When you document one of them, remove it from `EXEMPT` in the same change, or the check fails on the stale entry. The check does not read the prose of an entry, so keeping a description true when a file's role changes is still up to you. +Every production source file, meaning a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`, must have an entry in `docs/architecture.md`: a relative markdown link to it, which has to render as a link (a path inside an HTML comment, a code block or a code span does not count, and neither does a malformed link such as one missing its `)`, or an image). Use an inline `[text](path)` link with its text on one line; reference-style links, a `?query` suffix and root-relative paths are not recognised. `build.rs` files and the `tests/` and `examples/` trees are out of scope. `cargo xtask check-architecture-doc` (`xtask/src/architecture_doc.rs`) enforces that in the `clippy` CI job and in the `architecture-doc` prek hook, which runs on every commit and every push, over the whole tree. A new file with no entry fails it. This is the opposite direction to the `lychee` check below, which only confirms that links already in the doc still resolve. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. `EXEMPT` in `xtask/src/architecture_doc.rs` is empty and its size is capped at zero by `EXEMPT_CEILING` in the same file, so adding to it is a visible, reviewed exception rather than a way to make the check pass. The check does not read the prose of an entry, so keeping a description true when a file's role changes is still up to you. Every local (relative-path) Markdown link and `#anchor` fragment in the repository (outside `docs/rocm-docs/`, which `docs-build` covers instead, and hidden directories such as `.github/`, which neither check walks) is checked by the `docs-links` CI job and by the `lychee` prek hook above, both configured by `lychee.toml` (which lists the current exclusions). Both run offline only, so they don't catch broken external `https://` links. diff --git a/Cargo.lock b/Cargo.lock index dc61a39a04..09d0937413 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3780,7 +3780,6 @@ dependencies = [ "bollard", "chrono", "csv", - "regex", "reqwest 0.12.28", "rocm-core", "rocm-dash-core", diff --git a/crates/rocm-dash-collectors/Cargo.toml b/crates/rocm-dash-collectors/Cargo.toml index f8e7a826eb..91add936f4 100644 --- a/crates/rocm-dash-collectors/Cargo.toml +++ b/crates/rocm-dash-collectors/Cargo.toml @@ -16,7 +16,6 @@ rocm-core = { path = "../rocm-core" } serde = { version = "1", features = ["derive"] } serde_json = "1" csv = "1" -regex = "1" thiserror = "2" chrono = { version = "0.4", features = ["serde"] } tracing = "0.1" diff --git a/crates/rocm-dash-collectors/src/lib.rs b/crates/rocm-dash-collectors/src/lib.rs index a58cdf851a..b4c0047e02 100644 --- a/crates/rocm-dash-collectors/src/lib.rs +++ b/crates/rocm-dash-collectors/src/lib.rs @@ -5,7 +5,6 @@ //! IO-heavy collector implementations. Used by `rocm-dash-daemon` and, on Windows, `rocm-dash-tui`. //! //! Every collector implements one of the traits in `rocm_dash_core::traits`. -//! Stubs return `CollectorError::Unsupported` so the daemon can start with nothing wired. pub mod amd_smi; pub mod bench_load; @@ -16,8 +15,5 @@ pub mod engine_registry; pub mod host; pub mod lemonade; pub mod parallel; -pub mod proc_scan; -pub mod sysfs; -pub mod vllm_log; pub mod vllm_prom; pub mod winperf; diff --git a/crates/rocm-dash-collectors/src/proc_scan.rs b/crates/rocm-dash-collectors/src/proc_scan.rs deleted file mode 100644 index ea4c4053fd..0000000000 --- a/crates/rocm-dash-collectors/src/proc_scan.rs +++ /dev/null @@ -1,27 +0,0 @@ -// Copyright © Advanced Micro Devices, Inc., or its affiliates. -// -// SPDX-License-Identifier: MIT - -//! /proc scan for non-container deployments (Strix Halo). Stub. - -use rocm_dash_core::traits::{CollectorError, DiscoveredService, Result, ServiceDiscovery}; - -#[derive(Debug, Default)] -pub struct ProcDiscovery; - -impl ProcDiscovery { - pub const fn new() -> Self { - Self - } -} - -impl ServiceDiscovery for ProcDiscovery { - fn name(&self) -> &'static str { - "proc" - } - - fn discover(&self) -> Result> { - // TODO: walk /proc, match comm == "llama-server" | "lemonade-server". - Err(CollectorError::Unsupported("proc discovery stub".into())) - } -} diff --git a/crates/rocm-dash-collectors/src/sysfs.rs b/crates/rocm-dash-collectors/src/sysfs.rs deleted file mode 100644 index d6381fbd17..0000000000 --- a/crates/rocm-dash-collectors/src/sysfs.rs +++ /dev/null @@ -1,39 +0,0 @@ -// Copyright © Advanced Micro Devices, Inc., or its affiliates. -// -// SPDX-License-Identifier: MIT - -//! sysfs/hwmon collector for Strix Halo (gfx1151). Stub. - -use rocm_dash_core::metrics::{GpuMetrics, GpuSystemInfo}; -use rocm_dash_core::traits::{CollectorError, GpuCollector, GpuDevice, GpuProcess, Result}; - -#[derive(Debug, Default)] -pub struct SysfsGpuCollector; - -impl SysfsGpuCollector { - pub const fn new() -> Self { - Self - } -} - -impl GpuCollector for SysfsGpuCollector { - fn name(&self) -> &'static str { - "sysfs" - } - - fn devices(&self) -> Result> { - Err(CollectorError::Unsupported("sysfs collector stub".into())) - } - - fn metrics(&self) -> Result> { - Err(CollectorError::Unsupported("sysfs collector stub".into())) - } - - fn system_info(&self) -> Result { - Err(CollectorError::Unsupported("sysfs collector stub".into())) - } - - fn processes(&self) -> Result> { - Err(CollectorError::Unsupported("sysfs collector stub".into())) - } -} diff --git a/crates/rocm-dash-collectors/src/vllm_log.rs b/crates/rocm-dash-collectors/src/vllm_log.rs deleted file mode 100644 index a501d72fb0..0000000000 --- a/crates/rocm-dash-collectors/src/vllm_log.rs +++ /dev/null @@ -1,106 +0,0 @@ -// Copyright © Advanced Micro Devices, Inc., or its affiliates. -// -// SPDX-License-Identifier: MIT - -//! vLLM log slicer — verbatim port of `inspect_bench/log_slicer.py`. -//! See `../wiki/entities/log-slicer.md` and `../wiki/concepts/log-derived-metrics.md`. - -use regex::Regex; - -#[derive(Debug, Clone, Default, PartialEq)] -pub struct VllmPeaks { - pub prompt_tps: Option, - pub gen_tps: Option, - pub max_running_reqs: Option, - pub max_waiting_reqs: Option, - pub n_requests: u32, -} - -#[allow(clippy::struct_field_names)] // `_re` suffix is meaningful (these are Regex fields) -pub struct VllmLogSlicer { - prompt_tps_re: Regex, - gen_tps_re: Regex, - running_re: Regex, - waiting_re: Regex, - req_re: Regex, -} - -impl Default for VllmLogSlicer { - fn default() -> Self { - Self::new() - } -} - -impl VllmLogSlicer { - pub fn new() -> Self { - Self { - prompt_tps_re: Regex::new(r"Avg prompt throughput: ([0-9.]+)").unwrap(), - gen_tps_re: Regex::new(r"Avg generation throughput: ([0-9.]+)").unwrap(), - running_re: Regex::new(r"Running: ([0-9]+)").unwrap(), - waiting_re: Regex::new(r"Waiting: ([0-9]+)").unwrap(), - req_re: Regex::new(r"POST /v1/(chat/completions|completions|messages)").unwrap(), - } - } - - /// Aggregate peaks over a byte-bounded slice of the vLLM log. - pub fn parse_slice(&self, text: &str) -> VllmPeaks { - let mut out = VllmPeaks::default(); - for cap in self.prompt_tps_re.captures_iter(text) { - if let Ok(v) = cap[1].parse::() { - out.prompt_tps = Some(out.prompt_tps.map_or(v, |p| p.max(v))); - } - } - for cap in self.gen_tps_re.captures_iter(text) { - if let Ok(v) = cap[1].parse::() { - out.gen_tps = Some(out.gen_tps.map_or(v, |p| p.max(v))); - } - } - for cap in self.running_re.captures_iter(text) { - if let Ok(v) = cap[1].parse::() { - out.max_running_reqs = Some(out.max_running_reqs.map_or(v, |p| p.max(v))); - } - } - for cap in self.waiting_re.captures_iter(text) { - if let Ok(v) = cap[1].parse::() { - out.max_waiting_reqs = Some(out.max_waiting_reqs.map_or(v, |p| p.max(v))); - } - } - out.n_requests = self.req_re.find_iter(text).count() as u32; - out - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn parses_throughput_peaks_and_request_count() { - let log = r" -[2026-05-26 10:00:00] Avg prompt throughput: 1234.5 tokens/s -[2026-05-26 10:00:01] Avg generation throughput: 67.8 tokens/s -[2026-05-26 10:00:02] Running: 8, Waiting: 2 -[2026-05-26 10:00:03] Avg prompt throughput: 2000.1 tokens/s -[2026-05-26 10:00:03] Avg generation throughput: 50.0 tokens/s -[2026-05-26 10:00:04] Running: 16, Waiting: 0 -INFO POST /v1/chat/completions 200 -INFO POST /v1/completions 200 -INFO POST /v1/messages 200 -INFO POST /v1/chat/completions 200 -"; - let slicer = VllmLogSlicer::new(); - let peaks = slicer.parse_slice(log); - assert_eq!(peaks.prompt_tps, Some(2000.1)); - assert_eq!(peaks.gen_tps, Some(67.8)); - assert_eq!(peaks.max_running_reqs, Some(16)); - assert_eq!(peaks.max_waiting_reqs, Some(2)); - assert_eq!(peaks.n_requests, 4); - } - - #[test] - fn empty_log_is_all_none() { - let slicer = VllmLogSlicer::new(); - let peaks = slicer.parse_slice(""); - assert_eq!(peaks, VllmPeaks::default()); - } -} diff --git a/crates/rocm-dash-tui/src/ui/bench.rs b/crates/rocm-dash-tui/src/ui/bench.rs deleted file mode 100644 index 23372f5ad5..0000000000 --- a/crates/rocm-dash-tui/src/ui/bench.rs +++ /dev/null @@ -1,5 +0,0 @@ -// Copyright © Advanced Micro Devices, Inc., or its affiliates. -// -// SPDX-License-Identifier: MIT - -//! Benchmark result table + filters. Stub. diff --git a/crates/rocm-dash-tui/src/ui/instance_list.rs b/crates/rocm-dash-tui/src/ui/instance_list.rs deleted file mode 100644 index 4f5e55f492..0000000000 --- a/crates/rocm-dash-tui/src/ui/instance_list.rs +++ /dev/null @@ -1,5 +0,0 @@ -// Copyright © Advanced Micro Devices, Inc., or its affiliates. -// -// SPDX-License-Identifier: MIT - -//! j/k-navigable instance list with status badges. Stub. diff --git a/crates/rocm-dash-tui/src/ui/mod.rs b/crates/rocm-dash-tui/src/ui/mod.rs index 8604392cc3..d3fd972891 100644 --- a/crates/rocm-dash-tui/src/ui/mod.rs +++ b/crates/rocm-dash-tui/src/ui/mod.rs @@ -4,7 +4,6 @@ pub mod approval; pub mod automations_manager; -pub mod bench; pub mod bench_run; pub mod command_screen; pub mod config_manager; @@ -21,7 +20,6 @@ mod header; mod help_modal; pub mod install_manager; mod instance_detail; -pub mod instance_list; pub mod job_console; pub mod launcher; pub mod logs_view; diff --git a/docs/architecture.md b/docs/architecture.md index c1e2af33af..0a0ba71bf7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,7 +10,7 @@ This is the living module map for rocm-cli. It's a contributor-facing reference > Before relying on any entry below, verify current file and function boundaries directly (e.g. `grep`) rather than trusting this doc's wording. Module boundaries shift as the codebase grows, and a stale-but-plausible-looking note is worse than an explicit prompt to check. Every concrete file citation in the Module map's per-crate inventories below — except "the old agent.rs" in the `rocm-dash-tui` entry (historical prose about a deleted file) and `report.json` in the `crates/e2e-report` entry (cucumber's untracked generated output) — is a relative markdown link rather than a bare filename, checked offline by [lychee](https://github.com/lycheeverse/lychee) in CI (the `docs-links` job, config in `lychee.toml`) — a moved or deleted file fails CI immediately, naming the broken link. That check doesn't check that the surrounding prose (e.g. which extraction pattern a module follows) is still accurate; the caution above still applies to everything but link resolution. -A second check runs the other way: `cargo xtask check-architecture-doc` ([`architecture_doc.rs`](../xtask/src/architecture_doc.rs)) fails when a production source file — a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` — has no link in this document, so a new module cannot land without an entry. Only a link that renders as one counts: a path inside an HTML comment, a code block or a code span does not, and neither does a malformed link (one missing its `[` or `)`, or with an escaped bracket), an image, or the outer brackets of a link that contains another link. Write the entry as an inline `[text](path)` link with its text on one line; `` delimiters around the destination are accepted, but reference-style links, a `?query` suffix and root-relative paths are not. `build.rs` files and the `tests/` and `examples/` trees are out of scope. It runs in the `clippy` CI job and in the `architecture-doc` prek hook, at commit and at push. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. The files that were uncited when the check was added, mostly in the dashboard crates, are listed in its `EXEMPT` constant; documenting one means removing it from that list in the same change. +A second check runs the other way: `cargo xtask check-architecture-doc` ([`architecture_doc.rs`](../xtask/src/architecture_doc.rs)) fails when a production source file — a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` — has no link in this document, so a new module cannot land without an entry. Only a link that renders as one counts: a path inside an HTML comment, a code block or a code span does not, and neither does a malformed link (one missing its `[` or `)`, or with an escaped bracket), an image, or the outer brackets of a link that contains another link. Write the entry as an inline `[text](path)` link with its text on one line; `` delimiters around the destination are accepted, but reference-style links, a `?query` suffix and root-relative paths are not. `build.rs` files and the `tests/` and `examples/` trees are out of scope. It runs in the `clippy` CI job and in the `architecture-doc` prek hook, at commit and at push. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. Every production file has an entry, so the check's `EXEMPT` list is empty and capped at zero; adding a file to it to make the check pass is not the way, and growing it is a visible, reviewed exception. ## Module organization convention @@ -27,7 +27,7 @@ A flat size limit backs this convention: `cargo xtask check-module-size` ([`modu Scoped to the crates that make up the shipped CLI/daemon/dashboard/engine surface, plus `crates/e2e-report` (a shared exception: it's HTML/markdown reporting consumed only by `xtask` and `tests/e2e-cucumber`, but it's still one of the modularization effort's target files, so it's mapped below). Dev-tooling and test-harness workspace members (`xtask`, `tests/e2e-cucumber` themselves) are otherwise out of scope — they're not part of the modularization effort's inventory. -Each per-crate module inventory lists one file per line. Flat inventories (every one below except `rocm-dash-tui`'s directory-split `agent/` and `app/` lists, `e2e-report`'s directory-split `consolidated/` list and the `engines/aim` list) are sorted alphabetically by filename, so a newly-extracted module has a predictable insertion point and doesn't force a rewrite of its neighbors; the `agent/`, `app/` and `consolidated/` lists instead keep `mod.rs` first, then the other files in logical (not alphabetical) order. +Each per-crate module inventory lists one file per line. Flat inventories (every one below except `rocm-dash-tui`'s directory-split `agent/`, `app/` and `ui/tabs/` lists, `e2e-report`'s directory-split `consolidated/` list and the `engines/aim` list) are sorted alphabetically by filename, so a newly-extracted module has a predictable insertion point and doesn't force a rewrite of its neighbors; the `agent/`, `app/` and `consolidated/` lists instead keep `mod.rs` first, then the other files in logical (not alphabetical) order, and the `ui/tabs/` list keeps `mod.rs` first and then sorts the tabs alphabetically. ### `apps/rocm` — main CLI binary @@ -147,6 +147,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`daemon.rs`](../apps/rocmd/src/daemon.rs) — the foreground `rocmd run` loop (`run_daemon`: evaluate the watchers each tick, dispatch local webhook events, write the runtime state, stop on a signal; with no watcher enabled it exits by itself, deciding that under the autostart lock and publishing "stopped" and clearing its spawn claim before releasing it — `record_daemon_stop`, `release_own_autostart_claim`) and `rocmd status` (`print_status`), with the watcher tick interval, which `LoopHooks` makes injectable for tests. It uses [`watchers.rs`](../apps/rocmd/src/watchers.rs), [`webhook.rs`](../apps/rocmd/src/webhook.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs) calls it. - [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) — talking to an engine and guarding its endpoint key: `engine_request` (the stdio request to an engine process), the healthcheck call and its readiness and recoverability checks, and the endpoint-key helpers `apply_endpoint_key_env` and `ensure_public_service_has_endpoint_key`. It uses no other module of the crate; [`service.rs`](../apps/rocmd/src/service.rs), [`supervise.rs`](../apps/rocmd/src/supervise.rs) and [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) call it. - [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs) — the host snapshot: `build_bridge_snapshot` (the bridge snapshot printed by `rocmd bridge-snapshot` and returned by the `bridge_snapshot` MCP tool, assembled from the examine summary, the GPU snapshot, the automation runtime state and recent events, the engine inventory and the managed services), the amd-smi GPU snapshot (`gather_gpu_snapshot_for_config`, which honours the telemetry setting, over time-bounded amd-smi probes) and `bridge_engine_inventory`. It uses only [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs), [`mcp.rs`](../apps/rocmd/src/mcp.rs) and [`watchers.rs`](../apps/rocmd/src/watchers.rs) call it. +- [`main.rs`](../apps/rocmd/src/main.rs) — the `rocmd` binary's entry point: a synchronous `main` that calls `rocmd::run_bin_cli()`, which builds the Tokio runtime itself (`run_bin_cli` is `#[tokio::main]`) and dispatches the parsed `Cli`, so the binary holds no logic of its own. - [`mcp.rs`](../apps/rocmd/src/mcp.rs) — the MCP stdio server, tool schema table, tool dispatch, and the result-shaping helpers behind the MCP tools (the `rocm` subprocess itself is spawned through [`common.rs`](../apps/rocmd/src/common.rs), via `run_rocm_capture` here). The argument handling the dispatch calls is in [`mcp_argv.rs`](../apps/rocmd/src/mcp_argv.rs). - [`mcp_argv.rs`](../apps/rocmd/src/mcp_argv.rs) — the argument handling for the MCP tools that spawn `rocm`: `normalized_rocm_command_args` and the read-only allowlist `ensure_rocm_command_is_read_only` behind the `rocm_command` tool, and the argv builders for `install_sdk`, `install_engine`, `launch_server` and `watcher_enable` (`build_install_sdk_args`, `build_install_engine_args`, `build_launch_server_args`, `build_watcher_enable_args`, with the public-bind and system-prefix acknowledgement checks they apply). Functions over the tool's JSON arguments: it uses no other module of the crate, spawns nothing and writes nothing, and [`mcp.rs`](../apps/rocmd/src/mcp.rs) is its only caller. Its own `mod tests` holds the allowlist and argv-builder tests. - [`persistence.rs`](../apps/rocmd/src/persistence.rs) — `record_event`/`load_managed_services`/`load_service_record`, the automation-event/audit-log and managed-service-registry I/O shared across the daemon's sandbox, MCP, service-lifecycle, and watcher code. @@ -242,7 +243,7 @@ Further `rocm-core` modules, each with its own doc comment except where noted: ### `crates/rocm-dash-core`, `rocm-dash-collectors`, `rocm-dash-daemon`, `rocm-dash-tui` — dashboard/telemetry -The `agent/` and `app/` files below are every production file the dashboard-TUI split has touched in those two directories (the `ui/` files it has touched are listed after them); [`app/test_support.rs`](../crates/rocm-dash-tui/src/app/test_support.rs) holds only shared test helpers. +The `rocm-dash-tui` lists below cover every production file in the crate: the top-level modules, the `agent/` and `app/` directories, and `ui/` with its `ui/tabs/` subdirectory. [`app/test_support.rs`](../crates/rocm-dash-tui/src/app/test_support.rs) holds only shared test helpers. The crate depends on `rocm-dash-core` and never on `rocm-core` (the edges are listed in [`crate_edges.rs`](../xtask/src/crate_edges.rs); `rocm-dash-collectors` is a dependency on Windows only), so anything that needs `rocm-core` data reaches it through the `tool_exec` and `registry_probe` seams or through values the `rocm` binary plumbs in on `ResolvedArgs`. The old agent.rs was split (mechanical relocation — the shared seam types stay in [`agent/mod.rs`](../crates/rocm-dash-tui/src/agent/mod.rs), reached from the split-out files via `super::`) into: @@ -271,9 +272,21 @@ The old agent.rs was split (mechanical relocation — the shared seam types stay - [`app/slash.rs`](../crates/rocm-dash-tui/src/app/slash.rs) — previously extracted from [`app/mod.rs`](../crates/rocm-dash-tui/src/app/mod.rs). - [`app/summary.rs`](../crates/rocm-dash-tui/src/app/summary.rs) — previously extracted from [`app/mod.rs`](../crates/rocm-dash-tui/src/app/mod.rs). +The crate's remaining top-level modules, besides `agent/`, `app/` and `ui/`: + +- [`client.rs`](../crates/rocm-dash-tui/src/client.rs) — the connection task behind the dashboard's event stream, and `ClientMsg`, the one message type the app loop reads. `spawn` loops forever: it posts `Connecting`, runs `connect_and_run`, posts `Disconnected`, then sleeps for the next delay from [`reconnect.rs`](../crates/rocm-dash-tui/src/reconnect.rs) before trying again. On unix `connect_and_run` takes a `unix:/path` address (any other scheme is rejected), reads the daemon's `Welcome`, sends `Hello` and `Subscribe`, and forwards each event until `Bye` or EOF. On Windows there is no daemon socket, so the `cfg(windows)` arm collects host and GPU metrics in-process once a second through `rocm-dash-collectors` (the only use of that dependency) and posts them as `ClientMsg::HostGpuSnapshot`, which `AppState::push_host_gpu_snapshot` applies without treating its empty instance list as authoritative. Besides daemon events, `ClientMsg` carries the replay position and the TUI-internal chat, slash-command and approval results that [`replay.rs`](../crates/rocm-dash-tui/src/replay.rs) and the `agent/` and `app/` modules post. +- [`jobs.rs`](../crates/rocm-dash-tui/src/jobs.rs) — the one async process runner every operational screen shares. `run_effects` takes the `SideEffect`s the pure reducer in `rocm-dash-core` returns and starts a task for each `SideEffect::SpawnJob`. The task spawns the child with null stdin and piped stdout and stderr (`kill_on_drop`), posts each line as `StateEvent::JobLine` (keeping only the last `\r`-separated segment, so a progress bar that redraws in place reads as one line), and, once both pipes have closed, finishes with `JobDone` (the exit code, -1 when the child has none, as when a signal killed it) or `JobErr` for a wait failure; a spawn failure posts `JobErr` at once. It checks the shared cancel flag after every line and at least every 100 ms (`CANCEL_POLL`) while the child is idle, and kills the child when it flips; a cancelled job posts neither, because the reducer has already marked it `Cancelled`. Events go back over an unbounded channel on purpose: the reducer is synchronous and cannot apply backpressure, so output is bounded in the reducer instead (`JOB_OUTPUT_RING_CAP`). `app/event_loop.rs` owns the channel and folds the events into `AppState::jobs`; the screens in `ui/` start jobs only through `StateEvent::StartJob`. +- [`lib.rs`](../crates/rocm-dash-tui/src/lib.rs) — the library root: declares `agent`, `app`, `client`, `jobs`, `llm`, `reconnect`, `registry_probe`, `replay`, `tool_exec`, `transport` and `ui` as public modules, so `apps/rocm` runs the dashboard through `app::run` and the examples and integration tests drive the same modules in-process. +- [`llm.rs`](../crates/rocm-dash-tui/src/llm.rs) — chat endpoint configuration, with no HTTP client (the Rig-backed agents are in [`agent/clients.rs`](../crates/rocm-dash-tui/src/agent/clients.rs)). `resolve_llm_config` is a pure resolver: the base URL comes from the CLI, else the config, else the environment, else `DEFAULT_CHAT_BASE_URL` only if a TCP probe of it succeeded, and it returns `None` when none applies; the model comes from the CLI, else the config, else `DEFAULT_CHAT_MODEL`. `resolve_llm_config` strips the API key and custom auth header from a loopback base URL (`parse_host_port` handles bracketed IPv6, then `is_loopback_host`) and logs a warning when it discards one, so a configured local endpoint is never sent a gateway credential, and `LlmConfig`'s `Debug` redacts the key. `probe_endpoint(url, timeout)` is a TCP-only liveness check, and its callers pass `PROBE_TIMEOUT` (300 ms) for each resolved address, and `detect_local_endpoint` probes the Lemonade, vLLM and `rocm serve` default endpoints concurrently and returns the highest-priority one that answered; the last port is a literal copy of `rocm-core`'s default because this crate cannot depend on it. `pick_first_model` reads the first id of a `/v1/models` reply, which `apps/rocm`'s `dash.rs` also calls. The chat code in `app/chat_runtime.rs`, `app/chat.rs` and the Chat tab uses it. +- [`reconnect.rs`](../crates/rocm-dash-tui/src/reconnect.rs) — `Backoff`, the exponential delay `client.rs` sleeps between connection attempts: 250 ms, doubling up to 10 s, started over (a new `Backoff::default()`) after a connection that ended cleanly. +- [`registry_probe.rs`](../crates/rocm-dash-tui/src/registry_probe.rs) — the seam through which the dashboard reads the managed-service registry without depending on `rocm-core` or the daemon. `ManagedRegistryProbe::snapshot` returns a `ManagedRegistrySnapshot`: whether any record is live, and the ids `rocm services stop ` can reach. The binary implements it (`LiveManagedRegistryProbe` in `apps/rocm`'s `dash.rs`) and hands it in on `ResolvedArgs`. It is read live, at each quit attempt and each frame the quit-confirm prompt is open, rather than once at launch, so a model started mid-session and a Windows host with no instance snapshot are both seen; implementations must stay cheap (a local file read). It answers from each record's own status, so it can lag `rocm services list`, which also checks process liveness. `StaticManagedRegistrySnapshot` wraps an already-read snapshot for the launcher, which reads the registry once per menu pass. Compare [`tool_exec.rs`](../crates/rocm-dash-tui/src/tool_exec.rs), the same pattern for tool calls. +- [`replay.rs`](../crates/rocm-dash-tui/src/replay.rs) — the offline source behind `--replay`. `spawn` starts a task that posts `Connecting`, reads the whole session file up front (NDJSON `PersistedEntry` lines; a malformed line is skipped with a warning), posts `Connected`, and then posts the recorded events through the same `ClientMsg` channel as `client.rs`, so nothing downstream tells replay from a live daemon; it ends in `Disconnected` (the read error, or "end of recording"). `app/event_loop.rs` starts replay when `--replay` is given and the live client otherwise, except that a focused host without `--replay` starts neither. Between events it sleeps for the recorded gap capped at 2 s (`MAX_GAP_MS`) and then divided by the speed, with a 1 ms floor, so the cap bounds the recorded gap and not the wall-clock wait: at 0.25x a long gap sleeps 8 s. The returned `ReplayController` is what `app/actions.rs` drives: pause, resume, speed (`SPEED_STEPS`, 0.25x to 8x, stepped by `next_speed`/`prev_speed`) and jump. A forward jump emits the events it crosses at once; a backward jump posts `ClientMsg::ReplaySeek`, so the app drops state derived from past events, then re-emits from the start up to the new position. Every emitted event is followed by `ClientMsg::ReplayPosition` for the header's elapsed/total readout. +- [`tool_exec.rs`](../crates/rocm-dash-tui/src/tool_exec.rs) — the seam for chat tool calls that need `rocm-core`, in plain `serde_json` types. `RocmToolExecutor::execute` runs a read-only tool and returns `RocmToolOutcome::Result`, or returns `ApprovalRequired(ApprovalIntent)` for a mutating one without running it, or `Error`. `execute_approved` runs the same `(name, arguments)` only after the operator approves, re-validating first, and blocks, so callers run it off the async loop. The binary implements the trait (`BinToolExecutor` in `apps/rocm`'s `dash_seam.rs`); the dashboard holds it as `SharedRocmToolExecutor`, `None` under demo, replay and mock. The chat tools and slash commands use it; the operational screens in `ui/` do not, they spawn the `rocm` binary themselves through [`jobs.rs`](../crates/rocm-dash-tui/src/jobs.rs). +- [`transport.rs`](../crates/rocm-dash-tui/src/transport.rs) — newline-delimited JSON framing for `client.rs`: `write_line` serializes one value plus a newline and flushes, `read_line` returns `None` at EOF and an `InvalidData` error for a line that is not valid JSON for the requested type. The daemon's [`transport.rs`](../crates/rocm-dash-daemon/src/transport.rs) holds the same two functions, copied rather than shared so this crate does not depend on the daemon; the two sides serialize the `rocm-dash-core` protocol types. + [`crates/rocm-dash-tui/src/ui/approval.rs`](../crates/rocm-dash-tui/src/ui/approval.rs) is the shared component for approval-state prompts — reuse it rather than hand-rolling new approval UI. -The other shared `ui/` components the contributor rules in [`AGENTS.md`](../AGENTS.md) name — reuse these rather than hand-rolling a new console, box, spinner or status color (the directory holds further helper and per-screen modules that are not inventoried here): +The other shared `ui/` components the contributor rules in [`AGENTS.md`](../AGENTS.md) name — reuse these rather than hand-rolling a new console, box, spinner or status color (the overlays, widgets and tabs that use them are listed after the next list): - [`ui/job_console.rs`](../crates/rocm-dash-tui/src/ui/job_console.rs) — the shared "a job is running" surface: `draw_job_console` renders a job's status header, streamed output and key hints, and `on_console_key` handles its keys. - [`ui/panel.rs`](../crates/rocm-dash-tui/src/ui/panel.rs) — the shared box chrome: bordered, titled panels (`bento`, `popup`), scrollbars, and the popup geometry and framing (`centered_rect`, `centered_width`, `draw_popup_frame`, `grey_overlay`) that [`ui/approval.rs`](../crates/rocm-dash-tui/src/ui/approval.rs), [`ui/job_console.rs`](../crates/rocm-dash-tui/src/ui/job_console.rs) and the `Modal::*` renderers ([`ui/help_modal.rs`](../crates/rocm-dash-tui/src/ui/help_modal.rs), [`ui/warnings_modal.rs`](../crates/rocm-dash-tui/src/ui/warnings_modal.rs), [`ui/theme_picker.rs`](../crates/rocm-dash-tui/src/ui/theme_picker.rs) and [`ui/menu.rs`](../crates/rocm-dash-tui/src/ui/menu.rs)) import. @@ -294,11 +307,76 @@ The `ui/` modules the split moved code out of or into, beyond the shared compone - [`ui/warnings_modal.rs`](../crates/rocm-dash-tui/src/ui/warnings_modal.rs) — the Warnings modal (`draw_warnings`: the text behind the header warning badge, with each multi-line warning cut to a few lines), split out of [`ui/modal.rs`](../crates/rocm-dash-tui/src/ui/modal.rs). It imports `popup_height_for` from [`ui/help_modal.rs`](../crates/rocm-dash-tui/src/ui/help_modal.rs), which does not import it back. - [`ui/theme_picker.rs`](../crates/rocm-dash-tui/src/ui/theme_picker.rs) — the theme picker modal (`draw_theme_picker`: a scrollable list of the registered themes with a five-color swatch per entry, beside `draw_theme_preview`, a live preview of the highlighted theme), split out of [`ui/modal.rs`](../crates/rocm-dash-tui/src/ui/modal.rs). It imports the popup geometry from [`ui/panel.rs`](../crates/rocm-dash-tui/src/ui/panel.rs) and does not depend on `modal.rs`. -The other three dashboard crates are described at crate level here, not module by module: +Thirteen operational overlays share one lifecycle. Each has a state struct held as an `Option` on `AppState` (`None` is closed) and is opened by a `KeyAction::Open*` arm in [`app/actions.rs`](../crates/rocm-dash-tui/src/app/actions.rs), a slash command ([`app/slash.rs`](../crates/rocm-dash-tui/src/app/slash.rs)), the quit-confirm prompt's `s` key ([`app/quit_confirm.rs`](../crates/rocm-dash-tui/src/app/quit_confirm.rs)) or the startup focus ([`app/focus.rs`](../crates/rocm-dash-tui/src/app/focus.rs)), each of which first calls `close_overlays`, so at most one is open. `draw_active_manager` in [`ui/mod.rs`](../crates/rocm-dash-tui/src/ui/mod.rs) draws it into a centered card over a dimmed backdrop and shows the shared [job console](../crates/rocm-dash-tui/src/ui/job_console.rs) instead once the overlay's `active_job` is set. A screen's `on_key` mutates its own state and the shared job `State` and returns the reducer's `SideEffect`s, which `app/event_loop.rs` runs through [`jobs.rs`](../crates/rocm-dash-tui/src/jobs.rs). A mutating action stages an `ApprovalRequest` held in the screen's own state and shown by [`ui/approval.rs`](../crates/rocm-dash-tui/src/ui/approval.rs); the `rocm` path is resolved with `resolve_exe` when it is staged, the job starts only on Approve, and the CLI does the mutation, so the dashboard changes nothing itself. Read-only actions start their job directly. That is a separate path from a chat tool call's approval (`AppState::approval`, [`app/tool_approval.rs`](../crates/rocm-dash-tui/src/app/tool_approval.rs)). Exceptions are noted per screen. + +- [`ui/automations_manager.rs`](../crates/rocm-dash-tui/src/ui/automations_manager.rs) — lists the background checks from `AppState::automations` (`AutomationSummary`, plumbed in by the binary: id, summary, enabled, mode), refreshes with `rocm automations list`, and enables or disables the selected one through an approval. +- [`ui/bench_run.rs`](../crates/rocm-dash-tui/src/ui/bench_run.rs) — the form behind `rocm bench load`: endpoint (required; `https://` is rejected), optional model, a concurrency list or an auto-ramp toggle, and an output path that defaults to the daemon-tailed CSV (`bench_results_dir`) when there is one, so rows appear live in the Bench panel. Unlike the other overlays, submitting starts job `bench-run` directly, with no approval step, and closes the form. +- [`ui/command_screen.rs`](../crates/rocm-dash-tui/src/ui/command_screen.rs) — a freeform `rocm ...` runner. The input is split on whitespace (no quoting), and every command goes through approval showing the exact argv, since arbitrary subcommands cannot be classified as read-only; an argv containing key, token, secret or password adds a warning that arguments are visible in the process list and job log. +- [`ui/config_manager.rs`](../crates/rocm-dash-tui/src/ui/config_manager.rs) — `rocm config show` (read-only) plus approval-gated `rocm config enable-provider|disable-provider` for the fixed provider list (`PROVIDERS`: local, anthropic, openai). It has no key-entry field: provider keys come from the environment only. +- [`ui/engine_manager.rs`](../crates/rocm-dash-tui/src/ui/engine_manager.rs) — a fixed two-engine catalog (`ENGINE_CATALOG`: Lemonade and vLLM) with three approval-gated actions per engine: `rocm config set-default-engine `, `rocm engines install ` and the same with `--reinstall`. Job ids are `engine--`. +- [`ui/examine_manager.rs`](../crates/rocm-dash-tui/src/ui/examine_manager.rs) — runs `rocm examine`, read-only and streaming, with no approval; `open_running` starts it on open for the launcher's Diagnose host. It also prepares the Doctor report: `p` runs `rocm diagnose --report --json` (a preview that sends nothing) and `s` stages an approval whose body is the report, defaulting to Deny, after which `rocm diagnose --report --send` runs. Only a report whose output parsed as a JSON object with the `schema` number this build knows (`KNOWN_REPORT_SCHEMA`) can be sent; anything else is reported as unreadable and nothing is offered. That check looks only at the envelope because this crate cannot use `rocm-core`'s reader. +- [`ui/install_manager.rs`](../crates/rocm-dash-tui/src/ui/install_manager.rs) — the form behind `rocm install sdk`: channel, format (`wheel` or `tarball`), install prefix (typed or picked in the folder browser, then trimmed) and a dry-run toggle that is on by default. A dry run starts directly as job `install-sdk-dryrun`; a real install is approval-gated as `install-sdk` and adds `--approve-replacing-active-default` (not `--yes`, which would also approve `sudo` the dashboard's null-stdin child could not answer). +- [`ui/logs_view.rs`](../crates/rocm-dash-tui/src/ui/logs_view.rs) — `rocm logs`, optionally with `--search` terms from a search box; read-only, so no approval. A single job id means reopening re-attaches to the same console. +- [`ui/onboarding.rs`](../crates/rocm-dash-tui/src/ui/onboarding.rs) — the first-run flow, opened by `KeyAction::OpenOnboarding` (the `n` key on the Observe tab) and as the host for `Focus::Setup`; nothing auto-opens it at startup. Two paths, both approval-gated: install the ROCm SDK (a Configure step for the channel, Release or Nightly, an optional `--build-date` or `--version` pin and an optional prefix chosen in the folder browser, always `--format wheel` and `--approve-replacing-active-default`), or adopt an existing ROCm folder with the same `adopt_args` the runtime manager uses. +- [`ui/runtime_manager.rs`](../crates/rocm-dash-tui/src/ui/runtime_manager.rs) — lists the registered runtimes from `AppState::runtimes` (`RuntimeSummary`, plumbed in by the binary; the active default and the rollback target carry markers) and runs the `rocm runtimes` verbs: a read-only `list` refresh, and approval-gated activate, rollback, uninstall (which adds `--yes`), adopt (folder browser, argv from `adopt_args` in [`exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs)) and import (typed manifest path). One job id per verb. +- [`ui/serve_wizard.rs`](../crates/rocm-dash-tui/src/ui/serve_wizard.rs) — the Serve form. The default form is Model, an `Advanced settings` row and Launch; engine (`ENGINES`), host, port and managed/foreground sit behind the row, so the default invocation is exactly `rocm serve MODEL` and the CLI stays the authority for engine, GPU-required policy, host, port and managed mode. The model is typed, picked from [`model_picker.rs`](../crates/rocm-dash-tui/src/ui/model_picker.rs) (which fills only the id) or chosen as a local path in [`folder_browser.rs`](../crates/rocm-dash-tui/src/ui/folder_browser.rs). The host is limited to the loopback spellings in `LOOPBACK_HOSTS` and a custom host needs a custom port; a public bind is refused with a message pointing at the CLI's `--allow-public-bind`. Launch stages a `PendingServe` approval, then starts job `serve-`; a second launch of a model whose job is still running is refused with a message (the job id replaces every non-alphanumeric character in the model name with `-`, so names that differ only in punctuation share one). Unmanaged launches add `--foreground`. Its `is_loopback_host` accepts only the exact spellings in `LOOPBACK_HOSTS`, which is not the same function as the one in [`llm.rs`](../crates/rocm-dash-tui/src/llm.rs): that one decides whether to strip gateway credentials and also accepts any `127.*` address. +- [`ui/services_manager.rs`](../crates/rocm-dash-tui/src/ui/services_manager.rs) — lists the instances the daemon surfaces (`service_rows`, sorted by id: model, port, status, live generation throughput with a held-value marker) and stops or restarts one through an approval that runs `rocm services stop|restart --yes` as job `svc--`. It also shows a one-line note, from `AppState::services_past_attempts`, that records of servers no longer running exist and `rocm services list --all` shows them. +- [`ui/update_manager.rs`](../crates/rocm-dash-tui/src/ui/update_manager.rs) — wraps `rocm update` in four actions: check and `--apply --dry-run` preview start directly; `--apply` and `--apply --activate` go through approval. One job id per action. + +The remaining `ui/` files that are not overlays or tabs: + +- [`ui/core_bars.rs`](../crates/rocm-dash-tui/src/ui/core_bars.rs) — `CoreBars`, one braille column per CPU core filled from the bottom by utilization; cores that do not fit are dropped from the right edge. Used only by the Hardware panel. +- [`ui/dock.rs`](../crates/rocm-dash-tui/src/ui/dock.rs) — the rails of the wide layout, used when the terminal is at least `WIDE_COLS` x `WIDE_ROWS` (180 x 45; below that the dashboard stays single-column). `gpu_wall` is the left rail (`RAIL_W` 40): one instrument card per GPU from the latest snapshot. `draw_right_dock` is the right rail (`DOCK_W` 52): `logs_dock` (the recent output of every job in `AppState::jobs`, sorted by job id, scrolled by `dock_logs_scroll`) on the Observe, ROCm and Serving tabs, and `context_rail` (running services, GPU state, recent tools) on Home and Chat. Both rails are read-only; `ui/mod.rs`'s `wide_triptych` places them. +- [`ui/folder_browser.rs`](../crates/rocm-dash-tui/src/ui/folder_browser.rs) — the directory picker shared by the serve, install, runtime and onboarding screens: entries for use-current, new child, parent and sub-directories, with only `build_entries` touching the filesystem (an unreadable directory yields no children), and a `FolderOutcome` of chosen path or cancelled. +- [`ui/format.rs`](../crates/rocm-dash-tui/src/ui/format.rs) — pure value formatters every tab uses: memory in MiB promoted to GiB and TiB (`mib`, `mib_pair`, which never prints `1024.0`), `pct`, `si`, `bps`, `watts`, `celsius`, `mhz`, `duration`, and `-` or an em dash for missing values. The `*_cell` and `*_compact` variants (`gen_tps_cell`, `latency_cell`, `tokens_per_watt_cell`, ...) take the observation metadata and append `HELD_MARKER` to a value held over from a prior scrape window, with `HELD_LEGEND` for the footer. It also holds small shared list helpers: `running_instances` (serving first, then model and container id), `stage_text` and `overflow_line` (the "...and N more" line). +- [`ui/gradient.rs`](../crates/rocm-dash-tui/src/ui/gradient.rs) — `GradientGauge`, a horizontal gauge whose filled cells take their background from a three-stop ramp (ok, warn, err by default) across the bar's width, with the fill length tracking the ratio; and `lerp3_t`, the three-stop interpolation that [`sparkline.rs`](../crates/rocm-dash-tui/src/ui/sparkline.rs), [`core_bars.rs`](../crates/rocm-dash-tui/src/ui/core_bars.rs) and the menu's logo reuse. +- [`ui/launcher.rs`](../crates/rocm-dash-tui/src/ui/launcher.rs) — the bare-`rocm` front door, a synchronous screen with its own blocking key-read loop, not the dashboard's async event loop. `run_launcher` takes the running instances the binary read from the registry, shows a status strip over the menu (`ROWS`: set up, serve, diagnose and fix, chat, open the full dashboard, plus a display-only "Optimize a model" row with a `soon` badge) and returns a `LauncherChoice`, or `None` on quit. `apps/rocm`'s `dash.rs` (`launcher_route`) turns the choice into a focused host (Set up, Serve and Diagnose each open one overlay through `ui::draw_focused` and `app/focus.rs`), the chat, or the full dashboard. Quit goes through the same quit-confirm gate as the dashboard, with a `StaticManagedRegistrySnapshot` built from the instances it was given. +- [`ui/model_picker.rs`](../crates/rocm-dash-tui/src/ui/model_picker.rs) — a filterable list of model recipes for the Serve form: case-insensitive substring match over id and aliases, and a `PickerOutcome` of chosen, cancelled or none. `ModelRecipeSummary` is a flattened copy of the recipe fields it shows, built by `apps/rocm`'s `dash.rs` so the crate needs no `rocm-core`. It displays each recipe's `preferred_engine`; choosing one fills only the model id. +- [`ui/sparkline.rs`](../crates/rocm-dash-tui/src/ui/sparkline.rs) — `BrailleSparkline`, a history graph that packs two samples into each character column at four dot rows per cell, right-aligned so the newest samples show, optionally colored per column by the larger of its two samples. Used by Home, Observe, Hardware, Bench, the dock and the theme preview. + +The five tabs are drawn by `ui/mod.rs`'s `draw` into the folder-tab panel from [`tabs/mod.rs`](../crates/rocm-dash-tui/src/ui/tabs/mod.rs); the tabs only read `AppState` while drawing, except the Chat tab, which writes its measured scroll bounds back (`chat_max_scroll`, `chat_scroll`, `chat_follow`). The tabs list keeps `mod.rs` first, then the tabs alphabetically: + +- [`ui/tabs/mod.rs`](../crates/rocm-dash-tui/src/ui/tabs/mod.rs) — declares the tab modules and `TAB_LABELS` (Home 1, ROCm 2, Serving 3, Observe 4, Chat 5), and draws the outlined folder-tab panel (`draw_tab_panel`). `outlined_chip_spans` is the single source of tab geometry: the painter and `compute_chip_layout`, which `app/actions.rs` uses to turn a click into a tab, both go through it so they cannot drift. +- [`ui/tabs/bench.rs`](../crates/rocm-dash-tui/src/ui/tabs/bench.rs) — the Observe tab's benchmark panel: Pass^N and Pass@N rollups (`rollup_pass_n` from `rocm-dash-core`), a scrollable table of `AppState::bench_rows` with a pass/fail verdict per row, and a sparkline. With no rows it shows the `rocm bench load` command and the key that opens the bench-run form. +- [`ui/tabs/chat.rs`](../crates/rocm-dash-tui/src/ui/tabs/chat.rs) — the Chat tab's view of state the `app/` modules own. Until `chat_consent` is `Accepted` it shows the consent gate (the detected or configured endpoint, and the key to detect a local engine); afterwards a transcript that follows the tail until the user scrolls away (`ChatViewport`) over a three-row input line. While a reply is pending the input shows a static waiting line (a braille glyph with the wait reason, or "waiting for the agent"), and when the endpoint is a loopback address that matches a known instance (`chat_backend_wait_reason`, using `llm.rs`'s parsing) it says whether that model is still starting, stopped or in error. +- [`ui/tabs/hardware.rs`](../crates/rocm-dash-tui/src/ui/tabs/hardware.rs) — the Observe tab's host and GPU panel: CPU (aggregate sparkline plus per-core bars), memory and swap gauges, a disk and network throughput row, then one block per GPU. `plan_gpu_rows` chooses full bordered panels when every GPU fits at `FULL_PANEL_MIN_H` rows, otherwise one compact line per GPU, which scrolls (`scroll_to_show`) and reserves an overflow row only when the GPUs outnumber the rows. Shows "waiting for first snapshot" until a snapshot arrives. +- [`ui/tabs/home.rs`](../crates/rocm-dash-tui/src/ui/tabs/home.rs) — the landing view, three bands: a hero GPU gauge and sparkline with a VRAM, temperature and power cluster; Running, Health and Updates tiles (Updates reads the background update check's `UpdateStatus`); and an Activity feed of running services and recent jobs. Missing telemetry shows a placeholder with a hint at what would produce it, never a synthetic number, and the hero title keeps "Detecting GPU..." and "Unknown GPU" distinct. +- [`ui/tabs/observe.rs`](../crates/rocm-dash-tui/src/ui/tabs/observe.rs) — the serving-telemetry tab, stacked top to bottom: a hero band (node efficiency in tokens per watt with a trend, and node throughput with total power), the per-instance table (`tabs/instances.rs`), the Hardware panel and the Bench panel. A one-line banner shows when the daemon connection or the first snapshot is missing; simulated sessions are marked by the header chip, not here. +- [`ui/tabs/pane.rs`](../crates/rocm-dash-tui/src/ui/tabs/pane.rs) — the shared renderer for the ROCm and Serving tabs: an Actions list (46%) beside a `Details: ` pane (54%), from a `Verb` table (icon, label, `KeyAction`, summary, steps, underlying command, read-only flag, optional badge). `draw`, `hit_test` and `actions_rect` share one column split, so clicks resolve against what was painted. `live_lines` adds live data from the latest snapshot and `AppState::runtimes` to the Details pane for the install, engine, serve and services verbs only (for example the ROCm and driver versions, registered runtimes and running instances) and returns nothing for the others. +- [`ui/tabs/rocm.rs`](../crates/rocm-dash-tui/src/ui/tabs/rocm.rs) — the ROCm tab's verb table (`VERBS`): install, check for updates, diagnose and fix, runtimes, command runner, and a display-only Uninstall row (`KeyAction::Nothing`; the CLI does it). Each actionable row maps to a `KeyAction::Open*` for the overlay of the same name; rendering and hit-testing are in `pane.rs`. +- [`ui/tabs/serving.rs`](../crates/rocm-dash-tui/src/ui/tabs/serving.rs) — the Serving tab's verb table: serve a model, engines, running instances and services, providers and keys (the config manager), logs (marked read-only), and a display-only "Optimize a model" row with a `soon` badge. + +The other three dashboard crates each have a crate-level bullet, followed by a bullet for each source file the crate-level bullet does not already name: - [`rocm-dash-collectors`](../crates/rocm-dash-collectors/src/lib.rs) — the IO-heavy collector implementations behind `rocm-dash-core`'s traits (amd-smi, Docker, host metrics, Lemonade and vLLM scrapers, and on Windows the [`winperf.rs`](../crates/rocm-dash-collectors/src/winperf.rs) perf-counter/ADLX GPU collector) plus the benchmark load generator; used by `rocm-dash-daemon` and, on Windows, by `rocm-dash-tui` in-process. + - [`amd_smi.rs`](../crates/rocm-dash-collectors/src/amd_smi.rs) — the amd-smi collector. `AmdSmiCollector::detect_with_binary` returns `Some` only when a pre-flight passes and `amd-smi version` answers in time; its async methods (`metrics`, `power_limits`, `processes`, `system_info`) run `amd-smi ... --json` under a timeout with `kill_on_drop` and pass the output to pure parsers (`parse_metrics`, `parse_power_limits`, `parse_processes`, `parse_system_info`) that tolerate differences between amd-smi releases and return an empty or default value for a shape they do not recognise. The pre-flight exists because amd-smi can block in uninterruptible kernel sleep on a host with no readable `/dev/kfd`: unless the caller says the GPU is already known to be reachable without one (a verdict `apps/rocm` computes once, true only on WSL with a usable GPU, and hands to the daemon through `RunnerOptions`), it runs `rocm_core::device_readable_bounded` on the blocking pool. `detect_with_binary_skipping_device_preflight` exists for tests that point at a fake script. + - [`bench_load.rs`](../crates/rocm-dash-collectors/src/bench_load.rs) — the concurrency-sweep load generator behind `rocm bench load`, reached through `rocm_dash_daemon::bench_load`. `run_cell` sends `LoadSpec.requests` chat-completion requests at one concurrency and returns a `CellReport`: the `BenchmarkRow` plus attempted, succeeded and failed counts, the first error and an optional `EngineSplitNotice`. `run_and_append_csv` and `run_auto_ramp` (which walks `RAMP_SEQUENCE` and stops after the last level, on a throughput plateau against the previous cell, or when the queue is backed up, `should_stop_ramp`) append each cell to a CSV, taking an exclusive file lock for each row, and refuse a file whose first line differs from `CSV_HEADER` when the first row is appended. During a cell it scrapes the endpoint's Prometheus `/metrics` with a short timeout and sets the row's `engine` only when the scrape is recognisably vLLM; because `engine` is a rollup key, appending a row that carries an `engine` to a file that already holds blank-`engine` rows for that cell is reported, once per sweep, through `EngineSplitNotice` for the CLI to print, not printed here. + - [`bench_tail.rs`](../crates/rocm-dash-collectors/src/bench_tail.rs) — `CsvBenchTailer`, the `BenchTailer` the daemon runner uses on the configured benchmark CSV. Each `drain()` re-reads the whole file and returns only the rows past the count it has already returned, resets that count (and so re-emits every row) only when the file now has fewer rows than before, so a rewrite that is not shorter goes undetected, and returns nothing while the file does not exist; a row that fails to parse makes that drain return an error. Re-reading avoids tracking offsets and inodes, which is affordable because benchmark CSVs are small. + - [`cgroup.rs`](../crates/rocm-dash-collectors/src/cgroup.rs) — attributing GPU processes to containers. `parse_container_id_from_cgroup` is a pure scan of `/proc//cgroup` text for the first run of exactly 64 lowercase hex characters (cgroup v1, v2 and containerd layouts), `container_id_for_pid` reads that file (Linux only; a stub returns `None` elsewhere) and `is_container_id` tests the shape. amd-smi reports host PIDs and a tensor-parallel server's workers are descendants of the container's init process, so the join goes through the cgroup and not through PID equality; a managed `rocm serve` service id never has this shape, so such a service can never be a key of the per-container VRAM map. + - [`docker.rs`](../crates/rocm-dash-collectors/src/docker.rs) — `DockerDiscovery`, discovery through `bollard`. `detect` returns `Some` only when the local Docker daemon answers a ping within two seconds; `discover_async` lists running containers whose image matches the glob-style patterns (default `vllm/*` and `rocm/vllm*`, `*` being the only wildcard) and inspects each under a timeout; `parse_container` builds the `DiscoveredService` from the container's environment and command line: GPU ids from `HIP_VISIBLE_DEVICES` (unless it is empty or `all`), else the first non-empty of `AMD_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES` and `CUDA_VISIBLE_DEVICES`, tensor-parallel size, dtype, quantization (falling back to `--dtype`), model name and a published host port taken from an unordered map of bindings (so with several published ports which one is used is arbitrary; default 8000). Docker gives no readiness signal, so a discovered container is `InstanceStatus::Starting` with no phase. The sync `ServiceDiscovery::discover` returns `Unsupported`. + - [`engine_registry.rs`](../crates/rocm-dash-collectors/src/engine_registry.rs) — `EngineKind` (`Vllm`, `Lemonade`) maps an engine to its label, conventional port and sample parser: `label` and `from_label` (case-insensitive), `default_port` (a fallback only; a managed service's port comes from its registry record) and `parse_sample`, which dispatches to `vllm_prom::parse` or `lemonade::parse_stats`. In production only `label` and `from_label` are called: the daemon's `registry.rs` reaches `from_label` through `engine_kind_for` to leave managed records that are not vLLM out of the vLLM scrape, and `bench_load.rs` takes the `vllm` label so the CSV `engine` column cannot drift from the registry's spelling. `default_port` and `parse_sample` are exercised only by tests, since the runner builds its collectors directly. + - [`host.rs`](../crates/rocm-dash-collectors/src/host.rs) — `HostCollector`, CPU (overall and per core), memory and swap from `sysinfo`, sampled once per tick into `SystemMetrics`; the constructor primes the CPU counters because the first reading is meaningless. Disk and network rates are not collected yet and are always reported as zero. It is cross-platform, and `rocm-dash-tui` also runs it in-process on Windows, where there is no daemon socket. + - [`lemonade.rs`](../crates/rocm-dash-collectors/src/lemonade.rs) — the Lemonade Server collector. The parsers are pure: `parse_stats` maps `/api/v1/stats` `tokens_per_second` to `gen_tps` (Lemonade reports a rate, not a counter, and no KV-cache or request-queue figures) and `parse_health_model` reads the loaded model name from `/api/v1/health`; a body that is not a JSON object parses as empty. `LemonadeCollector::discover` builds a `DiscoveredService` (synthetic `lemonade--` id, status `Ready`) only when `/api/v1/health` answers, and `LEMONADE_PORT` (13305) is the fallback port. The daemon runs it only when `RunnerOptions::enable_lemonade` is set; it defaults to false and the embedded daemon that `rocm dash` starts leaves it off. + - [`parallel.rs`](../crates/rocm-dash-collectors/src/parallel.rs) — `parallel_scrape`, which runs an async closure against every target concurrently on a `JoinSet` and returns `(target, result)` pairs in completion order, not input order, leaving out a task that panics; the daemon runner uses it for the per-instance vLLM `/metrics` scrapes. + - [`vllm_prom.rs`](../crates/rocm-dash-collectors/src/vllm_prom.rs) — the vLLM `/metrics` scraper. `VllmPrometheusCollector::fetch_async` GETs `http://:/metrics` for a service's port and `parse` (crate-private; reached through `EngineKind::parse_sample` and by `bench_load.rs`) reads the running and waiting request gauges, KV-cache usage (rescaled from 0..1 to 0..100) and the cumulative generation-token counter and TTFT and TPOT histogram sums and counts, each from the first matching sample line (label sets are not summed). It leaves `gen_tps` empty because vLLM uses the counter path: the runner differences the counters into rates. The sync `InstanceMetrics::fetch` returns `Unsupported`. - [`rocm-dash-core`](../crates/rocm-dash-core/src/lib.rs) — pure types, traits, schemas and the reducer, with no rendering dependencies. [`chat_template.rs`](../crates/rocm-dash-core/src/chat_template.rs) also holds the chat-template defaults a locally served model needs on a chat request (`local_model_should_disable_thinking`), kept here as the leaf crate both front ends already depend on so that `apps/rocm`'s [`providers/wire.rs`](../apps/rocm/src/providers/wire.rs) and the TUI's [`agent/clients.rs`](../crates/rocm-dash-tui/src/agent/clients.rs) cannot drift apart on it. + - [`bench_rollup.rs`](../crates/rocm-dash-core/src/bench_rollup.rs) — Pass^N / Pass@N rollups: `rollup_pass_n` groups rows into trial sets keyed by `(cell, model, engine, tp, dtype, concurrency)` and returns one `PassNRollup` per group, sorted by that key, with a strict verdict (all N trials passed), a lenient one (at least one passed) and mean prompt and generation throughput; `row_verdict` falls back from `pass_fail` to the judge verdict when the rollup is `Unknown`. Because `engine` is in the key, rows with a blank `engine` do not group with rows that have one. It is pure, and the Bench tab ([`ui/tabs/bench.rs`](../crates/rocm-dash-tui/src/ui/tabs/bench.rs)) and the agent's telemetry helpers ([`agent/snapshot.rs`](../crates/rocm-dash-tui/src/agent/snapshot.rs)) recompute it from the rows rather than reading the CSV's `pass_n_of_n` and `pass_at_n` columns. + - [`bench_schema.rs`](../crates/rocm-dash-core/src/bench_schema.rs) — `BenchmarkRow`, the one row type for a benchmark result (identity, platform, backend configuration, request statistics, throughput, latency, quality, rollup and provenance columns), and `PassFail`. Almost every field is an `Option` and the struct is `#[serde(default)]`, so a CSV or a recording that omits columns still deserializes. `CsvBenchTailer` deserializes CSV rows into it, `bench_load.rs` builds it, and the daemon forwards it in `Event::BenchmarkRowsAppended`. + - [`config.rs`](../crates/rocm-dash-core/src/config.rs) — the standalone dashboard configuration read from `/rocm-dash/config.toml` (`default_config_path`; a missing file means defaults and `Config::load` fails only on a read or parse error): `Config` with `DaemonConfig` (listen address, token, tick intervals, benchmark CSV path), `TuiConfig` (connect address, theme, chat endpoint, model and auth-header name) and per-engine `EngineConfig`. It is separate from the unified `rocm` configuration (the `dashboard` section of `rocm-core`'s `RocmCliConfig`, which `rocm dash` reads); in the tree only the TUI's chat-endpoint save (`persist_chat_endpoint` in [`app/chat.rs`](../crates/rocm-dash-tui/src/app/chat.rs)) loads and writes this file. Its default socket path (`$XDG_RUNTIME_DIR`, then `~/.rocm/data/telemetry`, then a user-named directory under the temp dir) is a copy of `rocm-core`'s on purpose, because this crate may have no first-party dependencies (`cargo xtask check-crate-edges` enforces that). + - [`efficiency.rs`](../crates/rocm-dash-core/src/efficiency.rs) — the pure join of throughput and power. `normalize_gpu_id` strips the `gpu-` prefix so the bare indices Docker discovery records in `Instance.gpu_ids` match amd-smi's `gpu-N` device ids; `attribute_power_w` splits each GPU's board power among the instances that occupy it, weighted by per-container VRAM, equally when any occupant has no reading, and yields `None` for an instance none of whose GPUs reported a positive power reading; `tokens_per_watt` divides generation throughput by that attributed power. The daemon runner calls `attribute_power_w`, `tokens_per_watt` and `gpu_ids_overlap`, which it uses to warn when the id join finds nothing. + - [`metrics.rs`](../crates/rocm-dash-core/src/metrics.rs) — the live metric types every other dashboard crate shares: `Snapshot` (one tick: `SystemMetrics`, a `GpuMetrics` per GPU, an optional `GpuSystemInfo`, the `Instance` list and warnings), `Instance` with its `InstanceStatus` (`Starting` carries an optional `StartupPhase`; `is_serving`, `is_starting`, `counts_as_running` and `label` are the one definition of those predicates and chip labels) and `ObservationMetadata`/`ObservationFreshness`, which mark a throughput or latency value as freshly measured or held over from an earlier scrape. The optional fields added after the first recordings (`power_limit_w`, `gen_tps`, `tokens_per_watt`, `power_w`, `ttft_ms`, `tpot_ms` and the three `*_observation` fields) are `#[serde(default)]`, so older recordings still deserialize. + - [`observation.rs`](../crates/rocm-dash-core/src/observation.rs) — how a generation-throughput counter becomes a rate that stays honest. `GenerationObservationTracker` keeps a bounded window of counter samples, computes a rate over `rate_window(instance_tick)` (`clamp(5 x tick, 10 s, 30 s)`), reports the value as fresh or held for up to `observation_validity(instance_tick)` (`clamp(3 x tick, 6 s, 30 s)`), clears on a counter reset or on expiry, and accepts a rate an engine reports directly through `observe_direct`. All time is passed in, so the module reads no clock. The runner keeps one tracker per instance for throughput and one each for TTFT and TPOT; it drops them when a service is retired and calls `invalidate` when a service's port changes under the same id. + - [`partition.rs`](../crates/rocm-dash-core/src/partition.rs) — the `ComputePartitionMode` (`Spx`, `Dpx`, `Qpx`, `Cpx`) and `MemoryPartitionMode` (`Nps1`, `Nps2`, `Nps4`) enums, each defaulting to `Unknown` and serialized in upper case. `GpuSystemInfo` in `metrics.rs` holds them and `amd_smi.rs` parses them from amd-smi output. + - [`persist.rs`](../crates/rocm-dash-core/src/persist.rs) — `PersistedEntry`, one recorded `Event` plus its `ts_us` (microseconds since the epoch): the daemon stamps wall-clock time, the demo generator a fixed synthetic one. The daemon's [`persist.rs`](../crates/rocm-dash-daemon/src/persist.rs) writes one per NDJSON line, the demo generator emits the same format, and the TUI's [`replay.rs`](../crates/rocm-dash-tui/src/replay.rs) reads them back and paces playback by the `ts_us` gaps, each capped at 2 s and then divided by the playback speed. + - [`protocol.rs`](../crates/rocm-dash-core/src/protocol.rs) — the NDJSON wire protocol between the daemon and a TUI client: `Command` (client to daemon: `Hello`, `Subscribe`, `RequestSnapshot`, `RescanInstances`, `Pause`, `Resume`, `Goodbye`) and `Event` (daemon to client: `Welcome`, `Snapshot`, `InstanceDiscovered`, `InstanceGone`, `BenchmarkRowsAppended`, `Warning`, `Error`, `Bye`), both internally tagged with `kind`, and `PROTOCOL_VERSION`, which is bumped when the JSON shape of an existing variant changes. Internal tagging rejects a newtype variant that wraps a sequence, so every `Event` variant is a struct variant or wraps a struct, and a test round-trips each variant to hold that. + - [`state.rs`](../crates/rocm-dash-core/src/state.rs) — the pure reducer, `State::apply(StateEvent) -> Vec`, holding the latest snapshot, a bounded history (`SNAPSHOT_RING_CAP`), instances keyed by container id, bounded benchmark rows (`BENCH_RING_CAP`), the paused flag and the background-job table (`JobState`/`JobStatus`, output bounded by `JOB_OUTPUT_RING_CAP`), with no tokio or ratatui types. `JobStatus::glyph`, `label` and `legend_glyphs` are the one definition of how a job status is drawn. It has two kinds of user: the daemon runner applies `Tick`, `InstanceUpserted`, `InstanceRemoved` and `BenchmarkRows`, discards the returned effects and reads back only the instance map, and the TUI keeps one `State` for all of its jobs, `AppState::jobs`: `app/event_loop.rs` passes `&mut state.jobs` to the key handler of every screen that runs `rocm` commands (for example [`ui/update_manager.rs`](../crates/rocm-dash-tui/src/ui/update_manager.rs) and [`ui/runtime_manager.rs`](../crates/rocm-dash-tui/src/ui/runtime_manager.rs)), applies the job events that come back from [`jobs.rs`](../crates/rocm-dash-tui/src/jobs.rs) to that same instance, and runs the `SideEffect::SpawnJob`s the handlers return through `jobs.rs`. The table outlives any one overlay: closing a screen leaves its jobs in it. Only `SpawnJob` is interpreted: the `Persist` and `Broadcast*` effects are produced but nothing consumes them. + - [`traits.rs`](../crates/rocm-dash-core/src/traits.rs) — the collector-side contracts and the data they exchange: `DiscoveredService` (a found serving process or container), `InstanceSample` (one scrape's KV-cache usage, request counts, cumulative token counter and latency-histogram sums, or a rate the engine reports directly), `GpuProcess`, `CollectorError`, and `merge_instance`, which turns a service, a sample and VRAM figures into an `Instance`, copying a directly reported `gen_tps` and leaving `tokens_per_watt`, `power_w`, TTFT, TPOT and the freshness fields `None` for the runner to fill. The traits are synchronous, and the runner calls only `BenchTailer::drain`, on a concrete `CsvBenchTailer`; the Docker and vLLM-Prometheus implementations of `ServiceDiscovery` and `InstanceMetrics` return `CollectorError::Unsupported`, because their real API is the async inherent methods `discover_async` and `fetch_async`. + - [`vram.rs`](../crates/rocm-dash-core/src/vram.rs) — the pure VRAM attribution maths. `aggregate_process_vram` folds per-process readings into a per-container map through an injected PID-to-container resolver (the runner passes `cgroup::container_id_for_pid`), and `resolve_instance_vram` returns an instance's `(used, total)`: the per-container figure when there is one, otherwise the device-summed used over its GPUs, with `total` always device-summed and `(0, 0)` for an instance with no map entry and no matching GPU. It reuses `efficiency.rs`'s id matching and does no `/proc` or process I/O. - [`rocm-dash-daemon`](../crates/rocm-dash-daemon/src/lib.rs) — the daemon library: [`runner.rs`](../crates/rocm-dash-daemon/src/runner.rs) drives the collectors on tick cadences and broadcasts snapshots, and [`server.rs`](../crates/rocm-dash-daemon/src/server.rs) streams them to TUI clients as NDJSON. [`registry.rs`](../crates/rocm-dash-daemon/src/registry.rs) reads the managed-service records `rocm serve` writes and tells a registry that is confirmed empty (no directory, or no records) from one that could not be read: the run loop retires a managed service that an empty registry lacks, but when a pass cannot read the registry in full (a directory that cannot be listed, or a record file that cannot be opened) it retires nothing: the known services and their data stay, as when Docker discovery errors, the records that did read still add and refresh their services, and the header warns until a read succeeds. `apps/rocm`'s [`dash.rs`](../apps/rocm/src/dash.rs) drives it. + - [`bench_ring.rs`](../crates/rocm-dash-daemon/src/bench_ring.rs) and [`snapshot_ring.rs`](../crates/rocm-dash-daemon/src/snapshot_ring.rs) — the daemon's two bounded histories for late-joining clients, `BenchRing` (200 rows) and `SnapshotRing` (30 snapshots), each a `VecDeque` that drops its oldest entry when full. On `Command::Subscribe`, `server.rs` replays the snapshot ring one `Snapshot` event at a time, then the benchmark ring as a single `BenchmarkRowsAppended` event (if it holds any rows), to that client before it sends live events. + - [`demo.rs`](../crates/rocm-dash-daemon/src/demo.rs) — the synthetic-session generator behind `rocm dash --demo`. `generate_to_writer` and `generate_file` write a recording as `PersistedEntry` lines (a `Welcome`, one `Snapshot` per simulated second, the instance discoveries on the first tick, scheduled benchmark rows and a closing `Bye`) of an 8-GPU node serving three vLLM containers, so the dashboard replays populated data with no GPU and no daemon. The same `DemoOptions` (seconds and seed) produce byte-identical output, from seeded jitter and a fixed wall-clock anchor; the file is created `0o600` on Unix (an existing file keeps its mode) and removed again if generation fails. + - [`persist.rs`](../crates/rocm-dash-daemon/src/persist.rs) — `SessionWriter`, the append-only session recorder. `server.rs` opens one `session-YYYYMMDD-HHMMSS.ndjson` for the daemon's lifetime when `RunnerOptions::persist_dir` is set, which `rocm dash` always does, to the `telemetry` directory under the data dir; if the file cannot be created, persistence is switched off with a warning. Every broadcast `Event` is appended as a `PersistedEntry` line and flushed to the OS (there is no fsync, and a failed write is logged and skipped), so a process crash loses at most the in-flight tick. There is no rotation or compression; the TUI's replay mode reads the file. + - [`transport.rs`](../crates/rocm-dash-daemon/src/transport.rs) — NDJSON framing for the daemon's socket: `write_line` serializes a value as one JSON line and flushes, `read_line` reads one line and returns `None` at end of input. [`rocm-dash-tui/src/transport.rs`](../crates/rocm-dash-tui/src/transport.rs) is a copy of it, because the TUI does not depend on the daemon crate, so a framing change must be made in both. ### `crates/rocm-engine-protocol` — engine IPC protocol @@ -308,7 +386,7 @@ A contract surface: verify all impacted engines after any protocol change here ( ### `crates/rocm-deps` -Pinned versions of the third-party runtimes rocm-cli manages (from workspace-root [`runtime-deps.toml`](../runtime-deps.toml), turned into constants by [`build.rs`](../crates/rocm-deps/build.rs)). Small and already single-purpose; not part of the modularization effort's target list. +Pinned versions of the third-party runtimes rocm-cli manages (from workspace-root [`runtime-deps.toml`](../runtime-deps.toml), turned into constants by [`build.rs`](../crates/rocm-deps/build.rs)). Small and already single-purpose; not part of the modularization effort's target list. [`lib.rs`](../crates/rocm-deps/src/lib.rs) includes the generated constants and holds the helpers that derive Lemonade release names from them (`strip_v`, `lemonade_archive_name`, `lemonade_download_url`, `LEMONADE_GITHUB_REPO`). Only `rocm` and `rocm-engine-lemonade` depend on it; the dashboard TUI does not. ### `crates/e2e-report` @@ -340,7 +418,7 @@ These modules are private `mod` declarations — [`lib.rs`](../crates/e2e-report - [`state.rs`](../engines/vllm/src/state.rs) — service state file I/O, log tailing, and health/inference probing. - [`wheel_builds.rs`](../engines/vllm/src/wheel_builds.rs) — which vLLM wheels to install for a ROCm SDK version: the static build table (`VLLM_ROCM_BUILD_TABLE`), the install-route choice (`vllm_install_route`, which consults the discovery table in `discover_builds.rs`), target resolution including the `ROCM_CLI_VLLM_ROCM_INDEX_URL` override (`vllm_install_target`), and the static-table SDK-version matchers. It also holds `verified_torchaudio_release`, which the crate re-exports from `lib.rs` for the `rocm` binary. It imports only `discover_builds.rs` and runs no process. -[`lib.rs`](../engines/vllm/src/lib.rs) keeps only CLI/envelope dispatch and the few items shared across those modules. +[`lib.rs`](../engines/vllm/src/lib.rs) keeps only CLI/envelope dispatch and the few items shared across those modules. [`main.rs`](../engines/vllm/src/main.rs) is the `rocm-engine-vllm` binary's entry point and only calls `rocm_engine_vllm::run_cli()`. `engines/lemonade` follows the same pattern: @@ -355,7 +433,7 @@ These modules are private `mod` declarations — [`lib.rs`](../crates/e2e-report - [`runtime_dir.rs`](../engines/lemonade/src/runtime_dir.rs) — `XDG_RUNTIME_DIR` tier1/2/3 hardening for the spawned child. - [`state.rs`](../engines/lemonade/src/state.rs) — service state file I/O and port/endpoint helpers. -[`lib.rs`](../engines/lemonade/src/lib.rs) keeps only CLI/envelope dispatch and the service-lifecycle orchestration that calls across those modules. +[`lib.rs`](../engines/lemonade/src/lib.rs) keeps only CLI/envelope dispatch and the service-lifecycle orchestration that calls across those modules. [`main.rs`](../engines/lemonade/src/main.rs) is the `rocm-engine-lemonade` binary's entry point and only calls `rocm_engine_lemonade::run_cli()`. `engines/aim` runs models as Docker containers instead of a host process: @@ -365,4 +443,4 @@ These modules are private `mod` declarations — [`lib.rs`](../crates/e2e-report - [`state.rs`](../engines/aim/src/state.rs) — service state file I/O, including the model name the container actually serves (`served_model_id`), which differs from the image reference, and `endpoint_url`, the `/v1` base URL for a host and port. `state.rs` uses no `process.rs` item: `process.rs` imports `state.rs` and the dependency does not run the other way. - [`install.rs`](../engines/aim/src/install.rs) — much smaller than vLLM's/Lemonade's `install.rs`: there is no venv, wheel, or runtime archive to manage, only a Docker-reachability check. -[`lib.rs`](../engines/aim/src/lib.rs) keeps only CLI/envelope dispatch (including the `builtin_*` entry points the `rocm` binary calls in-process), the detect and capabilities responses, and the few items shared across those modules. +[`lib.rs`](../engines/aim/src/lib.rs) keeps only CLI/envelope dispatch (including the `builtin_*` entry points the `rocm` binary calls in-process), the detect and capabilities responses, and the few items shared across those modules. [`main.rs`](../engines/aim/src/main.rs) is the `rocm-engine-aim` binary's entry point and only calls `rocm_engine_aim::run_cli()`. diff --git a/docs/testing.md b/docs/testing.md index d984e61639..cbe31a2650 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -63,7 +63,7 @@ cargo xtask check-architecture-doc It fails on a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` that the doc does not link, and on a stale entry in the `EXEMPT` list in -[architecture_doc.rs](../xtask/src/architecture_doc.rs). Test-only helpers need no entry. The +[architecture_doc.rs](../xtask/src/architecture_doc.rs), which is empty. Test-only helpers need no entry. The `lychee` check is complementary, not redundant: it confirms that links already in the doc resolve, and says nothing about a file the doc never mentions. diff --git a/xtask/src/architecture_doc.rs b/xtask/src/architecture_doc.rs index 27b96e15f8..5b1f7262e8 100644 --- a/xtask/src/architecture_doc.rs +++ b/xtask/src/architecture_doc.rs @@ -33,14 +33,14 @@ //! `?query` suffix and a root-relative path render as working links but are not //! recognised, so the failure message asks for the inline form. //! -//! [`EXEMPT`] is the files that existed, uncited, when the check was added. It -//! follows the pattern of `ALLOWLIST` in [`crate::crate_edges`] and is checked -//! in both directions: a source file in neither the doc nor the list fails, -//! and so does a list entry that is now cited or no longer exists. The list is -//! meant to shrink as files are documented. Nothing can stop a determined edit -//! from adding to it, so [`EXEMPT_CEILING`] pins its size in a test: growing -//! the list means raising that constant too, a second, visible line in the diff -//! that a reviewer has to accept. +//! [`EXEMPT`] is the files that may go without an entry. It is empty: every +//! production file is documented. It follows the pattern of `ALLOWLIST` in +//! [`crate::crate_edges`] and is checked in both directions: a source file in +//! neither the doc nor the list fails, and so does a list entry that is now +//! cited or no longer exists. Nothing can stop a determined edit from adding to +//! it, so [`EXEMPT_CEILING`] pins its size in a test: growing the list means +//! raising that constant too, a second, visible line in the diff that a +//! reviewer has to accept. //! //! What this does not check: whether an entry's prose still describes the file, //! and functions that move between two files that are both already cited. @@ -62,95 +62,15 @@ const DOC_DIR: &str = "docs"; /// Source roots whose `/src` trees are in scope. const SOURCE_ROOTS: [&str; 3] = ["apps", "crates", "engines"]; -/// Production files that have no entry in `docs/architecture.md` yet: everything -/// that was uncited when this check was added, mostly the dashboard crates. -/// Do not add to this list to make the check pass: document the file in the doc -/// instead. Growing it is a reviewed exception and also needs -/// [`EXEMPT_CEILING`] raised. Remove an entry when the file gets one. Keep it -/// sorted. -const EXEMPT: &[&str] = &[ - "apps/rocmd/src/main.rs", - "crates/rocm-dash-collectors/src/amd_smi.rs", - "crates/rocm-dash-collectors/src/bench_load.rs", - "crates/rocm-dash-collectors/src/bench_tail.rs", - "crates/rocm-dash-collectors/src/cgroup.rs", - "crates/rocm-dash-collectors/src/docker.rs", - "crates/rocm-dash-collectors/src/engine_registry.rs", - "crates/rocm-dash-collectors/src/host.rs", - "crates/rocm-dash-collectors/src/lemonade.rs", - "crates/rocm-dash-collectors/src/parallel.rs", - "crates/rocm-dash-collectors/src/proc_scan.rs", - "crates/rocm-dash-collectors/src/sysfs.rs", - "crates/rocm-dash-collectors/src/vllm_log.rs", - "crates/rocm-dash-collectors/src/vllm_prom.rs", - "crates/rocm-dash-core/src/bench_rollup.rs", - "crates/rocm-dash-core/src/bench_schema.rs", - "crates/rocm-dash-core/src/config.rs", - "crates/rocm-dash-core/src/efficiency.rs", - "crates/rocm-dash-core/src/metrics.rs", - "crates/rocm-dash-core/src/observation.rs", - "crates/rocm-dash-core/src/partition.rs", - "crates/rocm-dash-core/src/persist.rs", - "crates/rocm-dash-core/src/protocol.rs", - "crates/rocm-dash-core/src/state.rs", - "crates/rocm-dash-core/src/traits.rs", - "crates/rocm-dash-core/src/vram.rs", - "crates/rocm-dash-daemon/src/bench_ring.rs", - "crates/rocm-dash-daemon/src/demo.rs", - "crates/rocm-dash-daemon/src/persist.rs", - "crates/rocm-dash-daemon/src/snapshot_ring.rs", - "crates/rocm-dash-daemon/src/transport.rs", - "crates/rocm-dash-tui/src/client.rs", - "crates/rocm-dash-tui/src/jobs.rs", - "crates/rocm-dash-tui/src/lib.rs", - "crates/rocm-dash-tui/src/llm.rs", - "crates/rocm-dash-tui/src/reconnect.rs", - "crates/rocm-dash-tui/src/registry_probe.rs", - "crates/rocm-dash-tui/src/replay.rs", - "crates/rocm-dash-tui/src/tool_exec.rs", - "crates/rocm-dash-tui/src/transport.rs", - "crates/rocm-dash-tui/src/ui/automations_manager.rs", - "crates/rocm-dash-tui/src/ui/bench.rs", - "crates/rocm-dash-tui/src/ui/bench_run.rs", - "crates/rocm-dash-tui/src/ui/command_screen.rs", - "crates/rocm-dash-tui/src/ui/config_manager.rs", - "crates/rocm-dash-tui/src/ui/core_bars.rs", - "crates/rocm-dash-tui/src/ui/dock.rs", - "crates/rocm-dash-tui/src/ui/engine_manager.rs", - "crates/rocm-dash-tui/src/ui/examine_manager.rs", - "crates/rocm-dash-tui/src/ui/folder_browser.rs", - "crates/rocm-dash-tui/src/ui/format.rs", - "crates/rocm-dash-tui/src/ui/gradient.rs", - "crates/rocm-dash-tui/src/ui/install_manager.rs", - "crates/rocm-dash-tui/src/ui/instance_list.rs", - "crates/rocm-dash-tui/src/ui/launcher.rs", - "crates/rocm-dash-tui/src/ui/logs_view.rs", - "crates/rocm-dash-tui/src/ui/model_picker.rs", - "crates/rocm-dash-tui/src/ui/onboarding.rs", - "crates/rocm-dash-tui/src/ui/runtime_manager.rs", - "crates/rocm-dash-tui/src/ui/serve_wizard.rs", - "crates/rocm-dash-tui/src/ui/services_manager.rs", - "crates/rocm-dash-tui/src/ui/sparkline.rs", - "crates/rocm-dash-tui/src/ui/tabs/bench.rs", - "crates/rocm-dash-tui/src/ui/tabs/chat.rs", - "crates/rocm-dash-tui/src/ui/tabs/hardware.rs", - "crates/rocm-dash-tui/src/ui/tabs/home.rs", - "crates/rocm-dash-tui/src/ui/tabs/mod.rs", - "crates/rocm-dash-tui/src/ui/tabs/observe.rs", - "crates/rocm-dash-tui/src/ui/tabs/pane.rs", - "crates/rocm-dash-tui/src/ui/tabs/rocm.rs", - "crates/rocm-dash-tui/src/ui/tabs/serving.rs", - "crates/rocm-dash-tui/src/ui/update_manager.rs", - "crates/rocm-deps/src/lib.rs", - "engines/aim/src/main.rs", - "engines/lemonade/src/main.rs", - "engines/vllm/src/main.rs", -]; - -/// The most entries [`EXEMPT`] may hold. Lower it when entries are removed; do -/// not raise it to make the check pass, since a file with no entry in the module -/// map should get one. Raising it is the visible, reviewed way to grow the list. -const EXEMPT_CEILING: usize = 76; +/// Production files that deliberately have no entry in `docs/architecture.md`. +/// Empty: every production file is documented. Do not add to this list to make +/// the check pass: document the file in the doc instead. Growing it is a +/// reviewed exception and also needs [`EXEMPT_CEILING`] raised. Keep it sorted. +const EXEMPT: &[&str] = &[]; + +/// The most entries [`EXEMPT`] may hold. Zero: a file with no entry in the module +/// map should get one. Raising it is the visible, reviewed way to add an exemption. +const EXEMPT_CEILING: usize = 0; /// Whether `path` is a tracked-file path this check covers: a `.rs` file under /// `//src/` for one of the [`SOURCE_ROOTS`]. From 6f1f5346599670dcfba7734abd0ebc85706fe694 Mon Sep 17 00:00:00 2001 From: Eugene Volen <150254791+volen-silo@users.noreply.github.com> Date: Mon, 9 Nov 2026 12:55:44 +0000 Subject: [PATCH 014/102] test(skill): drop the operating system from the replayed-CLI prompts (#977) The two replayed-CLI cases asked about hipErrorNoBinaryForGpu "on Linux". That symptom is not Linux-specific, and the recorded CLI answers the same on any host. On a Windows host the agent declined to run the commands at all, saying they had to run on the user's Linux machine, so the recorded replies never printed and the logs_contain pins could not be met. Without the operating system the agent runs the commands. In WSL both cases passed three runs out of three. On Windows the agent now runs them too, but a real rocm on PATH can answer before the stub, so a full pass there is only expected on a host with none, such as a grading runner. Only these two prompts change. The permission-denied case is Linux-only by nature, and the two vLLM cases use catalog entries that apply on Linux and WSL2 only. The two vLLM cases get a note saying they name Linux on purpose and are not expected to pass on a Windows lane. Signed-off-by: Eugene Volen (cherry picked from commit a2330bfa14c2841314b6313faadacf06e18059ca) Signed-off-by: juhovainio --- skills/rocm-doctor/evals/evals.json | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/skills/rocm-doctor/evals/evals.json b/skills/rocm-doctor/evals/evals.json index 844265fc35..bac1ae4674 100644 --- a/skills/rocm-doctor/evals/evals.json +++ b/skills/rocm-doctor/evals/evals.json @@ -72,8 +72,8 @@ { "id": "rocm-older-cli-stops-before-diagnosing", "skill_should_trigger": true, - "note": "The one case that needs a `rocm` on PATH, which the grading hosts do not have. `evals/hooks.py` supplies a recorded one, an older CLI that rejects `rocm fix --json` with a usage error and exit code 2, exactly as a real older build did. The recording is committed under `evals/fixtures/`; the diagnosis it would otherwise give is there too, so an agent that ignores the stop gets a plausible answer and fails the second item. What is graded is the stop, not a diagnosis. The `logs_contain` pin is the stub's own usage error, which `SKILL.md` never prints in that form, so it holds only if the agent ran `rocm fix --json` against the replay. Its `--version` line (`rocm 0.1.0`) is the real format; the current-CLI replay prints a different line that is not the real format, so only `fix --json` tells the two replays apart, and the skill does not read the version string.", - "prompt": "My PyTorch job fails with hipErrorNoBinaryForGpu on my AMD GPU on Linux. How do I fix it?", + "note": "The one case that needs a `rocm` on PATH, which the grading hosts do not have. `evals/hooks.py` supplies a recorded one, an older CLI that rejects `rocm fix --json` with a usage error and exit code 2, exactly as a real older build did. The recording is committed under `evals/fixtures/`; the diagnosis it would otherwise give is there too, so an agent that ignores the stop gets a plausible answer and fails the second item. What is graded is the stop, not a diagnosis. The `logs_contain` pin is the stub's own usage error, which `SKILL.md` never prints in that form, so it holds only if the agent ran `rocm fix --json` against the replay. Its `--version` line (`rocm 0.1.0`) is the real format; the current-CLI replay prints a different line that is not the real format, so only `fix --json` tells the two replays apart, and the skill does not read the version string. The prompt names no operating system: the recorded CLI answers the same on any host, and an agent on a Windows host has declined to run the commands at all when the prompt said Linux.", + "prompt": "My PyTorch job fails with hipErrorNoBinaryForGpu on my AMD GPU. How do I fix it?", "expected_behavior": [ "Tell the user that the installed `rocm` CLI is older than the skill requires, so the CLI is the side to update, instead of diagnosing the failure", "Offer to update the CLI by re-running the installer, and ask for the user's consent before running it" @@ -89,8 +89,8 @@ { "id": "rocm-current-cli-carries-on", "skill_should_trigger": true, - "note": "The control for the older-CLI case: the same prompt against a recorded current CLI, which publishes the catalog manifest. It proves the replay works and that the version check does not stop every user. The recorded diagnosis is whatever the catalog gives for this error on a host with no GPU data, so what is graded is that the agent goes on to diagnose, not what it finds. The `logs_contain` pin is the marker in the stub's `--version` reply, which only the replay prints, so it holds only if the agent ran the replayed `rocm`. That line (`rocm-cli 0.1.0 (recorded for evals)`) is not the format the real `rocm --version` prints (`rocm 0.1.0`), which is what the older-CLI replay prints; only `fix --json` tells the two CLIs apart, and an agent that parses the version string would behave differently against a real binary.", - "prompt": "My PyTorch job fails with hipErrorNoBinaryForGpu on my AMD GPU on Linux. How do I fix it?", + "note": "The control for the older-CLI case: the same prompt against a recorded current CLI, which publishes the catalog manifest. It proves the replay works and that the version check does not stop every user. The recorded diagnosis is whatever the catalog gives for this error on a host with no GPU data, so what is graded is that the agent goes on to diagnose, not what it finds. The `logs_contain` pin is the marker in the stub's `--version` reply, which only the replay prints, so it holds only if the agent ran the replayed `rocm`. That line (`rocm-cli 0.1.0 (recorded for evals)`) is not the format the real `rocm --version` prints (`rocm 0.1.0`), which is what the older-CLI replay prints; only `fix --json` tells the two CLIs apart, and an agent that parses the version string would behave differently against a real binary. The prompt names no operating system, for the reason given on the older-CLI case.", + "prompt": "My PyTorch job fails with hipErrorNoBinaryForGpu on my AMD GPU. How do I fix it?", "expected_behavior": [ "Run `rocm diagnose` and relay what it returned, instead of stopping at the version check" ], @@ -105,7 +105,7 @@ { "id": "rocm-vllm-oom-at-startup", "skill_should_trigger": true, - "note": "Replays a current CLI. The catalog matches the vLLM out-of-memory entry only when `vllm` is on the same line as the error, and the prompt deliberately keeps the two apart (the pasted error line has no `vllm` in it), so the agent has to compose a one-line symptom itself. The replay stub answers `diagnose --symptom` with the recorded diagnosis whatever the symptom is, so the stub cannot fail a badly composed symptom; the composition expectation is graded by the judge from the command in the transcript. The real guarantee that a two-line symptom misses is the CLI behaviour the skill text describes. The user shares the card with another job, so lowering the reservation is a right answer for them; what is graded is that the agent relays both cases and does not claim it helps a model that does not fit.", + "note": "Replays a current CLI. The catalog matches the vLLM out-of-memory entry only when `vllm` is on the same line as the error, and the prompt deliberately keeps the two apart (the pasted error line has no `vllm` in it), so the agent has to compose a one-line symptom itself. The replay stub answers `diagnose --symptom` with the recorded diagnosis whatever the symptom is, so the stub cannot fail a badly composed symptom; the composition expectation is graded by the judge from the command in the transcript. The real guarantee that a two-line symptom misses is the CLI behaviour the skill text describes. The user shares the card with another job, so lowering the reservation is a right answer for them; what is graded is that the agent relays both cases and does not claim it helps a model that does not fit. The prompt names Linux on purpose: this catalog entry applies on Linux and WSL2 only and native Windows gets no match, so the case is not expected to pass on a Windows lane.", "prompt": "I serve a model with vLLM on my AMD GPU on Linux. It dies at startup and the last line of the log is 'torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 GiB'. Another job is also using the card. What should I do?", "expected_behavior": [ "Run `rocm diagnose` with the error line as the symptom and relay the matched cause", @@ -123,7 +123,7 @@ { "id": "rocm-vllm-import-error", "skill_should_trigger": true, - "note": "Replays a current CLI. The import error matches the torch DLPack entry (Linux and WSL2) at the likely level (score 50), not the high-confidence level. The recorded plan opens with steps that confirm the torch version in the engine's environment before the reinstall, so the agent should present the match as likely and have the user check that first rather than state the cause as certain. The user has not named a group or permission problem, and the agent must not invent one.", + "note": "Replays a current CLI. The import error matches the torch DLPack entry (Linux and WSL2) at the likely level (score 50), not the high-confidence level. The recorded plan opens with steps that confirm the torch version in the engine's environment before the reinstall, so the agent should present the match as likely and have the user check that first rather than state the cause as certain. The user has not named a group or permission problem, and the agent must not invent one. The prompt names Linux on purpose: this catalog entry applies on Linux and WSL2 only and native Windows gets no match, so the case is not expected to pass on a Windows lane.", "prompt": "vLLM fails to start on my AMD GPU on Linux with 'ImportError: libtorch_cuda.so: cannot open shared object file: No such file or directory'. How do I fix it?", "expected_behavior": [ "Run `rocm diagnose` with the error line as the symptom and relay the matched cause and the engine reinstall the CLI returned", From c4de88aa831395a8666540cae805ed23a33f6fa8 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 13:05:16 +0000 Subject: [PATCH 015/102] fix(install): preview a ROCm 10 SDK install without fetching uv (#954) * feat(core): look for an installed uv without acquiring one ensure_uv_binary returns the binary named by ROCM_CLI_UV_BINARY or the managed copy, and otherwise takes a lock under the data folder's tools folder and downloads one. A caller that must leave the machine untouched (an install preview) had no way to ask whether a uv is already there. find_uv_binary answers that question and nothing more: the override, or the managed copy when it runs, and None when ensure_uv_binary would have had to fetch. It creates no folder and takes no lock. The managed copy's location now comes from one helper shared with the install, and a test shows the lookup finds the copy the install reuses. No caller yet; ensure_uv_binary behaves exactly as before. Signed-off-by: Jussi Elo * refactor(install): write the wheel install plan from one module install_wheel_runtime wrote the sdk install plan inline, from the provenance lines down to the dry-run command and manifest path. Move those writes into therock/wheel_plan.rs behind a WheelPlan value: write_wheel_plan for the lines a preview and a real install share, write_dry_run_tail for the lines only a preview adds. The lines, their order and their wording are unchanged; the existing tests pass as they were. This lets the next change print a plan whose package versions are not chosen yet through the same writer instead of a second copy of it. Signed-off-by: Jussi Elo * test(e2e): pin that a ROCm 10 preview still runs a uv that is already present A preview of a ROCm 10 or newer wheel has uv pip compile choose the torch and torchvision versions, and the plan's package specs, runtime key and command are made of them. Add runtime-27: with a uv present (the recording stand-in named by ROCM_CLI_UV_BINARY, answering the compile with the fixture's four pins) the preview runs it once, for the compile and nothing else, prints the resolved plan and reports no skipped step. It passes on the current code and is the half of the contract a later change must keep when a preview stops fetching uv on a host that has none. The stand-in gains an opt-in FAKE_UV_DRAIN_STDIN, because the CLI writes the requirements to uv's stdin before it reads the answer and a stand-in that exits without reading can fail that write. The two host setups that put an empty PATH and a cache folder of their own in the scenario now share one helper. Signed-off-by: Jussi Elo * fix(install): preview a ROCm 10 SDK install without fetching uv On the ROCm 10 and newer wheel layout, rocm install sdk --dry-run (and the install_sdk_dry_run assistant tool) resolved the torch and torchvision versions with uv pip compile through the same resolver a real install uses, which acquires the managed uv: on a clean host the preview downloaded uv and created the data folder's tools folder although it promises to change nothing. The canonical layout needs no uv and was not affected. The resolver now takes a UvAccess. Acquire is what a real install and rocm update pass, and they decide and call exactly what they did before. PresentOnly is the preview's: it runs a uv that is already there (the binary ROCM_CLI_UV_BINARY names, or the managed copy a previous install left) and, with none, stops before the compile and returns an UnresolvedPipRuntime holding what needs no uv (the ROCm version, the torchaudio pin, the requirements uv would be given) and no package versions. The preview prints that plan through the same writer as every other plan, with target, runtime_key, package_specs, command and manifest marked not resolved by this preview, and says why in a resolution_note line and a progress line, in the voice of python_note. It cannot say whether the published stack installs together either, and the note says that. The decision is one pure function, tested over both layouts, both accesses and a uv present or not; a second test runs the real resolution against a served index and a stand-in uv and counts what each row looked for, acquired and ran. Scenario runtime-28 asserts the note together with the state on a host with no uv: no tools folder, no tool archive, no saved config and no torch or torchvision version in the output. The previews in therock-next-02, -10, -11 and -13 that print resolved specs start from an installed uv. Signed-off-by: Jussi Elo * refactor(install): move the uv access decision out of the wheel resolver The previous commit took therock/resolve/wheel.rs past the 1500 production-line limit. Move the part that decides who may fetch uv (UvAccess, UvPolicy, VersionResolution, UvChoice, plan_version_resolution and uv_for_compile) and its state-matrix tests into therock/resolve/uv_access.rs. It builds only on SourceLayout, so wheel.rs keeps acting on the answer and nothing else changes: the code and the tests are moved as they were, with the items widened to pub(super) so wheel.rs can name them. Signed-off-by: Jussi Elo * docs: say that an install preview fetches no uv on any layout The notes written when the preview stopped preparing Python limited the no-uv, no-tools-folder guarantee to the canonical layout, because a ROCm 10 preview could still download the managed uv. It no longer does, so make every place that called the preview read-only or described its uv behaviour say what is now true: the --dry-run help, the README's ROCm 10 section, docs/testing.md and docs/manual-testing.md (including the new scenarios and the one gap no test reaches without a network), and the dash-TUI tool-group comment for install_sdk_dry_run. The assistant tool descriptions, docs/llm-tool-use.md and the assistant skill make no claim about the preview's uv handling, so they are unchanged. Signed-off-by: Jussi Elo * test(install): satisfy clippy in the uv access tests and the plan helper Clippy under -D warnings flags a function that can be const, a test helper that appends a format! to a String, and an import that only the tests use since the decision moved to uv_access.rs. Fix the three; no behaviour changes. Signed-off-by: Jussi Elo * docs: name the real update check where the testing notes advise one docs/testing.md said rocm update --check does not preview in the install-preview sense, but rocm update has no --check flag: the plain rocm update (without --apply) is the check. The advised-commands test rejects the spelling. Name the update_check tool and the plain rocm update check instead. Signed-off-by: Jussi Elo * docs(e2e): point runtime-28's comment at the test that holds the decision matrix The comment said the decision matrix and the plan wording are unit-tested in wheel.rs and wheel_plan.rs. The matrix test, the_version_step_is_decided_by_layout_access_and_a_present_uv_alone, lives in therock/resolve/uv_access.rs; the plan wording tests are in wheel.rs and exercise what wheel_plan.rs writes. docs/testing.md and docs/architecture.md already name the right files. Signed-off-by: Jussi Elo * refactor(install): resolve a recorded device target in one place The rule that a recorded target (an update apply's recorded arch, or the one a runtime's manifest was built with) replaces the detected one, and is judged against the targets the source publishes, was written three times: in install_wheel_runtime, in the preview of a plan that stopped before uv, and in an update's plan. A later change to one would have missed the others. AggregateDeviceTarget::prefer_recorded is now the one place, and all three call it; behaviour is unchanged. Tests pin the rule itself (nothing recorded keeps the detected answer, undetermined included; a published target wins; an unpublished one is undetermined with its reason), show that the preview applies it through the helper, and scan the three files so that a call site that resolves a recorded target by hand again fails. Signed-off-by: Jussi Elo * docs(install): tighten comments so wheel.rs stays within the module-size limit The recorded-device-target helper took therock/resolve/wheel.rs 13 production lines over the 1500-line limit. Shorten the doc comments on the helper, UnresolvedPipRuntime and the two resolver entry points; no code changes. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 784429277fc2417a950fb167fa0c2c90f30ca98f) Signed-off-by: juhovainio --- README.md | 8 + apps/rocm/src/cli.rs | 5 + apps/rocm/src/therock.rs | 250 ++-- apps/rocm/src/therock/resolve.rs | 7 +- apps/rocm/src/therock/resolve/uv_access.rs | 215 ++++ apps/rocm/src/therock/resolve/wheel.rs | 1038 +++++++++++++++-- apps/rocm/src/therock/update.rs | 17 +- apps/rocm/src/therock/wheel_plan.rs | 333 ++++++ crates/rocm-core/src/lib.rs | 7 +- crates/rocm-core/src/uv.rs | 106 +- crates/rocm-dash-tui/src/agent/tools.rs | 9 +- docs/architecture.md | 6 +- docs/manual-testing.md | 14 +- docs/testing.md | 50 +- .../features/runtime_setup.feature | 53 + .../features/therock_next_generation.feature | 11 + tests/e2e-cucumber/src/bin/fake-uv.rs | 12 + tests/e2e-cucumber/tests/e2e/therock_steps.rs | 173 ++- 18 files changed, 2010 insertions(+), 304 deletions(-) create mode 100644 apps/rocm/src/therock/resolve/uv_access.rs create mode 100644 apps/rocm/src/therock/wheel_plan.rs diff --git a/README.md b/README.md index bf24be610c..5a22e810ef 100644 --- a/README.md +++ b/README.md @@ -654,6 +654,14 @@ a family of GPUs, or none), or you want to target a different GPU, also pass rocm install sdk --version 10.1.0 --family gfx1200 --dry-run ``` +The `--dry-run` preview downloads no `uv` and creates no `tools` folder. For ROCm +10 and newer, `uv pip compile` is what chooses the `torch` and `torchvision` +versions, so the preview runs a `uv` that is already installed and gives the +complete plan. With none installed it can't, and it says so in a +`resolution_note:` line: the package specs, runtime key, install folder, manifest +and `command:` (everything made from those versions) read `not resolved by this +preview`, and a real install fetches `uv` and resolves them. + A family label such as `--family gfx120X-all` is rejected for those versions instead of being resolved to a guess, because the ROCm 10 packages publish one payload per exact arch and there is no bucket payload to fall back to. Run diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index eeed2127f8..7cc1ea8986 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -807,6 +807,11 @@ rocm install sdk --devel")] #[arg(long)] devel: bool, /// Resolve the install plan without changing files. + /// + /// Downloads no uv and prepares no Python. For ROCm 10 and newer on a + /// host with no uv installed, the plan marks the package versions it + /// would need uv to choose, and everything made from them, as not + /// resolved. #[arg(long)] dry_run: bool, /// Approve replacing the current active default ROCm runtime (and diff --git a/apps/rocm/src/therock.rs b/apps/rocm/src/therock.rs index 3371bbdb52..2de4ecb64a 100644 --- a/apps/rocm/src/therock.rs +++ b/apps/rocm/src/therock.rs @@ -22,6 +22,7 @@ mod source; mod test_support; mod update; mod version; +mod wheel_plan; use self::consent::{ SdkInstallApproval, active_default_runtime_relation, confirm_overwrite_existing_sdk, @@ -43,8 +44,9 @@ use self::python::{ ensure_uv_venv, managed_python_version, python_for_install, python_requirement, }; use self::resolve::{ - AggregateDeviceTarget, canonical_auto_select_needs_next_python, resolve_pip_runtime, - resolve_tarball_artifact, unverified_torchaudio_note, wheel_runtime_composition, + AggregateDeviceTarget, PipRuntimePlan, canonical_auto_select_needs_next_python, + preview_pip_runtime, resolve_pip_runtime, resolve_tarball_artifact, unverified_torchaudio_note, + wheel_runtime_composition, }; pub(crate) use self::source::RuntimeVersionSelector; #[cfg(test)] @@ -61,6 +63,10 @@ pub(crate) use self::version::runtime_version_display; use self::version::{ parse_host_version, parse_version_for_ordering, runtime_version_build_date, version_relation, }; +use self::wheel_plan::{ + PlanStack, ResolvedStack, UNRESOLVED_PROGRESS, UnresolvedPreview, WheelPlan, + unresolved_wheel_preview, write_dry_run_tail, write_wheel_plan, +}; #[cfg(test)] pub(crate) use crate::runtime_registry::RocmSdkLibraryProbe; #[cfg(test)] @@ -73,12 +79,13 @@ use crate::runtime_registry::{ ManifestOverwrite, runtime_manifest_path, wheel_composition_includes_devel, write_runtime_manifest, }; +#[cfg(test)] use crate::shell_quote::shell_quote; use anyhow::{Context, Result, bail}; use rocm_core::{ AppPaths, detect_host_gpu_diagnostics, detect_legacy_rocm_summary, ensure_uv_binary, interactive_terminal, runtime_is_windows, runtime_python_executable_in_env, unix_time_millis, - uv_pip_install_base, uv_venv_args, + uv_pip_install_base, }; #[cfg(test)] use rocm_core::{disk_space, managed_uv_cache_dir, normalize_therock_family}; @@ -384,24 +391,55 @@ fn install_wheel_runtime( "Checking TheRock {} packages for this AMD GPU...", channel.as_str() )); - let resolution = resolve_pip_runtime( - paths, - channel, - recovered_family_override.as_deref(), - &wheel_compatibility, - version_selector, - layout_override, - include_devel, - )?; - let device_target = device_target_override.map_or_else( - || resolution.device_target.clone(), - |target| { - AggregateDeviceTarget::resolve( - Some(target), - &resolution.family, - &resolution.published_device_targets, - ) - }, + // A preview must not fetch `uv` either: on the ROCm 10 layout resolving the + // package versions runs `uv pip compile`, and a real install gets the managed + // `uv` for that by downloading it. A preview uses a `uv` that is already there + // and otherwise stops with what it can know without one, which says so. A real + // install and an update resolve, and so fetch, exactly as they always did. + let resolution = if dry_run { + match preview_pip_runtime( + paths, + channel, + recovered_family_override.as_deref(), + &wheel_compatibility, + version_selector, + layout_override, + include_devel, + )? { + PipRuntimePlan::Resolved(resolution) => resolution, + PipRuntimePlan::NeedsUv(unresolved) => { + progress_line(UNRESOLVED_PROGRESS); + let host_version_newer = host_rocm_version_newer_than(&unresolved.rocm_version); + return Ok(SdkInstallResult::plan(unresolved_wheel_preview( + &UnresolvedPreview { + channel, + version_selector, + unresolved: &unresolved, + device_target_override, + prefix: prefix.as_deref(), + host_version_newer: host_version_newer.as_deref(), + python: &python, + wheel_compatibility: &wheel_compatibility, + }, + ))); + } + } + } else { + resolve_pip_runtime( + paths, + channel, + recovered_family_override.as_deref(), + &wheel_compatibility, + version_selector, + layout_override, + include_devel, + )? + }; + let device_target = AggregateDeviceTarget::prefer_recorded( + &resolution.device_target, + device_target_override, + &resolution.family, + &resolution.published_device_targets, ); // The exact device payload, not the version alone, decides what this runtime // can run, so it is what identifies the runtime. A preview on a host with no @@ -417,146 +455,41 @@ fn install_wheel_runtime( let install_root = resolved_install_root(paths, "wheel", &runtime_key, prefix); let manifest_path = runtime_manifest_path(paths, &runtime_key); - let mut output = String::new(); - let _ = writeln!(output, "sdk install"); - let _ = writeln!( - output, - " summary: rocm-cli will install the ROCm SDK and matching PyTorch packages for this Python and operating system" - ); - render_canonical_provenance( - &mut output, - channel, - &resolution.index_url, - resolution.layout.generation(), - &resolution.latest_version, - ); - let _ = writeln!(output, " format: wheel"); - if let Some(selector) = version_selector { - let _ = writeln!(output, " requested: {}", selector.describe()); - } - let _ = writeln!(output, " target_family: {}", resolution.family); - let _ = writeln!( - output, - " target_family_source: {}", - resolution.family_source - ); - let _ = writeln!(output, " device_target: {}", device_target.as_str()); - if let Some(reason) = device_target.reason() { - let _ = writeln!(output, " device_target_reason: {reason}"); - } - let _ = writeln!(output, " index_url: {}", resolution.index_url); - let _ = writeln!( - output, - " latest_compatible_version: {}", - runtime_version_display(&resolution.latest_version) - ); // Probed once and reused by the progress line below: each call is a full // filesystem scan for a legacy ROCm, and the summary block and the visible // note report the same answer about the same resolved version. let host_version_newer = host_rocm_version_newer_than(&resolution.latest_version); - if let Some(host_version) = host_version_newer.as_deref() { - let _ = writeln!( - output, - " version_note: {}", - wheel_host_version_note( - host_version, - &runtime_version_display(&resolution.latest_version) - ) - ); - } - let _ = writeln!( - output, - " compatibility_key: {}", - runtime_version_display(&resolution.package_versions.compatibility_key) - ); - let _ = writeln!(output, " target: {}", install_root.display()); - let _ = writeln!(output, " runtime_key: {runtime_key}"); - let _ = writeln!(output, " python_launcher: {}", python.launcher_display()); - let _ = writeln!(output, " python_source: {}", python.source().as_str()); - if let Some(note) = python.note() { - let _ = writeln!(output, " python_note: {note}"); - } - let _ = writeln!( - output, - " python_wheel_tag: {}", - wheel_compatibility.python_tag - ); - let _ = writeln!( - output, - " platform_wheel_tags: {}", - wheel_compatibility.platform_tags.join(",") - ); - // What resolution asked for, printed beside what the install will ask for. - // The two are produced by different code paths from the same `include_devel` - // and must name the same extras; showing only the second would hide a - // resolve that constrained the version choice by a toolchain the user - // declined. Absent on the canonical layout, which resolves by scraping each - // package's index instead of compiling a requirement set. - if let Some(specs) = resolution.version_resolution_specs.as_ref() { - let _ = writeln!(output, " version_resolution_specs: {}", specs.join(" ")); - } - // Printed from the same value that chose the pin, so the line cannot claim - // a verified build the requirements above do not name. - if let Some(pin) = resolution.torchaudio_pin.as_ref() { - let _ = writeln!(output, " torchaudio_selection: {}", pin.describe()); - } - let _ = writeln!( - output, - " package_specs: {}", - wheel_composition.package_specs.join(" ") - ); - let _ = writeln!( - output, - " package_policy: resolve the pinned target-complete rocm, torch, torchvision, and torchaudio plan from published package metadata, then install it in one uv transaction" - ); - let no_wheel_warning = repo_version_without_wheels( + let no_wheel_warning = no_wheel_warning_for( resolution.newest_repo_version.as_deref(), &resolution.latest_version, - ) - .map(|newest| { - no_wheel_warning_message( - &newest, - &runtime_version_display(&resolution.latest_version), - ) - }); - if let Some(warning) = no_wheel_warning.as_deref() { - let _ = writeln!(output, " warning: {warning}"); - } + ); + let plan = WheelPlan { + channel, + version_selector, + index_url: &resolution.index_url, + layout_generation: resolution.layout.generation(), + latest_version: &resolution.latest_version, + family: &resolution.family, + family_source: &resolution.family_source, + device_target: &device_target, + host_version_newer: host_version_newer.as_deref(), + python: &python, + wheel_compatibility: &wheel_compatibility, + version_resolution_specs: resolution.version_resolution_specs.as_deref(), + torchaudio_pin: resolution.torchaudio_pin.as_ref(), + no_wheel_warning: no_wheel_warning.as_deref(), + stack: PlanStack::Resolved(ResolvedStack { + compatibility_key: &resolution.package_versions.compatibility_key, + install_root: &install_root, + runtime_key: &runtime_key, + package_specs: &wheel_composition.package_specs, + manifest_path: &manifest_path, + }), + }; + let mut output = String::new(); + write_wheel_plan(&mut output, &plan); if dry_run { - let env_python = venv_python_path(&install_root); - let mut install_args = uv_pip_install_base(&env_python); - install_args.extend(["--index-url".to_owned(), resolution.index_url.clone()]); - if matches!(channel, TheRockChannel::Nightly) { - install_args.extend(["--prerelease".to_owned(), "allow".to_owned()]); - } - install_args.extend(wheel_composition.package_specs.iter().cloned()); - let venv_args = uv_venv_args(&python.venv_python_arg(), &install_root); - // This line is offered as the command to run, so it has to be one a - // shell reads back as the argv it previews. The install root in it - // comes from a `--prefix` the local assistant can choose - // (`install_sdk_dry_run` is a model-callable tool), so a bare or - // double-quoted value here would be a model-chosen value in a line a - // user is invited to paste. - let venv_args_display = venv_args - .iter() - .map(|arg| shell_quote(arg)) - .collect::>() - .join(" "); - let install_args_display = install_args - .iter() - .map(|arg| shell_quote(arg)) - .collect::>() - .join(" "); - let _ = writeln!(output, " mode: dry-run"); - let _ = writeln!( - output, - " command: uv {venv_args_display} && uv {install_args_display}" - ); - let _ = writeln!( - output, - " activation: use the managed venv Python; TheRock libraries are resolved from that venv by rocm_sdk.initialize_process" - ); - let _ = writeln!(output, " manifest: {}", manifest_path.display()); + write_dry_run_tail(&mut output, &plan); return Ok(SdkInstallResult::plan(output)); } @@ -817,6 +750,15 @@ fn no_wheel_warning_message(newest: &str, resolved_display: &str) -> String { ) } +/// The `warning` surfaced when the repository's newest version has no installable +/// PyTorch wheels, or `None` when there is no such gap: [`repo_version_without_wheels`] +/// decides whether there is one and [`no_wheel_warning_message`] words it. One call +/// for a plan that is resolved and one that is not, so they cannot disagree. +fn no_wheel_warning_for(newest_repo: Option<&str>, resolved_version: &str) -> Option { + repo_version_without_wheels(newest_repo, resolved_version) + .map(|newest| no_wheel_warning_message(&newest, &runtime_version_display(resolved_version))) +} + /// The repo's newest version when it is strictly newer than the version we are /// about to install — i.e. the newest release exists in the index but has no /// installable PyTorch wheels for this Python/platform, so an older one is used. diff --git a/apps/rocm/src/therock/resolve.rs b/apps/rocm/src/therock/resolve.rs index 9efff586db..6e1aca6340 100644 --- a/apps/rocm/src/therock/resolve.rs +++ b/apps/rocm/src/therock/resolve.rs @@ -5,15 +5,18 @@ //! How a request is turned into the artifact an install fetches. Only module //! declarations and re-exports live here, no logic: `tarball` resolves the SDK //! tarball from the catalog's index, `wheel` resolves the wheel stack from the -//! wheel indexes, and the rest of `therock` reads the names below. +//! wheel indexes (with `uv_access` deciding who may fetch `uv` while it does), +//! and the rest of `therock` reads the names below. mod tarball; +mod uv_access; mod wheel; pub(super) use self::tarball::{resolve_tarball_artifact, resolve_tarball_artifact_with_timeout}; #[cfg(test)] pub(super) use self::wheel::select_rocm_version; pub(super) use self::wheel::{ - AggregateDeviceTarget, canonical_auto_select_needs_next_python, resolve_pip_runtime, + AggregateDeviceTarget, PipRuntimePlan, TorchaudioPin, UnresolvedPipRuntime, + canonical_auto_select_needs_next_python, preview_pip_runtime, resolve_pip_runtime, resolve_pip_runtime_with_timeout, unverified_torchaudio_note, wheel_runtime_composition, }; diff --git a/apps/rocm/src/therock/resolve/uv_access.rs b/apps/rocm/src/therock/resolve/uv_access.rs new file mode 100644 index 0000000000..3ad8dbdb42 --- /dev/null +++ b/apps/rocm/src/therock/resolve/uv_access.rs @@ -0,0 +1,215 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Who may fetch `uv` while a wheel stack is resolved. +//! +//! On the ROCm 10 layout `uv pip compile` is the one step of resolution that can +//! download the managed `uv` and create the data folder's `tools` folder. A real +//! install and an update ([`UvAccess::Acquire`]) take it as they always did; a +//! preview ([`UvAccess::PresentOnly`]) runs a `uv` that is already there and +//! otherwise skips the step. [`plan_version_resolution`] decides which, as a pure +//! function of the layout, the access and whether a `uv` is present, so that every +//! combination is tested without a network. `wheel.rs` acts on the answer. + +use super::super::source::SourceLayout; +use anyhow::{Context, Result}; +use std::path::PathBuf; + +/// Whether a resolution may fetch `uv` to choose the ROCm 10 package versions. +/// +/// Only the ROCm 10 (`next`) layout needs `uv`: it has `uv pip compile` pick the +/// `torch` and `torchvision` builds. That is the one step of resolution that can +/// download the managed `uv` and create the data folder's `tools` folder. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum UvAccess { + /// A real install or an update: take the managed `uv`, downloading it if no + /// copy is installed yet. This is what resolution has always done. + Acquire, + /// A preview: use a `uv` that is already there and never fetch one. + PresentOnly, +} + +/// What a resolution may do about `uv`: how far it may go ([`UvAccess`]) and the two +/// things it can ask for. The real ones, `find_uv_binary` and `ensure_uv_binary`, +/// are built where a resolution starts; they are fields so that a test can count +/// the calls instead of reaching the network. +pub(super) struct UvPolicy<'a> { + pub(super) access: UvAccess, + /// A `uv` that is already there, without acquiring one. + pub(super) find_present: &'a dyn Fn() -> Option, + /// The managed `uv`, downloading it when none is installed. + pub(super) acquire: &'a dyn Fn() -> Result, +} + +/// How a resolution chooses the package versions, given [`UvAccess`] and whether a +/// `uv` is present. A pure function of those inputs: see [`plan_version_resolution`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum VersionResolution { + /// The canonical layout: read each package's simple index. No `uv`. + ScrapeIndexes, + /// The ROCm 10 layout: run `uv pip compile` with this `uv`. + Compile(UvChoice), + /// The ROCm 10 layout with no `uv` to run and none to fetch: stop with what is + /// known without one, and leave the versions `uv` would choose unresolved. + Skip, +} + +/// Which `uv` the compile step runs. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum UvChoice { + /// Ask `ensure_uv_binary`, which downloads the managed `uv` when none is installed. + Acquire, + /// A `uv` that was already there. + Existing(PathBuf), +} + +/// Decide how the versions are chosen, without doing it. +/// +/// `find_present` looks for a `uv` that is already there. It runs only when the +/// answer depends on it: never for the canonical layout, which needs no `uv`, and +/// never for [`UvAccess::Acquire`], whose `ensure_uv_binary` does its own looking. +/// So a real install and an update decide, and call, exactly what they did before a +/// preview could skip the step, and no answer for them is [`VersionResolution::Skip`]. +pub(super) fn plan_version_resolution( + layout: SourceLayout, + access: UvAccess, + find_present: impl FnOnce() -> Option, +) -> VersionResolution { + match (layout, access) { + (SourceLayout::Canonical, _) => VersionResolution::ScrapeIndexes, + (SourceLayout::Next, UvAccess::Acquire) => VersionResolution::Compile(UvChoice::Acquire), + (SourceLayout::Next, UvAccess::PresentOnly) => find_present() + .map_or(VersionResolution::Skip, |uv| { + VersionResolution::Compile(UvChoice::Existing(uv)) + }), + } +} + +/// The `uv` the compile step runs: the one already there, or the managed `uv` that +/// `acquire` (`ensure_uv_binary`) fetches when none is installed. +pub(super) fn uv_for_compile( + choice: UvChoice, + acquire: impl FnOnce() -> Result, +) -> Result { + match choice { + UvChoice::Existing(uv) => Ok(uv), + UvChoice::Acquire => { + acquire().context("failed to acquire uv for ROCm X metadata resolution") + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use anyhow::bail; + + /// Asks [`plan_version_resolution`] for one cell of the matrix and reports + /// the answer together with how many times a present `uv` was looked for. + fn decide( + layout: SourceLayout, + access: UvAccess, + uv_present: bool, + ) -> (VersionResolution, usize) { + let looked = std::cell::Cell::new(0); + let decision = plan_version_resolution(layout, access, || { + looked.set(looked.get() + 1); + uv_present.then(|| PathBuf::from("/opt/uv")) + }); + (decision, looked.get()) + } + + /// The state matrix of the decision: both layouts, both accesses, a `uv` + /// present or not. Only the ROCm 10 layout needs `uv`; a real install and an + /// update (`Acquire`) always go on to acquire it and never ask whether one is + /// present, because `ensure_uv_binary` does its own looking; a preview + /// (`PresentOnly`) asks, and with none present stops. `Skip` therefore exists + /// for exactly one row. + #[test] + fn the_version_step_is_decided_by_layout_access_and_a_present_uv_alone() { + use SourceLayout::{Canonical, Next}; + use UvAccess::{Acquire, PresentOnly}; + let existing = || VersionResolution::Compile(UvChoice::Existing(PathBuf::from("/opt/uv"))); + let rows = [ + // (layout, access, uv present, decision, times a present uv was looked for) + ( + Canonical, + Acquire, + false, + VersionResolution::ScrapeIndexes, + 0, + ), + ( + Canonical, + Acquire, + true, + VersionResolution::ScrapeIndexes, + 0, + ), + ( + Canonical, + PresentOnly, + false, + VersionResolution::ScrapeIndexes, + 0, + ), + ( + Canonical, + PresentOnly, + true, + VersionResolution::ScrapeIndexes, + 0, + ), + ( + Next, + Acquire, + false, + VersionResolution::Compile(UvChoice::Acquire), + 0, + ), + ( + Next, + Acquire, + true, + VersionResolution::Compile(UvChoice::Acquire), + 0, + ), + (Next, PresentOnly, false, VersionResolution::Skip, 1), + (Next, PresentOnly, true, existing(), 1), + ]; + for (layout, access, uv_present, expected, looked) in rows { + let (decision, calls) = decide(layout, access, uv_present); + let row = format!("{layout:?}, {access:?}, uv present: {uv_present}"); + assert_eq!(decision, expected, "decision for {row}"); + assert_eq!(calls, looked, "times a present uv was looked for in {row}"); + } + } + + /// The compile step runs the `uv` it was given, and asks for one only on + /// `Acquire`, with the context a failed fetch has always carried. + #[test] + fn the_compile_step_acquires_uv_only_when_told_to() { + let acquired = std::cell::Cell::new(0); + let acquire = || { + acquired.set(acquired.get() + 1); + Ok(PathBuf::from("/managed/uv")) + }; + assert_eq!( + uv_for_compile(UvChoice::Acquire, acquire).unwrap(), + PathBuf::from("/managed/uv") + ); + assert_eq!(acquired.get(), 1); + assert_eq!( + uv_for_compile(UvChoice::Existing(PathBuf::from("/opt/uv")), acquire).unwrap(), + PathBuf::from("/opt/uv") + ); + assert_eq!(acquired.get(), 1, "a present uv is not acquired again"); + + let error = uv_for_compile(UvChoice::Acquire, || bail!("no network")).unwrap_err(); + assert_eq!( + format!("{error:#}"), + "failed to acquire uv for ROCm X metadata resolution: no network" + ); + } +} diff --git a/apps/rocm/src/therock/resolve/wheel.rs b/apps/rocm/src/therock/resolve/wheel.rs index cf5c1f1a92..dc9c5ff937 100644 --- a/apps/rocm/src/therock/resolve/wheel.rs +++ b/apps/rocm/src/therock/resolve/wheel.rs @@ -15,7 +15,10 @@ //! `torch`, `torchvision` and `torchaudio` listings agree; on the next layout //! `torchaudio_pin` pins a torchaudio first (the hardware-verified release when //! it is published) and `resolve_published_pip_package_versions` has -//! `uv pip compile` choose the rest. `wheel_runtime_composition` turns a +//! `uv pip compile` choose the rest. That compile is the one step that may fetch +//! `uv`: a real install and an update acquire it; a preview +//! (`preview_pip_runtime`) runs a present one or stops with an +//! `UnresolvedPipRuntime`, as `uv_access` decides. `wheel_runtime_composition` turns a //! resolution into the exact package specs (`therock_pip_package_specs`) an //! install gives `uv` and records in the manifest, and //! `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` @@ -45,13 +48,17 @@ use super::super::source::{ use super::super::version::{ ParsedVersion, compare_version_strings, is_stable_runtime_version, parse_version, }; +use super::uv_access::{ + UvAccess, UvPolicy, VersionResolution, plan_version_resolution, uv_for_compile, +}; use crate::runtime_registry::WheelRuntimeComposition; use anyhow::{Context, Result, bail}; use rocm_core::{ - AppPaths, detect_host_gfx_target, ensure_uv_binary, extract_first_gfx_token, + AppPaths, detect_host_gfx_target, ensure_uv_binary, extract_first_gfx_token, find_uv_binary, normalize_therock_family, runtime_is_windows, uv_command_env, }; use std::io::Write; +use std::path::Path; use std::process::{Command, Stdio}; use std::time::Duration; @@ -192,6 +199,22 @@ impl AggregateDeviceTarget { Self::Exact(target) } + /// A recorded target (an update apply's, or a runtime manifest's) wins over + /// `detected` and is checked against what the source publishes; with none the + /// detected answer stands, `Undetermined` included. The one place this rule is + /// written, so a plan, a preview and an update's plan cannot disagree. + pub(in crate::therock) fn prefer_recorded( + detected: &Self, + recorded: Option<&str>, + family: &str, + published: &[String], + ) -> Self { + recorded.map_or_else( + || detected.clone(), + |target| Self::resolve(Some(target), family, published), + ) + } + pub(in crate::therock) fn as_str(&self) -> &str { match self { Self::Exact(target) => target, @@ -480,6 +503,37 @@ pub(in crate::therock) struct PipRuntimeResolution { pub(in crate::therock) published_device_targets: Vec, } +/// What a resolution knows once it has been asked for with a given [`UvAccess`]. +#[derive(Debug, Clone)] +pub(in crate::therock) enum PipRuntimePlan { + Resolved(PipRuntimeResolution), + /// A preview of the ROCm 10 layout on a host with no `uv`. Only + /// [`UvAccess::PresentOnly`] yields it. + NeedsUv(UnresolvedPipRuntime), +} + +/// A ROCm 10 resolution stopped before `uv pip compile`: what was chosen without +/// `uv` (family, device target, index, `rocm` version, torchaudio pin, the +/// requirements `uv` would be given) and nothing made from the `torch` and +/// `torchvision` versions only `uv` produces. It carries no +/// [`TheRockPipPackageVersions`], so such a value cannot be mistaken for one. +#[derive(Debug, Clone)] +pub(in crate::therock) struct UnresolvedPipRuntime { + pub(in crate::therock) family: String, + pub(in crate::therock) family_source: String, + pub(in crate::therock) index_url: String, + pub(in crate::therock) layout: SourceLayout, + /// The `rocm` version the install would be for. + pub(in crate::therock) rocm_version: String, + pub(in crate::therock) newest_repo_version: Option, + /// The requirement lines `uv pip compile` would be given, as + /// [`PipRuntimeResolution::version_resolution_specs`] holds them once it ran. + pub(in crate::therock) version_resolution_specs: Vec, + pub(in crate::therock) torchaudio_pin: TorchaudioPin, + pub(in crate::therock) device_target: AggregateDeviceTarget, + pub(in crate::therock) published_device_targets: Vec, +} + #[derive(Debug, Clone, Eq, PartialEq)] pub(in crate::therock) struct TheRockPipPackageVersions { rocm: String, @@ -618,6 +672,9 @@ pub(in crate::therock) fn resolve_pip_runtime( /// `layout_override` is how an update keeps a runtime on the stream it was /// installed from; a fresh install passes `None` and lets the request decide. +/// +/// Always [`UvAccess::Acquire`], as a real install and an update always were; a +/// preview asks [`preview_pip_runtime`]. #[allow(clippy::too_many_arguments)] pub(in crate::therock) fn resolve_pip_runtime_with_timeout( paths: &AppPaths, @@ -629,6 +686,67 @@ pub(in crate::therock) fn resolve_pip_runtime_with_timeout( include_devel: bool, download_timeout_secs: Option, ) -> Result { + match resolve_pip_runtime_plan( + paths, + channel, + family_override, + wheel_compatibility, + version_selector, + layout_override, + include_devel, + download_timeout_secs, + UvAccess::Acquire, + )? { + PipRuntimePlan::Resolved(resolution) => Ok(resolution), + // `Skip` is only ever answered to `PresentOnly` (a test holds it to that). + PipRuntimePlan::NeedsUv(_) => { + bail!("internal error: a resolution that may acquire uv stopped for want of one") + } + } +} + +/// [`resolve_pip_runtime`] for a preview: never downloads `uv` or creates the +/// folders that hold it. A `uv` that is already there is used; with none, a ROCm 10 +/// request comes back as [`PipRuntimePlan::NeedsUv`]. +pub(in crate::therock) fn preview_pip_runtime( + paths: &AppPaths, + channel: TheRockChannel, + family_override: Option<&str>, + wheel_compatibility: &WheelCompatibility, + version_selector: Option<&RuntimeVersionSelector>, + layout_override: Option, + include_devel: bool, +) -> Result { + resolve_pip_runtime_plan( + paths, + channel, + family_override, + wheel_compatibility, + version_selector, + layout_override, + include_devel, + None, + UvAccess::PresentOnly, + ) +} + +#[allow(clippy::too_many_arguments)] +fn resolve_pip_runtime_plan( + paths: &AppPaths, + channel: TheRockChannel, + family_override: Option<&str>, + wheel_compatibility: &WheelCompatibility, + version_selector: Option<&RuntimeVersionSelector>, + layout_override: Option, + include_devel: bool, + download_timeout_secs: Option, + uv_access: UvAccess, +) -> Result { + let uv = UvPolicy { + access: uv_access, + find_present: &|| find_uv_binary(paths), + acquire: &|| ensure_uv_binary(paths), + }; let family_resolution = resolve_family(paths, family_override)?; let layout = match layout_override { Some(layout) => layout, @@ -680,6 +798,7 @@ pub(in crate::therock) fn resolve_pip_runtime_with_timeout( version_selector, include_devel, download_timeout_secs, + &uv, ) .with_context(|| { format!( @@ -704,7 +823,8 @@ fn resolve_pip_runtime_from_index( version_selector: Option<&RuntimeVersionSelector>, include_devel: bool, download_timeout_secs: Option, -) -> Result { + uv: &UvPolicy<'_>, +) -> Result { let index_url = source.index_url.as_str(); let mut version_resolution_specs = None; let mut torchaudio_selection = None; @@ -720,99 +840,6 @@ fn resolve_pip_runtime_from_index( "release channel only installs stable TheRock wheel versions, but no stable `rocm` package versions were found in {index_url}; try `rocm install sdk --channel release --format tarball` for stable release artifacts, or use `--channel nightly --format wheel` for preview builds" ); } - let package_versions = if matches!(source.layout, SourceLayout::Next) { - let device_target = match &source.device_target { - AggregateDeviceTarget::Exact(target) => target.as_str(), - AggregateDeviceTarget::Undetermined(reason) => { - bail!( - "cannot resolve ROCm X package metadata without an exact device target: {reason}" - ) - } - }; - let rocm_version = select_rocm_version(channel, &rocm_versions, version_selector) - .with_context(|| { - let requested = version_selector.map_or_else( - || "latest compatible version".to_owned(), - RuntimeVersionSelector::describe, - ); - format!("no TheRock rocm package was found for {requested} in {index_url}") - })?; - // Chosen here, not by `uv`: see `published_pip_requirements`. - let torchaudio_versions = load_simple_index_versions( - paths, - index_url, - "torchaudio", - Some(wheel_compatibility), - download_timeout_secs, - )?; - let pin = torchaudio_pin( - &torchaudio_versions, - &rocm_version, - rocm_engine_vllm::verified_torchaudio_release(&rocm_version), - index_url, - )?; - // The canonical branch below budgets one `download_timeout_secs` per - // package (rocm, then torch, torchvision, torchaudio: 4 sequential HTTP - // fetches). `uv pip compile` still fetches metadata for all four - // packages in one subprocess call — the torchaudio listing above only - // chose its version — so it needs a comparable aggregate budget, not - // the single-fetch one: reusing the bare per-fetch value here would make - // a startup check that budgets 2s per fetch reliably time out a call - // doing 4 fetches' worth of work. That listing is one more fetch on top, - // with its own single-fetch budget. - let requirements = - published_pip_requirements(&rocm_version, &pin.version, device_target, include_devel); - let versions = resolve_published_pip_package_versions( - paths, - index_url, - &requirements, - wheel_compatibility, - download_timeout_secs.map(|secs| secs.saturating_mul(4)), - )?; - version_resolution_specs = Some(requirements); - torchaudio_selection = Some(pin); - versions - } else { - let torch_versions = load_simple_index_versions( - paths, - index_url, - "torch", - Some(wheel_compatibility), - download_timeout_secs, - )?; - let torchvision_versions = load_simple_index_versions( - paths, - index_url, - "torchvision", - Some(wheel_compatibility), - download_timeout_secs, - )?; - let torchaudio_versions = load_simple_index_versions( - paths, - index_url, - "torchaudio", - Some(wheel_compatibility), - download_timeout_secs, - )?; - select_matching_pip_package_versions( - channel, - &rocm_versions, - &torch_versions, - &torchvision_versions, - &torchaudio_versions, - version_selector, - ) - .with_context(|| { - let requested = version_selector.map_or_else( - || "latest compatible version".to_owned(), - RuntimeVersionSelector::describe, - ); - format!( - "no mutually compatible TheRock rocm, torch, torchvision, and torchaudio versions were found for {requested} in {index_url}" - ) - })? - }; - let latest_version = package_versions.rocm.clone(); // The repo's newest version for this channel, ignoring wheel availability. // Only meaningful when we auto-selected "latest" (no explicit request), so a // caller can warn when that newest version has no installable wheels. @@ -823,7 +850,126 @@ fn resolve_pip_runtime_from_index( } else { None }; - Ok(PipRuntimeResolution { + let package_versions = match plan_version_resolution(source.layout, uv.access, uv.find_present) + { + VersionResolution::ScrapeIndexes => { + let torch_versions = load_simple_index_versions( + paths, + index_url, + "torch", + Some(wheel_compatibility), + download_timeout_secs, + )?; + let torchvision_versions = load_simple_index_versions( + paths, + index_url, + "torchvision", + Some(wheel_compatibility), + download_timeout_secs, + )?; + let torchaudio_versions = load_simple_index_versions( + paths, + index_url, + "torchaudio", + Some(wheel_compatibility), + download_timeout_secs, + )?; + select_matching_pip_package_versions( + channel, + &rocm_versions, + &torch_versions, + &torchvision_versions, + &torchaudio_versions, + version_selector, + ) + .with_context(|| { + let requested = version_selector.map_or_else( + || "latest compatible version".to_owned(), + RuntimeVersionSelector::describe, + ); + format!( + "no mutually compatible TheRock rocm, torch, torchvision, and torchaudio versions were found for {requested} in {index_url}" + ) + })? + } + resolution @ (VersionResolution::Compile(_) | VersionResolution::Skip) => { + let device_target = match &source.device_target { + AggregateDeviceTarget::Exact(target) => target.as_str(), + AggregateDeviceTarget::Undetermined(reason) => { + bail!( + "cannot resolve ROCm X package metadata without an exact device target: {reason}" + ) + } + }; + let rocm_version = select_rocm_version(channel, &rocm_versions, version_selector) + .with_context(|| { + let requested = version_selector.map_or_else( + || "latest compatible version".to_owned(), + RuntimeVersionSelector::describe, + ); + format!("no TheRock rocm package was found for {requested} in {index_url}") + })?; + // Chosen here, not by `uv`: see `published_pip_requirements`. + let torchaudio_versions = load_simple_index_versions( + paths, + index_url, + "torchaudio", + Some(wheel_compatibility), + download_timeout_secs, + )?; + let pin = torchaudio_pin( + &torchaudio_versions, + &rocm_version, + rocm_engine_vllm::verified_torchaudio_release(&rocm_version), + index_url, + )?; + let requirements = published_pip_requirements( + &rocm_version, + &pin.version, + device_target, + include_devel, + ); + let VersionResolution::Compile(choice) = resolution else { + // No `uv` to run and none to fetch: report what is known, and + // leave the versions `uv` would choose to the install. + return Ok(PipRuntimePlan::NeedsUv(UnresolvedPipRuntime { + family: family_resolution.family.clone(), + family_source: family_resolution.source.clone(), + index_url: index_url.to_owned(), + layout: source.layout, + rocm_version, + newest_repo_version, + version_resolution_specs: requirements, + torchaudio_pin: pin, + device_target: source.device_target.clone(), + published_device_targets: source.published_device_targets.clone(), + })); + }; + // The canonical branch above budgets one `download_timeout_secs` per + // package (rocm, then torch, torchvision, torchaudio: 4 sequential HTTP + // fetches). `uv pip compile` still fetches metadata for all four + // packages in one subprocess call — the torchaudio listing above only + // chose its version — so it needs a comparable aggregate budget, not + // the single-fetch one: reusing the bare per-fetch value here would make + // a startup check that budgets 2s per fetch reliably time out a call + // doing 4 fetches' worth of work. That listing is one more fetch on top, + // with its own single-fetch budget. + let compile_uv = uv_for_compile(choice, uv.acquire)?; + let versions = resolve_published_pip_package_versions( + paths, + &compile_uv, + index_url, + &requirements, + wheel_compatibility, + download_timeout_secs.map(|secs| secs.saturating_mul(4)), + )?; + version_resolution_specs = Some(requirements); + torchaudio_selection = Some(pin); + versions + } + }; + let latest_version = package_versions.rocm.clone(); + Ok(PipRuntimePlan::Resolved(PipRuntimeResolution { family: family_resolution.family.clone(), family_source: family_resolution.source.clone(), index_url: index_url.to_owned(), @@ -835,7 +981,7 @@ fn resolve_pip_runtime_from_index( package_versions, device_target: source.device_target.clone(), published_device_targets: source.published_device_targets.clone(), - }) + })) } pub(in crate::therock) fn select_rocm_version( @@ -937,19 +1083,22 @@ fn parse_uv_compiled_package_versions(output: &str) -> Result, ) -> Result { - let uv = - ensure_uv_binary(paths).context("failed to acquire uv for ROCm X metadata resolution")?; let python_version = uv_python_version(compatibility)?; let python_platform = uv_python_platform(compatibility)?; let requirements = format!("{}\n", requirements.join("\n")); - let mut child = Command::new(&uv) + let mut child = Command::new(uv) .args([ "pip", "compile", @@ -1369,6 +1518,7 @@ mod tests { use super::super::super::test_support::test_runtime_manifest; use super::*; use crate::runtime_registry::{InstalledRuntimeManifest, wheel_composition_includes_devel}; + use std::path::PathBuf; /// A representative slice of what the canonical aggregate index publishes. /// Notably it has no `gfx943`: MI300 steppings other than `gfx942` have no @@ -2322,4 +2472,690 @@ mod tests { assert!(hint.contains("--format tarball")); } } + + // ---- who may fetch `uv`: the real resolution --------------------------- + + /// Serves fixed pages over HTTP until dropped. A request for a path it does + /// not hold is a 404. + #[cfg(unix)] + struct IndexServer { + base: String, + stop: std::sync::Arc, + thread: Option>, + } + + #[cfg(unix)] + impl IndexServer { + fn start(pages: Vec<(String, String)>) -> Self { + use std::io::{Read, Write}; + use std::sync::atomic::Ordering; + + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind"); + let port = listener.local_addr().expect("local addr").port(); + listener.set_nonblocking(true).expect("nonblocking"); + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let stopped = std::sync::Arc::clone(&stop); + let thread = std::thread::spawn(move || { + while !stopped.load(Ordering::SeqCst) { + let Ok((mut stream, _)) = listener.accept() else { + std::thread::sleep(Duration::from_millis(5)); + continue; + }; + let _ = stream.set_nonblocking(false); + let _ = stream.set_read_timeout(Some(Duration::from_secs(5))); + let mut request = Vec::new(); + let mut buffer = [0_u8; 1024]; + while !request.windows(4).any(|window| window == b"\r\n\r\n") { + match stream.read(&mut buffer) { + Ok(0) | Err(_) => break, + Ok(read) => request.extend_from_slice(&buffer[..read]), + } + } + let request = String::from_utf8_lossy(&request); + let path = request.split_whitespace().nth(1).unwrap_or_default(); + let page = pages.iter().find(|(served, _)| served == path); + let (status, body) = match page { + Some((_, body)) => ("200 OK", body.as_str()), + None => ("404 Not Found", ""), + }; + let _ = write!( + stream, + "HTTP/1.1 {status}\r\nContent-Type: text/html\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + let _ = stream.flush(); + } + }); + Self { + base: format!("http://127.0.0.1:{port}"), + stop, + thread: Some(thread), + } + } + } + + #[cfg(unix)] + impl Drop for IndexServer { + fn drop(&mut self) { + self.stop.store(true, std::sync::atomic::Ordering::SeqCst); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } + } + + #[cfg(unix)] + fn wheel_links(package: &str, versions: &[&str]) -> String { + use std::fmt::Write as _; + + versions.iter().fold(String::new(), |mut html, version| { + let file = format!("{package}-{version}-py3-none-any.whl"); + let _ = writeln!(html, "{file}"); + html + }) + } + + /// One canonical and one ROCm 10 index, as the preview's resolution reads them. + #[cfg(unix)] + fn two_layout_index() -> IndexServer { + let mut pages = Vec::new(); + for (layout, stack) in [ + ( + "current", + [ + ("rocm", "7.10.0"), + ("torch", "2.9.0+rocm7.10.0"), + ("torchvision", "0.24.0+rocm7.10.0"), + ("torchaudio", "2.9.0+rocm7.10.0"), + ], + ), + ( + "next", + [ + ("rocm", "10.0.0"), + ("torch", "2.10.0+rocm10.0.0"), + ("torchvision", "0.25.0+rocm10.0.0"), + ("torchaudio", "2.9.1+rocm10.0.0"), + ], + ), + ] { + for (package, version) in stack { + pages.push(( + format!("/{layout}/{package}/"), + wheel_links(package, &[version]), + )); + } + } + IndexServer::start(pages) + } + + /// A `uv` that logs its arguments, reads the requirements it is piped, and + /// answers `uv pip compile` with the ROCm 10 stack above. + #[cfg(unix)] + fn write_logging_uv(dir: &Path) -> (PathBuf, PathBuf) { + let log = dir.join("uv-runs.log"); + let uv = dir.join("uv"); + std::fs::create_dir_all(dir).expect("create the uv folder"); + let script = format!( + "#!/bin/sh\necho \"$@\" >> '{}'\ncat > /dev/null\nprintf '%s\\n' 'rocm==10.0.0' 'torch==2.10.0+rocm10.0.0' 'torchvision==0.25.0+rocm10.0.0' 'torchaudio==2.9.1+rocm10.0.0'\n", + log.display() + ); + rocm_core::fake_executable::write_fake_executable(&uv, script) + .expect("plant the stand-in uv"); + (uv, log) + } + + /// What one cell of the matrix did: the answer, and what it touched. + #[cfg(unix)] + struct UvAccessRun { + plan: Result, + /// Times a present `uv` was looked for. + looked: usize, + /// Times the managed `uv` was acquired (downloaded, for a real one). + acquired: usize, + /// The arguments `uv` was run with, one run per line. + uv_runs: String, + /// Whether the managed tools folder, where a download goes, exists. + tools_folder: bool, + } + + /// Runs the real resolution against the served index for `layout`, with `uv` + /// "present" at the logging stand-in or not, and `acquire` answering for the + /// managed `uv` with that stand-in or with a failure. + #[cfg(unix)] + fn resolve_through_policy( + layout: SourceLayout, + access: UvAccess, + uv_present: bool, + acquire_succeeds: bool, + ) -> UvAccessRun { + let (root, paths) = super::super::super::test_support::test_paths("uv-access"); + let (uv, log) = write_logging_uv(&root.join("bin")); + let server = two_layout_index(); + let (index, python_tag) = match layout { + SourceLayout::Canonical => (format!("{}/current", server.base), "cp312"), + SourceLayout::Next => (format!("{}/next", server.base), "cp314"), + }; + let source = ResolvedAggregateWheelSource { + index_url: index, + layout, + device_target: AggregateDeviceTarget::Exact("gfx1200".to_owned()), + published_device_targets: vec!["gfx1200".to_owned()], + }; + let compatibility = WheelCompatibility { + python_tag: python_tag.to_owned(), + platform_tags: vec!["linux_x86_64".to_owned(), "any".to_owned()], + }; + let looked = std::cell::Cell::new(0); + let acquired = std::cell::Cell::new(0); + let find_present = || { + looked.set(looked.get() + 1); + uv_present.then(|| uv.clone()) + }; + let acquire = || { + acquired.set(acquired.get() + 1); + if acquire_succeeds { + Ok(uv.clone()) + } else { + bail!("the managed uv could not be downloaded") + } + }; + let policy = UvPolicy { + access, + find_present: &find_present, + acquire: &acquire, + }; + let plan = resolve_pip_runtime_from_index( + &paths, + TheRockChannel::Release, + &super::super::super::test_support::family_for_test("gfx1200", Some("gfx1200")), + &source, + &compatibility, + None, + false, + None, + &policy, + ); + let run = UvAccessRun { + plan, + looked: looked.get(), + acquired: acquired.get(), + uv_runs: std::fs::read_to_string(&log).unwrap_or_default(), + tools_folder: rocm_core::managed_tools_dir(&paths.data_dir).exists(), + }; + let _ = std::fs::remove_dir_all(&root); + run + } + + #[cfg(unix)] + fn stack_of(plan: &PipRuntimePlan) -> &TheRockPipPackageVersions { + match plan { + PipRuntimePlan::Resolved(resolution) => &resolution.package_versions, + PipRuntimePlan::NeedsUv(_) => { + panic!("the plan stopped before the versions were chosen") + } + } + } + + /// The resolution itself, every layout and access with a `uv` present or not, + /// each row asserting the outcome, what was touched, and for the row that stops + /// the message the preview prints about it. It runs the real resolution against + /// a served index, so a preview that fetched `uv` would show up as an acquire + /// or a tools folder, and a real install that stopped fetching it would show up + /// as a plan with no versions. + #[cfg(unix)] + #[test] + fn a_preview_never_acquires_uv_and_a_real_resolution_still_does() { + use SourceLayout::{Canonical, Next}; + use UvAccess::{Acquire, PresentOnly}; + + // ROCm 10, a preview, no uv present: the one row that stops. It asks + // whether one is present and touches nothing, so no `uv` ran, none was + // acquired and no tools folder was made. + let run = resolve_through_policy(Next, PresentOnly, false, false); + let Ok(PipRuntimePlan::NeedsUv(unresolved)) = &run.plan else { + panic!("expected a plan that stopped before uv, got {:?}", run.plan); + }; + assert_eq!( + ( + run.looked, + run.acquired, + run.uv_runs.as_str(), + run.tools_folder + ), + (1, 0, "", false), + "a preview with no uv present must acquire nothing and run nothing" + ); + // What it knows without `uv`: the ROCm version, the pinned torchaudio and + // the requirement lines `uv` would be given. + assert_eq!(unresolved.rocm_version, "10.0.0"); + assert_eq!(unresolved.torchaudio_pin.version, "2.9.1+rocm10.0.0"); + assert_eq!( + unresolved.version_resolution_specs, + [ + "rocm[libraries,device-gfx1200]==10.0.0", + "torch[device-gfx1200]", + "torchvision[device-gfx1200]", + "torchaudio==2.9.1+rocm10.0.0", + ] + ); + // And the message says what it did not do, in the same row. + let text = preview_text_with_prefix(unresolved, None); + for line in [ + " runtime_key: not resolved by this preview", + " package_specs: not resolved by this preview", + " command: not resolved by this preview", + " manifest: not resolved by this preview", + ] { + assert!( + text.lines().any(|l| l == line), + "missing `{line}` in:\n{text}" + ); + } + assert!( + text.contains("resolution_note: no uv is installed"), + "the preview must say why:\n{text}" + ); + assert!( + !text.contains("2.10.0+rocm10.0.0") && !text.contains("0.25.0+rocm10.0.0"), + "no version only uv could have chosen may be printed:\n{text}" + ); + + // ROCm 10, a preview, a uv present: it runs, once, and the plan is the + // one a real install reaches. Nothing is acquired. + let run = resolve_through_policy(Next, PresentOnly, true, false); + let stack = stack_of(run.plan.as_ref().expect("resolves with the present uv")); + assert_eq!( + (stack.torch.as_str(), stack.compatibility_key.as_str()), + ("2.10.0+rocm10.0.0", "10.0.0") + ); + assert_eq!( + (run.looked, run.acquired, run.tools_folder), + (1, 0, false), + "a preview with a uv present must not acquire another" + ); + assert_eq!(run.uv_runs.lines().count(), 1, "{}", run.uv_runs); + assert!(run.uv_runs.starts_with("pip compile "), "{}", run.uv_runs); + + // ROCm 10, a real install or an update: the managed uv is acquired, once, + // whether or not one is present (acquiring is what finds an installed + // one), and never looked for first. This is what must not change. + for uv_present in [false, true] { + let run = resolve_through_policy(Next, Acquire, uv_present, true); + let stack = stack_of(run.plan.as_ref().expect("resolves with the acquired uv")); + assert_eq!(stack.torch, "2.10.0+rocm10.0.0"); + assert_eq!( + (run.looked, run.acquired), + (0, 1), + "an install acquires uv, with one present: {uv_present}" + ); + assert_eq!(run.uv_runs.lines().count(), 1, "{}", run.uv_runs); + } + // A failed acquisition fails the resolution with the context it always had, + // and no plan stands in for it. + let run = resolve_through_policy(Next, Acquire, false, false); + let error = format!( + "{:#}", + run.plan.expect_err("an install cannot go without uv") + ); + assert!( + error.contains("failed to acquire uv for ROCm X metadata resolution: the managed uv could not be downloaded"), + "{error}" + ); + assert_eq!((run.looked, run.acquired), (0, 1)); + + // The canonical layout needs no uv for either access, present or not: the + // versions are read off the simple indexes, and nothing is looked for, + // acquired or run. + for access in [Acquire, PresentOnly] { + for uv_present in [false, true] { + let run = resolve_through_policy(Canonical, access, uv_present, true); + let stack = stack_of(run.plan.as_ref().expect("resolves from the indexes")); + assert_eq!( + (stack.rocm.as_str(), stack.torch.as_str()), + ("7.10.0", "2.9.0+rocm7.10.0") + ); + assert_eq!( + ( + run.looked, + run.acquired, + run.uv_runs.as_str(), + run.tools_folder + ), + (0, 0, "", false), + "{access:?}, uv present: {uv_present}" + ); + } + } + } + + // ---- the plan a preview without uv prints -------------------------------- + // + // Built by hand rather than from a served index, so that these run on every + // platform, the Windows CI job included: the wording and the shape of the plan + // do not depend on a stand-in `uv`. + + const PLAN_INDEX: &str = "http://index.test/next"; + + fn rocm10_specs() -> Vec { + published_pip_requirements("10.0.0", "2.9.1+rocm10.0.0", "gfx1200", false) + } + + fn rocm10_pin() -> TorchaudioPin { + torchaudio_pin(&["2.9.1+rocm10.0.0".to_owned()], "10.0.0", None, PLAN_INDEX) + .expect("the one published torchaudio is pinned") + } + + /// What a ROCm 10 resolution knows before `uv` has run. + fn unresolved_runtime() -> UnresolvedPipRuntime { + UnresolvedPipRuntime { + family: "gfx1200".to_owned(), + family_source: "manifest".to_owned(), + index_url: PLAN_INDEX.to_owned(), + layout: SourceLayout::Next, + rocm_version: "10.0.0".to_owned(), + newest_repo_version: None, + version_resolution_specs: rocm10_specs(), + torchaudio_pin: rocm10_pin(), + device_target: AggregateDeviceTarget::Exact("gfx1200".to_owned()), + published_device_targets: vec!["gfx1200".to_owned()], + } + } + + /// The same resolution once `uv` has chosen the versions. + fn resolved_runtime() -> PipRuntimeResolution { + let unresolved = unresolved_runtime(); + PipRuntimeResolution { + family: unresolved.family, + family_source: unresolved.family_source, + index_url: unresolved.index_url, + layout: unresolved.layout, + latest_version: "10.0.0".to_owned(), + newest_repo_version: None, + version_resolution_specs: Some(unresolved.version_resolution_specs), + torchaudio_pin: Some(unresolved.torchaudio_pin), + package_versions: TheRockPipPackageVersions { + rocm: "10.0.0".to_owned(), + torch: "2.10.0+rocm10.0.0".to_owned(), + torchvision: "0.25.0+rocm10.0.0".to_owned(), + torchaudio: "2.9.1+rocm10.0.0".to_owned(), + compatibility_key: "10.0.0".to_owned(), + }, + device_target: unresolved.device_target, + published_device_targets: unresolved.published_device_targets, + } + } + + /// The Python a clean host would prepare, and the wheel tags that follow from it. + fn clean_host_python() -> ( + super::super::super::python::PlannedPython, + WheelCompatibility, + ) { + use super::super::super::python::{PlannedPython, TagCheck, python_requirement}; + ( + PlannedPython::WouldPrepareManaged { + version: "3.14".to_owned(), + requirement: python_requirement(SourceLayout::Next), + tag_check: TagCheck::Checked, + }, + WheelCompatibility { + python_tag: "cp314".to_owned(), + platform_tags: vec!["linux_x86_64".to_owned(), "any".to_owned()], + }, + ) + } + + fn preview_text_with_prefix( + unresolved: &UnresolvedPipRuntime, + prefix: Option<&Path>, + ) -> String { + let (python, wheel_compatibility) = clean_host_python(); + super::super::super::wheel_plan::unresolved_wheel_preview( + &super::super::super::wheel_plan::UnresolvedPreview { + channel: TheRockChannel::Release, + version_selector: None, + unresolved, + device_target_override: None, + prefix, + host_version_newer: None, + python: &python, + wheel_compatibility: &wheel_compatibility, + }, + ) + } + + /// The whole text, so that no line can be added, dropped or reworded without + /// this test saying so. Everything made from the torch and torchvision + /// versions reads `not resolved by this preview`, and `resolution_note` says + /// why, in the voice of `python_note` beside it. + #[test] + fn the_plan_a_preview_without_uv_prints_is_exactly_this() { + let unresolved = unresolved_runtime(); + let text = preview_text_with_prefix(&unresolved, None); + let generation = unresolved.layout.generation(); + let pin = unresolved.torchaudio_pin.describe(); + let expected = format!( + "\ +sdk install + summary: rocm-cli will install the ROCm SDK and matching PyTorch packages for this Python and operating system + channel: release + canonical_source: {PLAN_INDEX} + selected_rocm_version: 10.0.0 + build_date: not encoded in stable version + source_layout_generation: {generation} + format: wheel + target_family: gfx1200 + target_family_source: manifest + device_target: gfx1200 + index_url: {PLAN_INDEX} + latest_compatible_version: 10.0.0 + compatibility_key: 10.0.0 + target: not resolved by this preview + runtime_key: not resolved by this preview + python_launcher: not prepared by this preview + python_source: managed + python_note: no usable Python was found; a real install would use uv to install managed Python 3.14 and save its path in the configuration, which this preview did not do + resolution_note: no uv is installed, so a real install would fetch it and run `uv pip compile` to choose the torch and torchvision versions and to check that the published packages install together, which this preview did not do; the package specs, runtime key, install folder, manifest and command depend on those versions and are not resolved here + python_wheel_tag: cp314 + platform_wheel_tags: linux_x86_64,any + version_resolution_specs: rocm[libraries,device-gfx1200]==10.0.0 torch[device-gfx1200] torchvision[device-gfx1200] torchaudio==2.9.1+rocm10.0.0 + torchaudio_selection: {pin} + package_specs: not resolved by this preview + package_policy: resolve the pinned target-complete rocm, torch, torchvision, and torchaudio plan from published package metadata, then install it in one uv transaction + mode: dry-run + command: not resolved by this preview + activation: use the managed venv Python; TheRock libraries are resolved from that venv by rocm_sdk.initialize_process + manifest: not resolved by this preview +" + ); + assert_eq!(text, expected); + } + + /// A `--prefix` does not depend on the versions, so it is the plan's target + /// even though the runtime key is not known; the key is what would otherwise + /// name the folder. + #[test] + fn an_explicit_prefix_is_the_target_of_a_preview_without_uv() { + let (root, _) = super::super::super::test_support::test_paths("uv-access-prefix"); + std::fs::create_dir_all(&root).expect("create the prefix parent"); + let prefix = root.join("my runtime"); + let text = preview_text_with_prefix(&unresolved_runtime(), Some(&prefix)); + let resolved = rocm_core::resolve_path_through_symlinks(&prefix); + let _ = std::fs::remove_dir_all(&root); + assert!( + text.lines() + .any(|line| line == format!(" target: {}", resolved.display())), + "{text}" + ); + assert!( + text.lines() + .any(|line| line == " runtime_key: not resolved by this preview"), + "{text}" + ); + } + + /// The keys a plan prints, in order. + fn plan_keys(text: &str) -> Vec<&str> { + text.lines() + .filter_map(|line| line.strip_prefix(" ")) + .filter_map(|line| line.split_once(": ").map(|(key, _)| key)) + .collect() + } + + /// One writer serves both plans, so a line the resolved plan gains cannot be + /// missing from the other: the two print the same keys in the same order, and + /// the unresolved one adds only `resolution_note`. + #[test] + fn a_resolved_and_an_unresolved_plan_print_the_same_keys() { + use super::super::super::wheel_plan::{ + PlanStack, ResolvedStack, WheelPlan, write_dry_run_tail, write_wheel_plan, + }; + + let unresolved = preview_text_with_prefix(&unresolved_runtime(), None); + let resolution = resolved_runtime(); + let composition = wheel_runtime_composition(&resolution, &resolution.device_target, false); + let runtime_key = super::super::super::wheel_runtime_key( + TheRockChannel::Release, + &resolution.latest_version, + &composition, + ); + let install_root = PathBuf::from("/runtimes").join(&runtime_key); + let manifest_path = PathBuf::from("/manifests").join(&runtime_key); + let (python, compatibility) = clean_host_python(); + let plan = WheelPlan { + channel: TheRockChannel::Release, + version_selector: None, + index_url: &resolution.index_url, + layout_generation: resolution.layout.generation(), + latest_version: &resolution.latest_version, + family: &resolution.family, + family_source: &resolution.family_source, + device_target: &resolution.device_target, + host_version_newer: None, + python: &python, + wheel_compatibility: &compatibility, + version_resolution_specs: resolution.version_resolution_specs.as_deref(), + torchaudio_pin: resolution.torchaudio_pin.as_ref(), + no_wheel_warning: None, + stack: PlanStack::Resolved(ResolvedStack { + compatibility_key: &resolution.package_versions.compatibility_key, + install_root: &install_root, + runtime_key: &runtime_key, + package_specs: &composition.package_specs, + manifest_path: &manifest_path, + }), + }; + let mut resolved = String::new(); + write_wheel_plan(&mut resolved, &plan); + write_dry_run_tail(&mut resolved, &plan); + + let mut expected = plan_keys(&resolved); + let at = expected + .iter() + .position(|key| *key == "python_note") + .expect("a python_note: this host would prepare Python") + + 1; + expected.insert(at, "resolution_note"); + assert_eq!(plan_keys(&unresolved), expected); + // And the resolved plan carries what the unresolved one cannot: the + // versions `uv` chose, in the package specs, and a real folder and key. + assert!( + resolved.contains("torch[device-gfx1200]==2.10.0+rocm10.0.0"), + "{resolved}" + ); + assert!(!resolved.contains("not resolved"), "{resolved}"); + assert!(!resolved.contains("resolution_note"), "{resolved}"); + } + + // ---- the recorded device target ------------------------------------------ + + /// The rule in one place: nothing recorded leaves the detected answer as it + /// was, a recorded target is judged against what the source publishes, and an + /// unpublished one is `Undetermined` with the reason rather than a guess. + #[test] + fn a_recorded_device_target_is_judged_against_what_the_source_publishes() { + let published = vec!["gfx1200".to_owned()]; + let undetected = AggregateDeviceTarget::Undetermined("no GPU".to_owned()); + let exact = AggregateDeviceTarget::Exact("gfx1200".to_owned()); + // Nothing recorded: the detected answer stands, undetermined included. + assert_eq!( + AggregateDeviceTarget::prefer_recorded(&undetected, None, "gfx120X-all", &published), + undetected + ); + assert_eq!( + AggregateDeviceTarget::prefer_recorded(&exact, None, "gfx120X-all", &published), + exact + ); + // Recorded and published: it wins over an undetermined host probe. + assert_eq!( + AggregateDeviceTarget::prefer_recorded( + &undetected, + Some("gfx1200"), + "gfx120X-all", + &published + ), + exact + ); + // Recorded but not published: undetermined, whatever the host probe said. + let refused = AggregateDeviceTarget::prefer_recorded( + &exact, + Some("gfx1201"), + "gfx120X-all", + &published, + ); + assert!( + matches!(&refused, AggregateDeviceTarget::Undetermined(reason) if reason.contains("gfx1201")), + "{refused:?}" + ); + } + + /// The same rule reaches a preview that stopped before `uv`: an override the + /// source does not publish shows as `` with its reason, not as the + /// target the resolution detected. Red if `unresolved_wheel_preview` stops + /// using the helper and keeps the detected target. + #[test] + fn a_preview_without_uv_applies_a_recorded_device_target_through_the_helper() { + let unresolved = unresolved_runtime(); + let (python, wheel_compatibility) = clean_host_python(); + let text = super::super::super::wheel_plan::unresolved_wheel_preview( + &super::super::super::wheel_plan::UnresolvedPreview { + channel: TheRockChannel::Release, + version_selector: None, + unresolved: &unresolved, + device_target_override: Some("gfx1201"), + prefix: None, + host_version_newer: None, + python: &python, + wheel_compatibility: &wheel_compatibility, + }, + ); + assert!( + text.contains("\n device_target: \n"), + "{text}" + ); + assert!(text.contains("\n device_target_reason: "), "{text}"); + assert!(!text.contains("device_target: gfx1200"), "{text}"); + } + + /// No call site writes the rule by hand again: every one goes through + /// `AggregateDeviceTarget::prefer_recorded`, so a change to the rule reaches + /// the install plan, the preview and an update's plan together. A scan of the + /// three files that call it, red if one resolves a recorded target itself. + #[test] + fn the_recorded_device_target_rule_is_written_once() { + for (name, source) in [ + ("therock.rs", include_str!("../../therock.rs")), + ("therock/update.rs", include_str!("../update.rs")), + ("therock/wheel_plan.rs", include_str!("../wheel_plan.rs")), + ] { + assert!( + source.contains("AggregateDeviceTarget::prefer_recorded("), + "{name} no longer resolves its device target through the helper" + ); + assert!( + !source.contains("AggregateDeviceTarget::resolve("), + "{name} resolves a device target by hand; use AggregateDeviceTarget::prefer_recorded" + ); + } + } } diff --git a/apps/rocm/src/therock/update.rs b/apps/rocm/src/therock/update.rs index 08060f6deb..ccfea9278a 100644 --- a/apps/rocm/src/therock/update.rs +++ b/apps/rocm/src/therock/update.rs @@ -460,17 +460,12 @@ fn resolve_latest_for_manifest( // a fresh host probe. Planning must predict the key an apply will // produce, and re-probing would disagree with the installed runtime on // any host whose GPU is absent, hidden, or simply a second card. - let device_target = - wheel_composition_device_target(manifest.wheel_composition.as_ref()).map_or_else( - || resolution.device_target.clone(), - |target| { - AggregateDeviceTarget::resolve( - Some(target), - &resolution.family, - &resolution.published_device_targets, - ) - }, - ); + let device_target = AggregateDeviceTarget::prefer_recorded( + &resolution.device_target, + wheel_composition_device_target(manifest.wheel_composition.as_ref()), + &resolution.family, + &resolution.published_device_targets, + ); // No exact target means no reproducible composition, so freshness // falls back to the version comparison rather than demanding a repair // this host could not perform. diff --git a/apps/rocm/src/therock/wheel_plan.rs b/apps/rocm/src/therock/wheel_plan.rs new file mode 100644 index 0000000000..fe60fafad7 --- /dev/null +++ b/apps/rocm/src/therock/wheel_plan.rs @@ -0,0 +1,333 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The `sdk install` plan a wheel install prints, key by key. +//! +//! A preview prints it and stops; a real install prints it as the head of its +//! own output. [`write_wheel_plan`] writes the lines both share, and +//! [`write_dry_run_tail`] the lines only a preview adds. +//! +//! One writer serves a resolved plan and one that could not be resolved, so the +//! two cannot drift apart: a line the plan gains appears in both. They differ +//! only in the lines made from the package versions `uv pip compile` chooses on +//! the ROCm 10 layout (`package_specs`, `runtime_key`, `target`, `manifest` and +//! `command`). [`PlanStack::NeedsUv`] says those are not resolved, and why, +//! where [`PlanStack::Resolved`] carries their values. + +use super::python::{PlannedPython, WheelCompatibility}; +use super::resolve::{AggregateDeviceTarget, TorchaudioPin, UnresolvedPipRuntime}; +use super::source::{RuntimeVersionSelector, TheRockChannel}; +use super::{ + no_wheel_warning_for, render_canonical_provenance, runtime_version_display, venv_python_path, + wheel_host_version_note, +}; +use crate::shell_quote::shell_quote; +use rocm_core::{uv_pip_install_base, uv_venv_args}; +use std::fmt::Write as _; +use std::path::Path; + +/// The value printed for a line that depends on the package versions `uv` has not +/// chosen. +pub(super) const NOT_RESOLVED: &str = "not resolved by this preview"; + +/// The facts the plan prints, from a resolution and the Python decision. +pub(super) struct WheelPlan<'a> { + pub(super) channel: TheRockChannel, + pub(super) version_selector: Option<&'a RuntimeVersionSelector>, + pub(super) index_url: &'a str, + pub(super) layout_generation: &'a str, + /// The `rocm` version the plan is for. + pub(super) latest_version: &'a str, + pub(super) family: &'a str, + pub(super) family_source: &'a str, + pub(super) device_target: &'a AggregateDeviceTarget, + /// The host's ROCm when it is newer than `latest_version`. + pub(super) host_version_newer: Option<&'a str>, + pub(super) python: &'a PlannedPython, + pub(super) wheel_compatibility: &'a WheelCompatibility, + /// The requirement lines version resolution asked `uv` about. `None` on the + /// canonical layout, which resolves without `uv`. + pub(super) version_resolution_specs: Option<&'a [String]>, + pub(super) torchaudio_pin: Option<&'a TorchaudioPin>, + pub(super) no_wheel_warning: Option<&'a str>, + pub(super) stack: PlanStack<'a>, +} + +/// Everything in the plan that is made from the package versions. +pub(super) enum PlanStack<'a> { + /// The versions are chosen; these are what they produce. + Resolved(ResolvedStack<'a>), + /// A preview of the ROCm 10 layout with no `uv`: the versions were not chosen, + /// so nothing made from them can be printed. + NeedsUv { + /// The install folder when the user named one with `--prefix`: it does not + /// depend on the versions. Otherwise the runtime key names it. + explicit_install_root: Option<&'a Path>, + }, +} + +pub(super) struct ResolvedStack<'a> { + pub(super) compatibility_key: &'a str, + pub(super) install_root: &'a Path, + pub(super) runtime_key: &'a str, + pub(super) package_specs: &'a [String], + pub(super) manifest_path: &'a Path, +} + +/// The `compatibility_key` the install is recorded under is the `rocm` build every +/// package in the stack shares, so a plan that has not chosen the stack still +/// knows it: the `rocm` version. +const fn compatibility_key<'a>(plan: &'a WheelPlan<'a>) -> &'a str { + match &plan.stack { + PlanStack::Resolved(stack) => stack.compatibility_key, + PlanStack::NeedsUv { .. } => plan.latest_version, + } +} + +/// Writes the plan from `sdk install` to `package_policy` and the repository +/// warning. A real install carries on from here; a preview adds +/// [`write_dry_run_tail`]. +pub(super) fn write_wheel_plan(output: &mut String, plan: &WheelPlan<'_>) { + let _ = writeln!(output, "sdk install"); + let _ = writeln!( + output, + " summary: rocm-cli will install the ROCm SDK and matching PyTorch packages for this Python and operating system" + ); + render_canonical_provenance( + output, + plan.channel, + plan.index_url, + plan.layout_generation, + plan.latest_version, + ); + let _ = writeln!(output, " format: wheel"); + if let Some(selector) = plan.version_selector { + let _ = writeln!(output, " requested: {}", selector.describe()); + } + let _ = writeln!(output, " target_family: {}", plan.family); + let _ = writeln!(output, " target_family_source: {}", plan.family_source); + let _ = writeln!(output, " device_target: {}", plan.device_target.as_str()); + if let Some(reason) = plan.device_target.reason() { + let _ = writeln!(output, " device_target_reason: {reason}"); + } + let _ = writeln!(output, " index_url: {}", plan.index_url); + let _ = writeln!( + output, + " latest_compatible_version: {}", + runtime_version_display(plan.latest_version) + ); + if let Some(host_version) = plan.host_version_newer { + let _ = writeln!( + output, + " version_note: {}", + wheel_host_version_note(host_version, &runtime_version_display(plan.latest_version)) + ); + } + let _ = writeln!( + output, + " compatibility_key: {}", + runtime_version_display(compatibility_key(plan)) + ); + match &plan.stack { + PlanStack::Resolved(stack) => { + let _ = writeln!(output, " target: {}", stack.install_root.display()); + let _ = writeln!(output, " runtime_key: {}", stack.runtime_key); + } + PlanStack::NeedsUv { + explicit_install_root, + } => { + match explicit_install_root { + Some(root) => { + let _ = writeln!(output, " target: {}", root.display()); + } + None => { + let _ = writeln!(output, " target: {NOT_RESOLVED}"); + } + } + let _ = writeln!(output, " runtime_key: {NOT_RESOLVED}"); + } + } + let _ = writeln!( + output, + " python_launcher: {}", + plan.python.launcher_display() + ); + let _ = writeln!(output, " python_source: {}", plan.python.source().as_str()); + if let Some(note) = plan.python.note() { + let _ = writeln!(output, " python_note: {note}"); + } + if matches!(plan.stack, PlanStack::NeedsUv { .. }) { + let _ = writeln!(output, " resolution_note: {UNRESOLVED_NOTE}"); + } + let _ = writeln!( + output, + " python_wheel_tag: {}", + plan.wheel_compatibility.python_tag + ); + let _ = writeln!( + output, + " platform_wheel_tags: {}", + plan.wheel_compatibility.platform_tags.join(",") + ); + // What resolution asked for, printed beside what the install will ask for. + // The two are produced by different code paths from the same `include_devel` + // and must name the same extras; showing only the second would hide a + // resolve that constrained the version choice by a toolchain the user + // declined. Absent on the canonical layout, which resolves by scraping each + // package's index instead of compiling a requirement set. + if let Some(specs) = plan.version_resolution_specs { + let _ = writeln!(output, " version_resolution_specs: {}", specs.join(" ")); + } + // Printed from the same value that chose the pin, so the line cannot claim + // a verified build the requirements above do not name. + if let Some(pin) = plan.torchaudio_pin { + let _ = writeln!(output, " torchaudio_selection: {}", pin.describe()); + } + match &plan.stack { + PlanStack::Resolved(stack) => { + let _ = writeln!(output, " package_specs: {}", stack.package_specs.join(" ")); + } + PlanStack::NeedsUv { .. } => { + let _ = writeln!(output, " package_specs: {NOT_RESOLVED}"); + } + } + let _ = writeln!( + output, + " package_policy: resolve the pinned target-complete rocm, torch, torchvision, and torchaudio plan from published package metadata, then install it in one uv transaction" + ); + if let Some(warning) = plan.no_wheel_warning { + let _ = writeln!(output, " warning: {warning}"); + } +} + +/// What a preview says about the versions it did not choose. Written beside +/// `python_note`, in the same voice: what a real install would do, and that this +/// preview did not. +pub(super) const UNRESOLVED_NOTE: &str = "no uv is installed, so a real install would fetch it and run `uv pip compile` to choose the torch and torchvision versions and to check that the published packages install together, which this preview did not do; the package specs, runtime key, install folder, manifest and command depend on those versions and are not resolved here"; + +/// The progress line for the same decision as [`UNRESOLVED_NOTE`], in the voice of +/// the Python line (`... a real install would prepare Python X. This preview does +/// not.`). +pub(super) const UNRESOLVED_PROGRESS: &str = "No uv is installed; a real install would fetch it and resolve the torch and torchvision versions with `uv pip compile`. This preview does not."; + +/// The lines a preview adds after [`write_wheel_plan`]: the commands a real install +/// would run, and where it would record the runtime. +pub(super) fn write_dry_run_tail(output: &mut String, plan: &WheelPlan<'_>) { + let _ = writeln!(output, " mode: dry-run"); + match &plan.stack { + PlanStack::Resolved(stack) => { + let env_python = venv_python_path(stack.install_root); + let mut install_args = uv_pip_install_base(&env_python); + install_args.extend(["--index-url".to_owned(), plan.index_url.to_owned()]); + if matches!(plan.channel, TheRockChannel::Nightly) { + install_args.extend(["--prerelease".to_owned(), "allow".to_owned()]); + } + install_args.extend(stack.package_specs.iter().cloned()); + let venv_args = uv_venv_args(&plan.python.venv_python_arg(), stack.install_root); + // This line is offered as the command to run, so it has to be one a + // shell reads back as the argv it previews. The install root in it + // comes from a `--prefix` the local assistant can choose + // (`install_sdk_dry_run` is a model-callable tool), so a bare or + // double-quoted value here would be a model-chosen value in a line a + // user is invited to paste. + let venv_args_display = venv_args + .iter() + .map(|arg| shell_quote(arg)) + .collect::>() + .join(" "); + let install_args_display = install_args + .iter() + .map(|arg| shell_quote(arg)) + .collect::>() + .join(" "); + let _ = writeln!( + output, + " command: uv {venv_args_display} && uv {install_args_display}" + ); + } + PlanStack::NeedsUv { .. } => { + let _ = writeln!(output, " command: {NOT_RESOLVED}"); + } + } + let _ = writeln!( + output, + " activation: use the managed venv Python; TheRock libraries are resolved from that venv by rocm_sdk.initialize_process" + ); + match &plan.stack { + PlanStack::Resolved(stack) => { + let _ = writeln!(output, " manifest: {}", stack.manifest_path.display()); + } + PlanStack::NeedsUv { .. } => { + let _ = writeln!(output, " manifest: {NOT_RESOLVED}"); + } + } +} + +/// What [`unresolved_wheel_preview`] is made from. +pub(super) struct UnresolvedPreview<'a> { + pub(super) channel: TheRockChannel, + pub(super) version_selector: Option<&'a RuntimeVersionSelector>, + pub(super) unresolved: &'a UnresolvedPipRuntime, + /// An update apply's recorded exact arch, resolved against the targets the + /// source publishes exactly as a resolved plan does. + pub(super) device_target_override: Option<&'a str>, + /// The `--prefix` the user named. It does not depend on the versions, so it is + /// the plan's `target`; without one the runtime key names the install folder + /// and the `target` is not resolved. + pub(super) prefix: Option<&'a Path>, + /// The host's ROCm when it is newer than the version the plan is for. + pub(super) host_version_newer: Option<&'a str>, + pub(super) python: &'a PlannedPython, + pub(super) wheel_compatibility: &'a WheelCompatibility, +} + +/// The whole plan for a preview of the ROCm 10 layout that stopped before +/// `uv pip compile`: what was chosen without `uv`, written by the same writer as +/// every other plan, with the lines made from the unchosen versions marked. +pub(super) fn unresolved_wheel_preview(preview: &UnresolvedPreview<'_>) -> String { + let UnresolvedPreview { + channel, + version_selector, + unresolved, + device_target_override, + prefix, + host_version_newer, + python, + wheel_compatibility, + } = *preview; + let device_target = AggregateDeviceTarget::prefer_recorded( + &unresolved.device_target, + device_target_override, + &unresolved.family, + &unresolved.published_device_targets, + ); + let explicit_install_root = prefix.map(rocm_core::resolve_path_through_symlinks); + let no_wheel_warning = no_wheel_warning_for( + unresolved.newest_repo_version.as_deref(), + &unresolved.rocm_version, + ); + let plan = WheelPlan { + channel, + version_selector, + index_url: &unresolved.index_url, + layout_generation: unresolved.layout.generation(), + latest_version: &unresolved.rocm_version, + family: &unresolved.family, + family_source: &unresolved.family_source, + device_target: &device_target, + host_version_newer, + python, + wheel_compatibility, + version_resolution_specs: Some(&unresolved.version_resolution_specs), + torchaudio_pin: Some(&unresolved.torchaudio_pin), + no_wheel_warning: no_wheel_warning.as_deref(), + stack: PlanStack::NeedsUv { + explicit_install_root: explicit_install_root.as_deref(), + }, + }; + let mut output = String::new(); + write_wheel_plan(&mut output, &plan); + write_dry_run_tail(&mut output, &plan); + output +} diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index c8075f47a2..6f0601bb56 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -226,9 +226,10 @@ pub use service_record::{ pub use terminal::{quotable_in_single_quotes, strip_terminal_control_sequences}; pub use uv::{ DEFAULT_UV_TIMEOUT_SECS, DependencyViolation, UV_CACHE_DIR_ENV, UV_CACHE_DIR_OVERRIDE_ENV, - UvCacheSource, ViolationSubject, check_dependencies, ensure_uv_binary, split_local_version, - uv_binary_name, uv_cache_source, uv_command_env, uv_http_timeout_secs, uv_pip_check_args, - uv_pip_freeze_args, uv_pip_install_base, uv_venv_args, violation_subject, violations_requiring, + UvCacheSource, ViolationSubject, check_dependencies, ensure_uv_binary, find_uv_binary, + split_local_version, uv_binary_name, uv_cache_source, uv_command_env, uv_http_timeout_secs, + uv_pip_check_args, uv_pip_freeze_args, uv_pip_install_base, uv_venv_args, violation_subject, + violations_requiring, }; pub use vllm_oom::{ VLLM_OOM_DETECTED_PREFIX, VLLM_OOM_DIAGNOSE_COMMAND, vllm_oom_diagnose_symptom, diff --git a/crates/rocm-core/src/uv.rs b/crates/rocm-core/src/uv.rs index 5f3b17b34c..e89a0d1121 100644 --- a/crates/rocm-core/src/uv.rs +++ b/crates/rocm-core/src/uv.rs @@ -422,6 +422,38 @@ pub fn ensure_uv_binary(paths: &AppPaths) -> Result { ) } +/// A `uv` that is already available, without acquiring one: the binary named by +/// [`UV_BINARY_ENV`], or else the managed copy a previous install left behind. +/// +/// This is [`ensure_uv_binary`] stopped short of its download. It never downloads, +/// takes a lock, or creates a folder (not even the managed tools folder), and `None` +/// is the answer when `ensure_uv_binary` would have had to fetch one. A caller that +/// must leave the machine untouched (a preview) asks this instead and goes without +/// `uv` on `None`. A managed copy is only reported once it runs, which is the same +/// test `ensure_uv_binary` applies before it reuses one. +pub fn find_uv_binary(paths: &AppPaths) -> Option { + if let Some(path) = uv_binary_override() { + return Some(path); + } + find_managed_uv(paths, &uv_version()) +} + +/// The installed managed `uv` of `version`, when it runs. Looks and creates nothing. +fn find_managed_uv(paths: &AppPaths, version: &str) -> Option { + usable_uv_in(&managed_uv_install_dir(paths, version), uv_binary_name()) +} + +/// Where the managed `uv` of `version` is installed: `tools/uv/` under the +/// data folder. Shared by the install and by [`find_uv_binary`] so that a copy one +/// installs is the copy the other finds. +fn managed_uv_install_dir(paths: &AppPaths, version: &str) -> PathBuf { + managed_uv_root(paths).join(slug(version)) +} + +fn managed_uv_root(paths: &AppPaths) -> PathBuf { + managed_tools_dir(&paths.data_dir).join("uv") +} + /// Which `uv` release to install, resolved from the environment by /// [`ensure_uv_binary`] so the install itself never reads it. struct ManagedUvRelease { @@ -444,9 +476,9 @@ struct ManagedUvRelease { /// The lock is a leaf: nothing below acquires another lock, so it cannot take part in /// a lock-order inversion with the managed-launch or daemon-autostart locks. fn install_managed_uv(paths: &AppPaths, release: &ManagedUvRelease) -> Result { - let uv_root = managed_tools_dir(&paths.data_dir).join("uv"); + let uv_root = managed_uv_root(paths); let version_slug = slug(&release.version); - let install_dir = uv_root.join(&version_slug); + let install_dir = managed_uv_install_dir(paths, &release.version); let binary_name = uv_binary_name(); if let Some(existing) = usable_uv_in(&install_dir, binary_name) { @@ -1216,6 +1248,76 @@ All installed packages are compatible Ok(std::fs::read(&archive)?) } + /// Plants `script` as the managed `uv` of `version`, where an install would have + /// left it. + #[cfg(unix)] + fn plant_managed_uv(paths: &AppPaths, version: &str, script: &str) -> PathBuf { + use std::os::unix::fs::PermissionsExt; + + let dir = managed_uv_install_dir(paths, version); + std::fs::create_dir_all(&dir).expect("create the managed uv folder"); + let binary = dir.join(uv_binary_name()); + std::fs::write(&binary, script).expect("write the stand-in uv"); + std::fs::set_permissions(&binary, std::fs::Permissions::from_mode(0o755)) + .expect("make the stand-in uv executable"); + binary + } + + // The preview's question, answered on every platform: on a clean data folder + // there is no managed `uv`, and asking leaves no trace. `ensure_uv_binary` would + // have created `tools/uv` (for its lock) before it downloaded anything, which is + // the state this pins as untouched. + #[test] + fn looking_for_a_managed_uv_on_a_clean_host_finds_none_and_creates_nothing() { + let (root, paths) = crate::test_support::temp_app_paths("uv-find-clean"); + assert_eq!(find_managed_uv(&paths, "latest"), None); + assert!( + !managed_tools_dir(&paths.data_dir).exists(), + "looking for uv must not create the managed tools folder" + ); + let _ = std::fs::remove_dir_all(&root); + } + + // Unix-only: the stand-in `uv` is a shell script, and Windows can only run a real + // executable as `uv.exe`. + #[cfg(unix)] + #[test] + fn an_installed_managed_uv_is_found_where_the_install_puts_it() { + let (root, paths) = crate::test_support::temp_app_paths("uv-find-installed"); + let planted = plant_managed_uv(&paths, "latest", "#!/bin/sh\nexit 0\n"); + assert_eq!(find_managed_uv(&paths, "latest"), Some(planted.clone())); + // The install reuses the same copy rather than downloading: the lookup and the + // install agree on the location, so a copy one leaves is the copy the other + // finds. The release names a URL nothing listens on, so reaching for it would + // fail the call. + let reused = install_managed_uv( + &paths, + &ManagedUvRelease { + version: "latest".to_owned(), + asset: "uv-test.tar.gz".to_owned(), + url: "http://127.0.0.1:9/uv-test.tar.gz".to_owned(), + }, + ) + .expect("an installed uv is reused without a download"); + assert_eq!(reused, planted); + // Another version is a different copy. + assert_eq!(find_managed_uv(&paths, "0.0.1"), None); + let _ = std::fs::remove_dir_all(&root); + } + + #[cfg(unix)] + #[test] + fn a_managed_uv_that_does_not_run_is_not_reported() { + let (root, paths) = crate::test_support::temp_app_paths("uv-find-broken"); + plant_managed_uv(&paths, "latest", "#!/bin/sh\nexit 1\n"); + assert_eq!( + find_managed_uv(&paths, "latest"), + None, + "a copy `ensure_uv_binary` would replace is not one a preview can use" + ); + let _ = std::fs::remove_dir_all(&root); + } + // Unix-only: the fake `uv` is a shell script, and Windows can only run a real // executable as `uv.exe`. The lock and the re-check under test are // platform-independent. diff --git a/crates/rocm-dash-tui/src/agent/tools.rs b/crates/rocm-dash-tui/src/agent/tools.rs index 0c96d22d6a..0cf7d08d56 100644 --- a/crates/rocm-dash-tui/src/agent/tools.rs +++ b/crates/rocm-dash-tui/src/agent/tools.rs @@ -211,9 +211,12 @@ pub const TELEMETRY_TOOL_NAMES: [&str; 4] = [ // `update_check` can also, on a host with no usable Python on PATH, download a // managed `uv` and Python into the data folder and record that Python in the // saved config; `install_sdk_dry_run` prepares no Python and saves no Python -// path: on such a host it reports the Python a real install would prepare. (A -// ROCm 10 preview still resolves package versions with `uv` and can fetch the -// managed `uv` into the data folder.) The boundary type +// path: on such a host it reports the Python a real install would prepare. It +// fetches no `uv` either: a ROCm 10 preview runs a `uv` that is already +// installed to resolve the package versions and, with none, prints the lines +// that depend on them as not resolved, so it creates no `tools` folder in the +// data folder. (`update_check` still resolves with `uv` and can fetch it.) The +// boundary type // (`SharedRocmToolExecutor`) is plain data — importing it does NOT pull // `rocm-core` into this crate; the `agent` module stays the sole `rig` namer. // --------------------------------------------------------------------------- diff --git a/docs/architecture.md b/docs/architecture.md index 0a0ba71bf7..2e51fd54c3 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -101,7 +101,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) — shared CLI-output component (`shell_quote`): the CLI's name for `rocm-core`'s POSIX quoter, [`posix_shell_quote`](../crates/rocm-core/src/quoting.rs). `rocm-dash-tui` keeps a copy as `ui::exec::quote_display_arg` in [`exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs), since the crate-edge check forbids it `rocm-core`; a per-character walk and a property test here hold the two to the same output. It covers the remote `ssh` command line, the chat approval card and the plan for a free-form request, the install dry-run `command:` line, the [`service_hints.rs`](../apps/rocm/src/service_hints.rs) hints, the runtime-activation report, the `rocm services logs ` lines `rocm services` lists and the `rocm comfyui install --runtime-id` advice a failed ComfyUI reinstall prints — not every `rocm …` hint the CLI prints; older ones such as `rocm logs ` still interpolate their value bare. Also holds `has_terminal_control_character`, the predicate the chat tool-call guards refuse on. - [`storage.rs`](../apps/rocm/src/storage.rs) — full domain extraction. - [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and reads the os-release every one of these plans is built from through `rocm_core::hardware_root::real_os_release`, which has no copy here. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs). `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `rocm_core::hardware_root::real_os_release` and the three OpenMPI/libatomic/libnuma install hints ignore a simulated host root, and a scan that no source file of the binary reads os-release by hand. -- [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token` and `slugify`, plus `ensure_tarball_install_target_is_clean`, the refusal to extract a tarball into a root that already holds a payload. What "holds a payload" means is not defined here: it is `rocm_core::install_root_has_payload`, the one rule shared with `runtime_registry.rs` and `therock/update.rs`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read five of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key` and `wheel_runtime_key` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). +- [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the plan itself being written by `therock/wheel_plan.rs`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token` and `slugify`, plus `ensure_tarball_install_target_is_clean`, the refusal to extract a tarball into a root that already holds a payload. What "holds a payload" means is not defined here: it is `rocm_core::install_root_has_payload`, the one rule shared with `runtime_registry.rs` and `therock/update.rs`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read five of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key` and `wheel_runtime_key` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). - [`therock/version.rs`](../apps/rocm/src/therock/version.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): the version grammar behind runtime selection and update checks. `parse_version` is the strict gate an install is routed on; `parse_version_for_ordering`, `compare_version_strings` and `version_relation` implement the PEP 440 ordering and the "is this a different version" question; `parse_host_version` reads a host-reported ROCm version through the same parser; `runtime_version_display`, `runtime_version_build_date`, `normalize_requested_build_date` and the calendar helpers behind them handle build dates; `is_stable_runtime_version` and the `ParsedVersion`/`VersionOrderKey`/`VersionStage` types back them. The items `therock.rs` and its property-test module use are `pub(super)` (including the `ParsedVersion` and `VersionOrderKey` fields they read); the parsing and calendar helpers behind them are private; and `runtime_version_display` is `pub(crate)`, re-exported by `therock.rs` for the other modules that render versions. Depends on nothing else in `therock`. `therock_version_proptests.rs` stays declared from `therock.rs` because it also reaches update, wheel and registry code. - [`therock/io.rs`](../apps/rocm/src/therock/io.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): unique scratch directories (`linux_temp_dir`, `windows_temp_dir`), both `pub(super)`. The atomic file writes that used to live here are rocm-core's shared [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs), which [`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs), [`therock/update.rs`](../apps/rocm/src/therock/update.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) call directly. - [`therock/process.rs`](../apps/rocm/src/therock/process.rs) — full domain extraction, a child of `therock.rs` that depends on `therock/io.rs` for scratch directories (split out of `therock.rs`): running and capturing child commands (`run_command*`, `capture_command_output*`, `capture_python_stdout*`, `run_uv_progress_command`, and the Windows temp-file variants with `windows_child_path`), the bounded wait (`wait_with_output_bounded`), the progress-line output with its `--json` suppression guard (`SuppressProgressOutput`, `progress_line`), and unpacking the SDK tarball (`extract_tarball`, `extract_tarball_and_discard_archive`, which is why it sits here and not in `io.rs`: it spawns `tar` through `run_command`). Its `archive_properties` property tests run on Unix only. The process-wide environment lock these tests take is the single `PROCESS_ENV_TEST_LOCK` in [`test_support.rs`](../apps/rocm/src/test_support.rs), shared with the tests left in `therock.rs`. @@ -112,8 +112,10 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/probe.rs`](../apps/rocm/src/therock/probe.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/process.rs` and `therock/python.rs` (split out of `therock.rs`): asking an installed runtime about itself through its own interpreter. `probe_rocm_sdk_runtime` runs `ROCM_SDK_PROBE_SCRIPT` and parses what `rocm_sdk` reports into the `RocmSdkPythonProbe` that [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) owns and validates; `probe_runtime_devices` runs `RUNTIME_DEVICE_PROBE_SCRIPT` to count the devices the runtime's torch can open and whether one can run a kernel (`RuntimeDeviceProbe`); `probe_torch_alignment` reads the installed torch and the one an engine pins from distribution metadata without importing torch (`TorchAlignmentProbe`); and `split_local_version`, `requirement_pinned_version` and `install_pinned_package` are the torch-pin helpers [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) uses to put the SDK's torch back after an engine install. The `pub(crate)` items are re-exported by `therock.rs` for [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs); `probe_rocm_sdk_runtime_for_target`, which `install_wheel_runtime` calls with the device payload it just chose, is `pub(super)`. - [`therock/resolve.rs`](../apps/rocm/src/therock/resolve.rs) — a child of `therock.rs` that holds no logic (split out of `therock.rs`): only the `mod` declarations for the artifact-resolution modules under `therock/resolve/` and the `pub(super) use` lines through which `therock.rs` reads them. It is the parent the `resolve/` files hang from, so a module there is `pub(in crate::therock)` for the items `therock.rs` reaches and private otherwise. - [`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/fetch.rs`, `therock/family.rs`, `therock/source.rs` and `therock/version.rs` (split out of `therock.rs`): resolving the SDK tarball a request installs. `resolve_tarball_artifact` (and `resolve_tarball_artifact_with_timeout`, which takes the metadata download timeout) resolves the family and source layout, downloads the catalog through the cached metadata fetch, and returns the `TarballArtifact` it chose; `parse_tarball_index_html` reads the catalog's embedded file list into `TarballIndexFile`s and `validate_tarball_file_name` refuses an entry that is not a plain file name; `select_tarball_candidate` picks the newest archive the request can install, with `tarball_family_token` spelling the family the way the next layout's catalog does. `platform_tarball_token`, which names the archives' platform, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. Only `resolve_tarball_artifact*` and `TarballArtifact` are visible beyond this file. -- [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/family.rs`, `therock/fetch.rs`, `therock/process.rs`, `therock/python.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for `WheelRuntimeComposition` (split out of `therock.rs`): resolving the wheel stack a request installs. `resolve_pip_runtime` (and `resolve_pip_runtime_with_timeout`, which takes the metadata download timeout and an optional source layout) resolves the family and the layout, reads the aggregate root, decides the device payload and returns a `PipRuntimeResolution`; on the canonical layout `select_matching_pip_package_versions` then takes the newest `rocm` whose `torch`, `torchvision` and `torchaudio` listings agree, and on the next layout `torchaudio_pin` pins a torchaudio first (the hardware-verified release when it is published, read from the vLLM engine's table) and `resolve_published_pip_package_versions` has `uv pip compile` choose the rest. `wheel_runtime_composition` turns a resolution into the exact package specs (`therock_pip_package_specs`) an install gives `uv` and records in the manifest, `published_pip_requirements` is the matching requirement set for `uv pip compile`, and `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` listing so the interpreter can be picked before resolution. Beneath that, `parse_aggregate_package_links` and `parse_aggregate_device_targets` read the package links of the canonical aggregate root and the exact device targets it publishes, and `validate_aggregate_index_layout` refuses a root that lacks the expected packages or any device payload; `AggregateDeviceTarget::resolve` picks the one exact target a host gets, or records why it could not (`Undetermined`); `load_simple_index_versions` fetches a package's simple-index listing through the cached metadata download and `parse_simple_index_versions` reads the versions out of it, dropping wheels the interpreter's tags do not accept when given them (`wheel_stem_matches_compatibility` and the `wheel_*_tag_matches` helpers); `package_release_key` orders the PEP 440 versions found there (`PackageReleaseKey`). `slugify`, which keys the cached listing, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The functions and types `therock.rs` and `therock/update.rs` name are `pub(in crate::therock)` and reach them through `therock/resolve.rs`; `PipRuntimeResolution` and `TheRockPipPackageVersions` are `pub(in crate::therock)` with the fields they read, and `TorchaudioPin` with its `describe`, so they can use what `resolve_pip_runtime` returns without naming the types; the rest is private. +- [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/family.rs`, `therock/fetch.rs`, `therock/process.rs`, `therock/python.rs`, `therock/source.rs` and `therock/version.rs`, and on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for `WheelRuntimeComposition` (split out of `therock.rs`): resolving the wheel stack a request installs. `resolve_pip_runtime` (and `resolve_pip_runtime_with_timeout`, which takes the metadata download timeout and an optional source layout) resolves the family and the layout, reads the aggregate root, decides the device payload and returns a `PipRuntimeResolution`; on the canonical layout `select_matching_pip_package_versions` then takes the newest `rocm` whose `torch`, `torchvision` and `torchaudio` listings agree, and on the next layout `torchaudio_pin` pins a torchaudio first (the hardware-verified release when it is published, read from the vLLM engine's table) and `resolve_published_pip_package_versions` has `uv pip compile` choose the rest. That compile is the one step that may fetch `uv`, so who may is decided in one pure function, [`therock/resolve/uv_access.rs`](../apps/rocm/src/therock/resolve/uv_access.rs)'s `plan_version_resolution`, which takes the layout, a `UvAccess` and whether a `uv` is present, and answers scrape the indexes (canonical), compile with a `UvChoice` (acquire the managed `uv`, or use the present one) or skip. `resolve_pip_runtime` and `resolve_pip_runtime_with_timeout` (a real install and `rocm update`) are always `UvAccess::Acquire`, which never skips and never asks whether a `uv` is present; `preview_pip_runtime` is `UvAccess::PresentOnly`, which uses a present `uv` (`rocm_core::find_uv_binary`) and otherwise returns `PipRuntimePlan::NeedsUv` holding an `UnresolvedPipRuntime`: what was chosen without `uv` (the `rocm` version, the torchaudio pin, the requirements `uv` would be given) and no package versions, so nothing made from them can be built from it. `wheel.rs` acts on the answer; the real `find_uv_binary` and `ensure_uv_binary` reach it through a `UvPolicy` so that a test can count the calls. `wheel_runtime_composition` turns a resolution into the exact package specs (`therock_pip_package_specs`) an install gives `uv` and records in the manifest, `published_pip_requirements` is the matching requirement set for `uv pip compile`, and `canonical_auto_select_needs_next_python` peeks at the channel's own `rocm` listing so the interpreter can be picked before resolution. Beneath that, `parse_aggregate_package_links` and `parse_aggregate_device_targets` read the package links of the canonical aggregate root and the exact device targets it publishes, and `validate_aggregate_index_layout` refuses a root that lacks the expected packages or any device payload; `AggregateDeviceTarget::resolve` picks the one exact target a host gets, or records why it could not (`Undetermined`), and `AggregateDeviceTarget::prefer_recorded` is the one place a recorded target (an update apply's, or the one a runtime's manifest was built with) replaces the detected one, called by `therock.rs`, `therock/update.rs` and `therock/wheel_plan.rs`; `load_simple_index_versions` fetches a package's simple-index listing through the cached metadata download and `parse_simple_index_versions` reads the versions out of it, dropping wheels the interpreter's tags do not accept when given them (`wheel_stem_matches_compatibility` and the `wheel_*_tag_matches` helpers); `package_release_key` orders the PEP 440 versions found there (`PackageReleaseKey`). `slugify`, which keys the cached listing, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The functions and types `therock.rs` and `therock/update.rs` name are `pub(in crate::therock)` and reach them through `therock/resolve.rs`; `PipRuntimeResolution` and `TheRockPipPackageVersions` are `pub(in crate::therock)` with the fields they read, and `TorchaudioPin` with its `describe`, so they can use what `resolve_pip_runtime` returns without naming the types; the rest is private. +- [`therock/resolve/uv_access.rs`](../apps/rocm/src/therock/resolve/uv_access.rs) — full domain extraction, a leaf child of `therock/resolve.rs` that builds only on `therock/source.rs` for `SourceLayout` (split out of `therock/resolve/wheel.rs` to keep it within the module-size limit): who may fetch `uv` while a wheel stack is resolved. `UvAccess` says how far a resolution may go (`Acquire`, what a real install and `rocm update` pass, or `PresentOnly`, a preview's), `plan_version_resolution` turns that, the layout and whether a `uv` is present into a `VersionResolution` (scrape the indexes, compile with a `UvChoice`, or skip) without touching anything, and `uv_for_compile` hands the compile step the `uv` it runs, acquiring it only on `Acquire`, with the context a failed fetch has always carried. `UvPolicy` carries the access and the two real lookups (`find_uv_binary`, `ensure_uv_binary`) as closures so a test can count what a resolution asks for. Everything is `pub(super)` and read by `wheel.rs`; the state matrix over layout, access and presence lives in this module's tests. - [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` and `therock/version.rs` and reaches the registry through [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (split out of `therock.rs`): the consent gate for an install that would displace the active default runtime. `active_default_runtime_relation` says what the install would replace (an upgrade, downgrade or reinstall when it is the same family and channel, otherwise the different runtime it displaces) or `None` when nothing is the active default, and it fails closed, with a relation naming what could not be resolved, when the config names an active default the registry cannot resolve; `SdkInstallConsent` and `SdkInstallApprovalSource` say how consent is obtained (asked for, or already granted by `--yes`, `--approve-replacing-active-default` or `rocm update --apply`); `sdk_install_approval` turns those and whether a terminal is attached into proceed, prompt or refuse (`SdkInstallApproval`); `preapproved_install_line`, `fresh_install_line`, `refuse_non_interactive_message` and `confirm_overwrite_existing_sdk` are the wording of each outcome, and the prompt reads its answer through `prompt_yes_no` and the shared [`confirm.rs`](../crates/rocm-core/src/confirm.rs). `SdkInstallConsent`, `SdkInstallApprovalSource` and `preapproved_install_line` are re-exported by `therock.rs`; the other items `therock.rs` calls are `pub(super)`, as is `active_default_relation_text`, which the version property tests call, and the rest is private. +- [`therock/wheel_plan.rs`](../apps/rocm/src/therock/wheel_plan.rs) — full domain extraction, a leaf child of `therock.rs` (split out of `install_wheel_runtime`) that builds on `therock/python.rs` and `therock/resolve.rs` for the Python decision and the resolution it prints: the `sdk install` plan a wheel install prints, key by key. `WheelPlan` carries the facts, `write_wheel_plan` writes the lines a preview and a real install share, and `write_dry_run_tail` the lines only a preview adds (the `uv` commands it would run and the manifest path). `PlanStack` is the one place the plan differs: `Resolved` carries the package specs, runtime key, install folder and manifest path, and `NeedsUv` (a ROCm 10 preview on a host with no `uv`, see `UnresolvedPipRuntime` in `therock/resolve/wheel.rs`) prints those as `not resolved by this preview` and adds the `resolution_note:` line, so one writer serves both and a line the plan gains appears in both. `unresolved_wheel_preview` renders the second kind whole. `therock.rs` reads it through `pub(super)` items; it reads the provenance, host-note and `venv_python_path` helpers back from `therock.rs`. - [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests and on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for the startup-check record (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. - [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. - [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_serve_command_safety`, the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), and `validate_chat_rocm_command_safety`, which applies them to a `serve` argv and also refuses a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_serve_command_safety` and `validate_chat_rocm_command_safety` here instead of through `chat`, and keeps no copy of the serve rules. `chat.rs`'s child modules call into it and `chat.rs` re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). diff --git a/docs/manual-testing.md b/docs/manual-testing.md index c8528d7118..bdf3308237 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -169,11 +169,15 @@ Expected result: plan's `command:` line shows `uv venv --python 3.12 ...` (a version, as no interpreter path exists yet), so it is not copy-pasteable verbatim: `uv` resolves `3.12` itself and may pick a different Python 3.12 than the managed - one a real install prepares. This holds for the canonical layout. A preview - pinned to ROCm 10 or newer (`--version 10.0.0 --family gfx1200`) still needs - `uv` to resolve the package versions, so on a clean host it can download the - managed `uv` and create the `tools` folder, although it prepares no Python and - saves no Python path. + one a real install prepares. This holds for every layout. A preview pinned + to ROCm 10 or newer (`--version 10.0.0 --family gfx1200`) needs `uv` to choose + the `torch` and `torchvision` versions, so on a host with no `uv` installed it + stops before that step: the data folder still has no `tools` folder, a + `resolution_note:` line says why, and `target:`, `runtime_key:`, + `package_specs:`, `command:` and `manifest:` read `not resolved by this + preview`. Run the real install (or any command that fetches `uv`), then the + preview again, and it prints the complete plan; so does a preview with + `ROCM_CLI_UV_BINARY` naming a `uv`. - With `ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP=1`, or a `ROCM_CLI_MANAGED_PYTHON_VERSION` whose Python the layout does not accept (such as `3.13` for the canonical layout), on the same host the preview fails diff --git a/docs/testing.md b/docs/testing.md index cbe31a2650..8d645dd46d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -763,12 +763,47 @@ download could start. The printed `command:` line then shows `uv venv --python 3.12 ...` (a version, because no interpreter path exists yet), so it is not copy-pasteable verbatim: `uv` resolves `3.12` itself, and on a host with a different Python 3.12 it can choose that one rather than the managed one -a real install prepares. That scenario is a canonical-layout preview. A preview -of a ROCm 10 or newer wheel (a `--version` pin of 10 or later) does still -need `uv`: it resolves the published package versions with `uv pip compile`, so -on a clean host it can download the managed `uv` and create the data folder's -`tools` folder, although it still prepares no Python. The "no `uv`, no `tools` -folder" guarantee is therefore the canonical layout's only. +a real install prepares. That scenario is a canonical-layout preview. + +A preview of a ROCm 10 or newer wheel (a `--version` pin of 10 or later) is held +to the same rule, and never fetches `uv` either. On that layout `uv pip compile` +is the one step that chooses the `torch` and `torchvision` versions (the CLI +picks only the ROCm version and the torchaudio pin), so the preview runs a `uv` +that is already there (the one `ROCM_CLI_UV_BINARY` names, or the managed copy a +previous install left) and, with none, stops before that step. It then prints +what needs no `uv` (the ROCm version, the torchaudio pin, the requirements `uv` +would be asked about), marks `target`, `runtime_key`, `package_specs`, `command` +and `manifest` as `not resolved by this preview`, and says why in a +`resolution_note:` line and a progress line, in the voice of `python_note:`. It +cannot say whether the published stack installs together either. A real install +and `rocm update` still fetch `uv` as before, and a preview made after either +has run is the complete plan. Scenario `runtime-28` checks the note against the +state it left on a host with no `uv`: no managed-tools folder, no tool archive in +its own cache folder (here the real detector, since no stand-in `uv` is named), +no saved config, and no `torch` or `torchvision` version anywhere in the output. +`runtime-27` is the other polarity: with a `uv` present (the recording stand-in, +which answers `uv pip compile` with the fixture's four pins, drains its stdin +under `FAKE_UV_DRAIN_STDIN`, and logs its arguments under `FAKE_UV_LOG`), the +preview runs it exactly once, for the compile, and prints the resolved plan. The +`therock-next` scenarios that print resolved package specs start from `And a uv +is already installed`, which installs the managed `uv` into the scenario's data +folder the way a real install would have. +The decision is unit-tested on every platform, Windows included +(`the_version_step_is_decided_by_layout_access_and_a_present_uv_alone` in +[uv_access.rs](../apps/rocm/src/therock/resolve/uv_access.rs): the state matrix of +both layouts, a preview or an install, and a `uv` present or not), as are the +plan's whole text and its key order against a resolved plan's +(`the_plan_a_preview_without_uv_prints_is_exactly_this` and +`a_resolved_and_an_unresolved_plan_print_the_same_keys` in +[wheel.rs](../apps/rocm/src/therock/resolve/wheel.rs)). On Unix, +`a_preview_never_acquires_uv_and_a_real_resolution_still_does` there runs the +real resolution against a served index and a stand-in `uv`, counting what each +row looked for, acquired and ran. What no test reaches without the network is the +real download a real install makes when no `uv` is installed, and that the public +entry points a real install and `rocm update` call pass `UvAccess::Acquire`: the +first is a network call, and the second is held only by the types (those entry +points return a resolved plan and refuse the unresolved one) and by the lanes +that install a ROCm 10 runtime for real. The preview does not announce a Python the install would refuse either, and it prints nothing that predicts what a real install would do on a path that ends in @@ -803,6 +838,9 @@ progress output and that `uv` was not run and no file was written. Beside it are `a_dry_run_refuses_a_managed_python_version_the_install_would_refuse` and `a_dry_run_reads_every_spelling_of_a_managed_python_version_the_same_way`. The preview does still write its package-index cache under the cache folder. +The `update_check` tool and the plain `rocm update` check (no `--apply`) are not +previews in this sense: they resolve with `uv` and can fetch it, and prepare a +managed Python too. Then it verifies: diff --git a/tests/e2e-cucumber/features/runtime_setup.feature b/tests/e2e-cucumber/features/runtime_setup.feature index 4f820a1099..35be0b0678 100644 --- a/tests/e2e-cucumber/features/runtime_setup.feature +++ b/tests/e2e-cucumber/features/runtime_setup.feature @@ -581,3 +581,56 @@ Feature: Runtime configuration Then the adoption is refused And the refusal asks for the environment folder with --root And no runtime is registered + + # A preview of a ROCm 10 or newer wheel has `uv pip compile` choose the torch + # and torchvision versions: nothing else does, and the plan's package specs, its + # runtime key and its command are made of them. When a `uv` is already there the + # preview must still run it, so the plan is the one a real install would print. + # This pins that half of the contract next to runtime-28, where no `uv` is there, + # so that skipping the step on a clean host cannot become skipping it everywhere. + # + # The recording `uv` is the one named by `ROCM_CLI_UV_BINARY`, so it counts as + # present, and it answers `uv pip compile` with the fixture's four pinned + # versions. Its log is the evidence the compile ran, once, and the plan + # carries the versions it printed. + # + # Linux only, for the same empty-`PATH` host as runtime-23 (the Python the + # preview would prepare is not what this scenario is about). + @id:runtime-install-preview-of-rocm-10-uses-the-uv-that-is-present @requires-os:linux + Scenario: runtime-27 - Previewing a ROCm 10 SDK install with a uv already present resolves the package versions with it + Given a canonical release pip index fixture and a ROCm 10 pip index fixture + And a host with no usable Python and a recording uv + And the recording uv answers uv pip compile with the ROCm 10 fixture's versions + When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0 + Then the preview requests the gfx1200 device extras + And the preview resolved the versions with the uv that was present + + # A preview is a promise that nothing on the machine changes, and runtime-23 held + # the Python half of it. The other half is `uv`: a preview of a ROCm 10 or newer + # wheel used to go and fetch the managed `uv` (creating the data folder's `tools` + # folder) because `uv pip compile` is the one step that knows the torch and + # torchvision versions. On a host with no `uv` the preview now runs nothing and + # downloads nothing. It prints what it can know without `uv` (the ROCm version, + # the torchaudio pin, the requirements `uv` would be asked about), says in a + # `resolution_note:` line and a progress line what it did not do, and marks the + # lines made from the missing versions as not resolved rather than printing a + # guess. A real install and `rocm update` still fetch `uv` as they always did. + # + # The Then checks the message and the state together: with no `uv` to log, the + # evidence is the folders. No managed `tools` folder, no tool archive in the + # scenario's own cache, no saved configuration, and no torch version anywhere in + # the output, so the unresolved lines are not covering for a guess. runtime-27 + # is the other polarity: with a `uv` present the compile still runs. + # + # Linux only, for the same empty-`PATH` host as runtime-23. The decision is + # unit-tested for every combination of layout, who may fetch `uv` and whether one + # is present by `the_version_step_is_decided_by_layout_access_and_a_present_uv_alone` + # in `apps/rocm/src/therock/resolve/uv_access.rs`, and the plan's wording (written + # by `apps/rocm/src/therock/wheel_plan.rs`) by the tests in + # `apps/rocm/src/therock/resolve/wheel.rs`, on every platform. + @id:runtime-install-preview-of-rocm-10-fetches-no-uv @requires-os:linux + Scenario: runtime-28 - Previewing a ROCm 10 SDK install on a host with no uv fetches none and says what it did not resolve + Given a canonical release pip index fixture and a ROCm 10 pip index fixture + And a host with no uv and no usable Python + When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0 + Then the preview says it did not resolve the ROCm 10 package versions, as it says diff --git a/tests/e2e-cucumber/features/therock_next_generation.feature b/tests/e2e-cucumber/features/therock_next_generation.feature index 35c683eccf..58ce4a24c4 100644 --- a/tests/e2e-cucumber/features/therock_next_generation.feature +++ b/tests/e2e-cucumber/features/therock_next_generation.feature @@ -11,6 +11,13 @@ Feature: TheRock "next" ROCm 10 install layout # Scenarios 01-05 point source overrides at loopback fixtures through the # explicit trust gate. Scenario 06 proves the same overrides are inert when # the trust gate is absent; scenario 07 supplies the live GPU evidence. + # + # A ROCm 10 preview has a real `uv pip compile` choose the torch and torchvision + # versions, and a preview never fetches `uv` itself. The scenarios below that + # print resolved package specs (02, 10, 11 and 13) therefore start from an + # installed `uv`, which is the precondition they always had when the preview + # fetched one for them. runtime-27 and runtime-28 in `runtime_setup.feature` pin + # the preview's behaviour with a `uv` present and with none. # The regression this pins: a "next-first" dispatch that probes the ROCm 10 # sources before the canonical ones. An unpinned release install must never @@ -30,6 +37,7 @@ Feature: TheRock "next" ROCm 10 install layout @id:therock-next-02-wheel-pins-raw-device-extras Scenario: therock-next-02 - A pinned ROCm 10 wheel install resolves the next index with raw device extras Given a canonical release pip index fixture and a ROCm 10 pip index fixture + And a uv is already installed When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0 Then the preview resolves the ROCm 10 pip index And the preview reports the next source layout generation @@ -195,6 +203,7 @@ Feature: TheRock "next" ROCm 10 install layout @id:therock-next-09-wheel-devel-adds-the-toolchain Scenario: therock-next-10 - A pinned ROCm 10 wheel install adds the toolchain when asked Given a canonical release pip index fixture and a ROCm 10 pip index fixture + And a uv is already installed When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0 with the toolchain Then the preview resolves the ROCm 10 pip index And the preview requests the gfx1200 device extras with the toolchain @@ -210,6 +219,7 @@ Feature: TheRock "next" ROCm 10 install layout @id:therock-next-11-older-pin-keeps-one-rocm-build Scenario: therock-next-11 - A ROCm 10 wheel install pinned below the newest build installs only that build Given a ROCm 10 pip index fixture that also publishes a newer ROCm 10 build + And a uv is already installed When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0 Then the preview resolves the ROCm 10 pip index And every package in the preview is built for ROCm 10.0.0 @@ -235,6 +245,7 @@ Feature: TheRock "next" ROCm 10 install layout @id:therock-next-13-verified-torchaudio-over-newer-rebuild Scenario: therock-next-13 - A ROCm 10 wheel install takes the hardware-verified torchaudio over a newer rebuild Given a ROCm 10 pip index fixture where several torchaudio rebuilds share one ROCm label + And a uv is already installed When the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.1.0 Then the preview installs the hardware-verified torchaudio for ROCm 10.1.0, not the newer rebuild diff --git a/tests/e2e-cucumber/src/bin/fake-uv.rs b/tests/e2e-cucumber/src/bin/fake-uv.rs index 3227ae5abc..0bd0a4860e 100644 --- a/tests/e2e-cucumber/src/bin/fake-uv.rs +++ b/tests/e2e-cucumber/src/bin/fake-uv.rs @@ -14,10 +14,22 @@ //! arguments it was given (unset: nothing is written). A scenario that must //! show `uv` was never run asserts the file does not exist. //! +//! `FAKE_UV_DRAIN_STDIN` — when set, read standard input to its end before +//! answering (unset: stdin is left alone). The CLI writes the requirements to +//! `uv pip compile -` on stdin before it reads anything back, so a stand-in that +//! answers and exits without reading can make that write fail on a closed pipe, +//! depending on timing. Unset by default because a scenario that never pipes +//! anything in may leave stdin attached to something that never ends. +//! //! A Rust binary rather than a shell script so the scenarios run on Windows //! too. fn main() { + if std::env::var_os("FAKE_UV_DRAIN_STDIN").is_some() { + use std::io::Read as _; + let mut sink = Vec::new(); + let _ = std::io::stdin().read_to_end(&mut sink); + } if let Some(log) = std::env::var_os("FAKE_UV_LOG").map(std::path::PathBuf::from) { use std::io::Write as _; let args: Vec = std::env::args().skip(1).collect(); diff --git a/tests/e2e-cucumber/tests/e2e/therock_steps.rs b/tests/e2e-cucumber/tests/e2e/therock_steps.rs index d07a243d34..9a3ee87d27 100644 --- a/tests/e2e-cucumber/tests/e2e/therock_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/therock_steps.rs @@ -269,6 +269,40 @@ const SCRATCH_CACHE_DIR: &str = "scratch-cache"; /// for a reason it does not name, so they are refused up front. #[given("a host with no usable Python and a recording uv")] async fn host_with_no_python_and_recording_uv(world: &mut E2eWorld) { + let scenario_root = host_without_python(world); + let uv = scenario_root + .join("recording-uv-bin") + .join(e2e_cucumber::fixture_binary::fixture_target_name("uv")); + crate::install_fixture_binary("fake-uv", &uv); + world + .command_env + .push(("ROCM_CLI_UV_BINARY", uv.into_os_string())); + world.command_env.push(( + "FAKE_UV_LOG", + scenario_root.join(UV_INVOCATION_LOG).into_os_string(), + )); +} + +/// The same host with no `uv` anywhere: none named by `ROCM_CLI_UV_BINARY` and no +/// managed copy under the scenario's data folder, so a CLI that needs one would +/// have to download it. Nothing here can observe a `uv` run, since there is none to +/// run; what it can observe is the state a download leaves, which is why the +/// scenario's data and cache folders are the evidence. +#[given("a host with no uv and no usable Python")] +async fn host_with_no_uv_and_no_python(world: &mut E2eWorld) { + assert!( + std::env::var_os(rocm_core::uv::UV_BINARY_ENV).is_none(), + "{} is set in the environment running the suite, which would put a uv on this host; \ + unset it", + rocm_core::uv::UV_BINARY_ENV + ); + host_without_python(world); +} + +/// What the hosts these steps build share: `PATH` is one empty folder, so no Python is found +/// and none has been saved, and the CLI has a cache folder of its own. Returns +/// the scenario root. +fn host_without_python(world: &mut E2eWorld) -> std::path::PathBuf { for var in [ "ROCM_CLI_PYTHON", "ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP", @@ -282,10 +316,6 @@ async fn host_with_no_python_and_recording_uv(world: &mut E2eWorld) { let scenario_root = root(world).to_path_buf(); let empty_path = scenario_root.join("no-python-on-path"); std::fs::create_dir_all(&empty_path).expect("failed to create the empty PATH folder"); - let uv = scenario_root - .join("recording-uv-bin") - .join(e2e_cucumber::fixture_binary::fixture_target_name("uv")); - crate::install_fixture_binary("fake-uv", &uv); world .command_env .push(("PATH", empty_path.into_os_string())); @@ -298,13 +328,7 @@ async fn host_with_no_python_and_recording_uv(world: &mut E2eWorld) { world .command_env .push(("ROCM_CLI_CACHE_DIR", scratch_cache.into_os_string())); - world - .command_env - .push(("ROCM_CLI_UV_BINARY", uv.into_os_string())); - world.command_env.push(( - "FAKE_UV_LOG", - scenario_root.join(UV_INVOCATION_LOG).into_os_string(), - )); + scenario_root } #[given("a canonical release pip index fixture and a ROCm 10 pip index fixture")] @@ -865,6 +889,18 @@ async fn preview_unpinned_wheel_install(world: &mut E2eWorld) { ); } +/// A `uv` the preview can use without fetching one. The resolution these +/// scenarios assert on is a real `uv pip compile` against the loopback fixtures, +/// so the stand-in `uv` of runtime-27 would not do; this installs the managed one +/// the way a real install would have (from the network, into the scenario's own +/// data folder, or, when `ROCM_CLI_UV_BINARY` names one, that one). +#[given("a uv is already installed")] +async fn a_uv_is_already_installed(world: &mut E2eWorld) { + let paths = world.app_paths(); + rocm_core::ensure_uv_binary(&paths) + .unwrap_or_else(|error| panic!("failed to install uv for the scenario: {error:#}")); +} + #[when("the user previews a wheel SDK install for arch gfx1200 pinned to ROCm 10.0.0")] async fn preview_pinned_wheel_install(world: &mut E2eWorld) { preview_ok( @@ -1156,10 +1192,11 @@ async fn preview_prepared_no_python(world: &mut E2eWorld) { /// The state half of the claim: the recording `uv` was never run (the evidence /// that nothing was prepared), no managed tools folder, no saved configuration, -/// and, as a backstop only, no tool archive in the scenario's cache. Every -/// scenario that calls this previews the canonical layout (no ROCm 10 pin), the -/// only layout whose preview needs no `uv`; a ROCm 10 preview can legitimately -/// create the managed tools folder to resolve package versions. +/// and, as a backstop only, no tool archive in the scenario's cache. It holds for +/// both layouts, because a preview never fetches `uv`: the canonical layout needs +/// none, and the ROCm 10 layout runs one only when it is already present (see +/// runtime-27 and runtime-28). A scenario that gives the preview a `uv` and expects +/// it to run must not call this. fn assert_no_python_prepared(world: &E2eWorld) { let paths = world.app_paths(); let scenario_root = root(world).to_path_buf(); @@ -1300,6 +1337,112 @@ async fn preview_refused_for_python_version(world: &mut E2eWorld) { assert_no_python_prepared(world); } +/// The four pins `uv pip compile` would print for the ROCm 10 fixture, one +/// `name==version` per line, which the CLI reads back as the chosen stack. +#[given("the recording uv answers uv pip compile with the ROCm 10 fixture's versions")] +async fn recording_uv_answers_compile(world: &mut E2eWorld) { + let pins = [ + ("rocm", NEXT_ROCM_VERSION), + ("torch", NEXT_TORCH_VERSION), + ("torchvision", NEXT_TORCHVISION_VERSION), + ("torchaudio", NEXT_TORCHAUDIO_VERSION), + ] + .map(|(name, version)| format!("{name}=={version}")) + .join("\n"); + world.command_env.push(("FAKE_UV_STDOUT", pins.into())); + // The CLI pipes the requirements into `uv pip compile -`; see the stand-in. + world.command_env.push(("FAKE_UV_DRAIN_STDIN", "1".into())); +} + +/// A `uv` that is present is still used: it ran `uv pip compile`, once and for +/// nothing else, and the preview did not report the step as skipped. The log is +/// the evidence, since the plan's versions alone could not tell a compile from a +/// guess. +#[then("the preview resolved the versions with the uv that was present")] +async fn preview_resolved_with_present_uv(world: &mut E2eWorld) { + let log_path = root(world).join(UV_INVOCATION_LOG); + let log = std::fs::read_to_string(&log_path).unwrap_or_else(|error| { + panic!( + "the recording uv has no log at {}, so the preview never ran it: {error}", + log_path.display() + ) + }); + let runs: Vec<&str> = log.lines().collect(); + assert!( + matches!(runs.as_slice(), [run] if run.starts_with("pip compile ")), + "the preview should run uv once, for `uv pip compile`, and for nothing else:\n{log}" + ); + let out = stdout(world); + for unresolved in ["resolution_note:", "not resolved by this preview"] { + assert!( + !out.contains(unresolved), + "the preview ran uv but reported the step as skipped ({unresolved:?}):\n{out}" + ); + } +} + +/// What a preview of the ROCm 10 layout says when no `uv` is there to run, and the +/// state it left. The claim and the filesystem are checked together: the note alone +/// would pass with a preview that also downloaded `uv`, and the empty folders alone +/// would pass if the preview had stopped before it got to the question. +#[then("the preview says it did not resolve the ROCm 10 package versions, as it says")] +async fn preview_did_not_resolve_without_uv(world: &mut E2eWorld) { + assert_contains( + world, + "No uv is installed; a real install would fetch it and resolve the torch and torchvision \ + versions with `uv pip compile`. This preview does not.", + "the progress line about what the preview skipped", + ); + assert_contains( + world, + "resolution_note: no uv is installed, so a real install would fetch it and run `uv pip \ + compile` to choose the torch and torchvision versions and to check that the published \ + packages install together, which this preview did not do", + "the note about what the preview skipped", + ); + // Everything made from the versions `uv` would choose is marked, not guessed. + for key in [ + "target", + "runtime_key", + "package_specs", + "command", + "manifest", + ] { + assert_contains( + world, + &format!("\n {key}: not resolved by this preview\n"), + "a line made from the unchosen versions", + ); + } + // What needs no `uv` is still printed, from the same decisions a real install + // takes: the exact device payload, the requirements `uv` would be asked about, + // and the pinned torchaudio. + assert_contains( + world, + &format!("device_target: {RAW_ARCH}"), + "the device target", + ); + assert_version_resolution_extras(world, false); + assert_contains( + world, + &format!("selected_rocm_version: {NEXT_ROCM_VERSION}"), + "the ROCm version", + ); + assert_contains( + world, + &format!("torchaudio_selection: torchaudio {NEXT_TORCHAUDIO_VERSION}, "), + "the torchaudio pin", + ); + for chosen_by_uv in [NEXT_TORCH_VERSION, NEXT_TORCHVISION_VERSION] { + assert!( + !stdout(world).contains(chosen_by_uv), + "the preview named {chosen_by_uv}, which only `uv pip compile` could have chosen:\n{}", + stdout(world) + ); + } + assert_no_python_prepared(world); +} + #[then("the preview resolves the canonical release pip index")] async fn preview_resolves_canonical_pip_index(world: &mut E2eWorld) { let base = current_pip_base(world); From c0d0a9ef79641c6e1bcab9233a41ffd50cbb1485 Mon Sep 17 00:00:00 2001 From: Roman Inflianskas Date: Mon, 9 Nov 2026 13:06:07 +0000 Subject: [PATCH 016/102] fix(xtask): check the pinned-key assignment the installer uses (#830) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(xtask): check the pinned-key assignment the installer uses `cargo xtask verify-pinned-keys` is meant to fail CI when a public key pinned in install.sh, install.ps1 or apps/rocm/src/therock.rs differs from docs/keys/. It located each key by the first textual occurrence of its variable name and compared the next quoted string. The shell and PowerShell keep the last value they assign, so the check could pass while the installer trusted a different key: a commented-out line holding the canonical key above the real assignment, a correct assignment overwritten later, a longer identifier sharing the name as a prefix, or a PowerShell reassignment in a different case. Each source must now assign the name exactly once, as a plain declaration on its own line, and the value is read from that assignment. Names match as whole identifiers (case-insensitively for PowerShell) and whole-line comments are skipped. Other writes count towards the limit: assignments elsewhere on a line, compound assignment, ${NAME:=...}, and mentions of the name outside a variable reference, quoted or not (unset, read, Set-Variable, Remove-Variable), plus Rust let/static shadowing. Errors name the file, the variable and the lines. The value compared must also be the value the installer ends up with: the statement has to end with the string (only `;` or a comment may follow), the string must be one well-formed PEM block (BEGIN line, lines of base64, END line) with no `$`, backtick or backslash (Rust may write line breaks as \n) and, outside install.ps1 (which trims each line), no line indented. The closing delimiter must match the opening one (`"@` only after `@"`), and only plain PowerShell scopes count as the declaration (`$env:Name` is an environment variable). A slot with no canonical key must be exactly "", since the installers trust any non-empty slot. install.sh now marks both slots readonly, and the check requires it. This catches a key that drifts by mistake or slips past review. It is a textual check, not an interpreter, and does not defend against deliberately obfuscated installer code; whoever can commit that can also change docs/keys/ in the same commit. install.ps1 is not made read-only: `irm ... | iex` runs it in the caller's session, where a read-only variable would outlive the run and make the next install in that session fail. Signed-off-by: Roman Inflianskas * fix(xtask): follow the moved metadata key and tighten readonly marking The metadata key's source moved to apps/rocm/src/therock/fetch.rs on main, but the pinned-key tests hard-coded the old therock.rs path, so every scratch-repo test and the real-sources test failed once the branch met main. The tests now take each source path from PINNED_KEYS, so they follow the constant wherever it lives. A readonly command now marks a slot only when it names the slot in the command itself and carries no flags. Before, a name in a trailing comment (`readonly OTHER # NAME`) or after `-f`/`-p` counted, although the shell leaves the variable writable in each case. The readonly test now asserts the readonly message for those cases and the second-write message for `readonly OTHER=1 # NAME`, which it previously could not tell apart. The error messages say what to do: a repeated assignment names the declaration to keep, and a diverged key gives its line and which side to fix. docs/release-trust.md now describes the check, the module doc lists the PowerShell forms a textual check cannot follow, and the `$env:` comment states that such a write is counted on purpose. Signed-off-by: Roman Inflianskas * fix(xtask): tighten the readonly rule and name the real reasons A `readonly` line counted wherever it sat, so one indented inside an `if false` block or an uncalled function passed although nothing was made read-only. A `readonly` command, or a `readonly NAME="…"` declaration, now counts only when it starts its line; the module doc names the unindented case a textual check still cannot see. A trailing comment after a tab was not recognised, so `readonly NAME# note` was reported as a second write of the slot. A comment now starts at any `#` that follows whitespace. The missing-readonly error now states what a qualifying line looks like, so it no longer reads as wrong when a flagged or commented `readonly` is already present. The indented-key error names the rule (only install.ps1 trims each line) instead of `openssl`, which the Rust source never uses. Signed-off-by: Roman Inflianskas * docs(release-trust): say when pull-request CI runs the pinned-key check Pull-request CI runs `cargo xtask verify-pinned-keys` only when a change matches the `heavy` path filter, while the release and nightly workflows run it every time. Say so, and name what the filter covers. Signed-off-by: Roman Inflianskas --------- Signed-off-by: Roman Inflianskas (cherry picked from commit b1cc645708db7940b35af3742ff58ea87072fe1d) Signed-off-by: juhovainio --- docs/release-trust.md | 17 + install.sh | 5 + xtask/src/main.rs | 4 +- xtask/src/verify_pinned_keys.rs | 1763 +++++++++++++++++++++++++++++-- 4 files changed, 1706 insertions(+), 83 deletions(-) diff --git a/docs/release-trust.md b/docs/release-trust.md index b5d4429b9b..907962695b 100644 --- a/docs/release-trust.md +++ b/docs/release-trust.md @@ -376,6 +376,23 @@ documents. Do not generate those production keys inside CI. Once published, the installers, metadata cache, and model recipe index can make release-channel signature verification mandatory by default with those keys as trust roots. +Those trust roots are pinned copies of the published keys: the +`PINNED_RELEASE_PUBLIC_KEY_CURRENT`/`_NEXT` slots in +[install.sh](../install.sh), `$PinnedReleasePublicKeyCurrent`/`Next` in +[install.ps1](../install.ps1), and +`PINNED_METADATA_PUBLIC_KEY_PEM` in +[fetch.rs](../apps/rocm/src/therock/fetch.rs). `cargo xtask verify-pinned-keys` +fails when a pinned copy differs from its canonical file under `docs/keys/`. The +release and nightly workflows always run it; pull-request CI runs it when a +change matches the `heavy` path filter in +[ci.yml](../.github/workflows/ci.yml), which covers `docs/keys/`, the +installers, every Rust source and `xtask/`. It reads the one assignment +the installer actually uses, so each slot must be assigned exactly once as a +plain declaration, a slot whose key is not published yet must stay `""`, and +`install.sh` must mark both of its slots `readonly`. The full rules, and what a +textual check cannot catch, are in +[verify_pinned_keys.rs](../xtask/src/verify_pinned_keys.rs). + When those owner-controlled inputs exist, release operators can add `--require-production-trust` or set `ROCM_CLI_REQUIRE_PRODUCTION_TRUST=1` so the dist verifier fails unless the release signing public key, runtime metadata diff --git a/install.sh b/install.sh index ec31a401bc..48d1325ccc 100755 --- a/install.sh +++ b/install.sh @@ -121,6 +121,11 @@ ultUtiz4emTfXDCb2RptOgoydBA+mu9z6O4eVF8S5dVr/S834SK6dD2fWHNnT0dc JwIDAQAB -----END PUBLIC KEY-----" PINNED_RELEASE_PUBLIC_KEY_NEXT="" +# Keep each slot to the single assignment above: `cargo xtask verify-pinned-keys` +# compares exactly that assignment with docs/keys/ and requires this readonly +# line, which makes any later write fail instead of silently replacing the +# trust root. +readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT has_pinned_release_keys() { [ -n "${PINNED_RELEASE_PUBLIC_KEY_CURRENT}" ] || [ -n "${PINNED_RELEASE_PUBLIC_KEY_NEXT}" ] diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 3d3581e2d5..404facf47e 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -97,7 +97,9 @@ pub(crate) enum Command { /// Assert the release/metadata public keys pinned in the installers and /// `apps/rocm/src/therock/fetch.rs` match the canonical published keys under /// `docs/keys/`, and that any configured CI signing key matches the pinned - /// current release key. A no-op while no canonical keys are published. + /// current release key. Each slot must be assigned exactly once (install.sh + /// must also mark it `readonly`), and a slot whose canonical key is not + /// published yet must be empty. VerifyPinnedKeys, /// Print the workspace crates affected by a git range (changed crates plus /// their transitive dependents) as `cargo` package-selection flags, so CI diff --git a/xtask/src/verify_pinned_keys.rs b/xtask/src/verify_pinned_keys.rs index 219379b771..5f96f52ceb 100644 --- a/xtask/src/verify_pinned_keys.rs +++ b/xtask/src/verify_pinned_keys.rs @@ -15,17 +15,67 @@ //! //! Design notes: //! -//! - **Dormant by default.** Until a canonical `docs/keys/` file exists and is -//! non-empty, its key is skipped. So this passes as a no-op today (empty pinned -//! sentinels, no canonical files) and only starts enforcing once real keys are -//! published. +//! - **What it guarantees.** This catches a pinned key that drifts from +//! `docs/keys/` by mistake or slips past review: a stale or commented-out copy, +//! a second assignment, a key pasted into the wrong slot. It is a textual check +//! of plain installer code, not an interpreter, and does not defend against +//! deliberately obfuscated code (a name assembled at runtime through `eval` or +//! `Invoke-Expression`, an assignment hidden inside a multi-line string, a +//! PowerShell `<# … #>` block comment or a block that never runs, a PowerShell +//! multiple assignment with the name first (`$Name, $x = …`), `foreach` or +//! `[ref]`). Anyone who can commit that can also change +//! `docs/keys/` in the same commit, so review is the control there. +//! - **Dormant slots stay empty.** Until a canonical `docs/keys/` file exists and +//! is non-empty, its slot is not compared, but it must be exactly `""`: the +//! installers trust any non-empty slot, so a key pasted into one with nothing +//! published would be trusted with nothing to compare it against. //! - **Per-constant equality.** Each pinned key is embedded in a specific named //! constant (shell `NAME="…"`, PowerShell string/here-string, Rust `const`). We //! isolate *that constant's own value span*, reduce it to its base64 body, and //! compare it for equality to the canonical key. Comparing the whole file (rather //! than the specific constant) would let a wrong/tampered constant pass whenever //! the canonical body appears anywhere else in the file — e.g. in the `NEXT` slot -//! mid-rotation or a stale comment. +//! mid-rotation or a stale comment. The value must be one well-formed PEM block +//! (the BEGIN line, non-empty lines of base64, the END line, nothing else), +//! free of `$`, backticks and backslashes (Rust may write line breaks as `\n`), +//! and the statement must end with the string (only `;` or a comment may +//! follow), so the text compared is the text the installer ends up holding. +//! - **The assignment the runtime uses.** The shell and PowerShell keep the *last* +//! value they assign, so reading the first textual occurrence of the name could +//! compare a commented-out line, a longer identifier sharing the prefix, or a +//! value later overwritten, while the installer trusts another key. So each +//! source must assign the name exactly once, and that assignment must be the +//! plain declaration as its own statement (leading whitespace allowed): shell +//! `NAME="…"` (optionally after `export`/`readonly`), PowerShell `$Name = "…"` +//! or `$Name = @"…"@` (any case, optionally `$script:`, `$global:`, `$local:` +//! or `$private:`; `$env:Name` is an environment variable, so it never counts +//! as the declaration, only as a write), Rust `const NAME: &str = "…";` +//! (optionally `pub` or `pub(crate)`). Names match as whole identifiers, case-insensitively for +//! PowerShell, and whole-line comments are skipped. Every other write of the +//! name counts towards "exactly once": shell `NAME=` anywhere on a line, +//! `NAME+=`, `${NAME:=…}`, and any other mention of the name, quoted or not, +//! except a `readonly`/`export` command that only names variables (this is how +//! `unset` and `read` name it); PowerShell plain and compound assignment to any +//! reference spelling (`$Name`, `$scope:Name`, `${Name}`), and any mention of +//! the name outside a `$` reference, quoted or not (how `Set-Variable`, +//! `Remove-Variable` and `Set-Item variable:` name it); Rust +//! `const`/`static`/`let` bindings in any scope. Over-counting only fails the +//! check, so prose naming the variable outside a comment line is rejected +//! rather than guessed at. +//! - **Read-only at runtime.** `install.sh` must mark each slot `readonly` after +//! assigning it, so a later write fails instead of silently replacing the trust +//! root. The `readonly` command, or the `readonly NAME="…"` declaration, must +//! start its line (an indented one sits in a block or function that may never +//! run), and a separate command must name the slot itself, with no flags +//! (`-f` names a function, `-p` only prints) and not in a trailing comment. +//! A textual check cannot tell whether an unindented `readonly` line runs, so +//! one inside an `if false` block or an uncalled function whose body is not +//! indented still counts; review is the control there, as above. +//! +//! `install.ps1` does not mark its slots read-only: `irm … | iex` runs it in +//! the caller's session, where a read-only variable would outlive the run and +//! make the next install in that session fail (`-Force` would avoid that, but +//! also defeats the point). //! - **CI cross-check.** When a CI signing public key is configured — the file //! named by `ROCM_CLI_SIGNING_PUBLIC_KEY_PATH`, else the inline //! `ROCM_CLI_SIGNING_PUBLIC_KEY_PEM` release/nightly CI wires from the secret — @@ -39,7 +89,7 @@ use std::ffi::OsString; use std::path::{Path, PathBuf}; -use anyhow::{Context, Result, bail}; +use anyhow::{Context, Result, anyhow, bail}; const PEM_BEGIN: &str = "-----BEGIN PUBLIC KEY-----"; const PEM_END: &str = "-----END PUBLIC KEY-----"; @@ -156,48 +206,447 @@ fn pem_body(text: &str) -> Option { if body.is_empty() { None } else { Some(body) } } -/// Isolate the raw text span assigned to `token` under `embedding`, or `None` if the -/// assignment is not found. PEM payloads contain no `"`, so the first quote after the -/// assignment reliably closes the value. -fn extract_constant<'a>(source: &'a str, token: &str, embedding: Embedding) -> Option<&'a str> { - let key = source.find(token)?; - let after = &source[key + token.len()..]; +impl Embedding { + /// What a whole-line comment starts with in this language. + const fn comment_prefix(self) -> &'static str { + match self { + Self::Shell | Self::PowerShell => "#", + Self::Rust => "//", + } + } + + /// Why a second write defeats the comparison, for error messages. + const fn why_one(self) -> &'static str { + match self { + Self::Shell => "the shell keeps the last value it assigns", + Self::PowerShell => "PowerShell keeps the last value it assigns", + Self::Rust => "another binding can shadow the constant that is compared", + } + } + + /// The one assignment shape the value can be read from, for error messages. + fn supported_form(self, name: &str) -> String { + match self { + Self::Shell => format!("`{name}=\"…\"` as its own statement"), + Self::PowerShell => { + format!("`${name} = \"…\"` or `${name} = @\"…\"@` as its own statement") + } + Self::Rust => format!("`const {name}: &str = \"…\";` as its own statement"), + } + } +} + +/// One occurrence of a pinned name that can set its value. +struct Assignment { + /// 1-based line number, for error messages. + line: usize, + /// Byte offset in the whole source just past the name. + after_name: usize, + /// Whether it is the supported declaration shape whose value can be read. + declaration: bool, +} + +const fn is_ident_byte(byte: u8) -> bool { + byte.is_ascii_alphanumeric() || byte == b'_' +} + +/// The part of a shell line before a trailing comment: a `#` that starts a word, +/// so after any whitespace (a space or a tab), not one inside a word like `a#b`. +fn shell_code(line: &str) -> &str { + line.char_indices() + .find(|&(index, c)| c == '#' && line[..index].ends_with(char::is_whitespace)) + .map_or(line, |(index, _)| &line[..index]) +} + +/// Whether `line` is `readonly`/`export` followed only by names and flags, which +/// sets attributes without changing any value. `export NAME; unset NAME` is not. +fn is_attribute_command(line: &str) -> bool { + let mut words = shell_code(line).split_whitespace(); + matches!(words.next(), Some("readonly" | "export")) + && words.all(|word| { + word.starts_with('-') + || word.bytes().all(is_ident_byte) && !word.as_bytes()[0].is_ascii_digit() + }) +} + +/// Start offsets of `name` in `line` as a whole identifier: a longer identifier +/// that merely starts or ends with `name` is a different variable. +fn identifier_offsets(line: &str, name: &str, ignore_case: bool) -> Vec { + let (haystack, needle) = if ignore_case { + (line.to_ascii_lowercase(), name.to_ascii_lowercase()) + } else { + (line.to_owned(), name.to_owned()) + }; + let bytes = haystack.as_bytes(); + haystack + .match_indices(&needle) + .map(|(start, _)| start) + .filter(|&start| { + let end = start + needle.len(); + (start == 0 || !is_ident_byte(bytes[start - 1])) + && (end == bytes.len() || !is_ident_byte(bytes[end])) + }) + .collect() +} + +/// Classify one whole-identifier occurrence of `name` at `start` in `line`: +/// `None` if it cannot change the value, otherwise whether it is the supported +/// declaration shape. +fn classify(line: &str, start: usize, name: &str, embedding: Embedding) -> Option { + let head = &line[..start]; + let tail = &line[start + name.len()..]; match embedding { Embedding::Shell => { - // `NAME="…"` — the token is immediately followed by `="`. - let open = after.find('"')?; - let rest = &after[open + 1..]; - let close = rest.find('"')?; - Some(&rest[..close]) + // `NAME=`, `NAME+=` (bash) and `${NAME=…}`/`${NAME:=…}` all assign. + if tail.starts_with('=') || tail.starts_with("+=") || tail.starts_with(":=") { + let prefix = head.trim(); + let declaration = tail.starts_with('=') + && (prefix.is_empty() || prefix == "export" || prefix == "readonly"); + return Some(declaration); + } + // `$NAME`, `${NAME}`, `${#NAME}` and `${!NAME}` only read it. + if ["$", "${", "${#", "${!"].iter().any(|r| head.ends_with(r)) { + return None; + } + // `readonly NAME …` / `export NAME …` naming variables only set + // attributes. + if is_attribute_command(line) { + return None; + } + // Any other bare mention is how `unset`, `read`, `getopts` and + // `printf -v` name the variable; treat it as a write and fail closed. + Some(false) + } + Embedding::PowerShell => { + // A variable reference is `$Name`, `$scope:Name`, `${Name}` or + // `${scope:Name}`. Anything else naming the variable — `Set-Variable + // -Name Name`, `New-Variable Name`, `Set-Item variable:Name` — is + // treated as a write: the cmdlet forms are too varied to tell apart, + // and over-counting fails closed. + let mut reference = head; + let mut scope = ""; + if let Some(scoped) = reference.strip_suffix(':') { + reference = + scoped.trim_end_matches(|c: char| c.is_ascii_alphanumeric() || c == '_'); + scope = &scoped[reference.len()..]; + } + let braced = reference.ends_with("${"); + let Some(before) = reference + .strip_suffix("${") + .or_else(|| reference.strip_suffix('$')) + else { + return Some(false); + }; + let mut after = tail; + if braced { + after = after.strip_prefix('}')?; + } + let after = after.trim_start(); + let plain = after.starts_with('=') && !after.starts_with("=="); + let compound = ["+=", "-=", "*=", "/=", "%=", "??="] + .iter() + .any(|op| after.starts_with(op)); + if !(plain || compound) { + return None; + } + // Only a variable scope names this variable, so only it can be the + // declaration. `$env:Name` and other drive-qualified names leave the + // variable unset, but are still counted as a write: a lone one is + // reported as an unsupported form (most likely a mistyped + // declaration), and next to the declaration it fails closed rather + // than being guessed harmless. + let variable_scope = ["", "script", "global", "local", "private"] + .iter() + .any(|s| scope.eq_ignore_ascii_case(s)); + Some(plain && !braced && variable_scope && before.trim().is_empty()) } Embedding::Rust => { - // `const NAME: &str = "…";` — skip to the `=`, then the opening quote. - let eq = after.find('=')?; - let open = after[eq..].find('"')? + eq; - let rest = &after[open + 1..]; - let close = rest.find('"')?; - Some(&rest[..close]) + // `const NAME:`, `static NAME:`, `let NAME =` / `let NAME:` (a shadowing + // binding) all introduce a value under the name; a path `NAME::`, + // comparison `NAME ==` or match arm `NAME =>` does not. + let after = tail.trim_start(); + let binds = (after.starts_with(':') && !after.starts_with("::")) + || (after.starts_with('=') && !after.starts_with("==") && !after.starts_with("=>")); + if !binds { + return None; + } + let words: Vec<&str> = head.split_whitespace().collect(); + let declaration = after.starts_with(':') + && matches!( + words.as_slice(), + ["const"] | ["pub" | "pub(crate)", "const"] + ); + Some(declaration) } - Embedding::PowerShell => { - let here = after.find("@\""); - let quote = after.find('"'); - match (here, quote) { - // Here-string `= @"…"@`: the `@"` quote is the first quote seen. - (Some(h), Some(q)) if q == h + 1 => { - let rest = &after[h + 2..]; - let close = rest.find("\"@")?; - Some(&rest[..close]) - } - // Plain `= "…"`. - (_, Some(q)) => { - let rest = &after[q + 1..]; - let close = rest.find('"')?; - Some(&rest[..close]) + } +} + +/// Whether the occurrence of a name at `start` in `line` is named by a `readonly` +/// command that makes it read-only: at the start of the line (an indented one +/// sits in a block or function that may never run), no flags (`-f` names a +/// function, `-p` only prints), and the name in the command itself rather than +/// its trailing comment. +fn marks_readonly(line: &str, start: usize) -> bool { + let code = shell_code(line); + let mut words = code.split_whitespace(); + // `starts_with` is the start-of-line test; the word check below alone would + // also accept an indented command. + code.starts_with("readonly") + && is_attribute_command(line) + && words.next() == Some("readonly") + && words.all(|word| !word.starts_with('-')) + && start < code.len() +} + +/// Every occurrence of `name` in `source` that can set its value, skipping +/// whole-line comments, plus the lines of `readonly` commands naming it. +/// PowerShell names are case-insensitive. +fn find_assignments( + source: &str, + name: &str, + embedding: Embedding, +) -> (Vec, Vec) { + let ignore_case = matches!(embedding, Embedding::PowerShell); + let mut found = Vec::new(); + let mut readonly_lines = Vec::new(); + let mut offset = 0; + for (index, line) in source.split_inclusive('\n').enumerate() { + if !line.trim_start().starts_with(embedding.comment_prefix()) { + for start in identifier_offsets(line, name, ignore_case) { + match classify(line, start, name, embedding) { + Some(declaration) => found.push(Assignment { + line: index + 1, + after_name: offset + start + name.len(), + declaration, + }), + None if matches!(embedding, Embedding::Shell) + && marks_readonly(line, start) => + { + readonly_lines.push(index + 1); + } + None => {} } - _ => None, } } + offset += line.len(); } + (found, readonly_lines) +} + +/// The single declaration of a pinned constant and the raw text it assigns. +struct Constant<'a> { + value: &'a str, + /// 1-based line of the declaration. + line: usize, + /// Whether the declaration is `readonly NAME=…` or a later `readonly` command + /// names it. Only meaningful for the shell. + readonly: bool, +} + +/// Isolate the raw text span assigned to `token` under `embedding`. +/// +/// The value is read from the *single* assignment of the name, because that is +/// the one the runtime uses. A first-occurrence search read whatever came first — +/// a commented-out line, a longer identifier sharing the prefix, or an assignment +/// later overwritten — while the shell and PowerShell keep the last value they +/// execute. So zero or several assignments is an error, as is a single one in a +/// shape the value cannot be read from. PEM payloads contain no `"`, so the first +/// quote after the opening one reliably closes the value. +fn extract_constant<'a>( + source: &'a str, + token: &str, + embedding: Embedding, +) -> std::result::Result, String> { + let name = match embedding { + Embedding::PowerShell => token.strip_prefix('$').unwrap_or(token), + Embedding::Shell | Embedding::Rust => token, + }; + let (assignments, readonly_lines) = find_assignments(source, name, embedding); + let assignment = match assignments.as_slice() { + [] => return Err(format!("has no assignment of {token}")), + [one] => one, + many => { + let lines = many + .iter() + .map(|a| a.line.to_string()) + .collect::>() + .join(", "); + let declarations: Vec = many + .iter() + .filter(|a| a.declaration) + .map(|a| a.line) + .collect(); + let fix = match declarations.as_slice() { + [line] => format!( + "keep the declaration on line {line} and remove or reword the \ + other lines{}", + match embedding { + Embedding::Rust => "", + Embedding::Shell | Embedding::PowerShell => { + " (any mention of the name outside a variable reference \ + counts as a write)" + } + } + ), + _ => format!("leave exactly one, as {}", embedding.supported_form(name)), + }; + return Err(format!( + "assigns {token} {} times (lines {lines}); {}, so exactly one \ + assignment is allowed: {fix}", + many.len(), + embedding.why_one() + )); + } + }; + if !assignment.declaration { + return Err(format!( + "assigns {token} on line {} in an unsupported form; expected {}", + assignment.line, + embedding.supported_form(name) + )); + } + + let after = &source[assignment.after_name..]; + let until_quote = |value: &'a str| value.find('"').map(|close| &value[..close]); + let unreadable = || { + format!( + "assigns {token} on line {} to a value that is not a double-quoted string", + assignment.line + ) + }; + // The value and the length of the delimiter that closes it: `"@` only for a + // here-string that opened with `@"`, a plain `"` everywhere else. + let plain = |value: &'a str| (value, 1); + let value = match embedding { + // `NAME="…"` — the quote follows the `=` directly. + Embedding::Shell => after.strip_prefix("=\"").and_then(until_quote).map(plain), + // `const NAME: &str = "…";` — skip the type to the `=`, then the quote. + Embedding::Rust => after + .find('=') + .and_then(|eq| after[eq + 1..].trim_start().strip_prefix('"')) + .and_then(until_quote) + .map(plain), + // `$Name = @"…"@` (here-string) or `$Name = "…"`. + Embedding::PowerShell => after + .trim_start() + .strip_prefix('=') + .map(str::trim_start) + .and_then(|rhs| match rhs.strip_prefix("@\"") { + Some(body) => body.find("\"@").map(|close| (&body[..close], 2)), + None => rhs.strip_prefix('"').and_then(until_quote).map(plain), + }), + }; + let (value, closer) = value.ok_or_else(unreadable)?; + + // The statement must end with the string: anything after it (`-replace …`, + // `+ "…"`, a second quoted word, a command) would change the value the + // runtime ends up with, so the text compared here would not be it. + let value_end = value.as_ptr() as usize - source.as_ptr() as usize + value.len(); + let line_rest = source[value_end + closer..].lines().next().unwrap_or(""); + let semicolon = line_rest.trim_start().strip_prefix(';'); + let after_semicolon = semicolon.unwrap_or(line_rest); + let rest = after_semicolon.trim(); + // A shell or PowerShell comment starts a word: `""#x` is the string `#x`, + // not an empty string and a comment, while `"" #x` and `"";#x` are. + let comment = rest.starts_with(embedding.comment_prefix()) + && (matches!(embedding, Embedding::Rust) + || semicolon.is_some() + || after_semicolon.starts_with(char::is_whitespace)); + if !(rest.is_empty() || comment) { + return Err(format!( + "assigns {token} on line {} to a string followed by `{rest}`; the statement \ + must end with the string, so the value compared is the value used", + assignment.line + )); + } + + let line_start = source[..assignment.after_name] + .rfind('\n') + .map_or(0, |n| n + 1); + Ok(Constant { + value, + line: assignment.line, + // Same start-of-line rule as a separate `readonly` command: an indented + // `readonly NAME="…"` may sit in a block that never runs. + readonly: source[line_start..].starts_with("readonly") + || readonly_lines.iter().any(|&line| line > assignment.line), + }) +} + +/// Reduce a pinned constant's raw value to the base64 body of its key, or `None` +/// for an empty slot (`""` exactly: the shell tests a slot with `[ -n … ]`, so a +/// lone space would count as a key there). +/// +/// The value must be one well-formed PUBLIC KEY PEM block and nothing else: the +/// BEGIN line, non-empty lines of base64, the END line. Comparing a looser +/// reading would pass a key that `openssl` cannot load (stray punctuation, a +/// blank line in the body) or, with a second key prepended through `${…}`, one +/// it loads instead. Outside PowerShell, whose installer trims each line, a line +/// may not start with whitespace either. `$`, backticks and backslashes are +/// refused in the installers, +/// which would expand them or keep them verbatim; Rust may write line breaks as +/// `\n`, since a Rust string literal turns them into real ones. +fn pinned_pem_body( + value: &str, + embedding: Embedding, +) -> std::result::Result, String> { + let text = match embedding { + Embedding::Rust => value.replace("\\n", "\n"), + Embedding::Shell | Embedding::PowerShell => value.to_owned(), + }; + if let Some(found) = text.chars().find(|c| matches!(c, '$' | '`' | '\\')) { + return Err(format!( + "holds `{found}`, which the installer would expand or keep verbatim; a \ + pinned key is plain PEM text" + )); + } + if value.is_empty() { + return Ok(None); + } + let malformed = || { + "is not one well-formed PUBLIC KEY PEM block (a BEGIN line, lines of base64, an \ + END line, and nothing else)" + .to_owned() + }; + // install.ps1 trims each line before decoding, so indentation is harmless + // there. install.sh and the Rust source hand the value on verbatim, so outside + // PowerShell a line may not start with whitespace: one rule for both, kept + // stricter than any one PEM reader rather than tracking what each tolerates. + if !matches!(embedding, Embedding::PowerShell) + && text + .lines() + .any(|line| line.starts_with(char::is_whitespace)) + { + return Err( + "has a key line that starts with whitespace; only install.ps1, which trims \ + each line, may indent a pinned key" + .to_owned(), + ); + } + let lines: Vec<&str> = text.lines().map(str::trim).collect(); + // A here-string opens and closes on its own lines; nothing else may be blank. + let start = lines + .iter() + .position(|line| !line.is_empty()) + .ok_or_else(malformed)?; + let end = lines + .iter() + .rposition(|line| !line.is_empty()) + .ok_or_else(malformed)? + + 1; + let [first, body @ .., last] = &lines[start..end] else { + return Err(malformed()); + }; + let base64_line = |line: &&str| { + !line.is_empty() + && line + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '/' | '=')) + }; + if *first != PEM_BEGIN || *last != PEM_END || body.is_empty() || !body.iter().all(base64_line) { + return Err(malformed()); + } + Ok(Some(body.concat())) } /// Check every populated canonical key against its pinned sources. `ci_public_key` @@ -210,24 +659,17 @@ fn check_pinned_keys(root: &Path, ci_public_key: Option<&str>) -> Result text, - _ => { - messages.push(format!( - "{}: no canonical key yet — skipped (dormant)", - entry.name - )); - continue; - } + let canonical_body = match std::fs::read_to_string(&canonical_path) { + Ok(text) if !text.trim().is_empty() => Some(pem_body(&text).with_context(|| { + format!( + "{}: {} is not a valid PUBLIC KEY PEM", + entry.name, entry.canonical + ) + })?), + _ => None, }; - let canonical_body = pem_body(&canonical_text).with_context(|| { - format!( - "{}: {} is not a valid PUBLIC KEY PEM", - entry.name, entry.canonical - ) - })?; if entry.name == CURRENT_RELEASE_KEY { - current_release_body = Some(canonical_body.clone()); + current_release_body.clone_from(&canonical_body); } for source in entry.sources { @@ -235,29 +677,72 @@ fn check_pinned_keys(root: &Path, ci_public_key: Option<&str>) -> Result {} + (Some(_), _) => bail!( + "{}: {} does not embed the canonical key {} in {} (line {}) — the \ + pinned constant and the published key have diverged; copy the key \ + from {} into {}, or, if the published key is the one that is \ + wrong, fix {} and every source that pins it", + entry.name, + source.path, + entry.canonical, + source.token, + constant.line, + entry.canonical, + source.token, + entry.canonical + ), + (None, None) => {} + (None, Some(_)) => bail!( + "{}: {} pins a key in {} on line {}, but {} holds no canonical key; \ + publish the key there or leave the slot empty", + entry.name, + source.path, + source.token, + constant.line, + entry.canonical + ), + } } - messages.push(format!( - "{}: pinned in {}", - entry.name, - entry - .sources - .iter() - .map(|s| s.path) - .collect::>() - .join(", ") - )); + let sources = entry + .sources + .iter() + .map(|s| s.path) + .collect::>() + .join(", "); + messages.push(if canonical_body.is_some() { + format!("{}: pinned in {sources}", entry.name) + } else { + format!( + "{}: no canonical key yet — slots empty in {sources} (dormant)", + entry.name + ) + }); } messages.extend(check_ci_public_key( @@ -424,29 +909,34 @@ mod tests { let shell = "PINNED_RELEASE_PUBLIC_KEY_CURRENT=\"the-current\"\n\ PINNED_RELEASE_PUBLIC_KEY_NEXT=\"the-next\"\n"; assert_eq!( - extract_constant(shell, "PINNED_RELEASE_PUBLIC_KEY_CURRENT", Embedding::Shell), - Some("the-current") + extract_constant(shell, "PINNED_RELEASE_PUBLIC_KEY_CURRENT", Embedding::Shell) + .map(|c| c.value), + Ok("the-current") ); assert_eq!( - extract_constant(shell, "PINNED_RELEASE_PUBLIC_KEY_NEXT", Embedding::Shell), - Some("the-next") + extract_constant(shell, "PINNED_RELEASE_PUBLIC_KEY_NEXT", Embedding::Shell) + .map(|c| c.value), + Ok("the-next") ); let rust = "const PINNED_METADATA_PUBLIC_KEY_PEM: &str = \"the-meta\";\n"; assert_eq!( - extract_constant(rust, "PINNED_METADATA_PUBLIC_KEY_PEM", Embedding::Rust), - Some("the-meta") + extract_constant(rust, "PINNED_METADATA_PUBLIC_KEY_PEM", Embedding::Rust) + .map(|c| c.value), + Ok("the-meta") ); let ps = "$PinnedReleasePublicKeyCurrent = @\"\nthe-here\n\"@\n\ $PinnedReleasePublicKeyNext = \"the-plain\"\n"; assert_eq!( - extract_constant(ps, "$PinnedReleasePublicKeyCurrent", Embedding::PowerShell), - Some("\nthe-here\n") + extract_constant(ps, "$PinnedReleasePublicKeyCurrent", Embedding::PowerShell) + .map(|c| c.value), + Ok("\nthe-here\n") ); assert_eq!( - extract_constant(ps, "$PinnedReleasePublicKeyNext", Embedding::PowerShell), - Some("the-plain") + extract_constant(ps, "$PinnedReleasePublicKeyNext", Embedding::PowerShell) + .map(|c| c.value), + Ok("the-plain") ); } @@ -465,11 +955,1120 @@ mod tests { "PINNED_RELEASE_PUBLIC_KEY_CURRENT", Embedding::Shell, ) - .and_then(pem_body) + .ok() + .and_then(|c| pem_body(c.value)) .unwrap(); assert_ne!(current, pem_body(SAMPLE).unwrap()); } + /// A second and third distinct key, used as the canonical next-release and + /// metadata keys. + const NEXT_KEY: &str = "-----BEGIN PUBLIC KEY-----\n\ + MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnextnextnextnext+/\n\ + abcDEF0123456789nextnextnextnextnextnextnextnextnextnextABCD==\n\ + -----END PUBLIC KEY-----\n"; + const META: &str = "-----BEGIN PUBLIC KEY-----\n\ + MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmetametametameta+/\n\ + abcDEF0123456789metametametametametametametametametametaABCD==\n\ + -----END PUBLIC KEY-----\n"; + + const SH_CURRENT: &str = "PINNED_RELEASE_PUBLIC_KEY_CURRENT"; + const PS_CURRENT: &str = "$PinnedReleasePublicKeyCurrent"; + const RS_META: &str = "PINNED_METADATA_PUBLIC_KEY_PEM"; + + /// The file the check reads `token` from, taken from [`PINNED_KEYS`] so the + /// tests follow the constant when it moves to another file. + fn source_path(token: &str) -> &'static str { + PINNED_KEYS + .iter() + .flat_map(|key| key.sources) + .find(|source| source.token == token) + .map_or_else( + || panic!("{token} is not a pinned source"), + |source| source.path, + ) + } + + /// The Rust file holding the metadata key. + fn rs_path() -> &'static str { + source_path(RS_META) + } + + /// The Rust file's name, as the shorter error fragments name it. + fn rs_file() -> &'static str { + rs_path().rsplit('/').next().expect("a file name") + } + + /// A scratch repository with all three canonical keys published, so every + /// slot is live rather than dormant. + fn scratch_repo(install_sh: &str, install_ps1: &str, therock_rs: &str) -> tempfile::TempDir { + let dir = tempfile::tempdir().expect("tempdir"); + let root = dir.path(); + std::fs::create_dir_all(root.join("docs/keys")).expect("mkdir docs/keys"); + let rust = root.join(rs_path()); + std::fs::create_dir_all(rust.parent().expect("a parent")) + .expect("mkdir the Rust source dir"); + for (file, key) in [ + ("rocm-cli-release-current-public.pem", SAMPLE), + ("rocm-cli-release-next-public.pem", NEXT_KEY), + ("rocm-cli-metadata-public.pem", META), + ] { + std::fs::write(root.join("docs/keys").join(file), key).expect("write canonical key"); + } + std::fs::write(root.join("install.sh"), install_sh).expect("write install.sh"); + std::fs::write(root.join("install.ps1"), install_ps1).expect("write install.ps1"); + std::fs::write(rust, therock_rs).expect("write the Rust source"); + dir + } + + /// A key that differs from every canonical key in its base64 body. + fn wrong() -> String { + SAMPLE.replace("test", "evil") + } + + /// An install.sh whose CURRENT slot is `current` (the caller's lines), with a + /// correct NEXT slot and both marked readonly after them. + fn sh(current: &str) -> String { + format!( + "{current}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n" + ) + } + + fn sh_current() -> String { + format!("PINNED_RELEASE_PUBLIC_KEY_CURRENT=\"{SAMPLE}\"\n") + } + + fn good_sh() -> String { + sh(&sh_current()) + } + + /// An install.ps1 whose CURRENT slot is `current`, with a correct NEXT slot. + fn ps1(current: &str) -> String { + format!("{current}$PinnedReleasePublicKeyNext = @\"\n{NEXT_KEY}\"@\n") + } + + fn ps1_current() -> String { + format!("$PinnedReleasePublicKeyCurrent = @\"\n{SAMPLE}\"@\n") + } + + fn good_ps1() -> String { + ps1(&ps1_current()) + } + + fn good_rs() -> String { + format!("const PINNED_METADATA_PUBLIC_KEY_PEM: &str = \"{META}\";\n") + } + + /// Run the check over a scratch repo, require it to fail, and require the + /// error to contain every `expected` fragment, so a test cannot pass on an + /// unrelated failure in another slot. + #[track_caller] + fn rejected(install_sh: &str, install_ps1: &str, therock_rs: &str, expected: &[&str]) { + let repo = scratch_repo(install_sh, install_ps1, therock_rs); + let error = match check_pinned_keys(repo.path(), None) { + Ok(messages) => panic!( + "the check passed, but the source trusts a different key than \ + docs/keys; reported {messages:?}" + ), + Err(error) => format!("{error:#}"), + }; + for fragment in expected { + assert!( + error.contains(fragment), + "expected {fragment:?} in: {error}" + ); + } + } + + #[track_caller] + fn accepted(install_sh: &str, install_ps1: &str, therock_rs: &str) -> Vec { + let repo = scratch_repo(install_sh, install_ps1, therock_rs); + check_pinned_keys(repo.path(), None).unwrap_or_else(|error| panic!("{error:#}")) + } + + #[test] + fn scratch_repo_with_matching_keys_passes() { + let messages = accepted(&good_sh(), &good_ps1(), &good_rs()); + for expected in [ + "release-current: pinned in install.sh, install.ps1", + "release-next: pinned in install.sh, install.ps1", + &format!("metadata: pinned in {}", rs_path()), + ] { + assert!(messages.iter().any(|m| m == expected), "{messages:?}"); + } + } + + /// A slot without a canonical key must be empty: the installers trust any + /// non-empty slot, so a key pasted there would be trusted with nothing to + /// compare it against. + #[test] + fn a_dormant_slot_must_be_empty() { + let empty_next = |sh_next: &str, ps_next: &str, rs: &str| { + let sh = format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{sh_next}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ); + let ps = format!( + "{}$PinnedReleasePublicKeyNext = \"{ps_next}\"\n", + ps1_current() + ); + let repo = scratch_repo(&sh, &ps, rs); + for file in [ + "rocm-cli-release-next-public.pem", + "rocm-cli-metadata-public.pem", + ] { + std::fs::remove_file(repo.path().join("docs/keys").join(file)).expect("rm"); + } + check_pinned_keys(repo.path(), None).map_err(|error| format!("{error:#}")) + }; + let messages = empty_next( + "", + "", + "const PINNED_METADATA_PUBLIC_KEY_PEM: &str = \"\";\n", + ) + .expect("empty dormant slots pass"); + assert!( + messages.iter().any(|m| m + == "release-next: no canonical key yet — slots empty in install.sh, \ + install.ps1 (dormant)"), + "{messages:?}" + ); + + let error = empty_next( + NEXT_KEY, + "", + "const PINNED_METADATA_PUBLIC_KEY_PEM: &str = \"\";\n", + ) + .expect_err("a key pasted into a dormant shell slot must fail"); + assert!( + error.contains("install.sh pins a key in PINNED_RELEASE_PUBLIC_KEY_NEXT on line 6") + && error.contains("holds no canonical key"), + "{error}" + ); + let error = empty_next( + "", + NEXT_KEY, + "const PINNED_METADATA_PUBLIC_KEY_PEM: &str = \"\";\n", + ) + .expect_err("a key pasted into a dormant PowerShell slot must fail"); + assert!( + error.contains("install.ps1 pins a key in $PinnedReleasePublicKeyNext"), + "{error}" + ); + let error = empty_next("", "", &good_rs()) + .expect_err("a key pasted into a dormant Rust slot must fail"); + assert!( + error.contains(&format!("{} pins a key in {RS_META}", rs_path())), + "{error}" + ); + } + + /// A commented-out line holding the canonical key, above the real assignment + /// holding another key. The runtime never executes the comment, so the value + /// compared must be the real assignment's. + #[test] + fn shell_commented_canonical_above_a_different_assignment_is_rejected() { + let current = format!( + "# {SH_CURRENT}=\"{SAMPLE}\"\n{SH_CURRENT}=\"{}\"\n", + wrong() + ); + rejected( + &sh(¤t), + &good_ps1(), + &good_rs(), + &["install.sh does not embed the canonical key", SH_CURRENT], + ); + } + + #[test] + fn powershell_commented_canonical_above_a_different_assignment_is_rejected() { + let current = format!( + "# {PS_CURRENT} = @\"\n{SAMPLE}\"@\n{PS_CURRENT} = @\"\n{}\"@\n", + wrong() + ); + rejected( + &good_sh(), + &ps1(¤t), + &good_rs(), + &["install.ps1 does not embed the canonical key", PS_CURRENT], + ); + } + + #[test] + fn rust_commented_canonical_above_a_different_const_is_rejected() { + let rs = format!( + "// const {RS_META}: &str = \"{META}\";\nconst {RS_META}: &str = \"{}\";\n", + wrong() + ); + rejected( + &good_sh(), + &good_ps1(), + &rs, + &[ + &format!("{} does not embed the canonical key", rs_path()), + &format!("{RS_META} (line 6)"), + ], + ); + } + + /// The canonical key assigned first and a different one assigned later: the + /// shell keeps the last value it executed. + #[test] + fn shell_reassignment_after_the_canonical_key_is_rejected() { + let current = format!("{}{SH_CURRENT}=\"{}\"\n", sh_current(), wrong()); + rejected( + &sh(¤t), + &good_ps1(), + &good_rs(), + &[ + "install.sh assigns PINNED_RELEASE_PUBLIC_KEY_CURRENT 2 times (lines 1, 6)", + "leave exactly one, as `PINNED_RELEASE_PUBLIC_KEY_CURRENT=\"…\"`", + ], + ); + } + + #[test] + fn powershell_reassignment_after_the_canonical_key_is_rejected() { + let current = format!("{}{PS_CURRENT} = \"{}\"\n", ps1_current(), wrong()); + rejected( + &good_sh(), + &ps1(¤t), + &good_rs(), + &["install.ps1 assigns $PinnedReleasePublicKeyCurrent 2 times (lines 1, 7)"], + ); + } + + /// A longer identifier that merely starts with the token is a different + /// variable, and must not stand in for the real assignment after it. + #[test] + fn shell_identifier_sharing_the_token_as_a_prefix_is_not_the_assignment() { + let current = format!( + "{SH_CURRENT}_FPR=\"{SAMPLE}\"\n{SH_CURRENT}=\"{}\"\n", + wrong() + ); + rejected( + &sh(¤t), + &good_ps1(), + &good_rs(), + &["install.sh does not embed the canonical key", SH_CURRENT], + ); + } + + #[test] + fn powershell_identifier_sharing_the_token_as_a_prefix_is_not_the_assignment() { + let current = format!( + "{PS_CURRENT}Fpr = \"{SAMPLE}\"\n{PS_CURRENT} = \"{}\"\n", + wrong() + ); + rejected( + &good_sh(), + &ps1(¤t), + &good_rs(), + &["install.ps1 does not embed the canonical key", PS_CURRENT], + ); + } + + /// PowerShell variable names are case-insensitive, so a reassignment spelled + /// in another case is the same variable. + #[test] + fn powershell_reassignment_in_a_different_case_is_rejected() { + let current = format!( + "{}$pinnedreleasepublickeycurrent = \"{}\"\n", + ps1_current(), + wrong() + ); + rejected( + &good_sh(), + &ps1(¤t), + &good_rs(), + &["install.ps1", PS_CURRENT, "2 times"], + ); + } + + /// Comments, expansions and other reads of the name around the single real + /// assignment are not assignments and must not disturb the check; nor are + /// longer identifiers that contain the name. + #[test] + fn mentions_and_uses_around_the_single_assignment_pass() { + let sh = format!( + "# {SH_CURRENT} is the \"current\" trust root.\n\ + # {SH_CURRENT}=\"a retired key\"\n\ + {}\ + [ -n \"${{{SH_CURRENT}}}\" ] || exit 1\n\ + [ ${{#{SH_CURRENT}}} -gt 0 ] || exit 1\n\ + : ${{!{SH_CURRENT}}} \"${{!{SH_CURRENT}}}\"\n\ + for pem in \"${SH_CURRENT}\" \"${{PINNED_RELEASE_PUBLIC_KEY_NEXT}}\"; do :; done\n\ + export PINNED_RELEASE_PUBLIC_KEY_NEXT\n\ + {SH_CURRENT}_FPR=\"other\"\n\ + OLD_{SH_CURRENT}=\"other\"\n", + good_sh() + ); + let ps1 = format!( + "# {PS_CURRENT} holds the \"current\" trust root.\n\ + # {PS_CURRENT} = \"a retired key\"\n\ + {}\ + if ([string]::IsNullOrWhiteSpace({PS_CURRENT})) {{ exit 1 }}\n\ + foreach ($p in @({PS_CURRENT}, $PinnedReleasePublicKeyNext)) {{ }}\n\ + $copy = {PS_CURRENT}\n\ + $trimmed = \"$({PS_CURRENT}.Trim())\"\n\ + if ({PS_CURRENT} -eq \"\") {{ }}\n\ + {PS_CURRENT}Fpr = \"other\"\n\ + Set-Variable -Name OldPinnedReleasePublicKeyCurrent -Value \"other\"\n\ + Set-Variable -Name PinnedReleasePublicKeyCurrentFpr -Value \"other\"\n", + good_ps1() + ); + let rs = format!( + "// {RS_META} stays \"empty\" until published.\n\ + // const {RS_META}: &str = \"a retired key\";\n\ + {}\ + fn key() -> &'static str {{ {RS_META}.trim() }}\n\ + fn empty() -> bool {{ {RS_META} == \"\" }}\n\ + fn arm(k: &str) {{ match k {{ {RS_META} => {{}} _ => {{}} }} }}\n\ + const QUOTE: char = '\"';\n\ + const OLD_{RS_META}: &str = \"other\";\n", + good_rs() + ); + accepted(&sh, &ps1, &rs); + } + + /// Every shape that writes the variable counts towards the single-assignment + /// rule, before or after the canonical declaration. + #[test] + fn every_write_form_counts_as_an_assignment() { + let w = wrong(); + let shell_writes = [ + format!("export {SH_CURRENT}=\"{w}\"\n"), + format!("readonly {SH_CURRENT}=\"{w}\"\n"), + format!(" {SH_CURRENT}=\"{w}\"\n"), + format!("true && {SH_CURRENT}=\"{w}\"\n"), + format!(": \"${{{SH_CURRENT}:={w}}}\"\n"), + format!("{SH_CURRENT}+=\"x\"\n"), + format!("unset {SH_CURRENT}\n"), + format!("read -r {SH_CURRENT} < /dev/null\n"), + format!("export {SH_CURRENT}; unset {SH_CURRENT}\n"), + format!("echo \"$(unset {SH_CURRENT})\"\n"), + format!("[ ${{#{SH_CURRENT}}} -gt 0 ] && unset {SH_CURRENT}\n"), + // Quoting the name does not stop `unset` or `read` from writing it. + format!("unset '{SH_CURRENT}'\n"), + format!("read \"{SH_CURRENT}\" < /dev/null\n"), + ]; + for write in &shell_writes { + for current in [ + format!("{}{write}", sh_current()), + format!("{write}{}", sh_current()), + ] { + rejected( + &sh(¤t), + &good_ps1(), + &good_rs(), + &["install.sh", SH_CURRENT, " times (lines "], + ); + } + } + + let mut powershell_writes = vec![ + format!("$script:PinnedReleasePublicKeyCurrent = \"{w}\"\n"), + format!("$Global:PINNEDRELEASEPUBLICKEYCURRENT=\"{w}\"\n"), + format!("${{PinnedReleasePublicKeyCurrent}} = \"{w}\"\n"), + format!("if ($true) {{ {PS_CURRENT} = \"{w}\" }}\n"), + format!("Set-Variable -Name PinnedReleasePublicKeyCurrent -Value \"{w}\"\n"), + format!("New-Variable PinnedReleasePublicKeyCurrent \"{w}\" -Force\n"), + format!("Set-Item variable:PinnedReleasePublicKeyCurrent \"{w}\"\n"), + "Remove-Variable PinnedReleasePublicKeyCurrent\n".to_owned(), + // Quoting the name does not stop a cmdlet from writing it. + format!("Set-Variable -Name 'PinnedReleasePublicKeyCurrent' -Value '{w}'\n"), + format!("Set-Variable -Name \"PinnedReleasePublicKeyCurrent\" -Value \"{w}\"\n"), + format!("Set-Item 'variable:PinnedReleasePublicKeyCurrent' '{w}'\n"), + // A drive-qualified name is not this variable, but counting it fails closed. + format!("$env:PinnedReleasePublicKeyCurrent = \"{w}\"\n"), + ]; + powershell_writes.extend( + ["+=", "-=", "*=", "/=", "%=", "??="] + .iter() + .map(|op| format!("{PS_CURRENT} {op} \"x\"\n")), + ); + for write in &powershell_writes { + for current in [ + format!("{}{write}", ps1_current()), + format!("{write}{}", ps1_current()), + ] { + rejected( + &good_sh(), + &ps1(¤t), + &good_rs(), + &["install.ps1", PS_CURRENT, "2 times"], + ); + } + } + + for write in [ + format!("fn f() {{ let {RS_META} = \"{w}\"; }}\n"), + format!("static {RS_META}: &str = \"{w}\";\n"), + format!("mod m {{ const {RS_META}: &str = \"{w}\"; }}\n"), + format!("/*\nconst {RS_META}: &str = \"{w}\";\n*/\n"), + ] { + for rs in [ + format!("{}{write}", good_rs()), + format!("{write}{}", good_rs()), + ] { + rejected( + &good_sh(), + &good_ps1(), + &rs, + &[rs_path(), RS_META, "2 times"], + ); + } + } + } + + /// The declaration shapes the value is read from, beyond the plain ones. + #[test] + fn supported_declaration_forms_are_read() { + // `readonly NAME=` marks itself; `export NAME=` still needs the readonly line. + let sh = format!( + "readonly {SH_CURRENT}=\"{SAMPLE}\"\n\ + export PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_NEXT\n" + ); + let ps1 = format!( + "$script:pinnedReleasePublicKeyCurrent = \"{SAMPLE}\"\n\ + $Global:PinnedReleasePublicKeyNext = \"{NEXT_KEY}\" # rotation slot\n" + ); + for rs in [ + format!("pub const {RS_META}: &str = \"{META}\"; // metadata trust root\n"), + format!("pub(crate) const {RS_META}: &str = \"{META}\";\n"), + ] { + accepted(&sh, &ps1, &rs); + } + for scope in ["local", "private", "SCRIPT", "global"] { + accepted( + &good_sh(), + &good_ps1().replacen( + "$PinnedReleasePublicKeyCurrent = ", + &format!("${scope}:PinnedReleasePublicKeyCurrent = "), + 1, + ), + &good_rs(), + ); + } + // CRLF line endings, as a Windows checkout of install.ps1 may have. + accepted(&good_sh(), &good_ps1().replace('\n', "\r\n"), &good_rs()); + } + + /// No assignment at all, or a single one the value cannot be read from, is an + /// error naming the file and the variable rather than a silent mismatch. + #[test] + fn missing_or_unreadable_assignment_is_reported() { + rejected( + &sh(""), + &good_ps1(), + &good_rs(), + &["install.sh has no assignment of PINNED_RELEASE_PUBLIC_KEY_CURRENT"], + ); + + // A lone write in a shape the value cannot be read from: conditional, + // compound, braced, a cmdlet, or a drive-qualified name such as `$env:` + // that is not this variable at all. + for lone in [ + format!("Set-Variable -Name PinnedReleasePublicKeyCurrent -Value \"{SAMPLE}\"\n"), + format!("if ($true) {{ {PS_CURRENT} = \"{SAMPLE}\" }}\n"), + format!("{PS_CURRENT} += \"{SAMPLE}\"\n"), + format!("${{PinnedReleasePublicKeyCurrent}} = \"{SAMPLE}\"\n"), + format!("$env:PinnedReleasePublicKeyCurrent = \"{SAMPLE}\"\n"), + format!("$function:PinnedReleasePublicKeyCurrent = \"{SAMPLE}\"\n"), + ] { + rejected( + &good_sh(), + &ps1(&lone), + &good_rs(), + &[ + "install.ps1 assigns $PinnedReleasePublicKeyCurrent on line", + "unsupported form", + ], + ); + } + for lone in [ + format!("[ -n \"$CI\" ] && {SH_CURRENT}=\"{SAMPLE}\"\n"), + format!(": \"${{{SH_CURRENT}:={SAMPLE}}}\"\n"), + format!("{SH_CURRENT}+=\"{SAMPLE}\"\n"), + ] { + rejected( + &sh(&lone), + &good_ps1(), + &good_rs(), + &[ + "install.sh assigns PINNED_RELEASE_PUBLIC_KEY_CURRENT on line", + "unsupported form", + ], + ); + } + for rs in [ + format!("fn f() {{ let {RS_META}: &str = \"{META}\"; }}\n"), + format!("static {RS_META}: &str = \"{META}\";\n"), + ] { + rejected( + &good_sh(), + &good_ps1(), + &rs, + &[&format!("{} assigns", rs_file()), "unsupported form"], + ); + } + + rejected( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = '{SAMPLE}'\n")), + &good_rs(), + &["not a double-quoted string"], + ); + rejected( + &sh(&format!("{SH_CURRENT}=$(printf '%s' \"{SAMPLE}\")\n")), + &good_ps1(), + &good_rs(), + &["not a double-quoted string"], + ); + } + + /// The value must be the key the runtime sees: one well-formed PEM block, + /// and nothing the installer would expand or keep verbatim. + #[test] + fn the_value_must_be_exactly_the_pem_the_runtime_sees() { + const MALFORMED: &str = "is not one well-formed PUBLIC KEY PEM block"; + // A key prepended through an expansion: openssl uses the first PEM. + rejected( + &sh(&format!("{SH_CURRENT}=\"${{EXTRA_PEM}}{SAMPLE}\"\n")), + &good_ps1(), + &good_rs(), + &["install.sh PINNED_RELEASE_PUBLIC_KEY_CURRENT on line 1 holds `$`"], + ); + rejected( + &sh(&format!("{SH_CURRENT}=\"`cat extra.pem`{SAMPLE}\"\n")), + &good_ps1(), + &good_rs(), + &["install.sh PINNED_RELEASE_PUBLIC_KEY_CURRENT on line 1 holds ```, which"], + ); + rejected( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = @\"\n$($extra){SAMPLE}\"@\n")), + &good_rs(), + &["install.ps1 $PinnedReleasePublicKeyCurrent on line 1 holds `$`"], + ); + // In a shell double-quoted string `\n` stays a backslash and an `n`. + rejected( + &sh(&format!( + "{SH_CURRENT}=\"{}\"\n", + SAMPLE.replace('\n', "\\n") + )), + &good_ps1(), + &good_rs(), + &["holds `\\`"], + ); + // Accidental edits that leave the base64 letters intact but a key openssl + // cannot load, or a second block after the first. + let body_line = SAMPLE.lines().nth(1).expect("a body line"); + for broken in [ + SAMPLE.replacen("MIIBIjAN", "MIIBIjAN.-,;", 1), + SAMPLE.replacen(body_line, &format!("{body_line}\n"), 1), + SAMPLE.replacen(body_line, &format!("{body_line}\n----"), 1), + format!("{SAMPLE}{}", wrong()), + format!("junk\n{SAMPLE}"), + format!("{SAMPLE}junk"), + format!("{PEM_BEGIN}\n{PEM_END}\n"), + // A different key type's armour around the right body. + SAMPLE.replace(PEM_BEGIN, "-----BEGIN RSA PUBLIC KEY-----"), + SAMPLE.replace(PEM_END, "-----END RSA PUBLIC KEY-----"), + ] { + rejected( + &sh(&format!("{SH_CURRENT}=\"{broken}\"\n")), + &good_ps1(), + &good_rs(), + &[ + "install.sh PINNED_RELEASE_PUBLIC_KEY_CURRENT on line 1", + MALFORMED, + ], + ); + } + rejected( + &good_sh(), + &ps1(&format!( + "{PS_CURRENT} = @\"\n{}\"@\n", + SAMPLE.replacen("MIIB", "MI;IB", 1) + )), + &good_rs(), + &[ + "install.ps1 $PinnedReleasePublicKeyCurrent on line 1", + MALFORMED, + ], + ); + // A key cut short is well formed but not the canonical key. + let truncated = SAMPLE.replacen(&format!("{body_line}\n"), "", 1); + rejected( + &sh(&format!("{SH_CURRENT}=\"{truncated}\"\n")), + &good_ps1(), + &good_rs(), + &["install.sh does not embed the canonical key"], + ); + // An indented key: install.sh and the Rust source pass the value on + // verbatim, while install.ps1 trims each line before decoding. The message + // names that rule rather than one PEM reader's limits, since the Rust + // source never goes through `openssl`. + for indent in [" ", "\t"] { + let indented = SAMPLE.replace('\n', &format!("\n{indent}")); + let indented = indented.trim_end(); + rejected( + &sh(&format!("{SH_CURRENT}=\"{indent}{indented}\n\"\n")), + &good_ps1(), + &good_rs(), + &[ + "install.sh PINNED_RELEASE_PUBLIC_KEY_CURRENT on line 1 has a key line that starts with whitespace", + "only install.ps1, which trims each line, may indent a pinned key", + ], + ); + rejected( + &good_sh(), + &good_ps1(), + &format!( + "const {RS_META}: &str = \"{}\";\n", + META.replace('\n', &format!("\n{indent}")).trim_end() + ), + &[ + &format!( + "{} {RS_META} on line 1 has a key line that starts with whitespace", + rs_file() + ), + "only install.ps1, which trims each line, may indent a pinned key", + ], + ); + accepted( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = @\"\n{indent}{indented}\n\"@\n")), + &good_rs(), + ); + } + // Rust turns `\n` into a line break, so that form is the same key; any + // other escape is not. + accepted( + &good_sh(), + &good_ps1(), + &format!( + "const {RS_META}: &str = \"{}\";\n", + META.replace('\n', "\\n") + ), + ); + rejected( + &good_sh(), + &good_ps1(), + &format!( + "const {RS_META}: &str = \"{}\";\n", + META.replace('\n', "\\t") + ), + &[&format!("{} {RS_META} on line 1 holds `\\`", rs_file())], + ); + } + + /// Publishing a canonical key while a slot is still empty must fail: an empty + /// slot turns signature verification off in the installers. + #[test] + fn a_published_key_with_an_empty_slot_is_rejected() { + rejected( + &sh(&format!("{SH_CURRENT}=\"\"\n")), + &good_ps1(), + &good_rs(), + &["install.sh does not embed the canonical key", SH_CURRENT], + ); + rejected( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = \"\"\n")), + &good_rs(), + &["install.ps1 does not embed the canonical key", PS_CURRENT], + ); + rejected( + &good_sh(), + &good_ps1(), + &format!("const {RS_META}: &str = \"\";\n"), + &[ + &format!("{} does not embed the canonical key", rs_file()), + RS_META, + ], + ); + } + + /// A canonical file that is empty means "not published yet"; one that holds + /// something other than a key is an error rather than a silent skip. + #[test] + fn canonical_files_are_read_strictly() { + let repo = scratch_repo(&good_sh(), &good_ps1(), &good_rs()); + let next = repo + .path() + .join("docs/keys/rocm-cli-release-next-public.pem"); + std::fs::write(&next, "not a key\n").expect("write garbage"); + let error = format!( + "{:#}", + check_pinned_keys(repo.path(), None).expect_err("garbage") + ); + assert!( + error.contains( + "docs/keys/rocm-cli-release-next-public.pem is not a valid PUBLIC KEY PEM" + ), + "{error}" + ); + std::fs::write(&next, " \n").expect("write blank"); + let error = format!( + "{:#}", + check_pinned_keys(repo.path(), None).expect_err("blank") + ); + assert!( + error.contains("pins a key in PINNED_RELEASE_PUBLIC_KEY_NEXT") + && error.contains("holds no canonical key"), + "a blank canonical file is unpublished, so the filled slot must fail: {error}" + ); + } + + /// The statement must end with the string. Anything after the closing quote + /// changes the value the runtime ends up with: a PowerShell operator or + /// method rewrites it, a second quoted word appends to it, and a trailing + /// shell command turns the assignment into a prefix that leaves it unset. + #[test] + fn nothing_may_follow_the_closing_quote() { + // `"@` closes only a here-string that opened with `@"`. + rejected( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = \"{SAMPLE}\"@\n")), + &good_rs(), + &[ + "install.ps1 assigns $PinnedReleasePublicKeyCurrent on line 1 to a string followed by `@", + ], + ); + for tail in [ + " -replace \"MIIB\", \"MIIC\"", + ".Replace(\"MIIB\", \"MIIC\")", + ".Substring(5)", + " + \"x\"", + ] { + rejected( + &good_sh(), + &ps1(&format!("{PS_CURRENT} = @\"\n{SAMPLE}\"@{tail}\n")), + &good_rs(), + &[ + "install.ps1 assigns $PinnedReleasePublicKeyCurrent on line 1 to a string followed by", + ], + ); + } + for tail in [" true", " | cat", "'extra'", "x", "; unset X", "@"] { + rejected( + &sh(&format!("{SH_CURRENT}=\"{SAMPLE}\"{tail}\n")), + &good_ps1(), + &good_rs(), + &[ + "install.sh assigns PINNED_RELEASE_PUBLIC_KEY_CURRENT on line 1 to a string followed by", + ], + ); + } + rejected( + &good_sh(), + &good_ps1(), + &format!("const {RS_META}: &str = \"{META}\"; const X: u8 = 1;\n"), + &[&format!( + "{} assigns {RS_META} on line 1 to a string followed by", + rs_file() + )], + ); + + // An empty-looking slot that the runtime fills: the dormant NEXT slot has + // no canonical key to compare against, so this is the case that matters. + let dormant = |sh: &str, ps: &str| { + let repo = scratch_repo(sh, ps, &good_rs()); + std::fs::remove_file( + repo.path() + .join("docs/keys/rocm-cli-release-next-public.pem"), + ) + .expect("rm"); + check_pinned_keys(repo.path(), None).map_err(|error| format!("{error:#}")) + }; + let error = dormant( + &format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"\"'{NEXT_KEY}'\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ), + &good_ps1(), + ) + .expect_err("a quoted key glued to an empty shell slot must fail"); + assert!( + error.contains("PINNED_RELEASE_PUBLIC_KEY_NEXT on line 6 to a string followed by"), + "{error}" + ); + let error = dormant( + &format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ), + &format!( + "{}$PinnedReleasePublicKeyNext = \"\" + @\"\n{NEXT_KEY}\"@\n", + ps1_current() + ), + ) + .expect_err("a here-string added to an empty PowerShell slot must fail"); + assert!( + error.contains("$PinnedReleasePublicKeyNext on line 7 to a string followed by"), + "{error}" + ); + + // `""#junk` is the string `#junk`, and `" "` is not empty to `[ -n … ]`. + for next in ["\"\"#junk", "\" \""] { + let error = dormant( + &format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT={next}\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ), + &good_ps1(), + ) + .expect_err("a slot that only looks empty must fail"); + assert!( + error.contains("PINNED_RELEASE_PUBLIC_KEY_NEXT on line 6"), + "{error}" + ); + } + + // A trailing comment, or `;` before one, is fine. + accepted( + &sh(&format!("{SH_CURRENT}=\"{SAMPLE}\" # current trust root\n")), + &ps1(&format!("{PS_CURRENT} = @\"\n{SAMPLE}\"@ # current\n")), + &format!("const {RS_META}: &str = \"{META}\"; // metadata trust root\n"), + ); + accepted( + &sh(&format!("{SH_CURRENT}=\"{SAMPLE}\";# current\n")), + &ps1(&format!("{PS_CURRENT} = \"{SAMPLE}\"; # current\n")), + &good_rs(), + ); + } + + /// install.sh must mark each slot readonly after assigning it, so a later + /// write fails at runtime rather than silently swapping the trust root. + #[test] + fn install_sh_must_mark_each_slot_readonly() { + let missing = format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n", + sh_current() + ); + rejected( + &missing, + &good_ps1(), + &good_rs(), + &[ + "install.sh does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT readonly after assigning it on line 1", + ], + ); + let only_next = format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\nreadonly PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ); + rejected( + &only_next, + &good_ps1(), + &good_rs(), + &["does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT"], + ); + let before = format!( + "readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n{}\ + PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n", + sh_current() + ); + rejected( + &before, + &good_ps1(), + &good_rs(), + &["does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT"], + ); + let commented = format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + # readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ); + rejected( + &commented, + &good_ps1(), + &good_rs(), + &["does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT"], + ); + // `export` sets an attribute too, but not this one. + let exported = format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + export PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ); + rejected( + &exported, + &good_ps1(), + &good_rs(), + &["does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT"], + ); + + // A `readonly` command must name the slot itself, without flags: a read + // of it, a mention in the trailing comment, `-f` (which names a + // function) and `-p` (which only prints) leave the variable writable. + let with_line = |line: &str| { + format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n{line}\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_NEXT\n", + sh_current() + ) + }; + // An indented `readonly` sits in a block or function that may never run + // (`if false; then … fi`), so it does not count either. Each of these + // is told what a qualifying line looks like, since the slot's name is + // already on the line it rejected. + for line in [ + "readonly COPY=\"${PINNED_RELEASE_PUBLIC_KEY_CURRENT}\"", + "readonly OTHER # PINNED_RELEASE_PUBLIC_KEY_CURRENT", + "readonly OTHER\t# PINNED_RELEASE_PUBLIC_KEY_CURRENT", + "readonly -f PINNED_RELEASE_PUBLIC_KEY_CURRENT", + "readonly -p PINNED_RELEASE_PUBLIC_KEY_CURRENT", + "if false; then\n readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT\nfi", + "f() {\n\treadonly PINNED_RELEASE_PUBLIC_KEY_CURRENT\n}", + ] { + rejected( + &with_line(line), + &good_ps1(), + &good_rs(), + &[ + "install.sh does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT readonly", + "add a line `readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT` after the assignment", + "it must start the line, with no flags, and name the slot outside any comment", + ], + ); + } + // A trailing comment after an assignment is not an attribute command at + // all, so the mention in it counts as a second write of the slot. + rejected( + &with_line("readonly OTHER=1 # PINNED_RELEASE_PUBLIC_KEY_CURRENT"), + &good_ps1(), + &good_rs(), + &[ + "install.sh assigns PINNED_RELEASE_PUBLIC_KEY_CURRENT 2 times (lines 1, 11)", + "keep the declaration on line 1", + ], + ); + + // Accepted: `readonly NAME=` marks itself, and a trailing comment is fine. + accepted( + &format!( + "readonly {SH_CURRENT}=\"{SAMPLE}\"\nPINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_NEXT # rotation slot\n" + ), + &good_ps1(), + &good_rs(), + ); + // The declaration form follows the same start-of-line rule: indented, it + // may sit in a block that never runs, so it does not mark the slot. + rejected( + &format!( + "if true; then\n readonly {SH_CURRENT}=\"{SAMPLE}\"\nfi\n\ + PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_NEXT\n" + ), + &good_ps1(), + &good_rs(), + &[ + "install.sh does not mark PINNED_RELEASE_PUBLIC_KEY_CURRENT readonly after assigning it on line 2", + "it must start the line", + ], + ); + // A comment after a tab is a comment too, so the command still only + // names variables: it marks both slots rather than counting as a second + // write of them. + accepted( + &format!( + "{}PINNED_RELEASE_PUBLIC_KEY_NEXT=\"{NEXT_KEY}\"\n\ + readonly PINNED_RELEASE_PUBLIC_KEY_CURRENT PINNED_RELEASE_PUBLIC_KEY_NEXT\t# note\n", + sh_current() + ), + &good_ps1(), + &good_rs(), + ); + // `#` inside a word does not start a comment, so the command is not one + // that only names variables, and the mention counts as a write. + rejected( + &with_line("readonly OTHER a#b PINNED_RELEASE_PUBLIC_KEY_CURRENT"), + &good_ps1(), + &good_rs(), + &["install.sh assigns PINNED_RELEASE_PUBLIC_KEY_CURRENT 2 times"], + ); + } + + /// The real installers and Rust source pass as committed, and the check still + /// bites on them. The copy takes every canonical key the repository + /// publishes, so this keeps holding once the NEXT or metadata key is added. + #[test] + fn the_real_sources_pass_and_their_hardening_is_enforced() { + let real = repo_root(); + let copy = |install_sh: &str| { + let dir = tempfile::tempdir().expect("tempdir"); + let published = PINNED_KEYS + .iter() + .map(|key| key.canonical) + .filter(|path| real.join(path).exists()); + // Every source the check reads except install.sh, which the caller + // supplies, so a source that moves is copied from where it now lives. + let sources = PINNED_KEYS + .iter() + .flat_map(|key| key.sources) + .map(|source| source.path) + .filter(|&path| path != "install.sh"); + for path in published.chain(sources) { + let to = dir.path().join(path); + std::fs::create_dir_all(to.parent().unwrap()).expect("mkdir"); + std::fs::copy(real.join(path), to).expect("copy"); + } + std::fs::write(dir.path().join("install.sh"), install_sh).expect("write install.sh"); + let result = check_pinned_keys(dir.path(), None).map_err(|e| format!("{e:#}")); + (dir, result) + }; + let installer = std::fs::read_to_string(real.join("install.sh")).expect("read install.sh"); + copy(&installer).1.expect("the committed sources pass"); + + // Without its readonly line(s), install.sh fails. + let without_readonly: String = installer + .lines() + .filter(|line| !(is_attribute_command(line) && line.starts_with("readonly "))) + .map(|line| format!("{line}\n")) + .collect::>() + .concat(); + assert_ne!( + without_readonly, installer, + "install.sh has no readonly line" + ); + let error = copy(&without_readonly).1.expect_err("readonly enforced"); + assert!(error.contains("readonly after assigning it"), "{error}"); + + // A commented copy of the real key above a different one fails. + let declaration = "PINNED_RELEASE_PUBLIC_KEY_CURRENT=\"-----BEGIN PUBLIC KEY-----\n"; + let start = installer + .find(declaration) + .expect("the CURRENT declaration"); + let end = start + installer[start..].find("-----\"\n").expect("its end") + 7; + let commented = installer[start..end] + .lines() + .map(|line| format!("# {line}\n")) + .collect::>() + .concat(); + let tampered = format!( + "{}{commented}{}", + &installer[..start], + installer[start..].replacen("MIIB", "MIIC", 1) + ); + let error = copy(&tampered) + .1 + .expect_err("commented canonical above a wrong key"); + assert!( + error.contains("does not embed the canonical key"), + "{error}" + ); + } + #[test] fn ci_public_key_cross_check() { let current = pem_body(SAMPLE).unwrap(); From 6a8b45edde979e53c4681c7c0a882807d3b23ac3 Mon Sep 17 00:00:00 2001 From: Eugene Volen <150254791+volen-silo@users.noreply.github.com> Date: Mon, 9 Nov 2026 13:32:05 +0000 Subject: [PATCH 017/102] fix(skill): run examine for an unnamed vendor and finish the vocabulary (#976) Review of #909 found four things left over. The rule that asks which GPU the machine has when rocm examine reports no-amd-gpu had no step that ran examine, so a user who named no vendor on a host without an AMD GPU still got has_match false and an upstream route. Step 1 now runs rocm examine --json first when no vendor is named. The scope gate's closing line, "only continue when the GPU is AMD", could be read as overriding the unnamed-vendor rule, and nothing brought the decline back when the user then named another vendor. It now continues for an AMD or an unnamed vendor and declines for NVIDIA, Intel or Apple. "The auto-applicable set" was still in the contract-check paragraph, and step 1 showed the auto_applicable flag while step 3 said "has a runner". The paragraph now says which fixes have a runner, and step 3 says what the flag means: true only where the fix class is auto on this host. The closing pointer to reference.md lost "the full closed catalog", and the skill card's Description had become a run-on that dropped "print-only" for the two vLLM failures. Both are restored. Review then added three more. The Workflow intro still said the GPU is AMD, which stopped an unnamed vendor there too; it now says AMD or no vendor was named. After asking which GPU, the flow had no exit for an answer of AMD, or for a user who cannot say, such as a container without GPU passthrough. It now continues with rocm diagnose, which only reads. On WSL2 examine reports status wsl whatever the GPU, so step 1 reads has_amd_gpu there. fix-2 is print-only on WSL2 as well as Linux, so the notes say Linux and WSL2. A second review pass found two more. A failed rocm examine fell under "any other result: go on". examine exits 0 whenever it ran, so a non-zero exit is an error from the CLI; step 1 now relays the error and stops, as the contract check does. The scope gate's "treat ROCm / HIP wording as AMD and continue" read as in tension with step 1, which then asks which GPU. It now says to continue to step 1, which checks the host. Signed-off-by: Eugene Volen (cherry picked from commit b48c4ae8a2ea7373ce538c704e7386358515d840) Signed-off-by: juhovainio --- skills/rocm-doctor/SKILL.md | 52 ++++++++++++++++++++------------ skills/rocm-doctor/skill-card.md | 8 ++--- 2 files changed, 36 insertions(+), 24 deletions(-) diff --git a/skills/rocm-doctor/SKILL.md b/skills/rocm-doctor/SKILL.md index cffb776470..c3377ee89b 100644 --- a/skills/rocm-doctor/SKILL.md +++ b/skills/rocm-doctor/SKILL.md @@ -45,12 +45,10 @@ advice for a stack this skill does not cover is worse than no advice. A fresh install on a clean machine is out of scope too: that is a setup task, not a diagnosis. Exit cleanly and say so. -If the user names no GPU vendor, treat ROCm / HIP wording as AMD and continue. -Should `rocm examine` then report `no-amd-gpu`, the CLI found no AMD GPU: say so, -ask which GPU the machine has, and propose no fix until that is settled. That -status alone does not prove the GPU is not AMD. - -Only continue past this gate when the GPU is AMD. +If the user names no GPU vendor, treat ROCm / HIP wording as AMD and continue to +step 1, which checks what the machine has. Continue when the GPU is AMD or no +vendor is named. If the user then says it is NVIDIA, Intel or Apple, decline as +above. **WSL2 is in scope** and does not stop this gate. It is a platform family of its own, with its own catalog entries covering `/dev/dxg`, the DXCore handoff, @@ -76,7 +74,7 @@ offered there. ## Workflow Only start here once the [Scope gate](#scope-gate--check-before-anything-else) -passes — the GPU is AMD. +passes — the GPU is AMD or no vendor was named. 0. **Ensure the `rocm` CLI is present.** Everything below shells out to it, so check first and install it if missing: @@ -134,11 +132,23 @@ passes — the GPU is AMD. Any other failure, such as a different exit code or output with no `contract_version`, is an error from the CLI and not proof that it is old. Relay the error text and stop, because reinstalling the same version would not - clear it. Against a CLI without this contract, the entry ids, the - auto-applicable set, and the exit codes this skill documents are guesses, and a - guess presented as a fact is the failure this check prevents. - -1. **Diagnose.** Pass the user's error text as the symptom: + clear it. Against a CLI without this contract, the entry ids, which fixes have + a runner, and the exit codes this skill documents are guesses, and a guess + presented as a fact is the failure this check prevents. + +1. **Diagnose.** If the user named no GPU vendor, run `rocm examine --json` + first. Status `no-amd-gpu` means the CLI found no AMD GPU. On WSL2 the status + is `wsl` whatever the GPU, so read `has_amd_gpu` there: false means the same. + Either way, say so, ask which GPU the machine has, and propose no fix until + that is settled. That result alone does not prove the GPU is not AMD: a + container without GPU passthrough, or the very failure being diagnosed, can + hide it. If the user confirms AMD or cannot say, continue with `rocm diagnose` + below, which only reads; if the answer is NVIDIA, Intel or Apple, the scope + gate declines. Any other result: go on. If `rocm examine` itself fails (it + exits 0 whenever it ran, so a non-zero exit is an error from the CLI), relay + the error text and stop, as the contract check does. + + Pass the user's error text as the symptom: ``` rocm diagnose --symptom "" --json @@ -228,7 +238,9 @@ passes — the GPU is AMD. Only four fixes have a runner the CLI executes itself. The other 24 are **print-only** (bootloader, kernel, reinstall, Windows driver, …): `rocm fix ` just prints the plan for the user to run themselves — no prompt, and the - CLI never performs those. + CLI never performs those. In a diagnosis, `fix.auto_applicable` is true only + for a fix whose `fix.class` is auto on this host, so it is false for + `fix-9-igpu-dgpu` and for `fix-2-unset-override` on Linux and WSL2. One entry is in neither group. `info-20-gpu-no-rocm-support` is **diagnose-only**: `rocm fix` takes the id, names the problem, says there is @@ -236,11 +248,11 @@ passes — the GPU is AMD. Of the four, two do not always mutate: - - **`fix-2-unset-override` mutates on Windows only.** On Linux it reports - where the override is set and which rc files carry it, then stops — it - will not edit the user's dotfiles. So there is no prompt to answer and - nothing for `--dry-run` to preview. Tell the Linux user what to edit; do - not describe it as a change the CLI made or previewed. + - **`fix-2-unset-override` mutates on Windows only.** On Linux and WSL2 it + reports where the override is set and which rc files carry it, then stops + — it will not edit the user's dotfiles. So there is no prompt to answer and + nothing for `--dry-run` to preview. Tell the user what to edit; do not + describe it as a change the CLI made or previewed. - **`fix-9-igpu-dgpu` mutates only when `--device-index N` is passed.** Without it — on either platform — the runner just prints the `rocminfo`/`hipInfo` query that identifies the discrete GPU's index and @@ -308,5 +320,5 @@ start of the reply, before relaying a multi-step plan. first. New failure modes are added to the CLI catalog, not improvised here. Read [reference.md](reference.md) when `rocm fix` exits non-zero (its exit codes -are listed there), and before telling the user which platforms a fix id applies -to. +are listed there), before telling the user which platforms a fix id applies to, +and for the full closed catalog. diff --git a/skills/rocm-doctor/skill-card.md b/skills/rocm-doctor/skill-card.md index 96a01cdcaf..94e3dcf1d4 100644 --- a/skills/rocm-doctor/skill-card.md +++ b/skills/rocm-doctor/skill-card.md @@ -3,10 +3,10 @@ ## Description Diagnoses ROCm / HIP SDK / PyTorch / llama.cpp failures on AMD GPUs (Linux, -Windows, and WSL2) against a closed list of known misconfigurations, and applies a -low-risk fix with consent or routes the user to the right upstream channel, and -covers two vLLM startup failures (running out of GPU memory, an engine import -error) on Linux and WSL2. +Windows, and WSL2) against a closed list of known misconfigurations, including two +print-only vLLM startup failures on Linux and WSL2 (running out of GPU memory, an +engine import error), and applies a low-risk fix with consent or routes the user +to the right upstream channel. ## Owner From 38ba2a4b6904e2a81c1ddd69471a1c245d8caf19 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 13:35:43 +0000 Subject: [PATCH 018/102] test(e2e-report): make the gating-phrase source test require the constants in use, not only named (#952) * test(e2e-report): make the gating-phrase source test require the constants in use, not only named The test checked that markdown.rs and html.rs contained the constant names as substrings, which an import or a comment satisfies even after a renderer stops using the constant. Strip use statements and comments from the production source first and require each constant as a {NAME} format interpolation or a (NAME) macro splice. The negative half is unchanged. Signed-off-by: Jussi Elo * test(e2e-report): narrow the gating-phrase usage check and make its scanner fail loudly on raw strings references_constant now wants a {NAME} interpolation inside a string literal or a (NAME) splice outside one that does not follow an identifier, so a call argument or a spliced-in-text mention no longer counts. The stripper now reads escaped one-character char literals, documents that it is a scanner rather than a lexer, and the test asserts the renderers contain no raw string, which the scanner cannot read. The test comment states that placement in the sentence is carried by the verbatim pins. Signed-off-by: Jussi Elo * test(e2e-report): treat C raw strings as raw in the gating-phrase scanner guard contains_raw_string rejected a candidate whose r was preceded by c, so cr"..." and cr#"..."# reached the scanner unflagged. Recognise the cr prefix alongside br, and cover the cr forms, plain b and c strings and identifiers merely ending in br or cr in the fixtures. Signed-off-by: Jussi Elo * test(e2e-report): ignore escaped braces in the gating-phrase placeholder check and document the scanner's gaps references_constant counted {NAME} inside {{NAME}}, which a format string prints as literal text. Count the run of opening braces before the name and accept only an odd run. Also document that a string never passed to a format macro still satisfies the check, that contains_raw_string trips on ordinary strings ending in r such as "\r", and that the scanner does not read the hex char escapes '\x41' and '\u{41}', each with a test pinning the behavior. Signed-off-by: Jussi Elo * test(e2e-report): replace the source scanner by rendered-output assertions The comment and use stripper, the placeholder matcher and the raw-string guard decided by syntax rather than by use, and the positive half they served only closed the case of a constant's value changing while a renderer still hardcodes the old words. Assert instead, in the two verbatim pins, that the rendered markdown and HTML contain MOCK_GATES_PR and OTHER_LANES_NON_BLOCKING, which proves the placement directly. The negative half of the source test, which catches a renderer re-typing the current phrase, is unchanged. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 6531c79979a197bc7d4e618f1dc99e9af085e966) Signed-off-by: juhovainio --- crates/e2e-report/src/consolidated/html.rs | 11 +++++++++++ crates/e2e-report/src/consolidated/markdown.rs | 11 +++++++++++ crates/e2e-report/src/consolidated/mod.rs | 18 ++++++++---------- 3 files changed, 30 insertions(+), 10 deletions(-) diff --git a/crates/e2e-report/src/consolidated/html.rs b/crates/e2e-report/src/consolidated/html.rs index a58151b786..6c4717152f 100644 --- a/crates/e2e-report/src/consolidated/html.rs +++ b/crates/e2e-report/src/consolidated/html.rs @@ -341,6 +341,17 @@ mod tests {
  • Every other platform — real self-hosted GPU hardware; \ non-blocking while proven out.
  • "; assert!(html.contains(expected), "legend text changed:\n{html}"); + // The pin above would still pass if the legend hardcoded the words while + // a shared constant's value moved on; these tie the rendered text to the + // constants themselves. + assert!( + html.contains(MOCK_GATES_PR), + "legend must render MOCK_GATES_PR ({MOCK_GATES_PR:?}):\n{html}" + ); + assert!( + html.contains(OTHER_LANES_NON_BLOCKING), + "legend must render OTHER_LANES_NON_BLOCKING ({OTHER_LANES_NON_BLOCKING:?}):\n{html}" + ); } #[test] diff --git a/crates/e2e-report/src/consolidated/markdown.rs b/crates/e2e-report/src/consolidated/markdown.rs index d89ee11822..60fb5a4362 100644 --- a/crates/e2e-report/src/consolidated/markdown.rs +++ b/crates/e2e-report/src/consolidated/markdown.rs @@ -393,6 +393,17 @@ mod tests { md.contains(expected), "summary explanation text changed:\n{md}" ); + // The pin above would still pass if the paragraph hardcoded the words + // while a shared constant's value moved on; these tie the rendered text + // to the constants themselves. + assert!( + md.contains(MOCK_GATES_PR), + "summary must render MOCK_GATES_PR ({MOCK_GATES_PR:?}):\n{md}" + ); + assert!( + md.contains(OTHER_LANES_NON_BLOCKING), + "summary must render OTHER_LANES_NON_BLOCKING ({OTHER_LANES_NON_BLOCKING:?}):\n{md}" + ); } #[test] diff --git a/crates/e2e-report/src/consolidated/mod.rs b/crates/e2e-report/src/consolidated/mod.rs index 0ef063a7ff..911385029d 100644 --- a/crates/e2e-report/src/consolidated/mod.rs +++ b/crates/e2e-report/src/consolidated/mod.rs @@ -1764,12 +1764,14 @@ mod tests { fn renderers_take_the_pr_gating_phrases_from_the_shared_constants() { // What this proves, and what it does not. The rendered text is pinned // verbatim by `summary_explanation_paragraphs_are_unchanged` (markdown) - // and `legend_mock_and_other_platform_items_are_unchanged` (HTML); those - // catch any edit to the wording, but a renderer that re-typed the literal - // with identical words would still pass them and would silently stop - // following `MOCK_GATES_PR` / `OTHER_LANES_NON_BLOCKING`. This test - // closes that gap by reading each renderer's non-test source: it must - // name both constants and must not spell either phrase out itself. + // and `legend_mock_and_other_platform_items_are_unchanged` (HTML), and + // each of those also asserts that the rendered output contains + // `MOCK_GATES_PR` and `OTHER_LANES_NON_BLOCKING`; together they catch a + // renderer that hardcodes the words while a constant's value moves on. + // What they cannot catch is a renderer that re-types the CURRENT words + // with identical text, which renders the same bytes. This test closes + // that gap by reading each renderer's non-test source: it must not + // spell either phrase out itself. for (name, source) in [ ("markdown.rs", include_str!("markdown.rs")), ("html.rs", include_str!("html.rs")), @@ -1782,10 +1784,6 @@ mod tests { ("MOCK_GATES_PR", MOCK_GATES_PR), ("OTHER_LANES_NON_BLOCKING", OTHER_LANES_NON_BLOCKING), ] { - assert!( - production.contains(constant), - "{name} must build its gating sentence from `{constant}`" - ); assert!( !production.contains(phrase), "{name} re-types {phrase:?} instead of using `{constant}`" From 6fc5af8a5671be7be7c1ce832f1bda3ca5d9034e Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 14:22:04 +0000 Subject: [PATCH 019/102] test(e2e): judge the serve drain-wait by the pool the engine allocates from (#867) * test(e2e): judge the serve drain-wait by the pool the engine allocates from On an APU the VRAM total is only the BIOS carveout (512 MiB on the Strix Halo lane hosts) while the engine allocates from GTT (about 100 GiB), so the 90% floor taken from the carveout could never be met and every GPU serve waited out the full two-minute drain deadline. Choose GTT when the VRAM total is under the 8 GiB carveout limit and a non-empty GTT reading exists, as gpu-preflight.sh already does, and keep VRAM everywhere else. The choice is a pure function with unit tests built from the preflight log's real readings. Signed-off-by: Jussi Elo * test(e2e): bound the smi probes in the serve drain wait The drain wait ran amd-smi and rocm-smi with a blocking output() call, so a tool that stalled on a wedged driver blocked the test worker instead of letting the best-effort wait expire, and the first probe ran before the wait's deadline existed. Every probe in the wait now goes through bounded_stdout, which gives the call ten seconds, then kills and reaps the child and reports an unreadable sample (the wait falls back to the VRAM reading, as before). The deadline is created before the first probe, and the probes run under spawn_blocking so they do not occupy an async worker. Signed-off-by: Jussi Elo * test(e2e): tell an unreadable device probe from an absent one in the drain wait A GTT probe that stalled or failed on a carveout host used to look the same as "this host has no GTT pool", so the wait judged the small VRAM carveout and could report "drained" while the engine still held its GTT-backed memory. A probe now reports absent (tool not installed), unreadable (ran, no usable reading) or a reading. An unreadable GTT read makes the whole sample unreadable, and the "no GPU tooling" shortcut fires only when the tools are absent. The decision to read GTT at all moves next to the pool-selection rule, sharing one carveout predicate, so a test can reach it. The wait reuses its first sample instead of probing twice, the grandchild-holds-the-pipe limit of the bounded probe is documented, and the timeout test waits for the child's pid file instead of reading it once. Signed-off-by: Jussi Elo * test(e2e): make the drain wait's per-sample decision a tested function The choice between 'no GPU tooling', 'present but unreadable' and 'a reading to judge' lived inline in the async wait loop, where no unit test could reach it, so reverting an unreadable first probe to 'no tooling' would have stayed green. It is now judge() in device_memory.rs with tests, and the DeviceMemory docs state that an unreadable GTT read never reaches a sample. Signed-off-by: Jussi Elo * test(e2e): treat a zero VRAM total as an unreadable drain-wait sample A zero GTT total was already a failed read, but a parseable zero VRAM total was accepted as a pool. With no GTT tool on the host it was judged against a floor of zero, so the wait reported drained on its first sample. Both pools now share the rule, and the DeviceMemory docs say so. Signed-off-by: Jussi Elo * test(xtask): tie the serve drain carveout limit to the APU preflight floor CARVEOUT_MAX_MIB in the e2e harness is documented as the floor the APU lanes pass as GPU_PREFLIGHT_MIN_FREE_GIB, but nothing connected the two, so changing either left the other silently wrong with every unit test green. A workflow contract test now reads the constant as source text, with no new crate edge, and requires it to equal the smallest preflight floor any lane sets. Signed-off-by: Jussi Elo * test(e2e): name the test that keeps the carveout limit in step with the lanes The CARVEOUT_MAX_MIB doc says it is the APU lanes' preflight floor; it now also names the xtask contract test that fails when the two stop matching. Signed-off-by: Jussi Elo * test(e2e): bound the drain wait's probes, give up on a tool that never reads, and say what it saw Review of the serve drain wait raised five points, all addressed here. A host whose GPU tool is installed but never produces a reading waited the whole two-minute deadline on every serve. After three samples in a row with no reading, from a tool that has never read, the wait now ends and reports the device as unreadable; a tool that has read before is still waited on. One sample can make three probes of up to ten seconds each, so the last one could run well past the deadline while the line said 120s. Each probe is now held to the time left, the sleep is capped to it, and the line reports the elapsed time. A carveout-sized VRAM pool judged without GTT data (amd-smi present, rocm-smi absent) now says so on the timeout line, because that wait cannot follow the memory the engine uses. "NOT drained" is only printed for a reading below its floor. A sample the harness could not read is reported as "could not read device memory", with the last reading if there was one. The wait's decisions (`DrainWait`), the lines (`describe_verdict`, `DrainTarget::describe`) and the probe budget now live in `device_memory.rs` with unit tests that drive samples through the wait and assert the resulting line with it. Signed-off-by: Jussi Elo * test(e2e): show what a GPU tool printed when its output is not understood, and test the amd-smi parser A second review of the drain wait asked for two things. The `rocm-smi --showmeminfo gtt --csv` column names the Strix Halo fix relies on are only confirmed against the text output the preflight script parses. If they are wrong every sample on a carveout host is unreadable and the wait gives up after three samples with only a generic message, so the fix would silently do nothing. A probe whose output the parser rejects is now `Unparsed`, carrying ` printed ""` (the header row, printable ASCII only, cut to 120 characters). It reaches the final `device state:` line through `Judgement::Unparsed`, the wait's decision state and `Verdict::Unreadable`, is kept across later plain failures, and is dropped when a reading arrives. The amd-smi parser had been moved into a closure with no test. It is now `parse_amd_smi_vram` beside `parse_rocm_smi_csv_pool`, parses exactly as before, and has tests for both fields, a missing field, a zero total, non-numeric values, the first of two GPUs and the visible-VRAM lines it must not read. The probes now return a named `Pool` instead of a bare `(u64, u64)`, and `device_memory_mib`, a one-line delegate, is inlined into `probe_device_memory`. Signed-off-by: Jussi Elo * test(e2e): find the rocm-smi CSV header by its column names instead of taking the first line The drain wait read the first line of `rocm-smi --showmeminfo ... --csv` as the header. If the tool prints warning lines before the CSV, the first line names neither column, so every sample was unreadable and the wait could end as "could not read device memory" on a host whose device was fine. The header is now the first line that names both columns, the row after it is GPU 0, and a warning line that mentions only one of the columns is not taken for it. The comment on the GTT probe says which part of the column naming is known (the plain-text labels `gpu-preflight.sh` matches) and which is assumed (that the CSV headers use the same words). Signed-off-by: Jussi Elo * test(e2e): do not count a probe the deadline cut short, drop the unused parser, and document the floor and the probe bound A probe killed by the drain wait's own deadline gave no reading because the deadline stopped it, not because the tool stalled, yet it could replace a below-floor reading with "unreadable" in the final line. A sample that starts with less than a full probe timeout left and reads nothing is now ignored, so the wait ends on the last real reading; a truncated sample that does read still counts. `Probe::parse` was only used by its own test and is removed. The doc comments now say how much a 90% floor tolerates on the ~100 GiB GTT pool (about 10 GiB) and that the amd-smi bound is a behaviour change from the old unbounded call, softened by the rocm-smi fallback. Signed-off-by: Jussi Elo * test(e2e): give a probe that exited cleanly two seconds to hand over its output After a successful exit the probe's output is already in the pipe and the reader thread only has to be scheduled to pass it on. A 200 ms bound could expire on a loaded runner and turn a good sample into an unreadable one, and three of those in a row make the drain wait give up. The wait ends as soon as the bytes arrive, so a longer bound costs nothing in the normal case and matters only when the reader is late or a grandchild holds the pipe. The wait is now a function of the grace it is offered, so a test pins the bound without sleeping. Signed-off-by: Jussi Elo * docs(e2e): say why bounded_stdout repeats run_with_timeout instead of calling it rocm-core is only a dev-dependency of the harness crate, and bounded_stdout is library code, so it cannot call rocm_core::process::run_with_timeout. A normal dependency would be a new first-party edge for the crate-edge check to review. Record that where the dependency is declared and where the function is documented, so a change to either side's kill or drain behaviour looks at the other. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 88e6cc52667c9f89af39b2d67d393a3c9e4f50cc) Signed-off-by: juhovainio --- tests/e2e-cucumber/Cargo.toml | 5 + tests/e2e-cucumber/src/device_memory.rs | 1510 +++++++++++++++++ tests/e2e-cucumber/src/lib.rs | 1 + tests/e2e-cucumber/tests/e2e/serving_steps.rs | 186 +- xtask/src/workflow_contract.rs | 76 + 5 files changed, 1681 insertions(+), 97 deletions(-) create mode 100644 tests/e2e-cucumber/src/device_memory.rs diff --git a/tests/e2e-cucumber/Cargo.toml b/tests/e2e-cucumber/Cargo.toml index 8a46d60c7c..3efc130f5d 100644 --- a/tests/e2e-cucumber/Cargo.toml +++ b/tests/e2e-cucumber/Cargo.toml @@ -89,6 +89,11 @@ shlex.workspace = true # "inference verified" latch is the field the product reads), never the library # itself, and `xtask check-crate-edges` exempts dev edges by design — so keeping # it here needs no allowlist entry. +# +# For the same reason the library's `device_memory::bounded_stdout` cannot call +# `rocm_core::process::run_with_timeout`: it repeats that helper's spawn, poll, +# kill and reap steps on `std` alone, deliberately, and a normal dependency +# would be a new edge to review. A change to either one should look at the other. rocm-core = { path = "../../crates/rocm-core" } # `test-util` pauses tokio's clock in the retry loops' unit tests # (`#[tokio::test(start_paused = true)]`), so their timing assertions read a diff --git a/tests/e2e-cucumber/src/device_memory.rs b/tests/e2e-cucumber/src/device_memory.rs new file mode 100644 index 0000000000..fb3a513e6f --- /dev/null +++ b/tests/e2e-cucumber/src/device_memory.rs @@ -0,0 +1,1510 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Which device memory pool the serve drain-wait judges, and the floor it +//! demands of it. +//! +//! The wait between serve scenarios (`wait_for_free_vram` in the serving +//! steps) exists to see a just-killed engine hand its device memory back. It +//! has to look at the pool the engine allocates from. On a discrete card that +//! is VRAM. On an APU such as Strix Halo the VRAM total is only the fixed BIOS +//! carveout (512 MiB on the CI hosts) while the engine draws on GTT-backed +//! system memory (about 100 GiB there), so a floor taken from the carveout can +//! never be met and every GPU serve spent its whole drain deadline waiting. +//! `.github/scripts/gpu-preflight.sh` makes the same switch for the same reason. + +use std::io::Read; +use std::process::{Command, Stdio}; +use std::sync::mpsc; +use std::thread; +use std::time::{Duration, Instant}; + +/// Upper bound on the free-memory floor (MiB). +/// +/// Sized so the largest single +/// scenario model can allocate its engine's memory share without tripping its +/// startup check: Qwen3.6-27B plus vLLM's default fraction-of-total KV +/// reservation needs the MI300X mostly clear. Only a data-center GPU has this +/// much, so on smaller pools the floor is capped to 90% of the pool total (see +/// [`required_free_mib`]) — otherwise the check could never pass. Note the cap +/// no longer sits above what vLLM asks for: rocm-cli used to pin utilization +/// below it, and now defers to vLLM's own (higher) default, so the wait has +/// little headroom left and can time out on a device that is merely holding +/// display memory. The wait is best-effort and the serve proceeds regardless, +/// so this costs time rather than correctness; sizing a deliberate margin is +/// follow-up work. +pub const MAX_FREE_FLOOR_MIB: u64 = 150_000; + +/// A VRAM total below this is an APU's BIOS carveout, not private device memory. +/// +/// The engine's allocations come out of GTT instead. It is the +/// 8 GiB floor the Strix Halo preflight lanes pass as +/// `GPU_PREFLIGHT_MIN_FREE_GIB`, which `gpu-preflight.sh` compares the VRAM +/// total against for the same purpose. Every discrete card the lanes run on is +/// far above it (48 GiB and up). The xtask test +/// `the_serve_drain_carveout_limit_is_the_apu_lanes_preflight_floor` fails if +/// this stops matching the lanes' floor. +pub const CARVEOUT_MAX_MIB: u64 = 8 * 1024; + +/// Whether a VRAM total is an APU's carveout rather than private device memory. +/// +/// Both the decision to read GTT at all ([`sample_device_memory`]) and the +/// choice of pool ([`select_drain_target`]) go through this, so they cannot +/// disagree. +#[must_use] +pub const fn is_carveout(vram_total_mib: u64) -> bool { + vram_total_mib < CARVEOUT_MAX_MIB +} + +/// One attempt to read from a device tool. +/// +/// `Absent` and `Unreadable` are kept apart on purpose. With no tool installed +/// there is nothing to wait on (a mock or local run). A tool that is present +/// but stalled, failed, or printed something unparseable is a device that +/// could not be read right now, which must never be mistaken for either "no +/// tooling" or "drained". +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Probe { + /// The tool is not installed. + Absent, + /// The tool is there but gave no usable reading. + Unreadable, + /// The tool ran and exited successfully, but its output was not understood. + /// Like `Unreadable` for every decision, it also carries what the tool + /// printed (see [`output_hint`]) so a log can show what the parser met. + Unparsed(String), + Read(T), +} + +impl Probe { + /// Fall back to `next` when this probe produced no reading. + /// + /// A reading from either wins. Without one, the result is `Absent` only if + /// every tool tried was absent; if either tool ran and printed something not + /// understood the result keeps what it printed (`Unparsed`, the first tool's + /// if both); otherwise any tool that was present and gave nothing makes it + /// `Unreadable`. + #[must_use] + pub fn or_else(self, next: impl FnOnce() -> Self) -> Self { + if matches!(self, Self::Read(_)) { + return self; + } + match (self, next()) { + (_, read @ Self::Read(_)) => read, + (Self::Absent, Self::Absent) => Self::Absent, + (Self::Unparsed(hint), _) | (_, Self::Unparsed(hint)) => Self::Unparsed(hint), + _ => Self::Unreadable, + } + } +} + +impl Probe { + /// Parse a tool's stdout with `f`. Output `f` rejects is not `Unreadable` + /// but `Unparsed`, carrying the first line of it (see [`output_hint`]) so the + /// log says what the tool printed; `source` names the command. + #[must_use] + pub fn parse_text(self, source: &str, f: impl FnOnce(&str) -> Option) -> Probe { + match self { + Self::Absent => Probe::Absent, + Self::Unreadable => Probe::Unreadable, + Self::Unparsed(hint) => Probe::Unparsed(hint), + Self::Read(text) => { + f(&text).map_or_else(|| Probe::Unparsed(output_hint(source, &text)), Probe::Read) + } + } + } +} + +/// The longest first line of tool output [`output_hint`] repeats. +pub const HINT_MAX_CHARS: usize = 120; + +/// What a tool printed, as one bounded, printable line for a log. +/// +/// It is `source`, then the first non-blank line of `text` with anything outside +/// printable ASCII replaced by `?` and cut to [`HINT_MAX_CHARS`] characters. +/// Meant for the header row of a CSV a parser could not match, so the next run +/// shows the real column names. +#[must_use] +pub fn output_hint(source: &str, text: &str) -> String { + let Some(line) = text.lines().map(str::trim).find(|line| !line.is_empty()) else { + return format!("{source} printed no output"); + }; + let clean: String = line + .chars() + .map(|c| { + if c.is_ascii_graphic() || c == ' ' { + c + } else { + '?' + } + }) + .collect(); + let shown = if clean.chars().count() > HINT_MAX_CHARS { + format!( + "{}...", + clean.chars().take(HINT_MAX_CHARS).collect::() + ) + } else { + clean + }; + format!("{source} printed {shown:?}") +} + +/// One memory pool's size and free space, in MiB. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Pool { + pub total_mib: u64, + pub free_mib: u64, +} + +impl Pool { + #[must_use] + pub const fn new(total_mib: u64, free_mib: u64) -> Self { + Self { + total_mib, + free_mib, + } + } +} + +/// What the device tooling reported for GPU 0. +/// +/// `gtt` is `None` when GTT was not read: a discrete card, or a host with no +/// tool that reports it. A GTT read +/// that failed never gets here (the sample is `Unreadable`, see +/// [`sample_device_memory`]), so every pool here has a non-zero total, VRAM and +/// GTT alike. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct DeviceMemory { + pub vram: Pool, + pub gtt: Option, +} + +/// Which pool a [`DrainTarget`] was read from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PoolKind { + Vram, + Gtt, +} + +/// The pool to wait on, its current state, and the floor it must reach. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct DrainTarget { + pub kind: PoolKind, + pub total_mib: u64, + pub free_mib: u64, + pub floor_mib: u64, +} + +impl DrainTarget { + /// Whether the pool has handed back enough memory. + #[must_use] + pub const fn drained(&self) -> bool { + self.free_mib >= self.floor_mib + } + + /// `" MiB free of MiB[ GTT], floor MiB"`. VRAM is the + /// unlabelled default, so the line keeps the shape it always had on discrete + /// cards; only the GTT pool is named. + #[must_use] + pub fn describe(&self) -> String { + let pool = match self.kind { + PoolKind::Vram => "", + PoolKind::Gtt => " GTT", + }; + format!( + "{} MiB free of {} MiB{pool}, floor {} MiB", + self.free_mib, self.total_mib, self.floor_mib + ) + } +} + +/// The free-memory floor to wait for on a pool of `total_mib`: the +/// model-sized ceiling, but never more than 90% of the pool. +/// +/// The 90% cap is a tolerance, not a demand for an empty pool: on the CI Strix +/// Halo hosts' ~100 GiB GTT pool (102 400 MiB, floor 92 160 MiB) about 10 GiB +/// can still be held when the wait ends as drained. That is the accepted cost +/// of a floor that a pool with some memory permanently in use can still meet. +#[must_use] +pub fn required_free_mib(total_mib: u64) -> u64 { + MAX_FREE_FLOOR_MIB.min(total_mib / 100 * 90) +} + +/// What one sample means for the drain wait. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Judgement { + /// No device tool is installed: nothing to wait on (mock/local run). + NoTooling, + /// A tool is present but gave no usable reading. This is not "no tooling" + /// and not "drained": the wait keeps trying. + Unreadable, + /// As `Unreadable`, and the tool's output was not understood: what it + /// printed, for the report (see [`output_hint`]). + Unparsed(String), + /// A reading, and the pool and floor it is held to. + Target(DrainTarget), +} + +/// Turn a sample into what the wait does with it. +#[must_use] +pub fn judge(sample: Probe) -> Judgement { + match sample { + Probe::Absent => Judgement::NoTooling, + Probe::Unreadable => Judgement::Unreadable, + Probe::Unparsed(hint) => Judgement::Unparsed(hint), + Probe::Read(memory) => Judgement::Target(select_drain_target(memory)), + } +} + +/// Consecutive samples with no reading, from a tool that is installed and has +/// never once produced one, after which the wait stops. +/// +/// A tool that never reads is broken or wedged, and waiting out the whole +/// deadline on every serve would add that deadline to each of them. A tool that +/// has read before is not given up on: its next sample may well succeed. +pub const UNREAD_SAMPLES_BEFORE_GIVING_UP: u32 = 3; + +/// The least time worth starting a probe with. With less than this left before +/// the deadline the wait ends on what it has rather than spawn a probe that +/// would be killed before it could answer. +pub const MIN_PROBE_BUDGET: Duration = Duration::from_secs(1); + +/// How long one probe may run: [`SMI_PROBE_TIMEOUT`], but never past `deadline`. +/// +/// A sample makes up to three probes in a row, so without this the last sample +/// could run several probe timeouts beyond the wait's deadline. +#[must_use] +pub fn probe_budget(now: Instant, deadline: Instant) -> Duration { + SMI_PROBE_TIMEOUT.min(deadline.saturating_duration_since(now)) +} + +/// Whether a sample started at `now` has its probes held to less than a full +/// [`SMI_PROBE_TIMEOUT`] by `deadline` (see [`DrainWait::observe_truncated`]). +#[must_use] +pub fn sample_is_truncated(now: Instant, deadline: Instant) -> bool { + probe_budget(now, deadline) < SMI_PROBE_TIMEOUT +} + +/// How the drain wait ended. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Verdict { + /// No device tool is installed (mock/local run): nothing was waited on. + NoTooling, + /// The pool handed back enough memory. + Drained(DrainTarget), + /// The deadline passed with the pool still below its floor. + NotDrained(DrainTarget), + /// No usable reading at the end: `samples` samples in a row gave none, and + /// `last_reading` is the most recent one the wait did get, if any. `hint` is + /// what a tool printed that was not understood, when one did (the latest + /// such sample, kept until a reading arrives). This is not "not drained": + /// the harness did not observe that. + Unreadable { + last_reading: Option, + samples: u32, + hint: Option, + }, +} + +/// The drain wait's decision state, fed one [`Judgement`] per sample. +#[derive(Debug, Default)] +pub struct DrainWait { + samples: u32, + unread_in_a_row: u32, + last_reading: Option, + hint: Option, +} + +impl DrainWait { + /// Take the next sample. `Some` ends the wait; `None` means keep sampling. + pub fn observe(&mut self, judgement: Judgement) -> Option { + let first = self.samples == 0; + self.samples += 1; + match judgement { + Judgement::NoTooling if first => Some(Verdict::NoTooling), + Judgement::Target(target) => { + self.unread_in_a_row = 0; + self.last_reading = Some(target); + // A reading shows the tool's output is understood after all. + self.hint = None; + target.drained().then_some(Verdict::Drained(target)) + } + Judgement::Unparsed(hint) => { + self.hint = Some(hint); + self.unread() + } + // A tool that vanishes mid-wait is a sample nobody could read. + Judgement::NoTooling | Judgement::Unreadable => self.unread(), + } + } + + /// As [`observe`](Self::observe), for a sample taken when the deadline had + /// already cut its probes short (`truncated`, see [`probe_budget`]). + /// + /// A truncated sample that gave no reading says nothing about the tool: it + /// was stopped by the deadline, not stalled. Counting it would let a probe + /// killed in the last seconds replace a below-floor reading with + /// "unreadable" at the deadline. It is ignored, so the wait ends on the last + /// real reading. A truncated sample that does read counts as usual. + pub fn observe_truncated(&mut self, judgement: Judgement, truncated: bool) -> Option { + if truncated && matches!(judgement, Judgement::Unreadable | Judgement::Unparsed(_)) { + return None; + } + self.observe(judgement) + } + + fn unread(&mut self) -> Option { + self.unread_in_a_row += 1; + (self.last_reading.is_none() && self.unread_in_a_row >= UNREAD_SAMPLES_BEFORE_GIVING_UP) + .then(|| self.unreadable()) + } + + /// The verdict when the deadline passes: not drained if the last sample was + /// a reading, unreadable if it was not. + #[must_use] + pub fn timed_out(&self) -> Verdict { + match self.last_reading { + Some(target) if self.unread_in_a_row == 0 => Verdict::NotDrained(target), + _ => self.unreadable(), + } + } + + fn unreadable(&self) -> Verdict { + Verdict::Unreadable { + last_reading: self.last_reading, + samples: self.unread_in_a_row, + hint: self.hint.clone(), + } + } +} + +/// The one-line `device state:` report for a verdict, after `elapsed_secs` of waiting. +/// +/// It says only what the harness observed: "NOT drained" is for a reading below +/// its floor, never for a sample that gave none, and a carveout-sized VRAM pool +/// judged without GTT data says why that cannot show whether the engine's +/// memory came back. +#[must_use] +pub fn describe_verdict(verdict: Verdict, elapsed_secs: u64) -> String { + match verdict { + Verdict::NoTooling => "device state: no GPU tooling (mock/local run)".to_owned(), + Verdict::Drained(target) => format!("device state: drained ({})", target.describe()), + Verdict::NotDrained(target) => { + let hint = if target.kind == PoolKind::Vram && is_carveout(target.total_mib) { + "; only the BIOS carveout was read because no GTT reading was available (is \ + rocm-smi installed?), so this wait could not follow the memory the engine uses" + } else { + "" + }; + format!( + "device state: NOT drained after {elapsed_secs}s ({}) — a previous engine may \ + still be holding the GPU{hint}", + target.describe() + ) + } + Verdict::Unreadable { + last_reading: None, + samples, + hint, + } => format!( + "device state: could not read device memory ({samples} samples in {elapsed_secs}s; \ + the GPU tool is installed but gave no usable reading{})", + hint_suffix(hint.as_deref()) + ), + Verdict::Unreadable { + last_reading: Some(target), + hint, + .. + } => format!( + "device state: could not read device memory at the end of {elapsed_secs}s (last \ + reading: {}{})", + target.describe(), + hint_suffix(hint.as_deref()) + ), + } +} + +/// `"; "` for a report line, or nothing without one. +fn hint_suffix(hint: Option<&str>) -> String { + hint.map_or_else(String::new, |hint| format!("; {hint}")) +} + +/// Choose the pool the engine allocates from and the floor to hold it to. +/// +/// A VRAM total that [`is_carveout`] means the GTT pool is used, when one was +/// read. Without GTT data the answer is VRAM, which is what every host got +/// before this choice existed; [`sample_device_memory`] only leaves GTT out +/// when the host has no tool that reports it, never when a read failed. +#[must_use] +pub fn select_drain_target(memory: DeviceMemory) -> DrainTarget { + let (kind, pool) = match memory.gtt { + Some(gtt) if is_carveout(memory.vram.total_mib) => (PoolKind::Gtt, gtt), + _ => (PoolKind::Vram, memory.vram), + }; + DrainTarget { + kind, + total_mib: pool.total_mib, + free_mib: pool.free_mib, + floor_mib: required_free_mib(pool.total_mib), + } +} + +/// Read `(total, free)` in MiB from `rocm-smi --showmeminfo --csv` output. +/// +/// `total_column` and `used_column` are substrings of the two header +/// names (`Total Memory` and `Total Used Memory` for VRAM; the `GTT ` forms for +/// GTT); the first data row is the first GPU. Values are bytes in the output. +/// +/// The header is the first line that names both columns, not the first line: +/// the tool can print warning lines before the CSV, and taking one of those as +/// the header would make every sample unreadable. +#[must_use] +pub fn parse_rocm_smi_csv_pool(text: &str, total_column: &str, used_column: &str) -> Option { + let mut lines = text.lines(); + let (total_idx, used_idx) = lines.find_map(|line| { + let cols: Vec<&str> = line.split(',').collect(); + let total_idx = cols.iter().position(|c| c.contains(total_column))?; + let used_idx = cols.iter().position(|c| c.contains(used_column))?; + Some((total_idx, used_idx)) + })?; + let row = lines.next()?; + let vals: Vec<&str> = row.split(',').collect(); + let total: u64 = vals.get(total_idx)?.trim().parse().ok()?; + let used: u64 = vals.get(used_idx)?.trim().parse().ok()?; + let mib = 1024 * 1024; + Some(Pool::new(total / mib, total.saturating_sub(used) / mib)) +} + +/// Read VRAM from `amd-smi metric -m` output. +/// +/// The `TOTAL_VRAM:` and `FREE_VRAM:` lines look like `TOTAL_VRAM: 196592 MB`; +/// the number in front of the unit is taken as MiB. The first GPU's lines win. A +/// zero total parses here; it is [`sample_device_memory`] that treats it as a +/// failed read. +#[must_use] +pub fn parse_amd_smi_vram(text: &str) -> Option { + let field = |name: &str| { + text.lines() + .find_map(|line| line.trim().strip_prefix(name)) + .and_then(|value| value.split_whitespace().next()) + .and_then(|number| number.parse::().ok()) + }; + Some(Pool::new(field("TOTAL_VRAM:")?, field("FREE_VRAM:")?)) +} + +/// Assemble one device-memory sample from a VRAM probe and a GTT probe. +/// +/// `gtt` runs only when the VRAM total [`is_carveout`], so a discrete-card host +/// never makes the extra call. On a carveout host the GTT reading is what the +/// engine's memory comes out of, so: +/// +/// - a GTT reading makes the sample; +/// - a GTT probe that ran but gave nothing usable (a stall, a failure, output +/// not understood, or a zero total, which is a failed read rather than an +/// empty pool) makes the whole sample `Unreadable` (`Unparsed`, with what the +/// tool printed, when it ran but its output was not understood). Judging the tiny +/// carveout instead could report "drained" while the engine still holds its +/// GTT-backed memory. `gpu-preflight.sh` treats a zero GTT total the same way, +/// as an unreadable sample to retry; +/// - no tool that reports GTT at all (`Absent`) leaves `gtt` empty, and the +/// wait judges VRAM as it did before GTT was considered. +pub fn sample_device_memory( + vram: impl FnOnce() -> Probe, + gtt: impl FnOnce() -> Probe, +) -> Probe { + let vram = match vram() { + // A zero total is a failed read, not a pool: its floor would be zero and + // every sample would count as drained. + Probe::Read(pool) if pool.total_mib > 0 => pool, + Probe::Read(_) | Probe::Unreadable => return Probe::Unreadable, + Probe::Unparsed(hint) => return Probe::Unparsed(hint), + Probe::Absent => return Probe::Absent, + }; + if !is_carveout(vram.total_mib) { + return Probe::Read(DeviceMemory { vram, gtt: None }); + } + match gtt() { + Probe::Read(pool) if pool.total_mib > 0 => Probe::Read(DeviceMemory { + vram, + gtt: Some(pool), + }), + Probe::Read(_) | Probe::Unreadable => Probe::Unreadable, + Probe::Unparsed(hint) => Probe::Unparsed(hint), + Probe::Absent => Probe::Read(DeviceMemory { vram, gtt: None }), + } +} + +/// How long one `amd-smi` or `rocm-smi` call may run before it is an unreadable sample. +/// +/// These tools can stall on a wedged driver, and the drain wait is best-effort: +/// a stalled probe must cost one sample, not the worker. +/// +/// This is a behaviour change on discrete lanes: the previous probe ran +/// `amd-smi metric -m` with no timeout, so a slow call (an 8-GPU `metric` call +/// can be slow) simply took as long as it took, and a wedged one hung the +/// worker. Now `amd-smi` is cut off at this bound, or at the time left before +/// the deadline, and the `rocm-smi` fallback is asked next with its own bound, +/// so a sample costs at most two bounds and still gets a reading whenever +/// `rocm-smi` answers in time. It is not unbounded either way, and a tool that +/// never answers is reported as unreadable after a few samples. +pub const SMI_PROBE_TIMEOUT: Duration = Duration::from_secs(10); + +const SMI_POLL_INTERVAL: Duration = Duration::from_millis(25); + +/// How long to wait for the reader after the child has exited successfully. +/// +/// The output is already in the pipe by then and the reader only has to be +/// scheduled to hand it over, so the wait ends the moment it does and the bound +/// matters only when it cannot: a loaded runner that has not run the reader +/// thread yet, where a short bound would turn a good sample into an unreadable +/// one (and three of those in a row make the wait give up). A grandchild that +/// inherited the pipe can keep it open for good, so past this bound the reader +/// is abandoned rather than joined. The bound is generous because the cost of +/// that case, one grace per probe, is small next to [`SMI_PROBE_TIMEOUT`] and +/// the probes here do not fork helpers that outlive them. After a kill there is +/// no wait at all: a probe that outlived its deadline is unreadable at once. +const SMI_DRAIN_GRACE: Duration = Duration::from_secs(2); + +/// Run `command` and return its stdout if it exits successfully within `timeout`. +/// +/// A program that is not installed is [`Probe::Absent`]. Any other spawn +/// failure, a non-zero exit, or a timeout is [`Probe::Unreadable`]; on a +/// timeout the child is killed and reaped before this returns. +/// +/// Only the direct child is killed. A grandchild that inherited the stdout +/// pipe keeps the reader thread blocked in `read_to_end` until it exits, so a +/// timed-out probe can leak one thread. The probes here (`amd-smi`, +/// `rocm-smi`) do not fork helpers that outlive them, and the drain wait makes +/// a handful of calls, so this is accepted rather than guarded. +/// +/// Written against `std` alone, so it repeats the spawn, poll, kill and reap +/// steps of `rocm_core::process::run_with_timeout` on purpose: `rocm-core` is +/// only a dev-dependency of this crate (see its entry in `Cargo.toml`) and this +/// module is library code, and a normal dependency would be a new first-party +/// edge that `cargo xtask check-crate-edges` makes a reviewed addition. The two +/// can drift, so a change to either one's kill or drain behaviour should look at +/// the other. +pub fn bounded_stdout(mut command: Command, timeout: Duration) -> Probe { + let mut child = match command + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + { + Ok(child) => child, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Probe::Absent, + Err(_) => return Probe::Unreadable, + }; + let Some(mut pipe) = child.stdout.take() else { + let _ = child.kill(); + let _ = child.wait(); + return Probe::Unreadable; + }; + let (sender, receiver) = mpsc::channel(); + thread::spawn(move || { + let mut bytes = Vec::new(); + let _ = pipe.read_to_end(&mut bytes); + let _ = sender.send(bytes); + }); + + let deadline = Instant::now() + timeout; + let status = loop { + match child.try_wait() { + Ok(Some(status)) => break status, + Ok(None) if Instant::now() < deadline => thread::sleep(SMI_POLL_INTERVAL), + _ => { + let _ = child.kill(); + let _ = child.wait(); + return Probe::Unreadable; + } + } + }; + if !status.success() { + return Probe::Unreadable; + } + output_after_clean_exit(|grace| receiver.recv_timeout(grace).ok()) +} + +/// The reading from a probe that exited successfully, given a way to wait for +/// the reader's bytes for at most the grace it is passed. A function of the wait +/// so a test can pin the bound that is offered without sleeping. +fn output_after_clean_exit( + wait_for_reader: impl FnOnce(Duration) -> Option>, +) -> Probe { + wait_for_reader(SMI_DRAIN_GRACE).map_or(Probe::Unreadable, |bytes| { + Probe::Read(String::from_utf8_lossy(&bytes).into_owned()) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The pools `gpu-preflight.sh` printed on a Strix Halo lane host + /// (`VRAM 203321344/536870912, GTT 41787392/107374182400`, used/total in + /// bytes), and the same reading in MiB. + fn strix_halo() -> DeviceMemory { + DeviceMemory { + vram: Pool { + total_mib: 512, + free_mib: 318, + }, + gtt: Some(Pool { + total_mib: 102_400, + free_mib: 102_360, + }), + } + } + + fn mi300x() -> DeviceMemory { + DeviceMemory { + vram: Pool { + total_mib: 196_592, + free_mib: 196_309, + }, + gtt: Some(Pool { + total_mib: 1_000_000, + free_mib: 1_000_000, + }), + } + } + + #[test] + fn a_carveout_host_is_judged_by_gtt_and_a_healthy_strix_halo_counts_as_drained() { + let target = select_drain_target(strix_halo()); + assert_eq!(target.kind, PoolKind::Gtt); + assert_eq!(target.total_mib, 102_400); + assert_eq!(target.floor_mib, 92_160); + assert!( + target.drained(), + "an idle Strix Halo host must not wait out the drain deadline: {target:?}" + ); + } + + #[test] + fn a_carveout_host_whose_gtt_is_mostly_held_is_not_drained() { + let mut memory = strix_halo(); + memory.gtt = Some(Pool { + total_mib: 102_400, + free_mib: 60_000, + }); + let target = select_drain_target(memory); + assert_eq!(target.kind, PoolKind::Gtt); + assert!(!target.drained(), "{target:?}"); + } + + #[test] + fn a_discrete_card_is_still_judged_by_vram_with_the_model_sized_cap() { + let target = select_drain_target(mi300x()); + assert_eq!(target.kind, PoolKind::Vram); + assert_eq!(target.total_mib, 196_592); + assert_eq!(target.floor_mib, MAX_FREE_FLOOR_MIB); + assert!(target.drained()); + } + + #[test] + fn a_discrete_card_holding_most_of_its_vram_is_not_drained() { + let mut memory = mi300x(); + memory.vram.free_mib = 40_000; + assert!(!select_drain_target(memory).drained()); + } + + #[test] + fn a_host_without_gtt_data_keeps_the_vram_behaviour() { + let mut memory = strix_halo(); + memory.gtt = None; + let target = select_drain_target(memory); + assert_eq!(target.kind, PoolKind::Vram); + assert_eq!( + target.floor_mib, 450, + "the floor the CI log showed for the carveout" + ); + assert!(!target.drained()); + } + + #[test] + fn a_vram_total_at_the_carveout_limit_is_private_memory() { + let mut memory = strix_halo(); + memory.vram = Pool { + total_mib: CARVEOUT_MAX_MIB, + free_mib: CARVEOUT_MAX_MIB, + }; + assert_eq!(select_drain_target(memory).kind, PoolKind::Vram); + assert!(is_carveout(CARVEOUT_MAX_MIB - 1)); + assert!(!is_carveout(CARVEOUT_MAX_MIB)); + } + + /// Run `sample_device_memory` with canned probes, reporting whether the GTT + /// probe was consulted. + fn sample(vram: Probe, gtt: Probe) -> (Probe, bool) { + let mut gtt_asked = false; + let result = sample_device_memory( + || vram, + || { + gtt_asked = true; + gtt + }, + ); + (result, gtt_asked) + } + + #[test] + fn a_carveout_host_reads_gtt_and_the_sample_carries_it() { + let (result, gtt_asked) = sample( + Probe::Read(Pool::new(512, 318)), + Probe::Read(Pool::new(102_400, 102_360)), + ); + assert!(gtt_asked, "a carveout host must have its GTT read"); + assert_eq!(result, Probe::Read(strix_halo())); + } + + #[test] + fn a_discrete_host_never_makes_the_gtt_call() { + let (result, gtt_asked) = sample( + Probe::Read(Pool::new(196_592, 196_309)), + Probe::Read(Pool::new(1_000_000, 1_000_000)), + ); + assert!( + !gtt_asked, + "GTT must not be read when VRAM is private memory" + ); + let Probe::Read(memory) = result else { + panic!("a readable VRAM probe makes a sample: {result:?}"); + }; + assert_eq!(memory.gtt, None); + assert_eq!(select_drain_target(memory).kind, PoolKind::Vram); + } + + #[test] + fn a_carveout_host_whose_gtt_probe_failed_is_unreadable_not_judged_by_vram() { + // VRAM alone would read as "drained" here (free is above its floor), so + // falling back to it would hide a GTT-backed allocation still held. + let (result, gtt_asked) = sample(Probe::Read(Pool::new(512, 500)), Probe::Unreadable); + assert!(gtt_asked); + assert_eq!(result, Probe::Unreadable); + } + + #[test] + fn a_zero_gtt_total_is_a_failed_reading_not_a_pool() { + let (result, _) = sample( + Probe::Read(Pool::new(512, 500)), + Probe::Read(Pool::new(0, 0)), + ); + assert_eq!(result, Probe::Unreadable); + } + + #[test] + fn a_zero_vram_total_is_a_failed_reading_not_a_drained_host() { + // With no GTT tool the sample would otherwise be judged by a VRAM pool + // whose floor is zero, which is drained by definition. + let (result, gtt_asked) = sample(Probe::Read(Pool::new(0, 0)), Probe::Absent); + assert_eq!(result, Probe::Unreadable); + assert!(!gtt_asked); + } + + #[test] + fn a_carveout_host_with_no_gtt_tool_keeps_the_vram_behaviour() { + let (result, gtt_asked) = sample(Probe::Read(Pool::new(512, 318)), Probe::Absent); + assert!(gtt_asked); + let Probe::Read(memory) = result else { + panic!("a host without a GTT tool still has a VRAM sample: {result:?}"); + }; + assert_eq!(memory.gtt, None); + assert_eq!(select_drain_target(memory).kind, PoolKind::Vram); + } + + #[test] + fn a_vram_probe_that_is_absent_or_unreadable_is_passed_through_without_asking_for_gtt() { + assert_eq!( + sample(Probe::Absent, Probe::Read(Pool::new(102_400, 102_360))), + (Probe::Absent, false) + ); + assert_eq!( + sample(Probe::Unreadable, Probe::Read(Pool::new(102_400, 102_360))), + (Probe::Unreadable, false) + ); + } + + #[test] + fn an_absent_tool_is_no_tooling_but_an_unreadable_one_is_not() { + assert_eq!(judge(Probe::Absent), Judgement::NoTooling); + assert_eq!(judge(Probe::Unreadable), Judgement::Unreadable); + } + + #[test] + fn a_reading_is_judged_against_the_pool_the_engine_allocates_from() { + assert_eq!( + judge(Probe::Read(strix_halo())), + Judgement::Target(select_drain_target(strix_halo())) + ); + } + + #[test] + fn a_fallback_tool_supplies_the_reading_the_first_could_not() { + assert_eq!(Probe::Unreadable.or_else(|| Probe::Read(7)), Probe::Read(7)); + assert_eq!(Probe::Absent.or_else(|| Probe::Read(7)), Probe::Read(7)); + assert_eq!( + Probe::Read(1).or_else(|| panic!("the fallback must not run after a reading")), + Probe::Read(1) + ); + } + + #[test] + fn a_fallback_is_absent_only_when_every_tool_was_absent() { + assert_eq!(Probe::::Absent.or_else(|| Probe::Absent), Probe::Absent); + assert_eq!( + Probe::::Absent.or_else(|| Probe::Unreadable), + Probe::Unreadable + ); + assert_eq!( + Probe::::Unreadable.or_else(|| Probe::Absent), + Probe::Unreadable + ); + } + + #[test] + fn rocm_smi_csv_gtt_is_read_in_mib() { + let text = "device,GTT Total Memory (B),GTT Total Used Memory (B)\n\ + card0,107374182400,41787392\n"; + assert_eq!( + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory"), + Some(Pool::new(102_400, 102_360)) + ); + } + + #[test] + fn rocm_smi_csv_vram_keeps_its_original_column_match() { + let text = "device,VRAM Total Memory (B),VRAM Total Used Memory (B)\n\ + card0,536870912,203321344\n"; + assert_eq!( + parse_rocm_smi_csv_pool(text, "Total Memory", "Total Used Memory"), + Some(Pool::new(512, 318)) + ); + } + + #[test] + fn rocm_smi_csv_without_the_pool_is_none() { + let text = "device,VRAM Total Memory (B),VRAM Total Used Memory (B)\n\ + card0,536870912,203321344\n"; + assert_eq!( + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory"), + None + ); + assert_eq!( + parse_rocm_smi_csv_pool("", "Total Memory", "Total Used Memory"), + None + ); + } + + #[test] + fn rocm_smi_csv_with_warning_lines_before_the_header_is_still_read() { + let text = "WARNING: a device is in a low-power state\n\ + a warning that only says GTT Total Memory\n\ + \n\ + device,GTT Total Memory (B),GTT Total Used Memory (B)\n\ + card0,107374182400,41787392\n\ + card1,1,1\n"; + assert_eq!( + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory"), + Some(Pool::new(102_400, 102_360)), + "the header is the first line naming both columns, and the first row after it is GPU 0" + ); + } + + #[test] + fn rocm_smi_csv_with_a_header_but_no_data_row_is_none() { + let text = "WARNING: nothing to report\n\ + device,GTT Total Memory (B),GTT Total Used Memory (B)\n"; + assert_eq!( + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory"), + None + ); + } + + /// A Strix Halo host whose GTT pool has only `free_mib` MiB free of 102 400. + fn strix_holding(free_mib: u64) -> DeviceMemory { + DeviceMemory { + gtt: Some(Pool { + total_mib: 102_400, + free_mib, + }), + ..strix_halo() + } + } + + /// A sample whose GTT pool is nearly all held: below its 92 160 MiB floor. + fn undrained() -> Judgement { + Judgement::Target(select_drain_target(strix_holding(1_000))) + } + + /// The healthy Strix Halo reading: above its floor. + fn drained() -> Judgement { + Judgement::Target(select_drain_target(strix_halo())) + } + + #[test] + fn a_target_is_described_with_its_pool_and_only_gtt_is_named() { + assert_eq!( + select_drain_target(strix_halo()).describe(), + "102360 MiB free of 102400 MiB GTT, floor 92160 MiB" + ); + assert_eq!( + select_drain_target(mi300x()).describe(), + "196309 MiB free of 196592 MiB, floor 150000 MiB" + ); + } + + #[test] + fn a_drained_sample_ends_the_wait_and_the_line_names_the_pool_it_read() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(undrained()), None); + let verdict = wait.observe(drained()).expect("a drained sample ends it"); + assert_eq!(verdict, Verdict::Drained(select_drain_target(strix_halo()))); + assert_eq!( + describe_verdict(verdict, 6), + "device state: drained (102360 MiB free of 102400 MiB GTT, floor 92160 MiB)" + ); + } + + #[test] + fn no_tooling_ends_the_wait_on_the_first_sample_only() { + assert_eq!( + DrainWait::default().observe(Judgement::NoTooling), + Some(Verdict::NoTooling) + ); + assert_eq!( + describe_verdict(Verdict::NoTooling, 0), + "device state: no GPU tooling (mock/local run)" + ); + // A tool that disappears mid-wait is a sample nobody could read, not a + // reason to report a mock run. + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(undrained()), None); + for _ in 0..UNREAD_SAMPLES_BEFORE_GIVING_UP * 2 { + assert_eq!(wait.observe(Judgement::NoTooling), None); + } + } + + #[test] + fn a_tool_that_never_reads_is_given_up_on_and_called_unreadable_not_undrained() { + let mut wait = DrainWait::default(); + for _ in 1..UNREAD_SAMPLES_BEFORE_GIVING_UP { + assert_eq!(wait.observe(Judgement::Unreadable), None); + } + let verdict = wait + .observe(Judgement::Unreadable) + .expect("the last allowed unread sample ends the wait"); + assert_eq!( + verdict, + Verdict::Unreadable { + last_reading: None, + samples: UNREAD_SAMPLES_BEFORE_GIVING_UP, + hint: None, + } + ); + let line = describe_verdict(verdict, 24); + assert_eq!( + line, + "device state: could not read device memory (3 samples in 24s; the GPU tool is \ + installed but gave no usable reading)" + ); + assert!(!line.contains("NOT drained"), "{line}"); + } + + #[test] + fn a_probe_the_deadline_cut_short_does_not_replace_a_below_floor_reading() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe_truncated(undrained(), false), None); + // The last sample's probe was killed by the deadline and gave nothing. + assert_eq!( + wait.observe_truncated(Judgement::Unreadable, true), + None, + "a truncated sample with no reading is not counted" + ); + assert_eq!( + wait.observe_truncated(Judgement::Unparsed("rocm-smi: killed".to_owned()), true), + None + ); + assert_eq!( + wait.timed_out(), + Verdict::NotDrained(select_drain_target(strix_holding(1_000))), + "the wait ends on the last real reading, not on the probe the deadline stopped" + ); + } + + #[test] + fn an_unread_sample_that_was_not_cut_short_still_counts_and_a_truncated_reading_still_reads() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe_truncated(undrained(), false), None); + assert_eq!(wait.observe_truncated(Judgement::Unreadable, false), None); + assert!( + matches!(wait.timed_out(), Verdict::Unreadable { samples: 1, .. }), + "a stalled probe with the full budget is a real miss" + ); + let mut wait = DrainWait::default(); + let verdict = wait.observe_truncated(drained(), true); + assert!( + matches!(verdict, Some(Verdict::Drained(_))), + "a truncated sample that read is judged like any other: {verdict:?}" + ); + } + + #[test] + fn a_tool_that_has_read_is_waited_on_through_unreadable_samples() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(Judgement::Unreadable), None); + assert_eq!(wait.observe(Judgement::Unreadable), None); + assert_eq!(wait.observe(undrained()), None); + for _ in 0..UNREAD_SAMPLES_BEFORE_GIVING_UP * 3 { + assert_eq!(wait.observe(Judgement::Unreadable), None); + } + let verdict = wait.timed_out(); + assert_eq!( + verdict, + Verdict::Unreadable { + last_reading: Some(select_drain_target(strix_holding(1_000))), + samples: UNREAD_SAMPLES_BEFORE_GIVING_UP * 3, + hint: None, + } + ); + assert_eq!( + describe_verdict(verdict, 120), + "device state: could not read device memory at the end of 120s (last reading: \ + 1000 MiB free of 102400 MiB GTT, floor 92160 MiB)" + ); + } + + #[test] + fn the_deadline_after_a_reading_is_not_drained_and_after_a_miss_it_is_not() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(undrained()), None); + let target = select_drain_target(strix_holding(1_000)); + assert_eq!(wait.timed_out(), Verdict::NotDrained(target)); + assert_eq!( + describe_verdict(wait.timed_out(), 120), + "device state: NOT drained after 120s (1000 MiB free of 102400 MiB GTT, floor \ + 92160 MiB) — a previous engine may still be holding the GPU" + ); + // The most recent sample was one nobody could read: that is not a claim + // about the device. + assert_eq!(wait.observe(Judgement::Unreadable), None); + assert_eq!( + wait.timed_out(), + Verdict::Unreadable { + last_reading: Some(target), + samples: 1, + hint: None, + } + ); + } + + #[test] + fn an_undrained_carveout_judged_without_gtt_says_it_could_not_follow_the_engines_memory() { + // The real assembly path: amd-smi read the carveout, no tool reports GTT. + let (memory, _) = sample(Probe::Read(Pool::new(512, 318)), Probe::Absent); + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(judge(memory)), None); + let line = describe_verdict(wait.timed_out(), 120); + assert!( + line.starts_with( + "device state: NOT drained after 120s (318 MiB free of 512 MiB, floor 450 MiB)" + ), + "{line}" + ); + assert!( + line.contains( + "only the BIOS carveout was read because no GTT reading was available (is \ + rocm-smi installed?)" + ), + "{line}" + ); + // No such note when the pool judged is the one the engine uses. + for held in [ + Verdict::NotDrained(select_drain_target(strix_holding(1_000))), + Verdict::NotDrained(select_drain_target(DeviceMemory { + vram: Pool { + total_mib: 196_592, + free_mib: 1_000, + }, + gtt: None, + })), + ] { + let line = describe_verdict(held, 120); + assert!(!line.contains("carveout"), "{line}"); + } + } + + /// What `amd-smi metric -m` prints, per GPU, with the lines the parser reads + /// among others it must not confuse with them. + const AMD_SMI_GPU0: &str = "GPU: 0\n MEM_USAGE:\n TOTAL_VRAM: 196592 MB\n \ + USED_VRAM: 283 MB\n FREE_VRAM: 196309 MB\n \ + TOTAL_VISIBLE_VRAM: 196592 MB\n FREE_VISIBLE_VRAM: 196309 MB\n"; + + #[test] + fn amd_smi_vram_reads_the_total_and_free_lines_in_mib() { + assert_eq!( + parse_amd_smi_vram(AMD_SMI_GPU0), + Some(Pool::new(196_592, 196_309)) + ); + } + + #[test] + fn amd_smi_vram_takes_the_first_gpu_and_ignores_indentation_and_units() { + let two = format!("{AMD_SMI_GPU0}GPU: 1\n TOTAL_VRAM: 1 MB\n FREE_VRAM: 2 MB\n"); + assert_eq!(parse_amd_smi_vram(&two), Some(Pool::new(196_592, 196_309))); + // Tabs, wide gaps, any unit word and a missing space after the colon all + // read the same way: the number after the name is the value. + assert_eq!( + parse_amd_smi_vram("\tTOTAL_VRAM: 512 MiB\n FREE_VRAM:318\n"), + Some(Pool::new(512, 318)) + ); + assert_eq!( + parse_amd_smi_vram("TOTAL_VRAM: 512\r\nFREE_VRAM: 318 MB\r\n"), + Some(Pool::new(512, 318)) + ); + } + + #[test] + fn amd_smi_vram_needs_both_fields_as_numbers() { + assert_eq!(parse_amd_smi_vram("TOTAL_VRAM: 512 MB\n"), None); + assert_eq!(parse_amd_smi_vram("FREE_VRAM: 318 MB\n"), None); + assert_eq!( + parse_amd_smi_vram("TOTAL_VRAM: N/A\nFREE_VRAM: 318 MB\n"), + None + ); + assert_eq!(parse_amd_smi_vram("no such fields here"), None); + assert_eq!(parse_amd_smi_vram(""), None); + // The visible-VRAM lines are different fields, not the ones asked for. + assert_eq!( + parse_amd_smi_vram("TOTAL_VISIBLE_VRAM: 512 MB\nFREE_VISIBLE_VRAM: 318 MB\n"), + None + ); + } + + #[test] + fn a_zero_amd_smi_total_parses_and_is_then_rejected_as_a_failed_read() { + let zero = "TOTAL_VRAM: 0 MB\nFREE_VRAM: 0 MB\n"; + let parsed = parse_amd_smi_vram(zero).expect("parses"); + assert_eq!(parsed, Pool::new(0, 0)); + let (result, gtt_asked) = sample(Probe::Read(parsed), Probe::Absent); + assert_eq!(result, Probe::Unreadable); + assert!(!gtt_asked); + } + + #[test] + fn output_hint_is_the_first_nonblank_line_bounded_and_printable() { + assert_eq!( + output_hint( + "rocm-smi gtt", + "\n \ndevice,VRAM Total Memory (B)\ncard0,1\n" + ), + "rocm-smi gtt printed \"device,VRAM Total Memory (B)\"" + ); + assert_eq!(output_hint("tool", ""), "tool printed no output"); + assert_eq!(output_hint("tool", " \n\t\n"), "tool printed no output"); + // Non-ASCII and control characters are replaced, not repeated. + assert_eq!( + output_hint("tool", "GTT \u{1b}[31mT\u{f6}tal\u{0}"), + "tool printed \"GTT ?[31mT?tal?\"" + ); + // Quotes and backslashes are escaped by the quoting. + assert_eq!( + output_hint("tool", "a\"b\\c"), + "tool printed \"a\\\"b\\\\c\"" + ); + let long = "x".repeat(HINT_MAX_CHARS + 50); + let hint = output_hint("tool", &long); + assert_eq!( + hint, + format!("tool printed \"{}...\"", "x".repeat(HINT_MAX_CHARS)) + ); + let exact = "y".repeat(HINT_MAX_CHARS); + assert_eq!( + output_hint("tool", &exact), + format!("tool printed \"{exact}\"") + ); + } + + #[test] + fn output_a_parser_rejects_is_unparsed_with_what_it_printed_not_plain_unreadable() { + let header = "device,GTT Total (B),GTT Used (B)\ncard0,107374182400,41787392\n"; + let probe = Probe::Read(header.to_owned()).parse_text("rocm-smi gtt", |text| { + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory") + }); + assert_eq!( + probe, + Probe::Unparsed( + "rocm-smi gtt printed \"device,GTT Total (B),GTT Used (B)\"".to_owned() + ) + ); + // Right headers still read, and a tool that is absent or failed is not + // given a hint it has not earned. + let good = "device,GTT Total Memory (B),GTT Total Used Memory (B)\ncard0,1048576,0\n"; + assert_eq!( + Probe::Read(good.to_owned()).parse_text("x", |t| { + parse_rocm_smi_csv_pool(t, "GTT Total Memory", "GTT Total Used Memory") + }), + Probe::Read(Pool::new(1, 1)) + ); + assert_eq!( + Probe::::Absent.parse_text("x", |_| Some(0_u8)), + Probe::Absent + ); + assert_eq!( + Probe::::Unreadable.parse_text("x", |_| Some(0_u8)), + Probe::Unreadable + ); + } + + #[test] + fn a_fallback_tool_that_printed_something_unknown_keeps_what_it_printed() { + let unparsed = || Probe::::Unparsed("amd-smi printed \"GPU: 0\"".to_owned()); + assert_eq!( + unparsed().or_else(|| Probe::Absent), + unparsed(), + "an absent fallback leaves the first tool's output visible" + ); + assert_eq!(Probe::Absent.or_else(unparsed), unparsed()); + assert_eq!(Probe::Unreadable.or_else(unparsed), unparsed()); + assert_eq!(unparsed().or_else(|| Probe::Unreadable), unparsed()); + assert_eq!(unparsed().or_else(|| Probe::Read(7)), Probe::Read(7)); + // The first tool's output wins when both printed something. + assert_eq!( + unparsed().or_else(|| Probe::Unparsed("rocm-smi printed \"x\"".to_owned())), + unparsed() + ); + } + + #[test] + fn a_carveout_host_whose_gtt_output_was_not_understood_carries_what_it_printed() { + let hint = "rocm-smi gtt printed \"device,GTT Total (B)\""; + let (result, gtt_asked) = sample( + Probe::Read(Pool::new(512, 318)), + Probe::Unparsed(hint.to_owned()), + ); + assert!(gtt_asked); + assert_eq!(result, Probe::Unparsed(hint.to_owned())); + // And a VRAM probe's unparsed output passes through without asking for GTT. + let (result, gtt_asked) = sample(Probe::Unparsed(hint.to_owned()), Probe::Absent); + assert_eq!(result, Probe::Unparsed(hint.to_owned())); + assert!(!gtt_asked); + assert_eq!(judge(result), Judgement::Unparsed(hint.to_owned())); + } + + #[test] + fn what_a_tool_printed_reaches_the_final_line_when_it_never_reads() { + let hint = "rocm-smi --showmeminfo gtt --csv printed \"device,GTT Total (B)\""; + let mut wait = DrainWait::default(); + for _ in 1..UNREAD_SAMPLES_BEFORE_GIVING_UP { + assert_eq!(wait.observe(Judgement::Unparsed(hint.to_owned())), None); + } + let verdict = wait + .observe(Judgement::Unparsed(hint.to_owned())) + .expect("the third unread sample ends the wait"); + assert_eq!( + describe_verdict(verdict, 9), + "device state: could not read device memory (3 samples in 9s; the GPU tool is \ + installed but gave no usable reading; rocm-smi --showmeminfo gtt --csv printed \ + \"device,GTT Total (B)\")" + ); + } + + #[test] + fn what_a_tool_printed_is_kept_through_later_plain_failures_and_dropped_by_a_reading() { + let hint = "rocm-smi printed \"hdr\""; + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(Judgement::Unparsed(hint.to_owned())), None); + // A later timeout does not erase the evidence of the earlier sample. + assert_eq!(wait.observe(Judgement::Unreadable), None); + let Verdict::Unreadable { hint: kept, .. } = wait.timed_out() else { + panic!("two unread samples are unreadable at the deadline"); + }; + assert_eq!(kept.as_deref(), Some(hint)); + // A reading shows the output is understood after all. + assert_eq!(wait.observe(undrained()), None); + assert_eq!(wait.observe(Judgement::Unreadable), None); + let Verdict::Unreadable { + hint: after, + last_reading, + .. + } = wait.timed_out() + else { + panic!("the last sample was unreadable"); + }; + assert_eq!(after, None); + assert!(last_reading.is_some()); + } + + #[test] + fn a_hint_follows_the_last_reading_on_the_line_when_the_deadline_finds_it_unreadable() { + let mut wait = DrainWait::default(); + assert_eq!(wait.observe(undrained()), None); + assert_eq!( + wait.observe(Judgement::Unparsed("t printed \"h\"".to_owned())), + None + ); + assert_eq!( + describe_verdict(wait.timed_out(), 120), + "device state: could not read device memory at the end of 120s (last reading: \ + 1000 MiB free of 102400 MiB GTT, floor 92160 MiB; t printed \"h\")" + ); + } + + #[test] + fn a_probe_gets_the_probe_timeout_or_the_time_left_whichever_is_less() { + let now = Instant::now(); + assert_eq!( + probe_budget(now, now + Duration::from_secs(100)), + SMI_PROBE_TIMEOUT + ); + assert_eq!( + probe_budget(now, now + Duration::from_secs(4)), + Duration::from_secs(4) + ); + assert_eq!( + probe_budget(now + Duration::from_secs(5), now), + Duration::ZERO + ); + } + + #[test] + fn a_sample_is_truncated_only_when_less_than_a_full_probe_timeout_is_left() { + let now = Instant::now(); + assert!(!sample_is_truncated(now, now + Duration::from_secs(100))); + assert!( + !sample_is_truncated(now, now + SMI_PROBE_TIMEOUT), + "exactly a full probe timeout left is not truncated" + ); + assert!(sample_is_truncated( + now, + now + SMI_PROBE_TIMEOUT.saturating_sub(Duration::from_millis(1)) + )); + assert!( + sample_is_truncated(now + Duration::from_secs(5), now), + "past the deadline is truncated" + ); + } + + /// A probe that exited successfully has its output in the pipe, and a loaded + /// runner can take longer than the old 200 ms to run the reader thread that + /// hands it over. The wait it is offered must outlast that, the bytes it + /// returns are the reading, and only a reader that never answers (a + /// grandchild holding the pipe) is unreadable. + #[test] + fn a_clean_exit_waits_for_a_slow_reader_and_only_a_silent_one_is_unreadable() { + let mut offered = None; + let out = output_after_clean_exit(|grace| { + offered = Some(grace); + Some(b"MEM 1 2\n".to_vec()) + }); + assert_eq!(out, Probe::Read("MEM 1 2\n".to_owned())); + let offered = offered.expect("the wait was asked for"); + assert!( + offered >= Duration::from_secs(1), + "a reader on a loaded runner was offered only {offered:?}" + ); + assert!( + offered <= SMI_PROBE_TIMEOUT, + "the grace for one probe is {offered:?}, longer than the probe itself" + ); + assert_eq!(output_after_clean_exit(|_| None), Probe::Unreadable); + } + + #[cfg(unix)] + fn shell(script: &str, args: &[&str]) -> Command { + let mut command = Command::new("sh"); + command.arg("-c").arg(script).arg("sh").args(args); + command + } + + #[cfg(unix)] + #[test] + fn a_probe_that_succeeds_returns_its_stdout() { + let out = bounded_stdout(shell("echo hello", &[]), Duration::from_secs(30)); + assert_eq!(out, Probe::Read("hello\n".to_owned())); + } + + #[cfg(unix)] + #[test] + fn a_probe_that_exits_non_zero_is_unreadable() { + let out = bounded_stdout(shell("echo hello; exit 3", &[]), Duration::from_secs(30)); + assert_eq!(out, Probe::Unreadable); + } + + #[test] + fn a_probe_whose_program_is_not_installed_is_absent() { + let out = bounded_stdout( + Command::new("rocm-cli-test-no-such-smi-tool"), + Duration::from_secs(30), + ); + assert_eq!(out, Probe::Absent); + } + + /// A tool that never returns must cost one sample, not the worker, and the + /// child must not be left running or unreaped. A zombie still answers + /// `kill -0`, so a failing `kill -0` shows the child was reaped, not merely + /// signalled. + #[cfg(unix)] + #[test] + fn a_probe_that_outlives_its_deadline_is_killed_reaped_and_unreadable() { + let dir = tempfile::tempdir().expect("tempdir"); + let pid_file = dir.path().join("pid"); + let started = Instant::now(); + let out = bounded_stdout( + shell( + "echo $$ > \"$1\"; exec sleep 60", + &[pid_file.to_str().expect("utf-8 path")], + ), + Duration::from_secs(3), + ); + let elapsed = started.elapsed(); + assert_eq!( + out, + Probe::Unreadable, + "a timed-out probe is an unreadable sample" + ); + assert!( + elapsed < Duration::from_secs(30), + "returned after {elapsed:?}, far past the 3s deadline" + ); + // The shell creates the file and then writes the pid; give a loaded + // runner a moment to finish the write rather than read it half-done. + let mut pid = String::new(); + for _ in 0..100 { + pid = std::fs::read_to_string(&pid_file) + .unwrap_or_default() + .trim() + .to_owned(); + if !pid.is_empty() { + break; + } + thread::sleep(Duration::from_millis(50)); + } + assert!( + !pid.is_empty(), + "the child wrote its pid before the deadline" + ); + let still_there = Command::new("kill") + .args(["-0", &pid]) + .stderr(Stdio::null()) + .status() + .expect("kill runs") + .success(); + assert!(!still_there, "child {pid} was left running or unreaped"); + } + + /// Windows runs only the deadline half; the reaped-child check above needs + /// `kill -0`. + #[cfg(windows)] + #[test] + fn a_probe_that_outlives_its_deadline_returns_unreadable_in_time() { + let mut command = Command::new("ping.exe"); + command.args(["-n", "60", "127.0.0.1"]); + let started = Instant::now(); + let out = bounded_stdout(command, Duration::from_millis(1500)); + assert_eq!(out, Probe::Unreadable); + assert!(started.elapsed() < Duration::from_secs(20)); + } +} diff --git a/tests/e2e-cucumber/src/lib.rs b/tests/e2e-cucumber/src/lib.rs index 520ceb1574..33cc23da50 100644 --- a/tests/e2e-cucumber/src/lib.rs +++ b/tests/e2e-cucumber/src/lib.rs @@ -4,6 +4,7 @@ pub mod capability; pub mod chat_request; +pub mod device_memory; pub mod expectation; pub mod fixture_binary; pub mod harness_diagnostics; diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index d3e236b973..3f5182e41d 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -11,6 +11,11 @@ use cucumber::{given, then, when}; use crate::E2eWorld; use crate::e2e::tui_driver::{TuiSession, default_timeout}; +use e2e_cucumber::device_memory::{ + DeviceMemory, DrainWait, MIN_PROBE_BUDGET, Pool, Probe, bounded_stdout, describe_verdict, + judge, parse_amd_smi_vram, parse_rocm_smi_csv_pool, probe_budget, sample_device_memory, + sample_is_truncated, +}; use e2e_cucumber::mock_server::{MockServer, ServiceRecordOptions, write_service_record_with}; use e2e_cucumber::serve_log::{ ServeAttempt, archive_service_log, engine_from_serve_args, reported_engine, @@ -299,124 +304,111 @@ async fn ensure_serve_port_free() -> String { wait_for_free_vram().await } -/// Upper bound on the free-VRAM floor (MiB). Sized so the largest single -/// scenario model can allocate its engine's memory share without tripping its -/// startup check: Qwen3.6-27B plus vLLM's default fraction-of-total KV -/// reservation needs the MI300X mostly clear. Only a data-center GPU has this -/// much, so on smaller cards (e.g. Strix Halo's smaller unified-memory pool) the -/// floor is capped to 90% of the device total (see [`required_free_vram_mib`]) — -/// otherwise the check could never pass. Note the cap no longer sits above what -/// vLLM asks for: rocm-cli used to pin utilization below it, and now defers to -/// vLLM's own (higher) default, so the wait has little headroom left and can -/// time out on a device that is merely holding display memory. The wait is -/// best-effort and the serve proceeds regardless, so this costs time rather -/// than correctness; sizing a deliberate margin is follow-up work. -const MAX_FREE_VRAM_FLOOR_MIB: u64 = 150_000; - -/// The free-VRAM floor to wait for on this host: the model-sized ceiling, but -/// never more than 90% of the device's total VRAM. A hardcoded 150 GiB floor is -/// unreachable on a small card (e.g. Strix Halo's ~48 GiB unified VRAM), so -/// `wait_for_free_vram` would burn its full deadline on every serve scenario; -/// scaling to the device keeps the drain-check meaningful everywhere. -fn required_free_vram_mib(total_mib: u64) -> u64 { - MAX_FREE_VRAM_FLOOR_MIB.min(total_mib / 100 * 90) -} - /// How long to wait for a stopped engine to hand its device memory back. const VRAM_DRAIN_DEADLINE: Duration = Duration::from_mins(2); -/// Best-effort: wait until the GPU reports enough free VRAM (see -/// [`required_free_vram_mib`]), so a just-killed serve's memory is actually -/// reclaimed before the next serve starts. Queries `amd-smi` then `rocm-smi`; -/// if neither is present (mock/local, no ROCm), returns immediately so non-GPU -/// runs are unaffected. +/// Pause between two samples of the device. +const DRAIN_POLL_INTERVAL: Duration = Duration::from_secs(3); + +/// Best-effort: wait until the pool the engine allocates from reports enough +/// free memory (see [`select_drain_target`]: VRAM on a discrete card, GTT on an +/// APU whose VRAM is only a BIOS carveout), so a just-killed serve's memory is +/// actually reclaimed before the next serve starts. Queries `amd-smi` then +/// `rocm-smi`; if neither is present (mock/local, no ROCm), returns immediately +/// so non-GPU runs are unaffected. /// /// The wait is bounded and best-effort: on timeout the serve still starts, /// because a stale reading must not turn a slow drain into a hard failure. The -/// returned line records which of the two happened — an undrained device is the -/// single most likely reason the serve that follows never becomes ready, and -/// without it the failure looks identical to a genuinely broken serve. +/// returned line records what happened — an undrained device is the single most +/// likely reason the serve that follows never becomes ready, and without it the +/// failure looks identical to a genuinely broken serve. A tool that is installed +/// but never produces a reading is given up on after a few samples +/// (`UNREAD_SAMPLES_BEFORE_GIVING_UP`) and reported as unreadable, not as +/// undrained; the decisions and the wording live in `DrainWait` and +/// `describe_verdict`, where they are tested. async fn wait_for_free_vram() -> String { - // No GPU tooling → nothing to wait on (mock/local). Probe once up front. - let Some(total) = total_vram_mib() else { - return "device state: no GPU tooling (mock/local run)".to_owned(); - }; - let floor = required_free_vram_mib(total); - let deadline = Instant::now() + VRAM_DRAIN_DEADLINE; + // Started before the first probe so that probe counts against the wait. + let started = Instant::now(); + let deadline = started + VRAM_DRAIN_DEADLINE; + let mut wait = DrainWait::default(); loop { - let free = free_vram_mib(); - if let Some(free) = free - && free >= floor - { - return format!( - "device state: drained ({free} MiB free of {total} MiB, floor {floor} MiB)" - ); + // Every probe is held to the time left, so the wait ends at its deadline + // rather than a few probe timeouts after it. A sample started with less + // than a full probe timeout left is `truncated`: if it gives no reading + // that is the deadline's doing, not the tool's, and is not counted. + let truncated = sample_is_truncated(Instant::now(), deadline); + let sample = judge(probe_device_memory(deadline).await); + if let Some(verdict) = wait.observe_truncated(sample, truncated) { + return describe_verdict(verdict, started.elapsed().as_secs()); } - if Instant::now() >= deadline { - let free = free.map_or_else(|| "unreadable".to_owned(), |mib| format!("{mib} MiB")); - return format!( - "device state: NOT drained after {}s ({free} free of {total} MiB, floor \ - {floor} MiB) — a previous engine is still holding the GPU", - VRAM_DRAIN_DEADLINE.as_secs() - ); + let left = deadline.saturating_duration_since(Instant::now()); + if left < MIN_PROBE_BUDGET { + return describe_verdict(wait.timed_out(), started.elapsed().as_secs()); + } + tokio::time::sleep(left.min(DRAIN_POLL_INTERVAL)).await; + if deadline.saturating_duration_since(Instant::now()) < MIN_PROBE_BUDGET { + return describe_verdict(wait.timed_out(), started.elapsed().as_secs()); } - tokio::time::sleep(Duration::from_secs(3)).await; } } -/// Free device VRAM in MiB for GPU 0, via `amd-smi` (then `rocm-smi`). `None` -/// when no such tool exists or its output can't be parsed. -fn free_vram_mib() -> Option { - vram_mib().map(|(_total, free)| free) +/// Device memory for GPU 0 off the async worker: VRAM via `amd-smi` (then +/// `rocm-smi`), plus the GTT pool from `rocm-smi` where the host needs it (see +/// [`sample_device_memory`] for when GTT is read and what a failed read means). +/// `amd-smi` is not asked for GTT: its output has no field this harness has been +/// checked against, whereas the preflight step already reads `rocm-smi`'s GTT +/// rows on these hosts. It runs subprocesses, each bounded by `probe_budget` (at +/// most the probe timeout, never past `deadline`), so the blocking thread it +/// occupies is released. +async fn probe_device_memory(deadline: Instant) -> Probe { + tokio::task::spawn_blocking(move || { + sample_device_memory(|| vram_pool(deadline), || gtt_pool(deadline)) + }) + .await + .unwrap_or(Probe::Unreadable) } -/// Total device VRAM in MiB for GPU 0. `None` when no GPU tool is available — -/// used as the "is there a GPU to wait on at all?" probe in mock/local runs. -fn total_vram_mib() -> Option { - vram_mib().map(|(total, _free)| total) +/// GTT for GPU 0, via `rocm-smi`. The command is named in the report when its +/// output is not understood, so the first run on a host shows the CSV header the +/// column names were meant to match. +/// +/// What is and is not checked: `gpu-preflight.sh` matches the labels +/// `GTT Total Memory` and `GTT Total Used Memory` in `rocm-smi`'s plain-text +/// `--showmeminfo vram gtt` output, so those labels are known to exist there. +/// That `--showmeminfo gtt --csv` uses the same words as its column headers is +/// not checked against real output; it is assumed by analogy with the VRAM CSV. +/// Warning lines before the header are skipped (see [`parse_rocm_smi_csv_pool`]). +fn gtt_pool(deadline: Instant) -> Probe { + use std::process::Command; + let mut command = Command::new("rocm-smi"); + command.args(["--showmeminfo", "gtt", "--csv"]); + bounded_stdout(command, probe_budget(Instant::now(), deadline)) + .parse_text("rocm-smi --showmeminfo gtt --csv", |text| { + parse_rocm_smi_csv_pool(text, "GTT Total Memory", "GTT Total Used Memory") + }) } -/// `(total, free)` device VRAM in MiB for GPU 0, via `amd-smi` (then `rocm-smi`). -/// `None` when no such tool exists or its output can't be parsed. -fn vram_mib() -> Option<(u64, u64)> { +/// Device VRAM for GPU 0, via `amd-smi` (then `rocm-smi`). +fn vram_pool(deadline: Instant) -> Probe { use std::process::Command; // amd-smi: lines like " TOTAL_VRAM: 196592 MB" / "FREE_VRAM: 196309 MB". - if let Ok(out) = Command::new("amd-smi").args(["metric", "-m"]).output() - && out.status.success() - { - let text = String::from_utf8_lossy(&out.stdout); - let field = |name: &str| { - text.lines() - .find_map(|l| l.trim().strip_prefix(name)) - .and_then(|v| v.split_whitespace().next()) - .and_then(|n| n.parse::().ok()) - }; - if let (Some(total), Some(free)) = (field("TOTAL_VRAM:"), field("FREE_VRAM:")) { - return Some((total, free)); - } - } + let amd_smi = || { + let mut command = Command::new("amd-smi"); + command.args(["metric", "-m"]); + bounded_stdout(command, probe_budget(Instant::now(), deadline)) + .parse_text("amd-smi metric -m", parse_amd_smi_vram) + }; // rocm-smi fallback: `--showmeminfo vram --csv` → vram total/used per card. - if let Ok(out) = Command::new("rocm-smi") - .args(["--showmeminfo", "vram", "--csv"]) - .output() - && out.status.success() - { - let text = String::from_utf8_lossy(&out.stdout); + let rocm_smi = || { + let mut command = Command::new("rocm-smi"); + command.args(["--showmeminfo", "vram", "--csv"]); // CSV columns include "VRAM Total Memory (B)" and "VRAM Total Used Memory (B)". - // Parse the first data row's total and used to derive free. - let mut lines = text.lines(); - let header = lines.next()?; - let cols: Vec<&str> = header.split(',').collect(); - let total_idx = cols.iter().position(|c| c.contains("Total Memory"))?; - let used_idx = cols.iter().position(|c| c.contains("Total Used Memory"))?; - let row = lines.next()?; - let vals: Vec<&str> = row.split(',').collect(); - let total: u64 = vals.get(total_idx)?.trim().parse().ok()?; - let used: u64 = vals.get(used_idx)?.trim().parse().ok()?; - let mib = 1024 * 1024; - return Some((total / mib, total.saturating_sub(used) / mib)); - } - None + bounded_stdout(command, probe_budget(Instant::now(), deadline)) + .parse_text("rocm-smi --showmeminfo vram --csv", |text| { + parse_rocm_smi_csv_pool(text, "Total Memory", "Total Used Memory") + }) + }; + amd_smi().or_else(rocm_smi) } /// Kill whatever process is listening on `port`. Best-effort and diff --git a/xtask/src/workflow_contract.rs b/xtask/src/workflow_contract.rs index 484846c346..f3d8ce9a96 100644 --- a/xtask/src/workflow_contract.rs +++ b/xtask/src/workflow_contract.rs @@ -2695,6 +2695,82 @@ jobs: body.contains("GPU_PREFLIGHT_MIN_FREE_GIB:-8") || body.contains("else { 8 }") } + /// `CARVEOUT_MAX_MIB` in MiB, read from the harness source as text. + /// + /// Read as text rather than imported: `xtask` has no edge to `e2e-cucumber` + /// and this check is not a reason to add one. Only the two spellings the + /// constant can reasonably take are understood, ` * 1024` and a bare + /// number, and anything else fails loudly instead of guessing. + fn carveout_max_mib(source: &str) -> u64 { + let line = source + .lines() + .find(|l| { + l.trim_start() + .starts_with("pub const CARVEOUT_MAX_MIB: u64 =") + }) + .expect("device_memory.rs declares `pub const CARVEOUT_MAX_MIB: u64 = ...;`"); + let expr = line + .split_once('=') + .and_then(|(_, rhs)| rhs.split_once(';')) + .map(|(expr, _)| expr.trim().replace('_', "")) + .expect("the constant's value ends in a semicolon"); + let number = |s: &str| { + s.trim() + .parse::() + .unwrap_or_else(|_| panic!("cannot read {s:?} in `{expr}` as a number")) + }; + match expr.split_once('*') { + Some((a, b)) => number(a) * number(b), + None => number(&expr), + } + } + + #[test] + fn carveout_max_mib_reads_both_spellings() { + let spelled = |v: &str| format!("/// doc\npub const CARVEOUT_MAX_MIB: u64 = {v};\n"); + assert_eq!(carveout_max_mib(&spelled("8 * 1024")), 8192); + assert_eq!(carveout_max_mib(&spelled("8192")), 8192); + assert_eq!(carveout_max_mib(&spelled("8_192")), 8192); + } + + #[test] + fn the_serve_drain_carveout_limit_is_the_apu_lanes_preflight_floor() { + // The serve drain wait (`tests/e2e-cucumber/src/device_memory.rs`) treats + // a VRAM total under `CARVEOUT_MAX_MIB` as an APU's BIOS carveout and + // judges GTT instead. Its doc says that limit is the floor the APU lanes + // pass as `GPU_PREFLIGHT_MIN_FREE_GIB`, and nothing tied the two together: + // moving either one left the other quietly wrong, with every unit test + // green. A raised limit would send a small discrete card to GTT; a lowered + // floor with the limit left behind would judge a real carveout by VRAM. + // + // The APU floor is the smallest one any lane sets (the discrete lanes set + // a larger one), so this follows the lanes rather than a copy of 8. + let mut floors_gib = Vec::new(); + for (_, text) in self_hosted_workflows() { + for step in gpu_preflight_steps(&text) { + if let Some(floor) = preflight_floor(&step) { + floors_gib.push(floor.parse::().unwrap_or_else(|_| { + panic!("a preflight floor of {floor:?} is not a whole number of GiB") + })); + } + } + } + let apu_floor_gib = *floors_gib + .iter() + .min() + .expect("the self-hosted lanes set a preflight floor"); + let source = + std::fs::read_to_string(repo_root().join("tests/e2e-cucumber/src/device_memory.rs")) + .expect("reading tests/e2e-cucumber/src/device_memory.rs"); + assert_eq!( + carveout_max_mib(&source), + apu_floor_gib * 1024, + "CARVEOUT_MAX_MIB no longer equals the APU lanes' GPU_PREFLIGHT_MIN_FREE_GIB \ + ({apu_floor_gib} GiB); change them together, or the serve drain wait picks \ + the wrong memory pool" + ); + } + #[test] fn every_apu_preflight_block_measures_the_gtt_pool() { // A gfx1151 APU reports its BIOS carveout as VRAM total — 512 MiB on the From e01d7bd2d57e330d30690609551599c7c48d4326 Mon Sep 17 00:00:00 2001 From: Juho Date: Mon, 9 Nov 2026 14:35:40 +0000 Subject: [PATCH 020/102] feat(serve): expose an Anthropic-compatible /v1/messages endpoint (ROCMAI-597) (#883) * feat(serve): expose an Anthropic-compatible /v1/messages endpoint (ROCMAI-597) A served model now gets an Anthropic Messages endpoint beside its OpenAI one. The process that owns the engine waits for it to list its model, asks whether the engine already serves /v1/messages, and either publishes the engine's own endpoint or binds a translation front on a port of its own. The result goes in the engine state file, the service record adopts it, and `rocm serve` and `rocm services list` show it (marked "not responding" for a ready service whose endpoint does not answer). The translation covers text turns, system prompts, max_tokens, sampling parameters and stop sequences. Streaming, tool use and non-text content blocks are refused with an Anthropic invalid_request_error instead of being dropped. Whatever model name the client sends, the served model answers. Signed-off-by: juhovainio * fix(serve): address review feedback on the Anthropic messages endpoint - relay /v1/models upstream statuses (401 stays 401) via a status-returning GET helper - advertise the bind host, not the probe loopback, for a native endpoint - add expectations rows for the three serve-anthropic scenarios - unique per-test temp dirs (parallel tests shared a state file) - refuse tool_choice/thinking, require string text blocks, 504 on timeout, read FastAPI-style detail errors, fall back to Authorization on empty x-api-key - say text-only/non-streaming in README and serve --help; hint when the row is not published yet - drop the summary dump from the CodeQL-flagged assertions Signed-off-by: juhovainio * docs(architecture): add the anthropic_endpoint module to the module map Signed-off-by: juhovainio * fix(serve): always front the engine with the Anthropic proxy Engines that serve /v1/messages themselves (vLLM, Lemonade) were published as-is, but they reject any model name other than the one they serve, so a request naming a Claude model got "model not found" (serve-51 on the MI300X and Windows Strix Halo lanes). The proxy answers under any model name, so it now fronts every engine; the native probe and mode are removed. Signed-off-by: juhovainio * test(e2e): give the Anthropic messages request room for a reasoning model serve-51 sent max_tokens=32; Qwen3 spent the whole budget on reasoning and the reply held an empty text block with stop_reason max_tokens. Signed-off-by: juhovainio --------- Signed-off-by: juhovainio (cherry picked from commit 4b1edf2d6b5f2eb98f1fe530ee9f3156b67b4f13) Signed-off-by: juhovainio --- README.md | 21 +- apps/rocm/src/cli.rs | 4 +- apps/rocm/src/engine_ipc.rs | 10 + apps/rocm/src/providers.rs | 1 + apps/rocm/src/serve_cmd.rs | 19 + apps/rocm/src/serve_summary.rs | 38 +- apps/rocm/src/services_text.rs | 62 + apps/rocmd/src/anthropic_proxy.rs | 1139 +++++++++++++++++ apps/rocmd/src/lib.rs | 1 + crates/rocm-core/src/anthropic_endpoint.rs | 123 ++ crates/rocm-core/src/lib.rs | 7 +- crates/rocm-core/src/net.rs | 28 +- crates/rocm-core/src/service_record.rs | 57 +- docs/architecture.md | 2 + docs/manual-testing.md | 1 + docs/testing.md | 23 + tests/e2e-cucumber/expectations.toml | 23 + .../features/model_serving.feature | 29 + tests/e2e-cucumber/tests/e2e.rs | 4 + tests/e2e-cucumber/tests/e2e/serving_steps.rs | 139 ++ 20 files changed, 1722 insertions(+), 9 deletions(-) create mode 100644 apps/rocmd/src/anthropic_proxy.rs create mode 100644 crates/rocm-core/src/anthropic_endpoint.rs diff --git a/README.md b/README.md index 5a22e810ef..a83a1b2fc5 100644 --- a/README.md +++ b/README.md @@ -390,7 +390,7 @@ form works depends on the engine your GPU selects. | `rocm fix []` | Apply a fix reported by `rocm diagnose` | | `rocm install sdk` | Install TheRock ROCm wheels into a managed Python environment | | `rocm install driver` | Install the AMD kernel driver on Linux; on Windows, report the display driver version and age (installs nothing) | -| `rocm serve ` | Start a local OpenAI-compatible model server | +| `rocm serve ` | Start a local OpenAI-compatible model server (plus a text-only Anthropic `/v1/messages` endpoint) | | `rocm remote serve ` | Serve a model on another GPU machine over your private network | | `rocm dash` | Open the full-screen telemetry dashboard | | `rocm bench load --endpoint ` | Load-test a local OpenAI-compatible endpoint | @@ -1080,6 +1080,25 @@ rocm serve [--engine lemonade|vllm|aim] [--engine-env KEY=VALUE]... ``` +Alongside the OpenAI-compatible endpoint, a served model gets an +Anthropic-compatible `/v1/messages` endpoint, shown as `anthropic endpoint:` in +`rocm services list` and in the interactive `rocm serve` summary (a fresh launch +waits briefly for it; if it is not there yet, check `rocm services list`). It is +a text-only, non-streaming endpoint. It is a base URL: point an +Anthropic client at it (`ANTHROPIC_BASE_URL=`) and send your endpoint key, +if the service has one, as `x-api-key` (the translation front also accepts a +bearer token). `rocm` runs a +translation front on its own port, inside the serving process, so it starts and +stops with the service; it fronts the engine even when the engine has its own +`/v1/messages` route, because that route only accepts the served model's name. Whatever model name the client sends, the served model +answers, and the reply names it. The translation covers text turns, system +prompts, `max_tokens`, `temperature`, `top_p`, `top_k` and `stop_sequences`; +streaming, tool use (`tools`, `tool_choice`), extended thinking and image or +other non-text content blocks are refused with +an Anthropic `invalid_request_error` rather than ignored. `rocm services list` +marks the endpoint `(not responding)` when a ready service's endpoint does not +answer. + `--engine aim` runs the model in a Docker container instead of a host process; it requires a reachable Docker daemon and is Linux/WSL2 only. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the 5-minute readiness wait for a diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 7cc1ea8986..7cc3bd6cd1 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -423,7 +423,9 @@ rocm model --verbose" }, /// Start a local OpenAI-compatible model server. /// - /// Picks an engine and ROCm runtime automatically unless overridden. By default the + /// Picks an engine and ROCm runtime automatically unless overridden. A text-only, + /// non-streaming Anthropic `/v1/messages` endpoint is published beside the OpenAI + /// one (see `rocm services list`). By default the /// server runs as a managed background service and prints a deployment summary /// (status, endpoint, model, and smoke-test metrics); use --verbose to stream engine /// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the diff --git a/apps/rocm/src/engine_ipc.rs b/apps/rocm/src/engine_ipc.rs index 3e9565d00e..d1f4829f9d 100644 --- a/apps/rocm/src/engine_ipc.rs +++ b/apps/rocm/src/engine_ipc.rs @@ -362,6 +362,16 @@ pub(crate) fn run_builtin_engine_serve_http( engine_recipe: Option, ) -> Result<()> { let parsed_policy = parse_device_policy(Some(device_policy))?; + // Every engine is fronted the same way, so the Anthropic endpoint is started + // here once rather than inside each adapter. It runs on a thread of this + // process, which is the service's supervisor: it ends when the service does. + rocmd::anthropic_proxy::spawn_alongside_engine(rocmd::anthropic_proxy::AnthropicProxyConfig { + host: host.clone(), + engine_port: port, + served_model: model_ref.clone(), + state_path: state_path.clone(), + probe_api_key: rocm_engine_protocol::resolve_endpoint_api_key(), + }); match engine { "lemonade" => rocm_engine_lemonade::builtin_serve_http( service_id, diff --git a/apps/rocm/src/providers.rs b/apps/rocm/src/providers.rs index b40c3209c2..d35d738698 100644 --- a/apps/rocm/src/providers.rs +++ b/apps/rocm/src/providers.rs @@ -1918,6 +1918,7 @@ mod tests { requested_model: "Qwen/Qwen3.5".to_owned(), api_model: "Qwen/Qwen3.5".to_owned(), chat_endpoint: "http://127.0.0.1:8000/v1/chat/completions".to_owned(), + anthropic_endpoint: None, service_id: "svc-ready".to_owned(), status: "ready".to_owned(), already_running: false, diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 7147245f2c..0ffd69f010 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -1250,6 +1250,24 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { engine_serves_vllm, ); notes.extend(engine_env_notes.iter().cloned()); + // Published by the serving process shortly after the engine lists its + // model. A service that was already running published it long ago, so + // only a fresh launch is waited for. + let anthropic_endpoint = rocm_core::wait_for_published_anthropic_endpoint( + &paths.service_engine_state_path(&selected_engine, &report.service_id), + if report.already_running || report.status != "ready" { + std::time::Duration::ZERO + } else { + std::time::Duration::from_secs(5) + }, + ); + if anthropic_endpoint.is_none() && !report.already_running && report.status == "ready" { + notes.push( + "the Anthropic-compatible endpoint was not published yet; it appears as \ + `anthropic endpoint:` in `rocm services list` once the engine lists its model" + .to_owned(), + ); + } // A managed serve that failed on VRAM lands here not-ready with the OOM // traceback only in its own log. Read that log tail so the summary can // name the memory knobs, since the pre-launch low-VRAM warning cannot @@ -1282,6 +1300,7 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { requested_model: model, api_model: resolve.served_model_name().to_owned(), chat_endpoint: format!("{}/chat/completions", report.endpoint_url), + anthropic_endpoint, service_id: report.service_id.clone(), status: report.status.clone(), already_running: report.already_running, diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index 73db2a4b0d..cd0ab34bc6 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -53,6 +53,9 @@ pub(crate) struct DeploymentSummary { pub api_model: String, /// Full chat-completions endpoint, e.g. `http://127.0.0.1:1337/v1/chat/completions`. pub chat_endpoint: String, + /// Base URL of the Anthropic-compatible endpoint (clients append + /// `/v1/messages`), when the service has published one. + pub anthropic_endpoint: Option, pub service_id: String, /// `"ready"`, `"starting"`, or an existing service's status. pub status: String, @@ -122,14 +125,20 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ] }; + let anthropic_rows: Vec<(&str, String)> = summary + .anthropic_endpoint + .iter() + .map(|url| ("anthropic endpoint", url.clone())) + .collect(); let rows: Vec<(&str, String)> = [ vec![ ("status", summary.status.clone()), ("engine", summary.engine.clone()), ], model_rows, + vec![("endpoint", summary.chat_endpoint.clone())], + anthropic_rows, vec![ - ("endpoint", summary.chat_endpoint.clone()), ("time to first token", format_ttft(summary.metrics.ttft)), ("throughput (approx)", format_tps(summary.metrics.gen_tps)), ("service", summary.service_id.clone()), @@ -449,6 +458,7 @@ mod tests { requested_model: "Qwen/Qwen2.5-7B-Instruct".to_owned(), api_model: "Qwen/Qwen2.5-7B-Instruct".to_owned(), chat_endpoint: "http://127.0.0.1:1337/v1/chat/completions".to_owned(), + anthropic_endpoint: None, service_id: "vllm-qwen-1720000000".to_owned(), status: "ready".to_owned(), already_running: false, @@ -476,6 +486,32 @@ mod tests { assert_eq!(format_tps(Some(f64::INFINITY)), "n/a"); } + #[test] + fn summary_lists_the_anthropic_endpoint_beside_the_openai_one_when_published() { + let mut summary = base_summary(); + summary.anthropic_endpoint = Some("http://127.0.0.1:41000".to_owned()); + let rendered = render_summary(&summary); + let endpoint = rendered + .find("http://127.0.0.1:1337/v1/chat/completions") + .unwrap(); + let anthropic = rendered.lines().any(|line| { + line.trim_start().starts_with("anthropic endpoint") + && line.ends_with("http://127.0.0.1:41000") + }); + assert!( + anthropic, + "no `anthropic endpoint` row ending in the published URL" + ); + let anthropic = rendered.find("anthropic endpoint").unwrap(); + assert!( + endpoint < anthropic, + "the anthropic row must follow the endpoint row" + ); + // Nothing published (the service is not ready, or the wait ran out): + // no row rather than a made-up one. + assert!(!render_summary(&base_summary()).contains("anthropic")); + } + #[test] fn summary_shows_endpoint_model_and_metrics() { let rendered = render_summary(&base_summary()); diff --git a/apps/rocm/src/services_text.rs b/apps/rocm/src/services_text.rs index f1bee48f29..c281e6a474 100644 --- a/apps/rocm/src/services_text.rs +++ b/apps/rocm/src/services_text.rs @@ -360,6 +360,31 @@ fn write_past_attempts_hint(output: &mut String, past_attempts: usize, newest_id let _ = writeln!(output, " Reclaim the space: rocm services prune"); } +/// The `anthropic endpoint:` row of a service listing. +/// +/// Only a service that reports `ready` is probed: the endpoint is published when +/// the engine first lists its model and is gone with the process, so for any +/// other status the answer would say nothing the status does not. The probe is +/// the same cheap `/v1/models` listing the OpenAI endpoint is judged by. +fn anthropic_endpoint_line(paths: &AppPaths, record: &ManagedServiceRecord, url: &str) -> String { + let responding = record.status != "ready" + || rocm_core::anthropic_endpoint_ready( + url, + Some(record.served_model_name()), + crate::endpoint_keys::endpoint_api_key(paths, &record.service_id).as_deref(), + Duration::from_millis(750), + ); + format_anthropic_endpoint_line(url, responding) +} + +fn format_anthropic_endpoint_line(url: &str, responding: bool) -> String { + if responding { + format!(" anthropic endpoint: {url}") + } else { + format!(" anthropic endpoint: {url} (not responding)") + } +} + pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result { // Counts describe the WHOLE registry, not the rows this view keeps. The // default view hides records that are not live, and counting the filtered @@ -416,6 +441,13 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result Result<()> { + let (_guard, paths) = test_paths("services-anthropic-row"); + let mut record = crate::tests::plant_service_record(&paths, "svc-anthropic", "stopped", 0)?; + record.anthropic_endpoint_url = Some("http://127.0.0.1:1".to_owned()); + record.anthropic_endpoint_mode = Some("proxy".to_owned()); + record.write()?; + + let rendered = render_services_text(&paths, true)?; + + assert!( + rendered.contains("\n anthropic endpoint: http://127.0.0.1:1\n"), + "{rendered}" + ); + Ok(()) + } + #[test] fn render_services_text_lists_live_services_by_default_and_all_on_request() -> Result<()> { use std::io::{Read, Write}; diff --git a/apps/rocmd/src/anthropic_proxy.rs b/apps/rocmd/src/anthropic_proxy.rs new file mode 100644 index 0000000000..3e3c2e0d9b --- /dev/null +++ b/apps/rocmd/src/anthropic_proxy.rs @@ -0,0 +1,1139 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Anthropic Messages front for a served model. +//! +//! The engines speak the OpenAI API only. This module gives a managed service an +//! Anthropic-compatible `POST /v1/messages` by translating each request to the +//! engine's `/v1/chat/completions` and the reply back. It runs inside the process +//! that already owns the engine (`rocm`'s hidden `__engine-serve-http` entry), +//! as a background thread, so it starts, stops and restarts with the service and +//! adds no process of its own to track. +//! +//! [`spawn_alongside_engine`] first waits for the engine to list its model, then +//! binds a port of its own and publishes that address in the engine state file +//! under the keys in `rocm_core::anthropic_endpoint`, which the service record +//! adopts. The proxy fronts the engine even when the engine has a `/v1/messages` +//! route of its own (vLLM, Lemonade): that route only accepts the served model's +//! name, and an Anthropic client sends a Claude one. +//! +//! The translation is the text subset of the Messages API: system prompts and +//! text turns, sampling parameters, stop sequences, usage and stop reasons. +//! Streaming, tool use (`tools`, `tool_choice`), extended thinking and non-text +//! content blocks are refused with an Anthropic `invalid_request_error` that says +//! so, rather than being dropped silently. Fields that only annotate a request +//! (`metadata`) are ignored. + +use std::net::SocketAddr; +use std::path::PathBuf; +use std::time::Duration; + +use axum::body::Bytes; +use axum::extract::State; +use axum::http::{HeaderMap, StatusCode}; +use axum::response::{IntoResponse, Response}; +use axum::routing::{get, post}; +use axum::{Json, Router}; +use rocm_core::{ + ANTHROPIC_ENDPOINT_MODE_STATE_KEY, ANTHROPIC_ENDPOINT_URL_STATE_KEY, AnthropicEndpointMode, + format_http_base_url, http_get_with_auth, http_post_json_with_auth, merge_json_state_file, + openai_models_endpoint_has_model, +}; +use serde_json::{Value, json}; + +/// How long to wait for the engine to list its model before giving up on +/// publishing an Anthropic endpoint. Model loads can take many minutes. +const ENGINE_LISTING_TIMEOUT: Duration = Duration::from_mins(30); + +/// Budget for one proxied generation. +const UPSTREAM_REQUEST_TIMEOUT: Duration = Duration::from_mins(10); + +/// What the proxy needs to know about the engine it fronts. +#[derive(Debug, Clone)] +pub struct AnthropicProxyConfig { + /// Host the engine listens on, as given to `rocm serve`. The proxy binds the + /// same host, so it is never reachable from further away than the engine. + pub host: String, + pub engine_port: u16, + /// The model the engine was asked to serve. Every request is sent under the + /// name the engine actually serves it as — `served_model_id` in the state + /// file when the engine reported one, else this — whatever model the client + /// asked for: an Anthropic client names a Claude model, no local engine's. + pub served_model: String, + /// The engine state file the endpoint is published to. + pub state_path: PathBuf, + /// The service's endpoint key, used only for the proxy's own probes. Client + /// requests are never answered with it: a client brings its own. + pub probe_api_key: Option, +} + +impl AnthropicProxyConfig { + /// Where to reach the engine from this machine, even when it listens on a + /// wildcard address. + fn upstream_url(&self) -> String { + let host = match self.host.as_str() { + "0.0.0.0" => "127.0.0.1", + "::" => "::1", + other => other, + }; + format_http_base_url(host, self.engine_port) + } +} + +/// Start publishing an Anthropic-compatible endpoint for the engine described by +/// `config`, on a background thread that lives as long as the process. +pub fn spawn_alongside_engine(config: AnthropicProxyConfig) { + let _ = std::thread::Builder::new() + .name("anthropic-endpoint".to_owned()) + .spawn(move || { + if let Err(error) = run(&config) { + eprintln!("anthropic endpoint unavailable: {error:#}"); + } + }); +} + +fn run(config: &AnthropicProxyConfig) -> anyhow::Result<()> { + let upstream = config.upstream_url(); + wait_for_engine_listing(config, &upstream)?; + + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build()?; + runtime.block_on(async { + let listener = tokio::net::TcpListener::bind((config.host.as_str(), 0)).await?; + let address: SocketAddr = listener.local_addr()?; + // Published only once the socket is bound, so a reader never sees a URL + // nothing is listening on. + publish( + config, + &format_http_base_url(&config.host, address.port()), + AnthropicEndpointMode::Proxy, + )?; + axum::serve(listener, router(ProxyState::from_config(config))).await?; + Ok(()) + }) +} + +fn wait_for_engine_listing(config: &AnthropicProxyConfig, upstream: &str) -> anyhow::Result<()> { + let started = std::time::Instant::now(); + while started.elapsed() < ENGINE_LISTING_TIMEOUT { + if openai_models_endpoint_has_model( + upstream, + // Any listed model counts: a container engine lists its own name, + // not the reference it was asked to serve. + None, + config.probe_api_key.as_deref(), + Duration::from_secs(3), + ) + .unwrap_or(false) + { + return Ok(()); + } + std::thread::sleep(Duration::from_secs(2)); + } + anyhow::bail!("the engine did not list a model in time") +} + +fn publish( + config: &AnthropicProxyConfig, + url: &str, + mode: AnthropicEndpointMode, +) -> anyhow::Result<()> { + // The listener URL carries no `/v1`: Anthropic SDKs append `/v1/messages`. + let base = url.trim_end_matches('/').trim_end_matches("/v1"); + merge_json_state_file( + &config.state_path, + &json!({ + ANTHROPIC_ENDPOINT_URL_STATE_KEY: base, + ANTHROPIC_ENDPOINT_MODE_STATE_KEY: mode.as_str(), + }), + ) +} + +#[derive(Clone)] +struct ProxyState { + upstream: String, + served_model: String, +} + +impl ProxyState { + fn from_config(config: &AnthropicProxyConfig) -> Self { + let reported = std::fs::read(&config.state_path) + .ok() + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .and_then(|state| { + state + .get("served_model_id") + .and_then(Value::as_str) + .map(str::to_owned) + }) + .filter(|name| !name.trim().is_empty()); + Self { + upstream: config.upstream_url(), + served_model: reported.unwrap_or_else(|| config.served_model.clone()), + } + } +} + +fn router(state: ProxyState) -> Router { + Router::new() + .route("/v1/messages", post(messages)) + .route("/v1/models", get(models)) + .with_state(state) +} + +/// What went wrong, in the terms of the Anthropic error envelope. +#[derive(Debug, Clone, PartialEq, Eq)] +struct ProxyError { + status: u16, + kind: &'static str, + message: String, +} + +impl ProxyError { + fn invalid(message: impl Into) -> Self { + Self { + status: 400, + kind: "invalid_request_error", + message: message.into(), + } + } + + fn unsupported(what: &str) -> Self { + Self::invalid(format!( + "{what} is not supported by the rocm Anthropic-compatible endpoint yet" + )) + } + + /// An error the engine returned, relabelled with Anthropic's error types. + fn from_upstream(status: u16, body: &str) -> Self { + let message = serde_json::from_str::(body) + .ok() + .and_then(|value| { + // OpenAI-style `{"error": {"message": ...}}` or `{"error": "..."}`, + // and FastAPI-style `{"detail": "..."}`. + let error = value.get("error").or_else(|| value.get("detail"))?; + error + .get("message") + .or(Some(error)) + .and_then(Value::as_str) + .map(str::to_owned) + }) + .unwrap_or_else(|| format!("the engine answered HTTP {status}")); + let kind = match status { + 400 | 422 => "invalid_request_error", + 401 => "authentication_error", + 403 => "permission_error", + 404 => "not_found_error", + 413 => "request_too_large", + 429 => "rate_limit_error", + _ => "api_error", + }; + Self { + status, + kind, + message, + } + } + + fn unreachable(error: &anyhow::Error) -> Self { + let timed_out = error.chain().any(|cause| { + cause.downcast_ref::().is_some_and(|io| { + matches!( + io.kind(), + std::io::ErrorKind::TimedOut | std::io::ErrorKind::WouldBlock + ) + }) + }); + if timed_out { + return Self { + status: 504, + kind: "api_error", + message: format!("the engine did not answer in time: {error:#}"), + }; + } + Self { + status: 502, + kind: "api_error", + message: format!("the engine could not be reached: {error:#}"), + } + } +} + +impl IntoResponse for ProxyError { + fn into_response(self) -> Response { + let body = json!({ + "type": "error", + "error": { "type": self.kind, "message": self.message }, + }); + ( + StatusCode::from_u16(self.status).unwrap_or(StatusCode::BAD_GATEWAY), + Json(body), + ) + .into_response() + } +} + +/// The key a client presented, in either of the two places Anthropic clients put +/// it, as the bearer token the engine expects. +fn client_api_key(headers: &HeaderMap) -> Option { + let header = |name: &str| { + headers + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::trim) + .filter(|value| !value.is_empty()) + }; + header("x-api-key") + .or_else(|| { + header("authorization").and_then(|value| match value.split_once(' ') { + Some((scheme, token)) if scheme.eq_ignore_ascii_case("bearer") => Some(token), + // Another scheme (`Basic ...`) carries no key the engine accepts. + Some(_) => None, + None => Some(value), + }) + }) + .map(str::trim) + .filter(|key| !key.is_empty()) + .map(str::to_owned) +} + +async fn messages(State(state): State, headers: HeaderMap, body: Bytes) -> Response { + let key = client_api_key(&headers); + let result = + tokio::task::spawn_blocking(move || forward_messages(&state, key.as_deref(), &body)) + .await + .unwrap_or_else(|error| { + Err(ProxyError { + status: 500, + kind: "api_error", + message: format!("the proxy task failed: {error}"), + }) + }); + match result { + Ok(message) => Json(message).into_response(), + Err(error) => error.into_response(), + } +} + +fn forward_messages( + state: &ProxyState, + key: Option<&str>, + body: &[u8], +) -> Result { + let request: Value = serde_json::from_slice(body) + .map_err(|error| ProxyError::invalid(format!("the request body is not JSON: {error}")))?; + let upstream_request = anthropic_to_openai(&request, &state.served_model)?; + let response = http_post_json_with_auth( + &state.upstream, + "/v1/chat/completions", + &upstream_request, + key, + UPSTREAM_REQUEST_TIMEOUT, + ) + .map_err(|error| ProxyError::unreachable(&error))?; + if response.status != 200 { + return Err(ProxyError::from_upstream(response.status, &response.body)); + } + let reply: Value = serde_json::from_str(response.body.trim()).map_err(|error| ProxyError { + status: 502, + kind: "api_error", + message: format!("the engine's reply was not JSON: {error}"), + })?; + openai_to_anthropic(&reply, &state.served_model) +} + +async fn models(State(state): State, headers: HeaderMap) -> Response { + let key = client_api_key(&headers); + let result = tokio::task::spawn_blocking(move || { + http_get_with_auth( + &state.upstream, + "/v1/models", + key.as_deref(), + Duration::from_secs(10), + ) + }) + .await; + match result { + Ok(Ok(response)) if response.status == 200 => ( + StatusCode::OK, + [("content-type", "application/json")], + response.body, + ) + .into_response(), + Ok(Ok(response)) => { + ProxyError::from_upstream(response.status, &response.body).into_response() + } + Ok(Err(error)) => ProxyError::unreachable(&error).into_response(), + Err(error) => ProxyError { + status: 500, + kind: "api_error", + message: format!("the proxy task failed: {error}"), + } + .into_response(), + } +} + +/// The text of an Anthropic `system` or message `content` value. +/// +/// A string is itself; an array must hold only `text` blocks. +fn text_of(content: &Value, field: &str) -> Result { + match content { + Value::String(text) => Ok(text.clone()), + Value::Array(blocks) => { + let mut parts = Vec::with_capacity(blocks.len()); + for block in blocks { + match block.get("type").and_then(Value::as_str) { + Some("text") => parts.push( + block + .get("text") + .and_then(Value::as_str) + .ok_or_else(|| { + ProxyError::invalid(format!( + "a `text` content block in `{field}` needs a string `text`" + )) + })? + .to_owned(), + ), + Some(other) => { + return Err(ProxyError::unsupported(&format!( + "a `{other}` content block in `{field}`" + ))); + } + None => { + return Err(ProxyError::invalid(format!( + "a content block in `{field}` has no `type`" + ))); + } + } + } + Ok(parts.join("\n\n")) + } + _ => Err(ProxyError::invalid(format!( + "`{field}` must be a string or an array of content blocks" + ))), + } +} + +/// Translate an Anthropic Messages request into an OpenAI chat-completions one. +fn anthropic_to_openai(request: &Value, served_model: &str) -> Result { + let object = request + .as_object() + .ok_or_else(|| ProxyError::invalid("the request body must be a JSON object"))?; + if object.get("stream").and_then(Value::as_bool) == Some(true) { + return Err(ProxyError::unsupported("streaming (`stream: true`)")); + } + if object + .get("tools") + .and_then(Value::as_array) + .is_some_and(|tools| !tools.is_empty()) + { + return Err(ProxyError::unsupported("tool use (`tools`)")); + } + if object + .get("tool_choice") + .is_some_and(|value| !value.is_null()) + { + return Err(ProxyError::unsupported("`tool_choice`")); + } + if object + .get("thinking") + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + .is_some_and(|kind| kind != "disabled") + { + return Err(ProxyError::unsupported("extended thinking (`thinking`)")); + } + let max_tokens = object + .get("max_tokens") + .and_then(Value::as_u64) + .filter(|value| *value > 0) + .ok_or_else(|| ProxyError::invalid("max_tokens: a positive integer is required"))?; + let turns = object + .get("messages") + .and_then(Value::as_array) + .filter(|turns| !turns.is_empty()) + .ok_or_else(|| ProxyError::invalid("messages: at least one message is required"))?; + + let mut messages = Vec::with_capacity(turns.len() + 1); + if let Some(system) = object.get("system").filter(|value| !value.is_null()) { + let text = text_of(system, "system")?; + if !text.is_empty() { + messages.push(json!({ "role": "system", "content": text })); + } + } + for turn in turns { + let role = turn + .get("role") + .and_then(Value::as_str) + .filter(|role| matches!(*role, "user" | "assistant")) + .ok_or_else(|| { + ProxyError::invalid("messages: each message needs a `role` of user or assistant") + })?; + let content = turn + .get("content") + .ok_or_else(|| ProxyError::invalid("messages: each message needs `content`"))?; + messages.push(json!({ "role": role, "content": text_of(content, "messages")? })); + } + + let mut out = json!({ + "model": served_model, + "messages": messages, + "max_tokens": max_tokens, + "stream": false, + }); + for key in ["temperature", "top_p", "top_k"] { + if let Some(value) = object.get(key).filter(|value| value.is_number()) { + out[key] = value.clone(); + } + } + if let Some(stops) = object.get("stop_sequences").and_then(Value::as_array) + && !stops.is_empty() + { + out["stop"] = Value::Array(stops.clone()); + } + Ok(out) +} + +/// Translate an OpenAI chat-completions reply into an Anthropic Message. +fn openai_to_anthropic(reply: &Value, served_model: &str) -> Result { + let choice = reply + .get("choices") + .and_then(Value::as_array) + .and_then(|choices| choices.first()) + .ok_or_else(|| ProxyError { + status: 502, + kind: "api_error", + message: "the engine's reply had no choices".to_owned(), + })?; + let text = choice + .pointer("/message/content") + .and_then(Value::as_str) + .unwrap_or_default(); + let stop_reason = match choice.get("finish_reason").and_then(Value::as_str) { + Some("length") => "max_tokens", + _ => "end_turn", + }; + let id = reply + .get("id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + .map_or_else( + || "msg_rocm".to_owned(), + |id| format!("msg_{}", id.trim_start_matches("chatcmpl-")), + ); + let token_count = |key: &str| { + reply + .pointer(&format!("/usage/{key}")) + .and_then(Value::as_u64) + .unwrap_or(0) + }; + Ok(json!({ + "id": id, + "type": "message", + "role": "assistant", + "model": reply + .get("model") + .and_then(Value::as_str) + .filter(|model| !model.is_empty()) + .unwrap_or(served_model), + "content": [{ "type": "text", "text": text }], + "stop_reason": stop_reason, + "stop_sequence": null, + "usage": { + "input_tokens": token_count("prompt_tokens"), + "output_tokens": token_count("completion_tokens"), + }, + })) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Read, Write}; + use std::net::TcpListener; + use std::sync::{Arc, Mutex}; + + const MODEL: &str = "Org/Served-Model"; + + /// A directory no other test in this process shares. Process id plus a + /// millisecond clock is not enough: tests run in parallel threads and two can + /// start in the same millisecond, then overwrite each other's state file. + fn unique_temp_dir(label: &str) -> std::path::PathBuf { + static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + std::env::temp_dir().join(format!( + "rocmd-anthropic-{label}-{}-{}", + std::process::id(), + NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + )) + } + + fn translate(request: &Value) -> Result { + anthropic_to_openai(request, MODEL) + } + + #[test] + fn request_maps_system_turns_and_sampling_to_chat_completions() { + let out = translate(&json!({ + "model": "claude-sonnet-4", + "max_tokens": 64, + "system": [{"type": "text", "text": "Be brief."}, {"type": "text", "text": "Be kind."}], + "messages": [ + {"role": "user", "content": "hi"}, + {"role": "assistant", "content": [{"type": "text", "text": "hello"}]}, + {"role": "user", "content": "again"}, + ], + "temperature": 0.2, + "top_p": 0.9, + "top_k": 40, + "stop_sequences": ["END"], + })) + .unwrap(); + assert_eq!( + out, + json!({ + "model": MODEL, + "messages": [ + {"role": "system", "content": "Be brief.\n\nBe kind."}, + {"role": "user", "content": "hi"}, + {"role": "assistant", "content": "hello"}, + {"role": "user", "content": "again"}, + ], + "max_tokens": 64, + "stream": false, + "temperature": 0.2, + "top_p": 0.9, + "top_k": 40, + "stop": ["END"], + }) + ); + } + + #[test] + fn request_always_names_the_served_model_not_the_clients() { + let out = translate(&json!({ + "model": "claude-opus-4", + "max_tokens": 1, + "messages": [{"role": "user", "content": "x"}], + })) + .unwrap(); + assert_eq!(out["model"], MODEL); + } + + #[test] + fn request_refuses_what_it_cannot_translate_instead_of_dropping_it() { + let base = || json!({"max_tokens": 8, "messages": [{"role": "user", "content": "x"}]}); + let mut streaming = base(); + streaming["stream"] = json!(true); + let mut tools = base(); + tools["tools"] = json!([{"name": "get_weather", "input_schema": {"type": "object"}}]); + let image = json!({"max_tokens": 8, "messages": [{"role": "user", "content": [ + {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": ""}} + ]}]}); + for (request, needle) in [ + (streaming, "streaming"), + (tools, "tool use"), + (image, "`image` content block"), + ] { + let error = translate(&request).unwrap_err(); + assert_eq!(error.status, 400); + assert_eq!(error.kind, "invalid_request_error"); + assert!(error.message.contains(needle), "{}", error.message); + } + } + + #[test] + fn request_rejects_malformed_input_with_invalid_request_errors() { + for request in [ + json!([]), + json!({"messages": [{"role": "user", "content": "x"}]}), + json!({"max_tokens": 0, "messages": [{"role": "user", "content": "x"}]}), + json!({"max_tokens": 8, "messages": []}), + json!({"max_tokens": 8}), + json!({"max_tokens": 8, "messages": [{"role": "system", "content": "x"}]}), + json!({"max_tokens": 8, "messages": [{"role": "user"}]}), + ] { + let error = translate(&request).unwrap_err(); + assert_eq!(error.kind, "invalid_request_error", "{request}"); + } + } + + #[test] + fn response_becomes_a_valid_anthropic_message() { + let message = openai_to_anthropic( + &json!({ + "id": "chatcmpl-abc123", + "model": MODEL, + "choices": [{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}], + "usage": {"prompt_tokens": 11, "completion_tokens": 2}, + }), + "fallback", + ) + .unwrap(); + assert_eq!( + message, + json!({ + "id": "msg_abc123", + "type": "message", + "role": "assistant", + "model": MODEL, + "content": [{"type": "text", "text": "ok"}], + "stop_reason": "end_turn", + "stop_sequence": null, + "usage": {"input_tokens": 11, "output_tokens": 2}, + }) + ); + } + + #[test] + fn response_maps_a_length_stop_and_falls_back_to_the_served_model() { + let message = openai_to_anthropic( + &json!({"choices": [{"message": {"content": null}, "finish_reason": "length"}]}), + MODEL, + ) + .unwrap(); + assert_eq!(message["stop_reason"], "max_tokens"); + assert_eq!(message["model"], MODEL); + assert_eq!(message["content"][0]["text"], ""); + assert_eq!(message["usage"]["input_tokens"], 0); + } + + #[test] + fn response_without_choices_is_a_bad_gateway() { + let error = openai_to_anthropic(&json!({"choices": []}), MODEL).unwrap_err(); + assert_eq!(error.status, 502); + } + + #[test] + fn upstream_errors_keep_their_status_and_message_with_anthropic_types() { + let error = + ProxyError::from_upstream(404, r#"{"error": {"message": "model `x` does not exist"}}"#); + assert_eq!( + (error.status, error.kind, error.message.as_str()), + (404, "not_found_error", "model `x` does not exist") + ); + let error = ProxyError::from_upstream(401, "not json"); + assert_eq!(error.kind, "authentication_error"); + assert_eq!(error.message, "the engine answered HTTP 401"); + assert_eq!(ProxyError::from_upstream(500, "").kind, "api_error"); + } + + #[test] + fn client_key_is_read_from_either_header() { + let mut headers = HeaderMap::new(); + assert_eq!(client_api_key(&headers), None); + headers.insert("authorization", "Bearer from-bearer".parse().unwrap()); + assert_eq!(client_api_key(&headers).as_deref(), Some("from-bearer")); + headers.insert("x-api-key", "from-x-api-key".parse().unwrap()); + assert_eq!(client_api_key(&headers).as_deref(), Some("from-x-api-key")); + } + + #[test] + fn an_empty_x_api_key_falls_back_to_the_authorization_header() { + let mut headers = HeaderMap::new(); + headers.insert("x-api-key", " ".parse().unwrap()); + headers.insert("authorization", "bearer from-bearer".parse().unwrap()); + assert_eq!(client_api_key(&headers).as_deref(), Some("from-bearer")); + headers.insert("authorization", "Basic dXNlcjpwdw==".parse().unwrap()); + assert_eq!(client_api_key(&headers), None); + } + + #[test] + fn engine_errors_shaped_as_detail_keep_their_message() { + let error = ProxyError::from_upstream(404, r#"{"detail": "no such route"}"#); + assert_eq!(error.message, "no such route"); + } + + #[test] + fn a_timed_out_engine_is_a_gateway_timeout_not_unreachable() { + let timeout = anyhow::Error::new(std::io::Error::from(std::io::ErrorKind::TimedOut)) + .context("failed to read HTTP POST /v1/chat/completions"); + assert_eq!(ProxyError::unreachable(&timeout).status, 504); + let refused = + anyhow::Error::new(std::io::Error::from(std::io::ErrorKind::ConnectionRefused)); + assert_eq!(ProxyError::unreachable(&refused).status, 502); + } + + #[test] + fn requests_that_would_silently_lose_meaning_are_refused() { + let base = json!({"max_tokens": 4, "messages": [{"role": "user", "content": "x"}]}); + for (field, value) in [ + ("tool_choice", json!({"type": "auto"})), + ( + "thinking", + json!({"type": "enabled", "budget_tokens": 1024}), + ), + ] { + let mut request = base.clone(); + request[field] = value; + let error = translate(&request).unwrap_err(); + assert_eq!(error.status, 400, "{field}"); + } + let mut disabled = base; + disabled["thinking"] = json!({"type": "disabled"}); + assert!(translate(&disabled).is_ok()); + let missing_text = json!({ + "max_tokens": 4, + "messages": [{"role": "user", "content": [{"type": "text"}]}], + }); + let error = translate(&missing_text).unwrap_err(); + assert_eq!(error.status, 400); + assert!(error.message.contains("string `text`"), "{}", error.message); + } + + #[test] + fn wildcard_listen_hosts_are_reached_over_loopback() { + let config = |host: &str| AnthropicProxyConfig { + host: host.to_owned(), + engine_port: 8000, + served_model: MODEL.to_owned(), + state_path: PathBuf::new(), + probe_api_key: None, + }; + assert_eq!(config("0.0.0.0").upstream_url(), "http://127.0.0.1:8000"); + assert_eq!(config("::").upstream_url(), "http://[::1]:8000"); + assert_eq!(config("127.0.0.1").upstream_url(), "http://127.0.0.1:8000"); + } + + /// A stand-in OpenAI engine: lists its model, answers one chat completion, + /// and records the `Authorization` header and body of what it was sent. + struct FakeEngine { + port: u16, + seen: Arc>>, + } + + fn fake_engine() -> FakeEngine { + fake_engine_with(false) + } + + /// `native` makes it answer `/v1/messages` the way an engine that serves one + /// does (refusing an empty body); otherwise that route is a 404. + fn fake_engine_with(native: bool) -> FakeEngine { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let seen: Arc>> = Arc::default(); + let log = Arc::clone(&seen); + std::thread::spawn(move || { + for stream in listener.incoming() { + let Ok(mut stream) = stream else { break }; + let mut raw = Vec::new(); + let mut chunk = [0_u8; 4096]; + loop { + let read = stream.read(&mut chunk).unwrap_or(0); + raw.extend_from_slice(&chunk[..read]); + let text = String::from_utf8_lossy(&raw).into_owned(); + let Some((head, body)) = text.split_once("\r\n\r\n") else { + if read == 0 { + break; + } + continue; + }; + let length = head + .lines() + .find_map(|line| { + line.to_ascii_lowercase() + .strip_prefix("content-length:") + .map(|v| v.trim().parse::().unwrap_or(0)) + }) + .unwrap_or(0); + if body.len() >= length || read == 0 { + break; + } + } + let text = String::from_utf8_lossy(&raw).into_owned(); + let (head, body) = text.split_once("\r\n\r\n").unwrap_or((&text, "")); + let request_line = head.lines().next().unwrap_or_default().to_owned(); + let auth = head + .lines() + .find_map(|line| { + line.to_ascii_lowercase() + .starts_with("authorization:") + .then(|| line.split_once(':').map(|(_, v)| v.trim().to_owned())) + .flatten() + }) + .unwrap_or_default(); + let (status, reply) = if request_line.contains("/v1/messages") { + if native { + ("400 Bad Request", json!({"error": "invalid request"})) + } else { + ("404 Not Found", json!({"detail": "Not Found"})) + } + } else if auth == "Bearer wrong" { + ("401 Unauthorized", json!({"error": {"message": "bad key"}})) + } else if request_line.contains("/v1/models") { + ("200 OK", json!({"data": [{"id": MODEL}]})) + } else { + log.lock().unwrap().push((auth, body.to_owned())); + ( + "200 OK", + json!({ + "id": "chatcmpl-1", + "model": MODEL, + "choices": [{"message": {"content": "pong"}, "finish_reason": "stop"}], + "usage": {"prompt_tokens": 3, "completion_tokens": 1}, + }), + ) + }; + let reply = reply.to_string(); + let _ = write!( + stream, + "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{reply}", + reply.len() + ); + } + }); + FakeEngine { port, seen } + } + + /// Start the proxy router over a fake engine and return its base URL. + async fn proxy_over(engine: &FakeEngine) -> String { + let state = ProxyState { + upstream: format!("http://127.0.0.1:{}", engine.port), + served_model: MODEL.to_owned(), + }; + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { axum::serve(listener, router(state)).await }); + format!("http://127.0.0.1:{port}") + } + + /// POST `body` to the proxy and return `(status, json body)`. + fn post(base: &str, path: &str, key: Option<&str>, body: &Value) -> (u16, Value) { + let mut request = ureq::post(&format!("{base}{path}")); + if let Some(key) = key { + request = request.set("x-api-key", key); + } + let parse = |response: ureq::Response| { + let status = response.status(); + ( + status, + serde_json::from_str(&response.into_string().unwrap()).unwrap(), + ) + }; + match request + .set("content-type", "application/json") + .send_string(&body.to_string()) + { + Ok(response) => parse(response), + Err(ureq::Error::Status(_, response)) => parse(response), + Err(error) => panic!("{error}"), + } + } + + #[tokio::test(flavor = "multi_thread")] + async fn messages_round_trips_through_the_engine_with_the_clients_key() { + let engine = fake_engine(); + let base = proxy_over(&engine).await; + let (status, message) = tokio::task::spawn_blocking(move || { + post( + &base, + "/v1/messages", + Some("client-key"), + &json!({ + "model": "claude-sonnet-4", + "max_tokens": 16, + "messages": [{"role": "user", "content": "ping"}], + }), + ) + }) + .await + .unwrap(); + assert_eq!(status, 200); + assert_eq!(message["type"], "message"); + assert_eq!(message["model"], MODEL); + assert_eq!(message["content"][0]["text"], "pong"); + assert_eq!(message["stop_reason"], "end_turn"); + let seen = engine.seen.lock().unwrap(); + assert_eq!(seen.len(), 1); + assert_eq!(seen[0].0, "Bearer client-key"); + let forwarded: Value = serde_json::from_str(&seen[0].1).unwrap(); + assert_eq!(forwarded["model"], MODEL); + assert_eq!(forwarded["messages"][0]["content"], "ping"); + } + + #[tokio::test(flavor = "multi_thread")] + async fn engine_refusals_reach_the_client_as_anthropic_errors() { + let engine = fake_engine(); + let base = proxy_over(&engine).await; + let (status, error) = tokio::task::spawn_blocking(move || { + post( + &base, + "/v1/messages", + Some("wrong"), + &json!({"max_tokens": 4, "messages": [{"role": "user", "content": "x"}]}), + ) + }) + .await + .unwrap(); + assert_eq!(status, 401); + assert_eq!( + error, + json!({"type": "error", "error": {"type": "authentication_error", "message": "bad key"}}) + ); + } + + #[tokio::test(flavor = "multi_thread")] + async fn untranslatable_requests_are_refused_before_reaching_the_engine() { + let engine = fake_engine(); + let base = proxy_over(&engine).await; + let (status, error) = tokio::task::spawn_blocking(move || { + post( + &base, + "/v1/messages", + None, + &json!({"max_tokens": 4, "stream": true, "messages": [{"role": "user", "content": "x"}]}), + ) + }) + .await + .unwrap(); + assert_eq!(status, 400); + assert_eq!(error["error"]["type"], "invalid_request_error"); + assert!(engine.seen.lock().unwrap().is_empty()); + } + + #[tokio::test(flavor = "multi_thread")] + async fn a_models_auth_failure_is_relayed_not_reported_as_unreachable() { + let engine = fake_engine(); + let base = proxy_over(&engine).await; + let (status, error) = tokio::task::spawn_blocking(move || { + match ureq::get(&format!("{base}/v1/models")) + .set("x-api-key", "wrong") + .call() + { + Err(ureq::Error::Status(status, response)) => ( + status, + serde_json::from_str::(&response.into_string().unwrap()).unwrap(), + ), + other => panic!("expected an HTTP error, got {other:?}"), + } + }) + .await + .unwrap(); + assert_eq!(status, 401); + assert_eq!(error["error"]["type"], "authentication_error"); + assert_eq!(error["error"]["message"], "bad key"); + } + + #[tokio::test(flavor = "multi_thread")] + async fn models_are_forwarded_so_readiness_covers_the_proxy() { + let engine = fake_engine(); + let base = proxy_over(&engine).await; + let ready = tokio::task::spawn_blocking(move || { + rocm_core::anthropic_endpoint_ready(&base, Some(MODEL), None, Duration::from_secs(5)) + }) + .await + .unwrap(); + assert!(ready); + } + + #[test] + fn proxy_state_prefers_the_served_name_the_engine_reported() { + let dir = unique_temp_dir("served"); + std::fs::create_dir_all(&dir).unwrap(); + let mut config = AnthropicProxyConfig { + host: "127.0.0.1".to_owned(), + engine_port: 1, + served_model: "org/image:1".to_owned(), + state_path: dir.join("state.json"), + probe_api_key: None, + }; + assert_eq!(ProxyState::from_config(&config).served_model, "org/image:1"); + std::fs::write(&config.state_path, r#"{"served_model_id": "Org/Served"}"#).unwrap(); + assert_eq!(ProxyState::from_config(&config).served_model, "Org/Served"); + config.state_path = dir.join("missing.json"); + assert_eq!(ProxyState::from_config(&config).served_model, "org/image:1"); + let _ = std::fs::remove_dir_all(dir); + } + + fn published_endpoint_for(engine: &FakeEngine) -> (String, String) { + let dir = unique_temp_dir("run"); + std::fs::create_dir_all(&dir).unwrap(); + let state_path = dir.join("state.json"); + spawn_alongside_engine(AnthropicProxyConfig { + host: "127.0.0.1".to_owned(), + engine_port: engine.port, + served_model: MODEL.to_owned(), + state_path: state_path.clone(), + probe_api_key: None, + }); + let url = + rocm_core::wait_for_published_anthropic_endpoint(&state_path, Duration::from_secs(30)) + .expect("no Anthropic endpoint was published"); + let state: Value = serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); + let _ = std::fs::remove_dir_all(dir); + ( + url, + state["anthropic_endpoint_mode"] + .as_str() + .unwrap() + .to_owned(), + ) + } + + #[test] + fn an_engine_without_the_route_is_fronted_by_a_proxy_on_its_own_port() { + let engine = fake_engine_with(false); + let (url, mode) = published_endpoint_for(&engine); + assert_eq!(mode, "proxy"); + assert_ne!(url, format!("http://127.0.0.1:{}", engine.port)); + let (status, message) = post( + &url, + "/v1/messages", + None, + &json!({"max_tokens": 8, "messages": [{"role": "user", "content": "ping"}]}), + ); + assert_eq!( + (status, message["content"][0]["text"].as_str()), + (200, Some("pong")) + ); + } + + #[test] + fn an_engine_that_serves_the_route_is_still_fronted_by_the_proxy() { + // vLLM and Lemonade answer `/v1/messages` themselves but reject any model + // name other than the one they serve; the proxy answers under any name. + let engine = fake_engine_with(true); + let (url, mode) = published_endpoint_for(&engine); + assert_eq!(mode, "proxy"); + assert_ne!(url, format!("http://127.0.0.1:{}", engine.port)); + let (status, message) = post( + &url, + "/v1/messages", + None, + &json!({ + "model": "claude-sonnet-4-5", + "max_tokens": 8, + "messages": [{"role": "user", "content": "ping"}], + }), + ); + assert_eq!( + (status, message["content"][0]["text"].as_str()), + (200, Some("pong")) + ); + } + + #[test] + fn publish_records_the_endpoint_without_a_v1_suffix() { + let dir = unique_temp_dir("publish"); + let config = AnthropicProxyConfig { + host: "127.0.0.1".to_owned(), + engine_port: 1, + served_model: MODEL.to_owned(), + state_path: dir.join("state.json"), + probe_api_key: None, + }; + publish( + &config, + "http://127.0.0.1:8000/v1", + AnthropicEndpointMode::Proxy, + ) + .unwrap(); + let state: Value = + serde_json::from_slice(&std::fs::read(&config.state_path).unwrap()).unwrap(); + assert_eq!(state["anthropic_endpoint_url"], "http://127.0.0.1:8000"); + assert_eq!(state["anthropic_endpoint_mode"], "proxy"); + let _ = std::fs::remove_dir_all(dir); + } +} diff --git a/apps/rocmd/src/lib.rs b/apps/rocmd/src/lib.rs index 5b7d3d2d88..2a478a8b0f 100644 --- a/apps/rocmd/src/lib.rs +++ b/apps/rocmd/src/lib.rs @@ -4,6 +4,7 @@ #![allow(clippy::items_after_test_module)] +pub mod anthropic_proxy; mod cli; mod common; mod daemon; diff --git a/crates/rocm-core/src/anthropic_endpoint.rs b/crates/rocm-core/src/anthropic_endpoint.rs new file mode 100644 index 0000000000..744ae7d023 --- /dev/null +++ b/crates/rocm-core/src/anthropic_endpoint.rs @@ -0,0 +1,123 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The Anthropic-compatible `/v1/messages` surface of a managed service. +//! +//! An engine speaks the OpenAI API. A service additionally gets an Anthropic +//! Messages endpoint because `rocm` fronts the engine with a translation proxy +//! ([`AnthropicEndpointMode::Proxy`]), even when the engine has a `/v1/messages` +//! route of its own: an engine's route only accepts the model name it serves, +//! while an Anthropic client names a Claude model. The process that runs the +//! engine publishes the proxy's address under the state-file keys below, and [`crate::ManagedServiceRecord::refresh_from_engine_state`] adopts +//! it, so every reader (`rocm serve`, `rocm services list`, the readiness probe) +//! sees the same endpoint. + +use std::path::Path; +use std::time::{Duration, Instant}; + +use crate::openai_models_endpoint_has_model; + +/// Engine state-file key naming the Anthropic-compatible endpoint's base URL. +pub const ANTHROPIC_ENDPOINT_URL_STATE_KEY: &str = "anthropic_endpoint_url"; + +/// Engine state-file key recording how the Anthropic endpoint is provided. +pub const ANTHROPIC_ENDPOINT_MODE_STATE_KEY: &str = "anthropic_endpoint_mode"; + +/// How a service provides its Anthropic-compatible endpoint. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum AnthropicEndpointMode { + /// A `rocm`-hosted proxy translates Anthropic requests to the engine's + /// OpenAI API, on its own port. + Proxy, +} + +impl AnthropicEndpointMode { + /// The value stored under [`ANTHROPIC_ENDPOINT_MODE_STATE_KEY`]. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::Proxy => "proxy", + } + } +} + +/// Whether the Anthropic endpoint at `anthropic_url` is up and fronts `model`. +/// +/// The proxy answers `GET /v1/models` by forwarding to the engine. So the same cheap listing check that gates +/// the OpenAI endpoint also covers this one, without queuing a generation. +pub fn anthropic_endpoint_ready( + anthropic_url: &str, + model: Option<&str>, + endpoint_api_key: Option<&str>, + timeout: Duration, +) -> bool { + openai_models_endpoint_has_model(anthropic_url, model, endpoint_api_key, timeout) + .unwrap_or(false) +} + +/// The Anthropic endpoint the serving process has published in `state_path`, +/// waiting up to `wait` for it to appear. +/// +/// The endpoint is published a moment after the engine first lists its model, so +/// a caller that has just seen the service become ready can still find nothing +/// there. `Duration::ZERO` reads once. +pub fn wait_for_published_anthropic_endpoint(state_path: &Path, wait: Duration) -> Option { + let started = Instant::now(); + loop { + let published = std::fs::read(state_path) + .ok() + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .and_then(|state| { + state + .get(ANTHROPIC_ENDPOINT_URL_STATE_KEY) + .and_then(serde_json::Value::as_str) + .filter(|url| !url.trim().is_empty()) + .map(str::to_owned) + }); + if published.is_some() || started.elapsed() >= wait { + return published; + } + std::thread::sleep(Duration::from_millis(200)); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn published_endpoint_is_read_from_state_and_absent_when_unpublished() { + let dir = std::env::temp_dir().join(format!( + "rocm-core-anthropic-published-{}-{}", + std::process::id(), + crate::unix_time_millis() + )); + std::fs::create_dir_all(&dir).unwrap(); + let state = dir.join("state.json"); + assert_eq!( + wait_for_published_anthropic_endpoint(&state, Duration::ZERO), + None + ); + std::fs::write(&state, r#"{"status": "ready"}"#).unwrap(); + assert_eq!( + wait_for_published_anthropic_endpoint(&state, Duration::ZERO), + None + ); + std::fs::write( + &state, + r#"{"anthropic_endpoint_url": "http://127.0.0.1:41000"}"#, + ) + .unwrap(); + assert_eq!( + wait_for_published_anthropic_endpoint(&state, Duration::ZERO).as_deref(), + Some("http://127.0.0.1:41000") + ); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn mode_strings_are_the_values_published_in_state() { + assert_eq!(AnthropicEndpointMode::Proxy.as_str(), "proxy"); + } +} diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 6f0601bb56..0ccf9962e9 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -8,6 +8,7 @@ use std::path::Path; use std::time::{SystemTime, UNIX_EPOCH}; pub mod amd_smi; +pub mod anthropic_endpoint; #[cfg(all(test, unix))] mod archive_props; pub mod atomic_write; @@ -67,6 +68,10 @@ pub mod uv; pub mod vllm_oom; pub mod windows_driver; pub use amd_smi::{amd_smi_json, device_readable_bounded, resolve_amd_smi_binary}; +pub use anthropic_endpoint::{ + ANTHROPIC_ENDPOINT_MODE_STATE_KEY, ANTHROPIC_ENDPOINT_URL_STATE_KEY, AnthropicEndpointMode, + anthropic_endpoint_ready, wait_for_published_anthropic_endpoint, +}; pub use automation::{ AuditEventRecord, AutomationEventRecord, AutomationProposalRecord, AutomationRuntimeState, AutomationTriggerEvent, EVENT_LOG_ROTATE_BYTES, EVENT_LOG_ROTATED_GENERATIONS, @@ -165,7 +170,7 @@ pub use model_catalog::{ pub use net::{ DEFAULT_LOCAL_HOST, DEFAULT_LOCAL_PORT, HttpResponseParts, connect_tcp_stream, format_host_for_url, format_host_port, format_http_base_url, http_get_text, - http_get_text_with_auth, http_post_json_with_auth, parse_http_endpoint, + http_get_text_with_auth, http_get_with_auth, http_post_json_with_auth, parse_http_endpoint, read_http_response_bounded, write_all_tcp_stream, }; pub use paths::{AppPaths, engine_plugin_dirs}; diff --git a/crates/rocm-core/src/net.rs b/crates/rocm-core/src/net.rs index 2812c6d961..ca623360ff 100644 --- a/crates/rocm-core/src/net.rs +++ b/crates/rocm-core/src/net.rs @@ -67,6 +67,24 @@ pub fn http_get_text_with_auth( endpoint_api_key: Option<&str>, timeout: Duration, ) -> Result { + let response = http_get_with_auth(endpoint_url, path, endpoint_api_key, timeout)?; + if response.status != 200 { + bail!("HTTP endpoint returned status {}", response.status); + } + Ok(response.body) +} + +/// GET `path` and return the status and body, whatever the status. +/// +/// The GET sibling of [`http_post_json_with_auth`]: a caller that relays the +/// endpoint's answer (a proxy) must tell "answered with a refusal" apart from +/// "never answered", which [`http_get_text_with_auth`] folds into one error. +pub fn http_get_with_auth( + endpoint_url: &str, + path: &str, + endpoint_api_key: Option<&str>, + timeout: Duration, +) -> Result { let deadline = Instant::now() + timeout; let (host, port) = parse_http_endpoint(endpoint_url) .with_context(|| format!("unsupported endpoint URL `{endpoint_url}`"))?; @@ -87,10 +105,12 @@ pub fn http_get_text_with_auth( .split_once("\r\n\r\n") .context("HTTP response was missing a body")?; let status_line = headers.lines().next().unwrap_or_default(); - if !status_line.contains(" 200 ") { - bail!("HTTP endpoint returned {status_line}"); - } - Ok(body.to_owned()) + let status = http_status_code(status_line) + .with_context(|| format!("unparsable HTTP status line `{status_line}`"))?; + Ok(HttpResponseParts { + status, + body: body.to_owned(), + }) } /// POST a JSON body and return the response status line plus body. diff --git a/crates/rocm-core/src/service_record.rs b/crates/rocm-core/src/service_record.rs index aecab92d8c..ff6a6290d8 100644 --- a/crates/rocm-core/src/service_record.rs +++ b/crates/rocm-core/src/service_record.rs @@ -21,9 +21,9 @@ use std::fs; use std::path::PathBuf; use std::time::Duration; -use crate::AppPaths; use crate::format_http_base_url; use crate::unix_time_millis; +use crate::{ANTHROPIC_ENDPOINT_MODE_STATE_KEY, ANTHROPIC_ENDPOINT_URL_STATE_KEY, AppPaths}; use crate::{normalize_runtime_path_for_host, normalize_runtime_path_for_storage}; /// Engine state-file key recording when inference was first confirmed. @@ -127,6 +127,15 @@ pub struct ManagedServiceRecord { /// readiness was gated on inference. #[serde(default)] pub inference_verified_at_unix_ms: Option, + /// Base URL of the Anthropic-compatible `/v1/messages` endpoint, once the + /// serving process has published one. Adopted from the engine state file (see + /// [`crate::anthropic_endpoint`]); `None` while the service is still coming up + /// and on records written before the endpoint existed. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub anthropic_endpoint_url: Option, + /// How [`Self::anthropic_endpoint_url`] is provided: `proxy`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub anthropic_endpoint_mode: Option, pub manifest_path: PathBuf, pub log_path: PathBuf, pub engine_state_path: PathBuf, @@ -248,6 +257,8 @@ impl ManagedServiceRecord { startup_phase: None, inference_verified_at_unix_ms: None, inference_probe_attempted_at_unix_ms: None, + anthropic_endpoint_url: None, + anthropic_endpoint_mode: None, manifest_path, log_path, engine_state_path, @@ -283,6 +294,10 @@ impl ManagedServiceRecord { pub fn reset_for_restart(&mut self) { self.inference_verified_at_unix_ms = None; self.inference_probe_attempted_at_unix_ms = None; + // The respawned process publishes its own endpoint; the old proxy port + // died with the previous run. + self.anthropic_endpoint_url = None; + self.anthropic_endpoint_mode = None; self.restart_count = self.restart_count.saturating_add(1); self.last_restart_unix_ms = Some(unix_time_millis()); } @@ -433,6 +448,19 @@ impl ManagedServiceRecord { { self.inference_verified_at_unix_ms = Some(verified_at); } + // The endpoint belongs to the run that published it, so it is replaced + // rather than kept: a restarted proxy listens on a different port. + let published = |key: &str| { + state + .get(key) + .and_then(serde_json::Value::as_str) + .filter(|value| !value.trim().is_empty()) + .map(str::to_owned) + }; + if let Some(url) = published(ANTHROPIC_ENDPOINT_URL_STATE_KEY) { + self.anthropic_endpoint_url = Some(url); + self.anthropic_endpoint_mode = published(ANTHROPIC_ENDPOINT_MODE_STATE_KEY); + } Ok(self.status != previous) } @@ -645,6 +673,33 @@ mod tests { record } + /// The serving process publishes the Anthropic endpoint in the engine state; + /// the record adopts it, and a restart drops it so the next run's port wins. + #[test] + fn engine_refresh_adopts_the_published_anthropic_endpoint() { + let record = refresh_from_state( + "anthropic", + &serde_json::json!({ + "status": "ready", + "anthropic_endpoint_url": "http://127.0.0.1:40123", + "anthropic_endpoint_mode": "proxy", + }), + ); + assert_eq!( + record.anthropic_endpoint_url.as_deref(), + Some("http://127.0.0.1:40123") + ); + assert_eq!(record.anthropic_endpoint_mode.as_deref(), Some("proxy")); + + let mut record = record; + record.reset_for_restart(); + assert_eq!(record.anthropic_endpoint_url, None); + assert_eq!(record.anthropic_endpoint_mode, None); + + let record = refresh_from_state("no-anthropic", &serde_json::json!({ "status": "ready" })); + assert_eq!(record.anthropic_endpoint_url, None); + } + /// A container engine serves under a different name than the image it pulls; /// the record adopts that name from the engine state, and every readiness /// probe asks for it rather than for the image reference. diff --git a/docs/architecture.md b/docs/architecture.md index 2e51fd54c3..1b5f1c677b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -145,6 +145,7 @@ Further `apps/rocm` modules, not part of the `main.rs` split: Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`cli.rs`](../apps/rocmd/src/cli.rs) — the `Cli`/`Command` clap definitions, the bridge-snapshot diagnostic printer, the pretty-JSON printer for the subcommands that print a result (`print_json`), and the top-level dispatch in `run_cli`/`run_bin_cli`/`run_from_args`. +- [`anthropic_proxy.rs`](../apps/rocmd/src/anthropic_proxy.rs) — the Anthropic-compatible `/v1/messages` front for a served model: `spawn_alongside_engine` waits for the engine to list its model, then binds a translation proxy, even for an engine that has its own `/v1/messages` route because that route only accepts the served model's name (`anthropic_to_openai`, `openai_to_anthropic`, an axum router over `/v1/messages` and `/v1/models`) on a port of its own, publishing the result in the engine state file for the service record to adopt. It is a library module, not a daemon task: `rocm`'s `run_builtin_engine_serve_http` in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs) starts it on a thread of the process that owns the engine, so it lives and dies with the service. - [`common.rs`](../apps/rocmd/src/common.rs) — helpers the other modules call into: the read-only `rocm update` message, the `rocm`-subprocess capture wrapper (`run_rocm_capture_for_paths`) the sandbox and MCP clusters both call (it only chooses the child's `ROCM_CLI_*_DIR` environment; `CommandCapture` and the spawn itself live in [`command_capture.rs`](../crates/rocm-core/src/command_capture.rs), shared with `rocm`'s chat tool bridge, and the bounded-execution half in [`process.rs`](../crates/rocm-core/src/process.rs)), and small arg/JSON-payload utilities (`payload_string`, for example, is read by the [`watchers/cache_warm.rs`](../apps/rocmd/src/watchers/cache_warm.rs), [`watchers/driver_upgrade.rs`](../apps/rocmd/src/watchers/driver_upgrade.rs) and [`watchers/gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) handlers and by [`webhook.rs`](../apps/rocmd/src/webhook.rs)). - [`daemon.rs`](../apps/rocmd/src/daemon.rs) — the foreground `rocmd run` loop (`run_daemon`: evaluate the watchers each tick, dispatch local webhook events, write the runtime state, stop on a signal; with no watcher enabled it exits by itself, deciding that under the autostart lock and publishing "stopped" and clearing its spawn claim before releasing it — `record_daemon_stop`, `release_own_autostart_claim`) and `rocmd status` (`print_status`), with the watcher tick interval, which `LoopHooks` makes injectable for tests. It uses [`watchers.rs`](../apps/rocmd/src/watchers.rs), [`webhook.rs`](../apps/rocmd/src/webhook.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs) calls it. - [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) — talking to an engine and guarding its endpoint key: `engine_request` (the stdio request to an engine process), the healthcheck call and its readiness and recoverability checks, and the endpoint-key helpers `apply_endpoint_key_env` and `ensure_public_service_has_endpoint_key`. It uses no other module of the crate; [`service.rs`](../apps/rocmd/src/service.rs), [`supervise.rs`](../apps/rocmd/src/supervise.rs) and [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) call it. @@ -173,6 +174,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): Already-extracted subsystem modules, following the same full-domain-extraction pattern (and several further siblings beyond these): +- [`anthropic_endpoint.rs`](../crates/rocm-core/src/anthropic_endpoint.rs) — (ROCMAI-597) the Anthropic-compatible `/v1/messages` surface of a managed service: `AnthropicEndpointMode` (`rocm` fronts the OpenAI-speaking engine), and the readiness check and wait on the published endpoint. - [`amd_smi.rs`](../crates/rocm-core/src/amd_smi.rs) — (ROCMAI-723) locating the `amd-smi` binary: `resolve_amd_smi_binary` (re-exported at the crate root) prefers the newest managed ROCm SDK runtime recorded in the registry, then the home-directory venv and legacy locations, then the bare name for a `PATH` lookup. It only reads the registry records [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) exposes and [`runtime.rs`](../crates/rocm-core/src/runtime.rs)'s data/home directory lookups. As of EAI-8457 it also owns the *launch*: `amd_smi_json` (also re-exported at the crate root) is the one supported way for a synchronous caller to run `amd-smi`, because it is the only place the mandatory `/dev/kfd` pre-flight runs (`KFD_DEVICE`, `preflight_kfd_device`, `kfd_readable`/`kfd_readable_with`, and the thread-bounded, budget-capped `device_readable`/`device_readable_within_budget` probe behind `KFD_OPEN_TIMEOUT`). A device that cannot be opened declines the spawn rather than letting `amd-smi` park in uninterruptible sleep, where no timeout can reach it. The spawn itself still goes through [`process.rs`](../crates/rocm-core/src/process.rs)'s `run_with_timeout`, and the non-zero-exit quote is bounded with that module's `MAX_QUOTED_OUTPUT_CHARS` rather than a second copy of the constant. Callers: [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s `gpu_vram_usage_amd_smi`/`detect_gpu_count`, both through that module's own `amd_smi_probe_json` (which logs the discarded decline at `info`, since the default filter is `warn,rocm=info` and a decline on a healthy host is otherwise indistinguishable from "no GPU"), and `rocmd`'s [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs)`::capture_amd_smi_json`. That the list stays complete is enforced rather than asserted: [`amd_smi_launch_contract.rs`](../xtask/src/amd_smi_launch_contract.rs) is a static contract test failing any file outside its two-entry allowlist that calls `resolve_amd_smi_binary`, so a new probe cannot be spawned directly without a reviewed exemption. The `rocm dash` collectors keep their own pre-flight *decision* (`preflight_passes`), since they gate an async launch, but share this module's bounded device probe through the `pub fn device_readable_bounded` re-export — an allowlisted `rocm-dash-collectors -> rocm-core` edge recorded in [`crate_edges.rs`](../xtask/src/crate_edges.rs), so the one launch site outside this chokepoint cannot drift away from the bound and the budget. - [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) — the shared atomic file writer `rocm` and `rocmd` both use (unique exclusive temp file, then rename, or `ReplaceFileW` on Windows); atomic, not durable, since nothing is fsynced. Other hand-rolled temp-then-rename writers remain outside it, for example [`remote/session.rs`](../apps/rocm/src/remote/session.rs)'s session-file write, [`uninstall.rs`](../apps/rocm/src/uninstall.rs)'s profile replacement, `write_atomically` in [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs), and rocm-core's `write_atomic` in [`lib.rs`](../crates/rocm-core/src/lib.rs), which `RocmCliConfig::save` in [`config.rs`](../crates/rocm-core/src/config.rs), the automation event and proposal stores in [`automation.rs`](../crates/rocm-core/src/automation.rs), the JSONL stores in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and the service-record manifests in [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use; it is `pub` because `rocm-engine-lemonade` also writes through it, in [`install.rs`](../engines/lemonade/src/install.rs) (see the `write_atomic` entry below). #541, which builds on this writer, moves the config save onto it; #618 tracks that move. A new atomic write should call this module rather than add another copy. - [`automation.rs`](../crates/rocm-core/src/automation.rs) — (ROCMAI-151) automation-daemon runtime state and its on-disk event/proposal/audit records, and the policy applied to each store (which are rotated, and the proposal compaction) — see the module's own doc comment for what it owns. The generic JSONL machinery under those stores lives in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs). The dependency runs one way: this module reads `WatcherMode` from [`config.rs`](../crates/rocm-core/src/config.rs), and [`config.rs`](../crates/rocm-core/src/config.rs) imports nothing from here, because the built-in watcher catalog (whose entries carry a default `WatcherMode`, and which `RocmCliConfig::watcher_enabled` and `effective_watcher_mode` take as an argument) lives in [`config.rs`](../crates/rocm-core/src/config.rs) beside `WatcherMode`. diff --git a/docs/manual-testing.md b/docs/manual-testing.md index bdf3308237..9c06588f73 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -313,6 +313,7 @@ Expected result: - The run does not fall back to CPU or Vulkan. - If the GPU cannot be used, the command fails with a clear error. - The OpenAI-compatible endpoint answers a simple chat request. +- `rocm services list` shows an `anthropic endpoint:` URL, and `POST /v1/messages` with `{"model":"x","max_tokens":16,"messages":[{"role":"user","content":"Hello"}]}` returns an Anthropic `message` that names the served model. A request with `"stream":true` or `tools` is refused with an `invalid_request_error`. If the llama.cpp server exits at start with an access violation (`0xC0000005` in `MSVCP140.dll`), the Visual C++ runtime on the machine is probably older than diff --git a/docs/testing.md b/docs/testing.md index 8d645dd46d..ffa0790ae4 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2293,6 +2293,29 @@ serving on the Strix Halo Ubuntu lanes is broken (ROCMAI-898), so the helper che not reached on those lanes. The Strix Halo Windows lane runs it and passes, and the row also matches other Linux lemonade hosts, where a pass is tolerated). +Anthropic-compatible `/v1/messages` endpoint (ROCMAI-597). The translation and +its HTTP front are tested in +[anthropic_proxy.rs](../apps/rocmd/src/anthropic_proxy.rs) against a stand-in +OpenAI engine, including an engine that has its own `/v1/messages` route, which is +still fronted by the proxy so a Claude model name works. The record's adoption of the published endpoint are tested in +[anthropic_endpoint.rs](../crates/rocm-core/src/anthropic_endpoint.rs) and +[service_record.rs](../crates/rocm-core/src/service_record.rs). + +```bash +cargo test -p rocmd --lib anthropic_proxy +cargo test -p rocm-core --lib anthropic_endpoint +cargo test -p rocm-core --lib engine_refresh_adopts_the_published_anthropic_endpoint +cargo test -p rocm --bin rocm anthropic_endpoint +``` + +The user-visible side is `serve-50` (listed beside the OpenAI endpoint), `serve-51` +(a real Messages request is answered by the served model) and `serve-52` +(`services list` reports both endpoints healthy) in +[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature). +They run on the GPU lanes (`@requires-gpu`), with whichever engine the host +serves with. Streaming and tool use are not translated yet, so no scenario +covers them; a request for either is refused, and the unit tests above pin that. + Natural-language plan handoff tests: ```bash diff --git a/tests/e2e-cucumber/expectations.toml b/tests/e2e-cucumber/expectations.toml index 2307516266..465f9f4e2b 100644 --- a/tests/e2e-cucumber/expectations.toml +++ b/tests/e2e-cucumber/expectations.toml @@ -100,6 +100,29 @@ reason = "Lemonade managed serve never exposes the model: observed backend argum serve_timeout_secs = 90 flaky = true +# The Anthropic-endpoint scenarios start from `a model is being served on GPU` +# like the rows above, so they hit the same Linux lemonade serve failure. +[["serve-anthropic-endpoint-advertised"]] +when = { effective_engine = "lemonade", os = "linux" } +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model, so no endpoint is left to advertise." +serve_timeout_secs = 90 +flaky = true + +[["serve-anthropic-messages-inference"]] +when = { effective_engine = "lemonade", os = "linux" } +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model, so the Anthropic request never gets an answer." +serve_timeout_secs = 90 +flaky = true + +[["serve-anthropic-endpoint-readiness"]] +when = { effective_engine = "lemonade", os = "linux" } +bug = "ROCMAI-898" +reason = "Lemonade managed serve never exposes the model, so readiness never covers the Anthropic endpoint." +serve_timeout_secs = 90 +flaky = true + # Explicit Lemonade serving shares the same failure on the Ubuntu lane. [["serve-lemonade-inference"]] when = { effective_engine = "lemonade", os = "linux" } diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 64dd0d9028..2259478525 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -810,3 +810,32 @@ Feature: Model serving And a stopped managed service is recorded on the default serve port When the user restarts the stopped service Then the restart is refused naming the service that holds the port + + # ROCMAI-597: a served model is also reachable from Anthropic-format clients. + # The serving process publishes an Anthropic-compatible `/v1/messages` endpoint + # beside the OpenAI one — the engine's own when it serves one, else a + # translation front on its own port — and `services list` shows it. Engine- + # agnostic like `serve-readiness-contract`: the served engine follows the host. + # Streaming and tool use are not translated yet, so they have no scenario. + @id:serve-anthropic-endpoint-advertised @requires-gpu + Scenario: serve-50 - A served model advertises an Anthropic-compatible endpoint + Given a managed runtime is active + And a model is being served on GPU + When the user lists running services + Then the service details include an Anthropic-compatible messages endpoint + And the service details still include the existing OpenAI-compatible endpoint + + @id:serve-anthropic-messages-inference @requires-gpu + Scenario: serve-51 - A served model responds to an Anthropic Messages API request + Given a managed runtime is active + And a model is being served on GPU + When the user sends a request to the Anthropic messages endpoint + Then the response is a valid Anthropic Messages API response + And the Anthropic response identifies the correct model + + @id:serve-anthropic-endpoint-readiness @requires-gpu + Scenario: serve-52 - Readiness covers the Anthropic-compatible endpoint + Given a managed runtime is active + And a model is being served on GPU + When the CLI reports the service as ready + Then both the OpenAI-compatible and Anthropic-compatible endpoints are reported healthy diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index dee0ee502f..8ec3496528 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -79,6 +79,9 @@ pub struct E2eWorld { pub endpoint: Option, pub model_name: Option, pub chat_response: Option, + /// The reply to a request sent to the served model's Anthropic-compatible + /// `/v1/messages` endpoint. + pub anthropic_response: Option, pub cli_output: Option, pub cli_outputs: Option>, pub cli_stderr: Option, @@ -336,6 +339,7 @@ impl Default for E2eWorld { endpoint: None, model_name: None, chat_response: None, + anthropic_response: None, cli_output: None, cli_outputs: None, cli_stderr: None, diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 3f5182e41d..079f588e7c 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -2161,6 +2161,145 @@ async fn assert_lemonade_recovery_guidance(world: &mut E2eWorld) { ); } +/// The Anthropic endpoint `rocm services list` shows for the first service that +/// has one, waiting for it to appear. +/// +/// The serving process publishes the endpoint a moment after the engine lists its +/// model, so a service the CLI already calls ready may not show it on the first +/// listing. +async fn listed_anthropic_endpoint(world: &mut E2eWorld) -> Option { + let deadline = Instant::now() + Duration::from_secs(30); + loop { + let (stdout, _, _) = crate::run_rocm(world, &["services", "list"]); + let found = stdout.lines().find_map(|line| { + line.trim_start() + .strip_prefix("anthropic endpoint: ") + .map(str::to_owned) + }); + world.cli_output = Some(stdout); + if found.is_some() || Instant::now() >= deadline { + return found; + } + tokio::time::sleep(Duration::from_secs(1)).await; + } +} + +#[then("the service details include an Anthropic-compatible messages endpoint")] +async fn assert_service_lists_anthropic_endpoint(world: &mut E2eWorld) { + let listed = listed_anthropic_endpoint(world).await; + let output = world.cli_output.clone().unwrap_or_default(); + let endpoint = listed + .unwrap_or_else(|| panic!("no `anthropic endpoint:` row in `services list`:\n{output}")); + assert!( + endpoint.starts_with("http://") && !endpoint.contains("/v1"), + "the Anthropic endpoint is a base URL clients append `/v1/messages` to, got `{endpoint}`" + ); +} + +#[then("the service details still include the existing OpenAI-compatible endpoint")] +async fn assert_service_still_lists_openai_endpoint(world: &mut E2eWorld) { + let output = world.cli_output.as_deref().unwrap_or(""); + assert!( + output + .lines() + .any(|line| line.trim_start().starts_with("endpoint: http") && line.ends_with("/v1")), + "the OpenAI-compatible `endpoint:` row is missing from `services list`:\n{output}" + ); +} + +#[when("the user sends a request to the Anthropic messages endpoint")] +async fn user_sends_anthropic_messages_request(world: &mut E2eWorld) { + let base = listed_anthropic_endpoint(world) + .await + .expect("no Anthropic endpoint is listed for the served model"); + let budget = Duration::from_secs(crate::inference_timeout_for(world)); + // Sent under a Claude model name on purpose: an Anthropic client names a + // Claude model, and the endpoint answers for the model that is served. + // The budget leaves room for a reasoning model (Qwen3) to finish thinking + // before it answers: at 32 tokens the whole reply was spent on reasoning + // and the text block came back empty with `stop_reason: max_tokens`. + let body = serde_json::json!({ + "model": "claude-sonnet-4-5", + "max_tokens": 1024, + "messages": [{"role": "user", "content": "Hello"}], + }); + let reply = tokio::spawn(async move { + reqwest::Client::builder() + .timeout(budget) + .build() + .expect("failed to build HTTP client") + .post(format!("{base}/v1/messages")) + .header("anthropic-version", "2023-06-01") + .json(&body) + .send() + .await + .unwrap_or_else(|error| panic!("POST {base}/v1/messages failed: {error}")) + .json::() + .await + .unwrap_or_else(|error| panic!("POST {base}/v1/messages returned non-JSON: {error}")) + }) + .await + .expect("the Anthropic request task panicked"); + world.anthropic_response = Some(reply); +} + +#[then("the response is a valid Anthropic Messages API response")] +async fn assert_valid_anthropic_response(world: &mut E2eWorld) { + let reply = world + .anthropic_response + .as_ref() + .expect("no Anthropic response"); + assert_eq!(reply["type"], "message", "not a Message: {reply}"); + assert_eq!(reply["role"], "assistant", "{reply}"); + assert!( + reply["content"] + .as_array() + .and_then(|blocks| blocks.first()) + .is_some_and(|block| block["type"] == "text" + && block["text"].as_str().is_some_and(|text| !text.is_empty())), + "no text content block in the reply: {reply}" + ); + assert!(reply["stop_reason"].is_string(), "{reply}"); + assert!( + reply["usage"]["input_tokens"].is_u64() && reply["usage"]["output_tokens"].is_u64(), + "usage is missing token counts: {reply}" + ); +} + +#[then("the Anthropic response identifies the correct model")] +async fn assert_anthropic_response_model_correct(world: &mut E2eWorld) { + let reply = world + .anthropic_response + .as_ref() + .expect("no Anthropic response"); + let model = reply["model"].as_str().unwrap_or(""); + let expected = world.model_name.as_deref().unwrap_or(""); + assert!( + model_ids_match(model, expected), + "the Anthropic response model '{model}' does not identify '{expected}'" + ); +} + +#[then("both the OpenAI-compatible and Anthropic-compatible endpoints are reported healthy")] +async fn assert_both_endpoints_healthy(world: &mut E2eWorld) { + // `services list` probes each endpoint it shows and marks one that does not + // answer with `(not responding)`; a healthy service carries neither mark. + let listed = listed_anthropic_endpoint(world).await; + let output = world.cli_output.clone().unwrap_or_default(); + assert!( + listed.is_some(), + "no `anthropic endpoint:` row in `services list`:\n{output}" + ); + assert!( + output.contains("status: ready"), + "the OpenAI-compatible endpoint is not reported ready:\n{output}" + ); + assert!( + !output.contains("(not responding)"), + "the Anthropic-compatible endpoint is reported not responding:\n{output}" + ); +} + #[then("an inference request succeeds immediately")] async fn assert_inference_succeeds_now(world: &mut E2eWorld) { // No extra wait: the CLI already reported ready, so inference must work now. From fa4049db6c8ab128a0fff04926b3266cc9e55cfb Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 14:48:37 +0000 Subject: [PATCH 021/102] ROCMAI-887: make a managed-service recovery work from the embedded and the standalone daemon (#948) * ROCMAI-887: restart a managed service through rocmd when the daemon runs inside rocm `supervise` is a command of the `rocmd` binary only. The recovery restart re-executed the current executable with it, which is right for `rocmd run` but wrong for the daemon that `rocm daemon run` embeds: the current executable there is `rocm`, which answers `supervise` with an unrecognised-subcommand error, so the restarted supervisor died at once and the service stayed failed. The restart now asks a small `DaemonHost` for the binary that has the command, before it touches the record. The embedded entry point, `run_from_args`, marks the process explicitly, so the choice does not depend on what the binary is called (an upper-case or versioned `rocm` is still `rocm`). Embedded, the restart runs `supervise` on the `rocmd` installed beside `rocm` and, when there is none, says what it was looking for and leaves the record exactly as it found it instead of marking it recovering for a supervisor that was never started. Otherwise the current executable is used, as before, so the standalone daemon and the unit tests that stand the test binary in for `rocmd` are unchanged. The choice and the sibling lookup start from one executable path, through a new `rocm_core::sibling_binary_path_from`, so they cannot disagree about it. This makes the restart start the right supervisor. Whether the engine that supervisor then launches starts is a separate change. Tests cover the choice for each executable name, the name and starting path the sibling lookup is given, the marking by `run_from_args` (in a child run of the test binary, since the flag is process-wide), a restart on a host that is `rocm` that runs `supervise` on a stand-in `rocmd` and records its arguments and supervisor, and one with no `rocmd` that changes nothing. Signed-off-by: Jussi Elo * ROCMAI-887: run a managed service's engine through the rocm binary from the rocmd daemon A supervisor started the engine by re-executing its own executable with the hidden `__engine-serve-http` command. That command belongs to the `rocm` binary, and `rocmd` answers it with an unrecognised-subcommand error, so after a recovery restart the supervisor was running but its engine never started: the service log held the clap error, the record kept saying `running` with a dead engine PID, and the daemon's watcher restarted it again and again. It affected the standalone daemon as much as the embedded one, because the restart always starts the supervisor as `rocmd`. `DaemonHost` gains the mirror of the choice made for the restart: `rocm_binary` returns the current executable when the daemon is embedded in `rocm`, and the `rocm` installed beside `rocmd` when it is not, naming what was missing when there is none. `supervise_service` takes the engine's binary from it, before its first write, so a supervisor that cannot find `rocm` leaves the service's record as it was, and the lookup and the choice start from the same executable path as the restart's do. `rocmd` still does not depend on the `rocm` crate; the binary is found on disk. The supervise tests that reach the engine spawn now run on a host whose own executable stands in for `rocm`, as the test binary stood in for it before. Tests cover the choice for each executable name and the name and path the lookup is given, a supervisor run by `rocmd` that starts its engine as `rocm __engine-serve-http ` on a stand-in `rocm`, and one with no `rocm` beside it that changes nothing and says what it looked for. Signed-off-by: Jussi Elo * ROCMAI-887: ask an engine through the rocm binary from the rocmd daemon The daemon asked an engine a question, its readiness check or its stop, by re-executing its own executable with the hidden `__engine-stdio` command. That command belongs to the `rocm` binary, so run as `rocmd` the request failed with an unrecognised-subcommand error every time: the supervisor's readiness wait could never see its engine become ready, and a stop of a container-backed engine, whose confirmation depends on the engine's own answer, could never be confirmed. `engine_request` now runs the command on the binary `DaemonHost::rocm_binary` returns, the same choice the supervisor makes for its engine, so the current executable is used when the daemon is embedded in `rocm` and the `rocm` installed beside `rocmd` otherwise, with an error that names it when there is none. Tests run the request against a stand-in `rocm` that records its arguments, reads the request and answers, and against a `rocmd` with no `rocm` beside it. The script-writing helper and lock the earlier script tests carried now live with the other test support, and those tests use them. Signed-off-by: Jussi Elo * ROCMAI-887: keep the daemon's unit tests off the real rocm and rocmd binaries The supervise tests that stood in for `rocm` ran the supervisor on a thread that went on polling the engine's readiness through the real host, so in a workspace-wide test run with `target/debug/rocm` built, that thread could run the real `rocm __engine-stdio`. The readiness poll now goes through the host the supervisor started its engine on, and `DaemonHost::detect`, the real host, is refused under test: a test passes the code it exercises a host of its own, and anything that reaches the real one fails at once instead of running a binary. The supervise test's stand-in `rocm` now also answers the readiness check and lets its engine exit, so the supervisor finishes by itself and the test joins it, and the restart tests name the test binary as their `rocmd` rather than relying on the real lookup. The stand-in scripts no longer assume `/bin/sleep`: their shell and `sleep` are found on `PATH` once and asserted by name, as is the `setsid` the restart runs. A test that polls through the real host, one that polls through a different host than the engine ran on, and one that lets `detect` run under test each fail. Signed-off-by: Jussi Elo * ROCMAI-887: list the daemon host tests and describe which binary each spawn runs `docs/testing.md` lists the recovery and supervise tests but not the ones for choosing the binary, and nowhere describes that the daemon's spawns run on `rocmd` or `rocm` depending on the command and on where the daemon is running. It now lists the new tests and says which binary the restart, the engine start, the readiness check and the engine stop each run on, and how the unit tests keep off the real binaries. Signed-off-by: Jussi Elo * ROCMAI-887: name the real daemon command in the docs and keep the test helper from being dead code on Windows The docs and three code comments advised `rocm daemon run`, which is not a command: the embedded daemon is `rocm daemon`, which takes only `--status`. The repository's own check that parses every advised command failed on both doc lines, and so did the jobs that run it. The helper that stands a test binary in for `rocmd` is used only by Linux-gated tests, so a Windows build with warnings denied reported it as never used; it now allows dead code on other platforms. Signed-off-by: Jussi Elo * ROCMAI-887: name every module supervise.rs uses in production in its architecture entry The entry said supervise.rs's production code uses only common.rs, engine_ipc.rs and persistence.rs. It also imports DaemonHost from daemon_host.rs, and restart_managed_service_via calls stop_requested from watchers.rs, so both belong in the list. Signed-off-by: Jussi Elo * ROCMAI-887: make a test stand-in host refuse the binary it does not stand in for test_binary_standing_in_for_rocm and test_binary_standing_in_for_rocmd each name only one binary, and asking either for the other would run the real sibling lookup, which finds a rocm or rocmd built beside the test binary and lets a test run it. Nothing reaches that today, but it is the hazard the detect() stand-in was written to remove. The two stand-ins now look for no sibling, so the other binary is an error that never starts a lookup; a test pins both directions. Signed-off-by: Jussi Elo * ROCMAI-887: say what the daemon_host test isolation gives and no more DaemonHost::detect does not refuse to run under test: under test it returns the test binary for both commands and looks for no sibling. The two stand-in hosts name only the binary they stand in for and fail when asked for the other. Reword the daemon_host.rs entry in docs/architecture.md and the paragraph in docs/testing.md to match. Signed-off-by: Jussi Elo * ROCMAI-887: let a test reach the line where detect() takes the embedded flag The production body of DaemonHost::detect was compiled out of every unit test, so an inverted or constant embedded flag, which decides which binary each spawn runs, was caught by nothing. detect() now composes the host the same way in both builds, differing only in how the executable is found and in the test stand-in marker. A test reads the flag through a thread-local override, so the process-wide flag stays untouched for other tests, and pins both answers. The production arm that resolves the executable is still not reachable from a unit test. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit b3b4ddceda4cf21f0acc20eb927a7fcfb97e137d) Signed-off-by: juhovainio --- apps/rocmd/src/cli.rs | 3 + apps/rocmd/src/daemon_host.rs | 423 ++++++++++++++++++++++++ apps/rocmd/src/engine_ipc.rs | 122 ++++++- apps/rocmd/src/lib.rs | 1 + apps/rocmd/src/supervise.rs | 466 ++++++++++++++++++++++++++- apps/rocmd/src/test_support.rs | 63 ++++ crates/rocm-core/src/binary_paths.rs | 43 ++- crates/rocm-core/src/lib.rs | 4 +- docs/architecture.md | 5 +- docs/testing.md | 26 ++ 10 files changed, 1134 insertions(+), 22 deletions(-) create mode 100644 apps/rocmd/src/daemon_host.rs diff --git a/apps/rocmd/src/cli.rs b/apps/rocmd/src/cli.rs index dac75e120c..63d5fa0b66 100644 --- a/apps/rocmd/src/cli.rs +++ b/apps/rocmd/src/cli.rs @@ -166,6 +166,9 @@ pub fn parse_args(args: Vec) -> std::result::Result<(), clap::Error> { } pub fn run_from_args(args: Vec) -> Result<()> { + // `rocm daemon` is the one caller; see `daemon_host` for why the daemon + // has to know it is not running as the `rocmd` binary. + crate::daemon_host::mark_embedded_in_rocm(); let cli = Cli::try_parse_from(args)?; let runtime = tokio::runtime::Builder::new_current_thread() .enable_all() diff --git a/apps/rocmd/src/daemon_host.rs b/apps/rocmd/src/daemon_host.rs new file mode 100644 index 0000000000..9513e6a562 --- /dev/null +++ b/apps/rocmd/src/daemon_host.rs @@ -0,0 +1,423 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Which binary a daemon spawn runs. +//! +//! The daemon runs in one of two hosts: the `rocmd` binary (`rocmd run`), or +//! embedded in `rocm` (`rocm daemon`, which enters through +//! [`crate::run_from_args`]). The two binaries do not answer the same commands: +//! `supervise` belongs to `rocmd` alone, and the hidden engine commands +//! (`__engine-serve-http`, `__engine-stdio`) belong to `rocm` alone. A spawn +//! that re-executes the current executable is right only in the host that has +//! the command, so a spawn asks this module for the binary instead of reading +//! `current_exe` itself. + +use anyhow::{Context, Result}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; + +const ROCMD_BINARY: &str = "rocmd"; +const ROCM_BINARY: &str = "rocm"; + +/// Set once by the embedded entry point. An explicit signal rather than a guess +/// from the executable's name, which a renamed or upper-cased binary defeats. +static EMBEDDED_IN_ROCM: AtomicBool = AtomicBool::new(false); + +/// Record that this process is `rocm` running the daemon, for the rest of its life. +pub(crate) fn mark_embedded_in_rocm() { + EMBEDDED_IN_ROCM.store(true, Ordering::Relaxed); +} + +fn is_embedded_in_rocm() -> bool { + #[cfg(test)] + { + if let Some(flag) = EMBEDDED_OVERRIDE.with(std::cell::Cell::get) { + return flag; + } + } + EMBEDDED_IN_ROCM.load(Ordering::Relaxed) +} + +// Unit tests only: the answer `is_embedded_in_rocm` gives on this thread. The +// real flag is process-wide, so a test that set it would change the answer for +// every other test; a thread-local lets a test see both answers and no other +// test see either. +#[cfg(test)] +thread_local! { + static EMBEDDED_OVERRIDE: std::cell::Cell> = const { std::cell::Cell::new(None) }; +} + +/// The host the daemon is running in, with the one executable path that every +/// decision and every sibling lookup is made from, so they cannot disagree. +pub(crate) struct DaemonHost { + current_exe: PathBuf, + embedded_in_rocm: bool, + /// Unit tests only: the test binary has every command, so a spawn runs it + /// and no sibling is ever looked for. + #[cfg(test)] + test_binary_has_every_command: bool, + /// Unit tests only: a host that stands in for one binary looks for no + /// sibling, so asking it for the other is an error and not a lookup that + /// could find, and run, a build of the other binary beside the test binary. + #[cfg(test)] + test_looks_for_no_sibling: bool, +} + +impl DaemonHost { + /// The host this process is running in. + /// + /// Under test, the process is the test binary, which stands in for both + /// `rocm` and `rocmd` and looks for neither: code that reaches this runs + /// the test binary and nothing else, not even a `rocm` that happens to sit + /// in the build directory. A test that needs another binary passes the code + /// it exercises a host of its own. + pub(crate) fn detect() -> Result { + #[cfg(test)] + let current_exe = std::env::current_exe().context("the test binary has a path")?; + #[cfg(not(test))] + let current_exe = rocm_core::current_executable_path() + .context("failed to resolve the current executable path")?; + // The same composition in both builds, so a test reaches the line that + // takes the host's embedded flag from the process. + let host = Self::new(current_exe, is_embedded_in_rocm()); + #[cfg(test)] + let host = Self { + test_binary_has_every_command: true, + ..host + }; + Ok(host) + } + + pub(crate) const fn new(current_exe: PathBuf, embedded_in_rocm: bool) -> Self { + Self { + current_exe, + embedded_in_rocm, + #[cfg(test)] + test_binary_has_every_command: false, + #[cfg(test)] + test_looks_for_no_sibling: false, + } + } + + /// The binary that has the `supervise` command: this one when the daemon is + /// `rocmd`, the `rocmd` installed beside `rocm` when it is embedded there. + pub(crate) fn rocmd_binary(&self) -> Result { + self.rocmd_binary_via(rocm_core::sibling_binary_path_from) + } + + /// [`Self::rocmd_binary`] with the sibling lookup passed in, given the + /// executable and the binary's name. + fn rocmd_binary_via( + &self, + sibling: impl FnOnce(&Path, &str) -> Result, + ) -> Result { + #[cfg(test)] + { + if self.test_binary_has_every_command { + return Ok(self.current_exe.clone()); + } + } + if !self.embedded_in_rocm { + return Ok(self.current_exe.clone()); + } + #[cfg(test)] + { + self.refuse_sibling_lookup(ROCMD_BINARY)?; + } + sibling(&self.current_exe, ROCMD_BINARY).context( + "the daemon runs inside `rocm`, which has no `supervise` command, \ + and recovery restarts a service through the `rocmd` binary installed next to it", + ) + } + + /// The binary that has the hidden engine commands: this one when the daemon + /// is embedded in `rocm`, the `rocm` installed beside `rocmd` when it is not. + pub(crate) fn rocm_binary(&self) -> Result { + self.rocm_binary_via(rocm_core::sibling_binary_path_from) + } + + /// [`Self::rocm_binary`] with the sibling lookup passed in. + fn rocm_binary_via( + &self, + sibling: impl FnOnce(&Path, &str) -> Result, + ) -> Result { + #[cfg(test)] + { + if self.test_binary_has_every_command { + return Ok(self.current_exe.clone()); + } + } + if self.embedded_in_rocm { + return Ok(self.current_exe.clone()); + } + #[cfg(test)] + { + self.refuse_sibling_lookup(ROCM_BINARY)?; + } + sibling(&self.current_exe, ROCM_BINARY).context( + "the `rocmd` binary has no engine commands, and a managed service's engine \ + runs through the `rocm` binary installed next to it", + ) + } + + /// The error a test stand-in answers with when asked for the binary it does + /// not stand in for. + #[cfg(test)] + fn refuse_sibling_lookup(&self, binary: &str) -> Result<()> { + if self.test_looks_for_no_sibling { + anyhow::bail!( + "this test host stands in for one binary and looks for no sibling, \ + so it cannot name `{binary}`" + ); + } + Ok(()) + } + + /// A host whose own executable is taken to have every command, so a test + /// that reaches a spawn runs the test binary, as each one did before the + /// binary was chosen here. + #[cfg(test)] + pub(crate) fn test_binary_standing_in_for_rocm() -> Self { + let mut host = Self::new( + std::env::current_exe().expect("the test binary has a path"), + true, + ); + host.test_looks_for_no_sibling = true; + host + } + + /// The same for a test that restarts a service: the test binary is the + /// `rocmd` that is re-executed. + #[cfg(test)] + #[cfg_attr(not(target_os = "linux"), allow(dead_code))] + pub(crate) fn test_binary_standing_in_for_rocmd() -> Self { + let mut host = Self::new( + std::env::current_exe().expect("the test binary has a path"), + false, + ); + host.test_looks_for_no_sibling = true; + host + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::cell::RefCell; + + /// The names a binary can carry on disk. The choice is made from the + /// embedded signal, so none of them may change it. + const ROCM_NAMES: [&str; 4] = ["/opt/rocm-cli/bin/rocm", "rocm.exe", "ROCM.EXE", "rocm-0.5"]; + const ROCMD_NAMES: [&str; 4] = [ + "/opt/rocm-cli/bin/rocmd", + "rocmd.exe", + "ROCMD.EXE", + "/repo/target/debug/deps/rocmd-3f9a41c2d0e7b655", + ]; + + #[test] + fn embedded_in_rocm_runs_supervise_on_the_sibling_rocmd_whatever_rocm_is_called() { + for exe in ROCM_NAMES { + let asked = RefCell::new(None); + let chosen = DaemonHost::new(PathBuf::from(exe), true) + .rocmd_binary_via(|from, name| { + *asked.borrow_mut() = Some((from.to_path_buf(), name.to_owned())); + Ok(PathBuf::from("/opt/rocm-cli/bin/rocmd")) + }) + .unwrap(); + assert_eq!(chosen, PathBuf::from("/opt/rocm-cli/bin/rocmd"), "{exe}"); + // The lookup starts from the same executable the choice was made + // from, and asks for `rocmd`. + assert_eq!( + asked.into_inner(), + Some((PathBuf::from(exe), "rocmd".to_owned())), + "{exe}" + ); + } + } + + #[test] + fn the_rocmd_daemon_keeps_its_own_executable_whatever_it_is_called() { + for exe in ROCMD_NAMES.into_iter().chain(["rocm"]) { + let chosen = DaemonHost::new(PathBuf::from(exe), false) + .rocmd_binary_via(|_, _| { + panic!("{exe}: not embedded in rocm, so it has `supervise` itself") + }) + .unwrap(); + assert_eq!(chosen, PathBuf::from(exe)); + } + } + + #[test] + fn a_missing_rocmd_beside_rocm_is_named() { + let error = DaemonHost::new(PathBuf::from("/opt/rocm-cli/bin/rocm"), true) + .rocmd_binary_via(|_, _| { + anyhow::bail!("unable to locate sibling binary rocmd; checked /opt/rocm-cli/bin") + }) + .unwrap_err(); + let text = format!("{error:#}"); + assert!(text.contains("no `supervise` command"), "{text}"); + assert!( + text.contains("unable to locate sibling binary rocmd"), + "{text}" + ); + } + + /// The hidden engine commands are `rocm`'s: a daemon that is `rocmd`, under + /// any name, runs them on the `rocm` beside it, and says what it looked for. + #[test] + fn the_rocmd_daemon_runs_engine_commands_on_the_sibling_rocm() { + for exe in ROCMD_NAMES.into_iter().chain(["rocm"]) { + let asked = RefCell::new(None); + let chosen = DaemonHost::new(PathBuf::from(exe), false) + .rocm_binary_via(|from, name| { + *asked.borrow_mut() = Some((from.to_path_buf(), name.to_owned())); + Ok(PathBuf::from("/opt/rocm-cli/bin/rocm")) + }) + .unwrap(); + assert_eq!(chosen, PathBuf::from("/opt/rocm-cli/bin/rocm"), "{exe}"); + assert_eq!( + asked.into_inner(), + Some((PathBuf::from(exe), "rocm".to_owned())), + "{exe}" + ); + } + } + + #[test] + fn embedded_in_rocm_keeps_its_own_executable_for_engine_commands() { + for exe in ROCM_NAMES { + let chosen = DaemonHost::new(PathBuf::from(exe), true) + .rocm_binary_via(|_, _| { + panic!("{exe}: embedded in rocm, so it has the engine commands itself") + }) + .unwrap(); + assert_eq!(chosen, PathBuf::from(exe)); + } + } + + #[test] + fn a_missing_rocm_beside_rocmd_is_named() { + let error = DaemonHost::new(PathBuf::from("/opt/rocm-cli/bin/rocmd"), false) + .rocm_binary_via(|_, _| { + anyhow::bail!("unable to locate sibling binary rocm; checked /opt/rocm-cli/bin") + }) + .unwrap_err(); + let text = format!("{error:#}"); + assert!(text.contains("no engine commands"), "{text}"); + assert!( + text.contains("unable to locate sibling binary rocm"), + "{text}" + ); + } + + /// The real lookup, from the host's own executable: the `rocmd` next to a + /// `rocm` that is not the test binary. + #[test] + fn the_real_sibling_lookup_starts_from_the_hosts_executable() { + let dir = crate::test_support::unique_test_root("daemon-host-sibling-lookup"); + let rocm = dir.join(rocm_core::platform_binary_name("rocm")); + let rocmd = dir.join(rocm_core::platform_binary_name("rocmd")); + std::fs::write(&rocmd, b"").expect("write rocmd"); + + let chosen = DaemonHost::new(rocm, true).rocmd_binary(); + std::fs::remove_dir_all(&dir).ok(); + assert_eq!(chosen.unwrap(), rocmd); + } + + /// A stand-in host names only the binary it stands in for. Asked for the + /// other it fails, and it never starts the real sibling lookup, which would + /// find the other binary in the build directory and let a test run it. + #[test] + fn a_stand_in_host_looks_for_no_sibling() { + let this = std::env::current_exe().unwrap(); + let rocm = DaemonHost::test_binary_standing_in_for_rocm(); + assert_eq!(rocm.rocm_binary().unwrap(), this); + let error = rocm + .rocmd_binary_via(|_, _| panic!("a stand-in host must not look for a sibling")) + .unwrap_err(); + assert!( + format!("{error:#}").contains("looks for no sibling"), + "{error:#}" + ); + assert!(rocm.rocmd_binary().is_err()); + + let rocmd = DaemonHost::test_binary_standing_in_for_rocmd(); + assert_eq!(rocmd.rocmd_binary().unwrap(), this); + let error = rocmd + .rocm_binary_via(|_, _| panic!("a stand-in host must not look for a sibling")) + .unwrap_err(); + assert!( + format!("{error:#}").contains("looks for no sibling"), + "{error:#}" + ); + assert!(rocmd.rocm_binary().is_err()); + } + + /// `detect` takes the embedded flag from the process, and not its negation + /// or a constant: with the flag set on this thread the host it returns is + /// the embedded one, and with it clear it is not. The two answers are the + /// whole premise of the fix, since they decide which binary each spawn runs. + /// What a unit test still cannot reach is the production arm that resolves + /// the executable (`rocm_core::current_executable_path`). + #[test] + fn detect_takes_the_embedded_flag_from_the_process() { + let this = std::env::current_exe().unwrap(); + for flag in [false, true] { + EMBEDDED_OVERRIDE.with(|cell| cell.set(Some(flag))); + let host = DaemonHost::detect(); + EMBEDDED_OVERRIDE.with(|cell| cell.set(None)); + let host = host.unwrap(); + assert_eq!(host.embedded_in_rocm, flag, "embedded flag {flag}"); + // The stand-in still answers with the test binary for both + // commands whatever the flag says. + assert_eq!(host.rocmd_binary().unwrap(), this, "embedded flag {flag}"); + assert_eq!(host.rocm_binary().unwrap(), this, "embedded flag {flag}"); + } + assert!(!is_embedded_in_rocm(), "the override is cleared again"); + } + + /// A process that did not enter through `run_from_args` is not embedded. + #[test] + fn a_plain_process_is_not_embedded_in_rocm() { + assert!(!is_embedded_in_rocm()); + } + + /// The real host is not reachable from a unit test: under test, `detect` is + /// the test binary for both commands, and nothing beside it is looked for. + #[test] + fn under_test_detect_is_the_test_binary_and_looks_for_no_sibling() { + let host = DaemonHost::detect().unwrap(); + let this = std::env::current_exe().unwrap(); + assert_eq!(host.rocmd_binary().unwrap(), this); + assert_eq!(host.rocm_binary().unwrap(), this); + } + + /// `rocm daemon` enters through `run_from_args`, and that is what marks + /// the process. The flag is process-wide, so it is checked in a child run of + /// this test binary instead of being set here, where it would change the + /// answer for every other test. + #[test] + fn run_from_args_marks_the_process_as_embedded_in_rocm() { + const CHILD: &str = "ROCMD_TEST_RUN_FROM_ARGS_CHILD"; + let test_name = "daemon_host::tests::run_from_args_marks_the_process_as_embedded_in_rocm"; + if std::env::var_os(CHILD).is_some() { + // `--help` stops clap before anything runs; the marking comes first. + let _ = crate::run_from_args(vec!["rocmd".into(), "--help".into()]); + assert!(is_embedded_in_rocm()); + return; + } + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .args(["--exact", test_name, "--nocapture", "--test-threads=1"]) + .env(CHILD, "1") + .output() + .expect("run the child test"); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!( + output.status.success() && stdout.contains(&format!("test {test_name} ... ok")), + "child run failed:\n{stdout}\n{}", + String::from_utf8_lossy(&output.stderr) + ); + } +} diff --git a/apps/rocmd/src/engine_ipc.rs b/apps/rocmd/src/engine_ipc.rs index 32f1834547..b8ccefc354 100644 --- a/apps/rocmd/src/engine_ipc.rs +++ b/apps/rocmd/src/engine_ipc.rs @@ -2,6 +2,7 @@ // // SPDX-License-Identifier: MIT +use crate::daemon_host::DaemonHost; use anyhow::{Context, Result, bail}; use rocm_core::AppPaths; use rocm_engine_protocol::{ @@ -18,7 +19,19 @@ pub(crate) fn engine_healthcheck_response( engine: &str, service_id: &str, ) -> Result { - engine_request::<_, HealthcheckResponse>( + engine_healthcheck_response_on(&DaemonHost::detect()?, paths, engine, service_id) +} + +/// [`engine_healthcheck_response`] on a given host. The supervisor polls +/// through the host it started its engine on, so a test can supply both. +pub(crate) fn engine_healthcheck_response_on( + daemon_host: &DaemonHost, + paths: &AppPaths, + engine: &str, + service_id: &str, +) -> Result { + engine_request_on::<_, HealthcheckResponse>( + daemon_host, paths, engine, service_id, @@ -110,6 +123,31 @@ pub(crate) fn engine_request( method: EngineMethod, request: &T, ) -> Result +where + T: Serialize, + R: DeserializeOwned, +{ + engine_request_on( + &DaemonHost::detect()?, + paths, + engine, + service_id, + method, + request, + ) +} + +/// [`engine_request`] on a given host, so a test can supply the binary that +/// answers. The stdio command is `rocm`'s alone, so the daemon runs it on the +/// `rocm` binary, not on whatever it is itself. +fn engine_request_on( + daemon_host: &DaemonHost, + paths: &AppPaths, + engine: &str, + service_id: &str, + method: EngineMethod, + request: &T, +) -> Result where T: Serialize, R: DeserializeOwned, @@ -118,8 +156,7 @@ where method, payload: serde_json::to_value(request).context("failed to serialize engine request")?, }; - let engine_binary = - std::env::current_exe().context("failed to resolve current rocm executable path")?; + let engine_binary = daemon_host.rocm_binary()?; let mut command = ProcessCommand::new(&engine_binary); command .arg("__engine-stdio") @@ -341,4 +378,83 @@ mod tests { }; assert!(!healthcheck_response_ready(&listing_only)); } + + /// The stdio command is `rocm`'s alone: a daemon run as `rocmd` asks the + /// engine through the `rocm` beside it, which `rocmd` answered with an + /// unrecognised-subcommand error, so its readiness checks and its engine + /// stops could never succeed. + /// + /// The stand-in `rocm` records its arguments, reads the request line and + /// answers with an `ok` envelope. + #[cfg(target_os = "linux")] + #[test] + fn engine_request_runs_the_stdio_command_on_the_rocm_beside_rocmd() -> Result<()> { + use crate::test_support::{lock_script_tests, script_tools, write_executable_script}; + + let _serial = lock_script_tests(); + let tools = script_tools(); + let (root, paths) = temp_app_paths("engine-request-on-rocm"); + let seen = root.join("rocm-args.txt"); + write_executable_script( + &root.join("rocm"), + &tools, + &format!( + "printf '%s\\n' \"$@\" > '{}'\nread -r _request\n{}\n", + seen.display(), + r#"printf '%s\n' '{"ok":true,"data":{"status":"ready"}}'"# + ), + )?; + let daemon_host = DaemonHost::new(root.join("rocmd"), false); + + // The first exec of a new file can lose a race with an unrelated fork + // (ETXTBSY); the spawn then fails at once, and one more try is enough. + let mut outcome = Err(anyhow::anyhow!("not attempted")); + for _ in 0..10 { + outcome = engine_request_on::<_, serde_json::Value>( + &daemon_host, + &paths, + "vllm", + "svc-engine-request", + EngineMethod::Healthcheck, + &serde_json::json!({ "service_id": "svc-engine-request" }), + ); + match &outcome { + Err(error) if format!("{error:#}").contains("Text file busy") => { + std::thread::sleep(std::time::Duration::from_millis(20)); + } + _ => break, + } + } + let args = fs::read_to_string(&seen).unwrap_or_default(); + fs::remove_dir_all(&root).ok(); + + assert_eq!( + args.lines().collect::>(), + ["__engine-stdio", "vllm"], + "the engine must be asked through `rocm __engine-stdio `" + ); + assert_eq!(outcome?, serde_json::json!({ "status": "ready" })); + Ok(()) + } + + /// With no `rocm` beside `rocmd` the request fails, and says what it looked + /// for, instead of running a command `rocmd` does not have. + #[test] + fn engine_request_without_a_rocm_beside_rocmd_names_it() { + let (root, paths) = temp_app_paths("engine-request-no-rocm"); + let error = engine_request_on::<_, serde_json::Value>( + &DaemonHost::new(root.join("rocmd"), false), + &paths, + "vllm", + "svc-engine-request", + EngineMethod::Healthcheck, + &serde_json::json!({ "service_id": "svc-engine-request" }), + ) + .expect_err("no rocm to ask"); + fs::remove_dir_all(root).ok(); + + let text = format!("{error:#}"); + assert!(text.contains("no engine commands"), "{text}"); + assert!(text.contains("unable to locate sibling binary"), "{text}"); + } } diff --git a/apps/rocmd/src/lib.rs b/apps/rocmd/src/lib.rs index 2a478a8b0f..4fd74a7df3 100644 --- a/apps/rocmd/src/lib.rs +++ b/apps/rocmd/src/lib.rs @@ -8,6 +8,7 @@ pub mod anthropic_proxy; mod cli; mod common; mod daemon; +mod daemon_host; mod engine_ipc; mod host_snapshot; mod mcp; diff --git a/apps/rocmd/src/supervise.rs b/apps/rocmd/src/supervise.rs index eb594e2167..409895a844 100644 --- a/apps/rocmd/src/supervise.rs +++ b/apps/rocmd/src/supervise.rs @@ -2,6 +2,7 @@ // // SPDX-License-Identifier: MIT +use crate::daemon_host::DaemonHost; use crate::persistence::load_service_record; use crate::watchers::stop_requested; use anyhow::{Context, Result}; @@ -36,6 +37,41 @@ pub(crate) fn supervise_service( device_policy: String, gpu: Option, engine_recipe_json: Option, +) -> Result<()> { + supervise_service_on( + &DaemonHost::detect()?, + paths, + service_id, + engine, + model_ref, + canonical_model_id, + runtime_id, + env_id, + host, + port, + device_policy, + gpu, + engine_recipe_json, + ) +} + +/// [`supervise_service`] on a given host, so a test can supply the binary that +/// runs the engine. +#[allow(clippy::too_many_arguments)] +fn supervise_service_on( + daemon_host: &DaemonHost, + paths: &AppPaths, + service_id: String, + engine: String, + model_ref: String, + canonical_model_id: String, + runtime_id: Option, + env_id: Option, + host: String, + port: u16, + device_policy: String, + gpu: Option, + engine_recipe_json: Option, ) -> Result<()> { paths.ensure()?; fs::create_dir_all(paths.engine_logs_dir(&engine))?; @@ -108,6 +144,10 @@ pub(crate) fn supervise_service( .is_some(), record.requires_api_key, )?; + // The engine runs through the hidden commands of the `rocm` binary, which + // `rocmd` does not have. Found before the first write, like the guard above, + // so a supervisor that cannot find it leaves the record as it was. + let rocm_binary = daemon_host.rocm_binary()?; record.write()?; let log_file = fs::File::create(&record.log_path) @@ -116,8 +156,6 @@ pub(crate) fn supervise_service( .try_clone() .context("failed to clone service log file handle")?; - let rocm_binary = - std::env::current_exe().context("failed to resolve current rocm executable path")?; let mut command = ProcessCommand::new(rocm_binary); command .args(engine_serve_http_args( @@ -169,6 +207,7 @@ pub(crate) fn supervise_service( })?; let became_ready = wait_for_service_ready( + daemon_host, paths, &record.engine, &record.service_id, @@ -354,9 +393,14 @@ fn read_new_log_phase(log_path: &Path, pos: &mut u64) -> Option<&'static str> { phase } -fn engine_healthcheck_ready(paths: &AppPaths, engine: &str, service_id: &str) -> Result { +fn engine_healthcheck_ready( + daemon_host: &DaemonHost, + paths: &AppPaths, + engine: &str, + service_id: &str, +) -> Result { Ok(crate::engine_ipc::healthcheck_response_ready( - &crate::engine_ipc::engine_healthcheck_response(paths, engine, service_id)?, + &crate::engine_ipc::engine_healthcheck_response_on(daemon_host, paths, engine, service_id)?, )) } @@ -364,6 +408,7 @@ fn engine_healthcheck_ready(paths: &AppPaths, engine: &str, service_id: &str) -> /// timeout elapses), tailing its log file meanwhile and reporting each coarse /// startup phase transition via `on_phase`. fn wait_for_service_ready( + daemon_host: &DaemonHost, paths: &AppPaths, engine: &str, service_id: &str, @@ -381,7 +426,7 @@ fn wait_for_service_ready( last_phase = Some(phase); on_phase(phase); } - if engine_healthcheck_ready(paths, engine, service_id).unwrap_or(false) { + if engine_healthcheck_ready(daemon_host, paths, engine, service_id).unwrap_or(false) { return true; } thread::sleep(Duration::from_millis(200)); @@ -436,6 +481,44 @@ fn restart_managed_service_bounded( spawn_override: Option<&SpawnOverride<'_>>, lock_timeout: Duration, ) -> Result<()> { + restart_managed_service_via( + &DaemonHost::detect()?, + paths, + record, + spawn_override, + lock_timeout, + ) +} + +/// [`restart_managed_service`] on a given host, so a test can be `rocm`. +#[cfg(test)] +fn restart_managed_service_on( + paths: &AppPaths, + record: &mut ManagedServiceRecord, + host: &DaemonHost, +) -> Result<()> { + restart_managed_service_via( + host, + paths, + record, + None, + rocm_core::MANAGED_LAUNCH_LOCK_TIMEOUT, + ) +} + +/// The restart itself, on a given host. +/// +/// The binary that runs `supervise` is chosen first, before the launch lock is +/// taken and before anything is read or written: a restart that cannot find it +/// leaves every record as it was. +fn restart_managed_service_via( + host: &DaemonHost, + paths: &AppPaths, + record: &mut ManagedServiceRecord, + spawn_override: Option<&SpawnOverride<'_>>, + lock_timeout: Duration, +) -> Result<()> { + let rocmd_binary = host.rocmd_binary()?; let _launch_lock = rocm_core::FileLock::acquire_bounded(paths.managed_launch_lock_path(), lock_timeout)?; @@ -519,8 +602,6 @@ fn restart_managed_service_bounded( None }; - let rocmd_binary = - std::env::current_exe().context("failed to resolve current rocmd executable path")?; let log_file = fs::OpenOptions::new() .create(true) .append(true) @@ -838,9 +919,14 @@ fn detached_rocmd_command(rocmd_binary: &std::path::Path) -> ProcessCommand { #[cfg(test)] mod tests { use super::*; + use crate::daemon_host::DaemonHost; use crate::persistence::load_service_record; #[cfg(target_os = "linux")] use crate::test_support::UNCONFIRMED_STOP_PID; + #[cfg(target_os = "linux")] + use crate::test_support::{ + find_on_path, lock_script_tests, script_tools, write_executable_script, + }; use crate::test_support::{ free_loopback_port, identity_probe_record, temp_app_paths, unique_test_root, }; @@ -870,7 +956,8 @@ mod tests { let service_id = service_id.to_owned(); let (sender, receiver) = std::sync::mpsc::channel(); std::thread::spawn(move || { - let outcome = supervise_service( + let outcome = supervise_service_on( + &DaemonHost::test_binary_standing_in_for_rocm(), &paths, service_id, "llamacpp".to_owned(), @@ -937,7 +1024,8 @@ mod tests { let owned_id = service_id.to_owned(); let (sender, receiver) = std::sync::mpsc::channel(); std::thread::spawn(move || { - let outcome = supervise_service( + let outcome = supervise_service_on( + &DaemonHost::test_binary_standing_in_for_rocm(), &owned_paths, owned_id, "llamacpp".to_owned(), @@ -1238,7 +1326,8 @@ mod tests { // A directory where the supervisor expects to create a file. fs::create_dir_all(&log_path)?; - let outcome = supervise_service( + let outcome = supervise_service_on( + &DaemonHost::test_binary_standing_in_for_rocm(), &paths, service_id.to_owned(), "llamacpp".to_owned(), @@ -1358,7 +1447,8 @@ mod tests { // `supervise_service_persists_the_supervisor_identity_token`. fs::create_dir_all(&seeded.log_path)?; - let outcome = supervise_service( + let outcome = supervise_service_on( + &DaemonHost::test_binary_standing_in_for_rocm(), &paths, service_id.to_owned(), "llamacpp".to_owned(), @@ -1436,7 +1526,11 @@ mod tests { "precondition: the seeded record carries no token" ); - let _ = restart_managed_service(&paths, &mut record); + let _ = restart_managed_service_on( + &paths, + &mut record, + &DaemonHost::test_binary_standing_in_for_rocmd(), + ); let persisted = load_service_record(&paths, service_id); fs::remove_dir_all(root).ok(); @@ -1471,7 +1565,11 @@ mod tests { record.status = "failed".to_owned(); record.write()?; - let _ = restart_managed_service(&paths, &mut record); + let _ = restart_managed_service_on( + &paths, + &mut record, + &DaemonHost::test_binary_standing_in_for_rocmd(), + ); let persisted = load_service_record(&paths, service_id); fs::remove_dir_all(root).ok(); @@ -1530,7 +1628,11 @@ mod tests { Ok(false) }) }; - let outcome = restart_managed_service(&paths, &mut record); + let outcome = restart_managed_service_on( + &paths, + &mut record, + &DaemonHost::test_binary_standing_in_for_rocmd(), + ); let stop_landed = stop.join().expect("stop thread"); let persisted = load_service_record(&paths, service_id); fs::remove_dir_all(root).ok(); @@ -1550,6 +1652,262 @@ mod tests { Ok(()) } + /// The restart, when the daemon is embedded in `rocm`, launches `supervise` + /// on the `rocmd` it is given. Before the choice was made, it re-executed + /// `rocm` itself and the new supervisor died on an unrecognised subcommand. + /// + /// The stand-in is a script that records its arguments, so what the restart + /// actually ran can be read back. Its shell and `sleep` are found once and + /// asserted by name, and so is `setsid`, which the restart itself runs. + #[cfg(target_os = "linux")] + #[test] + fn restart_runs_supervise_on_the_rocmd_it_is_given() -> Result<()> { + let _serial = lock_script_tests(); + let tools = script_tools(); + assert!( + find_on_path("setsid").is_some(), + "this test needs `setsid` on PATH: the restart runs it" + ); + let (root, paths) = temp_app_paths("restart-runs-given-rocmd"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + let seen = root.join("rocmd-args.txt"); + // The daemon is `rocm`, which does not exist here; the `rocmd` beside it + // is the stand-in. + let host = DaemonHost::new(root.join("rocm"), true); + let stand_in = root.join("rocmd"); + write_executable_script( + &stand_in, + &tools, + &format!( + "printf '%s\\n' \"$@\" > '{}'\nexec '{}' 3\n", + seen.display(), + tools.sleep.display() + ), + )?; + + let service_id = "svc-restart-given-rocmd"; + let mut record = identity_probe_record(&paths, service_id, 11470); + record.status = "failed".to_owned(); + record.write()?; + + // The first exec of a new file can still lose a race with an unrelated + // fork (ETXTBSY): setsid then exits at once and nothing is recorded. + let mut outcome = Err(anyhow::anyhow!("not attempted")); + for _ in 0..10 { + outcome = restart_managed_service_on(&paths, &mut record, &host); + let deadline = std::time::Instant::now() + Duration::from_secs(2); + while !seen.exists() && std::time::Instant::now() < deadline { + thread::sleep(Duration::from_millis(20)); + } + if seen.exists() { + break; + } + } + let args = fs::read_to_string(&seen).unwrap_or_default(); + let persisted = load_service_record(&paths, service_id); + fs::remove_dir_all(&root).ok(); + + outcome?; + let args: Vec<&str> = args.lines().collect(); + assert_eq!( + args.get(..2), + Some(&["supervise", service_id][..]), + "the rocmd it is given must be started as `rocmd supervise `: {args:?}" + ); + let persisted = persisted?; + assert_eq!(persisted.status, "recovering"); + assert_ne!( + persisted.supervisor_pid, 0, + "the spawned supervisor's PID is recorded" + ); + Ok(()) + } + + /// With no `rocmd` to run, the restart reports it, and the record is exactly + /// as the restart found it: it is not marked `recovering` for a supervisor + /// that was never started. + #[test] + fn restart_without_a_rocmd_to_run_changes_nothing() -> Result<()> { + let (root, paths) = temp_app_paths("restart-no-rocmd"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + + let service_id = "svc-restart-no-rocmd"; + let mut record = identity_probe_record(&paths, service_id, 11471); + record.status = "failed".to_owned(); + record.write()?; + let before = load_service_record(&paths, service_id)?; + + let host = DaemonHost::new(root.join("rocm"), true); + let outcome = restart_managed_service_on(&paths, &mut record, &host); + let after = load_service_record(&paths, service_id); + fs::remove_dir_all(root).ok(); + + let error = format!( + "{:#}", + outcome.expect_err("no rocmd to run: the restart must fail") + ); + assert!(error.contains("no `supervise` command"), "{error}"); + assert!(error.contains("unable to locate sibling binary"), "{error}"); + let after = after?; + assert_eq!(after.status, "failed"); + assert_eq!(after.supervisor_pid, before.supervisor_pid); + assert_eq!(after.restart_count, before.restart_count); + Ok(()) + } + + /// A supervisor run by the `rocmd` binary starts its engine on the `rocm` + /// beside it: `__engine-serve-http` is a hidden command of `rocm` alone, and + /// `rocmd` answered it with an unrecognised-subcommand error. It polls the + /// engine's readiness through the same `rocm`. + /// + /// The stand-in `rocm` is the only binary involved: it records the engine + /// start, answers the readiness check, and lets the engine exit, so the + /// supervisor finishes by itself and nothing is left running. The test host + /// is the only host the supervisor has, so no other `rocm` can be reached. + #[cfg(target_os = "linux")] + #[test] + fn supervise_runs_the_engine_on_the_rocm_beside_rocmd() -> Result<()> { + let _serial = lock_script_tests(); + let tools = script_tools(); + let (root, paths) = temp_app_paths("supervise-engine-on-rocm"); + paths.ensure()?; + let served = root.join("rocm-serve-args.txt"); + let asked = root.join("rocm-stdio-args.txt"); + let ready = serde_json::to_string(&rocm_engine_protocol::EngineResponseEnvelope::success( + rocm_engine_protocol::HealthcheckResponse { + status: "ready".to_owned(), + model_loaded: true, + device: "cpu".to_owned(), + uptime_sec: 1, + queue_depth: 0, + last_error: None, + tokens_per_sec: None, + }, + ))?; + write_executable_script( + &root.join("rocm"), + &tools, + &format!( + "case \"$1\" in\n\ + __engine-serve-http)\n printf '%s\\n' \"$@\" > '{served}'\n exec '{sleep}' 2\n ;;\n\ + __engine-stdio)\n printf '%s\\n' \"$@\" > '{asked}'\n read -r _request\n printf '%s\\n' '{ready}'\n ;;\n\ + *) exit 2 ;;\nesac\n", + served = served.display(), + asked = asked.display(), + sleep = tools.sleep.display(), + ), + )?; + + let service_id = "svc-supervise-engine-on-rocm"; + let mut outcome = Err("not attempted".to_owned()); + for _ in 0..10 { + let (sender, receiver) = std::sync::mpsc::channel(); + // The daemon is `rocmd`, which does not exist here; the `rocm` beside + // it is the stand-in. + let (paths, daemon_host) = (paths.clone(), DaemonHost::new(root.join("rocmd"), false)); + thread::spawn(move || { + let outcome = supervise_service_on( + &daemon_host, + &paths, + service_id.to_owned(), + "llamacpp".to_owned(), + "a-model".to_owned(), + "a-model".to_owned(), + None, + None, + "127.0.0.1".to_owned(), + 11472, + "gpu_required".to_owned(), + None, + None, + ); + let _ = sender.send(outcome.map_err(|error| format!("{error:#}"))); + }); + outcome = receiver + .recv_timeout(Duration::from_secs(30)) + .unwrap_or_else(|_| { + panic!( + "the supervisor did not finish: it was not polling the stand-in `rocm`, \ + which answers its readiness check and lets the engine exit" + ) + }); + // The first exec of a new file can lose a race with an unrelated + // fork (ETXTBSY): the spawn fails at once, and the next attempt + // starts again. + if !matches!(&outcome, Err(error) if error.contains("Text file busy")) { + break; + } + } + let serve_args = fs::read_to_string(&served).unwrap_or_default(); + let stdio_args = fs::read_to_string(&asked).unwrap_or_default(); + fs::remove_dir_all(&root).ok(); + + assert_eq!(outcome, Ok(()), "the supervisor must finish cleanly"); + let serve_args: Vec<&str> = serve_args.lines().collect(); + assert_eq!( + serve_args.get(..3), + Some(&["__engine-serve-http", "llamacpp", service_id][..]), + "the engine must be started as `rocm __engine-serve-http `: {serve_args:?}" + ); + assert_eq!( + stdio_args.lines().collect::>(), + ["__engine-stdio", "llamacpp"], + "readiness must be polled through the same `rocm`" + ); + Ok(()) + } + + /// With no `rocm` beside `rocmd` the supervisor stops before it writes + /// anything, and says what it looked for: the service's record is as it was, + /// and no log was created for an engine that never started. + #[test] + fn supervise_without_a_rocm_beside_rocmd_changes_nothing() -> Result<()> { + let (root, paths) = temp_app_paths("supervise-no-rocm"); + let service_id = "svc-supervise-no-rocm"; + // A failed service whose supervisor is gone: a write by a supervisor + // that then stops would change both fields. + let mut seeded = identity_probe_record(&paths, service_id, 11434); + seeded.status = "failed".to_owned(); + seeded.supervisor_pid = 0; + seeded.write()?; + let before = load_service_record(&paths, service_id)?; + fs::remove_file(&before.log_path).ok(); + + let outcome = supervise_service_on( + &DaemonHost::new(root.join("rocmd"), false), + &paths, + service_id.to_owned(), + "llamacpp".to_owned(), + "a-model".to_owned(), + "a-model".to_owned(), + None, + None, + "127.0.0.1".to_owned(), + 11434, + "gpu_required".to_owned(), + None, + None, + ); + let after = load_service_record(&paths, service_id); + let log_created = before.log_path.exists(); + fs::remove_dir_all(root).ok(); + + let error = format!("{:#}", outcome.expect_err("no rocm to run the engine on")); + assert!(error.contains("no engine commands"), "{error}"); + assert!(error.contains("unable to locate sibling binary"), "{error}"); + let after = after?; + assert_eq!(after.status, before.status); + assert_eq!(after.supervisor_pid, before.supervisor_pid); + assert_eq!(after.restart_count, before.restart_count); + assert!( + !log_created, + "no engine log for an engine that never started" + ); + Ok(()) + } + /// Between a restart's first write, which clears the recorded PIDs, and the /// one recording the supervisor it spawned, a stop can persist its marker. /// Writing the restart's own copy back would erase it while the new @@ -1899,6 +2257,86 @@ mod tests { Ok(()) } + /// Both halves of a restart from the `rocm` binary, at the instant the + /// supervisor is spawned: the recorded port, which recovery leased to claim + /// it, is already released, and the program being started is the `rocmd` + /// beside `rocm`, not `rocm` itself. + #[cfg(target_os = "linux")] + #[test] + fn restart_from_the_rocm_binary_releases_the_port_and_runs_the_sibling_rocmd() -> Result<()> { + use std::cell::{Cell, RefCell}; + use std::net::TcpListener; + + let _serial = lock_script_tests(); + let tools = script_tools(); + let (root, paths) = temp_app_paths("restart-rocm-releases-lease"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + let stand_in = root.join("rocmd"); + write_executable_script( + &stand_in, + &tools, + &format!("exec '{}' 1\n", tools.sleep.display()), + )?; + // The daemon is `rocm`, which does not exist here; the `rocmd` beside it + // is the stand-in. + let host = DaemonHost::new(root.join("rocm"), true); + + let probe = TcpListener::bind(("127.0.0.1", 0))?; + let port = probe.local_addr()?.port(); + drop(probe); + let service_id = "svc-restart-rocm-releases-lease"; + let mut record = identity_probe_record(&paths, service_id, port); + record.status = "failed".to_owned(); + record.write()?; + + let bind_succeeded = Cell::new(false); + let spawned_args = RefCell::new(Vec::::new()); + let spawn_override = + |command: &mut ProcessCommand| -> std::io::Result { + // Retry briefly: a sibling test's concurrent fork can hold a + // copy of the just-released socket until its exec closes it. + let deadline = std::time::Instant::now() + Duration::from_secs(2); + while !bind_succeeded.get() && std::time::Instant::now() < deadline { + bind_succeeded.set(TcpListener::bind(("127.0.0.1", port)).is_ok()); + } + *spawned_args.borrow_mut() = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + command.spawn() + }; + + // Whether the stand-in then stays up does not matter here: what is read + // is what the restart was about to run. + let _ = restart_managed_service_via( + &host, + &paths, + &mut record, + Some(&spawn_override), + rocm_core::MANAGED_LAUNCH_LOCK_TIMEOUT, + ); + fs::remove_dir_all(&root).ok(); + + assert!( + bind_succeeded.get(), + "the leased port must already be released when the supervisor is spawned" + ); + let spawned_args = spawned_args.into_inner(); + assert_eq!( + spawned_args.get(..3), + Some( + &[ + stand_in.display().to_string(), + "supervise".to_owned(), + service_id.to_owned() + ][..] + ), + "the supervisor must be the sibling rocmd, started as `rocmd supervise `: {spawned_args:?}" + ); + Ok(()) + } + /// Recovery re-claims the concrete port already written in the record, so /// it must serialize on the same cross-process launch lock a fresh /// `rocm serve` launch takes, and must refuse a recorded port something diff --git a/apps/rocmd/src/test_support.rs b/apps/rocmd/src/test_support.rs index 759b892033..876adc5221 100644 --- a/apps/rocmd/src/test_support.rs +++ b/apps/rocmd/src/test_support.rs @@ -251,3 +251,66 @@ mod tests { assert_each_call_gets_its_own_root(|| temp_app_paths("same-label").0); } } + +/// Tests that run a script they just wrote take this lock. Writing an +/// executable while another thread forks can leave the new file "text file +/// busy" at its first exec; the lock keeps these tests from doing it to each +/// other, and each retries for the forks it cannot see. +#[cfg(unix)] +static SCRIPT_TESTS: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +#[cfg(unix)] +pub(crate) fn lock_script_tests() -> std::sync::MutexGuard<'static, ()> { + SCRIPT_TESTS + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) +} + +/// The tools a stand-in script needs, each found once on `PATH` as an absolute +/// path. A missing one fails the test by name, rather than being skipped or +/// surfacing as a confusing exec error. +#[cfg(unix)] +pub(crate) struct ScriptTools { + pub(crate) sh: PathBuf, + pub(crate) sleep: PathBuf, +} + +#[cfg(unix)] +pub(crate) fn find_on_path(name: &str) -> Option { + let path = std::env::var_os("PATH").unwrap_or_default(); + std::env::split_paths(&path) + .map(|dir| dir.join(name)) + .find(|candidate| candidate.is_file()) +} + +#[cfg(unix)] +pub(crate) fn script_tools() -> ScriptTools { + let find = |name: &str| { + find_on_path(name) + .unwrap_or_else(|| panic!("this test needs `{name}` on PATH to stand in for a binary")) + }; + ScriptTools { + sh: find("sh"), + sleep: find("sleep"), + } +} + +/// Write `body` as a script for `tools.sh` at `path` and make it executable. +/// Every writer is closed before this returns, so the first exec cannot find +/// the file open for writing. The script may use the shell's own commands and +/// the tools in `tools`. +#[cfg(unix)] +pub(crate) fn write_executable_script( + path: &std::path::Path, + tools: &ScriptTools, + body: &str, +) -> Result<()> { + use std::io::Write; + use std::os::unix::fs::PermissionsExt; + + let mut file = fs::File::create(path)?; + write!(file, "#!{}\n{body}", tools.sh.display())?; + file.set_permissions(fs::Permissions::from_mode(0o755))?; + file.sync_all()?; + Ok(()) +} diff --git a/crates/rocm-core/src/binary_paths.rs b/crates/rocm-core/src/binary_paths.rs index 3a4a64c752..48ded88aa4 100644 --- a/crates/rocm-core/src/binary_paths.rs +++ b/crates/rocm-core/src/binary_paths.rs @@ -12,9 +12,17 @@ use crate::require_nonempty; use crate::{current_executable_path, platform_binary_name}; pub fn sibling_binary_path(binary_name: &str) -> Result { + sibling_binary_path_from(¤t_executable_path()?, binary_name) +} + +/// [`sibling_binary_path`] for a caller that already holds the executable path. +/// +/// A caller that decides something from the current executable and then looks +/// for a sibling passes the one value to both, so the two cannot disagree: the +/// lookup falls back to `argv[0]` where `std::env::current_exe` fails. +pub fn sibling_binary_path_from(current_exe: &Path, binary_name: &str) -> Result { require_nonempty(binary_name, "binary_name")?; - let current_exe = current_executable_path()?; - let candidates = sibling_binary_candidates(¤t_exe, binary_name)?; + let candidates = sibling_binary_candidates(current_exe, binary_name)?; for candidate in &candidates { if candidate.is_file() { return Ok(candidate.clone()); @@ -76,3 +84,34 @@ pub fn daemon_binary_path() -> Result { } Ok(current_exe) } + +#[cfg(test)] +mod tests { + use super::*; + + /// The lookup starts from the executable it is given, not from the + /// process's own: a caller that chose something from one path must find the + /// sibling beside that same path. + #[test] + fn sibling_binary_path_from_looks_beside_the_given_executable() { + let dir = tempfile::tempdir().expect("tempdir"); + let exe = dir.path().join(platform_binary_name("rocm")); + let sibling = dir.path().join(platform_binary_name("rocmd")); + std::fs::write(&sibling, b"").expect("write sibling"); + + assert_eq!(sibling_binary_path_from(&exe, "rocmd").unwrap(), sibling); + + let error = sibling_binary_path_from(&exe, "rocm-engine-none").unwrap_err(); + let text = error.to_string(); + assert!(text.contains("unable to locate sibling binary"), "{text}"); + assert!( + text.contains(&dir.path().display().to_string()), + "the error names where it looked: {text}" + ); + } + + #[test] + fn sibling_binary_path_from_rejects_an_empty_name() { + assert!(sibling_binary_path_from(Path::new("/opt/bin/rocm"), "").is_err()); + } +} diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 0ccf9962e9..1a7f6606c6 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -81,7 +81,9 @@ pub use automation::{ load_recent_automation_events, load_recent_automation_proposals, update_automation_proposal_status, }; -pub use binary_paths::{daemon_binary_path, engine_binary_path, sibling_binary_path}; +pub use binary_paths::{ + daemon_binary_path, engine_binary_path, sibling_binary_path, sibling_binary_path_from, +}; pub use codex_bridge::{CodexBridgeEngine, CodexBridgeGpuSnapshot, CodexBridgeSnapshot}; pub use command_capture::{CommandCapture, command_capture_text, run_rocm_capture_with_env}; pub use config::{ diff --git a/docs/architecture.md b/docs/architecture.md index 1b5f1c677b..7b35533244 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -148,7 +148,8 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`anthropic_proxy.rs`](../apps/rocmd/src/anthropic_proxy.rs) — the Anthropic-compatible `/v1/messages` front for a served model: `spawn_alongside_engine` waits for the engine to list its model, then binds a translation proxy, even for an engine that has its own `/v1/messages` route because that route only accepts the served model's name (`anthropic_to_openai`, `openai_to_anthropic`, an axum router over `/v1/messages` and `/v1/models`) on a port of its own, publishing the result in the engine state file for the service record to adopt. It is a library module, not a daemon task: `rocm`'s `run_builtin_engine_serve_http` in [`engine_ipc.rs`](../apps/rocm/src/engine_ipc.rs) starts it on a thread of the process that owns the engine, so it lives and dies with the service. - [`common.rs`](../apps/rocmd/src/common.rs) — helpers the other modules call into: the read-only `rocm update` message, the `rocm`-subprocess capture wrapper (`run_rocm_capture_for_paths`) the sandbox and MCP clusters both call (it only chooses the child's `ROCM_CLI_*_DIR` environment; `CommandCapture` and the spawn itself live in [`command_capture.rs`](../crates/rocm-core/src/command_capture.rs), shared with `rocm`'s chat tool bridge, and the bounded-execution half in [`process.rs`](../crates/rocm-core/src/process.rs)), and small arg/JSON-payload utilities (`payload_string`, for example, is read by the [`watchers/cache_warm.rs`](../apps/rocmd/src/watchers/cache_warm.rs), [`watchers/driver_upgrade.rs`](../apps/rocmd/src/watchers/driver_upgrade.rs) and [`watchers/gpu_thermal_protect.rs`](../apps/rocmd/src/watchers/gpu_thermal_protect.rs) handlers and by [`webhook.rs`](../apps/rocmd/src/webhook.rs)). - [`daemon.rs`](../apps/rocmd/src/daemon.rs) — the foreground `rocmd run` loop (`run_daemon`: evaluate the watchers each tick, dispatch local webhook events, write the runtime state, stop on a signal; with no watcher enabled it exits by itself, deciding that under the autostart lock and publishing "stopped" and clearing its spawn claim before releasing it — `record_daemon_stop`, `release_own_autostart_claim`) and `rocmd status` (`print_status`), with the watcher tick interval, which `LoopHooks` makes injectable for tests. It uses [`watchers.rs`](../apps/rocmd/src/watchers.rs), [`webhook.rs`](../apps/rocmd/src/webhook.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs) calls it. -- [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) — talking to an engine and guarding its endpoint key: `engine_request` (the stdio request to an engine process), the healthcheck call and its readiness and recoverability checks, and the endpoint-key helpers `apply_endpoint_key_env` and `ensure_public_service_has_endpoint_key`. It uses no other module of the crate; [`service.rs`](../apps/rocmd/src/service.rs), [`supervise.rs`](../apps/rocmd/src/supervise.rs) and [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) call it. +- [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs) — which binary a daemon spawn runs. The daemon runs as the `rocmd` binary or embedded in `rocm` (`rocm daemon`, which enters through `run_from_args` and marks the process), and `supervise` is a command of `rocmd` alone. The hidden engine commands (`__engine-serve-http`, `__engine-stdio`) are `rocm`'s alone. `DaemonHost` holds the one executable path every choice and sibling lookup is made from (`rocm_core::sibling_binary_path_from`). `rocmd_binary` returns the current executable when the daemon is `rocmd` and the `rocmd` installed beside `rocm` when it is embedded; `rocm_binary` is the mirror, the current executable when embedded and the `rocm` installed beside `rocmd` when not. Each names what was missing when there is none. `detect` returns the real host. Under test it returns the test binary for both commands and looks for no sibling, and the two stand-in hosts (`test_binary_standing_in_for_rocm` and `test_binary_standing_in_for_rocmd`) name only the binary they stand in for and fail when asked for the other, so none of them can run a `rocm` or `rocmd` built beside the test binary; a test that needs another binary passes the code it exercises a host of its own. A leaf: it imports no other crate module. +- [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) — talking to an engine and guarding its endpoint key: `engine_request` (the stdio request to an engine process, run on the `rocm` binary that `DaemonHost::rocm_binary` finds, since `__engine-stdio` is `rocm`'s command), the healthcheck call and its readiness and recoverability checks, and the endpoint-key helpers `apply_endpoint_key_env` and `ensure_public_service_has_endpoint_key`. It uses no other module of the crate but [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs); [`service.rs`](../apps/rocmd/src/service.rs), [`supervise.rs`](../apps/rocmd/src/supervise.rs) and [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) call it. - [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs) — the host snapshot: `build_bridge_snapshot` (the bridge snapshot printed by `rocmd bridge-snapshot` and returned by the `bridge_snapshot` MCP tool, assembled from the examine summary, the GPU snapshot, the automation runtime state and recent events, the engine inventory and the managed services), the amd-smi GPU snapshot (`gather_gpu_snapshot_for_config`, which honours the telemetry setting, over time-bounded amd-smi probes) and `bridge_engine_inventory`. It uses only [`persistence.rs`](../apps/rocmd/src/persistence.rs); [`cli.rs`](../apps/rocmd/src/cli.rs), [`mcp.rs`](../apps/rocmd/src/mcp.rs) and [`watchers.rs`](../apps/rocmd/src/watchers.rs) call it. - [`main.rs`](../apps/rocmd/src/main.rs) — the `rocmd` binary's entry point: a synchronous `main` that calls `rocmd::run_bin_cli()`, which builds the Tokio runtime itself (`run_bin_cli` is `#[tokio::main]`) and dispatches the parsed `Cli`, so the binary holds no logic of its own. - [`mcp.rs`](../apps/rocmd/src/mcp.rs) — the MCP stdio server, tool schema table, tool dispatch, and the result-shaping helpers behind the MCP tools (the `rocm` subprocess itself is spawned through [`common.rs`](../apps/rocmd/src/common.rs), via `run_rocm_capture` here). The argument handling the dispatch calls is in [`mcp_argv.rs`](../apps/rocmd/src/mcp_argv.rs). @@ -157,7 +158,7 @@ Modularized (ROCMAI-83, Phase 5 of ROCMAI-27's sequencing): - [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) — bubblewrap/native sandbox execution, artifact prefetch policy gating, and the sandbox-tool result shaping for `check_updates`/`driver_plan`/`stop_server`. - [`sandbox_tool.rs`](../apps/rocmd/src/sandbox_tool.rs) — the sandbox-tool vocabulary: the `SandboxToolArg` value enum naming each sandbox tool, and `SandboxToolPolicy`, the artifact-download policy built from the `--allow-*` flags (including the Hugging Face token read from the environment when authenticated downloads are allowed). It uses no other module of the crate, so modules that need the vocabulary do not import [`cli.rs`](../apps/rocmd/src/cli.rs). - [`service.rs`](../apps/rocmd/src/service.rs) — managed-service stop: terminating the recorded processes of a service after verifying their identity, and the stop report (`stop_managed_service`, `stop_report_confirmed`) that the sandbox and MCP tools read. -- [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses, and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs) and [`persistence.rs`](../apps/rocmd/src/persistence.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. +- [`supervise.rs`](../apps/rocmd/src/supervise.rs) — launching and re-launching a managed service's supervisor: `supervise_service` (the `rocmd supervise` entry point: launch the engine child, record its identity, wait for readiness, record the exit) with the serve-log startup-phase polling and readiness wait it uses (its engine child runs on the `rocm` binary that `DaemonHost::rocm_binary` finds, before the first write, since `rocmd` has no engine commands, and it polls readiness through the same host), and `restart_managed_service` (the recovery respawn: re-launch `rocmd supervise` detached for a service and record the new supervisor; the binary that runs `supervise` comes from [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), and a restart that cannot find it fails before it writes the record). Writes after a supervisor's first go through the private `update_supervised_record`, which re-reads the record and leaves a stop marker or another supervisor's record alone. Its production code uses only [`common.rs`](../apps/rocmd/src/common.rs), [`daemon_host.rs`](../apps/rocmd/src/daemon_host.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and `stop_requested` from [`watchers.rs`](../apps/rocmd/src/watchers.rs). [`cli.rs`](../apps/rocmd/src/cli.rs) calls `supervise_service`; [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) (the server-recover handler) and [`sandbox.rs`](../apps/rocmd/src/sandbox.rs) (the `restart_server` tool) call `restart_managed_service`. - [`watchers.rs`](../apps/rocmd/src/watchers.rs) — the module root of the built-in watchers: the registry (`reconcile_watcher_snapshots`) and the event collector and dispatcher (`evaluate_watchers`, `collect_automation_events_with_gpu_snapshot`, which takes the GPU snapshot as a closure so tests can inject one, and `evaluate_watchers_for_events`), which route each event to its watcher's handler. `find_recoverable_service` and `stop_requested` are re-exported from here because [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s and [`service.rs`](../apps/rocmd/src/service.rs)'s tests reach them by the `crate::watchers::` path. Each watcher's handler lives in the child below that carries its name, together with the policy they share, and a child reaches only [`policy.rs`](../apps/rocmd/src/watchers/policy.rs) and, for the thermal-protect watcher, [`gpu_metrics.rs`](../apps/rocmd/src/watchers/gpu_metrics.rs) among the others. - [`watchers/policy.rs`](../apps/rocmd/src/watchers/policy.rs) — what a watcher does in each mode and the proposals it queues: `WatcherPolicyAction` and `watcher_policy_action` (observe, queue a proposal or run contained, chosen by the watcher mode), `queue_proposal` and `queue_proposal_with_arguments` with the action-to-tool and action-to-arguments tables behind them, and `pending_proposal_exists`, the check the thermal-protect and server-recover handlers use so they do not queue a duplicate of a proposal that is still pending. It uses no other watcher module. - [`watchers/server_recover.rs`](../apps/rocmd/src/watchers/server_recover.rs) — the `server-recover` watcher: finding a managed service that needs recovery (`find_recoverable_service`, from a failed or stale manifest, an unreachable endpoint or a failing engine healthcheck, skipping a service whose stop request still stands via `stop_requested`), the recovery event kinds and plain-language reasons, the backoff that spaces recovery attempts (`server_recover_due`), and the handler that restarts the service or queues a restart proposal by watcher mode. It uses [`policy.rs`](../apps/rocmd/src/watchers/policy.rs), [`engine_ipc.rs`](../apps/rocmd/src/engine_ipc.rs), [`persistence.rs`](../apps/rocmd/src/persistence.rs) and [`supervise.rs`](../apps/rocmd/src/supervise.rs)'s `restart_managed_service`. diff --git a/docs/testing.md b/docs/testing.md index ffa0790ae4..7498ca8a4b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2599,6 +2599,14 @@ cargo test -p rocmd supervise_service_first_write_keeps_a_standing_stop_marker cargo test -p rocmd restart_failure_write_keeps_a_stop_marker_set_meanwhile cargo test -p rocmd restart_records_its_supervisor_without_erasing_a_stop_marker cargo test -p rocmd restart_leaves_a_record_naming_another_supervisor_or_none +cargo test -p rocmd daemon_host +cargo test -p rocmd restart_runs_supervise_on_the_rocmd_it_is_given +cargo test -p rocmd restart_without_a_rocmd_to_run_changes_nothing +cargo test -p rocmd supervise_runs_the_engine_on_the_rocm_beside_rocmd +cargo test -p rocmd supervise_without_a_rocm_beside_rocmd_changes_nothing +cargo test -p rocmd engine_request_runs_the_stdio_command_on_the_rocm_beside_rocmd +cargo test -p rocmd engine_request_without_a_rocm_beside_rocmd_names_it +cargo test -p rocm-core sibling_binary_path_from cargo test -p rocm --bin rocm stop_records_its_marker_on_disk_before_any_process_is_signalled cargo test -p rocm --bin rocm stop_does_not_confirm_when_a_restart_recorded_new_processes_meanwhile cargo test -p rocm --bin rocm stop_final_write_keeps_what_others_wrote_while_it_ran @@ -2623,6 +2631,24 @@ it reports the unconfirmed stop and starts nothing. Its own later writes are applied to the record as it is on disk, and only while it still reads as the restart left it, so a stop asked for during the restart keeps its marker. +The daemon runs as the `rocmd` binary or embedded in `rocm` (`rocm daemon`), +and each binary has commands the other lacks: `supervise` is `rocmd`'s, and the +hidden engine commands (`__engine-serve-http`, `__engine-stdio`) are `rocm`'s. +Every daemon spawn therefore asks `DaemonHost` for the binary instead of +re-executing its own: the recovery restart runs `supervise` on `rocmd`, and the +supervisor starts its engine, polls its readiness and asks it to stop through +`rocm`, each the current executable when the daemon is that binary and the +sibling installed beside it otherwise, named in the error when there is none. +`rocmd`'s unit tests never reach the real host: under test `DaemonHost::detect` +returns the test binary for both commands and looks for no sibling, the two +stand-in hosts name only the binary they stand in for and fail when asked for +the other, each test that needs a different host passes the code under test one +of its own, and a test that needs a binary writes a stand-in script (its shell +and `sleep` found on `PATH` once and asserted by name), so none of them can run +a `rocm` or `rocmd` that happens to sit in the build directory. `run_from_args_marks_the_process_as_embedded_in_rocm` +checks the embedded marking in a child run of the test binary, because the flag +is process-wide. + Automation GPU-metrics event tests: ```bash From 71b0162d7367ec663c026eeb531cb8e3915fe22e Mon Sep 17 00:00:00 2001 From: Eugene Volen <150254791+volen-silo@users.noreply.github.com> Date: Mon, 9 Nov 2026 14:48:53 +0000 Subject: [PATCH 022/102] feat(lemonade): announce installs, show their progress, and report them in storage (#853) * feat(lemonade): announce a Lemonade install before it starts and show its progress on Windows ROCMAI-825. A fresh Lemonade install with its llama.cpp ROCm backend is a download of up to ~4.7 GB and a tree of up to ~25 GiB, and `rocm serve` starts one on its own whenever the install target has no current tree, for example the active runtime's own folder after `rocm install sdk`. It used to print one "Preparing..." line and then nothing for ten minutes. Before writing a new tree, the engine now prints where it installs, the ROCm backend's download and disk use as upper bounds, the free space on that drive, and any existing Lemonade install it is not reusing (never a record whose folder no longer holds an install). The backend is only chosen after this point, and Lemonade falls back to a smaller one where its ROCm build is unsupported, so the figures are "up to" and only warn. The one thing that refuses, before anything is downloaded, is a drive that cannot hold Lemonade itself (its archive and extracted files, about 16 MiB), the one figure that holds for every backend. The same figures are printed in the notice and in both warnings (the margin is used for the comparison only), and an install that replaces the existing tree (`--reinstall`, a new Lemonade version) deletes it before downloading, so its size is counted as free and the notice says so. `lemonade backends install` inherited the console but was started with CREATE_NO_WINDOW. A child started that way writes nothing to a console it inherited: from a real console, `cmd /c echo`, a Python child and `lemonade.exe --version` all printed without the flag and printed nothing with it. So the download progress vanished from an interactive console while still reaching a redirected log. The flag is now applied only when neither stdout nor stderr is a console. The window decision lives in the command builder (`backend_install_command`), which takes the console state and the hide step, so a unit test drives the real command for each console state and fails if the hide is dropped, unconditional, or ignores a stream. Scenarios lemonade-installs-01 (`rocm engines install lemonade`) and -02 (`rocm serve`) run on the mock lane with the e2e-test-hooks build. They report 1 MiB free through a test-only override, which also waives serve's no-GPU refusal as serve-18's failure hook does, and check against the disk that nothing was downloaded. The warning range is covered by unit tests, since a scenario would have to start the download. Signed-off-by: Eugene Volen * feat(storage): list Lemonade copies and clear a removed runtime's install record ROCMAI-825. Lemonade installs into the active runtime's folder when one is active and into the data folder otherwise, so installing Lemonade before the first SDK and serving after it leaves two full copies (24.3 GiB each on the release-test host). `rocm storage report` counted both but named neither: one hid inside a runtime's size, the other inside the data folder total, and once the record moved to the runtime's copy the data folder's copy was used by nothing and never cleaned up. The report now has an `Engine installs` section listing each Lemonade tree with its size (the install parts only, not the record, logs and state that share the data folder's tree). Reporting only; nothing is deleted. A copy is `in use` when the install record points into it. A copy is `not used right now` when the record points at another copy that really holds an install, which is not a promise it stays unused: Lemonade installs into the active runtime's folder, so a copy inside a registered runtime is reused as soon as that runtime becomes active and is never marked reclaimable. Only the data folder's copy, which no runtime owns, is. A record that names a folder with no install, or that cannot be read, leaves it unknown which copy works, so no copy is called unused: following the delete advice there would have removed the only working tree and forced a ~25 GiB reinstall. A copy inside an external or imported runtime is not said to be removed with it, using the same rule uninstall applies. `rocm runtimes uninstall` (and `storage remove-old-installs`, which delegates to it) deletes the runtime folder, Lemonade's copy included, but left Lemonade's install record naming the deleted files. The plan now says Lemonade goes with the runtime, and once the folder is gone the record is cleared and the result says so. A record that could not be deleted is reported with the file to delete. A record that could not be read is reported only when the removed runtime held a Lemonade tree, and names a plain `rocm engines install lemonade`, which writes the record without reading it and keeps a current tree (shown by a test that runs the real record-writing sequence over a corrupt record), not the wiping `--reinstall`. The install notice points at `rocm storage` for a copy that is not reused only when storage lists it (the data folder's tree or a tree inside a registered ROCm install), so a recorded install anywhere else is announced without a pointer that would be false for it. Scenarios lemonade-installs-03 (both copies listed, unused one flagged), -04 (uninstall clears the record), -05 (dangling record keeps the survivor off the reclaimable list) and -06 (a read-only runtime's copy survives uninstall and is not said to be removed), all on the mock lane. Signed-off-by: Eugene Volen --------- Signed-off-by: Eugene Volen (cherry picked from commit 5e3b313c6b207cbb7ef01d774e31b800a9968e97) Signed-off-by: juhovainio --- README.md | 34 + apps/rocm/src/runtime_uninstall.rs | 274 ++++- apps/rocm/src/runtimes_cmd.rs | 6 +- apps/rocm/src/serve_cmd.rs | 11 +- apps/rocm/src/storage.rs | 520 +++++++++ apps/rocm/src/test_support.rs | 40 + docs/architecture.md | 7 +- docs/manual-testing.md | 30 + docs/testing.md | 34 + engines/lemonade/src/footprint.rs | 1010 +++++++++++++++++ engines/lemonade/src/install.rs | 18 +- engines/lemonade/src/lib.rs | 74 +- engines/lemonade/src/process.rs | 146 ++- .../features/lemonade_installs.feature | 64 ++ tests/e2e-cucumber/tests/e2e.rs | 1 + .../tests/e2e/lemonade_install_steps.rs | 402 +++++++ tests/e2e-cucumber/tests/feature_naming.rs | 3 + 17 files changed, 2638 insertions(+), 36 deletions(-) create mode 100644 engines/lemonade/src/footprint.rs create mode 100644 tests/e2e-cucumber/features/lemonade_installs.feature create mode 100644 tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs diff --git a/README.md b/README.md index a83a1b2fc5..6a722243b2 100644 --- a/README.md +++ b/README.md @@ -883,6 +883,28 @@ followed. The two archive rows the report prints are tagged `note: can be downloaded again; safe to remove`, so it is visible which rows `remove-downloads` acts on. +The report also lists `Engine installs`: each Lemonade copy on disk and its +size. Lemonade installs into the active ROCm runtime's own folder when one is +active and into the data folder otherwise, so a machine that installed +Lemonade before its first SDK can hold two full copies (about 25 GiB each). The +copy Lemonade runs from is `in use`. A copy it is not using right now is marked +`not used right now`, which is not a promise it stays unused: Lemonade installs +into the active ROCm runtime's folder, so a copy inside a ROCm install is reused +as soon as that install becomes active, and is never marked reclaimable. Only +the data folder's copy, which no ROCm install owns, is marked reclaimable, and +nothing removes it automatically: delete its `runtime`, `downloads` and `extract` +folders to free the space. A copy is only called unused when Lemonade's install +record points at another copy that is really there: if the record names a folder +with no install, or cannot be read, the report says so and marks no copy +reclaimable. A copy inside a runtime folder +also counts toward that runtime's size. `rocm runtimes uninstall` removes it +with the runtime and clears Lemonade's install record, so Lemonade is no longer +recorded as installed; it says so when the record could not be cleared, and when +it could not be read it names `rocm engines install lemonade`, which rewrites +the record and keeps a current install. +For an external or imported runtime, uninstall only unregisters it, so its +Lemonade copy stays and the report does not say otherwise. + The report also lists `local server records`: one JSON record plus the engine's log for each `rocm serve --managed` launch, kept after the server stops. `rocm services list --all` lists them, and no `rocm storage` command removes @@ -912,6 +934,18 @@ and installs once more; see [Lemonade backend alignment](https://github.com/ROCm/rocm-cli/blob/main/docs/engine-plugins.md#lemonade-backend-alignment-on-engine-install) for the details and for `ROCM_CLI_DISABLE_LEMONADE_BACKEND_ALIGNMENT`, which turns both off. +Before Lemonade writes a new install, whether from `rocm engines install +lemonade` or the first `rocm serve` that needs it, it prints the folder it is +installing into, the download (up to about 4.4 GiB) and disk use (up to about +25 GiB) of its llama.cpp ROCm backend, and any Lemonade install it is not +reusing. Those are upper bounds: Lemonade picks the backend after the install +starts, and a smaller one needs less. So it stops before downloading only when +the drive cannot hold Lemonade itself (about 16 MiB), and with less room than +the ROCm backend or the finished install it warns and continues. An install that +replaces the existing one (`--reinstall`, or a new Lemonade version) deletes it +first, so its space counts as free and the notice says so. On Windows the +backend download's own progress appears in the console. + `--yes` approves the system-package installs vLLM needs (OpenMPI, libatomic and libnuma), including a `sudo` password prompt. `engines install` never asks for confirmation. Without `--yes`, a Linux host with root or passwordless `sudo` diff --git a/apps/rocm/src/runtime_uninstall.rs b/apps/rocm/src/runtime_uninstall.rs index 8b102383e6..e140cf1550 100644 --- a/apps/rocm/src/runtime_uninstall.rs +++ b/apps/rocm/src/runtime_uninstall.rs @@ -5,12 +5,16 @@ //! `rocm runtimes uninstall`: planning the removal of a registered runtime, //! re-validating that plan after the confirmation prompt, and applying it //! (unregistering the runtime, clearing the active and default state that named -//! it, and removing its install root when ROCm CLI owns it). +//! it, and removing its install root when ROCm CLI owns it). When that root +//! holds the Lemonade install its record points at (a self-managed engine +//! installs inside the active runtime's folder), the plan says so and applying +//! it clears the record, so Lemonade does not keep naming deleted files. //! -//! Relocated from `runtimes_cmd.rs` with no behavior change. `runtimes_cmd.rs` -//! calls into this module from `runtimes()` and re-exports `uninstall_runtime`, -//! `should_remove_runtime_install_root` and `InstallRootDecision` for -//! `storage.rs`. The active-runtime marker check this module makes +//! Relocated from `runtimes_cmd.rs`; the Lemonade record handling is new. +//! `runtimes_cmd.rs` calls into this module from `runtimes()` and re-exports +//! `uninstall_runtime`, `should_remove_runtime_install_root`, +//! `InstallRootDecision` and `lemonade_record_report_line` for `storage.rs`. +//! The active-runtime marker check this module makes //! (`active_runtime_marker_matches`, over the `ActiveRuntimeMarker` the //! activation writes) and the path comparison it needs live in //! `runtime_registry.rs`, so this module does not depend on `runtimes_cmd.rs`. @@ -20,6 +24,7 @@ use std::path::{Path, PathBuf}; use anyhow::{Context, Result, bail}; use rocm_core::{AppPaths, RocmCliConfig, runtime_install_root_is_protected}; +use rocm_engine_lemonade::RecordCleanup; use crate::cli_report; use crate::runtime_registry::{ @@ -34,6 +39,9 @@ pub(crate) struct RuntimeUninstallResult { pub(crate) runtime_key: String, pub(crate) registry_path: PathBuf, pub(crate) removed_install_root: Option, + /// What happened to Lemonade's install record once the folder was removed: + /// cleared, left dangling because deleting it failed, or not checkable. + pub(crate) lemonade_record: RecordCleanup, pub(crate) read_only: bool, pub(crate) manifest_mismatch: bool, pub(crate) was_active: bool, @@ -70,6 +78,14 @@ pub(crate) struct RuntimeUninstallPlan { /// does not by itself orphan the default while others remain). clears_default_runtime: bool, install_root_decision: InstallRootDecision, + /// Lemonade's runtime folder when its install record points inside this + /// runtime's folder: a self-managed engine installs there while the runtime + /// is active, so removing the folder removes that engine too. + lemonade_install_inside: Option, + /// Whether this runtime's folder holds a Lemonade tree at all, installed or + /// partly installed. Only then can an unreadable install record matter to + /// this uninstall. + holds_lemonade_tree: bool, } impl RuntimeUninstallPlan { @@ -78,6 +94,40 @@ impl RuntimeUninstallPlan { } } +/// The line `runtimes uninstall` (and `storage remove-old-installs`, which +/// delegates to it) report about Lemonade's install record, or `None` when the +/// removed runtime had nothing to do with it. +/// +/// Built from the outcome the record handling returned, so each claim is the one +/// that happened: "no longer recorded" only follows a record that was deleted, +/// and a record that could not be deleted or read says so and how to fix it. +pub(crate) fn lemonade_record_report_line(outcome: &RecordCleanup) -> Option { + match outcome { + RecordCleanup::NotAffected => None, + RecordCleanup::Cleared(lemonade) => Some(format!( + "its install at {} was inside this runtime and was removed with it; \ + Lemonade is no longer recorded as installed", + lemonade.display() + )), + RecordCleanup::Failed { + runtime_dir, + record, + error, + } => Some(format!( + "its install at {} was inside this runtime and was removed with it, but \ + Lemonade's install record could not be deleted ({error}); delete {} \ + so Lemonade is not left pointing at files that are gone", + runtime_dir.display(), + record.display() + )), + RecordCleanup::Unreadable(error) => Some(format!( + "Lemonade's install record could not be read ({error}), so it was not \ + checked against this runtime; `rocm engines install lemonade` rewrites it \ + and keeps a current Lemonade install" + )), + } +} + pub(crate) fn print_runtime_uninstall_plan(plan: &RuntimeUninstallPlan) { let install_folder = if plan.will_remove_install_root() { format!( @@ -104,6 +154,17 @@ pub(crate) fn print_runtime_uninstall_plan(plan: &RuntimeUninstallPlan) { if plan.clears_default_runtime { report = report.detail("default_runtime", "would be cleared"); } + if plan.will_remove_install_root() + && let Some(lemonade) = plan.lemonade_install_inside.as_ref() + { + report = report.detail( + "lemonade", + format!( + "its install at {} is inside this runtime folder and would be removed with it", + lemonade.display() + ), + ); + } print!("{}", report.render()); } @@ -127,12 +188,23 @@ pub(crate) fn plan_runtime_uninstall( && other.runtime_id.eq_ignore_ascii_case(&manifest.runtime_id) })); let install_root_decision = should_remove_runtime_install_root(&manifest)?; + let holds_lemonade_tree = + rocm_engine_lemonade::install_tree_dirs(&rocm_engine_lemonade::install_tree_root( + paths, + Some(&manifest.install_root.join("engines")), + )) + .iter() + .any(|part| part.exists()); + let lemonade_install_inside = rocm_engine_lemonade::recorded_runtime_dir(paths) + .filter(|dir| rocm_core::runtime_path_is_same_or_inside(dir, &manifest.install_root)); Ok(RuntimeUninstallPlan { manifest, registry_path, was_active, clears_default_runtime, install_root_decision, + lemonade_install_inside, + holds_lemonade_tree, }) } @@ -215,6 +287,8 @@ pub(crate) fn apply_runtime_uninstall( was_active, clears_default_runtime, install_root_decision, + lemonade_install_inside: _, + holds_lemonade_tree, } = plan; let mut removed_install_root = None; @@ -288,11 +362,27 @@ pub(crate) fn apply_runtime_uninstall( } } + // Decided from the folder actually removed, not from the plan: the record + // is only stale once its files are gone. + let lemonade_record = removed_install_root + .as_deref() + .map_or(RecordCleanup::NotAffected, |removed| { + rocm_engine_lemonade::forget_install_record_inside(paths, removed) + }); + // An unreadable record says nothing about this runtime unless the runtime + // could have held Lemonade; otherwise the line would repeat on every runtime + // uninstall for as long as the record stays unreadable. + let lemonade_record = match lemonade_record { + RecordCleanup::Unreadable(_) if !holds_lemonade_tree => RecordCleanup::NotAffected, + other => other, + }; + Ok(RuntimeUninstallResult { runtime_id: manifest.runtime_id, runtime_key: manifest.runtime_key, registry_path, removed_install_root, + lemonade_record, read_only: manifest.read_only, manifest_mismatch: matches!(install_root_decision, InstallRootDecision::ManifestMismatch), was_active, @@ -357,6 +447,180 @@ mod tests { use crate::test_support::write_test_pip_runtime; use crate::tests::test_paths; + /// While a runtime is active, `rocm serve` installs Lemonade inside that + /// runtime's folder and records it there. Uninstalling the runtime deletes + /// the folder; the record must go with it, or Lemonade keeps pointing at + /// files that no longer exist. A record pointing elsewhere is left alone. + #[test] + fn runtime_uninstall_clears_a_lemonade_record_pointing_into_the_removed_folder() -> Result<()> { + let (root, paths) = test_paths("runtime-uninstall-lemonade"); + let held = write_test_pip_runtime( + &paths, + "release-pip-gfx1151-7-14-1", + "therock-release:gfx1151", + "7.14.1", + 20, + )?; + let other = write_test_pip_runtime( + &paths, + "release-pip-gfx1151-7-13-0", + "therock-release:gfx1151", + "7.13.0", + 10, + )?; + let tree = rocm_engine_lemonade::install_tree_root( + &paths, + Some(&held.install_root.join("engines")), + ); + crate::test_support::plant_lemonade_tree(&tree, 16)?; + crate::test_support::record_lemonade_install(&paths, &tree)?; + let mut config = RocmCliConfig::default(); + + // Removing a runtime that does not hold the tree keeps the record. + let removed = uninstall_runtime(&paths, &mut config, &other.runtime_key)?; + assert_eq!(removed.lemonade_record, RecordCleanup::NotAffected); + assert_eq!( + rocm_engine_lemonade::recorded_runtime_dir(&paths), + Some(tree.join("runtime")) + ); + + let plan = plan_runtime_uninstall(&paths, &config, &held.runtime_key)?; + assert_eq!(plan.lemonade_install_inside, Some(tree.join("runtime"))); + let removed = apply_runtime_uninstall(&paths, &mut config, plan)?; + + assert_eq!( + removed.lemonade_record, + RecordCleanup::Cleared(tree.join("runtime")) + ); + assert!(!tree.exists()); + assert_eq!(rocm_engine_lemonade::recorded_runtime_dir(&paths), None); + let line = lemonade_record_report_line(&removed.lemonade_record).unwrap(); + assert!( + line.contains(&tree.join("runtime").display().to_string()) + && line.ends_with("Lemonade is no longer recorded as installed"), + "{line}" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// A record that cannot be read may well name the runtime being removed, so + /// the uninstall cannot tell whether it is left dangling. When the runtime + /// held a Lemonade tree it must say so rather than treat the record as + /// absent, name the cheap command that rewrites it (a plain + /// `rocm engines install lemonade`, shown to do so by + /// `a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree` in + /// the engine), not the wiping `--reinstall`, and leave the file alone. + #[test] + fn runtime_uninstall_reports_a_lemonade_record_it_could_not_read() -> Result<()> { + let (root, paths) = test_paths("runtime-uninstall-lemonade-unreadable"); + let held = write_test_pip_runtime( + &paths, + "release-pip-gfx1151-7-14-1", + "therock-release:gfx1151", + "7.14.1", + 20, + )?; + crate::test_support::plant_lemonade_tree( + &rocm_engine_lemonade::install_tree_root( + &paths, + Some(&held.install_root.join("engines")), + ), + 16, + )?; + let record = paths.engine_manifests_dir("lemonade").join("runtime.json"); + fs::create_dir_all(record.parent().expect("record has a parent"))?; + fs::write(&record, b"{ not json")?; + let mut config = RocmCliConfig::default(); + + let removed = uninstall_runtime(&paths, &mut config, &held.runtime_key)?; + + assert!( + matches!(removed.lemonade_record, RecordCleanup::Unreadable(_)), + "{:?}", + removed.lemonade_record + ); + let line = lemonade_record_report_line(&removed.lemonade_record).unwrap(); + assert!(line.contains("could not be read"), "{line}"); + assert!( + line.contains("`rocm engines install lemonade` rewrites it"), + "{line}" + ); + assert!(!line.contains("--reinstall"), "{line}"); + assert!(record.exists()); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// The same unreadable record must not be reported on a runtime that never + /// held Lemonade: nothing about that uninstall depends on it, and the line + /// would otherwise repeat on every runtime uninstall. + #[test] + fn an_unreadable_lemonade_record_is_not_reported_for_a_runtime_without_lemonade() -> Result<()> + { + let (root, paths) = test_paths("runtime-uninstall-lemonade-unrelated"); + let runtime = write_test_pip_runtime( + &paths, + "release-pip-gfx1151-7-14-1", + "therock-release:gfx1151", + "7.14.1", + 20, + )?; + let record = paths.engine_manifests_dir("lemonade").join("runtime.json"); + fs::create_dir_all(record.parent().expect("record has a parent"))?; + fs::write(&record, b"{ not json")?; + let mut config = RocmCliConfig::default(); + + let plan = plan_runtime_uninstall(&paths, &config, &runtime.runtime_key)?; + assert!(!plan.holds_lemonade_tree); + let removed = apply_runtime_uninstall(&paths, &mut config, plan)?; + + assert_eq!(removed.lemonade_record, RecordCleanup::NotAffected); + assert_eq!(lemonade_record_report_line(&removed.lemonade_record), None); + assert!( + removed.removed_install_root.is_some(), + "the runtime was removed" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// Each outcome's line says only what happened. A failed deletion must + /// name the record to delete and must not claim Lemonade is no longer + /// recorded; no outcome for an unrelated runtime prints anything. (That + /// deleting the named record really clears the condition is shown by + /// `a_record_that_cannot_be_deleted_is_reported_not_swallowed` in the + /// engine.) + #[test] + fn the_lemonade_record_line_matches_what_happened() { + assert_eq!( + lemonade_record_report_line(&RecordCleanup::NotAffected), + None + ); + let cleared = + lemonade_record_report_line(&RecordCleanup::Cleared(PathBuf::from("/rt/lemonade"))) + .unwrap(); + assert!( + cleared.contains("no longer recorded as installed"), + "{cleared}" + ); + let failed = lemonade_record_report_line(&RecordCleanup::Failed { + runtime_dir: PathBuf::from("/rt/lemonade"), + record: PathBuf::from("/data/engines/lemonade/manifests/runtime.json"), + error: "sharing violation".to_owned(), + }) + .unwrap(); + assert!(failed.contains("sharing violation"), "{failed}"); + assert!( + failed.contains("delete /data/engines/lemonade/manifests/runtime.json"), + "{failed}" + ); + assert!(!failed.contains("no longer recorded"), "{failed}"); + } + #[test] fn runtime_uninstall_unregisters_read_only_runtime_without_deleting_external_root() -> Result<()> { diff --git a/apps/rocm/src/runtimes_cmd.rs b/apps/rocm/src/runtimes_cmd.rs index 731c1afee1..0a43821f68 100644 --- a/apps/rocm/src/runtimes_cmd.rs +++ b/apps/rocm/src/runtimes_cmd.rs @@ -55,7 +55,8 @@ pub(crate) use crate::runtime_registry::{ validate_runtime_manifest_for_activation, }; pub(crate) use crate::runtime_uninstall::{ - InstallRootDecision, should_remove_runtime_install_root, uninstall_runtime, + InstallRootDecision, lemonade_record_report_line, should_remove_runtime_install_root, + uninstall_runtime, }; use crate::runtime_uninstall::{ RuntimeUninstallConfirmation, apply_runtime_uninstall, @@ -257,6 +258,9 @@ pub(crate) fn runtimes(command: Option) -> Result<()> { report = report.detail("folder_removed", "no"); } } + if let Some(line) = lemonade_record_report_line(&result.lemonade_record) { + report = report.detail("lemonade", line); + } if result.default_runtime_cleared { report = report .detail("default_runtime", "cleared") diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 0ffd69f010..95c22d6905 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -866,8 +866,17 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { // test reaches Lemonade's backend boundary without real GPU hardware, and the // E2E-only OOM-launch fault injection likewise stands in for the GPU it does // not have. + // + // The scripted free-space override opens the same waiver: it exists to refuse a + // Lemonade install on a drive that cannot hold it, which `rocm serve` only + // reaches past this bail. let scripted_backend_failure = cfg!(feature = "e2e-test-hooks") - && std::env::var_os("ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE").is_some(); + && [ + "ROCM_E2E_LEMONADE_BACKEND_INSTALL_FAILURE", + "ROCM_E2E_LEMONADE_FREE_SPACE_BYTES", + ] + .iter() + .any(|variable| std::env::var_os(variable).is_some()); if !cpu_only && !scripted_backend_failure && !e2e_simulate_oom_launch() diff --git a/apps/rocm/src/storage.rs b/apps/rocm/src/storage.rs index 3983cca0ba..b930874699 100644 --- a/apps/rocm/src/storage.rs +++ b/apps/rocm/src/storage.rs @@ -370,10 +370,94 @@ pub(crate) struct RuntimeUsage { pub hold_reason: Option, } +/// Whether Lemonade's install record points at a tree. +/// +/// `Unused` means unused right now: the record points at another tree that +/// really holds an install. It says nothing about the future. Lemonade installs +/// into the active ROCm install's folder, so a copy inside a registered ROCm +/// install is reused the moment that install becomes active, and only the data +/// folder's copy, which no ROCm install owns, is ever offered as reclaimable. A +/// record that names a folder with nothing in it, or that cannot be read, leaves +/// it unknown which copy works, so no tree is called unused. In every case the +/// advice must not tell a user to delete a copy that a later serve would reuse, +/// which would force a reinstall of roughly 25 GiB. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum EngineInstallStatus { + /// The install record points here, and the install is there. + InUse, + /// The record points at another tree that holds an install, so nothing uses + /// this one. + Unused, + /// The record names a folder that holds no install, so no copy is in use. + RecordWithoutInstall, + /// A record exists but could not be read, so it is not known which copy is + /// in use. + RecordUnreadable, + /// Lemonade has no install record at all. + NotRecorded, +} + +/// Decide a tree's status from the install record. +fn classify_engine_install( + record: &rocm_engine_lemonade::InstallRecord, + tree: &Path, +) -> EngineInstallStatus { + use rocm_engine_lemonade::InstallRecord; + match record { + InstallRecord::NotRecorded => EngineInstallStatus::NotRecorded, + InstallRecord::Unreadable(_) => EngineInstallStatus::RecordUnreadable, + InstallRecord::Recorded { + holds_install: false, + .. + } => EngineInstallStatus::RecordWithoutInstall, + InstallRecord::Recorded { runtime_dir, .. } + if rocm_core::runtime_path_is_same_or_inside(runtime_dir, tree) => + { + EngineInstallStatus::InUse + } + InstallRecord::Recorded { .. } => EngineInstallStatus::Unused, + } +} + +/// One Lemonade install tree found on disk. +/// +/// A self-managed engine installs into the active ROCm runtime's own folder +/// when one is active, and into the data folder otherwise, so the same host +/// can carry a full copy in each. Before this row existed the only trace of a +/// second copy was a larger runtime size and data-folder total. +#[derive(Debug, Clone, Serialize)] +pub(crate) struct EngineInstallUsage { + pub engine: String, + pub path: PathBuf, + /// The runtime whose folder holds this tree, `None` for the data folder. + #[serde(skip_serializing_if = "Option::is_none")] + pub runtime_key: Option, + pub size_bytes: u64, + pub size_complete: bool, + pub status: EngineInstallStatus, + /// Whether the report offers this copy for deletion: unused right now and + /// not inside a registered ROCm install, where it would be reused when that + /// install becomes active. + pub reclaimable: bool, + /// Why the status could not be decided, for the two record-problem states. + #[serde(skip_serializing_if = "Option::is_none")] + pub record_note: Option, + /// Whether `rocm runtimes uninstall` deletes this tree along with its + /// runtime. Decided by the same rule uninstall applies + /// (`should_remove_runtime_install_root`): an external or imported runtime + /// is only unregistered, so its Lemonade copy stays. `false` for the data + /// folder's tree, which belongs to no runtime. + pub removed_with_runtime: bool, +} + #[derive(Debug, Clone, Serialize)] pub(crate) struct StorageReport { pub runtimes: Vec, pub runtimes_total_bytes: u64, + /// Engine installs ROCm CLI made. A tree inside a runtime folder is also + /// part of that runtime's size above. + pub engines: Vec, pub rocm_cli: Vec, /// Caches ROCm CLI uses but does not own. Reported, never deleted. pub shared_with_other_tools: Vec, @@ -419,6 +503,81 @@ fn shared_model_cache_dir() -> Option { Some(home.join(".cache").join("huggingface")) } +/// Every Lemonade tree ROCm CLI could have installed: the data folder's, and +/// one per registered runtime (where it goes while that runtime is active). +/// Only the install parts are measured (`install_tree_dirs`), not the record, +/// logs and server state that share the data folder's tree. +fn lemonade_install_usages( + paths: &AppPaths, + manifests: &[therock::InstalledRuntimeManifest], +) -> Vec { + let record = rocm_engine_lemonade::install_record(paths); + let record_note = match &record { + rocm_engine_lemonade::InstallRecord::Unreadable(error) => Some(format!( + "Lemonade's install record could not be read ({error}), so it is not known which copy is in use" + )), + rocm_engine_lemonade::InstallRecord::Recorded { + runtime_dir, + holds_install: false, + } => Some(format!( + "Lemonade's install record names {}, which holds no install, so no copy is in use", + runtime_dir.display() + )), + _ => None, + }; + let candidates = std::iter::once(( + None, + rocm_engine_lemonade::install_tree_root(paths, None), + false, + )) + .chain(manifests.iter().map(|manifest| { + ( + Some(manifest.runtime_key.clone()), + rocm_engine_lemonade::install_tree_root( + paths, + Some(&manifest.install_root.join("engines")), + ), + should_remove_runtime_install_root(manifest) + .is_ok_and(InstallRootDecision::should_remove), + ) + })); + candidates + .filter_map(|(runtime_key, path, removed_with_runtime)| { + let parts = rocm_engine_lemonade::install_tree_dirs(&path); + let present = parts + .iter() + .filter(|part| part.exists()) + .collect::>(); + if present.is_empty() { + return None; + } + let (size_bytes, size_complete) = + present + .iter() + .fold((0_u64, true), |(bytes, complete), part| { + let measured = measure_path(part); + ( + bytes.saturating_add(measured.bytes), + complete && measured.complete, + ) + }); + let status = classify_engine_install(&record, &path); + let reclaimable = status == EngineInstallStatus::Unused && runtime_key.is_none(); + Some(EngineInstallUsage { + engine: "lemonade".to_owned(), + path, + runtime_key, + size_bytes, + size_complete, + status, + reclaimable, + record_note: record_note.clone(), + removed_with_runtime, + }) + }) + .collect() +} + pub(crate) fn build_report(paths: &AppPaths, config: &RocmCliConfig) -> Result { let manifests = therock::load_runtime_manifests(paths)?; let marker = read_active_runtime_marker(paths); @@ -532,9 +691,12 @@ pub(crate) fn build_report(paths: &AppPaths, config: &RocmCliConfig) -> Result String { format_bytes(report.runtimes_total_bytes) ); + let _ = writeln!(output); + let _ = writeln!(output, "Engine installs:"); + if report.engines.is_empty() { + let _ = writeln!(output, " none"); + } + for usage in &report.engines { + let size = if usage.size_complete { + format_bytes(usage.size_bytes) + } else { + format!( + "{} or more (part unreadable)", + format_bytes(usage.size_bytes) + ) + }; + let location = usage.runtime_key.as_deref().map_or_else( + || "data folder".to_owned(), + |key| format!("inside ROCm install {key}"), + ); + let _ = writeln!( + output, + " - {} ({location}): {size} ({})", + usage.engine, + usage.path.display() + ); + let status = match usage.status { + EngineInstallStatus::InUse => "in use", + EngineInstallStatus::Unused => "not used right now; Lemonade runs from another install", + EngineInstallStatus::RecordWithoutInstall => { + "not known to be used; the install record names a folder with no install" + } + EngineInstallStatus::RecordUnreadable => { + "not known to be used; the install record could not be read" + } + EngineInstallStatus::NotRecorded => "not recorded as installed", + }; + let _ = writeln!(output, " status: {status}"); + if usage.reclaimable { + let _ = writeln!( + output, + " note: reclaimable; Lemonade installs into the active ROCm install's folder, so this copy is only used while no ROCm install is active. Nothing removes it automatically: delete its runtime, downloads and extract folders to free the space" + ); + } else if usage.status == EngineInstallStatus::Unused { + let _ = writeln!( + output, + " note: reused if this ROCm install becomes the active one; not marked reclaimable" + ); + } + if let Some(note) = usage.record_note.as_deref() { + let _ = writeln!(output, " note: {note}; not marked reclaimable"); + } + if usage.runtime_key.is_some() { + let note = if usage.removed_with_runtime { + "also counted in that ROCm install's size above, and removed with it" + } else { + "also counted in that ROCm install's size above; ROCm CLI does not delete that install's folder (external or imported), so uninstalling it leaves this copy" + }; + let _ = writeln!(output, " note: {note}"); + } + } + let _ = writeln!(output); let _ = writeln!(output, "ROCm CLI folders:"); for usage in &report.rocm_cli { @@ -959,6 +1181,11 @@ pub(crate) fn storage(command: Option) -> Result<()> { format_bytes(entry.size_bytes), path.display() ); + if let Some(line) = + crate::runtimes_cmd::lemonade_record_report_line(&result.lemonade_record) + { + println!(" lemonade: {line}"); + } } else { println!( "removed the registry entry for {}, but left its folder {} in place", @@ -1468,6 +1695,299 @@ mod tests { Ok(()) } + /// The release-test layout (ROCMAI-825): Lemonade installed into the data + /// folder, then a runtime activated and `rocm serve` installing a second full + /// copy inside it, with the record now pointing at the second. Both trees + /// must be listed with their sizes, and the data folder's flagged as unused. + #[test] + fn report_lists_both_lemonade_installs_and_flags_the_unused_one() -> Result<()> { + let (root, paths) = test_paths("report-lemonade"); + let runtime = install_fixture(&paths, manifest("in-use", "gfx1151", "7.14.1", 20), 4096)?; + let base = rocm_engine_lemonade::install_tree_root(&paths, None); + let per_runtime = rocm_engine_lemonade::install_tree_root( + &paths, + Some(&runtime.install_root.join("engines")), + ); + crate::test_support::plant_lemonade_tree(&base, 3000)?; + crate::test_support::plant_lemonade_tree(&per_runtime, 5000)?; + crate::test_support::record_lemonade_install(&paths, &per_runtime)?; + // Record, logs and server state share the data folder's tree; they are + // not part of the install and must not be counted as reclaimable. + std::fs::create_dir_all(base.join("state"))?; + std::fs::write(base.join("state").join("big.log"), vec![0_u8; 100_000])?; + + let report = build_report(&paths, &RocmCliConfig::default())?; + + assert_eq!(report.engines.len(), 2, "{:?}", report.engines); + let base_row = &report.engines[0]; + assert_eq!(base_row.path, base); + assert_eq!(base_row.runtime_key, None); + assert_eq!(base_row.status, EngineInstallStatus::Unused); + assert!(base_row.size_bytes >= 3000 && base_row.size_bytes < 100_000); + let runtime_row = &report.engines[1]; + assert_eq!(runtime_row.path, per_runtime); + assert_eq!(runtime_row.runtime_key.as_deref(), Some("in-use")); + assert_eq!(runtime_row.status, EngineInstallStatus::InUse); + assert!(runtime_row.size_bytes >= 5000); + + let rendered = render_report(&report); + assert!( + rendered.contains("- lemonade (data folder): "), + "{rendered}" + ); + assert!( + rendered.contains("status: not used right now; Lemonade runs from another install"), + "{rendered}" + ); + assert!( + rendered.contains(&format!( + "- lemonade (inside ROCm install in-use): {} ({})", + format_bytes(runtime_row.size_bytes), + per_runtime.display() + )), + "{rendered}" + ); + assert!(rendered.contains("note: reclaimable"), "{rendered}"); + assert!(base_row.reclaimable && !runtime_row.reclaimable); + assert!( + rendered + .contains("also counted in that ROCm install's size above, and removed with it"), + "a runtime ROCm CLI owns is deleted with its copy:\n{rendered}" + ); + + // With no record at all, nothing is called unused: the next install may + // well reuse a current tree. + std::fs::remove_file(paths.engine_manifests_dir("lemonade").join("runtime.json"))?; + let report = build_report(&paths, &RocmCliConfig::default())?; + assert!( + report + .engines + .iter() + .all(|row| row.status == EngineInstallStatus::NotRecorded) + ); + assert!(!render_report(&report).contains("note: reclaimable")); + + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// Two registered runtimes that each hold a Lemonade copy, with the record + /// pointing at one. The other copy is unused right now but is reused as soon + /// as its runtime becomes active, so it must not be offered for deletion: + /// following that advice would force a ~25 GiB reinstall. Only the data + /// folder's copy, which no runtime owns, is reclaimable. + #[test] + fn a_copy_inside_another_registered_runtime_is_not_offered_for_deletion() -> Result<()> { + let (root, paths) = test_paths("report-lemonade-two-runtimes"); + let active = install_fixture(&paths, manifest("active", "gfx1151", "7.14.1", 20), 4096)?; + let idle = install_fixture(&paths, manifest("idle", "gfx1151", "7.13.0", 10), 4096)?; + let base = rocm_engine_lemonade::install_tree_root(&paths, None); + let tree_of = |runtime: &therock::InstalledRuntimeManifest| { + rocm_engine_lemonade::install_tree_root( + &paths, + Some(&runtime.install_root.join("engines")), + ) + }; + crate::test_support::plant_lemonade_tree(&base, 3000)?; + crate::test_support::plant_lemonade_tree(&tree_of(&active), 5000)?; + crate::test_support::plant_lemonade_tree(&tree_of(&idle), 6000)?; + crate::test_support::record_lemonade_install(&paths, &tree_of(&active))?; + + let report = build_report(&paths, &RocmCliConfig::default())?; + + let row = |runtime_key: Option<&str>| { + report + .engines + .iter() + .find(|row| row.runtime_key.as_deref() == runtime_key) + .expect("the copy is listed") + }; + assert_eq!(row(Some("active")).status, EngineInstallStatus::InUse); + assert_eq!(row(Some("idle")).status, EngineInstallStatus::Unused); + assert!( + !row(Some("idle")).reclaimable, + "reused when `idle` is activated" + ); + assert!( + row(None).reclaimable, + "no runtime owns the data folder's copy" + ); + let rendered = render_report(&report); + assert_eq!( + rendered.matches("note: reclaimable").count(), + 1, + "only the data folder's copy is offered:\n{rendered}" + ); + assert!( + rendered.contains( + "note: reused if this ROCm install becomes the active one; not marked reclaimable" + ), + "{rendered}" + ); + // Reporting only: every copy is still on disk. + assert!(tree_of(&idle).join("runtime").exists() && base.join("runtime").exists()); + + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// A record that names a runtime folder holding no install (a runtime + /// uninstalled by a build that left the record behind) means nothing is in + /// use. The surviving data-folder copy is then the only one a later serve can + /// reuse, so the report must not call it unused or reclaimable: following that + /// advice would force a reinstall of roughly 25 GiB. + #[test] + fn a_record_naming_a_folder_with_no_install_does_not_make_the_survivor_reclaimable() + -> Result<()> { + let (root, paths) = test_paths("report-lemonade-dangling"); + let base = rocm_engine_lemonade::install_tree_root(&paths, None); + crate::test_support::plant_lemonade_tree(&base, 3000)?; + let gone = paths + .data_dir + .join("runtimes") + .join("wheel") + .join("removed"); + crate::test_support::record_lemonade_install( + &paths, + &gone.join("engines").join("lemonade"), + )?; + + let report = build_report(&paths, &RocmCliConfig::default())?; + + assert_eq!(report.engines.len(), 1, "{:?}", report.engines); + let row = &report.engines[0]; + assert_eq!(row.status, EngineInstallStatus::RecordWithoutInstall); + let rendered = render_report(&report); + assert!(!rendered.contains("note: reclaimable"), "{rendered}"); + assert!(!rendered.contains("status: not used"), "{rendered}"); + assert!( + rendered.contains("install record names a folder with no install"), + "{rendered}" + ); + assert!( + rendered.contains("holds no install, so no copy is in use; not marked reclaimable") + ); + + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// An unparseable record says nothing about which copy works, so it is not + /// "no record" and no copy is called unused. + #[test] + fn an_unreadable_record_leaves_every_copy_unmarked() -> Result<()> { + let (root, paths) = test_paths("report-lemonade-unreadable"); + let base = rocm_engine_lemonade::install_tree_root(&paths, None); + crate::test_support::plant_lemonade_tree(&base, 3000)?; + let record = paths.engine_manifests_dir("lemonade").join("runtime.json"); + std::fs::create_dir_all(record.parent().expect("record has a parent"))?; + std::fs::write(&record, b"{ not json")?; + + let report = build_report(&paths, &RocmCliConfig::default())?; + + assert_eq!( + report.engines[0].status, + EngineInstallStatus::RecordUnreadable + ); + let rendered = render_report(&report); + assert!(!rendered.contains("note: reclaimable"), "{rendered}"); + assert!(rendered.contains("could not be read"), "{rendered}"); + assert!( + !rendered.contains("not recorded as installed"), + "{rendered}" + ); + + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + + /// The status rule on its own, over every record state and tree position. + #[test] + fn engine_install_status_follows_the_record() { + use rocm_engine_lemonade::InstallRecord; + let tree = PathBuf::from("data").join("engines").join("lemonade"); + let inside = InstallRecord::Recorded { + runtime_dir: tree.join("runtime"), + holds_install: true, + }; + let elsewhere = InstallRecord::Recorded { + runtime_dir: PathBuf::from("elsewhere").join("runtime"), + holds_install: true, + }; + let empty = InstallRecord::Recorded { + runtime_dir: PathBuf::from("elsewhere").join("runtime"), + holds_install: false, + }; + assert_eq!( + classify_engine_install(&inside, &tree), + EngineInstallStatus::InUse + ); + assert_eq!( + classify_engine_install(&elsewhere, &tree), + EngineInstallStatus::Unused + ); + assert_eq!( + classify_engine_install(&empty, &tree), + EngineInstallStatus::RecordWithoutInstall + ); + // A record at this tree whose server binary is gone is not "in use". + let hollow = InstallRecord::Recorded { + runtime_dir: tree.join("runtime"), + holds_install: false, + }; + assert_eq!( + classify_engine_install(&hollow, &tree), + EngineInstallStatus::RecordWithoutInstall + ); + assert_eq!( + classify_engine_install(&InstallRecord::Unreadable("x".to_owned()), &tree), + EngineInstallStatus::RecordUnreadable + ); + assert_eq!( + classify_engine_install(&InstallRecord::NotRecorded, &tree), + EngineInstallStatus::NotRecorded + ); + } + + /// Uninstalling an external or imported runtime only unregisters it, so its + /// Lemonade copy stays. The row must not say it is "removed with" the + /// runtime; the claim comes from the rule uninstall itself applies. + #[test] + fn a_read_only_runtimes_lemonade_copy_is_not_said_to_be_removed_with_it() -> Result<()> { + let (root, paths) = test_paths("report-lemonade-read-only"); + let mut record = manifest("external", "gfx1151", "7.14.1", 20); + record.read_only = true; + let runtime = install_fixture(&paths, record, 4096)?; + let tree = rocm_engine_lemonade::install_tree_root( + &paths, + Some(&runtime.install_root.join("engines")), + ); + crate::test_support::plant_lemonade_tree(&tree, 5000)?; + crate::test_support::record_lemonade_install(&paths, &tree)?; + + let report = build_report(&paths, &RocmCliConfig::default())?; + + let row = report + .engines + .iter() + .find(|row| row.runtime_key.as_deref() == Some("external")) + .expect("the runtime's copy is listed"); + assert!(!row.removed_with_runtime); + let rendered = render_report(&report); + assert!(!rendered.contains("and removed with it"), "{rendered}"); + assert!( + rendered.contains("does not delete that install's folder (external or imported)"), + "{rendered}" + ); + // The claim matches what uninstall then does to that copy. + let mut config = RocmCliConfig::default(); + crate::runtimes_cmd::uninstall_runtime(&paths, &mut config, "external")?; + assert!(tree.join("runtime").exists(), "uninstall deleted the copy"); + + let _ = std::fs::remove_dir_all(root); + Ok(()) + } + /// `rocm serve --managed` leaves one JSON record per launch, kept after the /// server exits, and no `rocm storage` command removes them. The report /// never named the folder, so a user asking what is on disk could not see diff --git a/apps/rocm/src/test_support.rs b/apps/rocm/src/test_support.rs index 1987de3679..caaa736810 100644 --- a/apps/rocm/src/test_support.rs +++ b/apps/rocm/src/test_support.rs @@ -485,3 +485,43 @@ mod tests { assert_eq!(raised_case_count(Some("lots"), 256), 256); } } + +/// Plant an extracted Lemonade tree under `tree_root` the way an install leaves +/// one: the server binary and `payload_bytes` of backend files. Black-box on +/// purpose (plain files, not the engine's own writer), so a test reads it the +/// way the CLI reads a real install. +pub(crate) fn plant_lemonade_tree(tree_root: &std::path::Path, payload_bytes: usize) -> Result<()> { + let runtime_dir = tree_root.join("runtime"); + let backend = runtime_dir.join("bin").join("llamacpp").join("rocm-stable"); + fs::create_dir_all(&backend)?; + let lemond = if rocm_core::runtime_is_windows() { + "lemond.exe" + } else { + "lemond" + }; + fs::write(runtime_dir.join(lemond), b"lemond")?; + fs::write(backend.join("payload.bin"), vec![0_u8; payload_bytes])?; + Ok(()) +} + +/// Write Lemonade's install record pointing at `tree_root`'s runtime, in the +/// schema the engine writes (`/engines/lemonade/manifests/runtime.json`). +pub(crate) fn record_lemonade_install(paths: &AppPaths, tree_root: &std::path::Path) -> Result<()> { + let runtime_dir = tree_root.join("runtime"); + let record = paths.engine_manifests_dir("lemonade").join("runtime.json"); + fs::create_dir_all(record.parent().expect("record has a parent"))?; + fs::write( + &record, + serde_json::to_vec_pretty(&serde_json::json!({ + "env_id": "lemonade-embeddable-test", + "version": "test", + "runtime_dir": runtime_dir, + "lemond": runtime_dir.join("lemond"), + "lemonade": runtime_dir.join("lemonade"), + "backend_recipe": "llamacpp", + "backend_name": "rocm", + "installed_at_unix_ms": 0, + }))?, + )?; + Ok(()) +} diff --git a/docs/architecture.md b/docs/architecture.md index 7b35533244..d563babc3a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,8 +85,8 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`runtime_install_finalize/torch_alignment.rs`](../apps/rocm/src/runtime_install_finalize/torch_alignment.rs) — full domain extraction, a child of `runtime_install_finalize.rs` that depends on its parent's device check (`RuntimeDeviceCheck`, `device_check_verdict`) and on `runtime_manifest_for_selector` in [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): what to do about the torch an engine install finds in a managed runtime. `plan_torch_alignment` decides whether the SDK's torch must be realigned to the engine's pin, `align_runtime_torch` runs the realignment, `classify_retained_torch` decides when a torch that already works is kept as an intended divergence instead, and the `render_*` and `report_*` functions with `TorchAlignment` and `TorchRetention` print and log the outcome. The manifest readers (`sdk_torch_build_for_key`, `sdk_torch_build_from_manifest`, `runtime_python_for_key`, `runtime_library_paths_for_key`, `runtime_index_url_for_key`) live here too. The parent stays the caller (`settle_engine_install` and `maybe_auto_install_sdk_preferred_engine`), so it could not be a crate-level sibling without a cycle with the parent's device check. The items the parent calls are `pub(super)`: `TorchAlignment`, `TorchRetention`, `report_torch_alignment`, `report_torch_retention`, `classify_retained_torch`, `retained_diverged_package`, `deliberately_diverged_package`, `sdk_torch_build_for_key`, `runtime_python_for_key` and `runtime_library_paths_for_key`; the rest is private. Its tests cover planning, rendering, the opt-out and the retention rules, and reach the parent's private engine-dependency check as a descendant of it. - [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) — full domain extraction, crate-level (split out of `therock.rs`): the installed-runtime manifest model (`InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `RocmSdkPackageProbe`, `RocmSdkLibraryProbe`, whose serde shape is the on-disk contract of the registry files and each install's `.rocm-cli-runtime.json`), where those files live (`runtime_registry_dir`, `runtime_manifest_path`), writing them (`write_runtime_manifest`, the one routine every way of registering a runtime goes through: an SDK install, `rocm runtimes import`, `rocm runtimes adopt` and the recovery of the runtime `rocm setup` created. It writes the registry entry, and for a runtime ROCm CLI owns (`InstalledRuntimeManifest::owns_install_root`, the same rule uninstall and prune use to decide whether they may delete the folder) the `.rocm-cli-runtime.json` in its install root; an imported or adopted runtime is someone else's folder, so it gets the registry entry only. The registry entry is written last, so a failed write of either file never leaves a new entry behind and an error that says a runtime could not be registered or restored is true. Its one parameter is `ManifestOverwrite`: an install replaces its own entry, import and adopt refuse an existing one unless `--replace` is given, and recovery refuses so it only fills a missing entry. Neither file is written atomically) and loading them (`load_runtime_manifests`, `load_runtime_manifests_reporting_unparsed`, `sort_manifests_newest_install_first`), the composition readers (`wheel_composition_includes_devel`, `wheel_composition_device_target`), the probe validator (`validate_rocm_sdk_runtime_probe`), and registry selection and validation (`current_runtime_manifest`, `default_runtime_id_matches`, `runtime_keys_text`, `select_runtime_manifest`, `runtime_usability_status`, `validate_runtime_manifest_for_activation`, and the wheel/tarball validators behind it; the tarball one asks `rocm_core::install_root_has_payload` whether the root holds a payload). Lives at the crate level and depends on neither `therock.rs` nor `runtimes_cmd.rs`, so both build on it. [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) still depends on `therock.rs` (it reaches the manifest types and loaders through `therock.rs`'s re-exports), but `therock.rs` no longer depends on `runtimes_cmd.rs` in production code (one of its tests still calls into it). It also holds `paths_equivalent` and `normalize_path_for_compare`, the path comparison that [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs), [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) and `runtimes_cmd.rs` need, and the on-disk active-runtime marker: `ActiveRuntimeMarker` (user state, so its serde shape is as fixed as the manifest's; its fields are `pub(crate)` because `runtimes_cmd.rs` builds it on activation and rollback), `active_runtime_marker_path` and `active_runtime_marker_matches`, which uninstall uses to decide whether the runtime it removes was the active one. `write_active_runtime_marker` stays in `runtimes_cmd.rs` with the activation transaction. `runtimes_cmd.rs` re-exports the selection and path-helper names that other modules import from it. Its own tests cover the manifest round-trip, the `devel`/composition readers, and runtime selector resolution. - [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) — full domain extraction; reads the live local servers at activation time, classifies each against the runtime being activated, renders the report, and moves stale ones with `--restart-services`. `fn runtimes()` lives in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) and `RuntimesCommand` in [`cli.rs`](../apps/rocm/src/cli.rs); this file owns `ServiceRuntimeState`/`RuntimeServiceEntry`/`FailedServiceRestart`/`UnconfirmedServiceRestart`/`ServiceRestartOutcome`/`RuntimeServiceReconciliation`/`ServiceRuntimePin`, reached from `runtimes_cmd.rs` via `crate::runtime_services`. -- [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): `rocm runtimes uninstall` and the storage pruning that removes a registered runtime. `plan_runtime_uninstall` builds the plan (`RuntimeUninstallPlan`, with the `InstallRootDecision` for its folder), `print_runtime_uninstall_plan` shows it, `confirm_and_revalidate_runtime_uninstall` asks and then checks that nothing changed while the prompt was open (`revalidate_runtime_uninstall_plan`), and `apply_runtime_uninstall` unregisters the runtime, clears the active and default state that named it and removes its install root when `should_remove_runtime_install_root` says ROCm CLI owns it (`local_runtime_manifest_matches`, `ensure_runtime_install_root_is_safe_to_remove`). `uninstall_runtime` is the non-interactive entry point. It reads the registry, the active-runtime marker and the path comparison from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and does not depend on `runtimes_cmd.rs`, which calls into it from `runtimes()` and re-exports `uninstall_runtime`, `should_remove_runtime_install_root` and `InstallRootDecision`. The planning and apply entry points, `RuntimeUninstallPlan`, `RuntimeUninstallConfirmation` and `RuntimeUninstallResult` (with its fields) are `pub(crate)` for `runtimes()`; `revalidate_runtime_uninstall_plan` and the folder-safety helpers are private. Its tests cover planning, revalidation, the folder-removal rules and uninstalling a read-only imported runtime; the uninstall tests that first activate a runtime stay with the activation tests in `runtimes_cmd.rs`. -- [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) — mechanical relocation, owns types (Phase 6c, ROCMAI-89): `runtimes()` and its activate/rollback handlers (the activation transaction, including `write_active_runtime_marker`), plus the runtime list rendering; the import and adopt handlers it dispatches to live in [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) and the uninstall plan, confirmation and apply in [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs); registry selection and manifest validation moved to [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), which this module re-exports for the callers that import them from here. `RuntimesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) (part of the `Command`/`Cli` clap definition, the same way `EnginesCommand` does). `RuntimeActivationResult` is `pub(crate)`, read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary`; `uninstall_runtime`, `should_remove_runtime_install_root` and `InstallRootDecision` (from `runtime_uninstall.rs`) and `ActiveRuntimeMarker` and `active_runtime_marker_path` (from `runtime_registry.rs`) are re-exported for [`storage.rs`](../apps/rocm/src/storage.rs) and [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs); `recover_setup_runtime_registration`, `newest_installed_runtime_manifest` and `setup_runtime_paths` (from `runtime_adopt.rs`) and `normalize_path_for_compare` (from `runtime_registry.rs`) are re-exported for the modules that import them from here, such as [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs), [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) and [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs). [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) is deliberately not one of them: it imports from `runtime_adopt.rs` and `runtime_registry.rs` directly, because going through this module would lead back to it via `runtime_services.rs`. The dependency on the sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) above runs the other way: that module calls into this one (`activate_runtime`, `newest_installed_runtime_manifest`, `recover_setup_runtime_registration`), while this module reaches it only from its test module. +- [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs) — full domain extraction, crate-level (split out of [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)): `rocm runtimes uninstall` and the storage pruning that removes a registered runtime. `plan_runtime_uninstall` builds the plan (`RuntimeUninstallPlan`, with the `InstallRootDecision` for its folder), `print_runtime_uninstall_plan` shows it, `confirm_and_revalidate_runtime_uninstall` asks and then checks that nothing changed while the prompt was open (`revalidate_runtime_uninstall_plan`), and `apply_runtime_uninstall` unregisters the runtime, clears the active and default state that named it and removes its install root when `should_remove_runtime_install_root` says ROCm CLI owns it (`local_runtime_manifest_matches`, `ensure_runtime_install_root_is_safe_to_remove`). `RuntimeUninstallPlan` also records whether Lemonade's install record points inside the runtime's folder (`lemonade_install_inside`, from `rocm_engine_lemonade::recorded_runtime_dir`), which `print_runtime_uninstall_plan` reports, and whether the folder holds a Lemonade tree at all (`holds_lemonade_tree`), which decides whether an unreadable record is worth reporting; `apply_runtime_uninstall` clears that record once the folder is removed (`RuntimeUninstallResult::lemonade_record`, a `RecordCleanup` that also carries a failed deletion or an unreadable record, worded by `lemonade_record_report_line`). `uninstall_runtime` is the non-interactive entry point. It reads the registry, the active-runtime marker and the path comparison from [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), and does not depend on `runtimes_cmd.rs`, which calls into it from `runtimes()` and re-exports `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line`. The planning and apply entry points, `RuntimeUninstallPlan`, `RuntimeUninstallConfirmation` and `RuntimeUninstallResult` (with its fields) are `pub(crate)` for `runtimes()`; `revalidate_runtime_uninstall_plan` and the folder-safety helpers are private. Its tests cover planning, revalidation, the folder-removal rules and uninstalling a read-only imported runtime; the uninstall tests that first activate a runtime stay with the activation tests in `runtimes_cmd.rs`. +- [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) — mechanical relocation, owns types (Phase 6c, ROCMAI-89): `runtimes()` and its activate/rollback handlers (the activation transaction, including `write_active_runtime_marker`), plus the runtime list rendering; the import and adopt handlers it dispatches to live in [`runtime_adopt.rs`](../apps/rocm/src/runtime_adopt.rs) and the uninstall plan, confirmation and apply in [`runtime_uninstall.rs`](../apps/rocm/src/runtime_uninstall.rs); registry selection and manifest validation moved to [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs), which this module re-exports for the callers that import them from here. `RuntimesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) (part of the `Command`/`Cli` clap definition, the same way `EnginesCommand` does). `RuntimeActivationResult` is `pub(crate)`, read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary`; `uninstall_runtime`, `should_remove_runtime_install_root`, `InstallRootDecision` and `lemonade_record_report_line` (from `runtime_uninstall.rs`) and `ActiveRuntimeMarker` and `active_runtime_marker_path` (from `runtime_registry.rs`) are re-exported for [`storage.rs`](../apps/rocm/src/storage.rs) and [`config_cmd.rs`](../apps/rocm/src/config_cmd.rs); `recover_setup_runtime_registration`, `newest_installed_runtime_manifest` and `setup_runtime_paths` (from `runtime_adopt.rs`) and `normalize_path_for_compare` (from `runtime_registry.rs`) are re-exported for the modules that import them from here, such as [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs), [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs), [`examine_cmd.rs`](../apps/rocm/src/examine_cmd.rs) and [`examine_report_text.rs`](../apps/rocm/src/examine_report_text.rs). [`engine_install_env.rs`](../apps/rocm/src/engine_install_env.rs) is deliberately not one of them: it imports from `runtime_adopt.rs` and `runtime_registry.rs` directly, because going through this module would lead back to it via `runtime_services.rs`. The dependency on the sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) above runs the other way: that module calls into this one (`activate_runtime`, `newest_installed_runtime_manifest`, `recover_setup_runtime_registration`), while this module reaches it only from its test module. - [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) — mechanical relocation, owns types (Phase 6a): `serve()`, plus the engine selection that runs ahead of it (`select_serve_engine`, `recipe_preferred_engine`, `serve_model_ref_for_engine`) and the AIM pre-flight that runs inside it: the Hugging Face token check, the GPU-family profile check and pin, the multi-GPU refusal (`aim_gpu_preflight`), and the board note. The engine-recipe overrides `serve()` applies to the launch hint live in [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs); this module imports them by name. It also holds `collect_serve_notes`, the GPU/device warnings folded into the interactive summary, which was relocated here from `main.rs` with its two tests. Its tests cover engine selection, the AIM pre-flight and the reuse pre-gate; the AIM tests build their hint with `protocol_engine_recipe_hint` and `engine_recipe_with_extra_env` from `serve_recipe.rs`, which `serve()` also calls. - [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs) — mechanical relocation, owns a type (split out of [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which imports the entry points by name): the pure transforms `serve()` applies to the selected engine's launch recipe hint. `protocol_engine_recipe_hint` maps the catalog's engine record onto the protocol hint, and `render_serve_engine_recipe_lines` prints it for the serve plan. The explicit controls are then layered on top: `--tool-call-parser` (`engine_recipe_with_tool_call_override`), `--gpu-memory-utilization` (`parse_gpu_memory_utilization`, `engine_recipe_with_gpu_memory_utilization_override`), `--temperature`/`--top-p`/`--max-tokens` (`ServeGenerationDefaults`, `engine_recipe_with_generation_defaults`) and `--engine-env` (`engine_recipe_with_extra_env`, `engine_env_ignored_note`, with the env names it refuses to override in `RESERVED_ENGINE_ENV_KEYS`); `engine_recipe_enables_tool_choice` reads the result back. The per-flag rewriters (`set_vllm_tool_call_parser`, `set_vllm_gpu_memory_utilization`, `set_vllm_override_generation_config`, `set_lemonade_generation_defaults`) are private. The entry points above, and `ServeGenerationDefaults`, are `pub(crate)` for `serve()` and for the AIM tests in `serve_cmd.rs`; `RESERVED_ENGINE_ENV_KEYS` is private. Among first-party crates the module depends only on `rocm_core` and `rocm_engine_protocol`, and nothing in it calls back into `serve_cmd.rs`. Its tests cover each override on its own. - [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs) — full domain extraction. @@ -99,7 +99,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service --follow` and `rocm services logs --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_tail_lines`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_tail_lines`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead. - [`setup_cmd.rs`](../apps/rocm/src/setup_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm setup`'s command handler (`setup()`, which runs `setup status` by default and `setup reset`), the text `rocm setup status` prints (`render_setup_status_text`) and the reset that clears the saved first-time prompt state (`reset_setup_prompt_state`). `SetupCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). `setup()` and `render_setup_status_text` are `pub(crate)` and re-imported in `main.rs`: `dispatch()` calls the first, and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only handling of `rocm setup` and `rocm setup status` reaches the second as `crate::render_setup_status_text`; `reset_setup_prompt_state` is private. The three regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` and `write_test_pip_runtime` through `crate::test_support::`. - [`shell_quote.rs`](../apps/rocm/src/shell_quote.rs) — shared CLI-output component (`shell_quote`): the CLI's name for `rocm-core`'s POSIX quoter, [`posix_shell_quote`](../crates/rocm-core/src/quoting.rs). `rocm-dash-tui` keeps a copy as `ui::exec::quote_display_arg` in [`exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs), since the crate-edge check forbids it `rocm-core`; a per-character walk and a property test here hold the two to the same output. It covers the remote `ssh` command line, the chat approval card and the plan for a free-form request, the install dry-run `command:` line, the [`service_hints.rs`](../apps/rocm/src/service_hints.rs) hints, the runtime-activation report, the `rocm services logs ` lines `rocm services` lists and the `rocm comfyui install --runtime-id` advice a failed ComfyUI reinstall prints — not every `rocm …` hint the CLI prints; older ones such as `rocm logs ` still interpolate their value bare. Also holds `has_terminal_control_character`, the predicate the chat tool-call guards refuse on. -- [`storage.rs`](../apps/rocm/src/storage.rs) — full domain extraction. +- [`storage.rs`](../apps/rocm/src/storage.rs) — full domain extraction. Its report also lists each Lemonade install tree (`EngineInstallUsage`, built by `lemonade_install_usages` from the read-only view in [`footprint.rs`](../engines/lemonade/src/footprint.rs)) with its size and whether the install record uses it (`EngineInstallStatus`, decided by `classify_engine_install`), and whether `rocm runtimes uninstall` would delete it with its runtime. - [`system_packages.rs`](../apps/rocm/src/system_packages.rs) — full domain extraction, a leaf of the crate (split out of [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)): the host system-package installs that `rocm install sdk` and `rocm engines install vllm` run through `sudo` apt, dnf or zypper. It owns the distro-aware OpenMPI, libatomic and libnuma flows (`ensure_openmpi_for_vllm`, `ensure_libatomic_for_torch`, `ensure_libnuma_for_torch`), the choice between printing the commands and running them (`SystemPackageInstallAction`, `system_package_install_action`), the guard that refuses an `apt-get install` which would remove ROCm packages, and reads the os-release every one of these plans is built from through `rocm_core::hardware_root::real_os_release`, which has no copy here. Production code uses only `std`, `anyhow` and `rocm_core`; the three `ensure_*` functions are `pub(crate)` and the rest is private. `runtime_install_finalize.rs` re-exports the three `ensure_*` names for [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs) and [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs). `SdkInstallConsents` stays in `runtime_install_finalize.rs` (it carries a `therock::SdkInstallConsent`); the functions here take its plain `system_packages` bool. `torch_runtime_dep_checks_disabled` is the E2E hook that skips the torch dependency checks; it only does anything under `e2e-test-hooks`. Its tests cover the consent decision (reaching `consents_for_install_sdk_argv` in `runtime_install_finalize.rs`'s test module) and, under `e2e-test-hooks` on Linux, that `rocm_core::hardware_root::real_os_release` and the three OpenMPI/libatomic/libnuma install hints ignore a simulated host root, and a scan that no source file of the binary reads os-release by hand. - [`therock.rs`](../apps/rocm/src/therock.rs) — full domain extraction; owns TheRock SDK install orchestration and is the parent of the `therock/` children below. It keeps `install_sdk` and `install_sdk_for_update` (with `SdkInstallRequest` and `SdkInstallResult`), the wheel and tarball install paths (`install_wheel_runtime`, `install_tarball_runtime`) with the provenance and notes they print (`render_canonical_provenance`, the plan itself being written by `therock/wheel_plan.rs`, the host-newer-ROCm and no-wheels messages, `quote_display_arg`), and the helpers that name an install and its folders: `runtime_key`, `wheel_runtime_key`, `managed_runtime_root`, `resolved_install_root`, `venv_python_path`, `platform_tarball_token` and `slugify`, plus `ensure_tarball_install_target_is_clean`, the refusal to extract a tarball into a root that already holds a payload. What "holds a payload" means is not defined here: it is `rocm_core::install_root_has_payload`, the one rule shared with `runtime_registry.rs` and `therock/update.rs`. The runtime-manifest model and registry loaders it writes through live in [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs); `therock.rs` re-exports the names other modules reach as `therock::...`: `InstalledRuntimeManifest`, `WheelRuntimeComposition`, `RocmSdkPythonProbe`, `load_runtime_manifests` and `validate_rocm_sdk_runtime_probe` from `runtime_registry.rs` (plus `RocmSdkLibraryProbe`, for tests only), and from its child modules the `pub(crate)` names each child's entry below lists. The children read five of them from here, so they stay in `therock.rs` for good: `slugify` ([`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) and [`therock/resolve/wheel.rs`](../apps/rocm/src/therock/resolve/wheel.rs)), `venv_python_path` ([`therock/python.rs`](../apps/rocm/src/therock/python.rs)), `platform_tarball_token` ([`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs)), and `runtime_key` and `wheel_runtime_key` ([`therock/update.rs`](../apps/rocm/src/therock/update.rs)). The `runtimes` subcommand's command enum (`RuntimesCommand`) lives in [`cli.rs`](../apps/rocm/src/cli.rs); its dispatch fn and most of its domain logic — rendering, activation, and manifest import — live in [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs). - [`therock/version.rs`](../apps/rocm/src/therock/version.rs) — full domain extraction, a leaf child of `therock.rs` (split out of it): the version grammar behind runtime selection and update checks. `parse_version` is the strict gate an install is routed on; `parse_version_for_ordering`, `compare_version_strings` and `version_relation` implement the PEP 440 ordering and the "is this a different version" question; `parse_host_version` reads a host-reported ROCm version through the same parser; `runtime_version_display`, `runtime_version_build_date`, `normalize_requested_build_date` and the calendar helpers behind them handle build dates; `is_stable_runtime_version` and the `ParsedVersion`/`VersionOrderKey`/`VersionStage` types back them. The items `therock.rs` and its property-test module use are `pub(super)` (including the `ParsedVersion` and `VersionOrderKey` fields they read); the parsing and calendar helpers behind them are private; and `runtime_version_display` is `pub(crate)`, re-exported by `therock.rs` for the other modules that render versions. Depends on nothing else in `therock`. `therock_version_proptests.rs` stays declared from `therock.rs` because it also reaches update, wheel and registry code. @@ -432,6 +432,7 @@ These modules are private `mod` declarations — [`lib.rs`](../crates/e2e-report - [`backend_alignment.rs`](../engines/lemonade/src/backend_alignment.rs) — ROCm-SDK-to-llama.cpp-backend Tier 1/Tier 2/revert alignment, including the `backend_versions.json` pin handling and the set-aside/restore of ROCm backend directories around a forced reinstall. It uses `llamacpp_backend.rs`; that module does not import it back. - [`device_policy.rs`](../engines/lemonade/src/device_policy.rs) — device-policy and GPU-selection handling: the GPU-required policy check (a CPU-only request is refused, never downgraded), `--device`/`--gpu` argument parsing, probing and resolving the GPU ordinal(s) to pin before a launch, the device availability reported by `detect`, and the device string reported for a loaded model. A leaf: it imports no other crate module. - [`direct_llama.rs`](../engines/lemonade/src/direct_llama.rs) — Hugging Face checkpoint/GGUF resolution and the direct-llama-server serve path (`serve_direct_llama_server`, which runs a packaged `llama-server` on a GGUF and waits for `/v1/models` to report the model). It imports `process.rs`; `process.rs` no longer imports it. +- [`footprint.rs`](../engines/lemonade/src/footprint.rs) — what an install costs on disk: the pre-install notice and free-space check, plus the read-only view of Lemonade's install trees and record that `apps/rocm` uses for `rocm storage` and `rocm runtimes uninstall` (re-exported at the crate root). - [`install.rs`](../engines/lemonade/src/install.rs) — embeddable-runtime download, extraction orchestration (the cache layer is in `archive_cache.rs`, the extractor in `archive_extract.rs`), and manifest persistence, plus the `LemonadeRuntime` handle and `resolve_runtime`, which wrap the persisted manifest (so `state.rs`, `process.rs` and `direct_llama.rs` reach it without importing `backend_alignment.rs`). - [`llamacpp_backend.rs`](../engines/lemonade/src/llamacpp_backend.rs) — llama.cpp backend discovery and selection: asking `lemonade backends` what the host supports, choosing the best GPU backend (`rocm` before `vulkan`, never `cpu`), installing it with one retry, the `ROCM_LLAMACPP_BACKEND_DIRS` name list, and `find_llama_server_binary_for_backend`. It imports `direct_llama.rs` for `find_binary_in`; `direct_llama.rs` does not import it. With this module the engine's modules form no import cycle outside test code. - [`process.rs`](../engines/lemonade/src/process.rs) — `lemond` spawn, env/path setup, readiness/health polling, and the background relaunch of the `serve-http` command (`serve_http_command_args`, `spawn_serve_http_background`). The direct `llama-server` serve loop is in `direct_llama.rs`. diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 9c06588f73..4092a27e7d 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -338,6 +338,36 @@ ROCm backend directory under `bin/llamacpp/`, and run - `therock.version` and `llamacpp.rocm-stable` in `backend_versions.json` hold the packaged values again, and the unrelated key is still there. +To check what a Lemonade install says about its size (no download needed for +the refusal): with a debug build, set +`ROCM_E2E_LEMONADE_FREE_SPACE_BYTES=1048576` (1 MiB free), then in a fresh data +folder run `rocm engines install lemonade --yes`. Expected result: + +- Before anything else it prints `Installing Lemonade into ` + and a line giving the ROCm backend's download (up to about 4.4 GiB), disk use + (up to about 25 GiB) and the free space on that drive. +- It then fails with `not enough free disk space to install Lemonade: ... + Nothing was downloaded.`, and `` has no `downloads` or `runtime` + subfolder. +- With `ROCM_E2E_LEMONADE_FREE_SPACE_BYTES=2147483648` (2 GiB) instead, it + warns that only 2.0 GiB is free for a backend of up to about 4.4 GiB and + carries on: Lemonade picks the backend after this point and a smaller one may + fit. (Stop it before the download if you do not want one.) + +With room to spare, on Windows from an interactive console: install Lemonade +with no runtime active, then `rocm install sdk`, then `rocm serve `. +Expected result: + +- The serve prints the install notice for the runtime's own + `engines\lemonade` folder, and says the data folder's copy is not reused. +- Lemonade's own backend download progress (`[1/2] ...`, `Total: ...`) appears + in the console instead of a silent wait. +- `rocm storage report` lists both copies under `Engine installs`, with the data + folder's one `not used right now` (and reclaimable) and the runtime's one `in use`. +- `rocm runtimes uninstall ` says Lemonade's install was inside the + runtime and removed with it, and a following `rocm storage report` shows the + remaining copy as `not recorded as installed`. + While the log stream is attached, verify detach and stop behave differently: - Press `Ctrl+D`. The stream ends and the shell prompt returns, printing a diff --git a/docs/testing.md b/docs/testing.md index 7498ca8a4b..229dfcabcb 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1433,6 +1433,40 @@ cargo test -p rocmd event_collector cargo test -p rocmd event_dispatcher ``` +## Lemonade Installs On Disk + +`features/lemonade_installs.feature` covers what a user is told about the +Lemonade copies ROCm CLI keeps (ROCMAI-825), all on the mock lane: + +- `lemonade-installs-01` and `-02` reach the engine's pre-install free-space + refusal from `rocm engines install lemonade` and from `rocm serve`, with 1 MiB + reported free (below what any Lemonade install needs). Room for Lemonade but + not for the multi-gigabyte ROCm backend only warns, which the unit tests cover + because a scenario would have to start the download. They use + `ROCM_E2E_LEMONADE_FREE_SPACE_BYTES`, compiled only into the + `rocm/e2e-test-hooks` build `cargo xtask e2e` makes, and check against the + disk that nothing was downloaded. That variable also waives serve's no-GPU + refusal, as serve-18's failure hook does. +- `lemonade-installs-03` plants a data-folder copy and a copy inside a runtime + and checks `rocm storage report` lists both and flags the unused one. +- `lemonade-installs-04` uninstalls that runtime and checks the install record + is cleared. +- `lemonade-installs-05` and `-06` cover the two cases the report must not + overstate: a record naming a folder with no install (the surviving copy is + not offered for deletion) and a read-only runtime (its copy is not said to be + removed with it, and uninstall then leaves it). A copy inside a registered + runtime is never marked reclaimable, because it is reused when that runtime + becomes active; two registered runtimes are covered by a unit test. + +Run them alone with `cargo xtask e2e -- -n "lemonade-installs-"`. + +The Windows console fix for the backend download's progress is unit-tested +through the real command builder (`backend_install_command` in +[process.rs](../engines/lemonade/src/process.rs)), which receives the console +state and the hide step, so a dropped or unconditional hide turns the test red. +Whether the output actually reaches a real console is a manual check in +[manual-testing.md](manual-testing.md). + ## Uninstall And Download Removal `rocm uninstall` and `rocm storage remove-downloads` both end in the same diff --git a/engines/lemonade/src/footprint.rs b/engines/lemonade/src/footprint.rs new file mode 100644 index 0000000000..9acdf26ccb --- /dev/null +++ b/engines/lemonade/src/footprint.rs @@ -0,0 +1,1010 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! What a Lemonade install costs on disk, and where its trees live. +//! +//! A fresh install is a multi-gigabyte download and a tree several times that +//! size: on a Windows gfx1151 host, Lemonade's `llamacpp:rocm` backend fetched +//! 4.7 GB (Lemonade's own `Total:` line) and the finished tree measured 24.3 GiB, +//! almost all of it the ROCm runtime Lemonade bundles with the backend. Before +//! this module the only thing a user saw was a one-line "Preparing..." and then, +//! from `rocm serve`, up to ten silent minutes. [`plan_install_preflight`] decides +//! whether an install is about to write a new tree, and [`InstallPreflight`] +//! renders what will be installed, where, how big it is, and refuses up front +//! when the drive cannot hold even the download. +//! +//! The second half is the read-only view `apps/rocm` needs to report Lemonade +//! trees (`rocm storage`) and to forget an install record whose tree a runtime +//! uninstall just deleted. The record and the tree layout belong to this crate, +//! so the knowledge of both stays here rather than being re-parsed by the CLI. + +use std::fs; +use std::path::{Path, PathBuf}; + +use rocm_core::{ + AppPaths, format_bytes, insufficient_space_message, runtime_path_is_same_or_inside, +}; + +use crate::install::{ + installed_runtime_version, lemonade_root, lemond_path_in, manifest_path, needs_extraction, + runtime_dir_in, +}; + +/// Download size of an install that fetches the llama.cpp ROCm backend: the +/// embeddable archive plus the backend. Lemonade printed `Total: 4.7 GB` +/// (decimal) for the backend on a Windows gfx1151 host; the embeddable archive +/// adds ~5 MB. +/// +/// An upper bound, never a promise. The backend is chosen after this preflight +/// (Lemonade reports per-GPU support, and `backend_alignment.rs` falls back to +/// `vulkan` where its ROCm build is unsupported), and the figure varies with the +/// GPU family and Lemonade's pins. So it is printed as "up to about" and only +/// ever warns; see [`MIN_INSTALL_BYTES`] for what can refuse. +pub(crate) const FRESH_INSTALL_DOWNLOAD_BYTES: u64 = 4_700_000_000; + +/// Disk space an install that fetches the ROCm backend ends up using: 24.3 GiB +/// measured for the same install, rounded up. An estimate and an upper bound, so +/// it only ever warns (the `rocm_core::disk_space` convention: exact +/// requirements may refuse, estimates only warn). +pub(crate) const FRESH_INSTALL_DISK_BYTES: u64 = 25 * 1024 * 1024 * 1024; + +/// What every Lemonade install needs whichever backend is chosen: the embeddable +/// archive (4.8 MB) plus the tree it extracts (10.6 MiB outside `bin`, measured on +/// the same host), rounded up. The one figure that is certain, so the one thing +/// a refusal may rest on: a drive without this much room cannot hold Lemonade at +/// all, however small a backend it picks. +pub(crate) const MIN_INSTALL_BYTES: u64 = 16 * 1024 * 1024; + +/// What an install is about to do, decided before anything is written. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct InstallPreflight { + /// The folder the install writes into (`/runtime`, its downloads...). + pub(crate) target_root: PathBuf, + /// The Lemonade install the record points at, when it really holds one and + /// is a different folder: the install is not going to reuse it. Never set + /// for a record whose folder no longer holds an install, because nothing is + /// installed there to be "not reused". + pub(crate) other_install: Option, + /// Whether `rocm storage` lists `other_install`, so the notice may point at + /// it. Storage lists only the data folder's tree and trees inside registered + /// ROCm installs; a recorded install anywhere else is announced but not + /// listed, and a pointer to it would be false. + pub(crate) other_install_listed: bool, + /// Bytes the install deletes before it extracts: the existing runtime + /// folder when this install replaces it (`--reinstall`, a new Lemonade + /// version, or a missing server binary), whose space is free by the time + /// the download starts. Zero when the tree is kept. + pub(crate) replaced_bytes: u64, + /// Free bytes on the drive holding `target_root`, when known. + pub(crate) available: Option, +} + +/// Outcome of comparing the preflight's free space with the install's size. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum SpaceVerdict { + /// Enough room, or free space could not be determined (never block on that). + Fits, + /// Room for the download but likely not for the finished tree. + Tight(String), + /// Not even room for the download: the install cannot finish. + Refuse(String), +} + +/// Whether installing into `env_root` will write a new Lemonade tree, and if so +/// what the user needs to be told. `None` when the tree there is already current +/// and has a llama.cpp backend, so the install only re-checks it. +pub(crate) fn plan_install_preflight( + paths: &AppPaths, + env_root: Option<&Path>, + reinstall: bool, + available_space: impl FnOnce(&Path) -> Option, +) -> Option { + let target_root = lemonade_root(paths, env_root); + let runtime_dir = runtime_dir_in(&target_root); + let extraction = needs_extraction( + reinstall, + installed_runtime_version(&runtime_dir).as_deref(), + rocm_deps::LEMONADE_VERSION, + lemond_path_in(&runtime_dir).is_file(), + ); + let backend_missing = !runtime_dir.join("bin").join("llamacpp").is_dir(); + if !extraction && !backend_missing { + return None; + } + let other_install = recorded_install_dir(paths) + .filter(|recorded| !runtime_path_is_same_or_inside(recorded, &target_root)); + let other_install_listed = other_install + .as_deref() + .is_some_and(|other| storage_lists(paths, other)); + // `prepare_embeddable` removes the whole runtime folder, backends included, + // before it extracts whenever `needs_extraction` says so, so that is the + // space the install frees first. + let replaced_bytes = if extraction && runtime_dir.exists() { + directory_bytes(&runtime_dir) + } else { + 0 + }; + let available = available_space(&target_root); + Some(InstallPreflight { + target_root, + other_install, + other_install_listed, + replaced_bytes, + available, + }) +} + +/// Total size of the files under `dir`, best effort: symlinks are not followed +/// and anything unreadable counts as zero, because this only sizes the space an +/// install is about to free. +fn directory_bytes(dir: &Path) -> u64 { + let mut total = 0_u64; + let mut stack = vec![dir.to_path_buf()]; + while let Some(current) = stack.pop() { + let Ok(entries) = std::fs::read_dir(¤t) else { + continue; + }; + for entry in entries.flatten() { + let Ok(metadata) = entry.metadata() else { + continue; + }; + if metadata.is_dir() && !metadata.is_symlink() { + stack.push(entry.path()); + } else { + total = total.saturating_add(metadata.len()); + } + } + } + total +} + +impl InstallPreflight { + /// The lines printed before the install starts. Every claim is derived from + /// the fields the install itself acts on: the target is where it writes, and + /// `other_install` is set only when the record points somewhere else. + pub(crate) fn notice_lines(&self) -> Vec { + let free = self + .available + .map(|bytes| format!(" ({} free on that drive)", format_bytes(bytes))) + .unwrap_or_default(); + let mut lines = vec![ + format!( + "Installing Lemonade {} into {}", + rocm_deps::LEMONADE_VERSION, + self.target_root.display() + ), + format!( + " With its llama.cpp ROCm backend this is a download of up to about {} and up to about {} of disk space{free}; a smaller backend, if Lemonade picks one for this GPU, needs less. It can take several minutes.", + download_text(), + disk_text(), + ), + ]; + if self.replaced_bytes > 0 { + lines.push(format!( + " The existing Lemonade install here ({}) is replaced first, and that space is counted as free.", + format_bytes(self.replaced_bytes) + )); + } + if let Some(other) = self.other_install.as_ref() { + let pointer = if self.other_install_listed { + " See `rocm storage`." + } else { + "" + }; + lines.push(format!( + " Lemonade is already installed at {}; that copy is not reused for this location.{pointer}", + other.display() + )); + } + lines + } + + /// Whether the drive can hold the install. + /// + /// Refuses only below [`MIN_INSTALL_BYTES`], the one figure that holds for + /// every backend. The ROCm backend download and the finished tree are upper + /// bounds, because the backend is not chosen until after the extraction this + /// runs before, so a host with less room than they need is warned and + /// allowed to try: refusing it could turn away a host whose smaller backend + /// would fit. + pub(crate) fn space_verdict(&self) -> SpaceVerdict { + const OPERATION: &str = "install Lemonade"; + let Some(free_now) = self.available else { + return SpaceVerdict::Fits; + }; + // The install deletes the existing runtime folder before it downloads, + // so its space counts. The warnings say so, so the figure shown is the + // one compared. + let available = free_now.saturating_add(self.replaced_bytes); + let room = if self.replaced_bytes > 0 { + format!( + "{} will be free on the drive holding {} once the existing install there is replaced", + format_bytes(available), + self.target_root.display() + ) + } else { + format!( + "only {} is free on the drive holding {}", + format_bytes(available), + self.target_root.display() + ) + }; + let minimum = rocm_core::with_margin(MIN_INSTALL_BYTES); + if available < minimum { + return SpaceVerdict::Refuse(format!( + "{} Nothing was downloaded.", + insufficient_space_message(OPERATION, &self.target_root, minimum, available) + )); + } + // The margin is working room for the comparison only. What is printed is + // the same figure the notice printed, so one run never shows two numbers + // for the same download. + if available < rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES) { + return SpaceVerdict::Tight(format!( + "Warning: {room}, and Lemonade's llama.cpp ROCm backend alone is a download of up to about {} (plus room to unpack it). The install will continue: if Lemonade picks a smaller backend for this GPU it may fit, otherwise it will fail partway through.", + download_text() + )); + } + if available < rocm_core::with_margin(FRESH_INSTALL_DISK_BYTES) { + return SpaceVerdict::Tight(format!( + "Warning: free disk space may be insufficient to install Lemonade's llama.cpp ROCm backend: up to about {} needed (plus room to work in), and {room}. The install will continue, but may fail partway through.", + disk_text() + )); + } + SpaceVerdict::Fits + } +} + +/// The ROCm backend's download as printed everywhere: by the notice and by both +/// warnings, so they cannot disagree. +fn download_text() -> String { + format_bytes(FRESH_INSTALL_DOWNLOAD_BYTES) +} + +/// The finished install's disk use as printed everywhere; see [`download_text`]. +fn disk_text() -> String { + format_bytes(FRESH_INSTALL_DISK_BYTES) +} + +/// Free space on the drive that will hold `path`, with the e2e-only override that +/// lets a black-box scenario reach the refusal without filling a real disk. +pub(crate) fn available_space_for_install(path: &Path) -> Option { + #[cfg(feature = "e2e-test-hooks")] + if let Some(bytes) = std::env::var_os(FREE_SPACE_TEST_ENV) + .and_then(|value| value.to_str().and_then(|value| value.parse::().ok())) + { + return Some(bytes); + } + rocm_core::available_space_for_path(path) +} + +/// Test-only override of the free space [`available_space_for_install`] reports. +#[cfg(feature = "e2e-test-hooks")] +const FREE_SPACE_TEST_ENV: &str = "ROCM_E2E_LEMONADE_FREE_SPACE_BYTES"; + +/// The folder a Lemonade install for `env_root` lives in: `/lemonade`, +/// or `/engines/lemonade` when there is none. +pub fn install_tree_root(paths: &AppPaths, env_root: Option<&Path>) -> PathBuf { + lemonade_root(paths, env_root) +} + +/// The parts of a Lemonade tree that make up the install itself. +/// +/// That is the extracted runtime with its backends, the cached archive, and +/// leftover extraction scratch. Deliberately not the whole folder: under +/// `/engines/lemonade` the same folder also holds the install record, logs +/// and server state. +pub fn install_tree_dirs(tree_root: &Path) -> [PathBuf; 3] { + [ + runtime_dir_in(tree_root), + tree_root.join("downloads"), + tree_root.join("extract"), + ] +} + +/// What Lemonade's install record says, as far as the CLI can tell. +/// +/// Three states rather than an `Option`, because they call for different +/// reporting: a record that cannot be read is not the same as no record, and a +/// record whose folder no longer holds an install means nothing is in use. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum InstallRecord { + /// No record: Lemonade has never been installed by ROCm CLI, or its record + /// was cleared. + NotRecorded, + /// A record file exists but could not be read or parsed. + Unreadable(String), + /// The record names `runtime_dir`; `holds_install` is whether the server + /// binary is still there. + Recorded { + runtime_dir: PathBuf, + holds_install: bool, + }, +} + +/// Read Lemonade's install record. +pub fn install_record(paths: &AppPaths) -> InstallRecord { + let record = manifest_path(paths); + match fs::read(&record) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => InstallRecord::NotRecorded, + Err(error) => InstallRecord::Unreadable(format!("{}: {error}", record.display())), + Ok(bytes) => { + match serde_json::from_slice::(&bytes) { + Ok(manifest) => InstallRecord::Recorded { + holds_install: lemond_path_in(&manifest.runtime_dir).is_file(), + runtime_dir: manifest.runtime_dir, + }, + Err(error) => InstallRecord::Unreadable(format!("{}: {error}", record.display())), + } + } + } +} + +/// The runtime folder the install record points at, if there is a readable one. +pub fn recorded_runtime_dir(paths: &AppPaths) -> Option { + match install_record(paths) { + InstallRecord::Recorded { runtime_dir, .. } => Some(runtime_dir), + InstallRecord::NotRecorded | InstallRecord::Unreadable(_) => None, + } +} + +/// The runtime folder Lemonade's record points at, only when it still holds an +/// install. +fn recorded_install_dir(paths: &AppPaths) -> Option { + match install_record(paths) { + InstallRecord::Recorded { + runtime_dir, + holds_install: true, + } => Some(runtime_dir), + InstallRecord::Recorded { .. } + | InstallRecord::NotRecorded + | InstallRecord::Unreadable(_) => None, + } +} + +/// The install roots of the ROCm installs registered with ROCm CLI, read from +/// the runtime registry the same way storage finds its runtimes. Best effort: an +/// unreadable entry is skipped. +fn registered_install_roots(paths: &AppPaths) -> Vec { + let registry = paths.data_dir.join("runtimes").join("registry"); + let Ok(entries) = fs::read_dir(registry) else { + return Vec::new(); + }; + entries + .flatten() + .filter(|entry| entry.path().extension().is_some_and(|ext| ext == "json")) + .filter_map(|entry| fs::read(entry.path()).ok()) + .filter_map(|bytes| serde_json::from_slice::(&bytes).ok()) + .filter_map(|record| record.get("install_root")?.as_str().map(PathBuf::from)) + .collect() +} + +/// Whether `rocm storage` lists the Lemonade install at `runtime_dir`: it lists +/// the data folder's tree and the tree inside each registered ROCm install, and +/// nothing else. Both use [`install_tree_root`], so the two cannot drift apart. +fn storage_lists(paths: &AppPaths, runtime_dir: &Path) -> bool { + runtime_path_is_same_or_inside(runtime_dir, &install_tree_root(paths, None)) + || registered_install_roots(paths).iter().any(|root| { + runtime_path_is_same_or_inside( + runtime_dir, + &install_tree_root(paths, Some(&root.join("engines"))), + ) + }) +} + +/// What [`forget_install_record_inside`] did, so a caller can report it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum RecordCleanup { + /// The record does not name anything inside the removed folder, or there is + /// no record. Nothing to report. + NotAffected, + /// The record named this runtime folder and has been removed. + Cleared(PathBuf), + /// The record names a folder that was removed, but deleting the record + /// failed: Lemonade still points at files that are gone. + Failed { + runtime_dir: PathBuf, + record: PathBuf, + error: String, + }, + /// The record could not be read, so it could not be compared with the + /// removed folder. + Unreadable(String), +} + +/// Forget the install record when the tree it points at was inside `removed_root`, +/// which the caller has just deleted. +/// +/// Leaving the record would point Lemonade at files that no longer exist. Without +/// it Lemonade reports itself as not installed, which is the true state. Every +/// outcome that leaves the user with something to know is returned rather than +/// printed, so the uninstall report can carry it. +pub fn forget_install_record_inside(paths: &AppPaths, removed_root: &Path) -> RecordCleanup { + forget_install_record_inside_with(paths, removed_root, |record| fs::remove_file(record)) +} + +/// [`forget_install_record_inside`] with the deletion injectable, so the failure +/// outcome can be tested without a file the OS refuses to delete. +fn forget_install_record_inside_with( + paths: &AppPaths, + removed_root: &Path, + remove_record: impl FnOnce(&Path) -> std::io::Result<()>, +) -> RecordCleanup { + let runtime_dir = match install_record(paths) { + InstallRecord::NotRecorded => return RecordCleanup::NotAffected, + InstallRecord::Unreadable(error) => return RecordCleanup::Unreadable(error), + InstallRecord::Recorded { runtime_dir, .. } => runtime_dir, + }; + if !runtime_path_is_same_or_inside(&runtime_dir, removed_root) { + return RecordCleanup::NotAffected; + } + let record = manifest_path(paths); + match remove_record(&record) { + Ok(()) => RecordCleanup::Cleared(runtime_dir), + Err(error) => RecordCleanup::Failed { + runtime_dir, + record, + error: error.to_string(), + }, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::install::{LemonadeInstallManifest, write_manifest}; + + fn paths_in(root: &Path) -> AppPaths { + AppPaths { + config_dir: root.join("config"), + data_dir: root.join("data"), + cache_dir: root.join("cache"), + } + } + + /// Plant an extracted tree the way `prepare_embeddable` leaves one: the + /// server binary, the version marker, and a llama.cpp backend folder. + fn plant_current_tree(tree_root: &Path) { + let runtime_dir = runtime_dir_in(tree_root); + fs::create_dir_all(runtime_dir.join("bin").join("llamacpp").join("rocm-stable")).unwrap(); + fs::write(lemond_path_in(&runtime_dir), b"").unwrap(); + crate::install::record_runtime_version(&runtime_dir, rocm_deps::LEMONADE_VERSION).unwrap(); + } + + fn record_install_at(paths: &AppPaths, runtime_dir: &Path) { + write_manifest( + paths, + &LemonadeInstallManifest { + env_id: "lemonade-embeddable-test".to_owned(), + version: rocm_deps::LEMONADE_VERSION.to_owned(), + runtime_dir: runtime_dir.to_path_buf(), + lemond: runtime_dir.join("lemond"), + lemonade: runtime_dir.join("lemonade"), + backend_recipe: "llamacpp".to_owned(), + backend_name: "rocm".to_owned(), + installed_at_unix_ms: 0, + }, + ) + .unwrap(); + } + + const GIB: u64 = 1024 * 1024 * 1024; + + /// The release-test sequence: Lemonade installed into the data folder, then + /// an SDK runtime activated, so the next install targets the runtime's own + /// folder. The notice must name the new folder and the copy it is not reusing. + #[test] + fn a_new_location_is_announced_with_the_install_it_does_not_reuse() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let base = install_tree_root(&paths, None); + plant_current_tree(&base); + record_install_at(&paths, &runtime_dir_in(&base)); + let runtime_engines = temp.path().join("runtimes").join("key").join("engines"); + + let preflight = + plan_install_preflight(&paths, Some(&runtime_engines), false, |_| Some(100 * GIB)) + .expect("a folder with no Lemonade in it needs a full install"); + + assert_eq!(preflight.target_root, runtime_engines.join("lemonade")); + assert_eq!(preflight.other_install, Some(runtime_dir_in(&base))); + assert!( + preflight.other_install_listed, + "the data folder's tree is listed" + ); + let notice = preflight.notice_lines().join("\n"); + assert!( + notice.contains(&preflight.target_root.display().to_string()), + "{notice}" + ); + assert!( + notice.contains( + "a download of up to about 4.4 GiB and up to about 25.0 GiB of disk space" + ), + "{notice}" + ); + assert!( + notice.contains("a smaller backend, if Lemonade picks one for this GPU, needs less"), + "{notice}" + ); + assert!(notice.contains("100.0 GiB free on that drive"), "{notice}"); + assert!( + notice.contains(&format!( + "Lemonade is already installed at {}; that copy is not reused", + runtime_dir_in(&base).display() + )), + "{notice}" + ); + assert_eq!(preflight.space_verdict(), SpaceVerdict::Fits); + } + + /// A current tree with a backend is not a new install: nothing to announce, + /// so a plain `rocm serve` on a ready host prints nothing extra. + #[test] + fn a_current_tree_is_not_announced() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let base = install_tree_root(&paths, None); + plant_current_tree(&base); + + assert_eq!(plan_install_preflight(&paths, None, false, |_| None), None); + // `--reinstall` wipes and re-extracts the same tree, so it is announced, + // and the record pointing into that same tree is not "another" install. + record_install_at(&paths, &runtime_dir_in(&base)); + let reinstall = plan_install_preflight(&paths, None, true, |_| None).unwrap(); + assert_eq!(reinstall.other_install, None); + assert!( + !reinstall + .notice_lines() + .join("\n") + .contains("already installed"), + "the tree being reinstalled must not be reported as a second copy" + ); + } + + #[test] + fn a_tree_without_a_backend_is_announced() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let base = install_tree_root(&paths, None); + plant_current_tree(&base); + fs::remove_dir_all(runtime_dir_in(&base).join("bin")).unwrap(); + + assert!(plan_install_preflight(&paths, None, false, |_| None).is_some()); + } + + /// A record whose folder lost its install (a runtime uninstalled by a build + /// that left the record behind) names nothing that is installed, so the + /// notice must not claim a copy "is already installed" or "not reused". + #[test] + fn a_record_naming_a_folder_with_no_install_is_not_announced_as_a_copy() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let gone = temp.path().join("runtimes").join("removed").join("engines"); + record_install_at( + &paths, + &runtime_dir_in(&install_tree_root(&paths, Some(&gone))), + ); + assert!( + matches!( + install_record(&paths), + InstallRecord::Recorded { + holds_install: false, + .. + } + ), + "the fixture must be a dangling record" + ); + + let preflight = plan_install_preflight(&paths, None, false, |_| Some(100 * GIB)) + .expect("an empty data folder needs a full install"); + + assert_eq!(preflight.other_install, None); + let notice = preflight.notice_lines().join("\n"); + assert!(!notice.contains("already installed"), "{notice}"); + assert!(!notice.contains("not reused"), "{notice}"); + } + + fn preflight_with(available: Option, replaced_bytes: u64) -> InstallPreflight { + InstallPreflight { + target_root: PathBuf::from("/data/engines/lemonade"), + other_install: None, + other_install_listed: false, + replaced_bytes, + available, + } + } + + /// Only the one figure that holds for every backend can refuse: the backend + /// is chosen after this runs, and a host Lemonade gives a smaller backend + /// must not be turned away for a download it does not need. + #[test] + fn only_space_below_the_smallest_install_refuses_and_less_than_the_rocm_backend_warns() { + const MIB: u64 = 1024 * 1024; + let preflight = |available| preflight_with(available, 0); + let SpaceVerdict::Refuse(message) = preflight(Some(2 * MIB)).space_verdict() else { + panic!("2 MiB cannot hold the Lemonade archive and its extracted files"); + }; + assert!( + message.contains("not enough free disk space to install Lemonade:"), + "{message}" + ); + assert!(message.contains("Nothing was downloaded."), "{message}"); + + // Room for Lemonade itself but not for the ROCm backend download: warned + // and allowed, because a smaller backend may still fit. + for available in [200 * MIB, 2 * GIB] { + let SpaceVerdict::Tight(warning) = preflight(Some(available)).space_verdict() else { + panic!("{available} bytes is below the ROCm backend download: warn, not refuse"); + }; + assert!( + warning.contains("if Lemonade picks a smaller backend for this GPU it may fit"), + "{warning}" + ); + } + // Room for the download but not the finished tree. + let SpaceVerdict::Tight(warning) = preflight(Some(10 * GIB)).space_verdict() else { + panic!("10 GiB is below the finished install: warn"); + }; + assert!(!warning.contains("smaller backend"), "{warning}"); + assert_eq!( + preflight(Some(40 * GIB)).space_verdict(), + SpaceVerdict::Fits + ); + assert_eq!(preflight(None).space_verdict(), SpaceVerdict::Fits); + } + + /// The notice and both warnings describe the same download and the same + /// disk use, so they must print the same figures. The margin is for the + /// comparison only: with it printed, the notice said 4.4 GiB and the warning + /// about 4.8 GiB for the same backend. + #[test] + fn the_notice_and_the_warnings_print_the_same_download_and_disk_figures() { + let notice = preflight_with(Some(100 * GIB), 0).notice_lines().join("\n"); + let SpaceVerdict::Tight(download_warning) = + preflight_with(Some(2 * GIB), 0).space_verdict() + else { + panic!("2 GiB is below the ROCm backend download"); + }; + let SpaceVerdict::Tight(disk_warning) = preflight_with(Some(10 * GIB), 0).space_verdict() + else { + panic!("10 GiB is below the finished install"); + }; + + // Derived from the constants, not retyped: if the notice or a warning + // ever prints a different number, one of these fails. + let download = format_bytes(FRESH_INSTALL_DOWNLOAD_BYTES); + let disk = format_bytes(FRESH_INSTALL_DISK_BYTES); + assert!( + notice.contains(&format!("a download of up to about {download}")), + "{notice}" + ); + assert!( + notice.contains(&format!("up to about {disk} of disk space")), + "{notice}" + ); + assert!( + download_warning.contains(&format!("a download of up to about {download}")), + "{download_warning}" + ); + assert!( + disk_warning.contains(&format!("up to about {disk} needed")), + "{disk_warning}" + ); + // And the margin-inflated values are not what is shown. + let inflated_download = format_bytes(rocm_core::with_margin(FRESH_INSTALL_DOWNLOAD_BYTES)); + let inflated_disk = format_bytes(rocm_core::with_margin(FRESH_INSTALL_DISK_BYTES)); + assert_ne!(inflated_download, download); + assert_ne!(inflated_disk, disk); + assert!( + !download_warning.contains(&inflated_download), + "{download_warning}" + ); + assert!(!disk_warning.contains(&inflated_disk), "{disk_warning}"); + // The figures themselves, so a constant change is a visible edit here too. + assert_eq!((download.as_str(), disk.as_str()), ("4.4 GiB", "25.0 GiB")); + } + + /// An install that replaces the existing runtime folder deletes it before + /// downloading (`prepare_embeddable` removes the whole folder, backends + /// included), so its space is free by then. The preflight sizes that folder, + /// counts it, and says so; an install that keeps the tree frees nothing. + #[test] + fn a_reinstall_counts_the_space_of_the_install_it_replaces() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let tree = install_tree_root(&paths, None); + plant_current_tree(&tree); + let payload = runtime_dir_in(&tree) + .join("bin") + .join("llamacpp") + .join("rocm-stable"); + fs::write(payload.join("payload.bin"), vec![0_u8; 123_456]).unwrap(); + + let reinstall = plan_install_preflight(&paths, None, true, |_| Some(2 * GIB)).unwrap(); + assert!( + reinstall.replaced_bytes >= 123_456, + "{}", + reinstall.replaced_bytes + ); + let notice = reinstall.notice_lines().join("\n"); + assert!( + notice.contains(&format!( + "The existing Lemonade install here ({}) is replaced first, and that space is counted as free.", + format_bytes(reinstall.replaced_bytes) + )), + "{notice}" + ); + + // A current tree whose backend is missing is topped up, not wiped. + fs::remove_dir_all(runtime_dir_in(&tree).join("bin")).unwrap(); + let kept = plan_install_preflight(&paths, None, false, |_| Some(2 * GIB)).unwrap(); + assert_eq!(kept.replaced_bytes, 0); + assert!(!kept.notice_lines().join("\n").contains("replaced first")); + + // The verdict counts it: 2 GiB free warns for the ROCm backend, but the + // same drive with a 24 GiB install about to be deleted does not. + assert!(matches!( + preflight_with(Some(2 * GIB), 0).space_verdict(), + SpaceVerdict::Tight(_) + )); + assert_eq!( + preflight_with(Some(2 * GIB), 30 * GIB).space_verdict(), + SpaceVerdict::Fits + ); + // When it still does not fit, the warning shows the figure compared. + let SpaceVerdict::Tight(warning) = preflight_with(Some(GIB), 2 * GIB).space_verdict() + else { + panic!("3 GiB is below the ROCm backend download"); + }; + assert!( + warning.contains("3.0 GiB will be free on the drive holding /data/engines/lemonade once the existing install there is replaced"), + "{warning}" + ); + } + + /// The notice points at `rocm storage` only for a copy storage lists: the + /// data folder's tree and trees inside registered ROCm installs. A recorded + /// install anywhere else is announced without the pointer, which would be + /// false for it. + #[test] + fn the_storage_pointer_is_only_given_for_a_copy_storage_lists() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let target = temp.path().join("target").join("engines"); + let notice_for = |paths: &AppPaths| { + let preflight = plan_install_preflight(paths, Some(&target), false, |_| None).unwrap(); + ( + preflight.other_install_listed, + preflight.notice_lines().join("\n"), + ) + }; + + // 1. In the data folder: listed. + let base = install_tree_root(&paths, None); + plant_current_tree(&base); + record_install_at(&paths, &runtime_dir_in(&base)); + let (listed, notice) = notice_for(&paths); + assert!( + listed + && notice + .contains("that copy is not reused for this location. See `rocm storage`."), + "{notice}" + ); + + // 2. Inside a registered ROCm install: listed. + let registered = temp.path().join("runtimes").join("registered"); + let registry = paths.data_dir.join("runtimes").join("registry"); + fs::create_dir_all(®istry).unwrap(); + fs::write( + registry.join("registered.json"), + serde_json::json!({ "runtime_key": "registered", "install_root": registered }) + .to_string(), + ) + .unwrap(); + let in_runtime = install_tree_root(&paths, Some(®istered.join("engines"))); + plant_current_tree(&in_runtime); + record_install_at(&paths, &runtime_dir_in(&in_runtime)); + let (listed, notice) = notice_for(&paths); + assert!(listed && notice.contains("See `rocm storage`."), "{notice}"); + + // 3. Anywhere else: announced, not pointed at. + let elsewhere = + install_tree_root(&paths, Some(&temp.path().join("custom").join("engines"))); + plant_current_tree(&elsewhere); + record_install_at(&paths, &runtime_dir_in(&elsewhere)); + let (listed, notice) = notice_for(&paths); + assert!(!listed, "{notice}"); + assert!( + notice.contains(&format!( + "Lemonade is already installed at {}", + runtime_dir_in(&elsewhere).display() + )), + "{notice}" + ); + assert!(!notice.contains("rocm storage"), "{notice}"); + } + + /// A runtime uninstall deletes the runtime's folder, Lemonade tree included. + /// The record pointing into it is forgotten; one pointing elsewhere is kept. + #[test] + fn the_record_is_forgotten_only_when_its_tree_was_removed() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let runtime_root = temp.path().join("runtimes").join("key"); + let per_runtime = install_tree_root(&paths, Some(&runtime_root.join("engines"))); + record_install_at(&paths, &runtime_dir_in(&per_runtime)); + + let other_runtime = temp.path().join("runtimes").join("other"); + assert_eq!( + forget_install_record_inside(&paths, &other_runtime), + RecordCleanup::NotAffected + ); + assert!(recorded_runtime_dir(&paths).is_some()); + + assert_eq!( + forget_install_record_inside(&paths, &runtime_root), + RecordCleanup::Cleared(runtime_dir_in(&per_runtime)) + ); + assert_eq!(recorded_runtime_dir(&paths), None); + assert!(!manifest_path(&paths).exists()); + // Nothing recorded: nothing to forget. + assert_eq!( + forget_install_record_inside(&paths, &runtime_root), + RecordCleanup::NotAffected + ); + } + + /// The folder is already gone when the record is cleared, so a failed + /// deletion leaves Lemonade pointing at nothing. That has to come back to + /// the caller with the record's path, not vanish into a log line. + #[test] + fn a_record_that_cannot_be_deleted_is_reported_not_swallowed() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let runtime_root = temp.path().join("runtimes").join("key"); + let per_runtime = install_tree_root(&paths, Some(&runtime_root.join("engines"))); + record_install_at(&paths, &runtime_dir_in(&per_runtime)); + + let outcome = forget_install_record_inside_with(&paths, &runtime_root, |_| { + Err(std::io::Error::other("sharing violation")) + }); + + assert_eq!( + outcome, + RecordCleanup::Failed { + runtime_dir: runtime_dir_in(&per_runtime), + record: manifest_path(&paths), + error: "sharing violation".to_owned(), + } + ); + assert!(manifest_path(&paths).exists(), "the record was not deleted"); + + // The uninstall report tells the user to delete that record. Doing so must + // clear the condition: Lemonade is then simply not recorded. + fs::remove_file(manifest_path(&paths)).unwrap(); + assert_eq!(install_record(&paths), InstallRecord::NotRecorded); + assert_eq!(recorded_runtime_dir(&paths), None); + } + + /// An unparseable record is not "no record": it may well name the removed + /// folder, and the caller must be told it could not be checked. + #[test] + fn an_unreadable_record_is_distinct_from_no_record() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + assert_eq!(install_record(&paths), InstallRecord::NotRecorded); + + let record = manifest_path(&paths); + fs::create_dir_all(record.parent().unwrap()).unwrap(); + fs::write(&record, b"{ not json").unwrap(); + + let InstallRecord::Unreadable(error) = install_record(&paths) else { + panic!("a corrupt record must not read as NotRecorded"); + }; + assert!(error.contains("runtime.json"), "{error}"); + assert_eq!(recorded_runtime_dir(&paths), None); + assert!(matches!( + forget_install_record_inside(&paths, temp.path()), + RecordCleanup::Unreadable(_) + )); + assert!(record.exists(), "an unreadable record must be left alone"); + } + + /// A record whose runtime folder lost its server binary names an install + /// that is not there; `holds_install` is what lets the storage report say so. + #[test] + fn a_record_reports_whether_its_folder_still_holds_an_install() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let tree = install_tree_root(&paths, None); + record_install_at(&paths, &runtime_dir_in(&tree)); + assert_eq!( + install_record(&paths), + InstallRecord::Recorded { + runtime_dir: runtime_dir_in(&tree), + holds_install: false + } + ); + plant_current_tree(&tree); + assert_eq!( + install_record(&paths), + InstallRecord::Recorded { + runtime_dir: runtime_dir_in(&tree), + holds_install: true + } + ); + } + + /// `rocm runtimes uninstall` tells a user whose install record cannot be read + /// to run a plain `rocm engines install lemonade`. That is only true if the + /// install path overwrites the record without reading it and keeps the + /// current tree, so this runs the real record-writing sequence + /// (`install_and_record`, with nothing to download) over a corrupt record and + /// asserts the state it leaves: a readable record naming an install that is + /// there, and the tree untouched (a `--reinstall` would have wiped it). + #[test] + fn a_plain_install_replaces_a_corrupt_record_and_keeps_the_current_tree() { + let temp = tempfile::tempdir().unwrap(); + let paths = paths_in(temp.path()); + let tree = install_tree_root(&paths, None); + plant_current_tree(&tree); + let runtime_dir = runtime_dir_in(&tree); + fs::write( + crate::install::lemonade_path_in(&runtime_dir), + b"fake lemonade", + ) + .unwrap(); + let sentinel = runtime_dir + .join("bin") + .join("llamacpp") + .join("backend.sentinel"); + fs::write(&sentinel, b"a backend already installed").unwrap(); + + let record = crate::install::manifest_path(&paths); + fs::create_dir_all(record.parent().unwrap()).unwrap(); + fs::write(&record, b"{ not json").unwrap(); + assert!(matches!( + install_record(&paths), + InstallRecord::Unreadable(_) + )); + + let archive = b"never parsed: the tree is current, so nothing is extracted"; + let sha256 = format!("{:x}", ::digest(archive)); + let (manifest, _) = crate::install_and_record( + &paths, + None, + false, + |paths, env_root, reinstall| { + crate::install::prepare_embeddable_with( + paths, + env_root, + reinstall, + rocm_deps::LEMONADE_VERSION, + &sha256, + |_, destination| { + fs::write(destination, archive)?; + Ok(()) + }, + ) + }, + |_, _| Ok(None), + ) + .expect("a plain install over a corrupt record succeeds"); + + assert_eq!( + install_record(&paths), + InstallRecord::Recorded { + runtime_dir: runtime_dir.clone(), + holds_install: true + }, + "the corrupt record was not replaced" + ); + assert_eq!(manifest.runtime_dir, runtime_dir); + assert!( + sentinel.is_file(), + "a plain install must keep the current tree" + ); + } +} diff --git a/engines/lemonade/src/install.rs b/engines/lemonade/src/install.rs index 5c67cbcad4..25728566d3 100644 --- a/engines/lemonade/src/install.rs +++ b/engines/lemonade/src/install.rs @@ -68,7 +68,7 @@ pub(crate) struct LemonadeInstallManifest { /// /// Pure so the upgrade case is testable without a real archive. `installed` /// is the version recorded for the tree already on disk, if any. -fn needs_extraction( +pub(crate) fn needs_extraction( reinstall: bool, installed: Option<&str>, wanted: &str, @@ -78,7 +78,7 @@ fn needs_extraction( } /// The Lemonade runtime version last extracted into `runtime_dir`, if any. -fn installed_runtime_version(runtime_dir: &Path) -> Option { +pub(crate) fn installed_runtime_version(runtime_dir: &Path) -> Option { fs::read_to_string(runtime_dir.join(RUNTIME_VERSION_MARKER)) .ok() .map(|s| s.trim().to_owned()) @@ -89,7 +89,7 @@ fn installed_runtime_version(runtime_dir: &Path) -> Option { /// part of the same wipe that removes the tree it describes — "marker present /// implies extraction completed" holds by construction, with no separate /// bookkeeping to keep in sync. -fn record_runtime_version(runtime_dir: &Path, version: &str) -> Result<()> { +pub(crate) fn record_runtime_version(runtime_dir: &Path, version: &str) -> Result<()> { fs::write(runtime_dir.join(RUNTIME_VERSION_MARKER), version).with_context(|| { format!( "failed to record installed Lemonade runtime version under {}", @@ -117,7 +117,7 @@ pub(crate) fn prepare_embeddable( /// sha256, and the download step all injectable — the seam a test uses to /// exercise the real decision of which source `installed_version` trusts, /// rather than reimplementing that decision inline. -fn prepare_embeddable_with( +pub(crate) fn prepare_embeddable_with( paths: &AppPaths, env_root: Option<&Path>, reinstall: bool, @@ -195,18 +195,18 @@ where }) } -fn manifest_path(paths: &AppPaths) -> PathBuf { +pub(crate) fn manifest_path(paths: &AppPaths) -> PathBuf { paths.engine_install_manifest_path(ENGINE_NAME) } -fn lemonade_root(paths: &AppPaths, env_root: Option<&Path>) -> PathBuf { +pub(crate) fn lemonade_root(paths: &AppPaths, env_root: Option<&Path>) -> PathBuf { env_root.map_or_else( || paths.engine_dir(ENGINE_NAME), |root| normalize_runtime_path_for_host(root).join(ENGINE_NAME), ) } -fn runtime_dir_in(root: &Path) -> PathBuf { +pub(crate) fn runtime_dir_in(root: &Path) -> PathBuf { root.join("runtime") } @@ -451,11 +451,11 @@ impl PinFileBackup { } } -fn lemond_path_in(runtime_dir: &Path) -> PathBuf { +pub(crate) fn lemond_path_in(runtime_dir: &Path) -> PathBuf { runtime_dir.join(platform_binary_name("lemond")) } -fn lemonade_path_in(runtime_dir: &Path) -> PathBuf { +pub(crate) fn lemonade_path_in(runtime_dir: &Path) -> PathBuf { runtime_dir.join(platform_binary_name("lemonade")) } diff --git a/engines/lemonade/src/lib.rs b/engines/lemonade/src/lib.rs index 48deb7d142..efa731264c 100644 --- a/engines/lemonade/src/lib.rs +++ b/engines/lemonade/src/lib.rs @@ -28,6 +28,7 @@ mod archive_props; mod backend_alignment; mod device_policy; mod direct_llama; +mod footprint; mod install; mod llamacpp_backend; mod process; @@ -41,6 +42,12 @@ mod test_support; // the crate root before the split, and `apps/rocm` depends on this crate's // lib target. pub use backend_alignment::LEMONADE_BACKEND_ALIGNMENT_DISABLED_ENV; +// The read-only view of Lemonade's install trees and record that `apps/rocm` +// needs for `rocm storage` and `rocm runtimes uninstall`; see `footprint`. +pub use footprint::{ + InstallRecord, RecordCleanup, forget_install_record_inside, install_record, install_tree_dirs, + install_tree_root, recorded_runtime_dir, +}; pub(crate) const ENGINE_NAME: &str = "lemonade"; pub(crate) const DEFAULT_HOST: &str = "127.0.0.1"; @@ -358,6 +365,35 @@ fn detect_response() -> DetectResponse { } } +/// The install proper: extract the embeddable, pick and align the llama.cpp +/// backend, then write the install record, replacing whatever was there. +/// +/// The record is written, never read, on this path: a corrupt record is +/// overwritten by the next plain install and the current tree is kept, which is +/// why that is what the CLI advises when it cannot read the record. The two +/// steps are parameters so a test can run this real sequence without +/// downloading anything or running `lemonade`. +fn install_and_record( + paths: &AppPaths, + env_root: Option<&std::path::Path>, + reinstall: bool, + prepare_embeddable: impl FnOnce( + &AppPaths, + Option<&std::path::Path>, + bool, + ) -> Result, + prepare_backend: impl FnOnce( + &AppPaths, + &mut crate::install::LemonadeInstallManifest, + ) -> Result>, +) -> Result<(crate::install::LemonadeInstallManifest, Option)> { + let mut manifest = prepare_embeddable(paths, env_root, reinstall)?; + eprintln!("Detecting best supported Lemonade llama.cpp backend..."); + let aligned_version = prepare_backend(paths, &mut manifest)?; + crate::install::write_manifest(paths, &manifest)?; + Ok((manifest, aligned_version)) +} + fn install_response(request: InstallRequest) -> Result { let paths = AppPaths::discover()?; paths.ensure()?; @@ -371,20 +407,38 @@ fn install_response(request: InstallRequest) -> Result { bail!("scripted Lemonade backend install failure") })?; } - eprintln!( - "Preparing Lemonade embeddable {}...", - rocm_deps::LEMONADE_VERSION - ); let env_root = request .env_root .as_deref() .map(normalize_runtime_path_for_host); - let mut manifest = - crate::install::prepare_embeddable(&paths, env_root.as_deref(), request.reinstall)?; - eprintln!("Detecting best supported Lemonade llama.cpp backend..."); - let aligned_version = - crate::backend_alignment::prepare_llamacpp_backend_for_active_rocm(&paths, &mut manifest)?; - crate::install::write_manifest(&paths, &manifest)?; + // Announced and space-checked before anything is downloaded: a fresh tree + // is a multi-gigabyte install, and `rocm serve` reaches this on its own. + if let Some(preflight) = crate::footprint::plan_install_preflight( + &paths, + env_root.as_deref(), + request.reinstall, + crate::footprint::available_space_for_install, + ) { + for line in preflight.notice_lines() { + eprintln!("{line}"); + } + match preflight.space_verdict() { + crate::footprint::SpaceVerdict::Fits => {} + crate::footprint::SpaceVerdict::Tight(warning) => eprintln!("{warning}"), + crate::footprint::SpaceVerdict::Refuse(message) => bail!(message), + } + } + eprintln!( + "Preparing Lemonade embeddable {}...", + rocm_deps::LEMONADE_VERSION + ); + let (manifest, aligned_version) = install_and_record( + &paths, + env_root.as_deref(), + request.reinstall, + crate::install::prepare_embeddable, + crate::backend_alignment::prepare_llamacpp_backend_for_active_rocm, + )?; let mut warnings = vec![ "Lemonade is installed as a rocm-cli managed embeddable runtime".to_owned(), format!( diff --git a/engines/lemonade/src/process.rs b/engines/lemonade/src/process.rs index e90c5cd25d..6d96eed836 100644 --- a/engines/lemonade/src/process.rs +++ b/engines/lemonade/src/process.rs @@ -159,6 +159,40 @@ pub(crate) fn hide_child_console_window(command: &mut ProcessCommand) { #[cfg(not(windows))] pub(crate) const fn hide_child_console_window(_command: &mut ProcessCommand) {} +/// Which of this process's standard output streams are a console. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct OutputConsoles { + pub(crate) stdout: bool, + pub(crate) stderr: bool, +} + +impl OutputConsoles { + pub(crate) fn detect() -> Self { + use std::io::IsTerminal; + Self { + stdout: std::io::stdout().is_terminal(), + stderr: std::io::stderr().is_terminal(), + } + } + + /// Whether a child that inherits both streams may be started without a + /// console window. + /// + /// `CREATE_NO_WINDOW` starts the child with no console at all, and a child + /// with no console writes nothing to a console handle it inherited: run + /// from a real Windows console, `cmd /c echo`, a Python child and + /// `lemonade.exe --version` all printed their output without the flag and + /// printed nothing with it. That is how `lemonade backends install`'s + /// progress for a multi-gigabyte download disappeared from an interactive + /// `rocm serve` while still showing up in a redirected log. When either + /// stream is a console this process has one, so the child shares it and no + /// window opens; when neither is (a pipe, a file, or no console at all) the + /// flag is still applied, so a console-less parent never flashes a window. + pub(crate) const fn child_may_hide_its_console(self) -> bool { + !(self.stdout || self.stderr) + } +} + fn child_process_path(path: &Path) -> String { if runtime_is_windows() { windows_child_path(path) @@ -309,6 +343,39 @@ pub(crate) fn run_lemonade_backend_install( backend: &str, process_env: &LemonadeProcessEnvironment, ) -> Result<()> { + let mut command = backend_install_command( + manifest, + host, + port, + backend, + process_env, + OutputConsoles::detect(), + hide_child_console_window, + )?; + let status = command + .status() + .with_context(|| format!("failed to run {}", manifest.lemonade.display()))?; + if !status.success() { + bail!("Lemonade backend install failed with status {status}"); + } + Ok(()) +} + +/// The `lemonade backends install` command, ready to run. +/// +/// It inherits stdout and stderr: the backend download's progress is the child's +/// own output. `consoles` and `hide_window` are parameters so a test can assert +/// whether the window is hidden for each console state; the one production +/// caller passes the real streams and [`hide_child_console_window`]. +fn backend_install_command( + manifest: &LemonadeInstallManifest, + host: &str, + port: u16, + backend: &str, + process_env: &LemonadeProcessEnvironment, + consoles: OutputConsoles, + hide_window: impl FnOnce(&mut ProcessCommand), +) -> Result { let mut command = ProcessCommand::new(&manifest.lemonade); command .arg("--host") @@ -322,14 +389,10 @@ pub(crate) fn run_lemonade_backend_install( .stdout(Stdio::inherit()) .stderr(Stdio::inherit()); apply_lemonade_process_environment(&mut command, process_env)?; - hide_child_console_window(&mut command); - let status = command - .status() - .with_context(|| format!("failed to run {}", manifest.lemonade.display()))?; - if !status.success() { - bail!("Lemonade backend install failed with status {status}"); + if consoles.child_may_hide_its_console() { + hide_window(&mut command); } - Ok(()) + Ok(command) } fn lemonade_pull_args(host: &str, port: u16, checkpoint_ref: &str) -> Vec { @@ -644,6 +707,75 @@ mod tests { use rocm_engine_protocol::DevicePolicy; use serde_json::json; + fn test_manifest() -> LemonadeInstallManifest { + LemonadeInstallManifest { + env_id: "lemonade-embeddable-test".to_owned(), + version: "test".to_owned(), + runtime_dir: PathBuf::from("rt"), + lemond: PathBuf::from("rt").join("lemond"), + lemonade: PathBuf::from("rt").join("lemonade"), + backend_recipe: "llamacpp".to_owned(), + backend_name: "rocm".to_owned(), + installed_at_unix_ms: 0, + } + } + + /// Build the real backend-install command for a console state and report + /// whether the window was hidden, along with the command itself. + fn backend_install_for(consoles: OutputConsoles) -> (ProcessCommand, bool) { + let mut hidden = false; + let command = backend_install_command( + &test_manifest(), + "127.0.0.1", + 8123, + "rocm", + &LemonadeProcessEnvironment::default(), + consoles, + |_| hidden = true, + ) + .expect("the command builds"); + (command, hidden) + } + + /// The backend install's progress is the child's own output. Hiding its + /// console while that output is a console discards it (see + /// `OutputConsoles::child_may_hide_its_console`), so the window is hidden + /// only when neither stream is one. The command is the real one: a dropped + /// call to the hide step, a hide step run unconditionally, or a console + /// check that ignores one of the streams each turn a case below red. + #[test] + fn the_backend_install_hides_its_window_only_when_no_output_is_a_console() { + for (stdout, stderr, expect_hidden) in [ + (true, true, false), + (true, false, false), + (false, true, false), + (false, false, true), + ] { + let (command, hidden) = backend_install_for(OutputConsoles { stdout, stderr }); + assert_eq!( + hidden, expect_hidden, + "stdout console={stdout}, stderr console={stderr}" + ); + assert_eq!(command.get_program(), PathBuf::from("rt").join("lemonade")); + let args: Vec<_> = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + assert_eq!( + args, + [ + "--host", + "127.0.0.1", + "--port", + "8123", + "backends", + "install", + "llamacpp:rocm" + ] + ); + } + } + #[test] fn models_ready_accepts_direct_serve_and_gates_router_backend() { // Direct-serve (stock llama-server): no `recipe_options` → ready by name. diff --git a/tests/e2e-cucumber/features/lemonade_installs.feature b/tests/e2e-cucumber/features/lemonade_installs.feature new file mode 100644 index 0000000000..e55ad7fe16 --- /dev/null +++ b/tests/e2e-cucumber/features/lemonade_installs.feature @@ -0,0 +1,64 @@ +Feature: Lemonade installs on disk + + # Lemonade manages its own runtime, and where it installs depends on whether a + # ROCm runtime is active: into the data folder when none is, into that + # runtime's own folder when one is. So `rocm engines install lemonade` before + # `rocm install sdk`, then `rocm serve` after it, leaves two full copies of + # roughly 25 GiB each, and the second one used to arrive silently from + # `rocm serve` (ROCMAI-825). These scenarios cover what the user is now told + # about that: what an install needs before it starts, both copies in the disk + # report, and the install record when a runtime holding a copy is removed. + # + # A new feature file rather than appended scenarios: these are about one + # behaviour area, Lemonade's footprint on disk, which spans `engines install`, + # `storage report` and `runtimes uninstall`. + # + # None downloads anything or needs a GPU, so all run on the mock lane every PR. + # lemonade-installs-01 and -02 need the `rocm/e2e-test-hooks` build + # `cargo xtask e2e` makes, to report a nearly full drive without filling a + # real one. + + @id:lemonade-installs-refused-without-space + Scenario: lemonade-installs-01 - A Lemonade install the drive cannot hold is refused before downloading + Given the drive for a new Lemonade install cannot hold even Lemonade itself + When the user installs the Lemonade engine + Then the Lemonade install is refused before anything is downloaded + And the user is told what the Lemonade install needs and where it would go + + # `rocm serve` reaches the same install on its own, which is how the second + # copy used to appear without a word. The free-space override also stands in + # for the GPU the no-GPU lane does not have, as serve-18's failure hook does. + @id:lemonade-installs-serve-refused-without-space + Scenario: lemonade-installs-02 - Serving with Lemonade on a drive that cannot hold it is refused before downloading + Given the drive for a new Lemonade install cannot hold even Lemonade itself + When the user serves a model with Lemonade + Then the Lemonade install is refused before anything is downloaded + And the user is told what the Lemonade install needs and where it would go + + @id:lemonade-installs-storage-lists-both-copies + Scenario: lemonade-installs-03 - The disk report lists both Lemonade copies and flags the unused one + Given Lemonade was installed in the data folder and again inside a ROCm install + When the user asks what ROCm CLI is keeping on disk + Then the report lists both Lemonade installs with their sizes + And the report flags the data folder's Lemonade install as unused + + @id:lemonade-installs-runtime-uninstall-clears-record + Scenario: lemonade-installs-04 - Removing the ROCm install that held Lemonade clears its install record + Given Lemonade was installed in the data folder and again inside a ROCm install + When the user uninstalls the ROCm install holding Lemonade + Then the CLI says Lemonade's install was removed with that ROCm install + And Lemonade is no longer recorded as installed + + @id:lemonade-installs-dangling-record-keeps-the-survivor + Scenario: lemonade-installs-05 - A record naming a folder with no install does not make the surviving copy reclaimable + Given the ROCm install that held Lemonade is gone and only the data folder's copy is left + When the user asks what ROCm CLI is keeping on disk + Then the report does not call the surviving Lemonade copy unused or reclaimable + + @id:lemonade-installs-read-only-runtime-keeps-its-copy + Scenario: lemonade-installs-06 - Uninstalling a read-only ROCm install leaves its Lemonade copy, and the report says so + Given a read-only ROCm install holds a Lemonade copy + When the user asks what ROCm CLI is keeping on disk + Then the report does not say that copy is removed with its ROCm install + When the user uninstalls the ROCm install holding Lemonade + Then that Lemonade copy is still on disk diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 8ec3496528..96ed4fb0a7 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -32,6 +32,7 @@ mod e2e { pub mod engines_steps; pub mod examine_steps; pub mod host_guard; + pub mod lemonade_install_steps; pub mod lifecycle_steps; pub mod logs_steps; pub mod model_catalog_steps; diff --git a/tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs b/tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs new file mode 100644 index 0000000000..10a98f0295 --- /dev/null +++ b/tests/e2e-cucumber/tests/e2e/lemonade_install_steps.rs @@ -0,0 +1,402 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Steps for `lemonade_installs.feature`: what the user is told about the +//! Lemonade copies ROCm CLI puts on disk (ROCMAI-825). +//! +//! Black-box throughout. The two Lemonade trees and the install record are +//! planted as plain files in the layout the engine writes, so no scenario +//! downloads anything; the refusal scenario reaches the engine's free-space +//! check through the `rocm/e2e-test-hooks` override instead of a full drive. + +use std::path::PathBuf; + +use cucumber::{given, then, when}; + +use crate::E2eWorld; + +/// Registry key of the planted runtime. Production-shaped (slug, no `:`), so it +/// is also a valid registry file name on Windows. +const RUNTIME_KEY: &str = "release-tarball-gfx1151"; +/// Payload planted in each tree, chosen so the two render as distinct sizes. +const DATA_FOLDER_TREE_BYTES: usize = 3 * 1024 * 1024; +const RUNTIME_TREE_BYTES: usize = 5 * 1024 * 1024; +/// Free space the refusal scenarios report for the install drive: 1 MiB, below +/// what any Lemonade install needs (its archive and extracted files), so the +/// engine refuses before fetching anything. Room for that but not for the +/// multi-gigabyte ROCm backend only warns, and is unit-tested. +const NEARLY_FULL_BYTES: &str = "1048576"; + +fn data_dir(world: &E2eWorld) -> PathBuf { + world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .join("data") +} + +/// Where Lemonade installs when no ROCm runtime is active. +fn data_folder_tree(world: &E2eWorld) -> PathBuf { + data_dir(world).join("engines").join("lemonade") +} + +fn runtime_install_root(world: &E2eWorld) -> PathBuf { + data_dir(world) + .join("runtimes") + .join("tarball") + .join(RUNTIME_KEY) +} + +/// Where Lemonade installs while the planted runtime is active. +fn runtime_tree(world: &E2eWorld) -> PathBuf { + runtime_install_root(world).join("engines").join("lemonade") +} + +/// Lemonade's install record, in the engine's own location. +fn install_record(world: &E2eWorld) -> PathBuf { + data_folder_tree(world) + .join("manifests") + .join("runtime.json") +} + +const fn lemond_name() -> &'static str { + if cfg!(windows) { + "lemond.exe" + } else { + "lemond" + } +} + +/// An extracted tree: the server binary plus a llama.cpp backend payload. +fn plant_lemonade_tree(tree: &std::path::Path, payload_bytes: usize) { + let runtime_dir = tree.join("runtime"); + let backend = runtime_dir.join("bin").join("llamacpp").join("rocm-stable"); + std::fs::create_dir_all(&backend).expect("failed to create backend dir"); + std::fs::write(runtime_dir.join(lemond_name()), b"").expect("failed to write lemond"); + std::fs::write(backend.join("payload.bin"), vec![0_u8; payload_bytes]) + .expect("failed to write backend payload"); +} + +fn stdout(world: &E2eWorld) -> &str { + world + .cli_output + .as_deref() + .expect("no CLI output captured - did the When step run?") +} + +fn combined_output(world: &E2eWorld) -> String { + format!( + "{}\n{}", + world.cli_output.as_deref().unwrap_or_default(), + world.cli_stderr.as_deref().unwrap_or_default() + ) +} + +#[given("the drive for a new Lemonade install cannot hold even Lemonade itself")] +async fn install_drive_nearly_full(world: &mut E2eWorld) { + world.command_env.push(( + "ROCM_E2E_LEMONADE_FREE_SPACE_BYTES", + NEARLY_FULL_BYTES.into(), + )); +} + +#[when("the user installs the Lemonade engine")] +async fn user_installs_lemonade(world: &mut E2eWorld) { + let (stdout, stderr, rc) = + crate::run_rocm_with_scenario_env(world, &["engines", "install", "lemonade", "--yes"]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the Lemonade install is refused before anything is downloaded")] +async fn assert_lemonade_install_refused(world: &mut E2eWorld) { + let output = combined_output(world); + // Without the hook the engine sees the real drive and starts a real + // multi-gigabyte install. Name that cause rather than a baffling mismatch. + assert!( + output.contains("Installing Lemonade"), + "no install notice: the binary under test was probably built without the \ + `rocm/e2e-test-hooks` feature, which `cargo xtask e2e` enables:\n{output}" + ); + assert_ne!( + world.cli_rc, + Some(0), + "the install must fail when the drive cannot hold it:\n{output}" + ); + let tree = data_folder_tree(world); + // The claim "Nothing was downloaded" is checked against the disk: no + // archive cache and no extracted runtime. + assert!( + !tree.join("downloads").exists(), + "an archive was downloaded despite the refusal:\n{output}" + ); + assert!( + !tree.join("runtime").exists(), + "a runtime was extracted despite the refusal:\n{output}" + ); + assert!( + !install_record(world).exists(), + "a refused install must not record Lemonade as installed:\n{output}" + ); +} + +#[then("the user is told what the Lemonade install needs and where it would go")] +async fn assert_lemonade_install_notice(world: &mut E2eWorld) { + let output = combined_output(world); + let tree = data_folder_tree(world); + for expected in [ + format!("into {}", tree.display()), + "a download of up to about".to_owned(), + "of disk space (1.0 MiB free on that drive)".to_owned(), + "not enough free disk space to install Lemonade:".to_owned(), + "Nothing was downloaded.".to_owned(), + ] { + assert!( + output.contains(&expected), + "missing `{expected}`:\n{output}" + ); + } +} + +/// Register the planted runtime: a removable install ROCm CLI made itself +/// (registry entry plus the in-tree copy `runtimes uninstall` checks before +/// deleting the folder), or, with `read_only`, an external one that uninstall +/// only unregisters. +fn plant_runtime(world: &E2eWorld, read_only: bool) { + let install_root = runtime_install_root(world); + std::fs::create_dir_all(&install_root).expect("failed to create runtime root"); + std::fs::write(install_root.join("payload.txt"), "payload") + .expect("failed to write runtime payload"); + let manifest = serde_json::to_string_pretty(&serde_json::json!({ + "runtime_key": RUNTIME_KEY, + "runtime_id": "therock-release:gfx1151", + "channel": "release", + "format": "tarball", + "family": "gfx1151", + "family_source": "manual", + "version": "1.0.0", + "install_root": install_root, + "selected_artifact_url": format!("https://example.invalid/{RUNTIME_KEY}.tar.gz"), + "read_only": read_only, + "installed_at_unix_ms": 1_700_000_000_000u64, + })) + .expect("failed to serialize runtime manifest"); + let registry = data_dir(world).join("runtimes").join("registry"); + std::fs::create_dir_all(®istry).expect("failed to create registry"); + std::fs::write(registry.join(format!("{RUNTIME_KEY}.json")), &manifest) + .expect("failed to write registry entry"); + if !read_only { + std::fs::write(install_root.join(".rocm-cli-runtime.json"), &manifest) + .expect("failed to write in-tree manifest"); + } +} + +/// Write Lemonade's install record naming `runtime_dir`, whether or not +/// anything is there. +fn record_install_at(world: &E2eWorld, runtime_dir: &std::path::Path) { + let record = install_record(world); + std::fs::create_dir_all(record.parent().expect("record has a parent")) + .expect("failed to create record dir"); + std::fs::write( + &record, + serde_json::to_string_pretty(&serde_json::json!({ + "env_id": "lemonade-embeddable-e2e", + "version": "e2e", + "runtime_dir": runtime_dir, + "lemond": runtime_dir.join(lemond_name()), + "lemonade": runtime_dir.join("lemonade"), + "backend_recipe": "llamacpp", + "backend_name": "rocm", + "installed_at_unix_ms": 0, + })) + .expect("failed to serialize install record"), + ) + .expect("failed to write install record"); +} + +#[given("Lemonade was installed in the data folder and again inside a ROCm install")] +async fn plant_two_lemonade_installs(world: &mut E2eWorld) { + plant_runtime(world, false); + // `rocm engines install lemonade` before any runtime, then `rocm serve` + // with the runtime active: one tree each, the record naming the second. + plant_lemonade_tree(&data_folder_tree(world), DATA_FOLDER_TREE_BYTES); + plant_lemonade_tree(&runtime_tree(world), RUNTIME_TREE_BYTES); + record_install_at(world, &runtime_tree(world).join("runtime")); +} + +#[given("the ROCm install that held Lemonade is gone and only the data folder's copy is left")] +async fn plant_surviving_data_folder_copy(world: &mut E2eWorld) { + // The record still names the removed runtime's folder, as it does after a + // runtime was uninstalled by a build that left the record behind. + plant_lemonade_tree(&data_folder_tree(world), DATA_FOLDER_TREE_BYTES); + record_install_at(world, &runtime_tree(world).join("runtime")); +} + +#[given("a read-only ROCm install holds a Lemonade copy")] +async fn plant_lemonade_in_read_only_runtime(world: &mut E2eWorld) { + plant_runtime(world, true); + plant_lemonade_tree(&runtime_tree(world), RUNTIME_TREE_BYTES); + record_install_at(world, &runtime_tree(world).join("runtime")); +} + +/// The report row for the Lemonade tree at `tree`, and the lines under it. +fn lemonade_row<'a>(report: &'a str, tree: &std::path::Path) -> Vec<&'a str> { + let lines: Vec<&str> = report.lines().collect(); + let path = format!("({})", tree.display()); + let at = lines + .iter() + .position(|line| line.trim_start().starts_with("- lemonade (") && line.ends_with(&path)) + .unwrap_or_else(|| panic!("no Lemonade row for {}:\n{report}", tree.display())); + lines[at..] + .iter() + .take_while(|line| { + *line == &lines[at] + || line.trim_start().starts_with("status:") + || line.trim_start().starts_with("note:") + }) + .copied() + .collect() +} + +#[then("the report lists both Lemonade installs with their sizes")] +async fn assert_both_lemonade_rows(world: &mut E2eWorld) { + let report = stdout(world).to_owned(); + let data_row = lemonade_row(&report, &data_folder_tree(world)); + assert!( + data_row[0].contains("- lemonade (data folder): 3.0 MiB"), + "the data folder's copy must be listed with its size:\n{report}" + ); + let runtime_row = lemonade_row(&report, &runtime_tree(world)); + assert!( + runtime_row[0].contains(&format!( + "- lemonade (inside ROCm install {RUNTIME_KEY}): 5.0 MiB" + )), + "the runtime's copy must be listed with its size:\n{report}" + ); + assert!( + runtime_row + .iter() + .any(|line| line.trim() == "status: in use"), + "the copy the record points at is the one in use:\n{report}" + ); +} + +#[then("the report flags the data folder's Lemonade install as unused")] +async fn assert_data_folder_copy_unused(world: &mut E2eWorld) { + let report = stdout(world).to_owned(); + let data_row = lemonade_row(&report, &data_folder_tree(world)); + assert!( + data_row + .iter() + .any(|line| line.trim() + == "status: not used right now; Lemonade runs from another install"), + "the data folder's copy must be flagged unused:\n{report}" + ); + assert!( + data_row + .iter() + .any(|line| line.trim_start().starts_with("note: reclaimable")), + "an unused copy must say its space can be reclaimed:\n{report}" + ); + // Reporting only: the tree is still on disk after the report. + assert!(data_folder_tree(world).join("runtime").exists()); +} + +#[then("the report does not call the surviving Lemonade copy unused or reclaimable")] +async fn assert_survivor_not_called_reclaimable(world: &mut E2eWorld) { + let report = stdout(world).to_owned(); + let row = lemonade_row(&report, &data_folder_tree(world)); + assert!( + row.iter().any(|line| line.trim() + == "status: not known to be used; the install record names a folder with no install"), + "a record naming a folder with no install leaves no copy known to be in use:\n{report}" + ); + assert!( + row.iter() + .all(|line| !line.contains("reclaimable") || line.contains("not marked reclaimable")), + "the only working copy must not be offered for deletion:\n{report}" + ); + assert!( + data_folder_tree(world).join("runtime").exists(), + "reporting must not delete anything" + ); +} + +#[then("the report does not say that copy is removed with its ROCm install")] +async fn assert_read_only_copy_not_removed_with_runtime(world: &mut E2eWorld) { + let report = stdout(world).to_owned(); + let row = lemonade_row(&report, &runtime_tree(world)); + assert!( + row.iter().all(|line| !line.contains("removed with it")), + "uninstalling an external ROCm install only unregisters it:\n{report}" + ); + assert!( + row.iter() + .any(|line| line + .contains("does not delete that install's folder (external or imported)")), + "the row must say why the copy outlives an uninstall:\n{report}" + ); +} + +#[then("that Lemonade copy is still on disk")] +async fn assert_lemonade_copy_still_on_disk(world: &mut E2eWorld) { + assert!( + runtime_tree(world).join("runtime").exists(), + "uninstalling an external ROCm install must leave its Lemonade copy" + ); +} + +#[when("the user uninstalls the ROCm install holding Lemonade")] +async fn user_uninstalls_runtime_holding_lemonade(world: &mut E2eWorld) { + let stdout = crate::run_rocm_ok(world, &["runtimes", "uninstall", RUNTIME_KEY, "--yes"]); + world.cli_output = Some(stdout); +} + +#[then("the CLI says Lemonade's install was removed with that ROCm install")] +async fn assert_lemonade_removed_with_runtime(world: &mut E2eWorld) { + let output = stdout(world).to_owned(); + let runtime_dir = runtime_tree(world).join("runtime"); + assert!( + output.contains(&format!( + "its install at {} is inside this runtime folder and would be removed with it", + runtime_dir.display() + )), + "the plan must warn that Lemonade goes with the runtime:\n{output}" + ); + assert!( + output.contains(&format!( + "its install at {} was inside this runtime and was removed with it", + runtime_dir.display() + )), + "the result must say Lemonade was removed with the runtime:\n{output}" + ); + assert!( + !runtime_install_root(world).exists(), + "the runtime folder, Lemonade tree included, must be gone:\n{output}" + ); +} + +#[then("Lemonade is no longer recorded as installed")] +async fn assert_lemonade_record_cleared(world: &mut E2eWorld) { + assert!( + !install_record(world).exists(), + "the record still names a deleted tree" + ); + // The CLI's own view agrees: the remaining copy is no longer called unused + // (that needs a record pointing elsewhere), and the deleted one is not listed. + let report = crate::run_rocm_ok(world, &["storage", "report"]); + let data_row = lemonade_row(&report, &data_folder_tree(world)); + assert!( + data_row + .iter() + .any(|line| line.trim() == "status: not recorded as installed"), + "with no record, the remaining copy is not recorded as installed:\n{report}" + ); + assert!( + !report.contains(&runtime_tree(world).display().to_string()), + "the removed copy must not be listed any more:\n{report}" + ); +} diff --git a/tests/e2e-cucumber/tests/feature_naming.rs b/tests/e2e-cucumber/tests/feature_naming.rs index 230f791c8b..ab4ad13c8e 100644 --- a/tests/e2e-cucumber/tests/feature_naming.rs +++ b/tests/e2e-cucumber/tests/feature_naming.rs @@ -36,6 +36,9 @@ const FEATURE_KEYS: &[(&str, &str)] = &[ ("engine_shell.feature", "engine-shell"), ("examine.feature", "examine"), ("install_lifecycle.feature", "lifecycle"), + // Not `lemonade`: the file is about the copies Lemonade leaves on disk, + // not about serving with it, which `model_serving.feature` covers. + ("lemonade_installs.feature", "lemonade-installs"), // Not `records`: a key has to say what kind of record, and this file is // about the ones a managed `rocm serve` leaves behind. ("local_server_records.feature", "server-records"), From cc26b58128230bc05b9b24bf7f9bb9b1de3aedb5 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 14:49:03 +0000 Subject: [PATCH 023/102] fix(uninstall): correct the dry-run footer and stop answering --force with --force-dev-binaries (#916) * fix(uninstall): do not create the folders a dry run then lists for removal `rocm uninstall` started the client file logger before it planned, and the logger creates `/logs`. On a machine ROCm CLI had never touched, `rocm uninstall --dry-run` therefore created the data folder and then reported it as the one thing to remove, so a dry run was not free of side effects and an empty machine looked installed. It happened with the default folders and with ROCM_CLI_DATA_DIR set alike. The `uninstall` command now skips the file logger. Other commands still log as before. Scenario uninstall-06 runs a dry run on a machine with none of the folders and asserts both that the plan says there is nothing to remove and that no folder exists afterwards; a unit test pins which invocations skip the logger. Signed-off-by: Jussi Elo * fix(uninstall): correct the dry-run footer and stop answering --force with --force-dev-binaries A dry run ended with "Choose Review uninstall to approve removal.", a control the command line does not have, and it printed that even when there was nothing to remove. A person at a terminal now gets "run it again without --dry-run" when there is something to remove, and nothing when there is not. Output that is captured (the dashboard's /uninstall and the daemon's rocm_command tool run the dry run as a subprocess) names no next step, since the real step belongs to whatever drives it. The assistant's preview has no flag to offer, so it says removal needs approval. The three wordings come from one audience value, chosen from whether the command has a terminal, rather than from strings that can drift. A mistyped --force was answered with "a similar argument exists: --force-dev-binaries", which removes the binaries of a development build, and so were near misses of it. clap ranks every declared flag by name, so --force is now declared, hidden, and refused with a pointer to --yes before anything is planned; near misses land on it instead of on the destructive flag, and a typo of the whole flag is still met with that flag. rocm uninstall --help now ends with examples that start with a dry run. Scenarios uninstall-07 to uninstall-11 cover the footer at a terminal (run again as it says, answering the prompt), outside one, with an empty plan, the --force cases, and the help examples. Every uninstall step now refuses to run without --keep-binaries. Signed-off-by: Jussi Elo * test(e2e): satisfy clippy::map_unwrap_or in the uninstall help step The step that finds the `EXAMPLES:` block in `rocm uninstall --help` used `map(..).unwrap_or_else(..)`, which the workspace's pedantic lints reject, so the clippy job and both Windows build jobs failed to compile the e2e harness. `map_or_else` keeps the same panic message and behavior. Signed-off-by: Jussi Elo * docs(uninstall): cite the test that backs the assistant's preview as a link, from the file it now lives in The testing guide named the test `lifecycle_read_mutate_split_is_honest` with a bare backticked path, which no link check can see go stale, and the path was already stale: the test moved from chat.rs to tool_policy.rs, so the guide and the two code comments that point at it named the wrong file. The guide now cites tool_policy.rs as a relative Markdown link, which the documentation link check resolves, and the comments name the file the test is in. Signed-off-by: Jussi Elo * test(uninstall): match only the unknown-argument error in the near-miss scenario The scenario asserted clap's exact wording for the unknown-argument error, including the argument it echoes. The exit code 2 and the absence of `--force-dev-binaries` already carry the point, so the step now matches only `unexpected argument`, and a clap upgrade that rewords the message no longer breaks the scenario while it still fails if the destructive flag is suggested again. Signed-off-by: Jussi Elo * refactor(uninstall): keep the refusal of the hidden --force beside the uninstall command `reject_uninstall_force` was domain logic in main.rs. It now lives in uninstall.rs, which owns `uninstall()`, as `reject_force`; the dispatch arm in main.rs calls it through the qualified `uninstall::reject_force` and the unit tests in cli.rs import it from there. uninstall.rs rather than uninstall_plan.rs, because the plan module holds the shared plan types and file-removal helpers that storage.rs reuses, while this is argument handling for the `uninstall` command alone. The architecture note for uninstall.rs names it. Signed-off-by: Jussi Elo * test(uninstall): make the near-miss guard fail whichever way clap's suggestions change The hidden `--force` works because clap's candidates for a near miss are every declared long name, hidden or not, so `--forc` is steered to `--force` and its refusal instead of to `--force-dev-binaries`. If a clap upgrade stopped suggesting hidden arguments, `--forc` would be answered with the destructive flag again. The guard test already failed on the destructive suggestion; it now also requires the near misses to name `--force`, and the two checks together do not depend on clap's wording around the suggestion. Signed-off-by: Jussi Elo * docs(uninstall): say the dry-run footer needs stdin and stdout to be terminals `interactive_terminal()` checks stdin and stdout, so a dry run with either one redirected, for example `rocm uninstall --dry-run < /dev/null` at a terminal, is answered for a captured caller and prints no footer. The doc comment on `PlanAudience::for_command` said "a terminal on both streams", and the README, manual-testing.md, testing.md and the comment above uninstall-08 spoke of output that is not read at a terminal. They now name both streams. Signed-off-by: Jussi Elo * docs(uninstall): list PlanAudience in the uninstall_plan.rs inventories `PlanAudience` is imported by `uninstall.rs`, which picks it from `interactive_terminal`, and by the `uninstall_case` property in `deletion_properties.rs`, which renders as `Captured`; `chat/mcp_bridge.rs` reaches the `Assistant` audience through `render_uninstall_dry_run` and does not import it. The module doc comment and the `uninstall_plan.rs` entry in docs/architecture.md listed the types the module owns and each consumer's imports, and named neither the type nor those two importers. Signed-off-by: Jussi Elo * test(uninstall): pin the assistant's uninstall preview with a scenario The footer "Nothing was removed. Removal needs your approval." reaches users through the assistant's `rocm_command` tool and was held only by a unit test. Scenario uninstall-12 drives the hidden `rocm mcp-call` entry point without a model, asserts the preview lists the config folder and says removal needs approval without offering a flag, and checks that the folder still holds the user's files. The call is made without `--allow-mutation`, and the isolated environment the other uninstall steps use is now one helper they share. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 951d2cbffeb393538cf97f76beab413397b212ad) Signed-off-by: juhovainio --- README.md | 14 +- apps/rocm/src/cli.rs | 129 ++++++- apps/rocm/src/deletion_properties.rs | 1 + apps/rocm/src/main.rs | 22 +- apps/rocm/src/uninstall.rs | 30 +- apps/rocm/src/uninstall_plan.rs | 164 +++++++- docs/architecture.md | 4 +- docs/manual-testing.md | 16 +- docs/testing.md | 15 +- tests/e2e-cucumber/features/uninstall.feature | 76 ++++ .../e2e-cucumber/tests/e2e/uninstall_steps.rs | 362 +++++++++++++++++- 11 files changed, 787 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index 6a722243b2..469757d321 100644 --- a/README.md +++ b/README.md @@ -1788,11 +1788,15 @@ download's progress: `rocm uninstall` removes four categories of files that ROCm CLI manages on this computer: its binaries, configuration, data, and caches. It lists what it will do and asks `[y/N]` (default: no) before doing it. Use `--dry-run` to only list, -and `--yes` to skip the confirmation. It does not create the folders it reports -on: with `--keep-binaries`, on a computer where ROCm CLI never ran, it says there -is nothing to remove. Outside an interactive terminal, when there is something -to remove, it refuses with exit code 1 and removes nothing unless you pass -`--yes`. Each `--keep-*` flag skips the removal of one category: +and `--yes` to skip the confirmation. There is no `--force`; `--force-dev-binaries` +is a separate opt-in for development builds. At a terminal, a dry run that finds +something to remove ends by saying to run the same command again without +`--dry-run`; with nothing to remove, or when stdin or stdout is not a terminal +(its output is piped or its input is redirected), it says nothing about a next step. It does not create the folders it reports on: +with `--keep-binaries`, on a computer where ROCm CLI never ran, it says there is +nothing to remove. Outside an interactive terminal, when there is something to +remove, it refuses with exit code 1 and removes nothing unless you pass `--yes`. +Each `--keep-*` flag skips the removal of one category: - `--keep-binaries` keeps the installed ROCm CLI binaries. - `--keep-config` keeps your saved settings. diff --git a/apps/rocm/src/cli.rs b/apps/rocm/src/cli.rs index 7cc3bd6cd1..ff922a0664 100644 --- a/apps/rocm/src/cli.rs +++ b/apps/rocm/src/cli.rs @@ -691,6 +691,10 @@ rocm logs --search error timeout")] /// PATH change the installer made (its marked shell-profile block, or the /// user PATH entry it recorded adding on Windows), unless other programs /// share the install folder. + #[command(after_help = "EXAMPLES:\n \ +rocm uninstall --dry-run\n \ +rocm uninstall\n \ +rocm uninstall --keep-data --yes")] Uninstall { /// Do not ask for interactive confirmation. Without it, the command /// asks `[y/N]` (default: no) in an interactive terminal; outside one, @@ -718,6 +722,11 @@ rocm logs --search error timeout")] /// Allow removing development binaries inside the current build tree. #[arg(long)] force_dev_binaries: bool, + /// Not an option; accepted only to say so. Without it clap answers a + /// mistyped `--force` (a likely guess for skipping the confirmation) + /// with `--force-dev-binaries`, which removes development binaries. + #[arg(long, hide = true, num_args = 0..=1, default_missing_value = "")] + force: Option, }, } @@ -1467,7 +1476,7 @@ impl PermissionsModeArg { #[cfg(test)] mod tests { use super::*; - use crate::{cli_command, device_policy_name, engine_inventory}; + use crate::{cli_command, device_policy_name, engine_inventory, uninstall::reject_force}; use clap::{CommandFactory, Parser}; use rocm_engine_protocol::DevicePolicy; @@ -2419,6 +2428,124 @@ mod tests { .expect_err("an abbreviated subcommand name must not reach uninstall"); } + /// `--force` is a likely guess for "skip the confirmation", and without a + /// declared `--force` clap answers it, and every near miss of it, with + /// `--force-dev-binaries`, which removes the binaries of a development build. + /// The scenario `uninstall-mistyped-force-is-not-answered-with-a-destructive-flag` + /// proves the real binary's output; this pins the parse behind it. + #[test] + fn a_mistyped_force_never_reaches_the_dev_binaries_flag() { + // `--force` itself parses, so the handler can say what to use instead. + let cli = Cli::try_parse_from(["rocm", "uninstall", "--force"]).unwrap(); + let Some(Command::Uninstall { + force, + force_dev_binaries, + .. + }) = cli.command + else { + panic!("not an uninstall command"); + }; + assert!(force.is_some() && !force_dev_binaries); + let refusal = reject_force(force.is_some()).unwrap_err().to_string(); + assert!(refusal.contains("`--force` is not an option"), "{refusal}"); + assert!(refusal.contains("--yes"), "{refusal}"); + assert!(!refusal.contains("force-dev-binaries"), "{refusal}"); + assert!(reject_force(false).is_ok()); + // `--force=x` is the same mistake, and gets the same answer rather than + // a usage line that lists every flag. + let cli = Cli::try_parse_from(["rocm", "uninstall", "--force=x"]).unwrap(); + let Some(Command::Uninstall { force, .. }) = cli.command else { + panic!("not an uninstall command"); + }; + assert!(reject_force(force.is_some()).is_err()); + + // Near misses and the forms clap suggests from elsewhere in the command + // line never offer the destructive flag. + for args in [ + vec!["uninstall", "--forc"], + vec!["uninstall", "--forcee"], + vec!["uninstall", "--keep-binaries", "--forc"], + vec!["uninstall", "-f"], + vec!["--force", "uninstall"], + ] { + let error = cli_command() + .try_get_matches_from(std::iter::once("rocm").chain(args.iter().copied())) + .unwrap_err(); + let rendered = error.render().to_string(); + assert!( + !rendered.contains("force-dev-binaries"), + "{args:?} was answered with the destructive flag:\n{rendered}" + ); + assert_eq!(error.exit_code(), 2, "{args:?}"); + } + + // The guard depends on clap offering the hidden `--force` as the + // suggestion for a near miss: its candidates are every declared long + // name, hidden or not. If a clap upgrade stopped suggesting hidden + // arguments, `--forc` would be answered with `--force-dev-binaries` + // again, so the near misses must name `--force` and not the + // destructive flag. The two checks together are independent of + // clap's wording around the suggestion. + for args in [ + vec!["uninstall", "--forc"], + vec!["uninstall", "--keep-binaries", "--forc"], + ] { + let rendered = cli_command() + .try_get_matches_from(std::iter::once("rocm").chain(args.iter().copied())) + .unwrap_err() + .render() + .to_string(); + assert!( + rendered.contains("--force") && !rendered.contains("force-dev-binaries"), + "{args:?} must be steered to the declared `--force`, not the destructive flag:\n{rendered}" + ); + } + + // The whole flag misspelled is still met with the flag it was meant to be. + let error = cli_command() + .try_get_matches_from(["rocm", "uninstall", "--force-dev-binary"]) + .unwrap_err() + .render() + .to_string(); + assert!(error.contains("--force-dev-binaries"), "{error}"); + + // It stays out of the help. + let mut command = cli_command(); + let help = command + .find_subcommand_mut("uninstall") + .unwrap() + .render_long_help() + .to_string(); + assert!(!help.contains("Not an option"), "{help}"); + } + + /// The help leads with the safe preview, then the real run, as the tip a + /// person reads before running a command that deletes things. + #[test] + fn uninstall_help_shows_a_dry_run_before_the_real_run() { + let mut command = cli_command(); + let help = command + .find_subcommand_mut("uninstall") + .expect("uninstall subcommand") + .render_long_help() + .to_string(); + let examples = help.split_once("EXAMPLES:").map_or_else( + || panic!("`rocm uninstall --help` has no examples:\n{help}"), + |(_, rest)| rest, + ); + let rows: Vec<&str> = examples + .lines() + .map(str::trim) + .filter(|line| !line.is_empty()) + .collect(); + assert_eq!( + rows.first().copied(), + Some("rocm uninstall --dry-run"), + "{examples}" + ); + assert!(rows.contains(&"rocm uninstall"), "{examples}"); + } + #[test] fn update_rejects_zero_timeout_secs() { let error = Cli::try_parse_from(["rocm", "update", "--json", "--timeout-secs", "0"]) diff --git a/apps/rocm/src/deletion_properties.rs b/apps/rocm/src/deletion_properties.rs index 243e2ab06c..a664dbef0b 100644 --- a/apps/rocm/src/deletion_properties.rs +++ b/apps/rocm/src/deletion_properties.rs @@ -673,6 +673,7 @@ fn uninstall_case( dry_run: true, ..options }, + crate::uninstall_plan::PlanAudience::Captured, ); let planned: BTreeSet = plan.actions.iter().map(|e| e.path.clone()).collect(); prop_assert_eq!(&listed_in_render(&rendered), &planned, "{}", rendered); diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 61a6f67ff8..ac9dd5d62a 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -955,15 +955,19 @@ fn dispatch(cli: Cli) -> Result<()> { keep_data, keep_cache, force_dev_binaries, - }) => uninstall(UninstallOptions { - yes, - dry_run, - keep_binaries, - keep_config, - keep_data, - keep_cache, - force_dev_binaries, - }), + force, + }) => { + uninstall::reject_force(force.is_some())?; + uninstall(UninstallOptions { + yes, + dry_run, + keep_binaries, + keep_config, + keep_data, + keep_cache, + force_dev_binaries, + }) + } Some(Command::Completions { shell }) => { let mut cmd = completion_command(); clap_complete::generate(shell, &mut cmd, "rocm", &mut std::io::stdout()); diff --git a/apps/rocm/src/uninstall.rs b/apps/rocm/src/uninstall.rs index aec8c1f57e..65aa4d7f96 100644 --- a/apps/rocm/src/uninstall.rs +++ b/apps/rocm/src/uninstall.rs @@ -18,10 +18,10 @@ use std::io::Write as _; use std::path::{Path, PathBuf}; use anyhow::{Context, Result, bail}; -use rocm_core::{AppPaths, require_yes_or_interactive}; +use rocm_core::{AppPaths, interactive_terminal, require_yes_or_interactive}; use crate::uninstall_plan::{ - UninstallOptions, UninstallPlan, build_uninstall_plan, confirm_uninstall, + PlanAudience, UninstallOptions, UninstallPlan, build_uninstall_plan, confirm_uninstall, is_rocm_install_entry_name, remove_path, render_uninstall_plan, }; @@ -63,10 +63,31 @@ impl PathEdit { } } +/// `rocm uninstall` has no `--force`; the flag is declared (hidden) only so this +/// can answer it. Left undeclared, clap's nearest-name guess for `--force` is +/// `--force-dev-binaries`, which removes development binaries, and a near miss +/// such as `--forc` gets that same suggestion. Declared, those guesses land here +/// instead, before anything is planned or removed. +pub(crate) fn reject_force(force: bool) -> Result<()> { + if force { + bail!( + "`--force` is not an option of `rocm uninstall`; use `--yes` to skip the confirmation" + ); + } + Ok(()) +} + pub(crate) fn uninstall(options: UninstallOptions) -> Result<()> { let paths = AppPaths::discover()?; let plan = build_uninstall_plan(&paths, &options)?; - print!("{}", render_uninstall_plan(&plan, &options)); + print!( + "{}", + render_uninstall_plan( + &plan, + &options, + PlanAudience::for_command(interactive_terminal()) + ) + ); if plan_is_empty(&plan) || options.dry_run { return Ok(()); @@ -837,7 +858,8 @@ mod tests { self_delete: Some(PathBuf::from("/opt/bin/rocm.exe")), ..UninstallPlan::default() }; - let output = render_uninstall_plan(&plan, &UninstallOptions::default()); + let output = + render_uninstall_plan(&plan, &UninstallOptions::default(), PlanAudience::Captured); for expected in [ "2 item(s) would be removed:", " - install record: /opt/bin/.rocm-cli-path-edit", diff --git a/apps/rocm/src/uninstall_plan.rs b/apps/rocm/src/uninstall_plan.rs index ea210c2eae..e553dc9e47 100644 --- a/apps/rocm/src/uninstall_plan.rs +++ b/apps/rocm/src/uninstall_plan.rs @@ -21,22 +21,26 @@ //! describe the `uninstall` command's own, exclusive logic. //! //! This module owns: the plan types (`UninstallOptions`, `UninstallPlan`, -//! `UninstallPlanEntry`), building the plan (`build_uninstall_plan`, +//! `UninstallPlanEntry`), the `PlanAudience` a rendered plan is written for +//! (a person at a terminal, a captured caller, or the assistant's preview), +//! which decides what the dry-run footer says comes next, building the plan (`build_uninstall_plan`, //! `plan_installed_binaries`, and the installed-binary/shared-cache //! discovery helpers behind them), rendering it (`render_uninstall_dry_run`, //! `render_uninstall_plan`), the `[y/N]` confirmation prompt //! (`confirm_uninstall`), and the filesystem removal primitive both //! `uninstall.rs` and `storage.rs` call to carry a plan out (`remove_path`, //! and the entry-resolution helpers behind it). `uninstall.rs` calls into it -//! for `UninstallOptions`/`UninstallPlan`/`build_uninstall_plan`/ +//! for `PlanAudience`/`UninstallOptions`/`UninstallPlan`/`build_uninstall_plan`/ //! `confirm_uninstall`/`is_rocm_install_entry_name`/`remove_path`/ //! `render_uninstall_plan`; `storage.rs` calls into it for //! `UninstallPlan`/`UninstallPlanEntry`/`confirm_uninstall`; `runtimes_cmd.rs` //! calls into it for `confirm_uninstall` (the same `[y/N]` prompt, reused for //! `rocm runtimes uninstall`); `chat/mcp_bridge.rs` calls into it for //! `render_uninstall_dry_run` (the deterministic uninstall-preview tool -//! result); `deletion_properties.rs`'s recursive-deletion property tests call -//! into it for `UninstallOptions`/`build_uninstall_plan`/`remove_path`/ +//! result, which renders for `PlanAudience::Assistant` itself and so does not +//! import the type); `deletion_properties.rs`'s recursive-deletion property +//! tests call into it for `PlanAudience` (the `uninstall_case` property renders +//! as `Captured`)/`UninstallOptions`/`build_uninstall_plan`/`remove_path`/ //! `render_uninstall_plan`. use std::fmt::Write as _; @@ -268,10 +272,61 @@ pub(crate) fn render_uninstall_dry_run(paths: &AppPaths) -> Result { ..UninstallOptions::default() }; let plan = build_uninstall_plan(paths, &options)?; - Ok(render_uninstall_plan(&plan, &options)) + Ok(render_uninstall_plan( + &plan, + &options, + PlanAudience::Assistant, + )) } -pub(crate) fn render_uninstall_plan(plan: &UninstallPlan, options: &UninstallOptions) -> String { +/// Who reads a rendered plan, which decides what its dry-run footer can truthfully +/// say comes next. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum PlanAudience { + /// A person at a terminal ran `rocm uninstall --dry-run` themselves: running + /// the same command again without `--dry-run` is the next step. + Terminal, + /// The output is captured by a caller (a script, the dashboard, the daemon's + /// tool). Whatever drives that caller has its own way to proceed, so there + /// is no next step to name; "run it again" would address the wrong reader. + Captured, + /// The assistant's read-only preview tool: it has no flag to offer, and the + /// removal itself goes through an approval step. + Assistant, +} + +impl PlanAudience { + /// The audience for `rocm uninstall` itself, from whether `interactive_terminal` + /// holds: stdin and stdout are both terminals. A dry run with either one + /// redirected, `--dry-run < /dev/null` at a terminal for one, is `Captured` + /// and prints no footer. + pub(crate) const fn for_command(interactive_terminal: bool) -> Self { + if interactive_terminal { + Self::Terminal + } else { + Self::Captured + } + } + + /// What a dry run says to do next, if anything. + const fn dry_run_footer(self) -> Option<&'static str> { + match self { + Self::Terminal => { + Some("Nothing was removed. To remove these items, run it again without --dry-run.") + } + Self::Captured => None, + // The assistant's own uninstall is an approval action; see + // `lifecycle_read_mutate_split_is_honest` in `tool_policy.rs`. + Self::Assistant => Some("Nothing was removed. Removal needs your approval."), + } + } +} + +pub(crate) fn render_uninstall_plan( + plan: &UninstallPlan, + options: &UninstallOptions, + audience: PlanAudience, +) -> String { let mut output = String::new(); let _ = writeln!(output, "Uninstall review"); let _ = writeln!(output); @@ -299,9 +354,13 @@ pub(crate) fn render_uninstall_plan(plan: &UninstallPlan, options: &UninstallOpt for skipped in &plan.skipped { let _ = writeln!(output, " - {skipped}"); } - if options.dry_run { + // An empty plan has nothing to approve, so a dry run says nothing about a next step. + if options.dry_run + && !uninstall::plan_is_empty(plan) + && let Some(footer) = audience.dry_run_footer() + { let _ = writeln!(output); - let _ = writeln!(output, "Choose Review uninstall to approve removal."); + let _ = writeln!(output, "{footer}"); } output } @@ -521,6 +580,95 @@ mod tests { use super::*; use crate::tests::test_paths; + fn plan_with_a_binary() -> UninstallPlan { + UninstallPlan { + self_delete: Some(PathBuf::from("/opt/bin/rocm")), + ..UninstallPlan::default() + } + } + + fn dry_run() -> UninstallOptions { + UninstallOptions { + dry_run: true, + ..UninstallOptions::default() + } + } + + /// A person at a terminal who dry-runs is told to run the same command again + /// without `--dry-run`: the next step is a rerun, not a control the command + /// line lacks. The scenario `uninstall-07` runs it again at a terminal and + /// checks that the listed folder goes. + #[test] + fn terminal_dry_run_footer_says_to_run_it_again_without_dry_run() { + let output = + render_uninstall_plan(&plan_with_a_binary(), &dry_run(), PlanAudience::Terminal); + assert!( + output.ends_with( + "\nNothing was removed. To remove these items, run it again without --dry-run.\n" + ), + "unexpected footer in:\n{output}" + ); + assert!(!output.contains("Review uninstall"), "{output}"); + } + + /// Output a caller captures is not addressed to someone who can rerun it, so + /// it carries no next step (the dashboard's `/uninstall` and the daemon's + /// `rocm_command` tool run the dry run as a subprocess, where the real step + /// is theirs to offer). + #[test] + fn captured_dry_run_names_no_next_step() { + let output = + render_uninstall_plan(&plan_with_a_binary(), &dry_run(), PlanAudience::Captured); + assert!(!output.contains("Nothing was removed."), "{output}"); + assert!(!output.contains("run it again"), "{output}"); + assert!(output.contains("1 item(s) would be removed:"), "{output}"); + } + + /// The command picks its audience from whether it has a terminal. + #[test] + fn the_command_is_addressed_to_a_terminal_only_when_it_has_one() { + assert_eq!(PlanAudience::for_command(true), PlanAudience::Terminal); + assert_eq!(PlanAudience::for_command(false), PlanAudience::Captured); + } + + /// An empty plan has nothing to approve, so a dry run adds no next step for + /// anyone; a real run adds none either, since its confirmation prompt follows. + #[test] + fn footer_is_only_for_a_dry_run_that_has_something_to_remove() { + for audience in [ + PlanAudience::Terminal, + PlanAudience::Captured, + PlanAudience::Assistant, + ] { + let empty = render_uninstall_plan(&UninstallPlan::default(), &dry_run(), audience); + assert!(!empty.contains("Nothing was removed."), "{empty}"); + let real_run = render_uninstall_plan( + &plan_with_a_binary(), + &UninstallOptions::default(), + audience, + ); + assert!(!real_run.contains("Nothing was removed."), "{real_run}"); + } + } + + /// The assistant's preview has no flag to offer, so it must not point at a + /// command line it cannot run; what it can truthfully say is that removal + /// needs approval (`lifecycle_read_mutate_split_is_honest` in `tool_policy.rs` pins + /// that its own `uninstall` is an approval action). It does not say "these + /// items": the preview always plans with the default options, while the + /// approval carries the user's own. + #[test] + fn assistant_dry_run_footer_names_no_command() { + let output = + render_uninstall_plan(&plan_with_a_binary(), &dry_run(), PlanAudience::Assistant); + assert!( + output.ends_with("\nNothing was removed. Removal needs your approval.\n"), + "unexpected footer in:\n{output}" + ); + assert!(!output.contains("rocm uninstall"), "{output}"); + assert!(!output.contains("these items"), "{output}"); + } + /// A cache that has moved inside a directory uninstall already removes must /// not be reported as "not removed" — the note would be false. #[test] diff --git a/docs/architecture.md b/docs/architecture.md index d563babc3a..e7f079d9bf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -119,8 +119,8 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests and on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for the startup-check record (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. - [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. - [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_serve_command_safety`, the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), and `validate_chat_rocm_command_safety`, which applies them to a `serve` argv and also refuses a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_serve_command_safety` and `validate_chat_rocm_command_safety` here instead of through `chat`, and keeps no copy of the serve rules. `chat.rs`'s child modules call into it and `chat.rs` re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). -- [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. -- [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) — mechanical relocation, shares crate-visible types (Phase 6f, ROCMAI-212): the uninstall-plan types (`UninstallOptions`, `UninstallPlan`, `UninstallPlanEntry`), building the plan (`build_uninstall_plan`, `plan_installed_binaries`, and the installed-binary/shared-cache discovery helpers behind them), rendering it (`render_uninstall_dry_run`, `render_uninstall_plan`), the `[y/N]` confirmation prompt (`confirm_uninstall`), and the filesystem removal primitive both [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`storage.rs`](../apps/rocm/src/storage.rs) call to carry a plan out (`remove_path`, and the entry-resolution helpers behind it). Kept separate from `uninstall.rs` rather than folded into it, since `uninstall.rs`'s own doc comment and name both describe `uninstall()`'s exclusive logic (the installer `PATH` undo and running-executable deletion), while this module's plan/render/removal helpers are shared with `storage.rs`. `main.rs` (`UninstallOptions`), `uninstall.rs`, `storage.rs`, `runtimes_cmd.rs`, `chat/mcp_bridge.rs`, and [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) (Unix-only property tests) each import the subset of these `pub(crate)` types and helpers they need; see this module's own doc comment for which symbol each one uses. +- [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, the refusal of the hidden `--force` (`reject_force`, which `main.rs` calls from the `uninstall` dispatch arm), plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. +- [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) — mechanical relocation, shares crate-visible types (Phase 6f, ROCMAI-212): the uninstall-plan types (`UninstallOptions`, `UninstallPlan`, `UninstallPlanEntry`), the `PlanAudience` a rendered plan is written for (a person at a terminal, a captured caller, or the assistant's preview), which decides what the dry-run footer says comes next, building the plan (`build_uninstall_plan`, `plan_installed_binaries`, and the installed-binary/shared-cache discovery helpers behind them), rendering it (`render_uninstall_dry_run`, which renders for the assistant, and `render_uninstall_plan`), the `[y/N]` confirmation prompt (`confirm_uninstall`), and the filesystem removal primitive both [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`storage.rs`](../apps/rocm/src/storage.rs) call to carry a plan out (`remove_path`, and the entry-resolution helpers behind it). Kept separate from `uninstall.rs` rather than folded into it, since `uninstall.rs`'s own doc comment and name both describe `uninstall()`'s exclusive logic (the installer `PATH` undo and running-executable deletion), while this module's plan/render/removal helpers are shared with `storage.rs`. `main.rs` (`UninstallOptions`), `uninstall.rs` (which also imports `PlanAudience` and picks it from `interactive_terminal`), `storage.rs`, `runtimes_cmd.rs`, `chat/mcp_bridge.rs` (which calls `render_uninstall_dry_run` and does not import `PlanAudience`), and [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) (Unix-only property tests, which render as `PlanAudience::Captured`) each import the subset of these `pub(crate)` types and helpers they need; see this module's own doc comment for which symbol each one uses. - [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types (Phase 6f, ROCMAI-210): `rocm update`'s command handler (`update()`, the body of the `update` arm of `dispatch()`) and its rendering/activation logic — the read-only status report (`render_update_text`, `append_update_surfaces`), the predicate deciding whether `update` should route into the mutating path instead of that report (`update_should_preview_or_apply`), applying a runtime update (`apply_runtime_update`, `append_update_activate_summary`), and the update-source/installed-runtime pickers behind it (`select_runtime_update_source`, `select_installed_update_runtime`). `const ROLLBACK_RECOVERY_HINT` stays in `main.rs` as a shared crate-root constant, because callers outside this module also use it ([`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimesCommand::Activate` handler and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `render_sdk_install_success`); this module reaches it via its `use crate::{...};` import, the same way `gpu_policy.rs`'s `device_policy_name` stays in `main.rs`. `update()` is `pub(crate)` and called from [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` through a re-import (`use crate::update_cmd::update;`), and calls `update_should_preview_or_apply`, `apply_runtime_update` and `render_update_text` itself; `render_update_text` is also `pub(crate)` and read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the read-only `rocm update` handling), while `apply_runtime_update` and `update_should_preview_or_apply` keep the `pub(crate)` they had when `dispatch()` called them directly, though only `update()` calls them now; `select_runtime_update_source` is also `pub(crate)`, read from [`therock_version_proptests.rs`](../apps/rocm/src/therock_version_proptests.rs)'s property tests via `crate::update_cmd::select_runtime_update_source`. `append_update_surfaces`, `append_update_activate_summary`, and `select_installed_update_runtime` have no callers outside this module and stay private. The cluster's regression tests live in `update_cmd.rs`'s own `mod tests`, reaching shared fixtures (`test_paths`, `test_runtime_manifest_for_update`, `write_test_pip_runtime`) through `crate::tests::`/`crate::test_support::` the same way other mechanically relocated modules do. Further `apps/rocm` modules, not part of the `main.rs` split: diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 4092a27e7d..062ec8d8cc 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1165,7 +1165,12 @@ Expected result: - The dry run lists the binaries (on Windows, `rocm.exe` is marked as deleted as the command exits) and, under `PATH changes from the installer that would be undone:`, the shell profile - (Linux) or the user `PATH` entry (Windows). It removes nothing. + (Linux) or the user `PATH` entry (Windows). It removes nothing, and at a + terminal ends with `Nothing was removed. To remove these items, run it again + without --dry-run.` Running the same command again without `--dry-run` asks + `[y/N]` and, on yes, removes what was listed (scenario `uninstall-07`). With + its output piped to a file or another command, or its input redirected, it + lists the same items and omits that last line (scenario `uninstall-08`). - After confirming, the install folder holds no `rocm`/`rocmd` files, and a new terminal no longer gets the install folder on `PATH` from the installer's change. @@ -1177,7 +1182,14 @@ Expected result: `Nothing managed by rocm-cli would be removed.` and leaves the home folder empty (scenario `uninstall-06`): run `HOME=$(mktemp -d) rocm uninstall --keep-binaries --dry-run`, then `ls -A` - on that folder. + on that folder. It adds no line about how to remove anything (scenario + `uninstall-09`). +- Passing `--force` to `rocm uninstall` is refused with a message that says it + is not an option and points to `--yes`; a near miss such as `--forc` is an + unknown-argument error (exit 2). Neither + suggests `--force-dev-binaries` (scenario `uninstall-10`). + `rocm uninstall --help` ends with examples that start with a dry run + (scenario `uninstall-11`). - Without `--yes` and without a terminal, the command prints the same review, then refuses with `uninstall requires --yes outside an interactive terminal` and exit code 1, and removes nothing. On Linux or WSL, check it with no diff --git a/docs/testing.md b/docs/testing.md index 229dfcabcb..a0d360b38b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1489,9 +1489,18 @@ The `uninstall-NN` E2E scenarios (`features/uninstall.feature`) check the same behaviour through the built binary, confined to the scenario's own folder, including a folder that cannot be checked stopping the uninstall before anything is removed, and a dry run on a machine ROCm CLI never touched -(`uninstall-06`) reporting nothing to remove and creating no folder. The -`uninstall` command does not start the file logger, which would otherwise create -the data folder's `logs` folder before the plan is built: +(`uninstall-06`) reporting nothing to remove and creating no folder. At a +terminal (a pseudo-terminal on the Linux lane), a dry run that lists something +ends by saying to run it again without `--dry-run`, and `uninstall-07` does that +and checks the listed folder goes; a run whose stdin or stdout is not a terminal, and an +empty plan even at one, name no next step (`uninstall-08`, `uninstall-09`). A +mistyped `--force` is refused with a pointer to `--yes`, and a near miss of it +is not answered with `--force-dev-binaries` (`uninstall-10`); the help starts +its examples with a dry run (`uninstall-11`). The assistant's uninstall preview +ends differently, since it has no flag to offer: it says removal needs approval, +which `lifecycle_read_mutate_split_is_honest` in [tool_policy.rs](../apps/rocm/src/tool_policy.rs) backs. +The `uninstall` command does not start the file logger, which would otherwise +create the data folder's `logs` folder before the plan is built: ```bash cargo xtask e2e -- -n "uninstall-" diff --git a/tests/e2e-cucumber/features/uninstall.feature b/tests/e2e-cucumber/features/uninstall.feature index 286a59c0c2..63e2568f33 100644 --- a/tests/e2e-cucumber/features/uninstall.feature +++ b/tests/e2e-cucumber/features/uninstall.feature @@ -80,3 +80,79 @@ Feature: Uninstall Then the uninstall succeeds And the uninstall reports that nothing would be removed And ROCm CLI has still not created its folders + + # A dry run run by a person at a terminal ends by saying to run it again + # without `--dry-run`. It once said "Choose Review uninstall", a control the + # command line does not have. The promise is the line and the state together: + # the dry run removes nothing, and running the same command again, as the + # line says and answering its prompt, removes what was listed. + @id:uninstall-dry-run-at-a-terminal-says-to-run-it-again-to-remove @requires-os:linux + Scenario: uninstall-07 - At a terminal, a dry run removes nothing and running it again without --dry-run removes what it listed + Given the config folder holds the user's files + When the user dry-runs an uninstall of only the config at a terminal + Then the uninstall succeeds + And the dry run says to run it again without --dry-run + And the config folder still holds the user's files + When the user runs it again without --dry-run at a terminal and answers yes + Then the uninstall succeeds + And the config folder is gone + + # A run whose stdin or stdout is not a terminal (a script, the dashboard's + # `/uninstall`, the daemon's `rocm_command` tool) has no one to tell to run it + # again: the real step belongs to whatever drives it. It lists what it would + # remove, offers no step, and removes nothing. + @id:uninstall-dry-run-outside-a-terminal-offers-no-next-step + Scenario: uninstall-08 - Outside a terminal, a dry run lists what it would remove and offers no step to remove it + Given the config folder holds the user's files + When the user dry-runs an uninstall of only the config + Then the uninstall succeeds + And the dry run lists the config folder + And the dry run offers no step to remove anything + And the config folder still holds the user's files + + # With nothing to remove there is nothing to approve, so even a person at a + # terminal is offered no step. A config folder that is kept and still holds + # the user's files shows the "nothing to remove" is true. + @id:uninstall-dry-run-with-nothing-to-remove-offers-no-next-step @requires-os:linux + Scenario: uninstall-09 - At a terminal, a dry run with nothing to remove offers no step to remove it + Given the config folder holds the user's files + When the user dry-runs an uninstall that keeps everything at a terminal + Then the uninstall succeeds + And the uninstall reports that nothing would be removed + And the dry run offers no step to remove anything + And the config folder still holds the user's files + + # `--force` is a natural guess for skipping the confirmation, and the nearest + # real flag by name is `--force-dev-binaries`, which removes the binaries of a + # development build. `--force` is refused with the flag to use instead, a near + # miss of it never suggests the destructive flag, and nothing is removed. + @id:uninstall-mistyped-force-is-not-answered-with-a-destructive-flag + Scenario: uninstall-10 - A mistyped --force is refused without pointing at the flag that removes development binaries + Given the config folder holds the user's files + When the user uninstalls the config with --force + Then the uninstall is refused and told to use --yes instead + And the config folder still holds the user's files + When the user uninstalls the config with a near miss of --force + Then the error does not point at the flag that removes development binaries + And the config folder still holds the user's files + + @id:uninstall-help-shows-a-dry-run-before-the-real-run + Scenario: uninstall-11 - The uninstall help shows a dry run before the real run + When the user asks for the uninstall help + Then the help shows a dry run before the real run + + # The assistant's uninstall preview (the `rocm_command` tool of the chat + # assistant and the dashboard, called here without a model through the hidden + # `rocm mcp-call`) renders the same plan as a dry run, but it has no flag to + # offer, so it says that removal needs approval. It is the one run here that + # cannot pass --keep-binaries: the preview takes no flags and removes nothing, + # and it is made without --allow-mutation, so a call that would remove + # something is refused. + @id:uninstall-assistant-preview-says-removal-needs-approval + Scenario: uninstall-12 - The assistant's uninstall preview lists what it would remove, says removal needs approval, and removes nothing + Given the config folder holds the user's files + When the assistant previews an uninstall + Then the preview succeeds + And the preview lists the config folder + And the preview says removal needs approval + And the config folder still holds the user's files diff --git a/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs b/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs index 3b818cffb5..0dbf34f51c 100644 --- a/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs @@ -14,6 +14,7 @@ use std::path::{Path, PathBuf}; use cucumber::{given, then, when}; use crate::E2eWorld; +use crate::e2e::tui_driver::{TuiSession, default_timeout}; const USER_FILE: &str = "keep.txt"; @@ -76,6 +77,19 @@ async fn cache_behind_looping_link(world: &mut E2eWorld) { E2eWorld::link_directory(&looping, &looping).expect("failed to plant looping link"); } +/// HOME, the uv cache and the Hugging Face cache inside the scenario's own +/// folder, so nothing a run resolves can reach the runner's shared caches. +fn isolated_env(world: &E2eWorld) -> Vec<(&'static str, String)> { + let root = root(world); + let home = root.join("home"); + std::fs::create_dir_all(&home).expect("failed to create isolated HOME"); + vec![ + ("HOME", home.display().to_string()), + ("UV_CACHE_DIR", root.join("uv-cache").display().to_string()), + ("HF_HOME", root.join("hf-home").display().to_string()), + ] +} + fn run_uninstall(world: &mut E2eWorld, keep: &[&str], settings: &[(&str, &str)]) { run_uninstall_with(world, true, keep, settings); } @@ -89,24 +103,25 @@ fn run_uninstall_with( keep: &[&str], settings: &[(&str, &str)], ) { - let root = root(world).to_path_buf(); - let home = root.join("home"); - std::fs::create_dir_all(&home).expect("failed to create isolated HOME"); - let home = home.display().to_string(); - let uv = root.join("uv-cache").display().to_string(); - let hf = root.join("hf-home").display().to_string(); let mut args = vec!["uninstall", "--keep-binaries"]; if confirmed { args.push("--yes"); } args.extend_from_slice(keep); - let mut envs = vec![ - ("HOME", home.as_str()), - ("UV_CACHE_DIR", &uv), - ("HF_HOME", &hf), - ]; + run_uninstall_args(world, &args, settings); +} + +/// Run `rocm` with exactly `args`, confined to the scenario's own folders like +/// every other uninstall here. +fn run_uninstall_args(world: &mut E2eWorld, args: &[&str], settings: &[(&str, &str)]) { + assert_keeps_binaries(args); + let isolated = isolated_env(world); + let mut envs = isolated + .iter() + .map(|(name, value)| (*name, value.as_str())) + .collect::>(); envs.extend_from_slice(settings); - let (stdout, stderr, rc) = crate::run_rocm_with_env(world, &args, &envs); + let (stdout, stderr, rc) = crate::run_rocm_with_env(world, args, &envs); world.cli_output = Some(stdout); world.cli_stderr = Some(stderr); world.cli_rc = Some(rc); @@ -451,3 +466,326 @@ async fn config_intact(world: &mut E2eWorld) { output(world) ); } + +/// Every run here keeps the binaries: the binary under test must survive, and an +/// uninstall that reached it would delete the program the scenario is running. +/// Checked at the point a run starts, so no step can reach it by accident. +fn assert_keeps_binaries(args: &[&str]) { + assert!( + args.contains(&"--keep-binaries"), + "an uninstall in a scenario must pass --keep-binaries: {args:?}" + ); +} + +/// The arguments for uninstalling only the config folder, as a dry run or for real. +fn config_only_args(dry_run: bool) -> Vec<&'static str> { + let mut args = vec![ + "uninstall", + "--keep-binaries", + "--keep-data", + "--keep-cache", + ]; + if dry_run { + args.push("--dry-run"); + } + args +} + +/// Run `rocm` under a pseudo-terminal, confined to the scenario's folders like +/// every other uninstall here, so the command sees a terminal on both streams as +/// a person typing it would. Answers its `[y/N]` prompt when `answer` is given. +async fn run_uninstall_at_terminal(world: &mut E2eWorld, args: &[&str], answer: Option<&str>) { + assert_keeps_binaries(args); + let root = root(world).to_path_buf(); + let home = root.join("home"); + std::fs::create_dir_all(&home).expect("failed to create isolated HOME"); + let home = home.display().to_string(); + let uv = root.join("uv-cache").display().to_string(); + let hf = root.join("hf-home").display().to_string(); + let config = config_dir(world).display().to_string(); + let mut session = TuiSession::spawn_with_env( + world, + args, + &[ + ("HOME", home.as_str()), + ("UV_CACHE_DIR", uv.as_str()), + ("HF_HOME", hf.as_str()), + ("ROCM_CLI_CONFIG_DIR", config.as_str()), + ], + ) + .unwrap_or_else(|e| panic!("failed to run {args:?} at a terminal: {e}")); + if let Some(answer) = answer { + session + .wait_for_screen("[y/N]:", default_timeout()) + .await + .unwrap_or_else(|e| panic!("the confirmation prompt never appeared: {e}")); + session + .send(&format!("{answer}\r")) + .unwrap_or_else(|e| panic!("failed to type the answer: {e}")); + } + let rc = session + .wait_for_exit_code(default_timeout()) + .await + .unwrap_or_else(|e| panic!("the uninstall never exited: {e}")); + world.cli_output = Some(session.screen_text()); + world.cli_stderr = None; + world.cli_rc = Some(rc); + world.tui = Some(session); +} + +#[when("the user dry-runs an uninstall of only the config")] +async fn dry_run_config_only(world: &mut E2eWorld) { + let config = config_dir(world).display().to_string(); + run_uninstall_args( + world, + &config_only_args(true), + &[("ROCM_CLI_CONFIG_DIR", &config)], + ); +} + +#[when("the user dry-runs an uninstall of only the config at a terminal")] +async fn dry_run_config_only_at_terminal(world: &mut E2eWorld) { + run_uninstall_at_terminal(world, &config_only_args(true), None).await; +} + +#[when("the user runs it again without --dry-run at a terminal and answers yes")] +async fn run_it_again_and_confirm(world: &mut E2eWorld) { + // The same arguments as the dry run, without `--dry-run`: the step the + // dry run's last line names. + run_uninstall_at_terminal(world, &config_only_args(false), Some("y")).await; +} + +#[when("the user dry-runs an uninstall that keeps everything at a terminal")] +async fn dry_run_keeping_everything_at_terminal(world: &mut E2eWorld) { + run_uninstall_at_terminal( + world, + &[ + "uninstall", + "--keep-binaries", + "--keep-config", + "--keep-data", + "--keep-cache", + "--dry-run", + ], + None, + ) + .await; +} + +#[then("the dry run says to run it again without --dry-run")] +async fn dry_run_says_to_run_it_again(world: &mut E2eWorld) { + let stdout = output(world); + assert!( + stdout.contains( + "Nothing was removed. To remove these items, run it again without --dry-run." + ), + "the dry run must say how to remove what it listed:\n{stdout}" + ); + assert!( + !stdout.contains("Review uninstall"), + "the command line has no such control:\n{stdout}" + ); +} + +#[then("the dry run lists the config folder")] +async fn dry_run_lists_the_config(world: &mut E2eWorld) { + let stdout = output(world); + let config = config_dir(world).display().to_string(); + assert!( + stdout.contains(&format!("config: {config}")), + "the plan must list the config folder ({config}):\n{stdout}" + ); +} + +#[then("the config folder is gone")] +async fn config_gone(world: &mut E2eWorld) { + let config = config_dir(world); + assert!( + std::fs::symlink_metadata(&config).is_err(), + "running it again as the dry run said did not remove {}:\n{}", + config.display(), + output(world) + ); +} + +#[then("the dry run offers no step to remove anything")] +async fn dry_run_offers_no_step(world: &mut E2eWorld) { + let stdout = output(world); + for phrase in [ + "Nothing was removed.", + "run it again", + "Review uninstall", + "approval", + ] { + assert!( + !stdout.contains(phrase), + "this dry run must not offer {phrase:?}:\n{stdout}" + ); + } +} + +/// The assistant's uninstall preview, called without a model through the hidden +/// `rocm mcp-call` entry point the dashboard and chat use for their tools. +/// +/// The one run here that cannot pass `--keep-binaries`: the preview takes no +/// flags, plans every folder and the binary under test, and renders the plan. It +/// removes nothing, and it is made without `--allow-mutation`, so the tool +/// refuses a call that would remove something instead of running it. +#[when("the assistant previews an uninstall")] +async fn assistant_previews_uninstall(world: &mut E2eWorld) { + let config = config_dir(world).display().to_string(); + let isolated = isolated_env(world); + let mut envs = isolated + .iter() + .map(|(name, value)| (*name, value.as_str())) + .collect::>(); + envs.push(("ROCM_CLI_CONFIG_DIR", &config)); + let arguments = serde_json::json!({ "args": ["uninstall", "--dry-run"] }).to_string(); + let (stdout, stderr, rc) = crate::run_rocm_with_env( + world, + &["mcp-call", "rocm_command", "--arguments-json", &arguments], + &envs, + ); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +/// What the preview printed: the command's own text inside the tool result. +fn preview_text(world: &E2eWorld) -> String { + let result: serde_json::Value = serde_json::from_str(output(world)).unwrap_or_else(|error| { + panic!("the tool result is not JSON ({error}):\n{}", output(world)) + }); + result["structuredContent"]["stdout"] + .as_str() + .unwrap_or_else(|| panic!("the tool result has no command output:\n{}", output(world))) + .to_owned() +} + +#[then("the preview succeeds")] +async fn preview_succeeds(world: &mut E2eWorld) { + let rc = world.cli_rc.expect("no exit code captured"); + assert!( + rc == 0, + "the preview must exit 0, got {rc}\nstdout:\n{}\nstderr:\n{}", + output(world), + world.cli_stderr.as_deref().unwrap_or("") + ); + let result: serde_json::Value = serde_json::from_str(output(world)).expect("tool result JSON"); + assert_eq!( + result["isError"], + serde_json::Value::Bool(false), + "the tool reported an error:\n{}", + output(world) + ); +} + +#[then("the preview lists the config folder")] +async fn preview_lists_the_config(world: &mut E2eWorld) { + let text = preview_text(world); + let config = config_dir(world).display().to_string(); + assert!( + text.contains(&format!("config: {config}")), + "the preview must list the config folder ({config}):\n{text}" + ); +} + +#[then("the preview says removal needs approval")] +async fn preview_says_removal_needs_approval(world: &mut E2eWorld) { + let text = preview_text(world); + assert!( + text.contains("Nothing was removed. Removal needs your approval."), + "the preview must say removal needs approval:\n{text}" + ); + // The preview has no flag to offer, so it must not point at one. + for phrase in ["run it again", "--dry-run.", "Review uninstall"] { + assert!( + !text.contains(phrase), + "the assistant's preview must not offer {phrase:?}:\n{text}" + ); + } +} + +#[when("the user uninstalls the config with --force")] +async fn uninstall_with_force(world: &mut E2eWorld) { + let config = config_dir(world).display().to_string(); + let mut args = config_only_args(false); + args.push("--force"); + run_uninstall_args(world, &args, &[("ROCM_CLI_CONFIG_DIR", &config)]); +} + +#[when("the user uninstalls the config with a near miss of --force")] +async fn uninstall_with_near_miss_of_force(world: &mut E2eWorld) { + let config = config_dir(world).display().to_string(); + let mut args = config_only_args(false); + args.push("--forc"); + run_uninstall_args(world, &args, &[("ROCM_CLI_CONFIG_DIR", &config)]); +} + +#[then("the uninstall is refused and told to use --yes instead")] +async fn refused_and_told_to_use_yes(world: &mut E2eWorld) { + let stderr = world.cli_stderr.as_deref().unwrap_or(""); + assert_eq!( + world.cli_rc, + Some(1), + "a refused --force exits 1\nstderr:\n{stderr}" + ); + assert!( + stderr.contains("`--force` is not an option of `rocm uninstall`") + && stderr.contains("use `--yes` to skip the confirmation"), + "expected the refusal to name --yes:\n{stderr}" + ); + assert!( + !stderr.contains("force-dev-binaries"), + "the refusal must not name the destructive flag:\n{stderr}" + ); +} + +#[then("the error does not point at the flag that removes development binaries")] +async fn error_does_not_suggest_dev_binaries(world: &mut E2eWorld) { + let stderr = world.cli_stderr.as_deref().unwrap_or(""); + assert_eq!( + world.cli_rc, + Some(2), + "an unknown flag exits 2\nstderr:\n{stderr}" + ); + assert!( + stderr.contains("unexpected argument"), + "expected the unknown-argument error on stderr:\n{stderr}" + ); + assert!( + !stderr.contains("force-dev-binaries"), + "the error must not name the destructive flag:\n{stderr}" + ); +} + +#[when("the user asks for the uninstall help")] +async fn ask_for_uninstall_help(world: &mut E2eWorld) { + let (stdout, stderr, rc) = crate::run_rocm_with_env(world, &["uninstall", "--help"], &[]); + world.cli_output = Some(stdout); + world.cli_stderr = Some(stderr); + world.cli_rc = Some(rc); +} + +#[then("the help shows a dry run before the real run")] +async fn help_shows_dry_run_first(world: &mut E2eWorld) { + let stdout = output(world); + let examples = stdout.split_once("EXAMPLES:").map_or_else( + || panic!("the help has no examples:\n{stdout}"), + |(_, rest)| rest, + ); + let rows: Vec<&str> = examples + .lines() + .map(str::trim) + .filter(|row| !row.is_empty()) + .collect(); + assert_eq!( + rows.first().copied(), + Some("rocm uninstall --dry-run"), + "the first example must be the dry run:\n{examples}" + ); + assert!( + rows.contains(&"rocm uninstall"), + "the examples must also show the real run:\n{examples}" + ); +} From 71cc5483db93def59db92d441cbb717792cb5981 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 16:18:30 +0000 Subject: [PATCH 024/102] ROCMAI-724: retire rocm-core's write_atomic onto the shared atomic_write module (#811) * refactor(core): share one atomic file writer between rocm and rocmd `apps/rocm/src/therock.rs` and `apps/rocmd/src/sandbox.rs` each carried their own copy of `write_file_atomically` and its staging/publish helpers. Move the `therock` copy, the more complete of the two, into `rocm_core::atomic_write` and call it from both binaries, so the next caller (the active-runtime marker) gets the same implementation rather than a third copy. The move is mechanical apart from the error text. rocmd gets two extra pieces of context it was missing: the full-disk mapping through `disk_space::map_write_error`, and "failed to publish " when the rename fails. The `failed to create ` context that only rocmd had is kept. The helper tests move with the code. The five rocmd tests that duplicated therock tests are dropped. The one rocmd test that drove the temp-name check through a real write is kept. Count: 15 tests before (two of them ignored), 10 now (one ignored), with no assertion lost. Signed-off-by: Roman Inflianskas * test(runtimes): pin the active-runtime marker's atomic replacement `write_active_runtime_marker` used to stage `runtimes/active.json` at `active.json.tmp-` with a truncating write, remove the marker, then rename the temp file into place (#605). Two activations against one data dir in the same millisecond therefore wrote through one shared temp file. One of them failed with "failed to move active runtime marker". The other could publish its payload followed by the tail of a longer one, which later readers such as `rocm runtimes uninstall` could not parse. Between the remove and the rename, the marker did not exist at all. write itself does not change here beyond going through `rocm_core::atomic_write::write_file_atomically` (the one writer the previous commit moved into rocm-core) for both the forward write and the failed-activation restore. Each writer stages into its own `create_new` temp file named with pid, millis and attempt, and the marker is replaced in one step rather than removed first. The doc comment now says what that does and does not guarantee for two concurrent activations. Two regression tests pin it: - a deterministic one pre-creates a file at every name the old scheme could pick during the test and asserts none is written through or moved; - a Unix-only stress test runs two writers in lock-step plus a reader, and asserts every write succeeds and the reader always finds a complete marker. Against the pre-#424 body, 190 of 400 writes failed. Re-adding only the remove before the publish made the reader miss the marker thousands of times. Signed-off-by: Roman Inflianskas * docs(architecture): point at rocm-core's shared atomic writer The rocmd sandbox module no longer carries its own atomic-write helpers; record where the shared one lives so the next caller reuses it. Signed-off-by: Roman Inflianskas * style(core): give the atomic writer's test seams short doc summaries clippy's too_long_first_doc_paragraph rejects the two seam doc comments carried over from therock, where the private functions were not linted for it. Signed-off-by: Roman Inflianskas * fix(rocmd): gate std::fs import to the targets that still use it With the atomic writer moved to rocm-core, sandbox.rs uses std::fs only in its Linux bubblewrap path and its tests, so the Windows build warned about an unused import. Signed-off-by: Roman Inflianskas * ci(coverage): lower rocmd's floor for the tests that moved to rocm-core Moving the atomic writer out of rocmd's sandbox.rs took its helper code and six tests with it. Those lines were close to fully covered, and the line count includes test modules, so rocmd's percentage drops although nothing it still owns lost coverage. The code and its tests are now measured under rocm-core. Only rocm's tests reach rocmd, so `cargo llvm-cov --no-cfg-coverage -p rocm -p rocmd` measures rocmd exactly as the workspace run does. Measured locally that way, rocmd went from 75.30% (7776 lines) on the base to 75.02% (7549 lines) here. That is a 0.28pp drop, past the 0.2pp tolerance. The floor is set to the branch measurement. Only this one line is changed. A full `--bless` from a local run would also rewrite every other crate's floor from a different host's numbers. rocm moved by -0.04pp, inside tolerance, and is left as is. Signed-off-by: Roman Inflianskas * fix(rocmd): drop the windows-sys dependency the move left unused `ReplaceFileW` was rocmd's only use of `windows-sys`; it moved into `rocm_core::atomic_write` with the rest of the atomic writer, and rocm-core declares its own dependency. Also narrow the marker regression test's doc comment: it guards against a return to the millis-only temp names, while collision safety under the current names is covered by the concurrent tests. Signed-off-by: Roman Inflianskas * build(rocm): drop the windows-sys dependency the move left unused ReplaceFileW was apps/rocm's only use of windows-sys, and it moved into rocm_core::atomic_write with the rest of the writer; rocm-core declares its own dependency. Same leftover as the rocmd one removed earlier. Signed-off-by: Roman Inflianskas * fix(core): state what the atomic writer guarantees, and test the full disk The module doc now gives the real reason for ReplaceFileW (a sharing violation while another process holds the destination open; std rename does replace), says Windows concurrent publishes are untested, and says atomic but not durable, since nothing is fsynced. The test seams no external production code calls are narrowed: write_file_atomically_with is private, and the two that apps/rocm's cross-crate tests drive stay pub but #[doc(hidden)] and say they are seams. The staging doc no longer implies an outside caller can publish and clean up on its own. The full-disk test called disk_space::map_write_error directly, so it stayed green if staging stopped mapping errors. It now drives the real staging path with the byte write swapped for one that fails like a full disk, and checks the message and that no temp file is left. Signed-off-by: Roman Inflianskas * test(runtimes): describe the marker tests as what they pin after #424 #424 already fixed the temp-name collision and the missing-marker window on main. The two tests now say they pin that the marker keeps those properties through the shared writer, not that they are the regression for #605. The collision test no longer fails when the write lands after its pre-created names: it retries with a window sized from how long the pre-creation and write took, and where the filesystem is too slow for any window to cover the write it reports itself inconclusive and passes. The marker writer's doc says the Windows replace is untested under concurrency. Signed-off-by: Roman Inflianskas * docs(architecture): call atomic_write the shared writer and name exceptions It is not the only temp-then-rename writer: remote/session.rs keeps its own, and RocmCliConfig::save keeps its own until #541 moves it, tracked in #618. Also say it is atomic, not durable. Signed-off-by: Roman Inflianskas * ROCMAI-724: retire rocm-core's write_atomic onto the shared atomic_write module The remaining callers of write_atomic (the automation event and proposal store rewrites, the service-record manifests, the config save, and the Lemonade pins restore and put-back) now publish through rocm_core::atomic_write::write_file_atomically. write_atomic and its two unit tests are deleted: the atomic_write module's own tests and the config save tests already cover what they asserted. Temp files now get a unique, exclusively created name instead of a deterministic clock-and-pid one, so two writers can no longer share a temp file and publish each other's bytes. The shared writer also creates a missing parent directory, so the Lemonade failed-put-back test now forces the failure with a non-empty directory at the target instead of a missing directory. Six per-caller tests pin that a writer never reuses another writer's temp file. Each was shown to fail with the old behaviour restored. docs/architecture.md no longer lists write_atomic as a remaining hand-rolled writer. Signed-off-by: Jussi Elo * ROCMAI-724: address review feedback on the write_atomic retirement The foreign-temp-file probe that backs the six per-caller tests could pass without proving anything: on a host that creates files at about a millisecond each, pre-creating the names took as long as the window they covered, and the probe then printed a notice and returned Ok. On the machine this was measured on, that was the normal case, not a corner. The window now starts a lead after the names are created and the write only runs once it has begun, so slow file creation no longer decides the outcome; a retry lengthens the lead or the window, and a host too slow for any of them fails the test off Windows instead of passing. On Windows the probe still tolerates an inconclusive run, since the cost of creating files there has not been measured; that is stated in its doc comment. The probe now has its own tests: it fails a writer that reuses a name fixed by the clock, spares one that picks its own, reports an inconclusive run, and turns that into a failure off Windows. The probe was duplicated in rocm-core and in the Lemonade engine, and the duplicate would have needed the same tests. It now lives once in rocm_core::foreign_temp_probe, compiled for rocm-core's own tests and under a new test-support feature that the Lemonade engine enables only from its dev-dependencies, so no shipped build contains it. Only retired_pin_temp_path stays in the engine. The atomic_write module header said the writer is shared by rocm and rocmd; it now names the rocm-core and Lemonade callers too. The Windows publish step is untouched: ReplaceFileW can leave the destination name briefly absent, which the lock-free automation readers read as an empty store. That property belongs to the shared writer, not to this change, and is left for a separate decision. Signed-off-by: Jussi Elo * ROCMAI-724: make the temp-file probe self-tests host-robust and fix two stale doc claims The foreign-temp-file probe's inconclusive handling now takes the Windows tolerance as a parameter, so both the tolerated and the failing branch are tested on every platform instead of only the failing one off Windows. The two self-tests that drive the probe against a writer that reuses a name and one that picks its own now rerun when a loaded host leaves the probe inconclusive, while still failing if the probe reports a clobbered name as spared or blames a clean writer. architecture.md no longer states an exact count of hand-rolled temp-then-rename writers outside atomic_write (the shell-profile rewrite in uninstall.rs is one more), and lists the probe module with the other rocm-core modules. The doc on the config save cleanup test now names the module the cleanup lives in. Signed-off-by: Jussi Elo * ROCMAI-724: publish with rename first on Windows so readers never see the destination absent The Windows publish step called ReplaceFileW whenever the destination existed. ReplaceFileW can leave the destination briefly absent, which the lock-free readers of the config and the automation stores read as an empty store. Now the publish tries rename first, which replaces in one step, and falls back to ReplaceFileW only when that rename fails and the destination exists (another process holding it open without delete sharing). The order lives in a helper that takes both steps as arguments, so its tests run on every host. A reader-polling test asserts the destination is never absent while it is replaced and runs on Unix and on the native Windows job. The stale comments that said ReplaceFileW is used whenever the destination exists are updated. Signed-off-by: Jussi Elo * fix(core): pass the borrowed-path rename wrapper to the Windows publish helper `publish_temp_file` on Windows passed `fs::rename` straight to `publish_rename_first`, whose parameters are `FnOnce(&Path, &Path)` for any two lifetimes. `fs::rename` is generic over `AsRef`, so the fn item is pinned to one lifetime per argument and rustc rejects it with "implementation of `FnOnce` is not general enough". The path is `cfg(windows)`, so only the Windows jobs compiled it and the Linux jobs stayed green. Pass the existing `rename_file` wrapper instead, the same non-generic fn `recover_partial_replace` already takes, and say in its doc why the bare function does not work. The real-publish tests (for example the reader-gap test) run on the Windows job and exercise this path. Signed-off-by: Jussi Elo * test(core): start the reader-gap test's writes only once the reader is polling `a_reader_never_finds_the_destination_absent_while_it_is_replaced` spawned the reader and immediately began its 300 replacements. If the reader thread was not scheduled until after the writes finished and `done` was set, it made zero reads and the final `reads > 0` assertion failed although publication was correct. The reader now signals its first successful read over a channel and the test waits for that signal (30s timeout) before writing. A reader that exits early because the destination was absent still reports that from its own outcome. Checked by giving the reader a 1.5s start delay: it fails with zero reads without the wait and passes with it. Signed-off-by: Jussi Elo * test(atomic-write): say what the reader-gap test covers and stop its reader on a panic The reader-gap test only discriminates the Windows publish order on Windows, and leaves the ReplaceFileW fallback untested; its doc comment now says so. A writer that panicked left the reader thread spinning, so a drop guard sets the stop flag on every path. The module doc also records that a rename publish does not carry over the replaced file's attributes and ACL, which no caller of the writer relies on. Signed-off-by: Jussi Elo * docs(jsonl-store): rewrap the module doc and state the error context exactly One reflowed line of the module doc ran past the width of its block. The test comment about layered errors now names the contexts the real steps add, instead of saying both wrap the OS error, which a full-disk staging failure does not. Signed-off-by: Jussi Elo * fix(lemonade): keep a removed runtime tree from being rebuilt around a pin file The shared atomic writer creates a missing parent directory, which the retired writer did not. The pin restore and put-back run after a failed install, so a runtime tree that is already gone would have been recreated as a bare resources directory holding one pin file, where both used to fail. They go through a helper that fails when the directory is missing, and the put-back failure test uses its original missing-directory trigger again. Signed-off-by: Jussi Elo --------- Signed-off-by: Roman Inflianskas Signed-off-by: Jussi Elo Co-authored-by: Roman Inflianskas (cherry picked from commit 48056708692f4cccb0cf881088f42cf547e8ce79) Signed-off-by: juhovainio --- apps/rocm/src/runtimes_cmd.rs | 23 +- apps/rocmd/src/supervise.rs | 4 +- crates/rocm-core/Cargo.toml | 4 + crates/rocm-core/src/atomic_write.rs | 216 +++++++++++++-- crates/rocm-core/src/automation.rs | 62 ++++- crates/rocm-core/src/config.rs | 47 +++- crates/rocm-core/src/foreign_temp_probe.rs | 290 +++++++++++++++++++++ crates/rocm-core/src/jsonl_store.rs | 53 ++-- crates/rocm-core/src/lib.rs | 126 +-------- crates/rocm-core/src/service_record.rs | 47 +++- crates/rocm-core/src/test_support.rs | 13 + docs/architecture.md | 6 +- engines/lemonade/Cargo.toml | 1 + engines/lemonade/src/install.rs | 229 +++++++++++++--- engines/lemonade/src/test_support.rs | 9 + 15 files changed, 902 insertions(+), 228 deletions(-) create mode 100644 crates/rocm-core/src/foreign_temp_probe.rs diff --git a/apps/rocm/src/runtimes_cmd.rs b/apps/rocm/src/runtimes_cmd.rs index 0a43821f68..950960a6b2 100644 --- a/apps/rocm/src/runtimes_cmd.rs +++ b/apps/rocm/src/runtimes_cmd.rs @@ -880,20 +880,22 @@ pub(crate) const fn toolchain_state_text(includes_devel: bool) -> &'static str { /// Two `rocm` processes can activate a runtime against the same data /// directory at once (two terminals, a script), so the write goes through the /// shared atomic writer: each writer stages into its own exclusively created -/// temp file, and the marker is replaced rather than removed first. On Unix, -/// and on Windows whenever `ReplaceFileW` succeeds, a reader therefore always -/// finds a complete marker. A `ReplaceFileW` that fails part-way is the -/// exception: it can leave the marker missing, which the writer repairs with a -/// plain rename when that rename succeeds (see `rocm_core::atomic_write`); two -/// replacements in flight at once on Windows are not tested. It does not order -/// the two activations: the last one to publish wins, as before. +/// temp file, and the marker is replaced rather than removed first, so a +/// reader always finds a complete marker: on Windows the publish renames first, +/// which replaces in one step. Only when that rename fails, because another +/// process holds the marker open without delete sharing, does it fall back to +/// `ReplaceFileW`, and a `ReplaceFileW` that fails part-way can leave the marker +/// missing, which the writer repairs with a plain rename when that succeeds +/// (see `rocm_core::atomic_write`); two replacements in flight at once through +/// the fallback are not tested. It does not order the two activations: the last +/// one to publish wins, as before. fn write_active_runtime_marker(paths: &AppPaths, marker: ActiveRuntimeMarker) -> Result<()> { let path = active_runtime_marker_path(paths); // The shared helper rather than a local temp-file-and-rename: it is the same // write the failed-activation restore uses, and it is strictly stronger than // a bare `fs::rename`, which on Windows is `MoveFileExW` and fails with a // sharing violation when another process holds the marker open. The helper - // uses `ReplaceFileW` there whenever the destination exists. The + // renames first there and falls back to `ReplaceFileW` when that fails. The // destination is never removed first, on any platform — that would leave a // window with no marker at all, which runtime resolution and the storage // retention holds both read as "nothing is active" — and a failed publish @@ -1280,9 +1282,8 @@ pub(crate) mod tests { /// probability per round rather than deterministically; against the /// current writer nothing in it is timing-dependent. Unix only: the /// property under test is that `rename` replaces in one step. The Windows - /// publish path goes through `ReplaceFileW` instead, whose behaviour with - /// two replacements of one file in flight this test has not been run - /// against. + /// publish tries `rename` first too, but this test has not been run + /// against it there, nor against the `ReplaceFileW` fallback. #[cfg(unix)] #[test] fn concurrent_active_runtime_marker_writes_keep_a_complete_marker_published() -> Result<()> { diff --git a/apps/rocmd/src/supervise.rs b/apps/rocmd/src/supervise.rs index 409895a844..023a00d313 100644 --- a/apps/rocmd/src/supervise.rs +++ b/apps/rocmd/src/supervise.rs @@ -2453,12 +2453,12 @@ mod tests { /// spawned yet: the rollback at that first write must undo the in-memory /// transition and leave the on-disk record exactly as it was, not a /// half-applied `recovering` row. The manifest write goes through - /// [`crate::write_atomic`][rocm_core_write_atomic]-style write-then-rename, + /// [`write_file_atomically`][rocm_core_atomic_write]-style write-then-rename, /// so a read-only *file* no longer blocks it (rename ignores the /// destination's own permission bits) — a read-only *directory* still does, /// since it can't hold the new temp file the rename needs. /// - /// [rocm_core_write_atomic]: rocm_core + /// [rocm_core_atomic_write]: rocm_core::atomic_write::write_file_atomically /// /// Unix only and skipped as root, where a read-only mode is not enforced /// and the write would simply succeed. diff --git a/crates/rocm-core/Cargo.toml b/crates/rocm-core/Cargo.toml index 48afe6fee4..e0cd9036fa 100644 --- a/crates/rocm-core/Cargo.toml +++ b/crates/rocm-core/Cargo.toml @@ -14,6 +14,10 @@ workspace = true # Lets the E2E suite re-root the hardware probes under a simulated host; see # `src/hardware_root.rs`. Never enabled in a release build. e2e-test-hooks = [] +# Compiles the `foreign_temp_probe` test probe so a dependent crate's tests can +# use it. Enabled only from a `[dev-dependencies]` entry; never in a shipped +# build. +test-support = [] [build-dependencies] cc = "1" diff --git a/crates/rocm-core/src/atomic_write.rs b/crates/rocm-core/src/atomic_write.rs index a3899fde64..0ba0d41032 100644 --- a/crates/rocm-core/src/atomic_write.rs +++ b/crates/rocm-core/src/atomic_write.rs @@ -2,7 +2,8 @@ // // SPDX-License-Identifier: MIT -//! Atomic file replacement, shared by `rocm` and `rocmd`. +//! Atomic file replacement, shared by the `rocm` and `rocmd` apps, `rocm-core`'s +//! own config, automation and service-record writers, and the Lemonade engine. //! //! [`write_file_atomically`] stages the bytes in a fresh sibling temp file and //! then publishes it over the destination in one step, so a reader sees either @@ -16,18 +17,25 @@ //! attempt. A failed write or publish removes its temp file, with the one //! exception described below. //! -//! On Windows, when the destination already exists, the publish uses -//! `ReplaceFileW` rather than `rename`. `std::fs::rename` does replace an -//! existing file there, but as `MoveFileExW` it fails with a sharing violation -//! while another process holds the destination open, which is the case -//! `ReplaceFileW` is used for. The concurrency tests below run on every -//! platform, but each starts with no destination, so on Windows at most one of -//! their publishes goes through `ReplaceFileW`: two `ReplaceFileW` publishes of -//! one file in flight at once are not exercised by any test. +//! On Windows the publish tries `rename` first. `std::fs::rename` replaces an +//! existing file in one step, so a lock-free reader (`RocmCliConfig::load`, the +//! automation stores) finds the old or the new file and never an absent one. +//! Only when that rename fails and the destination exists, which is the case of +//! another process holding it open without delete sharing, does the publish +//! fall back to `ReplaceFileW`. The concurrency tests below run on every +//! platform and publish by `rename`; two `ReplaceFileW` publishes of one file in +//! flight at once are not exercised by any test. //! -//! One failure leaves the destination absent: Microsoft documents that -//! `ReplaceFileW`, called without a backup name, can fail after it has already -//! moved the old file off the destination name +//! A publish by `rename` leaves the destination with the staged file's +//! attributes and the ACL its directory hands down, where `ReplaceFileW` carries +//! over those of the file it replaces. No caller of this module sets a hidden or +//! read-only flag or a custom ACL on a file it writes, so nothing is lost today. +//! A file that must keep such settings across a rewrite should not rely on this +//! writer to carry them over. +//! +//! The fallback is the one path that can leave the destination absent: +//! Microsoft documents that `ReplaceFileW`, called without a backup name, can +//! fail after it has already moved the old file off the destination name //! (`ERROR_UNABLE_TO_MOVE_REPLACEMENT` and `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2`). //! The publish then retries once as a plain rename. If that fails too, the //! temp file is kept rather than removed, since it holds the only copy of the @@ -396,15 +404,25 @@ pub fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { #[cfg(windows)] #[doc(hidden)] pub fn publish_temp_file(tmp: &Path, path: &Path) -> io::Result<()> { - if path.try_exists()? { - return replace_file_windows(path, tmp); - } + publish_rename_first(tmp, path, rename_file, replace_file_windows) +} - match fs::rename(tmp, path) { +/// The Windows publish order, with both steps passed in so a test can check it +/// on any host: `rename` first, because it replaces in one step and never +/// leaves the destination absent; `replace` only when the rename failed and the +/// destination exists. A failed rename onto a destination that does not exist is +/// returned as it is, since `replace` has nothing to replace. +#[cfg(any(windows, test))] +fn publish_rename_first(tmp: &Path, path: &Path, rename: R, replace: P) -> io::Result<()> +where + R: FnOnce(&Path, &Path) -> io::Result<()>, + P: FnOnce(&Path, &Path) -> io::Result<()>, +{ + match rename(tmp, path) { Ok(()) => Ok(()), Err(rename_error) => { if path.try_exists()? { - replace_file_windows(path, tmp) + replace(path, tmp) } else { Err(rename_error) } @@ -453,7 +471,9 @@ fn replace_file_windows(path: &Path, replacement: &Path) -> io::Result<()> { } /// `fs::rename` with both paths borrowed, the shape `recover_partial_replace` -/// takes. +/// and `publish_rename_first` take. `fs::rename` itself is generic over +/// `AsRef`, so passed directly it is pinned to one lifetime per argument +/// and does not satisfy their `FnOnce(&Path, &Path)` bounds. #[cfg(windows)] fn rename_file(from: &Path, to: &Path) -> io::Result<()> { fs::rename(from, to) @@ -591,6 +611,166 @@ mod tests { assert_eq!(names, vec!["artifact.tar.gz".to_owned()]); } + /// The Windows publish order: `rename` is tried first so the destination is + /// never absent, and `ReplaceFileW` is reached only after a failed rename + /// onto a destination that exists. Runs on every host because the order is + /// a plain function of the two injected steps. + #[test] + fn windows_publish_renames_first_and_skips_the_replace_when_it_succeeds() { + let root = scratch_dir("publish-rename-first"); + fs::create_dir_all(&root).unwrap(); + // The destination must exist: with it absent, no ordering of the two + // steps would reach the replace, so the test could not tell them apart. + let existing = root.join("existing"); + fs::write(&existing, b"old").unwrap(); + let replaced = std::cell::Cell::new(false); + let renamed = std::cell::Cell::new(false); + publish_rename_first( + Path::new("tmp"), + &existing, + |_, _| { + renamed.set(true); + Ok(()) + }, + |_, _| { + replaced.set(true); + Ok(()) + }, + ) + .expect("a successful rename is the whole publish"); + let _ = fs::remove_dir_all(&root); + assert!(renamed.get(), "the rename must be tried first"); + assert!( + !replaced.get(), + "ReplaceFileW must not run after a rename that worked" + ); + } + + #[test] + fn windows_publish_falls_back_to_the_replace_only_for_an_existing_destination() { + let root = scratch_dir("publish-fallback"); + fs::create_dir_all(&root).unwrap(); + let existing = root.join("existing"); + fs::write(&existing, b"old").unwrap(); + let sharing = || Err(io::Error::from(io::ErrorKind::PermissionDenied)); + + let replaced = std::cell::Cell::new(None); + publish_rename_first( + Path::new("tmp"), + &existing, + |_, _| sharing(), + |path, tmp| { + replaced.set(Some((path.to_path_buf(), tmp.to_path_buf()))); + Ok(()) + }, + ) + .expect("the fallback result is the publish result"); + assert_eq!( + replaced.take(), + Some((existing, PathBuf::from("tmp"))), + "the fallback replaces the destination with the staged file" + ); + + let missing = root.join("missing"); + let called = std::cell::Cell::new(false); + let error = publish_rename_first( + Path::new("tmp"), + &missing, + |_, _| sharing(), + |_, _| { + called.set(true); + Ok(()) + }, + ) + .expect_err("a failed rename onto a missing destination is an error"); + let _ = fs::remove_dir_all(&root); + assert_eq!(error.kind(), io::ErrorKind::PermissionDenied); + assert!( + !called.get(), + "there is nothing for ReplaceFileW to replace" + ); + } + + /// A reader that polls the destination while a writer replaces it over and + /// over must never find it missing: that is what a lock-free reader of the + /// config or an automation store would read as an empty store. A transient + /// permission error while a replace is in flight is tolerated; an absent + /// destination is not. + /// + /// This is a Windows-lane check. On Linux `rename` replaces atomically, so + /// the test passes with either publish order and would not notice the order + /// being reversed; it only discriminates on Windows, where a publish that + /// removes the destination before putting the new file in place leaves a + /// window the reader can fall into. What it still does not cover is the + /// `ReplaceFileW` fallback, which runs only when the rename fails and is the + /// one path that can leave the destination absent: it is reached by no test + /// that races a reader, and its order is pinned only by the injected-step + /// tests above. + #[test] + fn a_reader_never_finds_the_destination_absent_while_it_is_replaced() { + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; + + /// Tells the reader to stop when dropped, so a writer that panics + /// through `unwrap` still ends the polling thread instead of leaving it + /// to spin until the process exits. + struct StopReader(Arc); + impl Drop for StopReader { + fn drop(&mut self) { + self.0.store(true, Ordering::Relaxed); + } + } + + let root = scratch_dir("atomic-reader-gap"); + fs::create_dir_all(&root).unwrap(); + let destination = root.join("store.json"); + write_file_atomically(&destination, b"0").unwrap(); + + let done = Arc::new(AtomicBool::new(false)); + let (first_read, reader_started) = std::sync::mpsc::channel::<()>(); + let reader = { + let destination = destination.clone(); + let done = Arc::clone(&done); + std::thread::spawn(move || { + let mut reads = 0u32; + while !done.load(Ordering::Relaxed) { + match fs::read(&destination) { + Ok(_) => { + reads += 1; + if reads == 1 { + let _ = first_read.send(()); + } + } + Err(error) if error.kind() == io::ErrorKind::NotFound => { + return Err(error); + } + Err(_) => {} + } + } + Ok(reads) + }) + }; + // Start writing only once the reader is polling, so a late-scheduled + // reader cannot miss every replacement and fail on timing alone. A + // reader that exits before its first read (the destination was absent) + // disconnects the channel; that is reported from its outcome below. + let stop_reader = StopReader(Arc::clone(&done)); + let started = reader_started.recv_timeout(std::time::Duration::from_secs(30)); + if started.is_ok() { + for round in 1..=300u32 { + write_file_atomically(&destination, round.to_string().as_bytes()).unwrap(); + } + } + drop(stop_reader); + let outcome = reader.join().unwrap(); + let _ = fs::remove_dir_all(&root); + let reads = outcome.expect("the destination was absent while it was being replaced"); + assert!( + started.is_ok() && reads > 0, + "the reader never read the destination at all: {started:?}" + ); + } + #[test] fn concurrent_atomic_writes_do_not_remove_a_published_destination() { let root = scratch_dir("atomic-collision"); diff --git a/crates/rocm-core/src/automation.rs b/crates/rocm-core/src/automation.rs index 6cedd857a6..9bcf417207 100644 --- a/crates/rocm-core/src/automation.rs +++ b/crates/rocm-core/src/automation.rs @@ -402,7 +402,7 @@ fn write_automation_proposals( ); text.push('\n'); } - crate::write_atomic(&path, text.as_bytes(), "jsonl") + crate::atomic_write::write_file_atomically(&path, text.as_bytes()) } fn normalize_proposal_identity(proposal: &mut AutomationProposalRecord, index: usize) { @@ -451,9 +451,11 @@ pub fn load_recent_automation_events( #[cfg(test)] mod tests { use super::*; + use crate::foreign_temp_probe::assert_write_spares_foreign_temp_files; use crate::jsonl_store::jsonl_generation_path; use crate::test_support::{ - audit_line, audit_messages, audit_record, seed_audit_store, temp_app_paths, + audit_line, audit_messages, audit_record, retired_temp_path, seed_audit_store, + temp_app_paths, }; use std::path::Path; @@ -559,6 +561,62 @@ mod tests { Ok(()) } + /// A status update never writes through a file another writer already + /// staged next to the proposals store. The occupied names are the ones the + /// retired clock-and-pid temp naming would have picked, so a rewrite that + /// still used one would truncate a foreign file and rename it into place. + /// `proposal_status_update_rewrites_record` would pass for either writer; + /// the foreign files are what tell them apart. + #[test] + fn proposal_status_update_never_reuses_another_writers_temp_file() -> Result<()> { + let (root, paths) = temp_app_paths("proposal-status-foreign-temp"); + append_automation_proposal(&paths, &proposal_record("proposal-1"))?; + let store = paths.automation_proposals_path(); + + assert_write_spares_foreign_temp_files( + |millis| retired_temp_path(&store, millis), + || update_automation_proposal_status(&paths, "proposal-1", "rejected").map(|_| ()), + )?; + + let loaded = find_automation_proposal(&paths, "proposal-1")?; + let _ = fs::remove_dir_all(&root); + assert_eq!(loaded.status, "rejected"); + Ok(()) + } + + /// The torn-tail repair, the other place a store is replaced wholesale, + /// spares foreign staged files for the same reason. Each attempt seeds the + /// torn tail again, since a repair only happens once. + #[test] + fn torn_tail_repair_never_reuses_another_writers_temp_file() -> Result<()> { + let (root, paths) = temp_app_paths("torn-tail-foreign-temp"); + // Creates the store directory the foreign files go in. + seed_torn_tail_in_every_store(&paths)?; + let store = paths.automation_events_path(); + + assert_write_spares_foreign_temp_files( + |millis| retired_temp_path(&store, millis), + || { + seed_torn_tail_in_every_store(&paths)?; + append_automation_event(&paths, &event_record("after-repair")) + }, + )?; + + // Only the events store was appended to; the other two still hold the + // torn tails the seeding left in them. + let tags = fs::read_to_string(&store)? + .lines() + .map(|line| serde_json::from_str::(line).map(|r| r.message)) + .collect::, _>>()?; + let _ = fs::remove_dir_all(&root); + assert_eq!( + tags, + vec!["seed".to_owned(), "after-repair".to_owned()], + "the repair must keep the clean prefix, drop the torn tail and add the new record" + ); + Ok(()) + } + #[test] fn load_recent_audit_events_returns_tail() -> Result<()> { let (root, paths) = temp_app_paths("audit-tail"); diff --git a/crates/rocm-core/src/config.rs b/crates/rocm-core/src/config.rs index 62eaf05fad..e577241ff4 100644 --- a/crates/rocm-core/src/config.rs +++ b/crates/rocm-core/src/config.rs @@ -603,9 +603,11 @@ impl RocmCliConfig { .with_context(|| format!("failed to parse {}", path.display())) } - /// Persist the config by replacing it via the shared `write_atomic` - /// helper (write a sibling temp file, then rename it over `config.json`; - /// see that helper's own doc for the write/rename/cleanup guarantees). + /// Persist the config by replacing it via the shared + /// [`write_file_atomically`](crate::atomic_write::write_file_atomically) + /// helper (write a uniquely named sibling temp file, then publish it over + /// `config.json`; see that helper's module doc for the + /// write/publish/cleanup guarantees). /// /// The destination is never removed first, because `load` reads a /// missing `config.json` as "no config" and answers with defaults — a @@ -617,7 +619,7 @@ impl RocmCliConfig { .with_context(|| format!("failed to create {}", paths.config_dir.display()))?; let bytes = serde_json::to_vec_pretty(self).context("failed to serialize rocm-cli config")?; - crate::write_atomic(&path, &bytes, "json") + crate::atomic_write::write_file_atomically(&path, &bytes) .with_context(|| format!("failed to save rocm-cli config to {}", path.display())) } @@ -757,7 +759,8 @@ impl RocmCliConfig { #[cfg(test)] mod tests { use super::*; - use crate::test_support::temp_app_paths; + use crate::foreign_temp_probe::assert_write_spares_foreign_temp_files; + use crate::test_support::{retired_temp_path, temp_app_paths}; #[test] fn builtin_watchers_include_read_only_gpu_metrics() { @@ -1143,10 +1146,11 @@ last_installed_runtime_id = "therock-release" Ok(()) } - /// The `inspect_err` cleanup on the rename, which the happy-path test above + /// The temp-file cleanup when the publish step fails (in + /// `atomic_write::publish_staged_file_with`), which the happy-path test above /// cannot reach: it only ever asserts that a SUCCESSFUL save leaves nothing /// behind, so deleting the cleanup would go unnoticed until a user with a - /// failing disk accumulated one `config.json.tmp--` per attempt. + /// failing disk accumulated one `config.json.tmp---` per attempt. /// /// A directory at the destination is what makes the rename fail on both /// supported platforms without needing permissions a root CI lane ignores — @@ -1208,6 +1212,35 @@ last_installed_runtime_id = "therock-release" Ok(()) } + /// A save never writes through a file another writer already staged next + /// to `config.json`. The names occupied are the ones the retired + /// clock-and-pid temp naming would have picked, so a save that still wrote + /// to one would truncate a foreign file and rename it into place, taking + /// the other writer's bytes with it. The save-and-load assertions below + /// would pass for either writer; the foreign files are what tell them apart. + #[test] + fn saving_the_config_never_reuses_another_writers_temp_file() -> Result<()> { + let (root, paths) = temp_app_paths("config-save-foreign-temp"); + fs::create_dir_all(&paths.config_dir)?; + let config_path = paths.config_path(); + let mut config = RocmCliConfig { + default_engine: Some("vllm".to_owned()), + ..RocmCliConfig::default() + }; + config.save(&paths)?; + config.default_engine = Some("lemonade".to_owned()); + + assert_write_spares_foreign_temp_files( + |millis| retired_temp_path(&config_path, millis), + || config.save(&paths), + )?; + + let loaded = RocmCliConfig::load(&paths)?; + let _ = fs::remove_dir_all(&root); + assert_eq!(loaded.default_engine.as_deref(), Some("lemonade")); + Ok(()) + } + #[cfg(unix)] #[test] fn saving_the_config_replaces_the_file_rather_than_rewriting_it_in_place() -> Result<()> { diff --git a/crates/rocm-core/src/foreign_temp_probe.rs b/crates/rocm-core/src/foreign_temp_probe.rs new file mode 100644 index 0000000000..9d9fd4f520 --- /dev/null +++ b/crates/rocm-core/src/foreign_temp_probe.rs @@ -0,0 +1,290 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! A test probe for the claim "this writer never touches another writer's +//! temp file", shared by this crate's tests and by the tests of the crates +//! that publish through [`crate::atomic_write`]. +//! +//! Compiled only for this crate's own tests and under the `test-support` +//! feature, which a dependent crate enables from its `[dev-dependencies]` +//! entry, so it is absent from every shipped build. + +use crate::unix_time_millis; +use std::path::PathBuf; + +const FOREIGN: &[u8] = b"another writer's staged bytes"; +const ATTEMPTS: usize = 3; +const INITIAL_WINDOW_MS: u128 = 400; +/// Ceiling for both the window and the lead before the probe gives up. +const MAX_MS: u128 = 16_000; + +/// What one run of the probe established. +#[derive(Debug, PartialEq, Eq)] +enum Outcome { + /// The write ran inside the window of occupied names and left every one + /// of them alone. + Spared, + /// The host was too slow for any window to cover the write, so a name may + /// not have been in its way and the run says nothing about the writer. + Inconclusive(String), +} + +/// Runs `write` while a foreign file occupies every name `foreign_name` +/// returns for each millisecond of a window, and asserts that none of them was +/// written through, truncated or moved. +/// +/// This is what tells a writer that reserves its own temp file +/// (`atomic_write`) from one that writes to a clock-and-pid name with +/// `fs::write`: the second opens the foreign file, truncates it and renames +/// it away, taking another writer's staged bytes with it. Every name in the +/// window is taken, so a writer that picks one of them hits a foreign file +/// whatever millisecond its write runs in. +/// +/// The window starts a "lead" after the names are first created, and `write` +/// only runs once it has begun, so creating the names can take as long as the +/// host needs (about a millisecond per file is common) without the clock +/// running past them. `write` runs once per attempt, so it must set up +/// whatever it needs itself, inside the window. When the creation outlasts +/// the lead or the write outlasts the window, the attempt is retried with a +/// longer lead or window. A host too slow for any of them fails the test with +/// an "inconclusive" panic, because a probe that quietly passes there is a +/// test that does not test. The one exception is Windows, where the cost of +/// creating files has not been measured: there the probe says so on stderr +/// and passes, so a run on a Windows host can pass without the probe having +/// proven anything. The directory the names live in must exist already. +pub fn assert_write_spares_foreign_temp_files( + foreign_name: impl Fn(u128) -> PathBuf, + write: impl FnMut() -> anyhow::Result<()>, +) -> anyhow::Result<()> { + let outcome = probe(foreign_name, write, INITIAL_WINDOW_MS, MAX_MS, ATTEMPTS)?; + require_conclusive(outcome, cfg!(windows)); + Ok(()) +} + +/// Fails the test for an inconclusive `outcome` unless `tolerate_inconclusive` +/// is set, in which case it only says so on stderr. A parameter rather than a +/// `cfg!` inside, so both branches run in a test on every platform. +fn require_conclusive(outcome: Outcome, tolerate_inconclusive: bool) { + if let Outcome::Inconclusive(why) = outcome { + if tolerate_inconclusive { + eprintln!("inconclusive (tolerated on Windows): {why}"); + } else { + panic!( + "the foreign-temp-file probe is inconclusive, so it cannot show the writer is \ + safe: {why}" + ); + } + } +} + +fn probe( + foreign_name: impl Fn(u128) -> PathBuf, + mut write: impl FnMut() -> anyhow::Result<()>, + initial_window_ms: u128, + max_ms: u128, + attempts: usize, +) -> anyhow::Result { + let mut window_ms = initial_window_ms; + let mut lead_ms = 2 * initial_window_ms; + for attempt in 1..=attempts { + let created_from = unix_time_millis(); + let window_start = created_from + lead_ms; + let window_end = window_start + window_ms; + let foreign: Vec = (window_start..window_end).map(&foreign_name).collect(); + for path in &foreign { + std::fs::write(path, FOREIGN)?; + } + let setup_ms = unix_time_millis() - created_from; + while unix_time_millis() < window_start { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + + let write_started = unix_time_millis(); + write()?; + let finished = unix_time_millis(); + + for path in &foreign { + assert_eq!( + std::fs::read(path).ok().as_deref(), + Some(FOREIGN), + "{} belongs to another writer but was written through or moved", + path.display() + ); + } + for path in &foreign { + std::fs::remove_file(path)?; + } + if finished < window_end { + return Ok(Outcome::Spared); + } + window_ms = window_ms.max(4 * (finished - write_started)); + lead_ms = lead_ms.max(2 * setup_ms); + if window_ms.max(lead_ms) > max_ms { + return Ok(Outcome::Inconclusive(format!( + "creating {} names took {setup_ms} ms and the write took {} ms, past the \ + window; this host is too slow for the test to say anything", + foreign.len(), + finished - write_started + ))); + } + eprintln!( + "attempt {attempt}/{attempts}: the write finished after the pre-created names ran \ + out; retrying with a {window_ms} ms window after a {lead_ms} ms lead" + ); + } + Ok(Outcome::Inconclusive( + "on every attempt the write ran after the window of pre-created names, so this host \ + is too slow for this test to say anything" + .to_owned(), + )) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::Path; + + fn occupied_name(dir: &Path, millis: u128) -> PathBuf { + dir.join(format!("target.tmp-{millis}")) + } + + /// How many times a self-test below reruns the probe when the host was too + /// slow for it to be conclusive, before accepting that it cannot say. + const SELF_TEST_RUNS: usize = 3; + + /// Reruns `run` until it reaches a verdict about the writer, `None` when + /// every run was inconclusive. A panic from the probe propagates. + fn run_until_conclusive(mut run: impl FnMut() -> anyhow::Result) -> Option { + for _ in 0..SELF_TEST_RUNS { + match run().expect("probe ran") { + Outcome::Inconclusive(why) => eprintln!("self-test run inconclusive: {why}"), + Outcome::Spared => return Some(Outcome::Spared), + } + } + None + } + + fn panic_text(payload: &(dyn std::any::Any + Send)) -> String { + payload + .downcast_ref::() + .cloned() + .or_else(|| { + payload + .downcast_ref::<&str>() + .map(|text| (*text).to_owned()) + }) + .unwrap_or_default() + } + + #[test] + fn a_writer_that_reuses_a_retired_name_is_caught() { + let dir = tempfile::tempdir().expect("scratch directory"); + // A writer shaped like the retired `write_atomic`: a temp name fixed + // by the clock alone, opened with `fs::write`, which truncates + // whatever is already there. + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + run_until_conclusive(|| { + probe( + |millis| occupied_name(dir.path(), millis), + || { + std::fs::write( + occupied_name(dir.path(), unix_time_millis()), + b"clobbered", + )?; + Ok(()) + }, + INITIAL_WINDOW_MS, + MAX_MS, + ATTEMPTS, + ) + }) + })); + match result { + Err(payload) => { + let text = panic_text(payload.as_ref()); + assert!( + text.contains("belongs to another writer"), + "the probe failed for another reason: {text}" + ); + } + Ok(Some(verdict)) => { + panic!("a writer that truncates a foreign temp file was reported {verdict:?}") + } + // A host too slow for any window to cover the write says nothing + // about the probe; the run above already printed why. + Ok(None) => {} + } + } + + #[test] + fn a_writer_that_picks_its_own_name_is_spared() { + let dir = tempfile::tempdir().expect("scratch directory"); + let verdict = run_until_conclusive(|| { + probe( + |millis| occupied_name(dir.path(), millis), + || { + std::fs::write(dir.path().join("unrelated.tmp"), b"staged")?; + Ok(()) + }, + INITIAL_WINDOW_MS, + MAX_MS, + ATTEMPTS, + ) + }); + // `None` is a host too slow to say; a panic inside would be the probe + // wrongly blaming a writer that never touched a foreign name. + assert!( + matches!(verdict, None | Some(Outcome::Spared)), + "{verdict:?}" + ); + let leftovers: Vec<_> = std::fs::read_dir(dir.path()) + .expect("list scratch directory") + .map(|entry| entry.expect("directory entry").file_name()) + .collect(); + assert_eq!( + leftovers, + [std::ffi::OsString::from("unrelated.tmp")], + "the probe must remove the names it occupied" + ); + } + + #[test] + fn a_write_slower_than_every_window_is_reported_inconclusive() { + let dir = tempfile::tempdir().expect("scratch directory"); + let outcome = probe( + |millis| occupied_name(dir.path(), millis), + || { + std::thread::sleep(std::time::Duration::from_millis(60)); + Ok(()) + }, + 5, + 20, + ATTEMPTS, + ) + .expect("probe ran"); + assert!( + matches!(outcome, Outcome::Inconclusive(_)), + "a write that outlasts the window proves nothing: {outcome:?}" + ); + } + + #[test] + #[should_panic(expected = "inconclusive")] + fn an_inconclusive_probe_fails_when_not_tolerated() { + require_conclusive(Outcome::Inconclusive("too slow".to_owned()), false); + } + + #[test] + fn an_inconclusive_probe_is_tolerated_when_allowed() { + // Returning without a panic is the whole claim. The public entry point + // passes `cfg!(windows)` for this flag. + require_conclusive(Outcome::Inconclusive("too slow".to_owned()), true); + } + + #[test] + fn a_conclusive_probe_passes_either_way() { + require_conclusive(Outcome::Spared, false); + require_conclusive(Outcome::Spared, true); + } +} diff --git a/crates/rocm-core/src/jsonl_store.rs b/crates/rocm-core/src/jsonl_store.rs index c0b9dd0bd6..abd0ac0a91 100644 --- a/crates/rocm-core/src/jsonl_store.rs +++ b/crates/rocm-core/src/jsonl_store.rs @@ -11,22 +11,22 @@ //! with the store. A torn tail — crash debris, or left by a write that failed //! partway — is cut by the next append rather than truncated in place: that //! append writes the store's clean prefix plus its own new record as a single -//! rewrite via `write_atomic`, so a lock-free reader never observes a file -//! shrunk mid-repair. That rewrite costs O(store) rather than O(record), the -//! same as a proposal status update, which rewrites the whole store via -//! `write_atomic` too. Only when that rewrite itself fails (a full disk, or on -//! Windows another program holding the store open without delete sharing) does -//! the append cut the debris in place instead, so a store cannot be wedged by -//! a failed repair; otherwise nothing in this module ever shrinks a store in -//! place. Readers take no lock, and an unterminated last line that is not -//! valid JSON — an append caught mid-write — is skipped rather than reported -//! as corruption. +//! rewrite via [`crate::atomic_write::write_file_atomically`], so a lock-free +//! reader never observes a file shrunk mid-repair. That rewrite costs O(store) +//! rather than O(record), the same as a proposal status update, which rewrites +//! the whole store the same way. Only when that rewrite itself fails (a full +//! disk, or on Windows another program holding the store open without delete +//! sharing) does the append cut the debris in place instead, so a store cannot +//! be wedged by a failed repair; otherwise nothing in this module ever shrinks +//! a store in place. Readers take no lock, and an unterminated last line that +//! is not valid JSON — an append caught mid-write — is skipped rather than +//! reported as corruption. //! //! A store can additionally be capped by size-based rotation (see //! [`JsonlRotation`]): the append that finds the live file at the cap renames //! it whole to `.1` first, under the same lock — before any torn tail //! in it would otherwise be repaired, so an at-cap store is never paid for -//! with a `write_atomic` rewrite only to be rotated away immediately after — +//! with an atomic rewrite only to be rotated away immediately after — //! and the recent-record readers span the rotated generations. //! //! The engine is generic over the record type. Which stores exist, their @@ -200,7 +200,7 @@ pub(crate) fn append_jsonl_record( file, tail_start, &line, - |path, content| crate::write_atomic(path, content, "jsonl"), + crate::atomic_write::write_file_atomically, truncate_jsonl_in_place, )?, TailState::ValidUnterminated(start) => { @@ -259,12 +259,11 @@ fn repair_torn_tail_and_append( content.extend_from_slice(line); // Done with the pre-repair file's handle before replacing it, rather than // leaving that to whatever happens to still have `file` open at function - // return. The rename already succeeds with this handle open regardless — - // std opens files with `FILE_SHARE_DELETE` by default on Windows, so a - // rename isn't blocked by it (see `write_atomic`'s doc for the case that - // *isn't* covered: another process holding the destination open without - // delete-sharing) — but this states the intent rather than leaning on that - // default. + // return. The replace already succeeds with this handle open regardless — + // std opens files with `FILE_SHARE_DELETE` by default on Windows, so it + // isn't blocked by it (see the `atomic_write` module doc for how the publish + // treats a destination another process holds open) — but this states the + // intent rather than leaning on that default. drop(file); let replace_error = match replace(path, &content) { Ok(()) => return Ok(()), @@ -432,7 +431,7 @@ fn inspect_tail(path: &Path, file: &fs::File) -> Result { /// /// Returns fewer bytes if the file now ends before `end`. Nothing this /// module's own writers do shrinks a store in place — an append only ever -/// grows it, and a torn-tail repair replaces it wholesale via `write_atomic` +/// grows it, and a torn-tail repair replaces it wholesale via the atomic writer /// (see [`inspect_tail`] and [`TailState::Torn`]) — except the repair's /// fallback, taken only when that replacement fails and only under the store's /// lock (see [`repair_torn_tail_and_append`]); otherwise a short read here @@ -1164,11 +1163,9 @@ mod tests { seed_store(&path, &format!("{}\n{torn}", entry_line("a")))?; let before = file_identity(&path)?; - repair_store_with( - &path, - |path, content| crate::write_atomic(path, content, "jsonl"), - |_, _| bail!("truncate must not run when the replace succeeded"), - )?; + repair_store_with(&path, crate::atomic_write::write_file_atomically, |_, _| { + bail!("truncate must not run when the replace succeeded") + })?; let after = file_identity(&path)?; let text = fs::read_to_string(&path); @@ -1185,9 +1182,11 @@ mod tests { let seeded = format!("{}\n{torn}", entry_line("a")); seed_store(&path, &seeded)?; - // Layered like the real steps (`write_atomic` and - // `truncate_jsonl_in_place` both wrap the OS error in a context), so - // the assertion fails if either cause is formatted without its chain. + // Layered like the real steps (`write_file_atomically` puts a context + // such as `failed to publish ` over the OS error, and + // `truncate_jsonl_in_place` puts `failed to truncate ` over its + // own), so the assertion fails if either cause is formatted without its + // chain. let result = repair_store_with( &path, |_, _| { diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index 1a7f6606c6..a3462b8b6a 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -2,9 +2,7 @@ // // SPDX-License-Identifier: MIT -use anyhow::{Context, Result, bail}; -use std::fs; -use std::path::Path; +use anyhow::{Result, bail}; use std::time::{SystemTime, UNIX_EPOCH}; pub mod amd_smi; @@ -60,6 +58,8 @@ pub mod service_record; pub mod terminal; pub mod user_path; +#[cfg(any(test, feature = "test-support"))] +pub mod foreign_temp_probe; #[cfg(test)] mod test_env; #[cfg(test)] @@ -354,70 +354,10 @@ pub fn unix_time_millis() -> u128 { .as_millis() } -/// Replace `path`'s content with `bytes`, atomically. -/// -/// Writes a sibling temp file and renames it over the destination: an -/// interrupted write (full disk, crash, killed process) can never leave a -/// truncated file in place, and a reader sees -/// either the old content or the new content. The destination is NOT removed -/// first: `fs::rename` replaces an existing file on both supported platforms -/// (it is `MoveFileEx` with `MOVEFILE_REPLACE_EXISTING` on Windows), and -/// removing it would open a window with no file at all, which a concurrent -/// reader could see as "file doesn't exist" instead of the old or new -/// content. The temp file is removed if the write or the rename fails. -/// -/// `tmp_ext` names the temp file's extension (e.g. `"json"`, `"jsonl"`). -/// Both halves of the generated suffix matter: the timestamp keeps -/// successive writes to the same path apart, and the pid keeps two -/// processes writing in the same millisecond from clobbering each other's -/// temp file and losing one of the two updates. -/// -/// Not guaranteed: -/// - Serializing writers: this takes no lock, so two read-modify-write -/// cycles that overlap still lose one update (last rename wins). A caller -/// whose new content depends on the old must hold a lock across the whole -/// cycle, as the JSONL proposal store does (`jsonl_store::jsonl_lock_path`). -/// Two callers in the *same process* within the same millisecond are a -/// sharper case of this, not just a milder one: they generate the exact -/// same temp path below, so concurrent `fs::write`s truncating that one -/// path can publish mixed bytes, and the losing rename then fails outright -/// (its source file already gone) rather than merely losing an update — -/// this is the same-process collision -/// `automation::tests::concurrent_appends_from_many_threads_lose_no_records`'s -/// own comment describes ("two threads of one process even shared a -/// temp-file name") — fixed for its own callers by the per-store lock -/// `automation.rs` takes before ever calling into this function. -/// - Durability: the bytes are not fsynced before the rename, so a power -/// loss can still surface the new name with unflushed contents. -/// - Replacing a destination another process holds open without delete -/// sharing: on Windows the rename then fails with a sharing violation, -/// this returns that error, and the old destination stays in place. -pub fn write_atomic(path: &Path, bytes: &[u8], tmp_ext: &str) -> Result<()> { - let tmp_path = path.with_extension(format!( - "{tmp_ext}.tmp-{}-{}", - unix_time_millis(), - std::process::id() - )); - fs::write(&tmp_path, bytes) - .with_context(|| format!("failed to write {}", tmp_path.display())) - .inspect_err(|_| { - let _ = fs::remove_file(&tmp_path); - })?; - fs::rename(&tmp_path, path) - .with_context(|| format!("failed to replace {}", path.display())) - .inspect_err(|_| { - let _ = fs::remove_file(&tmp_path); - })?; - Ok(()) -} - #[cfg(test)] mod tests { use super::*; - use std::ffi::OsString; - use std::fs; - fn test_exit_status() -> std::process::ExitStatus { std::process::ExitStatus::default() } @@ -447,64 +387,4 @@ mod tests { let output = test_process_output(" \n", " "); assert_eq!(command_failure_detail(&output), "no output"); } - - #[test] - fn write_atomic_writes_bytes_and_leaves_no_temp_file() -> Result<()> { - let temp_root = - std::env::temp_dir().join(format!("rocm-cli-write-atomic-ok-{}", unix_time_millis())); - fs::create_dir_all(&temp_root)?; - let path = temp_root.join("store.jsonl"); - - write_atomic(&path, b"hello\n", "jsonl")?; - - let written = fs::read(&path)?; - let leftovers: Vec<_> = fs::read_dir(&temp_root)? - .filter_map(Result::ok) - .map(|entry| entry.file_name()) - .collect(); - let _ = fs::remove_dir_all(&temp_root); - - assert_eq!(written, b"hello\n"); - assert_eq!( - leftovers, - vec![OsString::from("store.jsonl")], - "a successful write must leave no `*.tmp-*` sibling behind" - ); - Ok(()) - } - - #[test] - fn write_atomic_rename_failure_leaves_destination_and_dir_untouched() -> Result<()> { - let temp_root = - std::env::temp_dir().join(format!("rocm-cli-write-atomic-fail-{}", unix_time_millis())); - fs::create_dir_all(&temp_root)?; - let path = temp_root.join("store.jsonl"); - // A file can't be renamed over an existing directory, so making the - // destination a directory forces `write_atomic`'s rename step to fail - // after the temp file has already been written. - fs::create_dir_all(&path)?; - - let result = write_atomic(&path, b"hello\n", "jsonl"); - let destination_is_still_a_dir = path.is_dir(); - let leftovers: Vec<_> = fs::read_dir(&temp_root)? - .filter_map(Result::ok) - .map(|entry| entry.file_name()) - .collect(); - let _ = fs::remove_dir_all(&temp_root); - - assert!( - result.is_err(), - "renaming a file onto an existing directory must fail" - ); - assert!( - destination_is_still_a_dir, - "a failed rename must leave the destination untouched" - ); - assert_eq!( - leftovers, - vec![OsString::from("store.jsonl")], - "the temp file must be removed when the rename fails" - ); - Ok(()) - } } diff --git a/crates/rocm-core/src/service_record.rs b/crates/rocm-core/src/service_record.rs index ff6a6290d8..eea3a73715 100644 --- a/crates/rocm-core/src/service_record.rs +++ b/crates/rocm-core/src/service_record.rs @@ -530,10 +530,11 @@ impl ManagedServiceRecord { let storage_record = host_record.with_storage_paths(); let bytes = serde_json::to_vec_pretty(&storage_record) .context("failed to serialize service record")?; - crate::write_atomic(&host_record.manifest_path, &bytes, "json").with_context(|| { + let manifest_path = &host_record.manifest_path; + crate::atomic_write::write_file_atomically(manifest_path, &bytes).with_context(|| { format!( "failed to write service manifest {}", - host_record.manifest_path.display() + manifest_path.display() ) }) } @@ -542,7 +543,8 @@ impl ManagedServiceRecord { #[cfg(test)] mod tests { use super::*; - use crate::test_support::{probe_test_record, temp_app_paths}; + use crate::foreign_temp_probe::assert_write_spares_foreign_temp_files; + use crate::test_support::{probe_test_record, retired_temp_path, temp_app_paths}; /// An unknown id reads as one plain sentence pointing at the listing /// command, with no OS error chained under it. @@ -800,6 +802,45 @@ mod tests { assert_eq!(record.engine_start_ticks, expected); } + /// Writing a service manifest never writes through a file another writer + /// already staged next to it. The occupied names are the ones the retired + /// clock-and-pid temp naming would have picked, so a write that still used + /// one would truncate a foreign file and rename it into place. Reading the + /// manifest back would pass for either writer; the foreign files are what + /// tell them apart. + #[test] + fn writing_a_service_manifest_never_reuses_another_writers_temp_file() -> Result<()> { + let (root, paths) = temp_app_paths("record-write-foreign-temp"); + let mut record = ManagedServiceRecord::new( + &paths, + "svc-foreign-temp", + "vllm", + "Qwen/Qwen3-0.6B", + "Qwen/Qwen3-0.6B", + "127.0.0.1", + 9, + "managed", + std::process::id(), + None, + None, + Some("gpu_required".to_owned()), + ); + record.write()?; + record.startup_phase = Some("loading".to_owned()); + let manifest_path = record.manifest_path.clone(); + + assert_write_spares_foreign_temp_files( + |millis| retired_temp_path(&manifest_path, millis), + || record.write(), + )?; + + let published: ManagedServiceRecord = serde_json::from_slice(&fs::read(&manifest_path)?)?; + let _ = fs::remove_dir_all(&root); + assert_eq!(published.service_id, "svc-foreign-temp"); + assert_eq!(published.startup_phase.as_deref(), Some("loading")); + Ok(()) + } + #[test] fn engine_state_refresh_keeps_the_runtime_selector_the_launch_requested() -> Result<()> { let (root, paths) = temp_app_paths("record-runtime-pin"); diff --git a/crates/rocm-core/src/test_support.rs b/crates/rocm-core/src/test_support.rs index f5f2c18410..19b1b11c42 100644 --- a/crates/rocm-core/src/test_support.rs +++ b/crates/rocm-core/src/test_support.rs @@ -211,6 +211,19 @@ fn raised_case_count(requested: Option<&str>, default: u32) -> u32 { .map_or(default, |requested: u32| requested.max(default)) } +/// The temp name the retired `write_atomic` picked for `destination` at +/// `millis`: the full file name plus `.tmp--`, with this +/// process's pid. Unlike the names `atomic_write` picks, it was fixed by the +/// clock and the pid alone, so a test can occupy it ahead of time. +pub(crate) fn retired_temp_path(destination: &std::path::Path, millis: u128) -> PathBuf { + let mut name = destination + .file_name() + .expect("a destination has a file name") + .to_os_string(); + name.push(format!(".tmp-{millis}-{}", std::process::id())); + destination.with_file_name(name) +} + #[cfg(test)] mod tests { use super::*; diff --git a/docs/architecture.md b/docs/architecture.md index e7f079d9bf..305fb5d47f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -177,7 +177,7 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`anthropic_endpoint.rs`](../crates/rocm-core/src/anthropic_endpoint.rs) — (ROCMAI-597) the Anthropic-compatible `/v1/messages` surface of a managed service: `AnthropicEndpointMode` (`rocm` fronts the OpenAI-speaking engine), and the readiness check and wait on the published endpoint. - [`amd_smi.rs`](../crates/rocm-core/src/amd_smi.rs) — (ROCMAI-723) locating the `amd-smi` binary: `resolve_amd_smi_binary` (re-exported at the crate root) prefers the newest managed ROCm SDK runtime recorded in the registry, then the home-directory venv and legacy locations, then the bare name for a `PATH` lookup. It only reads the registry records [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) exposes and [`runtime.rs`](../crates/rocm-core/src/runtime.rs)'s data/home directory lookups. As of EAI-8457 it also owns the *launch*: `amd_smi_json` (also re-exported at the crate root) is the one supported way for a synchronous caller to run `amd-smi`, because it is the only place the mandatory `/dev/kfd` pre-flight runs (`KFD_DEVICE`, `preflight_kfd_device`, `kfd_readable`/`kfd_readable_with`, and the thread-bounded, budget-capped `device_readable`/`device_readable_within_budget` probe behind `KFD_OPEN_TIMEOUT`). A device that cannot be opened declines the spawn rather than letting `amd-smi` park in uninterruptible sleep, where no timeout can reach it. The spawn itself still goes through [`process.rs`](../crates/rocm-core/src/process.rs)'s `run_with_timeout`, and the non-zero-exit quote is bounded with that module's `MAX_QUOTED_OUTPUT_CHARS` rather than a second copy of the constant. Callers: [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s `gpu_vram_usage_amd_smi`/`detect_gpu_count`, both through that module's own `amd_smi_probe_json` (which logs the discarded decline at `info`, since the default filter is `warn,rocm=info` and a decline on a healthy host is otherwise indistinguishable from "no GPU"), and `rocmd`'s [`host_snapshot.rs`](../apps/rocmd/src/host_snapshot.rs)`::capture_amd_smi_json`. That the list stays complete is enforced rather than asserted: [`amd_smi_launch_contract.rs`](../xtask/src/amd_smi_launch_contract.rs) is a static contract test failing any file outside its two-entry allowlist that calls `resolve_amd_smi_binary`, so a new probe cannot be spawned directly without a reviewed exemption. The `rocm dash` collectors keep their own pre-flight *decision* (`preflight_passes`), since they gate an async launch, but share this module's bounded device probe through the `pub fn device_readable_bounded` re-export — an allowlisted `rocm-dash-collectors -> rocm-core` edge recorded in [`crate_edges.rs`](../xtask/src/crate_edges.rs), so the one launch site outside this chokepoint cannot drift away from the bound and the budget. -- [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) — the shared atomic file writer `rocm` and `rocmd` both use (unique exclusive temp file, then rename, or `ReplaceFileW` on Windows); atomic, not durable, since nothing is fsynced. Other hand-rolled temp-then-rename writers remain outside it, for example [`remote/session.rs`](../apps/rocm/src/remote/session.rs)'s session-file write, [`uninstall.rs`](../apps/rocm/src/uninstall.rs)'s profile replacement, `write_atomically` in [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs), and rocm-core's `write_atomic` in [`lib.rs`](../crates/rocm-core/src/lib.rs), which `RocmCliConfig::save` in [`config.rs`](../crates/rocm-core/src/config.rs), the automation event and proposal stores in [`automation.rs`](../crates/rocm-core/src/automation.rs), the JSONL stores in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and the service-record manifests in [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use; it is `pub` because `rocm-engine-lemonade` also writes through it, in [`install.rs`](../engines/lemonade/src/install.rs) (see the `write_atomic` entry below). #541, which builds on this writer, moves the config save onto it; #618 tracks that move. A new atomic write should call this module rather than add another copy. +- [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) — the shared atomic file writer behind `rocm`, `rocmd`, this crate's own atomic replaces ([`config.rs`](../crates/rocm-core/src/config.rs)'s `RocmCliConfig::save`, the automation event and proposal stores in [`automation.rs`](../crates/rocm-core/src/automation.rs), the JSONL stores in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs), the service-record manifests in [`service_record.rs`](../crates/rocm-core/src/service_record.rs)) and the Lemonade engine's rewrite of `backend_versions.json` in [`install.rs`](../engines/lemonade/src/install.rs) (unique exclusive temp file, then rename, which on Windows falls back to `ReplaceFileW` only if the rename fails); atomic, not durable, since nothing is fsynced. Other hand-rolled temp-then-rename writers remain outside it, for example [`remote/session.rs`](../apps/rocm/src/remote/session.rs)'s session-file write, [`uninstall.rs`](../apps/rocm/src/uninstall.rs)'s profile replacement and `write_atomically` in [`comfyui/source_swap.rs`](../apps/rocm/src/comfyui/source_swap.rs). A new atomic write should call this module rather than add another copy. - [`automation.rs`](../crates/rocm-core/src/automation.rs) — (ROCMAI-151) automation-daemon runtime state and its on-disk event/proposal/audit records, and the policy applied to each store (which are rotated, and the proposal compaction) — see the module's own doc comment for what it owns. The generic JSONL machinery under those stores lives in [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs). The dependency runs one way: this module reads `WatcherMode` from [`config.rs`](../crates/rocm-core/src/config.rs), and [`config.rs`](../crates/rocm-core/src/config.rs) imports nothing from here, because the built-in watcher catalog (whose entries carry a default `WatcherMode`, and which `RocmCliConfig::watcher_enabled` and `effective_watcher_mode` take as an argument) lives in [`config.rs`](../crates/rocm-core/src/config.rs) beside `WatcherMode`. - [`binary_paths.rs`](../crates/rocm-core/src/binary_paths.rs) — (ROCMAI-151, Phase 7d) locating `rocm-engine-` siblings, and the CLI binary itself (a sibling `rocm` only when running from a cargo `deps/` directory). - [`codex_bridge.rs`](../crates/rocm-core/src/codex_bridge.rs) — (ROCMAI-662) the `CodexBridgeSnapshot`/`CodexBridgeGpuSnapshot`/`CodexBridgeEngine` host-snapshot DTOs. Plain serde data: `rocm` and `rocmd` each build their own snapshot of this shape, so the types live here rather than in either binary. @@ -223,9 +223,8 @@ What else stays in [`lib.rs`](../crates/rocm-core/src/lib.rs), beside module dec - `sanitize_component` — the lowercase-alphanumeric slugger shared by [`service_identity.rs`](../crates/rocm-core/src/service_identity.rs)'s `generate_service_id` and [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs)'s artifact cache-marker names. It is not model-catalog-specific, so it does not move there. - `TORCH_ALIGNMENT_DISABLED_ENV`/`torch_alignment_disabled` — the one read of the torch-alignment opt-out, shared by the CLI and the vLLM engine so the two cannot disagree. - `VLLM_GPU_MEMORY_UTILIZATION_HINT` — a single wording contract between the CLI, the vLLM engine, and this crate's own `fix-16-vllm-oom` diagnosis; `rocm-core` is the only crate all of them can reach. -- **`write_atomic`** — not permanent. It is the temp-then-rename writer [`config.rs`](../crates/rocm-core/src/config.rs), [`automation.rs`](../crates/rocm-core/src/automation.rs), [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) and [`service_record.rs`](../crates/rocm-core/src/service_record.rs) use for their atomic replaces. The shared, stronger writer `rocm` and `rocmd` use is [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs); [#541](https://github.com/ROCm/rocm-cli/pull/541), which builds on it, moves `RocmCliConfig::save` onto it. It is `pub`, visible to other crates, because `rocm-engine-lemonade` also writes Lemonade's `backend_versions.json` through it, in [`install.rs`](../engines/lemonade/src/install.rs). The automation, JSONL-store and service-record callers follow; `write_atomic` is deleted only after that Lemonade caller has moved to the shared writer too (ROCMAI-724). That is a behavior change, not a pure move, so it is not done here. -Once the `write_atomic` move lands, [`lib.rs`](../crates/rocm-core/src/lib.rs) holds only module declarations, re-exports, the crate-root primitives above, the private `use` shims that let sibling modules reach some `host_gpu`, `managed_runtime`, `rocm_install` and `runtime` items as `crate::` paths, and its test-only modules — the end state of the modularization effort for this crate (ROCMAI-27, planned last: highest fan-in, since every app and engine crate depends on it, but lowest novelty, since the sibling modules already prove the pattern). +[`lib.rs`](../crates/rocm-core/src/lib.rs) now holds only module declarations, re-exports, the crate-root primitives above, the private `use` shims that let sibling modules reach some `host_gpu`, `managed_runtime`, `rocm_install` and `runtime` items as `crate::` paths, and its test-only modules — the end state of the modularization effort for this crate (ROCMAI-27, planned last: highest fan-in, since every app and engine crate depends on it, but lowest novelty, since the sibling modules already prove the pattern). Further `rocm-core` modules, each with its own doc comment except where noted: @@ -233,6 +232,7 @@ Further `rocm-core` modules, each with its own doc comment except where noted: - [`disk_space.rs`](../crates/rocm-core/src/disk_space.rs) — free-space preflight checks and out-of-space error reporting. - [`fake_executable.rs`](../crates/rocm-core/src/fake_executable.rs) — writing stand-in executables that tests across the workspace then run. - [`fix.rs`](../crates/rocm-core/src/fix.rs) — `rocm fix`: remediations for diagnosed failure modes. Its `RECIPES` list (in [`fix/catalog.rs`](../crates/rocm-core/src/fix/catalog.rs)), together with the checkers under `diagnose/` (their table is [`catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs)), is the authoritative failure catalog. [`fix.rs`](../crates/rocm-core/src/fix.rs) stays the module root, as [`examine.rs`](../crates/rocm-core/src/examine.rs) does for `examine/`, so links to it stay valid, with its children in `fix/`; it keeps the dispatch (`apply`, `Plan`, `plan_of`, `act_on`) and re-exports the names the rest of the crate and the apps use, so `crate::fix::exit`, `rocm_core::fix_exit`, `rocm_core::fix::FixClass` and the `crate::fix::` oracles that the tests under `diagnose/` call keep resolving. The children are private modules: [`fix/options.rs`](../crates/rocm-core/src/fix/options.rs) (the named exit codes, `FixOptions` and `FixClass`); [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs) (the consent decision and `confirm`, the run and query timeouts, `current_os`, and the `fail!` helper, which it re-exports by path for its siblings); [`fix/runners.rs`](../crates/rocm-core/src/fix/runners.rs) (the host-mutating runners, one per auto-applicable fix, with their rc-file, PATH and device-pin helpers; only the four entry points the catalog names are visible to the root, and the `*_windows` variants are ordinary functions that every platform compiles); [`fix/catalog.rs`](../crates/rocm-core/src/fix/catalog.rs) (`FixRecipe`, the per-platform class table, `RECIPES`, the lookups by fix-id, and the `cfg(test)` oracles [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) and the e2e crate hold their own copies to); [`fix/render.rs`](../crates/rocm-core/src/fix/render.rs) (`list_recipes`, a recipe's plan and verify lines, `format_flags`, and the refusal for a diagnosis position passed as a fix-id); and [`fix/manifest.rs`](../crates/rocm-core/src/fix/manifest.rs) (the published JSON contract, `CATALOG_CONTRACT_VERSION` and `catalog_manifest`). `render` and `manifest` read `RECIPES` from `catalog`; nothing in `catalog` reads either. +- [`foreign_temp_probe.rs`](../crates/rocm-core/src/foreign_temp_probe.rs) — a test probe, compiled only for this crate's tests and under the `test-support` feature, that checks a writer leaves another writer's temp files alone; shared by this crate's tests and the tests of the crates that publish through `atomic_write.rs`. - [`hardware_root.rs`](../crates/rocm-core/src/hardware_root.rs) — the root the hardware probes resolve host paths through (`host_path`): the real host in a normal build, a simulated host tree for the E2E suite in an `e2e-test-hooks` build. It also owns `read_os_release` (the `/etc/os-release` read with its `/usr/lib/os-release` fallback) and `real_os_release`, the one reader every package-manager plan and install hint uses, which always reads the real host even under a simulated root; the resolver it passes is private, so a second copy of that reader cannot be written outside this module. - [`log_tail.rs`](../crates/rocm-core/src/log_tail.rs) — `tail_lines`, the bounded log-tail read shared by the lemonade and vLLM adapters (no module doc comment). - [`openmpi.rs`](../crates/rocm-core/src/openmpi.rs) — OpenMPI, libatomic and libnuma detection, the `libmpi_cxx` compatibility shim, and the per-library install plans (`build_openmpi_install_plan`, `build_libatomic_install_plan`, `build_libnuma_install_plan`) and hints, needed by vLLM's ROCm wheels. The distro-generic half of that planning lives in [`package_plan.rs`](../crates/rocm-core/src/package_plan.rs); [`openmpi.rs`](../crates/rocm-core/src/openmpi.rs) re-exports its public items, so `rocm_core::openmpi::running_as_root`, `plan_for_os_release_read` and the rest keep resolving, and `pub(crate)` re-exports `PackageManager` and `resolve_package_manager` for [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs). diff --git a/engines/lemonade/Cargo.toml b/engines/lemonade/Cargo.toml index 5e0e9208c2..98b56d0681 100644 --- a/engines/lemonade/Cargo.toml +++ b/engines/lemonade/Cargo.toml @@ -38,5 +38,6 @@ ureq = { version = "2.12", features = ["json", "native-certs"] } zip = { version = "9.0", default-features = false, features = ["deflate"] } [dev-dependencies] +rocm-core = { path = "../../crates/rocm-core", features = ["test-support"] } # Archive-extraction property tests; see `crates/rocm-core/src/archive_props.rs`. proptest = { version = "1", default-features = false, features = ["std"] } diff --git a/engines/lemonade/src/install.rs b/engines/lemonade/src/install.rs index 25728566d3..b6617fc215 100644 --- a/engines/lemonade/src/install.rs +++ b/engines/lemonade/src/install.rs @@ -364,7 +364,7 @@ fn restore_packaged_backend_versions_with( // Missing (or not JSON): there is nothing of the user's to keep. None => packaged, }; - rocm_core::write_atomic(&target, &restored, "pins") + write_pin_file(&target, &restored) .with_context(|| format!("failed to restore {}", target.display()))?; Ok(Some(PinFileBackup { target, @@ -372,6 +372,28 @@ fn restore_packaged_backend_versions_with( })) } +/// Replaces the pin file at `target` through +/// `rocm_core::atomic_write::write_file_atomically`, but only inside a +/// directory that is still there. +/// +/// The shared writer creates a missing parent directory. For this file that +/// would rebuild `resources/` in a runtime tree that has been removed, leaving +/// a directory holding nothing but a pin file that looks like the start of an +/// install. The restore and the put-back both run after a failed install, when +/// the tree may be gone, so a missing parent stays a failure, as it was before +/// the shared writer took over. +fn write_pin_file(target: &Path, bytes: &[u8]) -> Result<()> { + if let Some(parent) = target.parent() + && !parent.is_dir() + { + bail!( + "{} does not exist; not recreating it to hold the pin file", + parent.display() + ); + } + rocm_core::atomic_write::write_file_atomically(target, bytes) +} + /// The two entries of `resources/backend_versions.json` that backend alignment /// writes: the ROCm version a backend is built for and the llama.cpp build tag. const ALIGNMENT_PIN_KEYS: [(&str, &str); 2] = [("therock", "version"), ("llamacpp", "rocm-stable")]; @@ -416,7 +438,7 @@ fn set_or_remove_pin( /// What `resources/backend_versions.json` held before /// [`restore_packaged_backend_versions`] rewrote it. Written back through -/// `rocm_core::write_atomic`, like the restore itself. +/// `rocm_core::atomic_write::write_file_atomically`, like the restore itself. /// /// A restore is attempted after any failed install, including a transient /// network or disk error, so a restore whose retry fails too must not leave a @@ -439,7 +461,7 @@ impl PinFileBackup { /// Writes the previous bytes back, or removes the file if there were none. pub(crate) fn put_back(self) -> Result<()> { match self.previous { - Some(bytes) => rocm_core::write_atomic(&self.target, &bytes, "pins") + Some(bytes) => write_pin_file(&self.target, &bytes) .with_context(|| format!("failed to put back {}", self.target.display())), None => match fs::remove_file(&self.target) { Err(err) if err.kind() != std::io::ErrorKind::NotFound => { @@ -567,7 +589,8 @@ fn collect_embeddable_roots( #[cfg(test)] mod tests { use super::*; - use crate::test_support::scratch_dir; + use crate::test_support::{retired_pin_temp_path, scratch_dir}; + use rocm_core::foreign_temp_probe::assert_write_spares_foreign_temp_files; use sha2::{Digest, Sha256}; use std::io::Write; #[cfg(windows)] @@ -1097,40 +1120,64 @@ mod tests { ) } + /// A runtime tree holding a pin file, with a verified embeddable archive + /// cached that ships its own packaged pin file, ready to be restored over. + struct RestoreFixture { + dir: PathBuf, + pins: PathBuf, + manifest: LemonadeInstallManifest, + sha256: String, + } + + impl RestoreFixture { + fn new(tag: &str, packaged: &str, current: &str) -> Self { + let dir = scratch_dir(tag); + let root = dir.join("lemonade"); + let runtime_dir = runtime_dir_in(&root); + fs::create_dir_all(runtime_dir.join("resources")).unwrap(); + let pins = runtime_dir.join("resources/backend_versions.json"); + fs::write(&pins, current).unwrap(); + let source = dir.join("source-archive"); + write_embeddable_with_pins(&source, packaged); + let sha256 = format!("{:x}", Sha256::digest(fs::read(&source).unwrap())); + let manifest = LemonadeInstallManifest { + env_id: "test".to_owned(), + version: "11.5.1".to_owned(), + runtime_dir: runtime_dir.clone(), + lemond: lemond_path_in(&runtime_dir), + lemonade: lemonade_path_in(&runtime_dir), + backend_recipe: LLAMACPP_RECIPE.to_owned(), + backend_name: ROCM_BACKEND_NAME.to_owned(), + installed_at_unix_ms: 0, + }; + cache_archive(&root, &source); + Self { + dir, + pins, + manifest, + sha256, + } + } + + fn restore(&self) -> Result> { + restore_packaged_backend_versions_with( + &self.manifest, + "resources/backend_versions.json", + &self.sha256, + || {}, + ) + } + } + /// [`restore_over`] with the pin file the archive ships given too. fn restore_over_packaged( tag: &str, packaged: &str, current: &str, ) -> (Option, String) { - let dir = scratch_dir(tag); - let root = dir.join("lemonade"); - let runtime_dir = runtime_dir_in(&root); - fs::create_dir_all(runtime_dir.join("resources")).unwrap(); - let pins = runtime_dir.join("resources/backend_versions.json"); - fs::write(&pins, current).unwrap(); - let source = dir.join("source-archive"); - write_embeddable_with_pins(&source, packaged); - let sha256 = format!("{:x}", Sha256::digest(fs::read(&source).unwrap())); - let manifest = LemonadeInstallManifest { - env_id: "test".to_owned(), - version: "11.5.1".to_owned(), - runtime_dir: runtime_dir.clone(), - lemond: lemond_path_in(&runtime_dir), - lemonade: lemonade_path_in(&runtime_dir), - backend_recipe: LLAMACPP_RECIPE.to_owned(), - backend_name: ROCM_BACKEND_NAME.to_owned(), - installed_at_unix_ms: 0, - }; - cache_archive(&root, &source); - let backup = restore_packaged_backend_versions_with( - &manifest, - "resources/backend_versions.json", - &sha256, - || {}, - ) - .unwrap(); - let after = fs::read_to_string(&pins).unwrap(); + let fixture = RestoreFixture::new(tag, packaged, current); + let backup = fixture.restore().unwrap(); + let after = fs::read_to_string(&fixture.pins).unwrap(); // The backup, if any, still has to be able to put the file back. if let Some(backup) = &backup { assert!(backup.previous.is_some()); @@ -1373,6 +1420,124 @@ mod tests { fs::remove_dir_all(&dir).ok(); } + /// Restoring the packaged pins never writes through a file another writer + /// already staged next to `backend_versions.json`. The occupied names are + /// the ones the retired clock-and-pid temp naming would have picked + /// (`backend_versions.pins.tmp--`), so a restore that still + /// used one would truncate a foreign file and rename it into place. + /// The restore tests above would pass for either writer; the foreign files + /// are what tell them apart. + #[test] + fn restoring_the_packaged_pins_never_reuses_another_writers_temp_file() { + let drifted = r#"{"therock":{"version":"7.14.1"},"llamacpp":{"rocm-stable":"b11783"}}"#; + let fixture = RestoreFixture::new( + "restore-foreign-temp", + r#"{"therock":{"version":"7.13.0"},"llamacpp":{"rocm-stable":"b9752"}}"#, + drifted, + ); + + // Each attempt starts from the drifted file again, since a restore of + // an already restored file writes nothing. + let mut restored = None; + assert_write_spares_foreign_temp_files( + |millis| retired_pin_temp_path(&fixture.pins, millis), + || { + fs::write(&fixture.pins, drifted)?; + restored = fixture.restore()?; + Ok(()) + }, + ) + .unwrap(); + + let after = fs::read_to_string(&fixture.pins).unwrap(); + let value: serde_json::Value = serde_json::from_str(&after).unwrap(); + assert_eq!(value["therock"]["version"], "7.13.0", "{after}"); + assert_eq!(value["llamacpp"]["rocm-stable"], "b9752", "{after}"); + assert!(restored.is_some(), "a drifted pin file must be restored"); + assert_no_pin_temp_files(fixture.pins.parent().unwrap()); + fs::remove_dir_all(&fixture.dir).ok(); + } + + /// Putting a backup back spares foreign staged files for the same reason. + #[test] + fn putting_a_pin_file_backup_back_never_reuses_another_writers_temp_file() { + let dir = scratch_dir("pin-backup-foreign-temp"); + let pins = dir.join("backend_versions.json"); + let previous = br#"{"therock":{"version":"7.14.1"},"llamacpp":{"rocm-stable":"b11783"}}"#; + fs::write(&pins, br#"{"therock":{"version":"7.13.0"}}"#).unwrap(); + + assert_write_spares_foreign_temp_files( + |millis| retired_pin_temp_path(&pins, millis), + || PinFileBackup::new(pins.clone(), Some(previous.to_vec())).put_back(), + ) + .unwrap(); + + assert_eq!(fs::read(&pins).unwrap(), previous); + assert_no_pin_temp_files(&dir); + fs::remove_dir_all(&dir).ok(); + } + + /// A put-back onto a runtime tree that has been removed fails, as it did + /// before the shared atomic writer, and recreates nothing: the shared + /// writer would build `resources/` just to hold the pin file. The failure + /// message and the absent directory are asserted together, since the + /// warning the caller prints says the previous file could not be put back. + #[test] + fn putting_a_pin_file_backup_back_into_a_removed_tree_fails_and_recreates_nothing() { + let dir = scratch_dir("pin-backup-removed-tree"); + let resources = dir.join("runtime").join("resources"); + let pins = resources.join("backend_versions.json"); + + let error = PinFileBackup::new(pins, Some(b"{}".to_vec())) + .put_back() + .expect_err("the tree is gone, so the previous pins cannot be put back into it"); + + let text = format!("{error:#}"); + assert!( + text.contains("failed to put back") && text.contains("not recreating"), + "{text}" + ); + assert!( + !resources.exists(), + "put_back recreated {}", + resources.display() + ); + assert!(!dir.join("runtime").exists(), "put_back recreated the tree"); + fs::remove_dir_all(&dir).ok(); + } + + /// The restore has the same rule: a runtime tree removed while the archive + /// was being read is not rebuilt around a freshly written pin file. + #[test] + fn restoring_the_packaged_pins_into_a_removed_tree_fails_and_recreates_nothing() { + let fixture = RestoreFixture::new( + "restore-removed-tree", + r#"{"therock":{"version":"7.13.0"},"llamacpp":{"rocm-stable":"b9752"}}"#, + r#"{"therock":{"version":"7.14.1"},"llamacpp":{"rocm-stable":"b11783"}}"#, + ); + let runtime_dir = fixture.manifest.runtime_dir.clone(); + + let error = restore_packaged_backend_versions_with( + &fixture.manifest, + "resources/backend_versions.json", + &fixture.sha256, + || fs::remove_dir_all(&runtime_dir).unwrap(), + ) + .expect_err("the tree is gone, so the packaged pins cannot be written into it"); + + let text = format!("{error:#}"); + assert!( + text.contains("failed to restore") && text.contains("not recreating"), + "{text}" + ); + assert!( + !runtime_dir.exists(), + "the restore recreated {}", + runtime_dir.display() + ); + fs::remove_dir_all(&fixture.dir).ok(); + } + #[test] fn lemonade_root_uses_requested_engine_root() { let paths = AppPaths { diff --git a/engines/lemonade/src/test_support.rs b/engines/lemonade/src/test_support.rs index 7b79057c86..33ff89eb41 100644 --- a/engines/lemonade/src/test_support.rs +++ b/engines/lemonade/src/test_support.rs @@ -36,6 +36,15 @@ pub(crate) fn scratch_dir(tag: &str) -> PathBuf { dir } +/// The temp name the retired `rocm_core::write_atomic` picked when it +/// replaced `backend_versions.json`: `with_extension("pins.tmp--")` +/// swapped the `json` extension for `pins`, so the name was +/// `backend_versions.pins.tmp--`, with this process's pid. It was +/// fixed by the clock and the pid alone, so a test can occupy it ahead of time. +pub(crate) fn retired_pin_temp_path(pin_file: &Path, millis: u128) -> PathBuf { + pin_file.with_extension(format!("pins.tmp-{millis}-{}", std::process::id())) +} + mod tests { use super::*; From f752a2906e47e52a61c8efa333b413981e6d2111 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 16:40:08 +0000 Subject: [PATCH 025/102] refactor(rocm-core): one PATH lookup for the summary and the examination stacks (#953) * test(rocm-core): pin both PATH lookups' answers before they are merged The summary stack's tool_on_path and the examination stack's `which` answer the same question, is this program installed, and are about to become one. Pin what each returns for the same input first, so the change that merges them shows exactly which answers it moves. They agree on a PATH of plain directories, on an unset PATH and on a directory that has the program's name. They differ on an empty PATH entry (tool_on_path reads it as the current directory, `which` skips it) and on a PATH that is not valid Unicode (tool_on_path still searches it, `which` reads PATH with env::var and finds nothing). On Windows they also differ on PATHEXT spellings outside .exe, .bat and .cmd, on a quoted PATH entry holding a semicolon, and on the System32 location of cmd. Each case runs in a child copy of the test binary with its own PATH, PATHEXT and working directory, because neither lookup can be pointed elsewhere on a test thread without disturbing the tests that spawn programs by name. The Windows cases run only on the Windows job. Signed-off-by: Jussi Elo * refactor(rocm-core): one PATH lookup for the summary and the examination stacks The summary stack's tool_on_path (host_gpu/capture.rs) and the examination stack's `which` (probe_exec.rs) answered the same question and disagreed on an empty PATH entry, on a PATH that is not valid Unicode and, on Windows, on PATHEXT spellings, quoted PATH entries and the System32 locations. `which` now has the behaviour tool_on_path had, and tool_on_path is gone. tool_path_candidates moved to probe_exec.rs with it, because the capture runner and the Windows diagnostics capture start each spelling in turn; they import it from there. detect_wsl_summary asks `which` for rocminfo and cargo. An empty PATH entry is skipped rather than read as the current directory. That keeps the examination's answer, so a file named sudo or usermod in the directory the CLI runs from still does not read as installed, and moves only the summary's WSL answer. A program the CLI later starts by bare name can still resolve from the current directory through the operating system; the lookup no longer agrees with that, and its doc comment says so. Moves relative to the two old copies: the examination now searches a PATH that is not valid Unicode (it used to find nothing in any directory), follows every PATHEXT entry in both cases, keeps a quoted PATH entry holding a semicolon whole, and finds cmd, pnputil, powershell and pwsh at their System32 locations (no `which` caller asks for those four). The summary stops finding a file in the current directory through an empty entry. The Windows spellings are computed by path_candidates, which takes the platform, PATHEXT and the system root as arguments, and the search by found_in, which takes the directories and an existence test, so both are unit-tested on every host; the cases that need the real PATH, PATHEXT and working directory run in a child copy of the test binary. The characterisation tests of the previous commit move to probe_exec.rs and now assert the one answer: the cases that differed flip to the survivor's. Signed-off-by: Jussi Elo * test(e2e): hold the host summary and the examination level on whether rocminfo is installed `rocm examine --json` on WSL reports whether rocminfo is installed twice, as summary.wsl.rocminfo and as wsl.rocminfo, and the two used to come from different PATH lookups. They share one now, so three scenarios on the simulated WSL host that examine-32 uses hold the two answers level: a rocminfo in a PATH directory is found by both (examine-35), a rocminfo only in the directory the command runs from, behind an empty PATH entry, is found by neither (examine-36), and a rocminfo in a PATH directory whose name is not valid Unicode is found by both (examine-37). The runner's own PATH entries that hold a rocminfo are left out of the planted PATH, so a runner with ROCm installed cannot be why the tool reads as found. The scenario with the empty entry first shows that the stand-in rocminfo does resolve through that entry from the working directory, so its "not installed" cannot come from a fixture that never reached it. The harness gains command_dir, the working directory for the next rocm run, consumed like command_env. Signed-off-by: Jussi Elo * docs: record the PATH lookup decision in the host-probing note The PATH lookup is the first pair of the host-probing note to converge. Record what was decided: which survived, that an empty PATH entry is skipped and what that costs, every answer that moved with the test or scenario that pins it, and that the System32 candidates stay in the shared list. The analysis of the two old copies stays as the record of what the change moved; the entry for the empty-entry question leaves the open list, and the near copy in apps/rocm stays on it. Signed-off-by: Jussi Elo * test(e2e): add the control that shows examine-36 really runs from its working directory examine-36 expects both reports to say rocminfo is not installed when the only copy is in the directory the command runs from and PATH has an empty entry. That answer would also come out if the harness ignored command_dir and started rocm somewhere else, so nothing showed the working directory was applied. examine-38 is the same machine with `.` as the first PATH entry instead of an empty one. A `.` entry is an ordinary directory, so both reports find the stand-in, and only if rocm really runs from that directory. The two scenarios now differ in nothing but that entry. With command_dir ignored in the harness, examine-38 fails with false/false where true/true is expected; with the lookup keeping empty entries, examine-36 fails with true/true. The planting is shared between the two Given steps, and the testing guide and the host-probing note list the new scenario. Signed-off-by: Jussi Elo * docs(rocm-core): re-wrap the probe list in the host_gpu module doc The line that named the child modules had grown to about 140 characters when capture lost its mention of the PATH lookup. Wrap it like the neighbouring lines. Signed-off-by: Jussi Elo * test(e2e): resolve the runner's relative PATH entries from the harness directory before filtering them The scenarios that plant a rocminfo leave out the runner's own PATH entries that hold one, but they checked an inherited relative entry from the harness's directory while the rocm child runs from the scenario's command_dir. A runner whose PATH contains `.` and whose harness directory has no rocminfo kept that entry, and the child then read it as the planted work directory and found the stand-in, so examine-36 failed with true/true despite a correct lookup. planted_path_entries now makes each non-empty inherited entry absolute against the harness's own working directory, checks it there and keeps it in that resolved form, drops inherited empty entries, and leaves the fixture's own entries exactly as written, since the empty entry and `.` are what two scenarios are about. It is a pure function in the harness library with a row per case, so it runs on every host. With the old filter and `.` first on the harness PATH, examine-36 fails with true/true; with this change examine-35 to examine-38 pass both with and without it. Signed-off-by: Jussi Elo * test(e2e): leave a bare rocm name for PATH to resolve when a scenario moves the child When a scenario sets command_dir, the harness made the rocm binary absolute with canonicalize. A bare name such as rocm is found on PATH, but canonicalize also succeeds for a file called rocm in the harness's own directory, so the harness would have run that local file instead of the program PATH resolves. Only a path is pinned now: one that is absolute or contains a separator (./rocm, bin/rocm, ../x/rocm). A bare name is returned unchanged without being looked up on disk. The decision is a pure function in the harness library with a table test, including a canonicalize that succeeds for everything to stand for a local rocm file; making it canonicalise every binary again turns that test red. Signed-off-by: Jussi Elo * test(e2e): drop a runner PATH entry only when it holds a rocminfo file, as the lookup does The harness leaves out the runner's PATH entries that hold a rocminfo, so a runner with ROCm installed cannot be why the tool reads as found. It tested that with exists(), which a directory named rocminfo also satisfies, while the shared lookup in rocm-core requires a regular file. Such an entry would have been dropped without ever making rocminfo resolve. dir_holds_tool now uses is_file(), in the harness library next to planted_path_entries, and the step uses it. A test plants a directory called rocminfo in one entry and a file in another and checks the first is kept and the second dropped; going back to exists() turns it red. No other exists() or is_dir() check was added by the harness changes of this branch. Signed-off-by: Jussi Elo * fix(rocm-core): test an absolute PATH candidate once, so the lookup agrees with what the runner can start found_in joined every candidate onto each PATH directory, so an absolute candidate (the System32 location of cmd, pnputil or powershell on Windows) was only tried inside the loop over directories. which("cmd") was therefore false when PATH was unset or all empty, while the capture runner would still start cmd at its fixed location. An absolute candidate names its own location, so found_in now tests it once as it is, after the directories, whatever dirs holds, and which no longer returns early when PATH is unset. A pure test pins that an absolute candidate is asked about once, with no directories, one or two, and another that a relative candidate beside an absolute one is still joined onto the directories. On Linux no candidate is absolute unless the caller passes an absolute path, and every which call site passes a literal bare name, so nothing changes there. The doc comment on which also says that a file in the starting directory does not read as installed through an empty entry, which is what the lookup does; an explicit `.` or other relative entry is an ordinary directory and is searched. The host-probing note states the new System32 behaviour, says that guarding the runners against an empty PATH entry (they start a bare name through Command, so the operating system resolves it) belongs to the subprocess-runners pair and is not changed with the lookup, and lists that among the open questions. Signed-off-by: Jussi Elo * docs: say which PATHEXT spellings the framework probe cannot run and fix a scenario comment framework.rs: which follows PATHEXT on Windows, so python or llama-cli can read as found through a spelling such as .PS1 or .PY that the probe, which starts the bare name through Command, cannot run. Name that at the two probes that call which for those programs. examine.feature: the comment above examine-35 said three scenarios and four were added. Signed-off-by: Jussi Elo * docs: fix a cited test name in the host-probing note and record how dotted names are looked up on Windows The note cited found_in_asks_about_an_absolute_candidate_as_it_is_whatever_the_directory, which does not exist; the test is found_in_asks_about_an_absolute_candidate_once_as_it_is_whatever_the_directories. Fix the name. Add the case the shared lookup moved for a Windows name that already has an extension (python3.11, tool.py, wsl.exe). The examination's old copy also tried .exe, .bat and .cmd after such a name, as read from the deleted function; the summary's copy and the shared lookup try it as given only. No which call site passes such a name except wsl.exe, which is found as given, so no answer moves today. The System32 row also no longer lists pwsh, which has no fixed location. Signed-off-by: Jussi Elo * test(rocm-core): make the System32 candidate test non-vacuous, check the note's cited tests exist, and correct the which summary candidates_on_windows_add_the_fixed_system32_locations asserted pwsh.exe and powershell.exe with a PATHEXT that contains .EXE, which adds those spellings by itself, so deleting the arms of the fixed list changed nothing. It now uses a PATHEXT without .EXE, so they can only come from the list; deleting either goes red. The host-probing note cites tests by bare name, which the link checker does not see, and one name was wrong. the_host_probing_note_cites_only_tests_that_exist reads the note and requires every cited which_, candidates_, found_in_ and path_dirs_ name to be a function in probe_exec.rs; a misspelled name goes red. candidates_on_windows_for_a_name_with_an_extension_are_the_name_alone also pins python3.11: PATHEXT is not tried after a dotted name, which the doc comment on which now says, with the fact that no caller relied on the old copy doing so. The summary line of which no longer says it only resolves on PATH, and the doc stops listing pwsh among the names found at a System32 location. Signed-off-by: Jussi Elo * test(rocm-core): drop the test that read the host-probing note from inside the crate the_host_probing_note_cites_only_tests_that_exist read docs/host-probing.md from rocm-core. The pre-push narrowing rule in xtask does not know that read, so it rejects a rocm-core test that reads a file outside the crate (every_cross_crate_read_is_covered_by_the_widening_rule), and widening the rule for a citation tripwire is not worth it. The corrected test name stays in the note and the other changes of the previous commit are untouched; the names the note cites were checked against the sources by hand instead. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit f3c08f12df1398abd5c51e0ec2784bac9987d34b) Signed-off-by: juhovainio --- crates/rocm-core/src/examine/framework.rs | 7 + crates/rocm-core/src/host_gpu.rs | 14 +- crates/rocm-core/src/host_gpu/capture.rs | 61 +- crates/rocm-core/src/host_gpu/diagnostics.rs | 4 +- crates/rocm-core/src/host_gpu/wsl.rs | 9 +- crates/rocm-core/src/probe_exec.rs | 603 +++++++++++++++++- docs/architecture.md | 6 +- docs/host-probing.md | 32 +- docs/testing.md | 6 + tests/e2e-cucumber/features/examine.feature | 50 ++ tests/e2e-cucumber/src/binary_path.rs | 100 +++ tests/e2e-cucumber/src/lib.rs | 2 + tests/e2e-cucumber/src/planted_path.rs | 179 ++++++ tests/e2e-cucumber/tests/e2e.rs | 22 +- tests/e2e-cucumber/tests/e2e/examine_steps.rs | 175 +++++ 15 files changed, 1169 insertions(+), 101 deletions(-) create mode 100644 tests/e2e-cucumber/src/binary_path.rs create mode 100644 tests/e2e-cucumber/src/planted_path.rs diff --git a/crates/rocm-core/src/examine/framework.rs b/crates/rocm-core/src/examine/framework.rs index 420f7b0755..6817a16a9f 100644 --- a/crates/rocm-core/src/examine/framework.rs +++ b/crates/rocm-core/src/examine/framework.rs @@ -125,6 +125,9 @@ fn runtime_library_path_env( } fn probe_pytorch_on_path(e: &mut Examination) { + // On Windows `which` also accepts any `PATHEXT` spelling (`python.PY`, `.PS1`), + // while the probe starts the bare name through `Command`, which only starts an + // `.exe`: the same caveat as in `probe_llama_cpp`. let py = if which("python") { "python" } else if which("python3") { @@ -222,6 +225,10 @@ fn record_pytorch_probe(e: &mut Examination, rc: i32, out: &str, err: &str) { } fn probe_llama_cpp(e: &mut Examination) { + // `which` follows `PATHEXT` on Windows, so a name can read as found through an + // extension this probe cannot run (`llama-cli.PS1`): the probe starts the bare + // name through `Command`, which only starts an `.exe`. The report can then + // name a binary it cannot start; the later run fails and says so. let binary = ["llama-cli", "llama-server", "main"] .into_iter() .find(|name| which(name)); diff --git a/crates/rocm-core/src/host_gpu.rs b/crates/rocm-core/src/host_gpu.rs index fa08dd40c7..c1854ac0d8 100644 --- a/crates/rocm-core/src/host_gpu.rs +++ b/crates/rocm-core/src/host_gpu.rs @@ -12,13 +12,13 @@ //! [`crate::confirm`] and is re-exported here too, so the //! `crate::host_gpu::interactive_terminal` path keeps resolving. //! -//! The probes live in child modules: `capture` (optional-command capture and the -//! tool-on-`PATH` lookup), `os_facts` (kernel, distro, CPU and RAM), `linux` -//! (sysfs, KFD and DRM), `gpu_visibility` (which GPUs this process may use), -//! `windows_inventory` (the Windows display-adapter inventory), `wsl` (WSL -//! detection and the Windows host driver as seen from the guest), `render` (the -//! `rocm examine` text of an `ExamineSummary`), `diagnostics` (the GPU -//! detection diagnostics text) and `engine_default` (the default serve engine). +//! The probes live in child modules: `capture` (optional-command capture), +//! `os_facts` (kernel, distro, CPU and RAM), `linux` (sysfs, KFD and DRM), +//! `gpu_visibility` (which GPUs this process may use), `windows_inventory` (the +//! Windows display-adapter inventory), `wsl` (WSL detection and the Windows host +//! driver as seen from the guest), `render` (the `rocm examine` text of an +//! `ExamineSummary`), `diagnostics` (the GPU detection diagnostics text) and +//! `engine_default` (the default serve engine). //! //! The `examine` child modules and `fix.rs` still reach several items here //! through `lib.rs`'s flat `crate::` re-export paths rather than diff --git a/crates/rocm-core/src/host_gpu/capture.rs b/crates/rocm-core/src/host_gpu/capture.rs index fa2c1c68fe..7aef97001d 100644 --- a/crates/rocm-core/src/host_gpu/capture.rs +++ b/crates/rocm-core/src/host_gpu/capture.rs @@ -5,11 +5,12 @@ //! Optional-command capture for the host probes. //! //! Runs a tool that may not be installed under a short timeout and keeps its -//! stdout only on success, resolving the program the way the platform does -//! (`PATHEXT` and the fixed Windows system locations included). A missing, -//! slow or failing tool reads as `None`, never as an error. +//! stdout only on success, trying each spelling of the program that +//! [`crate::probe_exec::tool_path_candidates`] gives (`PATHEXT` and the fixed +//! Windows system locations included). A missing, slow or failing tool reads as +//! `None`, never as an error. -use crate::runtime_is_windows; +use crate::probe_exec::tool_path_candidates; use std::io::Read; use std::path::Path; use std::process::{Command, Stdio}; @@ -111,55 +112,3 @@ fn debug_command_capture_failure(program: &Path, stage: &str, detail: &str) { program.display() ); } - -pub(super) fn tool_on_path(program: &str) -> bool { - std::env::var_os("PATH").is_some_and(|path| { - std::env::split_paths(&path).any(|dir| { - tool_path_candidates(program) - .into_iter() - .any(|name| dir.join(name).is_file()) - }) - }) -} - -pub(super) fn tool_path_candidates(program: &str) -> Vec { - let path = Path::new(program); - if path.extension().is_some() || !runtime_is_windows() { - return vec![program.to_owned()]; - } - let mut names = Vec::new(); - names.push(program.to_owned()); - let pathext = std::env::var("PATHEXT").unwrap_or_else(|_| ".COM;.EXE;.BAT;.CMD".to_owned()); - for ext in pathext - .split(';') - .map(str::trim) - .filter(|ext| !ext.is_empty()) - { - names.push(format!("{program}{ext}")); - names.push(format!("{program}{}", ext.to_ascii_lowercase())); - } - names.extend(windows_absolute_tool_candidates(program)); - names.sort(); - names.dedup(); - names -} - -fn windows_absolute_tool_candidates(program: &str) -> Vec { - if !runtime_is_windows() { - return Vec::new(); - } - let program = program.trim().to_ascii_lowercase(); - let system_root = std::env::var("SystemRoot") - .or_else(|_| std::env::var("WINDIR")) - .unwrap_or_else(|_| r"C:\Windows".to_owned()); - match program.as_str() { - "cmd" | "cmd.exe" => vec![format!(r"{system_root}\System32\cmd.exe")], - "pnputil" | "pnputil.exe" => vec![format!(r"{system_root}\System32\pnputil.exe")], - "powershell" | "powershell.exe" => vec![ - format!(r"{system_root}\System32\WindowsPowerShell\v1.0\powershell.exe"), - "powershell.exe".to_owned(), - ], - "pwsh" | "pwsh.exe" => vec!["pwsh.exe".to_owned()], - _ => Vec::new(), - } -} diff --git a/crates/rocm-core/src/host_gpu/diagnostics.rs b/crates/rocm-core/src/host_gpu/diagnostics.rs index 1a456f361a..bc190d2f24 100644 --- a/crates/rocm-core/src/host_gpu/diagnostics.rs +++ b/crates/rocm-core/src/host_gpu/diagnostics.rs @@ -5,8 +5,6 @@ //! The text `rocm install sdk` prints when it cannot resolve a GPU: what the //! detection saw, and on Windows the raw output of each inventory query. -#[cfg(windows)] -use super::capture::tool_path_candidates; use super::detect_host_gpu_summary; use super::linux::{detect_linux_primary_gpu_name, detect_linux_sysfs_gfx_target}; #[cfg(windows)] @@ -21,6 +19,8 @@ use crate::gfx_target::{ amd_pci_device_id_from_pnp_id, gfx_target_from_amd_marketing_name, gfx_target_from_amd_pci_device_id, }; +#[cfg(windows)] +use crate::probe_exec::tool_path_candidates; use crate::{runtime_is_linux, runtime_is_windows, runtime_os_name}; #[cfg(windows)] use std::io::Read; diff --git a/crates/rocm-core/src/host_gpu/wsl.rs b/crates/rocm-core/src/host_gpu/wsl.rs index 098d99ef22..c5900a039a 100644 --- a/crates/rocm-core/src/host_gpu/wsl.rs +++ b/crates/rocm-core/src/host_gpu/wsl.rs @@ -5,9 +5,7 @@ //! WSL detection and the Windows host driver as seen from the guest: the WSL2 //! signals, the librocdxg chain, and the local and WSL host-driver probes. -use super::capture::{ - capture_optional_command, capture_optional_command_with_timeout, tool_on_path, -}; +use super::capture::{capture_optional_command, capture_optional_command_with_timeout}; use super::windows_inventory::{ WINDOWS_INVENTORY_QUERY_TIMEOUT, WINDOWS_VIDEO_CONTROLLER_INVENTORY_SCRIPT, WindowsExamineInventory, detect_windows_examine_inventory_with_driver_facts, @@ -15,6 +13,7 @@ use super::windows_inventory::{ }; use super::{DriverSummary, WslSummary}; use crate::gfx_target::parse_windows_display_gfx_target; +use crate::probe_exec::which; use crate::{discover_rocm_installs, host_path, runtime_is_linux, runtime_is_windows}; use std::fs; use std::path::{Path, PathBuf}; @@ -372,8 +371,8 @@ pub(crate) fn detect_wsl_summary() -> Option { // The roots are already re-rooted through `host_path`, so this existence check // takes the path as given. let librocdxg_beside_runtime = librocdxg_beside_runtime(&roots, Path::exists); - let rocminfo = tool_on_path("rocminfo"); - let cargo = tool_on_path("cargo"); + let rocminfo = which("rocminfo"); + let cargo = which("cargo"); let missing = missing_plumbing( dxg_device, dxcore, diff --git a/crates/rocm-core/src/probe_exec.rs b/crates/rocm-core/src/probe_exec.rs index 86f807cfa4..37dca2b437 100644 --- a/crates/rocm-core/src/probe_exec.rs +++ b/crates/rocm-core/src/probe_exec.rs @@ -11,9 +11,9 @@ //! importing the module that owns the `Examination` document. use crate::runtime_is_windows; -use std::ffi::OsString; +use std::ffi::{OsStr, OsString}; use std::io::Read; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::thread; use std::time::{Duration, Instant}; @@ -134,27 +134,140 @@ pub(crate) fn read_host_text(path: &str) -> String { std::fs::read_to_string(crate::host_path(path)).unwrap_or_default() } -/// Whether `program` resolves on `PATH` (best-effort, no execution). +/// Whether `program` is installed, by `PATH` lookup plus a few fixed Windows +/// locations (best-effort, no execution). +/// +/// The one PATH lookup of the host probes: the examination stack's capability +/// checks and the summary stack's WSL facts both ask it, so a tool reads as +/// installed or not the same way in every part of `rocm examine`. It tries +/// each spelling [`tool_path_candidates`] gives, in each directory of `PATH`, +/// and answers `true` for the first one that is a file. +/// +/// - An empty `PATH` entry is skipped, not read as the current directory as a +/// shell or `execvp` would. A file named like the tool in the directory the +/// CLI was started from therefore does not read as the tool being installed +/// through an empty entry, which matters for the checks that gate a +/// privileged step (`sudo`, `usermod`). An explicit `.` or other relative +/// entry is an ordinary directory and is searched. The cost is that a +/// program the CLI later starts by bare name can still resolve from the +/// current directory when `PATH` has an empty entry, because the operating +/// system resolves it, not this lookup. +/// - A `PATH` that is not valid Unicode is searched: it is split as an +/// `OsString`, and the entries that are not Unicode are looked in as they are. +/// - The directories are split the way [`std::env::split_paths`] splits them on +/// this platform, so on Windows an entry may be quoted to hold a `;`. +/// - On Windows a program without an extension is tried with every `PATHEXT` +/// entry in both cases. A name that already has an extension (`python3.11`, +/// `wsl.exe`) is tried as given and nothing else; the examination's old copy +/// also tried `.exe`, `.bat` and `.cmd` after it, and no caller passes a +/// dotted name that relied on that (`wsl.exe` is found as given). +/// - On Windows `cmd`, `pnputil` and `powershell` are also found at their fixed +/// System32 locations (`pwsh` has no fixed location: its only extra candidate +/// is the bare `pwsh.exe`). An absolute candidate is tested once as it is, not +/// joined onto each `PATH` directory, so it is found whether or not `PATH` is +/// set, on `PATH` or not: the lookup then agrees with what the capture runner, +/// which starts those locations, can run. No caller asks for those names +/// today; they stay in the shared candidate list because the runner needs +/// them and one list cannot drift from the other. pub(crate) fn which(program: &str) -> bool { - let Ok(path) = std::env::var("PATH") else { - return false; - }; - let (sep, exts): (char, &[&str]) = if runtime_is_windows() { - (';', &[".exe", ".bat", ".cmd", ""]) - } else { - (':', &[""]) - }; - for dir in path.split(sep) { - if dir.is_empty() { - continue; - } - for ext in exts { - if Path::new(dir).join(format!("{program}{ext}")).is_file() { - return true; - } - } + let dirs = std::env::var_os("PATH") + .map(|path| path_dirs(&path)) + .unwrap_or_default(); + found_in(&dirs, &tool_path_candidates(program), Path::is_file) +} + +/// The directories of a `PATH` value, empty entries dropped. +fn path_dirs(path: &OsStr) -> Vec { + std::env::split_paths(path) + .filter(|dir| !dir.as_os_str().is_empty()) + .collect() +} + +/// Whether any of `candidates` is a file according to `is_file`. +/// +/// A relative candidate is joined onto each of `dirs`: directories are tried in +/// order, and within a directory the candidates in order. An absolute candidate +/// names its own location, so it is tested once as it is, after the directories, +/// and does not depend on `dirs` at all. `is_file` is injected so a test can say +/// which files exist without a file system. +fn found_in(dirs: &[PathBuf], candidates: &[String], is_file: impl Fn(&Path) -> bool) -> bool { + let in_a_dir = dirs.iter().any(|dir| { + candidates + .iter() + .filter(|candidate| !Path::new(candidate).is_absolute()) + .any(|candidate| is_file(&dir.join(candidate))) + }); + in_a_dir + || candidates + .iter() + .map(Path::new) + .filter(|candidate| candidate.is_absolute()) + .any(is_file) +} + +/// The spellings of `program` to try, in the order to try them. +/// +/// On Linux, just `program`. On Windows a program with no extension is also +/// spelled with each `PATHEXT` entry, in both cases, and `cmd`, `pnputil`, +/// `powershell` and `pwsh` add their fixed locations. Used by [`which`] and by +/// the capture runner, which starts each spelling in turn. +pub(crate) fn tool_path_candidates(program: &str) -> Vec { + path_candidates( + program, + runtime_is_windows(), + std::env::var("PATHEXT").ok().as_deref(), + &system_root_from_env(), + ) +} + +/// `SystemRoot`, else `WINDIR`, else `C:\Windows`. +fn system_root_from_env() -> String { + std::env::var("SystemRoot") + .or_else(|_| std::env::var("WINDIR")) + .unwrap_or_else(|_| r"C:\Windows".to_owned()) +} + +/// [`tool_path_candidates`] with the platform, `PATHEXT` (`None` when unset or +/// not Unicode) and system root passed in, so the Windows spellings can be +/// checked on every host. +fn path_candidates( + program: &str, + windows: bool, + pathext: Option<&str>, + system_root: &str, +) -> Vec { + if Path::new(program).extension().is_some() || !windows { + return vec![program.to_owned()]; + } + let mut names = vec![program.to_owned()]; + let pathext = pathext.unwrap_or(".COM;.EXE;.BAT;.CMD"); + for ext in pathext + .split(';') + .map(str::trim) + .filter(|ext| !ext.is_empty()) + { + names.push(format!("{program}{ext}")); + names.push(format!("{program}{}", ext.to_ascii_lowercase())); + } + names.extend(windows_absolute_tool_candidates(program, system_root)); + names.sort(); + names.dedup(); + names +} + +/// The fixed locations of the Windows tools the host probes run. +fn windows_absolute_tool_candidates(program: &str, system_root: &str) -> Vec { + let program = program.trim().to_ascii_lowercase(); + match program.as_str() { + "cmd" | "cmd.exe" => vec![format!(r"{system_root}\System32\cmd.exe")], + "pnputil" | "pnputil.exe" => vec![format!(r"{system_root}\System32\pnputil.exe")], + "powershell" | "powershell.exe" => vec![ + format!(r"{system_root}\System32\WindowsPowerShell\v1.0\powershell.exe"), + "powershell.exe".to_owned(), + ], + "pwsh" | "pwsh.exe" => vec!["pwsh.exe".to_owned()], + _ => Vec::new(), } - false } pub(crate) const SHORT: Duration = Duration::from_secs(5); @@ -215,3 +328,451 @@ mod tests { assert_eq!(truncate_to_chars("short".to_owned(), 4000), "short"); } } + +/// The PATH lookup ([`which`]) and the spellings it tries +/// ([`tool_path_candidates`]), which the summary stack and the examination stack +/// share. Before they were one, each stack had its own copy and the copies +/// answered differently for an empty `PATH` entry, a `PATH` that is not Unicode +/// and the Windows spellings (see `docs/host-probing.md`, "PATH lookup"); the +/// tests here pin the one answer they now give. +/// +/// The pure functions are tested on every host with the platform, `PATHEXT` and +/// the files that exist passed in. The lookup as a whole reads the process's own +/// `PATH`, and one case needs a different working directory, which a test thread +/// cannot change without disturbing every other test in the process (they spawn +/// children by bare name and call `which` themselves). So those cases run in a +/// child: this test binary again, started with the `PATH`, `PATHEXT` and working +/// directory the case needs and asked to run only +/// [`which_child_reports_its_answer`]. In a normal run that test finds no +/// program to ask about and passes without doing anything. +#[cfg(test)] +mod path_lookup_tests { + use super::{found_in, path_candidates, path_dirs, which}; + use std::cell::RefCell; + use std::ffi::{OsStr, OsString}; + use std::path::{Path, PathBuf}; + use std::process::Command; + + const PROGRAM_ENV: &str = "ROCM_CORE_TEST_WHICH_PROGRAM"; + const ANSWER_PREFIX: &str = "which-answer:"; + /// A name no host has installed, so only the planted file can answer for it. + const PROGRAM: &str = "rocm-core-which-tool"; + + fn names(candidates: &[&str]) -> Vec { + candidates.iter().map(|name| (*name).to_owned()).collect() + } + + #[test] + fn candidates_off_windows_are_the_program_alone_whatever_pathext_says() { + assert_eq!( + path_candidates("tool", false, Some(".COM;.EXE"), r"C:\Windows"), + names(&["tool"]) + ); + } + + #[test] + fn candidates_on_windows_for_a_name_with_an_extension_are_the_name_alone() { + // The early return also keeps the System32 spelling of `cmd.exe` out of + // the list: only the extension-less `cmd` reaches it. + assert_eq!( + path_candidates("cmd.exe", true, Some(".COM;.EXE"), r"C:\Windows"), + names(&["cmd.exe"]) + ); + assert_eq!( + path_candidates("tool.ps1", true, Some(".PS1"), r"C:\Windows"), + names(&["tool.ps1"]) + ); + // A dotted name is not a bare name with an unknown extension: PATHEXT is + // not tried after it, so `python3.11` is looked for as `python3.11` only. + // The examination's old copy also tried `.exe`, `.bat` and `.cmd`; no + // caller passes such a name (they are all bare, or `wsl.exe`). + assert_eq!( + path_candidates( + "python3.11", + true, + Some(".COM;.EXE;.BAT;.CMD"), + r"C:\Windows" + ), + names(&["python3.11"]) + ); + } + + #[test] + fn candidates_on_windows_try_every_pathext_entry_in_both_cases() { + // Sorted as bytes, so the upper-case spelling of an extension comes + // before its lower-case one, and the bare name before both. `.PS1` is + // outside the old examination list (`.exe`, `.bat`, `.cmd`). + assert_eq!( + path_candidates("tool", true, Some(".COM;.EXE;.PS1"), r"C:\Windows"), + names(&[ + "tool", "tool.COM", "tool.EXE", "tool.PS1", "tool.com", "tool.exe", "tool.ps1", + ]) + ); + } + + #[test] + fn candidates_on_windows_trim_pathext_entries_and_skip_empty_ones() { + assert_eq!( + path_candidates("tool", true, Some(" .BAT ;; ;.CMD;"), r"C:\Windows"), + names(&["tool", "tool.BAT", "tool.CMD", "tool.bat", "tool.cmd"]) + ); + } + + #[test] + fn candidates_on_windows_fall_back_to_the_default_pathext_when_it_is_unset() { + assert_eq!( + path_candidates("tool", true, None, r"C:\Windows"), + names(&[ + "tool", "tool.BAT", "tool.CMD", "tool.COM", "tool.EXE", "tool.bat", "tool.cmd", + "tool.com", "tool.exe", + ]) + ); + } + + #[test] + fn candidates_on_windows_add_the_fixed_system32_locations() { + let root = r"C:\Win"; + // A `PATHEXT` without `.EXE`, so `powershell.exe` and `pwsh.exe` can only + // come from the fixed list: with `.EXE` in it the expansion adds them + // anyway and these assertions would not notice the list losing them. + let with = |program: &str| path_candidates(program, true, Some(".BAT"), root); + + assert!( + with("cmd").contains(&r"C:\Win\System32\cmd.exe".to_owned()), + "{:?}", + with("cmd") + ); + assert!(with("pnputil").contains(&r"C:\Win\System32\pnputil.exe".to_owned())); + let powershell = with("powershell"); + assert!( + powershell + .contains(&r"C:\Win\System32\WindowsPowerShell\v1.0\powershell.exe".to_owned()) + ); + assert!(powershell.contains(&"powershell.exe".to_owned())); + assert!(with("pwsh").contains(&"pwsh.exe".to_owned())); + // Matched without regard to case, like the file system. + assert!(with("CMD").contains(&r"C:\Win\System32\cmd.exe".to_owned())); + // The location comes first when it sorts first, which is the order the + // capture runner starts them in. + assert_eq!( + with("cmd").first().map(String::as_str), + Some(r"C:\Win\System32\cmd.exe") + ); + } + + #[test] + fn candidates_on_windows_for_other_tools_have_no_fixed_location() { + assert_eq!( + path_candidates("python", true, Some(".EXE"), r"C:\Win"), + names(&["python", "python.EXE", "python.exe"]) + ); + } + + #[test] + fn candidates_off_windows_never_name_a_system32_location() { + assert_eq!( + path_candidates("cmd", false, Some(".EXE"), r"C:\Win"), + names(&["cmd"]) + ); + } + + #[test] + fn path_dirs_drop_every_empty_entry() { + let path = std::env::join_paths([ + OsStr::new(""), + OsStr::new("first"), + OsStr::new(""), + OsStr::new("second"), + OsStr::new(""), + ]) + .expect("no entry holds a separator"); + assert_eq!( + path_dirs(&path), + vec![PathBuf::from("first"), PathBuf::from("second")] + ); + assert!(path_dirs(OsStr::new("")).is_empty()); + } + + #[cfg(unix)] + #[test] + fn path_dirs_keep_an_entry_that_is_not_unicode() { + use std::os::unix::ffi::OsStrExt as _; + + let odd = OsStr::from_bytes(b"/odd-\xff"); + let path = std::env::join_paths([OsStr::new("/usr/bin"), odd]).expect("no separator"); + assert_eq!( + path_dirs(&path), + vec![PathBuf::from("/usr/bin"), PathBuf::from(odd)] + ); + } + + #[test] + fn found_in_asks_each_candidate_in_each_directory_in_order_and_stops_at_the_first_file() { + let dirs = [PathBuf::from("first"), PathBuf::from("second")]; + let candidates = names(&["tool", "tool.exe"]); + let asked = RefCell::new(Vec::new()); + + let found = found_in(&dirs, &candidates, |file| { + asked.borrow_mut().push(file.to_owned()); + file == Path::new("second").join("tool") + }); + + assert!(found); + assert_eq!( + asked.into_inner(), + vec![ + Path::new("first").join("tool"), + Path::new("first").join("tool.exe"), + Path::new("second").join("tool"), + ] + ); + assert!(!found_in(&dirs, &candidates, |_| false)); + assert!(!found_in(&[], &candidates, |_| true)); + } + + #[test] + fn found_in_asks_about_an_absolute_candidate_once_as_it_is_whatever_the_directories() { + // This is why a System32 location is found "without being on PATH", and + // found even when PATH is unset or all empty: it is tested on its own, + // once, not joined onto each directory. + let absolute = std::env::temp_dir().join("rocm-core-absolute-candidate"); + let candidates = [absolute.to_string_lossy().into_owned()]; + + for dirs in [ + Vec::new(), + vec![PathBuf::from("anywhere")], + vec![PathBuf::from("anywhere"), PathBuf::from("elsewhere")], + ] { + let asked = RefCell::new(Vec::new()); + let found = found_in(&dirs, &candidates, |file| { + asked.borrow_mut().push(file.to_owned()); + true + }); + // With directories listed the absolute candidate is never joined + // onto them, so the only thing asked about is the candidate itself. + assert!(found, "{} directories", dirs.len()); + assert_eq!( + asked.into_inner(), + vec![absolute.clone()], + "{} directories", + dirs.len() + ); + } + assert!(!found_in(&[], &candidates, |_| false)); + } + + #[test] + fn found_in_still_tries_relative_candidates_in_directories_beside_an_absolute_one() { + let absolute = std::env::temp_dir().join("rocm-core-absolute-candidate"); + let candidates = [ + absolute.to_string_lossy().into_owned(), + "tool.exe".to_owned(), + ]; + let dirs = [PathBuf::from("first")]; + + assert!(found_in(&dirs, &candidates, |file| file + == Path::new("first").join("tool.exe"))); + assert!(!found_in(&[], &candidates, |file| file + == Path::new("first").join("tool.exe"))); + } + + /// The child's side of [`ask_for`]: print the answer for the program named in + /// the environment. Does nothing in a normal test run. + #[test] + fn which_child_reports_its_answer() { + let Some(program) = std::env::var_os(PROGRAM_ENV) else { + return; + }; + println!("{ANSWER_PREFIX} {}", which(&program.to_string_lossy())); + } + + /// Run [`which`] for `program` in a child whose `PATH` is `path` (unset when + /// `None`), working directory is `cwd` and extra environment is `env`. + fn ask_for(program: &str, path: Option<&OsStr>, cwd: &Path, env: &[(&str, &str)]) -> bool { + let this_test = module_path!() + .split_once("::") + .expect("module path has a crate prefix") + .1; + let mut child = Command::new(std::env::current_exe().expect("path of the test binary")); + child + .args([ + "--exact", + &format!("{this_test}::which_child_reports_its_answer"), + "--nocapture", + "--test-threads=1", + ]) + .env(PROGRAM_ENV, program) + .current_dir(cwd); + match path { + Some(path) => child.env("PATH", path), + None => child.env_remove("PATH"), + }; + for (key, value) in env { + child.env(key, value); + } + let output = child.output().expect("start the test binary again"); + let stdout = String::from_utf8_lossy(&output.stdout); + let answer = stdout + .lines() + .find_map(|line| line.split_once(ANSWER_PREFIX).map(|(_, answer)| answer)) + .unwrap_or_else(|| { + panic!( + "the child printed no answer (status {}):\n{stdout}\n{}", + output.status, + String::from_utf8_lossy(&output.stderr) + ) + }); + answer.trim() == "true" + } + + fn ask(path: Option<&OsStr>, cwd: &Path) -> bool { + ask_for(PROGRAM, path, cwd, &[]) + } + + fn join(dirs: &[&Path]) -> OsString { + std::env::join_paths(dirs).expect("no directory name holds a path separator") + } + + /// Create `name` under `parent`, holding a file called `file_name`. + fn dir_holding(parent: &Path, name: &OsStr, file_name: &str) -> PathBuf { + let dir = parent.join(name); + std::fs::create_dir_all(&dir).expect("create the planted directory"); + crate::fake_executable::write_fake_executable(&dir.join(file_name), "#!/bin/sh\n") + .expect("plant the program"); + dir + } + + fn empty_dir(parent: &Path, name: &str) -> PathBuf { + let dir = parent.join(name); + std::fs::create_dir_all(&dir).expect("create the empty directory"); + dir + } + + #[test] + fn which_finds_a_program_in_a_path_directory_and_not_otherwise() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let present = dir_holding(scratch.path(), OsStr::new("present"), PROGRAM); + let absent = empty_dir(scratch.path(), "absent"); + let cwd = empty_dir(scratch.path(), "cwd"); + + assert!( + ask(Some(&join(&[&absent, &present])), &cwd), + "a program in the second PATH directory is found" + ); + assert!( + !ask(Some(&join(&[&absent])), &cwd), + "a PATH without the program finds nothing" + ); + assert!(!ask(None, &cwd), "an unset PATH finds nothing"); + } + + #[test] + fn which_does_not_take_a_directory_named_like_the_program_for_the_program() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let bin = empty_dir(scratch.path(), "bin"); + std::fs::create_dir_all(bin.join(PROGRAM)).expect("plant a directory with the name"); + assert!(!ask(Some(&join(&[&bin])), scratch.path())); + } + + /// The decision this lookup was merged on: an empty `PATH` entry is skipped, + /// so a file in the directory the CLI runs from is not "installed". The + /// summary stack's old copy read the entry as the current directory and + /// answered `true` here; the control below shows the same file does count + /// once a real entry names its directory. + #[cfg(unix)] + #[test] + fn which_does_not_read_an_empty_path_entry_as_the_current_directory() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let cwd = dir_holding(scratch.path(), OsStr::new("cwd"), PROGRAM); + let bin = empty_dir(scratch.path(), "bin"); + let bin_text = bin.to_str().expect("UTF-8 scratch path"); + + for (shape, path) in [ + ("leading", format!(":{bin_text}")), + ("middle", format!("{bin_text}::{bin_text}")), + ("trailing", format!("{bin_text}:")), + ("the whole PATH", String::new()), + ] { + assert!( + !ask(Some(OsStr::new(&path)), &cwd), + "an empty entry ({shape}) must not read the current directory" + ); + } + assert!( + ask(Some(&join(&[&cwd])), &cwd), + "the same file is found when a PATH entry names its directory" + ); + } + + /// The other half of the decision: a `PATH` that is not valid Unicode is + /// still searched. The examination stack's old copy read `PATH` with + /// `std::env::var`, which fails on it, and answered `false` for every tool. + #[cfg(unix)] + #[test] + fn which_searches_a_path_that_is_not_unicode() { + use std::os::unix::ffi::OsStrExt as _; + + let scratch = tempfile::tempdir().expect("scratch directory"); + let cwd = empty_dir(scratch.path(), "cwd"); + let valid = dir_holding(scratch.path(), OsStr::new("valid"), PROGRAM); + // A directory whose own name is not Unicode (it holds the byte 0xff). + let odd = dir_holding(scratch.path(), OsStr::from_bytes(b"odd-\xff"), PROGRAM); + let missing_odd = scratch.path().join(OsStr::from_bytes(b"missing-\xff")); + + assert!( + ask(Some(&join(&[&odd])), &cwd), + "the program sits in a directory whose name is not Unicode" + ); + assert!( + ask(Some(&join(&[&missing_odd, &valid])), &cwd), + "a Unicode directory after an entry that is not Unicode is still searched" + ); + assert!( + !ask(Some(&join(&[&missing_odd])), &cwd), + "a PATH that is not Unicode does not find a program that is not on it" + ); + } + + /// A `.ps1` listed in `PATHEXT` is found, where the examination stack's old + /// copy tried only `.exe`, `.bat`, `.cmd` and the bare name. + #[cfg(windows)] + #[test] + fn which_tries_a_pathext_extension_outside_exe_bat_and_cmd() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let ps1 = dir_holding(scratch.path(), OsStr::new("ps1"), &format!("{PROGRAM}.ps1")); + let pathext = [("PATHEXT", ".COM;.EXE;.BAT;.CMD;.PS1")]; + assert!(ask_for( + PROGRAM, + Some(&join(&[&ps1])), + scratch.path(), + &pathext + )); + } + + /// An entry in quotes keeps a `;` inside it, as `std::env::split_paths` + /// reads it on Windows; the examination stack's old copy split on every `;`. + #[cfg(windows)] + #[test] + fn which_searches_a_quoted_path_entry_that_holds_a_semicolon() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let dir = dir_holding( + scratch.path(), + OsStr::new("semi;colon"), + &format!("{PROGRAM}.exe"), + ); + let quoted = format!("\"{}\"", dir.display()); + assert!(ask(Some(OsStr::new("ed)), scratch.path())); + } + + /// `cmd` is found at its System32 location without being on `PATH`, as the + /// summary stack's old copy found it; the examination stack's did not. + #[cfg(windows)] + #[test] + fn which_finds_cmd_at_its_system32_location_without_it_being_on_path() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let empty = empty_dir(scratch.path(), "empty"); + assert!(ask_for("cmd", Some(&join(&[&empty])), scratch.path(), &[])); + // And with no PATH at all, as the capture runner could still start it. + assert!(ask_for("cmd", None, scratch.path(), &[])); + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 305fb5d47f..15aa2820ee 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -194,8 +194,8 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`gguf.rs`](../crates/rocm-core/src/gguf.rs) — reads the shape of a GGUF model (layers, KV heads, head width, parameter count, data offset) from its header over any `Read + Seek`, never touching the weights. - [`gguf_fit.rs`](../crates/rocm-core/src/gguf_fit.rs) — pure arithmetic over that header: weights + KV cache at a context size and KV type + compute buffer + runtime reserve against dedicated VRAM, the largest context that fits, and what would make a model that does not fit, fit. - [`gguf_source.rs`](../crates/rocm-core/src/gguf_source.rs) — where a header comes from: a local file, or a Hugging Face file, found by one repository-listing request (and a `HEAD` request for a file the listing does not size) and read by HTTP range requests (`HF_ENDPOINT` selects a mirror; redirects are followed by hand, up to five, and never from https to plain http). The only code in the estimate that reads a file or the network. -- [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) — (ROCMAI-85, Phase 7a of ROCMAI-27's sequencing) GPU/host hardware detection — see the module's own doc comment for what it owns. Its leaf concerns are child modules under `host_gpu/`, with `host_gpu.rs` kept as the module root: [`capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs) (running an optional tool under a timeout and the tool-on-`PATH` lookup), [`os_facts.rs`](../crates/rocm-core/src/host_gpu/os_facts.rs) (kernel version, distro name, CPU model and installed RAM), [`linux.rs`](../crates/rocm-core/src/host_gpu/linux.rs) (the sysfs, KFD and DRM probes), [`gpu_visibility.rs`](../crates/rocm-core/src/host_gpu/gpu_visibility.rs) (which AMD GPUs this process may use, from the KFD/DRM counts and the `HIP_VISIBLE_DEVICES`/`ROCR_VISIBLE_DEVICES` masks, plus `windows_amd_gpu_listing`, the native-Windows adapter bound `rocm serve --gpu ` is checked against), [`windows_inventory.rs`](../crates/rocm-core/src/host_gpu/windows_inventory.rs) (the Windows display-adapter inventory and its parsers, including the AMD-adapter count behind `windows_amd_gpu_listing`), [`wsl.rs`](../crates/rocm-core/src/host_gpu/wsl.rs) (WSL detection, the librocdxg chain and the Windows host driver as seen from the guest), [`render.rs`](../crates/rocm-core/src/host_gpu/render.rs) (the `rocm examine` text of an `ExamineSummary`, as a second `impl` block), [`diagnostics.rs`](../crates/rocm-core/src/host_gpu/diagnostics.rs) (the GPU detection diagnostics text) and [`engine_default.rs`](../crates/rocm-core/src/host_gpu/engine_default.rs) (the default serve engine, and the vLLM preference for the TheRock families that serve through it). -- [`host-probing.md`](host-probing.md) — a design note, not a module: [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) and [`examine.rs`](../crates/rocm-core/src/examine.rs) are two separate host-probing stacks that still read some of the same facts twice (kernel version, CPU model, the Windows GPU query, the marketing-name table, the `PATH` lookup and the subprocess runners). It proposes which of each pair should survive, how the two differ today and the order in which they could converge; it changes no code. It is a snapshot, to be retired when the convergence tickets close. +- [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) — (ROCMAI-85, Phase 7a of ROCMAI-27's sequencing) GPU/host hardware detection — see the module's own doc comment for what it owns. Its leaf concerns are child modules under `host_gpu/`, with `host_gpu.rs` kept as the module root: [`capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs) (running an optional tool under a timeout, trying each spelling [`probe_exec.rs`](../crates/rocm-core/src/probe_exec.rs) gives for it), [`os_facts.rs`](../crates/rocm-core/src/host_gpu/os_facts.rs) (kernel version, distro name, CPU model and installed RAM), [`linux.rs`](../crates/rocm-core/src/host_gpu/linux.rs) (the sysfs, KFD and DRM probes), [`gpu_visibility.rs`](../crates/rocm-core/src/host_gpu/gpu_visibility.rs) (which AMD GPUs this process may use, from the KFD/DRM counts and the `HIP_VISIBLE_DEVICES`/`ROCR_VISIBLE_DEVICES` masks, plus `windows_amd_gpu_listing`, the native-Windows adapter bound `rocm serve --gpu ` is checked against), [`windows_inventory.rs`](../crates/rocm-core/src/host_gpu/windows_inventory.rs) (the Windows display-adapter inventory and its parsers, including the AMD-adapter count behind `windows_amd_gpu_listing`), [`wsl.rs`](../crates/rocm-core/src/host_gpu/wsl.rs) (WSL detection, the librocdxg chain and the Windows host driver as seen from the guest), [`render.rs`](../crates/rocm-core/src/host_gpu/render.rs) (the `rocm examine` text of an `ExamineSummary`, as a second `impl` block), [`diagnostics.rs`](../crates/rocm-core/src/host_gpu/diagnostics.rs) (the GPU detection diagnostics text) and [`engine_default.rs`](../crates/rocm-core/src/host_gpu/engine_default.rs) (the default serve engine, and the vLLM preference for the TheRock families that serve through it). +- [`host-probing.md`](host-probing.md) — a design note, not a module: [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) and [`examine.rs`](../crates/rocm-core/src/examine.rs) are two separate host-probing stacks that read some of the same facts twice (kernel version, CPU model, the Windows GPU query, the marketing-name table and the subprocess runners). It proposes which of each pair should survive, how the two differ and the order in which they could converge, and records what each convergence decided; it changes no code itself. The `PATH` lookup is the first pair converged: [`probe_exec.rs`](../crates/rocm-core/src/probe_exec.rs)'s `which` is now the only copy. It is a snapshot, to be retired when the convergence tickets close. - [`jsonl_store.rs`](../crates/rocm-core/src/jsonl_store.rs) — the JSONL store engine behind the automation stores: the sibling `.lock` file (`jsonl_lock_path`, re-exported at the crate root), single-line appends, torn-tail repair, size-based rotation (`JsonlRotation`) and the tail-first readers that span rotated generations. It is generic over the record type; [`automation.rs`](../crates/rocm-core/src/automation.rs) supplies the record types and each store's policy, and nothing in this module depends on it, in production or in its own tests: those drive the engine with a small record type of their own. The module is private; its engine items are `pub(crate)`. - [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) — (ROCMAI-85, Phase 7a) the TheRock managed-runtime-environment surface — see the module's own doc comment for what it owns. - [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs) — (ROCMAI-152, Phase 7c) the curated model catalog's types, built-in catalog JSON loading, signed external recipe indexes, and artifact cache-marker bookkeeping — see the module's own doc comment for what it owns. @@ -203,7 +203,7 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`net.rs`](../crates/rocm-core/src/net.rs) — (ROCMAI-154) networking primitives: the default local host and port, URL/endpoint formatting and parsing, and the hand-rolled loopback HTTP/TCP client (`http_get_text*`, `http_post_json_with_auth`, `read_http_response_bounded`). It depends on no other service module: the managed-service readiness probes built on it live in [`endpoint_readiness.rs`](../crates/rocm-core/src/endpoint_readiness.rs) and the streaming downloads in [`download.rs`](../crates/rocm-core/src/download.rs). - [`paths.rs`](../crates/rocm-core/src/paths.rs) — (ROCMAI-154, Phase 7b) `AppPaths` and its filesystem-location primitives. - [`placeholder.rs`](../crates/rocm-core/src/placeholder.rs) — the `` placeholder rendered reports print for an empty value: `empty_as_unknown` (returns the value untouched) and `trimmed_or_unknown` (trims first), shared by `apps/rocm`'s driver-install plan and chat tool summaries and by [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs)'s Windows GPU diagnostics, replacing the copy those two `apps/rocm` callers shared in [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and `host_gpu.rs`'s own trimming copy. -- [`probe_exec.rs`](../crates/rocm-core/src/probe_exec.rs) — the host probes' subprocess and file-read helpers, split out of [`examine.rs`](../crates/rocm-core/src/examine.rs) so `fix.rs`, `process.rs` and the child modules of [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) reach them without importing the module that owns the `Examination` document: `run`/`run_with_env`/`run_raw` (run a command under a timeout; `rc` is `127` when it cannot be spawned and `124` on timeout), `run_utf16le`/`decode_utf16le` (for `wsl.exe`'s UTF-16LE output), `read_text`/`read_host_text`, `which`, the `SHORT`/`MEDIUM` timeouts and `truncate_to_chars`. All are `pub(crate)` except `run_raw`, which stays private to the module. The module itself is private to the crate and is not re-exported. +- [`probe_exec.rs`](../crates/rocm-core/src/probe_exec.rs) — the host probes' subprocess and file-read helpers, split out of [`examine.rs`](../crates/rocm-core/src/examine.rs) so `fix.rs`, `process.rs` and the child modules of [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs) reach them without importing the module that owns the `Examination` document: `run`/`run_with_env`/`run_raw` (run a command under a timeout; `rc` is `127` when it cannot be spawned and `124` on timeout), `run_utf16le`/`decode_utf16le` (for `wsl.exe`'s UTF-16LE output), `read_text`/`read_host_text`, `which` (the one `PATH` lookup, used by both host-probing stacks: it skips an empty `PATH` entry, searches a `PATH` that is not valid Unicode and follows `PATHEXT` on Windows) with `tool_path_candidates`, the spellings it tries and the capture runner in [`host_gpu/capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs) starts in turn, the `SHORT`/`MEDIUM` timeouts and `truncate_to_chars`. All are `pub(crate)` except `run_raw`, which stays private to the module. The module itself is private to the crate and is not re-exported. - [`process.rs`](../crates/rocm-core/src/process.rs) — (ROCMAI-154, Phase 7b) process-spawn/terminate primitives, including `FileLock`, plus `run_with_timeout`: it drains both pipes on their own threads from the moment the child starts, so output size cannot stall a command into its own timeout, and it bounds how much of a timed-out child's output the error quotes, because that text reaches persisted automation logs. It is not yet the *only* such helper — [`therock/process.rs`](../apps/rocm/src/therock/process.rs)'s `wait_with_output_bounded`, [`probe_exec.rs`](../crates/rocm-core/src/probe_exec.rs)'s `run_raw`, and [`host_gpu/capture.rs`](../crates/rocm-core/src/host_gpu/capture.rs)'s `capture_optional_command_candidate_with_timeout` (plus the Windows-only `capture_diagnostic_command` in [`host_gpu/diagnostics.rs`](../crates/rocm-core/src/host_gpu/diagnostics.rs)) still join their reader threads with no time limit, so a grandchild holding the pipe can outlast their timeout; converging them onto `run_with_timeout` is follow-up work to ROCMAI-448 tracked in [#623](https://github.com/ROCm/rocm-cli/issues/623). - [`quoting.rs`](../crates/rocm-core/src/quoting.rs) — (ROCMAI-727) argument-quoting helpers with real execution stakes (POSIX shell, PowerShell), consolidated out of `apps/rocm`-side near-duplicates — see the module's own doc comment for what it owns. `posix_shell_quote` also renders the command lines the CLI shows (through `apps/rocm`'s `shell_quote`); only `comfyui.rs`'s log-line quoter stays separate. - [`report_rollup.rs`](../crates/rocm-core/src/report_rollup.rs) — (ROCMAI-492) counts read Doctor reports at every level of `Report::group_key`, with refused and unread reports kept apart. Pure, with no transport: the mailbox adapter that feeds it is maintainer tooling in `xtask`, behind `cargo xtask report-rollup`. diff --git a/docs/host-probing.md b/docs/host-probing.md index bfed94a007..057f5517e0 100644 --- a/docs/host-probing.md +++ b/docs/host-probing.md @@ -6,7 +6,7 @@ SPDX-License-Identifier: MIT # Host probing: which stack owns each fact -Status: proposal. This note changes no code and no behaviour. It decides nothing by itself: it lists the host facts that `rocm-core` reads twice (on the hosts each is read on, see "Which host reads which fact"), says how the two reads differ, and proposes which one should survive and in what order a later series of changes could remove the other. Each pair is meant to be a separate change with its own review. +Status: proposal, with the first pair delivered. This note changes no code and no behaviour itself, and what it proposes for a pair is decided by the change that converges that pair, which then records the decision here (the PATH lookup, pair 1, has: see "PATH lookup"). It lists the host facts that `rocm-core` reads twice (on the hosts each is read on, see "Which host reads which fact"), says how the two reads differ, and proposes which one should survive and in what order a later series of changes could remove the other. Each pair is meant to be a separate change with its own review. This note is a snapshot of `main` at the time of writing, not maintained documentation. It is to be retired (the file deleted, together with its entry in [architecture.md](architecture.md)) when the convergence tickets for these pairs close, so the inventory does not keep pointing at a stale design record. @@ -22,7 +22,7 @@ For each pair, the proposed survivor and the biggest difference a user can see t | [CPU model](#cpu-model) | one reader in `os_facts` returning model and vendor, called by `probe_cpu_linux` and `probe_cpu_windows` | on WSL the text shows the CPU while `cpu_model` is empty and `cpu_vendor` is `unknown`; the change leaves that as it is (option (a)) | | [Windows GPU query](#windows-gpu-query) | the summary stack's inventory, with `probe_gpus_windows` building `gpus[]` from it and NVIDIA kept in the JSON | an NVIDIA-only host reads "AMD GPU not detected yet" in the text beside `has_nvidia_gpu: true` in the JSON | | [Marketing name to gfx target](#marketing-name-to-gfx-target) | the `AMD_MARKETING_GFX_TARGETS` table, gaining the codenames `classify_amd_marketing_name` knows (finish [#521](https://github.com/ROCm/rocm-cli/pull/521)) | on Windows without a HIP SDK an RX 7900 XTX has an empty `gpus[].gfx_target` while `summary.detected_gfx_target` is `gfx1100` | -| [PATH lookup](#path-lookup) | the behaviour of `tool_on_path`, housed in `probe_exec.rs` under the name `which`; the empty-entry case is open | an empty `PATH` entry or a `PATH` that is not UTF-8 changes which tools read as present (for example `wsl.rocminfo`), so the change is not output-preserving | +| [PATH lookup](#path-lookup) | delivered: the behaviour of `tool_on_path`, housed in `probe_exec.rs` under the name `which`; an empty `PATH` entry is skipped | before the change, an empty `PATH` entry or a `PATH` that is not UTF-8 changed which tools read as present (for example `wsl.rocminfo`); now there is one answer, and the differences it moved are listed under "PATH lookup" | | [Subprocess runners](#subprocess-runners) | `run_with_timeout` as the one bounded executor, with `CommandCapture` extended by a typed outcome | a probe behind a hung grandchild returns at its deadline, and a successful capture can lose output a descendant writes more than 200 ms after the child exits | ## The two stacks @@ -50,7 +50,7 @@ Every pair below is read on a subset of the hosts, and the subset decides which | CPU model and vendor | both | summary only: the examination returns before `probe_cpu_linux`, so `cpu_model` is empty and `cpu_vendor` is `unknown` | both, plus the inventory row | neither: `cpu_model` empty | | Windows adapter list | not applicable | examination `gpus[]` stays empty (the WSL branch skips the GPU probes); the summary asks the Windows host for the display target only | both | not collected here; the host driver is read through the summary's inventory (`detect_local_windows_host_driver`) | | Marketing name to gfx target | examination only (the `lspci` scan); the summary reads sysfs and KFD, not names | summary only, through the Windows host's display name | both | not collected | -| PATH lookup | examination `which` only | both (`tool_on_path` and `which` for `rocminfo`) | examination `which` only | examination `which` for `wsl.exe` only | +| PATH lookup | `which` | `which`, asked by both stacks for `rocminfo` (before the change the summary asked `tool_on_path`, a second copy) | `which` | `which` for `wsl.exe` only | | Subprocess runners (which helpers exist on the host) | three: `capture_optional_command*`, `run`, `run_with_timeout` (`capture_diagnostic_command` is Windows only) | the same three | all four | `run` and `run_utf16le` (both over `run_raw`), plus the summary's capture helper through the inventory behind `detect_local_windows_host_driver` | A pair that one host reads with only one stack cannot disagree on that host, but it can still change when a shared helper is changed, so the characterisation tests below name the hosts they cover and say so where a row is empty. @@ -141,7 +141,7 @@ Proposal: the table survives. It is pure, data-driven and already the superset f ## PATH lookup -Verdict: confirmed duplicate. +Verdict: confirmed duplicate, now converged (see "Decided and delivered" below). The comparison that follows is how the two copies differed before that change, kept as the record of what it moved. | | Summary | Examination | |---|---|---| @@ -155,6 +155,26 @@ What a user can see: only on WSL, where both run on Linux: the summary's `wsl_gl Proposal: the behaviour of `tool_on_path` survives (it is the one that follows `PATHEXT`), housed where both stacks already import from, in `probe_exec.rs`, under the existing name `which` so the call sites do not change. `tool_path_candidates` moves with it, because the runner uses it to try each spelling. Keeping the call sites unchanged does not keep their answers unchanged: the differences listed above become the intended observable differences of this change (see the first item of the order below). The empty-entry case is a real choice, because either answer changes one side. Following `PATH` semantics matches what a shell or `execvp` does with an empty entry, and makes `which` read a file in the current directory as found, which widens what the examination calls installed (fifteen or so capability checks, including `sudo` and `usermod`). Skipping empty entries keeps the examination's answer and changes only the summary's on WSL, at the cost of disagreeing with `execvp` for a program the CLI later starts by bare name. The note does not pick one; the change has to, and its PR text states which. +### Decided and delivered + +The change took the survivor proposed above: `which` in [probe_exec.rs](../crates/rocm-core/src/probe_exec.rs) has the behaviour `tool_on_path` had, `tool_path_candidates` moved there with it (the capture runner in [capture.rs](../crates/rocm-core/src/host_gpu/capture.rs) and the Windows diagnostics capture import it from there), `detect_wsl_summary` asks `which` for `rocminfo` and `cargo`, and `tool_on_path` is gone. The Windows spellings are computed by `path_candidates`, which takes the platform, `PATHEXT` and the system root as arguments, and the search by `found_in`, which takes the directories and an existence test, so both are unit-tested on every host. + +The empty-entry question is decided as skip. An empty `PATH` entry is not read as the current directory. That keeps the examination's answer, which is what fifteen or so capability checks (including `sudo` and `usermod`) already relied on, so the only answer the empty-entry rule moves is the summary's on WSL, where a `rocminfo` or `cargo` file in the current directory no longer reads as installed. It also settles the Windows question the note could not answer: how `std::env::split_paths` treats an empty entry there no longer matters, because the entry is dropped either way. The cost is the one named above: a program the CLI later starts by bare name can still resolve from the current directory through the operating system when `PATH` has an empty entry, and the lookup, which does not start anything, no longer agrees with that. Guarding the runners against that (`capture_optional_command*` and `run` start a bare name through `Command`, so the operating system resolves an empty entry for them) is a property of how they spawn, not of the lookup, so it belongs to the subprocess-runners pair (item 6 below) and is not changed here. + +The differences that moved, each with the tests that pin it (the unit tests are in `probe_exec.rs`; the scenarios are in `examine.feature`): + +| Difference | Before | Now | Pinned by | +|---|---|---|---| +| Empty `PATH` entry, with a file of the program's name in the current directory | summary: found; examination: not found | not found | `which_does_not_read_an_empty_path_entry_as_the_current_directory`, `path_dirs_drop_every_empty_entry`, scenario `examine-36` (control `examine-35`) | +| `PATH` that is not valid Unicode | summary: searched; examination: not found for every program | searched | `which_searches_a_path_that_is_not_unicode`, `path_dirs_keep_an_entry_that_is_not_unicode`, scenario `examine-37` | +| Windows `PATHEXT` entries outside `.exe`, `.bat`, `.cmd`, in both cases | summary: tried; examination: not tried | tried | `which_tries_a_pathext_extension_outside_exe_bat_and_cmd` (Windows job), `candidates_on_windows_try_every_pathext_entry_in_both_cases`, `candidates_on_windows_fall_back_to_the_default_pathext_when_it_is_unset` | +| Windows `PATH` entry in quotes holding a `;` | summary: kept whole; examination: split | kept whole | `which_searches_a_quoted_path_entry_that_holds_a_semicolon` (Windows job) | +| A Windows name that already has an extension (`python3.11`, `tool.py`, `wsl.exe`) | examination: also tried `.exe`, `.bat` and `.cmd` after it; summary: as given only | as given only | `candidates_on_windows_for_a_name_with_an_extension_are_the_name_alone` | +| System32 locations of `cmd`, `pnputil` and `powershell` (`pwsh` has none; its only extra candidate is the bare `pwsh.exe`) | summary: found; examination: not found | found | `which_finds_cmd_at_its_system32_location_without_it_being_on_path` (Windows job), `candidates_on_windows_add_the_fixed_system32_locations`, `found_in_asks_about_an_absolute_candidate_once_as_it_is_whatever_the_directories` | +| A file in the current directory, with no empty entry | not found by either | not found | the control in `which_does_not_read_an_empty_path_entry_as_the_current_directory` | + +No `which` call site passes a name with an extension other than `wsl.exe` (found as given), so the dotted-name row moves no answer today: the call sites ask for python, python3, mokutil, lspci, rocminfo, dpkg, rpm, usermod, sudo, cargo and the llama.cpp names, all bare. The System32 candidates stay in the shared candidate list. The capture runner needs them to start `cmd`, `pnputil` and `powershell` at all on a host where they are not on `PATH`, and one list that both the runner and the lookup read cannot drift. No `which` call site asks for those four today, so the change is invisible there; a future caller of `which("powershell")` gets "would the runner find it", not "is it on `PATH`". An absolute candidate is tested once as it is and not joined onto each `PATH` directory, so it is found even when `PATH` is unset or all empty, which is what the runner can start. The near copy of `tool_path_candidates` in `apps/rocm` (`program_path_candidates`) did not join this change and stays open at the end of this note. The scenarios `examine-35`, `examine-36`, `examine-37` and `examine-38` in [examine.feature](../tests/e2e-cucumber/features/examine.feature) drive the first two rows on the Linux lane from a simulated WSL host, holding `summary.wsl.rocminfo` and `wsl.rocminfo` of one `rocm examine --json` run level: a tool in a `PATH` directory is found (the control), a tool only in the working directory behind an empty entry is not (with `.` as the entry instead it is, which `examine-38` holds as the control), and a tool in a directory whose name is not valid Unicode is found. The harness sets a `PATH` that is not Unicode without trouble, which settles the question the note left about it. The Windows rows cannot run on the Linux lane and are covered by the unit tests named above, which the Windows job runs. + ## Subprocess runners Verdict: confirmed duplicate, and a family rather than a pair. @@ -184,7 +204,7 @@ Proposal: `run_with_timeout` survives as the one bounded executor, with `Command One pair per change. Each begins with a commit that adds a characterisation test and is green against the code as it is: it pins what both copies return for the same input, and where they differ it pins each copy's answer, with a comment naming the pair. The convergence commit then flips the assertions that are meant to change and the PR text lists each one as a user-visible difference. Tests that need a program on `PATH` use the fake-executable helper in [fake_executable.rs](../crates/rocm-core/src/fake_executable.rs); Windows-only logic goes behind an injectable function so a unit test runs on the Windows job, as the inventory already does with `WindowsInventorySources`. -1. PATH lookup. Smallest, and it settles where `tool_path_candidates` lives before the runner work needs it. It is not output-preserving: the differences listed under "PATH lookup" change which tools are reported as present, and its PR text lists each one as intentional, with the characterisation test that pins it. Hosts covered: Linux, which is also WSL's code path, for the empty entry, the non-UTF-8 `PATH` (set through an `OsString` environment value) and the current-directory file; Windows through an injectable resolver for the `PATHEXT` spellings, the System32 candidates and the `;` splitting, since the Linux lane cannot run them. +1. PATH lookup (delivered: see "Decided and delivered" under "PATH lookup"). Smallest, and it settles where `tool_path_candidates` lives before the runner work needs it. It is not output-preserving: the differences listed under "PATH lookup" change which tools are reported as present, and its PR text lists each one as intentional, with the characterisation test that pins it. Hosts covered: Linux, which is also WSL's code path, for the empty entry, the non-UTF-8 `PATH` (set through an `OsString` environment value) and the current-directory file; Windows through an injectable resolver for the `PATHEXT` spellings, the System32 candidates and the `;` splitting, since the Linux lane cannot run them. 2. Kernel version. A single command; settles the timeout and the empty-versus-`None` question. Hosts covered as set out under "Kernel version". 3. CPU model. Adds the model-and-vendor reader; needs the Windows inventory row, which already exists. Its PR says which WSL option under "CPU model" it takes, (a) or (b); a PR that does not is not ready. 4. Marketing name to gfx target, by finishing [#521](https://github.com/ROCm/rocm-cli/pull/521). It must land before the next item, so that `gpus[]` is not rebuilt on top of two name tables. @@ -200,10 +220,10 @@ No item is output-preserving without qualification, and the PR for each says whi - How NVIDIA adapters are represented once the examination reads the inventory (a vendor on every row, or a separate check). - Whether one `rocm examine --json` run shares answers between the stacks (a single WSL summary and a single `ldconfig -p`, one inventory including its failure) or keeps calling shared code that asks again. - What an absent Windows inventory does once there is one inventory: remember the absence for the run, or drop the per-fact fallbacks. -- Whether an empty `PATH` entry counts as the current directory in the surviving lookup. - Whether `gpus[].name` should come from `pnputil` or from WMI. - Which timeout wins for the kernel and CPU reads, and whether `kernel_release` should be filled on Windows. - Whether the typed outcome belongs on `CommandCapture` or beside it. - Whether CPU facts should be collected on WSL at all (option (b) under "CPU model"), and if so how the Windows-hosted distro probe obtains them. - Whether the adapters over `run_with_timeout` take a per-call drain grace, so that a probe whose tool is known to leave a late-writing descendant keeps its output, or whether the 200 ms grace is accepted for every probe. - Whether the near copy of `tool_path_candidates` in `apps/rocm` joins the PATH lookup change. +- Whether the runners that replace the subprocess helpers should refuse to resolve a bare program name through an empty `PATH` entry, so that what the lookup calls absent is also what they do not start. diff --git a/docs/testing.md b/docs/testing.md index a0d360b38b..010bb8efb6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -3332,6 +3332,12 @@ cache nor the variable involved, which is the layout of a TheRock tarball stays `wsl_rocdxg_missing`, so the sibling runtime, and not the planted library, is what the verdict turns on. +`examine-35`, `examine-36`, `examine-37` and `examine-38` hold the two answers `rocm examine --json` gives to "is `rocminfo` installed" level on that same planted WSL machine: `summary.wsl.rocminfo` from the host summary and `wsl.rocminfo` from the examination, which share one `PATH` lookup. They plant a stand-in `rocminfo` that prints nothing, and build the `PATH` from the machine's stand-in directory followed by the runner's own entries that hold no `rocminfo`, so a runner with ROCm installed cannot be why the tool reads as found. `examine-35` is the control: the tool in a `PATH` directory is found by both. `examine-36` puts it only in the directory the command runs from, with an empty entry first on `PATH`: neither report finds it, because an empty entry is skipped and not read as the current directory (the harness's `command_dir` sets that directory). `examine-38` is its control: the same machine with `.` as the first entry instead, which both reports find, so it fails if `rocm` does not really run from that directory and `examine-36`'s answer cannot come from a harness that ignored `command_dir`. `examine-37` puts it in a `PATH` directory whose name is not valid Unicode: both find it, because a `PATH` that is not Unicode is still searched. The Windows spellings of the lookup (`PATHEXT`, a quoted entry holding a `;`, the System32 locations) cannot run on the Linux lane and are covered by the unit tests in `probe_exec.rs`, which the Windows job runs. + +```bash +RUSTC_WRAPPER= cargo xtask e2e -- -n 'examine-3[5-8] ' +``` + `serve-42` plants the same WSL machine with `LD_LIBRARY_PATH` empty and runs `rocm serve`: it is refused, its advice names the WSL `driver_status` and not `/dev/kfd`, and `rocm examine` on that machine reports `wsl_rocdxg_missing`, so the command the refusal names exists and shows the condition. The launch that follows when `LD_LIBRARY_PATH` does reach the library has no scenario, because it would start preparing a real engine; the serve gate's decision there is covered by unit tests in `gpu_visibility.rs`. ```bash diff --git a/tests/e2e-cucumber/features/examine.feature b/tests/e2e-cucumber/features/examine.feature index 70b6de0eb4..7620e2d04a 100644 --- a/tests/e2e-cucumber/features/examine.feature +++ b/tests/e2e-cucumber/features/examine.feature @@ -548,3 +548,53 @@ Feature: GPU detection and system inspection Then the inspection reports the ROCDXG driver status "wsl_rocdxg_ready" And the inspection reports the ROCDXG library as beside the runtime and in neither the linker cache nor LD_LIBRARY_PATH And the inspection's driver detail does not send the user to register the library with ldconfig + + # `rocm examine --json` asks twice whether `rocminfo` is installed on WSL: the + # host summary (`summary.wsl.rocminfo`) and the examination (`wsl.rocminfo`). + # They used to use two different PATH lookups and could disagree; they now use + # one, and these scenarios hold the two answers level on the cases where + # the old lookups differed. The planted WSL machine is the one `examine-32` uses + # (a simulated host root with `/dev/dxg`); the runner's own PATH entries that + # hold a `rocminfo` are left out, so a runner with ROCm installed cannot be the + # reason a tool reads as found. The stand-in `rocminfo` prints nothing, so the + # scenarios are about whether the tool is found, not whether it sees a GPU. + # + # The control: a `rocminfo` in a PATH directory is found by both. + @id:examine-rocminfo-on-path-is-found-by-both-reports @requires-os:linux + Scenario: examine-35 - Both reports find a rocminfo that sits in a PATH directory + Given a WSL machine whose rocminfo is in a PATH directory + When the user inspects that machine in machine-readable form + Then both reports say rocminfo is installed + + # An empty PATH entry is not the current directory for this lookup: a file named + # `rocminfo` in the directory the command runs from is not "installed". The + # summary's old lookup read the entry as the current directory and said it was, + # while the examination's said it was not. The cost is that a program the CLI + # starts by bare name can still resolve from the current directory through the + # operating system, which the lookup does not follow. + @id:examine-rocminfo-in-working-directory-is-not-installed @requires-os:linux + Scenario: examine-36 - Neither report reads an empty PATH entry as the working directory + Given a WSL machine whose only rocminfo is in the directory the command runs from, with an empty entry on PATH + When the user inspects that machine in machine-readable form + Then both reports say rocminfo is not installed + + # A PATH that is not valid Unicode is searched. The examination's old lookup read + # PATH as a `String`, which fails on it, and so found no tool at all while the + # summary's old lookup still searched it. + @id:examine-rocminfo-on-a-path-that-is-not-unicode-is-found @requires-os:linux + Scenario: examine-37 - Both reports search a PATH that is not valid Unicode + Given a WSL machine whose rocminfo sits in a PATH directory with a name that is not valid Unicode + When the user inspects that machine in machine-readable form + Then both reports say rocminfo is installed + + # The control for `examine-36`: the same machine, with `.` as the first PATH + # entry instead of an empty one. A `.` entry is an ordinary directory, so the + # lookup finds the stand-in in the working directory. That only happens if the + # `rocm` child really runs from it, so this scenario fails when the harness does + # not apply the working directory, and `examine-36`'s "not installed" is then + # meaningful: the two differ in nothing but the entry. + @id:examine-rocminfo-in-working-directory-named-on-path-is-found @requires-os:linux + Scenario: examine-38 - Both reports find a rocminfo in the working directory when PATH names it + Given a WSL machine whose only rocminfo is in the directory the command runs from, with that directory named on PATH + When the user inspects that machine in machine-readable form + Then both reports say rocminfo is installed diff --git a/tests/e2e-cucumber/src/binary_path.rs b/tests/e2e-cucumber/src/binary_path.rs new file mode 100644 index 0000000000..93c959f428 --- /dev/null +++ b/tests/e2e-cucumber/src/binary_path.rs @@ -0,0 +1,100 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! How the harness names the `rocm` binary when it starts the child somewhere +//! else. +//! +//! `ROCM_CLI_BINARY` is either a path (`target/debug/rocm`, `./rocm`, an +//! absolute one) or a bare name (`rocm`) the operating system finds on `PATH`. +//! A scenario that moves the child to another working directory must pin a +//! relative path first, because it was written relative to the harness's own +//! directory. A bare name must be left alone: canonicalising it would find a +//! file called `rocm` in the harness's directory, if there is one, and run that +//! instead of the program `PATH` resolves. + +use std::io; +use std::path::PathBuf; + +/// Whether `binary` names a path rather than a bare program name: it is +/// absolute, or contains a separator (so `./rocm`, `bin/rocm` and `../x/rocm` +/// are paths, and `rocm` is not). +#[must_use] +pub fn names_a_path(binary: &str) -> bool { + std::path::Path::new(binary).is_absolute() || binary.contains(['/', std::path::MAIN_SEPARATOR]) +} + +/// The program to start for `binary` when the child runs from another directory. +/// +/// A path is made absolute with `canonicalize` (and kept as written when that +/// fails); a bare name is returned unchanged without asking. +pub fn program_for_moved_child( + binary: &str, + canonicalize: impl Fn(&str) -> io::Result, +) -> PathBuf { + if names_a_path(binary) { + canonicalize(binary).unwrap_or_else(|_| PathBuf::from(binary)) + } else { + PathBuf::from(binary) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::cell::RefCell; + + #[test] + fn paths_are_told_from_bare_names() { + let abs = std::env::temp_dir().join("rocm"); + for (binary, is_path) in [ + ("rocm", false), + ("rocm.exe", false), + ("./rocm", true), + ("bin/rocm", true), + ("../x/rocm", true), + (abs.to_str().expect("UTF-8 temp dir"), true), + ] { + assert_eq!(names_a_path(binary), is_path, "{binary}"); + } + } + + #[test] + fn a_path_is_canonicalised_and_a_bare_name_is_not_even_when_a_local_file_would_match() { + // A canonicalise that succeeds for everything stands for a `rocm` file + // existing in the harness's directory. + let asked = RefCell::new(Vec::new()); + let pin = |binary: &str| { + asked.borrow_mut().push(binary.to_owned()); + Ok(PathBuf::from("/pinned").join(binary)) + }; + + assert_eq!( + program_for_moved_child("rocm", pin), + PathBuf::from("rocm"), + "a bare name stays for PATH to resolve" + ); + assert!( + asked.borrow().is_empty(), + "a bare name must not be looked up on disk" + ); + for binary in ["./rocm", "bin/rocm", "../x/rocm"] { + assert_eq!( + program_for_moved_child(binary, pin), + PathBuf::from("/pinned").join(binary), + "{binary}" + ); + } + assert_eq!(asked.borrow().len(), 3); + } + + #[test] + fn a_path_that_cannot_be_canonicalised_is_kept_as_written() { + assert_eq!( + program_for_moved_child("./gone/rocm", |_| Err(io::Error::from( + io::ErrorKind::NotFound + ))), + PathBuf::from("./gone/rocm") + ); + } +} diff --git a/tests/e2e-cucumber/src/lib.rs b/tests/e2e-cucumber/src/lib.rs index 33cc23da50..f3969cee22 100644 --- a/tests/e2e-cucumber/src/lib.rs +++ b/tests/e2e-cucumber/src/lib.rs @@ -2,6 +2,7 @@ // // SPDX-License-Identifier: MIT +pub mod binary_path; pub mod capability; pub mod chat_request; pub mod device_memory; @@ -18,6 +19,7 @@ pub mod monotonic_clock; pub mod paced_download; pub mod palette_rows; pub mod panic_capture; +pub mod planted_path; pub mod polled_deadline; pub mod powershell_text_rule; pub mod quit_and_wait; diff --git a/tests/e2e-cucumber/src/planted_path.rs b/tests/e2e-cucumber/src/planted_path.rs new file mode 100644 index 0000000000..e26c46d868 --- /dev/null +++ b/tests/e2e-cucumber/src/planted_path.rs @@ -0,0 +1,179 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The `PATH` a scenario hands the `rocm` child when it plants a machine that +//! holds a `rocminfo` of its own. +//! +//! The scenarios about whether `rocminfo` is installed must not be decided by a +//! `rocminfo` the runner happens to have, so the runner's own `PATH` entries +//! that hold one are left out. The entries are written by the harness process +//! but read by the child, and the two can run from different directories (the +//! scenario's `command_dir`), so a relative entry such as `.` means two +//! different directories. [`planted_path_entries`] settles that before it +//! filters: each inherited relative entry is made absolute against the +//! harness's own working directory, and is kept or dropped by what that +//! directory holds. The fixture's own entries are never touched, because a +//! scenario writes them to be read from the child's directory (an empty entry +//! and `.` are the point of two of them). + +use std::ffi::OsStr; +use std::path::{Path, PathBuf}; + +/// The program the runner's entries are checked for. +pub const PLANTED_TOOL: &str = "rocminfo"; + +/// Whether `dir` would make [`PLANTED_TOOL`] resolve. +/// +/// It holds a regular file of that name, as the shared `which` lookup in +/// `rocm-core` requires. A directory called `rocminfo` does not, so it must not +/// get its `PATH` entry dropped. +#[must_use] +pub fn dir_holds_tool(dir: &Path) -> bool { + dir.join(PLANTED_TOOL).is_file() +} + +/// The entries of the planted `PATH`, in order: `fixture` exactly as written, +/// then `fake_bin`, then each non-empty entry of `inherited` that holds no +/// [`PLANTED_TOOL`]. +/// +/// An inherited relative entry is resolved against `harness_cwd` first, checked +/// there with `holds_tool`, and kept in its resolved (absolute) form, so it still +/// names the directory it named when the harness read it after the child has +/// moved. An inherited empty entry is dropped: it would be read as the child's +/// working directory, which the scenario decides and the runner must not. +pub fn planted_path_entries( + fixture: &[PathBuf], + fake_bin: &Path, + inherited: &OsStr, + harness_cwd: &Path, + holds_tool: impl Fn(&Path) -> bool, +) -> Vec { + let kept = std::env::split_paths(inherited) + .filter(|entry| !entry.as_os_str().is_empty()) + // `join` leaves an absolute entry as it is and roots a relative one. + .map(|entry| harness_cwd.join(entry)) + .filter(|resolved| !holds_tool(resolved)); + fixture + .iter() + .cloned() + .chain(std::iter::once(fake_bin.to_owned())) + .chain(kept) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + /// An absolute directory on any host, so the rows are about the resolution + /// rule and not about one platform's path syntax. + fn abs(name: &str) -> PathBuf { + std::env::temp_dir().join("planted-path-rows").join(name) + } + + fn joined(entries: &[PathBuf]) -> std::ffi::OsString { + std::env::join_paths(entries).expect("no entry holds a separator") + } + + /// The planted entries for a host whose `rocminfo` is in exactly `holding`, + /// with the harness running from `abs("harness")`. + fn planted(fixture: &[PathBuf], inherited: &[PathBuf], holding: &[PathBuf]) -> Vec { + planted_path_entries( + fixture, + &abs("fake-bin"), + &joined(inherited), + &abs("harness"), + |dir| holding.iter().any(|held| held == dir), + ) + } + + #[test] + fn an_inherited_dot_without_a_rocminfo_survives_as_the_harness_directory() { + // The child runs from elsewhere, where a bare `.` would name the planted + // directory and expose its stand-in, so it must not survive as `.`. + assert_eq!( + planted(&[], &[PathBuf::from(".")], &[]), + vec![abs("fake-bin"), abs("harness")] + ); + } + + #[test] + fn an_inherited_dot_with_a_rocminfo_in_the_harness_directory_is_dropped() { + assert_eq!( + planted(&[], &[PathBuf::from(".")], &[abs("harness")]), + vec![abs("fake-bin")] + ); + } + + #[test] + fn an_inherited_absolute_entry_is_checked_and_kept_as_it_is() { + assert_eq!( + planted(&[], &[abs("usr-bin"), abs("rocm-bin")], &[abs("rocm-bin")]), + vec![abs("fake-bin"), abs("usr-bin")] + ); + } + + #[test] + fn an_inherited_relative_name_is_resolved_against_the_harness_directory() { + assert_eq!( + planted( + &[], + &[PathBuf::from("bin"), PathBuf::from("tools")], + &[abs("harness").join("tools")] + ), + vec![abs("fake-bin"), abs("harness").join("bin")] + ); + } + + #[test] + fn inherited_empty_entries_are_dropped_wherever_they_stand() { + assert_eq!( + planted(&[], &[PathBuf::new(), abs("usr-bin"), PathBuf::new()], &[]), + vec![abs("fake-bin"), abs("usr-bin")] + ); + } + + #[test] + fn fixture_entries_come_first_and_are_left_exactly_as_written() { + // The empty entry, `.` and a name are what the scenarios are about; the + // harness directory holding a `rocminfo` makes no difference to them. + let fixture = [PathBuf::new(), PathBuf::from("."), PathBuf::from("odd")]; + assert_eq!( + planted( + &fixture, + &[PathBuf::from(".")], + &[abs("harness"), PathBuf::from("."), PathBuf::from("odd")] + ), + vec![ + PathBuf::new(), + PathBuf::from("."), + PathBuf::from("odd"), + abs("fake-bin") + ] + ); + } + + #[test] + fn a_directory_named_like_the_tool_does_not_hold_it_and_a_file_does() { + let scratch = tempfile::tempdir().expect("scratch directory"); + let with_dir = scratch.path().join("with-dir"); + let with_file = scratch.path().join("with-file"); + std::fs::create_dir_all(with_dir.join(PLANTED_TOOL)).expect("plant a directory"); + std::fs::create_dir_all(&with_file).expect("create the directory"); + std::fs::write(with_file.join(PLANTED_TOOL), "").expect("plant a file"); + + assert!(!dir_holds_tool(&with_dir)); + assert!(dir_holds_tool(&with_file)); + // Through the planting itself: the entry with only a directory of that + // name is kept, the entry with a file is dropped. + let entries = planted_path_entries( + &[], + &abs("fake-bin"), + &joined(&[with_dir.clone(), with_file]), + &abs("harness"), + dir_holds_tool, + ); + assert_eq!(entries, vec![abs("fake-bin"), with_dir]); + } +} diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 96ed4fb0a7..63a8f41d00 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -151,6 +151,11 @@ pub struct E2eWorld { /// records a behavioral precondition here; the When step remains a plain user /// action and consumes the fixture without exposing its mechanism in Gherkin. pub command_env: Vec<(&'static str, std::ffi::OsString)>, + /// The working directory for this scenario's next `rocm` invocation, for a + /// scenario whose subject is what the CLI finds in the directory it is run + /// from. Consumed by [`run_rocm_with_scenario_env`] like `command_env`; `None` + /// leaves the harness's own working directory. + pub command_dir: Option, /// The interactive dash/chat TUI spawned under a pseudo-terminal for this /// scenario, if any (see `e2e::tui_driver`). Torn down in `Drop` before the /// mock server and isolated directory so the child process never outlives @@ -357,6 +362,7 @@ impl Default for E2eWorld { serve_timeout_override: None, expect_xfail: false, command_env: Vec::new(), + command_dir: None, tui: None, chat_use_mock: false, comfyui_baseline_torch_stack: None, @@ -1021,12 +1027,26 @@ pub fn run_rocm_with_env_and_unset( /// consume it so it cannot leak into a later action in the same scenario. pub fn run_rocm_with_scenario_env(world: &mut E2eWorld, args: &[&str]) -> (String, String, i32) { let binary = rocm_binary(); - let mut cmd = std::process::Command::new(&binary); + // A binary given as a relative path was meant relative to this process's + // directory, not to the scenario's, so it is made absolute before the child + // moves. A bare name (`rocm`, found on `PATH`) stays as it is, even if the + // harness's directory happens to hold a file of that name. + let program = if world.command_dir.is_some() { + e2e_cucumber::binary_path::program_for_moved_child(&binary, |path| { + std::fs::canonicalize(path) + }) + } else { + PathBuf::from(&binary) + }; + let mut cmd = std::process::Command::new(program); cmd.args(args); world.isolate_cmd(&mut cmd); for (key, value) in std::mem::take(&mut world.command_env) { cmd.env(key, value); } + if let Some(dir) = world.command_dir.take() { + cmd.current_dir(dir); + } let output = cmd .output() .unwrap_or_else(|e| panic!("failed to run {binary}: {e}")); diff --git a/tests/e2e-cucumber/tests/e2e/examine_steps.rs b/tests/e2e-cucumber/tests/e2e/examine_steps.rs index 93f0999f2f..8d8af1abea 100644 --- a/tests/e2e-cucumber/tests/e2e/examine_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/examine_steps.rs @@ -2369,6 +2369,181 @@ async fn assert_rocdxg_detail_names_the_cache(world: &mut E2eWorld) { ); } +// ── Both reports agree whether `rocminfo` is installed ───────────── + +/// Plant a stand-in `rocminfo` in `dir`. It prints nothing and succeeds, so the +/// probe finds the tool installed and, having seen no agent in its output, does +/// not claim the GPU works: the scenarios turn on whether it is found, not on +/// what it says. +fn plant_stand_in_rocminfo(dir: &std::path::Path) { + std::fs::create_dir_all(dir).expect("failed to create the stand-in rocminfo directory"); + rocm_core::fake_executable::write_fake_executable(&dir.join("rocminfo"), "#!/bin/sh\nexit 0\n") + .expect("failed to write the stand-in rocminfo"); +} + +/// Replace the `PATH` the planted WSL machine gives the next `rocm` run with +/// `entries`, then the machine's `fake-bin` (its stand-in `ldconfig`), then the +/// runner's own entries that hold no `rocminfo`, so a runner with ROCm installed +/// cannot be the reason the tool reads as found. How the runner's entries are +/// resolved and filtered is [`e2e_cucumber::planted_path::planted_path_entries`]. +/// +/// An entry that is an empty path is how a test writes the empty `PATH` entry +/// the shell reads as the current directory. +fn set_planted_wsl_path(world: &mut E2eWorld, entries: &[PathBuf]) { + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .to_owned(); + // The runner's relative entries are resolved from this process's directory + // before they are checked, because the child may run from another one. + let path = std::env::join_paths(e2e_cucumber::planted_path::planted_path_entries( + entries, + &root.join("fake-bin"), + &std::env::var_os("PATH").unwrap_or_default(), + &std::env::current_dir().expect("the harness has a working directory"), + e2e_cucumber::planted_path::dir_holds_tool, + )) + .expect("PATH entries cannot hold a separator"); + world.command_env.retain(|(key, _)| *key != "PATH"); + world.command_env.push(("PATH", path)); +} + +#[given("a WSL machine whose rocminfo is in a PATH directory")] +async fn wsl_machine_with_rocminfo_on_path(world: &mut E2eWorld) { + plant_wsl_machine_with_unregistered_rocdxg(world, ""); + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .to_owned(); + let tools = root.join("rocminfo-bin"); + plant_stand_in_rocminfo(&tools); + set_planted_wsl_path(world, &[tools]); +} + +/// The machine of `examine-36` and `examine-38`: the only `rocminfo` is in +/// `work`, the directory `rocm` is started from, and `path_entry` is the first +/// entry of its `PATH`. The two scenarios differ in nothing else, so the answer +/// can only turn on whether that entry names the working directory: `.` does, +/// and an empty entry does not (the lookup skips it). That is also what shows +/// `command_dir` reaches the `rocm` child: with `.` the stand-in is found only +/// if `rocm` really runs from `work`. +fn plant_wsl_machine_with_rocminfo_only_in_the_working_directory( + world: &mut E2eWorld, + path_entry: PathBuf, +) { + plant_wsl_machine_with_unregistered_rocdxg(world, ""); + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .to_owned(); + let work = root.join("work"); + plant_stand_in_rocminfo(&work); + set_planted_wsl_path(world, &[path_entry]); + // A "not installed" answer would also come out if the stand-in could not be + // reached at all, so show it can: started by bare name from `work` with this + // PATH, it must run, which only an entry that names `work` (or the empty + // entry, which `execvp` reads as `work`) makes possible. + let path = world + .command_env + .iter() + .rev() + .find_map(|(key, value)| (*key == "PATH").then(|| value.clone())) + .expect("the planted PATH was just set"); + let reached = std::process::Command::new("rocminfo") + .env("PATH", path) + .current_dir(&work) + .status() + .expect("the stand-in must resolve through the first PATH entry"); + assert!(reached.success(), "the stand-in rocminfo failed: {reached}"); + world.command_dir = Some(work); +} + +#[given( + "a WSL machine whose only rocminfo is in the directory the command runs from, with an empty entry on PATH" +)] +async fn wsl_machine_with_rocminfo_only_behind_an_empty_path_entry(world: &mut E2eWorld) { + plant_wsl_machine_with_rocminfo_only_in_the_working_directory(world, PathBuf::new()); +} + +#[given( + "a WSL machine whose only rocminfo is in the directory the command runs from, with that directory named on PATH" +)] +async fn wsl_machine_with_rocminfo_only_on_a_path_entry_naming_the_working_directory( + world: &mut E2eWorld, +) { + plant_wsl_machine_with_rocminfo_only_in_the_working_directory(world, PathBuf::from(".")); +} + +/// A directory name that is not valid Unicode: the byte 0xff is never UTF-8. +#[cfg(unix)] +fn directory_name_that_is_not_unicode() -> std::ffi::OsString { + use std::os::unix::ffi::OsStrExt as _; + + std::ffi::OsStr::from_bytes(b"rocminfo-bin-\xff").to_owned() +} + +/// Windows file names are UTF-16 and a lone surrogate would stand in for the +/// byte; the scenario that asks for this name is `@requires-os:linux`, so +/// reaching this is a harness mistake. +#[cfg(not(unix))] +fn directory_name_that_is_not_unicode() -> std::ffi::OsString { + panic!("a directory name that is not valid Unicode is planted on Linux only") +} + +#[given( + "a WSL machine whose rocminfo sits in a PATH directory with a name that is not valid Unicode" +)] +async fn wsl_machine_with_rocminfo_in_a_directory_that_is_not_unicode(world: &mut E2eWorld) { + plant_wsl_machine_with_unregistered_rocdxg(world, ""); + let root = world + .isolated_root + .as_ref() + .expect("no isolated root") + .path() + .to_owned(); + let tools = root.join(directory_name_that_is_not_unicode()); + plant_stand_in_rocminfo(&tools); + set_planted_wsl_path(world, &[tools]); +} + +/// Both reports of one `rocm examine --json` run say whether `rocminfo` is +/// installed: `summary.wsl.rocminfo` from the host summary and `wsl.rocminfo` +/// from the examination. Asked together so the pair cannot disagree unseen. +fn rocminfo_answers(world: &E2eWorld) -> (bool, bool) { + assert_eq!(world.cli_rc, Some(0), "`rocm examine --json` failed"); + let json = parsed_json(world); + let answer = |pointer: &str| { + json.pointer(pointer) + .and_then(serde_json::Value::as_bool) + .unwrap_or_else(|| panic!("`examine --json` has no boolean at {pointer}:\n{json:#}")) + }; + (answer("/summary/wsl/rocminfo"), answer("/wsl/rocminfo")) +} + +#[then("both reports say rocminfo is installed")] +async fn assert_rocminfo_installed_in_both_reports(world: &mut E2eWorld) { + assert_eq!( + rocminfo_answers(world), + (true, true), + "(host summary, examination)" + ); +} + +#[then("both reports say rocminfo is not installed")] +async fn assert_rocminfo_not_installed_in_either_report(world: &mut E2eWorld) { + assert_eq!( + rocminfo_answers(world), + (false, false), + "(host summary, examination)" + ); +} + // ── `rocm serve` on a WSL host whose ROCDXG library the loader cannot find ── #[when("the user serves a model under the GPU-required default on that machine")] From 21808e6748ecdaeb1a80adbdfee4843afcd8890a Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 16:48:55 +0000 Subject: [PATCH 026/102] test(xtask): fail on a hand-rolled text edit in the dash TUI (#985) * test(xtask): fail on a hand-rolled text edit in the dash TUI The shared apply_char_key / apply_char_key_filtered helper replaced the hand-rolled "Char appends, Backspace pops one char" copies in the dash TUI, but only prose in AGENTS.md and architecture.md kept the next copy from appearing. Add a source-scan test that reads the non-test code of every file under crates/rocm-dash-tui/src/ and reports a no-argument .pop() in a file that handles KeyCode::Backspace, and a .push(x) of a name the file binds as a key character or declares as a char parameter. The scan is per file rather than per match arm because one of the removed copies kept its push in a method the arm called. ui/widgets.rs is exempt, and the Chat input in app/actions.rs is the one allowlisted file; the allowlist pins how many offenses that file carries, so a further hand-rolled field there fails too, and a stale entry fails. Document the check and its limits in docs/testing.md and point the convention in AGENTS.md and docs/architecture.md at it. Closes #851 Signed-off-by: Jussi Elo * fix(xtask): see a typed-in char pushed with a trailing comma or a cast to char The text-edit contract read the argument of a `.push(` as the bound key character only when the next token after the name was `)` or `.`, so `q.push(c,)` (the form rustfmt writes for a call that spans lines), a spaced `q.push( c , )` and a no-op `q.push(c as char)` passed the guard and did not count against an allowlisted file's offenses. The argument may now be the bound name alone, a method on it, or a cast to `char`, each with an optional trailing comma. A second argument and a cast to another type (a `Vec` push) are still not a `String` append. Fixtures cover the single-line and multi-line trailing comma, the other spellings, and the two negative controls. The module doc and docs/testing.md list the shapes that are still not seen. Signed-off-by: Jussi Elo * test(xtask): do not pin the dash helper's offense count in the walk sanity test The sanity test asserted that the scan finds exactly two offenses in ui/widgets.rs, so any unrelated edit to that file failed it, although nothing else depends on the count (check skips the helper file). It now asserts that the scan still sees the helper's push and its pop, which keeps it red when the walk stops reading the file or the scanner stops seeing either half. Also rewraps two over-long lines in the module doc and docs/testing.md. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 2d9b5a1cdba28450c24efd41953b68999ee24998) Signed-off-by: juhovainio --- AGENTS.md | 2 +- docs/architecture.md | 2 +- docs/testing.md | 64 ++ xtask/src/dash_text_edit_contract.rs | 938 +++++++++++++++++++++++++++ xtask/src/main.rs | 1 + 5 files changed, 1005 insertions(+), 2 deletions(-) create mode 100644 xtask/src/dash_text_edit_contract.rs diff --git a/AGENTS.md b/AGENTS.md index 155b474510..43ec89c342 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -267,7 +267,7 @@ dash-TUI, in `crates/rocm-dash-tui/src/ui/`: - **[`spinner.rs`](crates/rocm-dash-tui/src/ui/spinner.rs)**'s `spinner_frame`/`parse_progress_pct` — the braille animation and percentage parsing `job_console.rs` renders with - **[`panel.rs`](crates/rocm-dash-tui/src/ui/panel.rs)**'s `bento`/`popup` — bordered, titled panels; prefer these over an inline `Block::default()`. Popup geometry and framing live here too: `centered_rect`/`centered_width` size and place a popup, `draw_popup_frame` clears and frames it, and `grey_overlay` dims the backdrop - **[`theme.rs`](crates/rocm-dash-tui/src/ui/theme.rs)**'s `StatusTone` and `Theme::tone_color` — the shared status→color mapping. A status enum gets one exhaustive `fn …_tone(status) -> StatusTone` beside them (`job_status_tone`, `instance_status_tone`) and a `Theme::…_color` wrapper (`job_status_color`, `instance_status_color`); screens call the wrapper (or `tone_color(…)` on a tone) instead of an inline `match … => theme.ok/warn/err`. To color a new status enum, add its tone function there rather than a bespoke match per screen -- **[`widgets.rs`](crates/rocm-dash-tui/src/ui/widgets.rs)**'s `apply_char_key`/`apply_char_key_filtered` — the append-only text-field edit (`Char` appends, `Backspace` pops one char); call it from a form's key handler instead of writing another `push`/`pop` match arm. The `_filtered` form takes a predicate to refuse some chars (a digits-only port); there is no cursor or mid-string editing. The Chat tab's input is the one exception: its keys are first mapped to a `KeyAction` by the pure, separately tested table in [`actions.rs`](crates/rocm-dash-tui/src/app/actions.rs), which also applies them +- **[`widgets.rs`](crates/rocm-dash-tui/src/ui/widgets.rs)**'s `apply_char_key`/`apply_char_key_filtered` — the append-only text-field edit (`Char` appends, `Backspace` pops one char); call it from a form's key handler instead of writing another `push`/`pop` match arm. The `_filtered` form takes a predicate to refuse some chars (a digits-only port); there is no cursor or mid-string editing. The Chat tab's input is the one exception: its keys are first mapped to a `KeyAction` by the pure, separately tested table in [`actions.rs`](crates/rocm-dash-tui/src/app/actions.rs), which also applies them. `cargo test -p xtask dash_text_edit_contract` ([`dash_text_edit_contract.rs`](xtask/src/dash_text_edit_contract.rs)) fails on a hand-rolled `push`/`pop` text edit under `crates/rocm-dash-tui/src/` outside `widgets.rs`; a new exception is a reviewed entry in that test's `ALLOWED` list. - **[`scroll.rs`](crates/rocm-dash-tui/src/app/scroll.rs)**'s `ScrollState` (in `app/`, not `ui/`, beside the `AppState` scroll methods) — an offset plus its upper bound, whose `scroll_by(delta)` is the one "add a signed delta, clamp into `[0, max]`, saturate to `u16`" implementation. The job console (both axes), the LOGS dock and the instance Detail modal clamp through it; a new scrollable surface that keeps a `u16` offset with a known bound builds a `ScrollState` from them and stores `pos` back instead of repeating the `i32` clamp arithmetic. The chat transcript and the GPU list clamp with a `min()` against a measured bound (chat also follows the tail) and stay out of it **Two different components are both informally "the spinner"; they are not interchangeable.** The CLI's [`cli_progress.rs`](apps/rocm/src/cli_progress.rs) `Spinner`/`AnimatedSpinner` above is TTY/stderr-gated for a plain terminal and lives in `apps/rocm`. The dash-TUI's [`spinner.rs`](crates/rocm-dash-tui/src/ui/spinner.rs) above is a pure frame/percentage helper rendered through `job_console.rs` inside a ratatui frame loop, and lives in `crates/rocm-dash-tui`. A dash-TUI screen needs the latter. diff --git a/docs/architecture.md b/docs/architecture.md index 15aa2820ee..643d712536 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -298,7 +298,7 @@ The other shared `ui/` components the contributor rules in [`AGENTS.md`](../AGEN - [`ui/exec.rs`](../crates/rocm-dash-tui/src/ui/exec.rs) — the helpers every operational screen uses to launch a `rocm` sub-command through the approval gate: `resolve_exe`, the approval-preview quoting (`display_args`, `quote_display_arg`), and the `rocm runtimes adopt` argv. `adopt_args` builds that argv for both the runtime manager's Adopt action and the onboarding wizard's Adopt-existing-folder path, with `--python` set to the Python executable inside the root (`Scripts\python.exe` on Windows, `bin/python` elsewhere, the layout `rocm-core` uses, kept here because the crate may not depend on `rocm-core`). Its unit tests reach both layouts through an injectable `windows` flag. - [`ui/spinner.rs`](../crates/rocm-dash-tui/src/ui/spinner.rs) — pure helpers for the braille spinner frame (`spinner_frame`) and progress-percentage parsing (`parse_progress_pct`) that `job_console.rs` renders with. - [`ui/theme.rs`](../crates/rocm-dash-tui/src/ui/theme.rs) — the semantic `Theme`, including the shared status-to-color mapping (`StatusTone` and `tone_color`, with `job_status_color` and `instance_status_color` for the job and instance status enums). -- [`ui/widgets.rs`](../crates/rocm-dash-tui/src/ui/widgets.rs) — small helpers shared across tabs, including `apply_char_key`/`apply_char_key_filtered`, the append-only text-field edit (`Char` appends, `Backspace` pops one char) that every form's key handler calls instead of writing its own `push`/`pop` arms. The Chat tab's input is the exception: its keys go through the `KeyAction` table in [`app/actions.rs`](../crates/rocm-dash-tui/src/app/actions.rs). +- [`ui/widgets.rs`](../crates/rocm-dash-tui/src/ui/widgets.rs) — small helpers shared across tabs, including `apply_char_key`/`apply_char_key_filtered`, the append-only text-field edit (`Char` appends, `Backspace` pops one char) that every form's key handler calls instead of writing its own `push`/`pop` arms. The Chat tab's input is the exception: its keys go through the `KeyAction` table in [`app/actions.rs`](../crates/rocm-dash-tui/src/app/actions.rs). A source-scan test in `xtask` ([`dash_text_edit_contract.rs`](../xtask/src/dash_text_edit_contract.rs)) fails if another dash-TUI file hand-rolls that `push`/`pop`. The `ui/` modules the split moved code out of or into, beyond the shared components above: diff --git a/docs/testing.md b/docs/testing.md index 010bb8efb6..45dd4d47b1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -191,6 +191,70 @@ test fails on a path that no longer exists rather than dropping it from the guard. When the planner needs something that lives in `chat.rs`, move it into a leaf module and import it from there. +### Dash-TUI text fields go through `apply_char_key` + +A dash-TUI form edits its `String` through `apply_char_key` / +`apply_char_key_filtered` in +[`ui/widgets.rs`](../crates/rocm-dash-tui/src/ui/widgets.rs), not through its +own "`Char` appends, `Backspace` pops one char" code. As with the other module +rules above, a test enforces it rather than an `xtask` subcommand: +`the_dash_tui_has_no_hand_rolled_text_edit_outside_the_helper` in `xtask` +([`dash_text_edit_contract.rs`](../xtask/src/dash_text_edit_contract.rs)) scans +the non-test code of every file under `crates/rocm-dash-tui/src/` and fails, +naming each hit as `file:line` with a pointer to the helper. Run it on its own +with: + +```bash +cargo test -p xtask dash_text_edit_contract +``` + +It is per file, not per `match` arm, because the copies the helper replaced were +not all arms (one had its `push` in a `type_char` method the arm called). In a +file other than `ui/widgets.rs` it reports a no-argument `.pop()` when the file +names `KeyCode::Backspace`, and `.push(x)` (also `.push(*x)`, +`.push(x.method())` and `.push(x as char)`, each with or without a trailing +comma) when `x` is a name the file binds as a key character +(`KeyCode::Char(x)`, `KeyCode::Char(mut x)`) or declares as `x: char`, in a file +that names `KeyCode::Char(`. A `#[cfg(test)] mod name { .. }` block is skipped, +so a unit test may drive a buffer by hand; a `#[cfg(test)]` on anything else +skips nothing, and an out-of-line test module (`#[cfg(test)] mod name;`) is a +file of its own, scanned in full. + +What it does not see: it matches the shape, not the type, so a `let top = +stack.pop()` in a file that handles `Backspace` is a false alarm, and an +unrelated `out.push(c)` is reported when the same file binds `KeyCode::Char(c)` +elsewhere. A push is read only when its argument is that name alone in one of the +spellings above: `.push((x))`, `.push(char::from(x))`, a `.push(if ..)` of the +name, and a call with a gap before the paren (`.push (x)`, which rustfmt never +writes) are not seen. Only `#[cfg(test)]` exactly is read as a test gate (a +`#[cfg(all(test, ..))]` is scanned, which can only cause a false alarm), and +`use KeyCode::*` or an alias of `KeyCode` defeats the scan. A typed key passed +from one file to a method in another is not followed. Only `.pop()` and +`.push(x)` are matched: `push_str`, `truncate`, `clear`, `.insert(..)`, +`.remove(..)` and `replace_range(..)` are not scanned, partly because a cursor editor (the serve wizard's Host field, `InlineEditor` in +[`serve_wizard.rs`](../crates/rocm-dash-tui/src/ui/serve_wizard.rs)) cannot use +the append-only helper and is a different thing. A `Backspace` arm that never +pops, such as the folder browser's "go to the parent directory", is not a text +edit and is not reported. + +The one file with its own `push` and `pop` is the Chat input in +[`app/actions.rs`](../crates/rocm-dash-tui/src/app/actions.rs), whose keys are +mapped to a `KeyAction` by a separately tested table. It is listed in `ALLOWED` +in the test with its reason and the number of offenses it is expected to carry +(its one push and its one pop). The test fails when the file carries a different +number, so a further hand-rolled field added there is caught, as is an entry that +stops being needed, or whose file moves. A new field reuses the helper; a new +exception is a reviewed entry in that list. + +Where it runs: the pre-push hook (`cargo xtask pre-push-check`) selects `xtask` +for any Rust change, the `windows-debug` job runs `cargo test --workspace` on +every pull request that changes Rust, and the merge queue and post-merge runs +test the whole workspace. The Linux pull-request lane (`cargo xtask affected`) +selects `xtask` only for a change in `xtask` or in a crate it depends on, or for +a file it reads, so a pull request that touches nothing but +`crates/rocm-dash-tui` does not run this test on Linux. The module-import +contract above has the same property. + ### Tests that run a fake executable Plant a stand-in `python`, `uv`, `amd-smi`, ... with diff --git a/xtask/src/dash_text_edit_contract.rs b/xtask/src/dash_text_edit_contract.rs new file mode 100644 index 0000000000..299ed08a36 --- /dev/null +++ b/xtask/src/dash_text_edit_contract.rs @@ -0,0 +1,938 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Static contract test: a dash-TUI text field edits its `String` through +//! `apply_char_key` / `apply_char_key_filtered` in `ui/widgets.rs`, not through +//! its own "`Char` appends, `Backspace` pops one char" code. +//! +//! The helper replaced the hand-rolled copies, and the convention lives in +//! `AGENTS.md` and `docs/architecture.md`. Prose does not stop the next copy, so +//! this scan does. +//! +//! The rule is per file, not per `match` arm, because the copies were not all +//! arms: `install_manager.rs` had `KeyCode::Char(c) => i.type_char(c)` with the +//! `push` in a separate `fn type_char(&mut self, c: char)`. In the production +//! code of every file under `crates/rocm-dash-tui/src/` other than `ui/widgets.rs`: +//! +//! * a no-argument `.pop()` call in a file that names `KeyCode::Backspace` is +//! reported, and +//! * `.push(x)` (or `.push(*x)`, `.push(x.method())`, `.push(x as char)`, and each +//! of them with a trailing comma such as `.push(x,)`) is reported where `x` is +//! a name the same file binds as a key character (`KeyCode::Char(x)`, +//! `KeyCode::Char(mut x)`) or declares as a `char` parameter (`x: char`), in a +//! file that names `KeyCode::Char(`. +//! +//! Production code means what is left after [`crate::rust_source`]'s +//! `strip_literals_and_comments` blanks comments and literals and every +//! `#[cfg(test)] mod name { .. }` block is blanked too, so a unit test may drive +//! a buffer by hand. A `#[cfg(test)]` on anything else exempts nothing. An +//! out-of-line test module (`#[cfg(test)] mod x;`) is a separate file, which is +//! scanned in full. [`crate::chat_import_contract`] blanks test modules over a +//! token stream instead; this one works over the stripped text because it needs +//! only the module's extent, and the two are not shared. +//! +//! Known limits, stated rather than hidden: +//! +//! * It looks for a `String` edit by shape, not by type. A `.pop()` that takes a +//! value (`let top = stack.pop()`) in a file that handles `Backspace` is a +//! false alarm; rename it, or add the file to `ALLOWED` with the reason. +//! Bindings are per file, not per function, so an unrelated `out.push(c)` is +//! also reported when the same file binds `KeyCode::Char(c)` somewhere else. +//! * Only `.pop()` and `.push(x)` are matched, and a push only when its argument +//! is the bound name alone, as listed above: `.push((x))`, `.push(char::from(x))`, +//! `.push(if shift { x.to_ascii_uppercase() } else { x })` and a call written +//! with a gap before the paren (`.push (x)`, which rustfmt never produces) are +//! not seen. `push_str`, `truncate`, `clear`, `String::pop(&mut b)`, +//! `.insert(..)`, `.remove(..)` and `replace_range(..)` are not scanned. A +//! cursor editor such as the serve wizard's Host field (`InlineEditor` in +//! `ui/serve_wizard.rs`) cannot use the append-only helper, so it is a +//! different thing, and a second copy of one goes unseen. +//! * A file that never names `KeyCode::Char(` is not checked for a `char` +//! push. A typed key can reach such a file through a call from another one (the +//! `install_manager` `type_char` shape, with the arm in a different file), and +//! that split is not seen. `x: char` counts as a binding wherever it appears, +//! as a parameter, a field or a `let`; `x: &char` does not. +//! * Only `#[cfg(test)]` exactly is read as a test gate. `#[cfg(all(test, ..))]` +//! and a file-level `#![cfg(test)]` read as production code, which fails toward +//! a false alarm. `use KeyCode::*` or an alias of `KeyCode` defeats every +//! trigger, since the scan looks for the path `KeyCode::`. +//! * A `Backspace` handler that never pops (`ui/folder_browser.rs` navigates to the +//! parent directory) is not a text edit and is not reported, so it needs no +//! `ALLOWED` entry. +//! * A buffer edited through a wrapper that is itself not in these files is not +//! followed, and a binding renamed on the way to the `push` (`let ch = c;`) is +//! not seen. +//! * An `ALLOWED` entry pins the number of offenses in its file, so a second +//! hand-rolled edit added there fails, but it does not say which one is the +//! documented one: swapping an edit for another keeps the count. + +#[cfg(test)] +mod tests { + use crate::rust_source::strip_literals_and_comments; + use std::path::{Path, PathBuf}; + + /// The one file allowed to own the append/pop edit. + const HELPER_FILE: &str = "crates/rocm-dash-tui/src/ui/widgets.rs"; + + /// The directory the scan covers, relative to the repository root. + const SCANNED_DIR: &str = "crates/rocm-dash-tui/src"; + + /// Files that legitimately keep their own `push` / `pop`: the file, how many + /// offenses it is expected to carry (the documented ones), and the reason. + /// The scan fails when a file carries a different number, so a file that + /// stops needing its entry, and a further hand-rolled edit added next to the + /// documented one, are both caught. + const ALLOWED: &[(&str, usize, &str)] = &[( + "crates/rocm-dash-tui/src/app/actions.rs", + 2, + "the Chat input: its keys are first mapped to a `KeyAction` by a pure, \ + separately tested table, and `apply_action` then pushes and pops the \ + chat buffer from that action, so there is no `KeyCode` match to route \ + through the helper. The two offenses are that push and that pop. The \ + push is found because the file also binds `KeyCode::Char(c)` elsewhere: \ + bindings are file-wide, not followed through `KeyAction::ChatInput(c)`", + )]; + + #[derive(Clone, Debug, PartialEq, Eq)] + struct Offense { + line: usize, + what: String, + } + + fn is_ident_char(c: char) -> bool { + c.is_alphanumeric() || c == '_' + } + + /// Blank `text[from..to]` to spaces, keeping newlines so line numbers hold. + fn blank(chars: &mut [char], from: usize, to: usize) { + for c in &mut chars[from..to] { + if *c != '\n' { + *c = ' '; + } + } + } + + fn skip_ws(chars: &[char], mut i: usize) -> usize { + while chars.get(i).is_some_and(|c| c.is_whitespace()) { + i += 1; + } + i + } + + fn starts_with_at(chars: &[char], i: usize, word: &str) -> bool { + word.chars() + .enumerate() + .all(|(k, w)| chars.get(i + k) == Some(&w)) + } + + /// The index past the bracket group opening at `open`, or the end of input. + fn past_balanced(chars: &[char], open: usize) -> usize { + let mut depth = 0usize; + for (i, c) in chars.iter().enumerate().skip(open) { + match c { + '(' | '[' | '{' => depth += 1, + ')' | ']' | '}' => { + depth = depth.saturating_sub(1); + if depth == 0 { + return i + 1; + } + } + _ => {} + } + } + chars.len() + } + + /// `source` with comments, literals and every `#[cfg(test)] mod` blanked, + /// line structure preserved. + fn production_text(source: &str) -> Vec { + const GATE: &str = "#[cfg(test)]"; + let mut chars: Vec = strip_literals_and_comments(source).chars().collect(); + let mut i = 0; + while i < chars.len() { + if !starts_with_at(&chars, i, GATE) { + i += 1; + continue; + } + let mut j = skip_ws(&chars, i + GATE.len()); + // Attributes stacked under the gate, then a visibility. + loop { + if starts_with_at(&chars, j, "#[") { + j = skip_ws(&chars, past_balanced(&chars, j + 1)); + } else if starts_with_at(&chars, j, "pub") + && !chars.get(j + 3).is_some_and(|c| is_ident_char(*c)) + { + j = skip_ws(&chars, j + 3); + if chars.get(j) == Some(&'(') { + j = skip_ws(&chars, past_balanced(&chars, j)); + } + } else { + break; + } + } + let is_mod = starts_with_at(&chars, j, "mod") + && chars.get(j + 3).is_some_and(|c| c.is_whitespace()); + if !is_mod { + i += GATE.len(); + continue; + } + j = skip_ws(&chars, j + 3); + while chars.get(j).is_some_and(|c| is_ident_char(*c)) { + j += 1; + } + j = skip_ws(&chars, j); + let end = match chars.get(j) { + Some('{') => past_balanced(&chars, j), + Some(';') => j + 1, + _ => { + i += GATE.len(); + continue; + } + }; + blank(&mut chars, i, end); + i = end; + } + chars + } + + fn line_of(chars: &[char], index: usize) -> usize { + 1 + chars[..index].iter().filter(|c| **c == '\n').count() + } + + fn find_all(chars: &[char], word: &str) -> Vec { + (0..chars.len()) + .filter(|i| starts_with_at(chars, *i, word)) + .collect() + } + + /// The identifier starting at `i`, if one does. + fn ident_at(chars: &[char], i: usize) -> Option<(String, usize)> { + let first = chars.get(i)?; + if !(first.is_alphabetic() || *first == '_') { + return None; + } + let mut end = i; + while chars.get(end).is_some_and(|c| is_ident_char(*c)) { + end += 1; + } + Some((chars[i..end].iter().collect(), end)) + } + + /// Names the file binds as a typed-in character: `KeyCode::Char(x)` and + /// `x: char`. + fn char_bindings(chars: &[char]) -> Vec { + let mut names = Vec::new(); + for at in find_all(chars, "KeyCode::Char(") { + let mut i = skip_ws(chars, at + "KeyCode::Char(".len()); + // `mut c` and `ref c` bind the name `c`; the modifier is not it. + while let Some((name, end)) = ident_at(chars, i) { + if name == "mut" || name == "ref" { + i = skip_ws(chars, end); + continue; + } + names.push(name); + break; + } + } + for at in find_all(chars, "char") { + let after = at + "char".len(); + if chars.get(after).is_some_and(|c| is_ident_char(*c)) { + continue; + } + // Walk back over `: ` to the name before it. A `char` that ends a + // longer name (`mychar`) has no `:` before it and stops there. + let mut k = at; + while k > 0 && chars[k - 1].is_whitespace() { + k -= 1; + } + if k == 0 || chars[k - 1] != ':' || (k >= 2 && chars[k - 2] == ':') { + continue; + } + k -= 1; + while k > 0 && chars[k - 1].is_whitespace() { + k -= 1; + } + let end = k; + while k > 0 && is_ident_char(chars[k - 1]) { + k -= 1; + } + if k < end { + names.push(chars[k..end].iter().collect()); + } + } + names + } + + /// Whether the text after a `.push(` argument's leading name (ending at `end`) + /// closes the call with that name as the whole argument: the bare name + /// (`push(c)`), a method on it (`push(c.to_ascii_lowercase())`), a no-op cast + /// to `char` (`push(c as char)`), and any of the first and last with a + /// trailing comma (`push(c,)`, which rustfmt writes when a call spans lines). + /// A second argument (`push(c, 1)`) or a cast to another type + /// (`push(c as u8)`, a `Vec` push) is not a `String` append. + fn is_whole_push_argument(chars: &[char], end: usize) -> bool { + let mut i = skip_ws(chars, end); + if starts_with_at(chars, i, "as") && chars.get(i + 2).is_some_and(|c| c.is_whitespace()) { + let ty = skip_ws(chars, i + 2); + // Any other cast (`as u8`) leaves `i` on the `as`, which closes nothing. + if starts_with_at(chars, ty, "char") + && !chars + .get(ty + "char".len()) + .is_some_and(|c| is_ident_char(*c)) + { + i = skip_ws(chars, ty + "char".len()); + } + } + match chars.get(i) { + Some(')' | '.') => true, + Some(',') => chars.get(skip_ws(chars, i + 1)) == Some(&')'), + _ => false, + } + } + + /// Every hand-rolled append or pop in `source`'s production code. + /// + /// Takes the raw file text, so a caller cannot forget to strip comments, + /// literals and test modules first. + fn hand_rolled_edits(source: &str) -> Vec { + let chars = production_text(source); + let mut offenses = Vec::new(); + let names_backspace = !find_all(&chars, "KeyCode::Backspace").is_empty(); + if names_backspace { + for at in find_all(&chars, ".pop(") { + let close = skip_ws(&chars, at + ".pop(".len()); + if chars.get(close) == Some(&')') { + offenses.push(Offense { + line: line_of(&chars, at), + what: "a `.pop()` in a file that handles `KeyCode::Backspace`".to_owned(), + }); + } + } + } + if !find_all(&chars, "KeyCode::Char(").is_empty() { + let bound = char_bindings(&chars); + for at in find_all(&chars, ".push(") { + let mut arg = skip_ws(&chars, at + ".push(".len()); + // `push(*c)` when the key is matched by reference. + while matches!(chars.get(arg), Some('*' | '&')) { + arg = skip_ws(&chars, arg + 1); + } + let Some((name, end)) = ident_at(&chars, arg) else { + continue; + }; + if is_whole_push_argument(&chars, end) && bound.contains(&name) { + offenses.push(Offense { + line: line_of(&chars, at), + what: format!("`.push({name})` of a typed-in character"), + }); + } + } + } + offenses.sort_by_key(|o| o.line); + offenses + } + + fn repo_root() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("xtask crate has a parent directory") + .to_path_buf() + } + + /// Every `.rs` file under the dash-TUI `src/`, as `(repo-relative `/` path, + /// text)`. + fn dash_tui_sources(root: &Path) -> Vec<(String, String)> { + fn walk(root: &Path, dir: &Path, out: &mut Vec<(String, String)>) { + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + walk(root, &path, out); + } else if path.extension().is_some_and(|ext| ext == "rs") { + let relative = path + .strip_prefix(root) + .expect("under the root") + .to_string_lossy() + .replace('\\', "/"); + let text = std::fs::read_to_string(&path) + .unwrap_or_else(|err| panic!("cannot read {relative}: {err}")); + out.push((relative, text)); + } + } + } + let mut found = Vec::new(); + walk(root, &root.join(SCANNED_DIR), &mut found); + found.sort(); + found + } + + /// The failure lines for `files`: each offense outside the helper file and + /// the allowlist, and each allowlist entry that no longer needs its place. + fn check(files: &[(String, String)]) -> Vec { + let mut failures = Vec::new(); + for (path, source) in files { + if path == HELPER_FILE { + continue; + } + let offenses = hand_rolled_edits(source); + let Some((_, expected, _)) = ALLOWED.iter().find(|(file, _, _)| file == path) else { + for o in offenses { + failures.push(format!( + "{path}:{}: {}. A text field edits its buffer with `apply_char_key` or \ + `apply_char_key_filtered` from ui/widgets.rs, not its own push/pop", + o.line, o.what + )); + } + continue; + }; + let lines = offenses + .iter() + .map(|o| o.line.to_string()) + .collect::>() + .join(", "); + if offenses.is_empty() { + failures.push(format!( + "ALLOWED entry `{path}` is stale: it no longer hand-rolls a text edit. \ + Remove the entry, or the exception outlives its reason" + )); + } else if offenses.len() > *expected { + failures.push(format!( + "ALLOWED entry `{path}` documents {expected} hand-rolled edit(s) and the \ + file now has {} (lines {lines}). A further text field belongs on \ + `apply_char_key`; raise the count only for a reviewed exception", + offenses.len() + )); + } else if offenses.len() < *expected { + failures.push(format!( + "ALLOWED entry `{path}` documents {expected} hand-rolled edit(s) but the \ + file now has {} (lines {lines}): part of the exception is gone. Lower \ + the count to match, or remove the entry if none is left", + offenses.len() + )); + } + } + for (file, _, _) in ALLOWED { + if !files.iter().any(|(path, _)| path == file) { + failures.push(format!( + "ALLOWED entry `{file}` names a file that does not exist; update it in \ + xtask/src/dash_text_edit_contract.rs" + )); + } + } + failures + } + + /// A file with one append and one pop: the shape of the Chat input's pair. + const TWO_EDITS: &str = "\ +fn push(c: KeyCode, q: &mut String) { if let KeyCode::Char(x) = c { q.push(x); } } +fn pop(c: KeyCode, q: &mut String) { if c == KeyCode::Backspace { q.pop(); } } +"; + + /// The failure lines that are about `name`, so a test about one file does + /// not depend on every other file in the tree being clean. + fn in_file<'a>(failures: &'a [String], name: &str) -> Vec<&'a String> { + failures.iter().filter(|f| f.contains(name)).collect() + } + + fn file(path: &str, source: &str) -> (String, String) { + (path.to_owned(), source.to_owned()) + } + + fn lines(offenses: &[Offense]) -> Vec { + offenses.iter().map(|o| o.line).collect() + } + + #[test] + fn the_dash_tui_has_no_hand_rolled_text_edit_outside_the_helper() { + let files = dash_tui_sources(&repo_root()); + let failures = check(&files); + assert!( + failures.is_empty(), + "dash-TUI text fields must edit through the shared helper:\n {}", + failures.join("\n ") + ); + } + + #[test] + fn the_scan_reaches_the_helper_and_the_converted_modules() { + let files = dash_tui_sources(&repo_root()); + for needed in [ + HELPER_FILE, + "crates/rocm-dash-tui/src/ui/logs_view.rs", + "crates/rocm-dash-tui/src/ui/install_manager.rs", + "crates/rocm-dash-tui/src/app/actions.rs", + "crates/rocm-dash-tui/src/ui/folder_browser.rs", + ] { + assert!( + files.iter().any(|(path, _)| path == needed), + "the walk did not reach {needed}" + ); + } + // The helper itself is the shape being banned elsewhere, so it must + // read as an offender here: that is what the exemption is for. Its count + // is not pinned (`check` skips the helper file, so nothing else depends on + // it, and an unrelated edit to `widgets.rs` must not fail this test): it + // must show a push and a pop, the two halves of the edit it implements. + let helper = &files.iter().find(|(p, _)| p == HELPER_FILE).unwrap().1; + let seen: Vec = hand_rolled_edits(helper) + .into_iter() + .map(|offense| offense.what) + .collect(); + assert!( + seen.iter().any(|what| what.contains(".push(")), + "the scan no longer sees the helper's push: {seen:?}" + ); + assert!( + seen.iter().any(|what| what.contains(".pop()")), + "the scan no longer sees the helper's pop: {seen:?}" + ); + } + + #[test] + fn every_allowed_file_carries_a_reason() { + for (file, expected, reason) in ALLOWED { + assert!(*expected > 0, "ALLOWED entry `{file}` expects no offenses"); + assert!( + reason.len() > 40, + "ALLOWED entry `{file}` has no real justification: {reason:?}" + ); + } + } + + #[test] + fn a_hand_rolled_arm_is_reported_with_its_line() { + let source = "\ +fn on_key(l: &mut Logs, key: KeyEvent) { + match key.code { + KeyCode::Char(c) => l.query.push(c), + KeyCode::Backspace => { + l.query.pop(); + } + _ => {} + } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![3, 5]); + } + + #[test] + fn the_helper_split_shape_that_the_arm_scan_would_miss_is_reported() { + // install_manager.rs before the shared helper: the arms call methods, + // and the push/pop live in those methods. + let source = "\ +fn on_key(i: &mut Install, key: KeyEvent) { + match key.code { + KeyCode::Backspace => i.backspace(), + KeyCode::Char(ch) => i.type_char(ch), + _ => {} + } +} +impl Install { + fn type_char(&mut self, c: char) { + match self.field { + Field::Channel => self.channel.push(c), + _ => {} + } + } + fn backspace(&mut self) { + match self.field { + Field::Channel => { + self.channel.pop(); + } + _ => {} + } + } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![11, 18]); + } + + #[test] + fn a_char_parameter_is_a_binding_even_when_no_arm_names_it() { + // Only `x: char` binds `c` here; `KeyCode::Char('x')` just makes this a + // file that handles typed keys. + let source = "\ +fn on_key(code: KeyCode, i: &mut Install) { + if code == KeyCode::Char('x') { i.clear_channel(); } +} +fn type_char(q: &mut String, c: char) { + q.push(c); +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![5]); + } + + #[test] + fn a_binding_by_mut_or_ref_or_a_pushed_deref_or_method_is_reported() { + // One file each: a name bound by any form is bound for the whole file, + // so a shared `c` would let one form's binding cover for another's. + for source in [ + "fn a(code: KeyCode, q: &mut String) {\n if let KeyCode::Char(mut c) = code { q.push(c); }\n}\n", + "fn b(code: &KeyCode, q: &mut String) {\n if let KeyCode::Char(c) = code { q.push(*c); }\n}\n", + "fn d(code: KeyCode, q: &mut String) {\n if let KeyCode::Char(ref c) = code { q.push(c.to_ascii_lowercase()); }\n}\n", + ] { + assert_eq!(lines(&hand_rolled_edits(source)), vec![2], "{source}"); + } + } + + #[test] + fn binding_names_are_whole_identifiers_not_single_letters() { + // `key_char` carries an underscore and `mychar` ends in the word `char`: + // only the first is a char parameter, and only it is pushed. + let source = "\ +fn on_key(code: KeyCode, i: &mut Install) { + if code == KeyCode::Char('x') { i.clear_channel(); } +} +fn typed(q: &mut String, key_char: char) { + q.push(key_char); +} +fn bytes(out: &mut Vec, mychar: u8) { + out.push(mychar); +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![5]); + } + + #[test] + fn a_push_through_two_dereferences_is_reported() { + let source = "\ +fn a(code: &&KeyCode, q: &mut String) { + if let KeyCode::Char(c) = code { q.push(**c); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![2]); + } + + /// A key arm that appends `shape` to a `String`, wrapped in a function so the + /// whole file is a valid fixture for [`hand_rolled_edits`]. + fn key_arm_pushing(shape: &str) -> String { + format!( + "fn f(code: KeyCode, q: &mut String) {{\n if let KeyCode::Char(c) = code {{\n \ + {shape};\n }}\n}}\n" + ) + } + + #[test] + fn a_push_with_a_trailing_comma_is_reported() { + // `q.push(c,)` is a valid call, and the multi-line form below is what + // rustfmt writes when a call is too long for one line. Both append a + // typed-in character, so both are offenses. + let one_line = key_arm_pushing("q.push(c,)"); + assert_eq!(lines(&hand_rolled_edits(&one_line)), vec![3], "{one_line}"); + let multi_line = key_arm_pushing("q.push(\n c,\n )"); + assert_eq!( + lines(&hand_rolled_edits(&multi_line)), + vec![3], + "{multi_line}" + ); + } + + #[test] + fn the_other_spellings_of_a_pushed_typed_in_char_are_reported() { + let mut missed = Vec::new(); + for (shape, call) in [ + ("spaces around the comma", "q.push( c , )"), + ("a dereference and a comma", "q.push(*c,)"), + ("a method and a comma", "q.push(c.to_ascii_lowercase(),)"), + ("a comment before the paren", "q.push(c /* typed */)"), + ("a cast to char", "q.push(c as char)"), + ("a cast to char and a comma", "q.push(c as char,)"), + ( + "the argument on its own line", + "q.push(\n c\n )", + ), + ] { + let source = key_arm_pushing(call); + if hand_rolled_edits(&source).len() != 1 { + missed.push(shape); + } + } + assert!(missed.is_empty(), "not reported: {missed:?}"); + } + + #[test] + fn a_push_with_more_than_the_typed_in_char_is_not_reported() { + // Not a `String` append of the typed key: two arguments, or a cast to a + // type a `String` cannot take. + for (shape, call) in [ + ("two arguments", "q.push(c, 1)"), + ("two arguments over lines", "q.push(c,\n 1)"), + ("a cast to a byte", "q.push(c as u8,)"), + ("a cast to a byte", "q.push(c as u8)"), + ] { + let source = key_arm_pushing(call); + assert!(hand_rolled_edits(&source).is_empty(), "{shape}: {source}"); + } + } + + #[test] + fn a_pop_with_an_argument_and_a_push_of_a_computed_value_are_ignored() { + let source = "\ +fn f(code: KeyCode, history: &mut History, bytes: &mut Vec) { + if code == KeyCode::Backspace { history.pop(2); } + if let KeyCode::Char(c) = code { bytes.push(c as u8); } +} +"; + assert!(hand_rolled_edits(source).is_empty()); + } + + #[test] + fn a_push_of_something_that_is_not_a_typed_in_char_is_ignored() { + let source = "\ +fn on_key(key: KeyEvent, out: &mut Vec, label: &str) { + match key.code { + KeyCode::Char('x') => out.push(label.to_owned()), + KeyCode::Char(c) if c.is_ascii_digit() => out.push(format!(\"{c}\")), + _ => {} + } + out.push(label); +} +"; + assert!(hand_rolled_edits(source).is_empty()); + } + + #[test] + fn a_pop_or_push_in_a_file_that_handles_no_text_key_is_ignored() { + // `exec.rs` and similar pop a Vec and render chars; neither is a field. + let source = "\ +fn pad(out: &mut String, c: char) { + out.push(c); +} +fn last(words: &mut Vec) -> Option { + words.pop() +} +"; + assert!(hand_rolled_edits(source).is_empty()); + } + + #[test] + fn a_pop_that_takes_a_value_is_still_reported_in_a_backspace_file() { + // The scan cannot tell a `Vec` pop from a `String` pop; the allowlist + // is the way out, documented in the module comment. + let source = "\ +fn f(stack: &mut Vec, code: KeyCode) { + if code == KeyCode::Backspace { + let _ = stack.pop(); + } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![3]); + } + + #[test] + fn code_inside_a_test_module_is_allowed_and_code_around_it_is_not() { + let source = "\ +fn on_key(code: KeyCode, q: &mut String) { + if let KeyCode::Char(c) = code { q.push(c); } +} +#[cfg(test)] +mod tests { + fn drive(code: KeyCode, q: &mut String) { + match code { + KeyCode::Char(c) => q.push(c), + KeyCode::Backspace => { q.pop(); } + _ => {} + } + } +} +fn late(code: KeyCode, q: &mut String) { + if code == KeyCode::Backspace { q.pop(); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![2, 15]); + } + + #[test] + fn a_test_module_with_stacked_attributes_and_a_visibility_is_still_a_test_module() { + let source = "\ +#[cfg(test)] +#[allow(dead_code)] +pub(crate) mod helpers { + fn drive(code: KeyCode, q: &mut String) { + if let KeyCode::Char(c) = code { q.push(c); } + } +} +#[cfg(test)] +pub mod more { + fn drive(code: KeyCode, q: &mut String) { + if code == KeyCode::Backspace { q.pop(); } + } +} +fn late(code: KeyCode, q: &mut String) { + if code == KeyCode::Backspace { q.pop(); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![15]); + } + + #[test] + fn an_out_of_line_test_module_declaration_leaves_the_rest_of_the_file_scanned() { + let source = "\ +#[cfg(test)] +mod tests; +fn late(code: KeyCode, q: &mut String) { + if code == KeyCode::Backspace { q.pop(); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![4]); + } + + #[test] + fn a_test_gate_on_a_function_exempts_nothing() { + let source = "\ +#[cfg(test)] +fn on_key(code: KeyCode, q: &mut String) { + if let KeyCode::Char(c) = code { q.push(c); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![3]); + } + + #[test] + fn a_test_module_ends_at_its_own_brace_not_at_one_in_a_literal() { + let source = "\ +#[cfg(test)] +mod tests { + fn s() -> &'static str { \"}}\" } +} +fn on_key(code: KeyCode, q: &mut String) { + if let KeyCode::Char(c) = code { q.push(c); } +} +"; + assert_eq!(lines(&hand_rolled_edits(source)), vec![6]); + } + + #[test] + fn mentions_in_comments_and_strings_are_not_edits() { + let source = "\ +// KeyCode::Backspace pops with q.pop(); +/// KeyCode::Char(c) pushes with q.push(c). +fn f(code: KeyCode) -> &'static str { + let _ = KeyCode::Backspace; + \"q.pop(); KeyCode::Char(c) q.push(c)\" +} +"; + assert!(hand_rolled_edits(source).is_empty()); + } + + #[test] + fn the_failure_names_the_file_the_line_and_the_helper() { + let files = [ + file( + "crates/rocm-dash-tui/src/ui/new_form.rs", + "fn k(c: KeyCode, q: &mut String) {\n if let KeyCode::Char(x) = c { q.push(x); }\n}\n", + ), + file(ALLOWED[0].0, TWO_EDITS), + ]; + let failures = check(&files); + assert_eq!(failures.len(), 1, "{failures:?}"); + assert!(failures[0].starts_with("crates/rocm-dash-tui/src/ui/new_form.rs:2:")); + assert!(failures[0].contains("apply_char_key")); + } + + #[test] + fn the_helper_file_and_an_allowed_file_are_not_reported() { + let files = [file(HELPER_FILE, TWO_EDITS), file(ALLOWED[0].0, TWO_EDITS)]; + assert_eq!(check(&files), Vec::::new()); + } + + #[test] + fn an_allowed_file_that_loses_part_of_its_exception_fails_with_the_right_advice() { + // One of the two documented edits is gone: the count is stale, and the + // advice must be to lower it, not to move a further field onto the helper. + let one = TWO_EDITS.lines().next().unwrap().to_owned() + "\n"; + let failures = check(&[file(ALLOWED[0].0, &one)]); + assert_eq!(failures.len(), 1, "{failures:?}"); + assert!(failures[0].contains("documents 2"), "{failures:?}"); + assert!(failures[0].contains("now has 1"), "{failures:?}"); + assert!( + failures[0].contains("part of the exception is gone"), + "{failures:?}" + ); + assert!(!failures[0].contains("further text field"), "{failures:?}"); + } + + #[test] + fn a_further_edit_in_an_allowed_file_fails() { + let three = format!( + "{TWO_EDITS}fn more(c: KeyCode, q: &mut String) {{ if let KeyCode::Char(x) = c {{ q.push(x); }} }}\n" + ); + let failures = check(&[file(ALLOWED[0].0, &three)]); + assert_eq!(failures.len(), 1, "{failures:?}"); + assert!(failures[0].contains("documents 2"), "{failures:?}"); + assert!(failures[0].contains("now has 3"), "{failures:?}"); + assert!(failures[0].contains("further text field"), "{failures:?}"); + assert!(failures[0].contains("apply_char_key"), "{failures:?}"); + } + + #[test] + fn an_allowed_file_that_stops_hand_rolling_fails() { + let files = [file(ALLOWED[0].0, "fn nothing() {}\n")]; + let failures = check(&files); + assert_eq!(failures.len(), 1, "{failures:?}"); + assert!(failures[0].contains("is stale")); + } + + #[test] + fn an_allowed_file_that_no_longer_exists_fails() { + let failures = check(&[]); + assert_eq!(failures.len(), 1, "{failures:?}"); + assert!(failures[0].contains("does not exist")); + } + + #[test] + fn the_guard_finds_a_hand_rolled_arm_added_to_a_converted_module() { + // Liveness on real files: the fixtures prove the scan on small strings; + // this proves it on a file it has to get through, in both of the + // spellings the helper replaced. + let files = dash_tui_sources(&repo_root()); + let logs = "crates/rocm-dash-tui/src/ui/logs_view.rs"; + let original = &files.iter().find(|(p, _)| p == logs).unwrap().1; + let helper_call = "apply_char_key(&mut l.query, code);"; + assert!(original.contains(helper_call), "the mutation target moved"); + assert!(hand_rolled_edits(original).is_empty(), "baseline is clean"); + + let inline = original.replace( + helper_call, + "match code { KeyCode::Char(c) => l.query.push(c), \ + KeyCode::Backspace => { l.query.pop(); } _ => {} }", + ); + let mut mutated = files.clone(); + mutated.iter_mut().find(|(p, _)| p == logs).unwrap().1 = inline; + let all = check(&mutated); + let failures = in_file(&all, "logs_view.rs"); + assert_eq!(failures.len(), 2, "{failures:?}"); + + let appended = format!( + "{original}\nfn late(code: KeyCode, q: &mut String) {{\n \ + if code == KeyCode::Backspace {{ q.pop(); }}\n}}\n" + ); + let mut mutated = files.clone(); + mutated.iter_mut().find(|(p, _)| p == logs).unwrap().1 = appended; + let all = check(&mutated); + let failures = in_file(&all, "logs_view.rs"); + assert_eq!( + failures.len(), + 1, + "after the file's test module: {failures:?}" + ); + + // The same edits inside the file's real `mod tests` are test code. + let in_tests = original.replacen( + "mod tests {", + "mod tests {\n fn drive(code: KeyCode, q: &mut String) {\n \ + match code { KeyCode::Char(c) => q.push(c), \ + KeyCode::Backspace => { q.pop(); } _ => {} }\n }", + 1, + ); + assert_ne!(&in_tests, original, "the test module moved"); + let mut mutated = files; + mutated.iter_mut().find(|(p, _)| p == logs).unwrap().1 = in_tests; + let all = check(&mutated); + let failures = in_file(&all, "logs_view.rs"); + assert!(failures.is_empty(), "{failures:?}"); + } +} diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 404facf47e..58a5066136 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -18,6 +18,7 @@ mod catalog; mod chat_import_contract; mod coverage; mod crate_edges; +mod dash_text_edit_contract; mod demos; mod e2e; mod e2e_prewarm; From d46768a1d80522b9bc3419643f8fd15eee09ffd7 Mon Sep 17 00:00:00 2001 From: Haribabu G Date: Mon, 9 Nov 2026 16:52:24 +0000 Subject: [PATCH 027/102] chore: add security scanning workflows (#461) * chore: add security scanning workflows Copies standardised security scanning config from ROCm/rocm-repo-template. Signed-off-by: juhovainio * chore: update dependabot.yml from rocm-repo-template Syncs .github/dependabot.yml with the latest version from ROCm/rocm-repo-template. Signed-off-by: juhovainio * chore: sync security scanning workflows from template Syncs 6 security scanning file(s) with the latest version from ROCm/rocm-repo-template. Signed-off-by: juhovainio * chore: update security scanning workflows Updates 8 security scanning file(s) to the latest version from ROCm/rocm-repo-template. Signed-off-by: juhovainio * chore: pin security baseline to v1.0.8 Pins the 'ROCm/rocm-security-gh' security baseline from the floating 'v1' tag to v1.0.8 (35d3060c4b2ed48f950f85bd414a1302a431fc0b) and removes the zizmor unpinned-uses allow list entry for it. 3 file(s) changed. Signed-off-by: juhovainio * ci: restore dependabot ecosystems, scan ready PRs, align weekly scan permissions Restore the cargo and pre-commit Dependabot entries, the cooldown and rebase-strategy settings, and the sha2 major-version ignore that the template sync overwrote. Run the PR security scan on ready_for_review, and grant the weekly scan the same actions: read the PR scan has. Signed-off-by: juhovainio --------- Signed-off-by: juhovainio Co-authored-by: haribabug Co-authored-by: juhovainio (cherry picked from commit 9ea181e9be34ac7f1aa959be7580fed8a1ee85b6) Signed-off-by: juhovainio --- .github/scan_tools_configs/.gitleaksignore | 13 ++++ .github/scan_tools_configs/.trivyignore | 19 +++++ .github/scan_tools_configs/bandit.yml | 32 +++++++++ .github/scan_tools_configs/gitleaks.toml | 82 ++++++++++++++++++++++ .github/scan_tools_configs/trivy.yml | 38 ++++++++++ .github/scan_tools_configs/zizmor.yml | 28 ++++++++ .github/workflows/pr-security-scan.yml | 42 +++++++++++ .github/workflows/weekly-security-scan.yml | 35 +++++++++ 8 files changed, 289 insertions(+) create mode 100644 .github/scan_tools_configs/.gitleaksignore create mode 100644 .github/scan_tools_configs/.trivyignore create mode 100644 .github/scan_tools_configs/bandit.yml create mode 100644 .github/scan_tools_configs/gitleaks.toml create mode 100644 .github/scan_tools_configs/trivy.yml create mode 100644 .github/scan_tools_configs/zizmor.yml create mode 100644 .github/workflows/pr-security-scan.yml create mode 100644 .github/workflows/weekly-security-scan.yml diff --git a/.github/scan_tools_configs/.gitleaksignore b/.github/scan_tools_configs/.gitleaksignore new file mode 100644 index 0000000000..1e3cd3c5f5 --- /dev/null +++ b/.github/scan_tools_configs/.gitleaksignore @@ -0,0 +1,13 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# Reference: https://github.com/gitleaks/gitleaks#gitleaksignore +# +# One finding fingerprint per line, copied from a gitleaks report +# (`:::`). A line starting with # is a +# comment and suppresses nothing. Remove the leading # to activate an +# example: +# +# 3f2a9c1e...:tests/fixtures/creds.py:generic-api-key:12 +# +# Entries here combine with a .gitleaksignore at the repository root. diff --git a/.github/scan_tools_configs/.trivyignore b/.github/scan_tools_configs/.trivyignore new file mode 100644 index 0000000000..26da6acf4d --- /dev/null +++ b/.github/scan_tools_configs/.trivyignore @@ -0,0 +1,19 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# Reference: https://trivy.dev/latest/docs/configuration/filtering/#trivyignore +# +# One vulnerability, misconfiguration, or secret rule ID per line. +# A line starting with # is a comment and suppresses nothing. Remove the +# leading # to activate an example: +# +# CVE-2019-14697 exp:2027-01-01 +# AVD-DS-0002 +# generic-unwanted-rule +# +# `exp:YYYY-MM-DD` limits a suppression to that date. After it, trivy +# reports the finding again. +# +# This file sits beside the configured trivy.yml, so the scan uses it. +# A .trivyignore at the repository root applies only when this file is +# absent; the two are never merged. diff --git a/.github/scan_tools_configs/bandit.yml b/.github/scan_tools_configs/bandit.yml new file mode 100644 index 0000000000..aa800efe64 --- /dev/null +++ b/.github/scan_tools_configs/bandit.yml @@ -0,0 +1,32 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# bandit configuration +# Reference: https://bandit.readthedocs.io/en/latest/config.html +# Local scan: bandit --recursive . --configfile .github/scan_tools_configs/bandit.yml +# +# Bandit configuration for this repository's security scans. +# +# This file defines the complete scanner policy; +# Keep repository-specific suppressions and tuning here. +# +# Bandit has no ignore file. Suppress a finding with: +# +# 1. `# nosec ` on the flagged line: +# subprocess.run(cmd, shell=True) # nosec B602 # cmd is a constant +# +# 2. `skips:` for test IDs disabled across the repository: +# skips: [B101] +# +# 3. `exclude_dirs:` below for paths Bandit should not scan. + +# Environment and generated-output exclusions support local scans; these +# paths are absent from clean CI checkouts. +exclude_dirs: + - .git + - .venv* + - venv* + - build + - dist + - node_modules + - __pycache__ diff --git a/.github/scan_tools_configs/gitleaks.toml b/.github/scan_tools_configs/gitleaks.toml new file mode 100644 index 0000000000..f50c51eb9d --- /dev/null +++ b/.github/scan_tools_configs/gitleaks.toml @@ -0,0 +1,82 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# gitleaks.toml +# Reference: https://github.com/gitleaks/gitleaks/blob/master/config/gitleaks.toml +# Local scan: gitleaks detect --source . --config .github/scan_tools_configs/gitleaks.toml +# +# Gitleaks configuration for this repository's security scans. +# +# This file defines the repository's complete Gitleaks policy. Recurring +# exclusions belong in this file. +# +# The `[extend]` section inherits Gitleaks' built-in rules, and the +# `[[allowlists]]` below apply to every rule. +# +# Ways to suppress a finding, narrowest first: +# +# 1. `gitleaks:allow` on the offending line: +# api_key = "not-a-real-key" # gitleaks:allow +# +# 2. `.gitleaksignore` with one finding fingerprint per line: +# 3f2a9c1e...:tests/fixtures/creds.py:generic-api-key:12 +# Reference: https://github.com/gitleaks/gitleaks#gitleaksignore +# Keep it beside this file (`.github/scan_tools_configs/.gitleaksignore`) +# or at the repository root. When both exist, their entries combine. +# The `.gitleaksignore` here is a commented placeholder. +# +# 3. `[[allowlists]]` below for recurring paths, regexes or stopwords. + +title = "gitleaks config" + +[extend] +# Inherit gitleaks' built-in ruleset (rotations, AWS, GCP, Slack, ...). +# Setting this to false would mean rolling our own detections, which +# is almost never what you want. +useDefault = true + + + +# Allowlist field-vs-target reference: +# stopwords -> always matched against the SECRET value. +# regexes -> matched against `regexTarget` (default: "secret"). +# regexTarget -> "secret" | "match" | "line". +# Use "match" or "line" when the "tell" is the variable name or +# surrounding syntax, not the value itself. + +[[allowlists]] +# Allowlist by location only where a real first-party secret structurally +# can't live. Dependency lock files qualify: their contents are generated +# from a manifest, and the high-entropy strings they carry are artifact +# digests rather than credentials. Add vendored third-party trees here as +# callers bring them in, one explicit path per entry. +description = "Generated dependency lock files" +paths = [ + '''.*\.lock$''', +] + +[[allowlists]] +description = "Test fixtures using well-known dummy variable names" +# The "tell" here is the variable name on the same line, not the value +# (the value is a random-looking string by design). Match against the +# full rule match (variable + value) so we anchor on the identifier. +regexTarget = "match" +regexes = [ + '''(?i)\bINVALID_TOKEN\s*=''', + '''(?i)\bwrong_secret\s*=''', + # Explicit test-only secret identifiers + '''(?i)\btest_secret\s*=''', + '''(?i)\bdummy_secret\s*=''', + '''(?i)\bfake_secret\s*=''', + '''(?i)\bexample_secret\s*=''' +] + +[[allowlists]] +description = "Public artifact digests (variable names ending in SHA256/512/MD5)" +# SHA-256/512/MD5 hashes of public artifacts are not secrets. +regexTarget = "line" +regexes = [ + '''[A-Z][A-Z0-9_]*SHA256\s*[=:]''', + '''[A-Z][A-Z0-9_]*SHA512\s*[=:]''', + '''[A-Z][A-Z0-9_]*MD5\s*[=:]''', +] diff --git a/.github/scan_tools_configs/trivy.yml b/.github/scan_tools_configs/trivy.yml new file mode 100644 index 0000000000..579a2de3c3 --- /dev/null +++ b/.github/scan_tools_configs/trivy.yml @@ -0,0 +1,38 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# trivy configuration +# Reference: https://trivy.dev/latest/docs/references/configuration/config-file/ +# Local scan: trivy fs --config .github/scan_tools_configs/trivy.yml . +# +# Trivy configuration for this repository's security scans. +# +# This file defines the complete scanner policy; +# Keep repository-specific suppressions and tuning here. +# +# Ways to suppress a finding, narrowest first: +# +# 1. `.trivyignore` -- one vulnerability, misconfiguration or secret rule +# ID per line. Add `exp:YYYY-MM-DD` for a temporary suppression: +# CVE-2019-14697 exp:2027-01-01 +# Reference: https://trivy.dev/latest/docs/configuration/filtering/#trivyignore +# Keep it beside this file (`.github/scan_tools_configs/.trivyignore`) +# or at the repository root. The file beside this config takes +# precedence; the two are never merged. The `.trivyignore` here is +# a commented placeholder. +# +# 2. `scan.skip-dirs` / `scan.skip-files` below -- paths trivy never +# reads, for vendored trees and generated output. Anything found only +# there is not scanned at all, so keep entries narrow. + +# Environment and generated-output exclusions support local scans; these +# paths are absent from clean CI checkouts. +scan: + skip-dirs: + - .git + - .venv* + - venv* + - build + - dist + - node_modules + - __pycache__ diff --git a/.github/scan_tools_configs/zizmor.yml b/.github/scan_tools_configs/zizmor.yml new file mode 100644 index 0000000000..16b1f501a6 --- /dev/null +++ b/.github/scan_tools_configs/zizmor.yml @@ -0,0 +1,28 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# zizmor configuration +# Reference: https://docs.zizmor.sh/configuration/ +# Local scan: zizmor --config .github/scan_tools_configs/zizmor.yml . +# +# Zizmor configuration for this repository's security scans. +# +# This file defines the complete scanner policy; +# Keep repository-specific suppressions and tuning here. +# +# Zizmor has no ignore file. Suppress a finding with: +# +# 1. An inline comment naming the rule: +# run: | # zizmor: ignore[template-injection] input is validated +# +# 2. `rules..ignore` with `[:[:]]`: +# rules: +# template-injection: +# ignore: +# - safe.yml +# +# 3. `rules..disable: true` to disable a rule repository-wide. +# +# Add rule-specific settings and suppressions under `rules:`. + +rules: {} diff --git a/.github/workflows/pr-security-scan.yml b/.github/workflows/pr-security-scan.yml new file mode 100644 index 0000000000..62560d492e --- /dev/null +++ b/.github/workflows/pr-security-scan.yml @@ -0,0 +1,42 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# Pre-commit security gate: fast, PR-side security scanners that run on +# every pull request. Scanner selection and policy are defined by the pinned +# ROCm/rocm-security-gh `security-baseline.yml` reusable workflow. +# +# Requests human-readable reports only: findings are uploaded as a build +# artifact and printed to the job summary, never to the Security tab, so +# fork PRs (which never receive elevated tokens) behave identically to +# same-repo PRs. See the scheduled counterpart (weekly-security-scan.yml) +# for the SARIF / code-scanning upload. + +name: Security scan (PR) + +on: + pull_request: + # The default types plus ready_for_review, so a draft marked ready without + # a new push still gets scanned. + types: [opened, synchronize, reopened, ready_for_review] + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + security: + name: Security scan + permissions: + actions: read + contents: read + pull-requests: write + uses: ROCm/rocm-security-gh/.github/workflows/security-baseline.yml@35d3060c4b2ed48f950f85bd414a1302a431fc0b # v1.0.8 + with: + report_formats: human + bandit_config_path: .github/scan_tools_configs/bandit.yml + gitleaks_config_path: .github/scan_tools_configs/gitleaks.toml + trivy_config_path: .github/scan_tools_configs/trivy.yml + zizmor_config_path: .github/scan_tools_configs/zizmor.yml diff --git a/.github/workflows/weekly-security-scan.yml b/.github/workflows/weekly-security-scan.yml new file mode 100644 index 0000000000..a78036ddaa --- /dev/null +++ b/.github/workflows/weekly-security-scan.yml @@ -0,0 +1,35 @@ +# Copyright Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT +# +# Weekly security scan: runs on a fixed cadence and pushes SARIF findings +# to the repository's code-scanning Security tab. Complements +# `pr-security-scan.yml`, which scans only what changed in a PR and +# uploads reviewer-readable artifacts for in-review browsing. + +name: Security scan (Weekly) + +on: + # Run every Saturday at 10:00 UTC. + schedule: + - cron: "0 10 * * 6" + workflow_dispatch: + +permissions: + contents: read + +jobs: + security: + name: Security scan + # Only the job that uploads SARIF gets write access to code scanning. + permissions: + actions: read + contents: read + security-events: write + uses: ROCm/rocm-security-gh/.github/workflows/security-baseline.yml@35d3060c4b2ed48f950f85bd414a1302a431fc0b # v1.0.8 + with: + scan_mode: all + report_formats: sarif + bandit_config_path: .github/scan_tools_configs/bandit.yml + gitleaks_config_path: .github/scan_tools_configs/gitleaks.toml + trivy_config_path: .github/scan_tools_configs/trivy.yml + zizmor_config_path: .github/scan_tools_configs/zizmor.yml From f5b581330be1e66e20c693969c7c083b251b3b58 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 17:09:42 +0000 Subject: [PATCH 028/102] chore(xtask): drop the exemption mechanism from check-architecture-doc (#967) * chore(xtask): remove the exemption mechanism from check-architecture-doc With every production file documented, EXEMPT is empty and EXEMPT_CEILING is 0, so the list, its ceiling and the logic that validates them are dead weight. A source file with no entry in docs/architecture.md now simply fails the check, as in check-module-size, which has no exemption mechanism either. check() loses its exempt argument and the stale-entry loop, and the ceiling_violation helper goes. Six tests that existed only to exercise the list are removed: the exempt-file, stale-entry and redundant-entry cases, the ceiling test and the sorted-and-unique test. The tests that cover what the check is for stay, including the one that runs it against the real tree. architecture.md, CONTRIBUTING.md, AGENTS.md and testing.md said the list was empty and capped; they now say there is no exemption list. Signed-off-by: Jussi Elo * docs(xtask): fix leftovers from removing the architecture-doc exemption list The command's --help text still said the check fails when an exemption list has gone stale, a condition that no longer exists. Also re-wrap the module doc line and the testing.md paragraph that the removal left ragged, say in testing.md that there is no exemption list as the other docs do, and fold the repeated "a file without an entry fails it" sentences in CONTRIBUTING.md into one. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit c007e985f310f37214dd6ec39905848bbca4f24b) Signed-off-by: juhovainio --- AGENTS.md | 2 +- CONTRIBUTING.md | 2 +- docs/architecture.md | 2 +- docs/testing.md | 4 +- xtask/src/architecture_doc.rs | 152 +++++----------------------------- xtask/src/main.rs | 3 +- 6 files changed, 27 insertions(+), 138 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 43ec89c342..17113230a6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -278,7 +278,7 @@ Guardrails: - `crates/rocm-engine-protocol` is a contract surface; verify all impacted engines after protocol changes - first-party crate-layering invariants (e.g. `rocmd` must never depend on `rocm`) are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`); a new first-party dependency edge failing that check means the edge needs review, not a bypass - source file size is checked by `cargo xtask check-module-size` ([`module_size.rs`](xtask/src/module_size.rs)): no source file may exceed 1,500 production lines (test-gated items excluded), with one limit for every file and no exemption mechanism. A PR that would push a file past it splits the file in the same change; do not raise the limit or opt a file out. Merely editing a file that is already over the limit does not by itself require a split: those files are split under their own tickets, and the check cannot become blocking until none is left. Today the CI step and pre-commit hook only report; see `docs/architecture.md` -- every production source file (a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`) has an entry in `docs/architecture.md`, enforced by `cargo xtask check-architecture-doc` ([architecture_doc.rs](xtask/src/architecture_doc.rs)) in the `clippy` CI job and in the `architecture-doc` prek hook at commit and push. Add the entry in the same change as the file. Do not add the file to the check's `EXEMPT` list to make it pass: the list is empty and `EXEMPT_CEILING` caps it at zero, so adding to it is a visible, reviewed exception. The check reads no prose, so keeping an entry's description true is still on you +- every production source file (a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`) has an entry in `docs/architecture.md`, enforced by `cargo xtask check-architecture-doc` ([architecture_doc.rs](xtask/src/architecture_doc.rs)) in the `clippy` CI job and in the `architecture-doc` prek hook at commit and push. Add the entry in the same change as the file. The check has no exemption list: document the file. The check reads no prose, so keeping an entry's description true is still on you - preserve strict GPU-required behavior; do not introduce silent CPU fallback - respect platform gates (for example, native Windows handling for vLLM) - pin third-party GitHub Actions to a full commit SHA with a trailing `# vX.Y.Z` comment, never a moving tag (`@v2`, `@main`); a retagged or compromised action otherwise enters CI silently. Bump the SHA and comment together when upgrading diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c7540bb9c6..450373aa83 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,7 +110,7 @@ The `docs-links` job below checks the file links in that document, including eve Crate-layering invariants, such as `rocmd` never depending on `rocm`, are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`). File size is checked by `cargo xtask check-module-size` (`xtask/src/module_size.rs`): no source file may exceed 1,500 production lines, test code excluded, and a change that would cross the limit splits the file instead. Editing a file that is already over the limit does not by itself require a split; those files are split under their own tickets. For now the CI step and the `cargo-module-size` pre-commit hook report offenders without failing. -Every production source file, meaning a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`, must have an entry in `docs/architecture.md`: a relative markdown link to it, which has to render as a link (a path inside an HTML comment, a code block or a code span does not count, and neither does a malformed link such as one missing its `)`, or an image). Use an inline `[text](path)` link with its text on one line; reference-style links, a `?query` suffix and root-relative paths are not recognised. `build.rs` files and the `tests/` and `examples/` trees are out of scope. `cargo xtask check-architecture-doc` (`xtask/src/architecture_doc.rs`) enforces that in the `clippy` CI job and in the `architecture-doc` prek hook, which runs on every commit and every push, over the whole tree. A new file with no entry fails it. This is the opposite direction to the `lychee` check below, which only confirms that links already in the doc still resolve. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. `EXEMPT` in `xtask/src/architecture_doc.rs` is empty and its size is capped at zero by `EXEMPT_CEILING` in the same file, so adding to it is a visible, reviewed exception rather than a way to make the check pass. The check does not read the prose of an entry, so keeping a description true when a file's role changes is still up to you. +Every production source file, meaning a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src`, must have an entry in `docs/architecture.md`: a relative markdown link to it, which has to render as a link (a path inside an HTML comment, a code block or a code span does not count, and neither does a malformed link such as one missing its `)`, or an image). Use an inline `[text](path)` link with its text on one line; reference-style links, a `?query` suffix and root-relative paths are not recognised. `build.rs` files and the `tests/` and `examples/` trees are out of scope. `cargo xtask check-architecture-doc` (`xtask/src/architecture_doc.rs`) enforces that in the `clippy` CI job and in the `architecture-doc` prek hook, which runs on every commit and every push, over the whole tree. A file with no entry fails it, and there is no exemption list. This is the opposite direction to the `lychee` check below, which only confirms that links already in the doc still resolve. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. The check does not read the prose of an entry, so keeping a description true when a file's role changes is still up to you. Every local (relative-path) Markdown link and `#anchor` fragment in the repository (outside `docs/rocm-docs/`, which `docs-build` covers instead, and hidden directories such as `.github/`, which neither check walks) is checked by the `docs-links` CI job and by the `lychee` prek hook above, both configured by `lychee.toml` (which lists the current exclusions). Both run offline only, so they don't catch broken external `https://` links. diff --git a/docs/architecture.md b/docs/architecture.md index 643d712536..34a7b4ff3e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,7 +10,7 @@ This is the living module map for rocm-cli. It's a contributor-facing reference > Before relying on any entry below, verify current file and function boundaries directly (e.g. `grep`) rather than trusting this doc's wording. Module boundaries shift as the codebase grows, and a stale-but-plausible-looking note is worse than an explicit prompt to check. Every concrete file citation in the Module map's per-crate inventories below — except "the old agent.rs" in the `rocm-dash-tui` entry (historical prose about a deleted file) and `report.json` in the `crates/e2e-report` entry (cucumber's untracked generated output) — is a relative markdown link rather than a bare filename, checked offline by [lychee](https://github.com/lycheeverse/lychee) in CI (the `docs-links` job, config in `lychee.toml`) — a moved or deleted file fails CI immediately, naming the broken link. That check doesn't check that the surrounding prose (e.g. which extraction pattern a module follows) is still accurate; the caution above still applies to everything but link resolution. -A second check runs the other way: `cargo xtask check-architecture-doc` ([`architecture_doc.rs`](../xtask/src/architecture_doc.rs)) fails when a production source file — a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` — has no link in this document, so a new module cannot land without an entry. Only a link that renders as one counts: a path inside an HTML comment, a code block or a code span does not, and neither does a malformed link (one missing its `[` or `)`, or with an escaped bracket), an image, or the outer brackets of a link that contains another link. Write the entry as an inline `[text](path)` link with its text on one line; `` delimiters around the destination are accepted, but reference-style links, a `?query` suffix and root-relative paths are not. `build.rs` files and the `tests/` and `examples/` trees are out of scope. It runs in the `clippy` CI job and in the `architecture-doc` prek hook, at commit and at push. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. Every production file has an entry, so the check's `EXEMPT` list is empty and capped at zero; adding a file to it to make the check pass is not the way, and growing it is a visible, reviewed exception. +A second check runs the other way: `cargo xtask check-architecture-doc` ([`architecture_doc.rs`](../xtask/src/architecture_doc.rs)) fails when a production source file — a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` — has no link in this document, so a new module cannot land without an entry. Only a link that renders as one counts: a path inside an HTML comment, a code block or a code span does not, and neither does a malformed link (one missing its `[` or `)`, or with an escaped bracket), an image, or the outer brackets of a link that contains another link. Write the entry as an inline `[text](path)` link with its text on one line; `` delimiters around the destination are accepted, but reference-style links, a `?query` suffix and root-relative paths are not. `build.rs` files and the `tests/` and `examples/` trees are out of scope. It runs in the `clippy` CI job and in the `architecture-doc` prek hook, at commit and at push. Test-only helpers (`test_support.rs`, `test_env.rs`, `*_proptests.rs`, `*_props.rs`, `*_properties.rs`) need no entry. There is no exemption list: a production file with no entry fails the check. ## Module organization convention diff --git a/docs/testing.md b/docs/testing.md index 45dd4d47b1..48eccb31c8 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -62,8 +62,8 @@ cargo xtask check-architecture-doc ``` It fails on a `.rs` file under `apps/*/src`, `crates/*/src` or `engines/*/src` that the doc -does not link, and on a stale entry in the `EXEMPT` list in -[architecture_doc.rs](../xtask/src/architecture_doc.rs), which is empty. Test-only helpers need no entry. The +does not link, and there is no exemption list; see +[architecture_doc.rs](../xtask/src/architecture_doc.rs). Test-only helpers need no entry. The `lychee` check is complementary, not redundant: it confirms that links already in the doc resolve, and says nothing about a file the doc never mentions. diff --git a/xtask/src/architecture_doc.rs b/xtask/src/architecture_doc.rs index 5b1f7262e8..9999970acf 100644 --- a/xtask/src/architecture_doc.rs +++ b/xtask/src/architecture_doc.rs @@ -18,9 +18,9 @@ //! inside an HTML comment, a code block or a code span does not count, and //! neither does a link missing its `[` or `)`, one with an escaped bracket, an //! image, or the outer brackets of a link that contains another link), -//! or be listed in [`EXEMPT`], or be a test-only helper (see -//! [`is_test_helper`]). Out of scope: `build.rs` files at a crate's root and the -//! `tests/` and `examples/` trees, which sit outside `/src`. +//! or be a test-only helper (see [`is_test_helper`]). Out of scope: `build.rs` +//! files at a crate's root and the `tests/` and `examples/` trees, which sit +//! outside `/src`. //! //! It reads the git index and the working tree, never a git diff, so the answer //! does not depend on a base ref. That is the same answer in CI and in the @@ -33,15 +33,6 @@ //! `?query` suffix and a root-relative path render as working links but are not //! recognised, so the failure message asks for the inline form. //! -//! [`EXEMPT`] is the files that may go without an entry. It is empty: every -//! production file is documented. It follows the pattern of `ALLOWLIST` in -//! [`crate::crate_edges`] and is checked in both directions: a source file in -//! neither the doc nor the list fails, and so does a list entry that is now -//! cited or no longer exists. Nothing can stop a determined edit from adding to -//! it, so [`EXEMPT_CEILING`] pins its size in a test: growing the list means -//! raising that constant too, a second, visible line in the diff that a -//! reviewer has to accept. -//! //! What this does not check: whether an entry's prose still describes the file, //! and functions that move between two files that are both already cited. @@ -62,16 +53,6 @@ const DOC_DIR: &str = "docs"; /// Source roots whose `/src` trees are in scope. const SOURCE_ROOTS: [&str; 3] = ["apps", "crates", "engines"]; -/// Production files that deliberately have no entry in `docs/architecture.md`. -/// Empty: every production file is documented. Do not add to this list to make -/// the check pass: document the file in the doc instead. Growing it is a -/// reviewed exception and also needs [`EXEMPT_CEILING`] raised. Keep it sorted. -const EXEMPT: &[&str] = &[]; - -/// The most entries [`EXEMPT`] may hold. Zero: a file with no entry in the module -/// map should get one. Raising it is the visible, reviewed way to add an exemption. -const EXEMPT_CEILING: usize = 0; - /// Whether `path` is a tracked-file path this check covers: a `.rs` file under /// `//src/` for one of the [`SOURCE_ROOTS`]. #[allow(clippy::case_sensitive_file_extension_comparisons)] // Rust sources are always lowercase `.rs` @@ -85,7 +66,7 @@ fn in_scope(path: &str) -> bool { /// Whether `path` is a test-only helper that does not need an architecture /// entry: shared test support, or a property-test module. Matched on the file -/// stem so a new helper of the same kind never needs an [`EXEMPT`] entry. +/// stem so a new helper of the same kind never needs an entry. fn is_test_helper(path: &str) -> bool { let name = path.rsplit('/').next().unwrap_or(path); let stem = name.strip_suffix(".rs").unwrap_or(name); @@ -396,9 +377,9 @@ fn cited_paths(doc: &str) -> BTreeSet { .collect() } -/// Check the tracked files against the doc's citations and `exempt`, returning -/// every violation found. -fn check(tracked: &[String], cited: &BTreeSet, exempt: &[&str]) -> Vec { +/// Check the tracked files against the doc's citations, returning every +/// violation found. +fn check(tracked: &[String], cited: &BTreeSet) -> Vec { let in_tree: BTreeSet<&str> = tracked .iter() .map(String::as_str) @@ -408,7 +389,7 @@ fn check(tracked: &[String], cited: &BTreeSet, exempt: &[&str]) -> Vec, exempt: &[&str]) -> Vec Result> { .collect()) } -/// The violation, if any, of `entries` exempt files exceeding `ceiling`. -fn ceiling_violation(entries: usize, ceiling: usize) -> Option { - (entries > ceiling).then(|| { - format!( - "EXEMPT holds {entries} entries but EXEMPT_CEILING is {ceiling}: a file with no entry \ - in the module map should get one, not an EXEMPT line. Growing the list means raising \ - EXEMPT_CEILING in xtask/src/architecture_doc.rs too, which is a reviewed exception" - ) - }) -} - /// Run the check against the real workspace. fn violations_in(root: &Path) -> Result> { let doc = std::fs::read_to_string(root.join(DOC)).with_context(|| format!("failed to read {DOC}"))?; - let mut violations = check(&tracked_files(root)?, &cited_paths(&doc), EXEMPT); - violations.extend(ceiling_violation(EXEMPT.len(), EXEMPT_CEILING)); - Ok(violations) + Ok(check(&tracked_files(root)?, &cited_paths(&doc))) } -/// Fail if a production source file has no entry in `docs/architecture.md`, or -/// [`EXEMPT`] has gone stale. +/// Fail if a production source file has no entry in `docs/architecture.md`. pub fn run() -> Result<()> { let root = paths::workspace_root()?; let violations = violations_in(&root)?; @@ -511,13 +459,13 @@ mod tests { #[test] fn cited_file_passes() { let tracked = files(&["apps/rocm/src/cli.rs"]); - assert!(check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]).is_empty()); + assert!(check(&tracked, &cites(&["apps/rocm/src/cli.rs"])).is_empty()); } #[test] fn new_uncited_file_fails_and_is_named() { let tracked = files(&["apps/rocm/src/cli.rs", "apps/rocm/src/brand_new.rs"]); - let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]); + let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"])); assert_eq!(violations.len(), 1); assert!(violations[0].contains("apps/rocm/src/brand_new.rs")); assert!(violations[0].contains("is not cited")); @@ -526,46 +474,11 @@ mod tests { #[test] fn a_citation_does_not_cover_the_same_name_in_another_crate() { let tracked = files(&["apps/rocm/src/cli.rs", "apps/rocmd/src/cli.rs"]); - let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"]), &[]); + let violations = check(&tracked, &cites(&["apps/rocm/src/cli.rs"])); assert_eq!(violations.len(), 1); assert!(violations[0].contains("apps/rocmd/src/cli.rs")); } - #[test] - fn exempt_file_passes() { - let tracked = files(&["crates/rocm-dash-core/src/state.rs"]); - let exempt = ["crates/rocm-dash-core/src/state.rs"]; - assert!(check(&tracked, &cites(&[]), &exempt).is_empty()); - } - - #[test] - fn exempt_entry_for_a_missing_file_is_stale() { - let exempt = ["crates/rocm-dash-core/src/gone.rs"]; - let violations = check(&files(&[]), &cites(&[]), &exempt); - assert_eq!(violations.len(), 1); - assert!(violations[0].contains("crates/rocm-dash-core/src/gone.rs")); - assert!(violations[0].contains("no tracked source file")); - } - - #[test] - fn exempt_entry_that_is_now_cited_is_stale() { - let tracked = files(&["crates/rocm-dash-core/src/state.rs"]); - let exempt = ["crates/rocm-dash-core/src/state.rs"]; - let cited = cites(&["crates/rocm-dash-core/src/state.rs"]); - let violations = check(&tracked, &cited, &exempt); - assert_eq!(violations.len(), 1); - assert!(violations[0].contains("now cites it")); - } - - #[test] - fn exempt_entry_for_a_test_helper_is_redundant() { - let tracked = files(&["crates/rocm-core/src/test_support.rs"]); - let exempt = ["crates/rocm-core/src/test_support.rs"]; - let violations = check(&tracked, &cites(&[]), &exempt); - assert_eq!(violations.len(), 1); - assert!(violations[0].contains("redundant")); - } - #[test] fn files_outside_the_source_trees_are_ignored() { let tracked = files(&[ @@ -576,13 +489,13 @@ mod tests { "apps/rocm/src/data.json", "tests/e2e-cucumber/src/lib.rs", ]); - assert!(check(&tracked, &cites(&[]), &[]).is_empty()); + assert!(check(&tracked, &cites(&[])).is_empty()); } #[test] fn nested_source_files_are_in_scope() { let tracked = files(&["crates/rocm-dash-tui/src/ui/tabs/home.rs"]); - let violations = check(&tracked, &cites(&[]), &[]); + let violations = check(&tracked, &cites(&[])); assert_eq!(violations.len(), 1); assert!(violations[0].contains("ui/tabs/home.rs")); } @@ -598,7 +511,7 @@ mod tests { "crates/rocm-core/src/examine_driver_proptests.rs", "apps/rocm/src/providers/local_http/proptests.rs", ]); - assert!(check(&tracked, &cites(&[]), &[]).is_empty()); + assert!(check(&tracked, &cites(&[])).is_empty()); } #[test] @@ -649,7 +562,7 @@ mod tests { fn commenting_out_the_only_entry_leaves_the_file_uncited() { let doc = format!(""); let tracked = files(&["apps/rocm/src/cli.rs"]); - let violations = check(&tracked, &cited_paths(&doc), &[]); + let violations = check(&tracked, &cited_paths(&doc)); assert_eq!(violations.len(), 1); assert!(violations[0].contains("apps/rocm/src/cli.rs")); } @@ -770,7 +683,7 @@ mod tests { "cli](../apps/rocm/src/cli.rs)", "\\[cli](../apps/rocm/src/cli.rs)", ] { - let violations = check(&tracked, &cited_paths(broken), &[]); + let violations = check(&tracked, &cited_paths(broken)); assert_eq!(violations.len(), 1, "{broken:?} must not cite the file"); } } @@ -789,11 +702,7 @@ mod tests { assert!(!cited_cli("![cli](../apps/rocm/src/cli.rs)")); assert!(!cited_cli("see ![cli](../apps/rocm/src/cli.rs \"title\")")); let tracked = files(&["apps/rocm/src/cli.rs"]); - let violations = check( - &tracked, - &cited_paths("![cli](../apps/rocm/src/cli.rs)"), - &[], - ); + let violations = check(&tracked, &cited_paths("![cli](../apps/rocm/src/cli.rs)")); assert_eq!(violations.len(), 1); } @@ -840,16 +749,6 @@ mod tests { assert!(cited_cli("\\")); } - #[test] - fn the_exempt_list_stays_within_its_ceiling() { - assert_eq!(ceiling_violation(EXEMPT.len(), EXEMPT_CEILING), None); - assert_eq!(ceiling_violation(77, 77), None); - assert_eq!(ceiling_violation(70, 77), None); - let violation = ceiling_violation(78, 77).expect("one over the ceiling fails"); - assert!(violation.contains("78 entries")); - assert!(violation.contains("EXEMPT_CEILING is 77")); - } - #[test] fn the_link_forms_the_failure_message_rules_out_are_not_citations() { // The message tells authors these forms are not recognised; this holds it @@ -857,7 +756,7 @@ mod tests { assert!(!cited_cli("[cli][c]\n\n[c]: ../apps/rocm/src/cli.rs")); assert!(!cited_cli("[cli](../apps/rocm/src/cli.rs?plain=1)")); assert!(!cited_cli("[cli](/apps/rocm/src/cli.rs)")); - let message = &check(&files(&["apps/rocm/src/new.rs"]), &cites(&[]), &[])[0]; + let message = &check(&files(&["apps/rocm/src/new.rs"]), &cites(&[]))[0]; assert!(message.contains("inline `[text](path)`")); assert!(message.contains("reference-style")); assert!(message.contains("`?query`")); @@ -866,10 +765,9 @@ mod tests { #[test] fn the_failure_message_for_an_uncited_file_only_asks_for_an_entry() { - let violations = check(&files(&["apps/rocm/src/new.rs"]), &cites(&[]), &[]); + let violations = check(&files(&["apps/rocm/src/new.rs"]), &cites(&[])); assert_eq!(violations.len(), 1); assert!(violations[0].contains("add an entry for it to the module map")); - assert!(!violations[0].contains("EXEMPT")); } #[test] @@ -885,14 +783,6 @@ mod tests { ); } - #[test] - fn exempt_list_is_sorted_and_unique() { - let mut sorted = EXEMPT.to_vec(); - sorted.sort_unstable(); - sorted.dedup(); - assert_eq!(sorted, EXEMPT, "keep EXEMPT sorted, without duplicates"); - } - #[test] fn current_tree_matches_the_architecture_doc() { let root = Path::new(env!("CARGO_MANIFEST_DIR")) diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 58a5066136..aa0ad51cf1 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -165,8 +165,7 @@ pub(crate) enum Command { /// the limit is enforced. CheckModuleSizeAdvisory, /// Fail if a production source file (a `.rs` file under `apps/*/src`, - /// `crates/*/src` or `engines/*/src`) has no entry in `docs/architecture.md`, - /// or the exemption list in `xtask/src/architecture_doc.rs` has gone stale. + /// `crates/*/src` or `engines/*/src`) has no entry in `docs/architecture.md`. CheckArchitectureDoc, /// Regenerate the published Doctor catalog manifest from the compiled catalog. Catalog { From 09501456b02d973c2025ff6188e06ba2047d8094 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 17:17:36 +0000 Subject: [PATCH 029/102] ROCMAI-872: keep the update check from preparing a managed Python (#951) * fix(install): preview an SDK install without preparing Python On a host with no usable Python, rocm install sdk --dry-run (and the install_sdk_dry_run assistant tool) went on to download a managed uv, have it install Python and save that Python's path in the config before printing a plan, although a preview promises to change nothing. The preview only needs the interpreter's wheel tags, so it now reports that a real install would prepare a managed Python (python_note) and prepares nothing; every refusal the real resolution raises is raised for the preview too. A real install resolves and prepares exactly as before. Adds unit tests for the no-Python, PATH Python, saved Python, stale saved Python and bootstrap-disabled cases, and scenario runtime-22, which checks the preview's note together with the empty tools, cache and config folders and a recording uv that is never run. Signed-off-by: Jussi Elo * fix(install): refuse a dry run for a managed Python the install would refuse With ROCM_CLI_MANAGED_PYTHON_VERSION naming a version whose wheel tag is not the one the layout needs (3.13 against cp312), a dry run on a host with no usable Python said it would prepare that Python, but a real install prepares it, finds the wrong tag and refuses. The preview now applies the same tag rule, shared with the check an existing interpreter must pass, and fails with the install's message instead. A version whose tag cannot be read off its name is announced as before. Adds a unit test and scenario runtime-23, which asserts the refusal together with the empty tools, cache and config folders. Signed-off-by: Jussi Elo * test(e2e): give runtime-22 and runtime-23 a cache folder of their own The harness points the CLI at a cache shared across scenarios and, on CI, across runs. runtime-22 asserted that the preview had downloaded no tool archive by looking in that shared folder, so a uv archive left by an earlier run failed it although the preview downloaded nothing. The scenarios now set ROCM_CLI_CACHE_DIR to a folder under the scenario root and assert against it. Signed-off-by: Jussi Elo * fix(install): read every spelling of a managed Python version in a dry run The dry run's check of ROCM_CLI_MANAGED_PYTHON_VERSION read only plain major.minor versions, so 3.13t, cpython-3.13 and cpython@3.13 were announced as a plan although the installed interpreter reports cp313 and the real install refuses it. The version is now read in those spellings. A version whose tag still cannot be read is announced as unchecked, in both the note and the progress line, rather than as a promise. The progress line before the search no longer says a real install would prepare a Python, since the search can end in a refusal; the line naming the Python, or the refusal, follows it. runtime-23 asserts the whole output: no promise before the refusal. Signed-off-by: Jussi Elo * test(e2e): present the uv log, not the cache folder, as runtime-22's evidence ROCM_CLI_UV_BINARY makes the CLI return the recording uv before any download could begin, so the check for a tool archive in the scenario's cache cannot fail by itself. It stays as a backstop and says so; the scenario comments and docs/testing.md now name the uv log as what shows nothing was prepared. Signed-off-by: Jussi Elo * fix(install): print a dry run's Python prediction only after the decision With an unusable PATH Python, no usable saved interpreter and managed Python disabled, the preview said a real install would use managed Python and then refused; the saved-interpreter line made the same premature claim. The two lines now state only what was found. The one line that predicts the preparation is built from the decision itself and printed by python_for_install, so it exists only for a decision that passed the bootstrap policy and the version check; the Using lines moved there with it. A real install prints exactly what it did. Adds a state-matrix unit test over PATH Python, saved Python, bootstrap policy and version override that asserts each row's whole output and the state left behind, and scenario runtime-25 for the policy refusal. Signed-off-by: Jussi Elo * docs: limit the dry run's no-uv guarantee to the canonical layout A preview of a ROCm 10 or newer wheel resolves its package versions with uv pip compile, so on a clean host it can still download the managed uv and create the data folder's tools folder, although it prepares no Python. docs/testing.md, docs/manual-testing.md and the dash-TUI tool-group comment now say so, and describe the state matrix and the refusal scenarios the preview is checked by. Signed-off-by: Jussi Elo * docs(install): make python_tag_for_version's comment match what it reads The doc comment said a wildcard yields None, but "3.13.*" reads as cp313: whatever follows the minor is not validated, because the major and minor alone decide the wheel tag. A wildcard that leaves the minor open ("3.*", "*") is what yields None. Reword the comment to say so and pin both unreadable wildcards in the existing reader test. Also say in the comment that this hand-rolled reader can drift from what uv accepts, and that the failure is benign: an unreadable spelling is announced as unchecked rather than refused. Signed-off-by: Jussi Elo * refactor(install): word a preview's unchecked-tag caveat in one place A preview that would prepare a managed Python whose wheel tag cannot be read off its name said so twice, in the python_note line and in the progress line, each with its own sentence. Build the caveat from one function so the two cannot drift. The note is unchanged; the progress line now reads "... would prepare Python pypy@3.10 (its wheel tag cannot be checked until it is installed; cp312 is required). This preview does not." instead of using a "whose ... " clause. The unit tests that pin both lines are updated to match. Signed-off-by: Jussi Elo * refactor(install): name a Python launcher's source with an enum PythonLauncher::source was a bare &'static str ("env", "path", "managed", "manifest") that three places matched or compared by spelling, with a catch-all arm in the progress line that would hide a new spelling. Make it a PythonSource enum with an as_str for the install plan's python_source line, and match it exhaustively in the progress text. The printed output is unchanged. Signed-off-by: Jussi Elo * refactor(install): let the Python resolver's environment carry the managed-Python policy PythonResolverEnv already injects the two inputs that locate an interpreter (ROCM_CLI_PYTHON and the search directories), but the two inputs that decide what happens when none is found were still read from the process environment inside the decision: whether the managed bootstrap is disabled and which managed Python version is requested. A test of that decision therefore had to mutate process-wide environment variables under a lock. Both now travel in PythonResolverEnv (read once by from_process_env, which the real resolver still uses), and find_python_launcher_in, plan_python_launcher_in and the real install's ensure_managed_python take them from it, so a preview and an install made from one environment cannot disagree about the version or the policy. managed_python_version keeps its signature for the progress line in therock.rs and still reads the process environment. No behaviour change: the existing tests, which build an environment from the process one, pass unchanged. Signed-off-by: Jussi Elo * test(install): run the preview's Python decision on every platform The tests of what a dry run decides about Python drive it with shell-script interpreters and are cfg(unix), and the scenarios that cover it are Linux-only, so the Windows CI job ran none of it although a fresh Windows install is one of the hosts the preview is for. The decision needs no interpreter: with nothing to find, python_for_install is a function of the resolver environment (now carrying the managed-Python policy) and the dry_run flag. The new test builds that environment directly and runs python_for_install through it, so it runs on the Windows job. It asserts, per row, the preview's complete progress output together with the state it leaves behind (no config, manifest, or folder created): the default, a patch release, a blank override and an unreadable spelling are announced (the last as unchecked); a version whose wheel tag is not the layout's is refused in every spelling that reads as one; a disabled bootstrap is refused with the install's own message. A control runs the real install on the same host and checks it reaches the managed uv, which is a file that cannot be launched, so nothing is downloaded or run and the preview's silence about uv is shown to mean something. The search order (python3 before python on Linux, python, python3, py with PATHEXT on Windows) and the interpreter probes are still covered by the Unix tests only. RestoredEnv and snapshot lose their cfg(unix) because the new test shares them. Signed-off-by: Jussi Elo * refactor(install): name whether a preview could check a Python's wheel tag PlannedPython::WouldPrepareManaged carried a tag_checked bool next to the requirement it was checked against. The bool is one of two named cases, so it is now a TagCheck enum (Checked, Unchecked), and unchecked_tag_caveat matches on it. The preview's note and progress line are unchanged; the existing tests, updated to the enum, pass as before. Signed-off-by: Jussi Elo * test(install): pin the wording of the unusable-Python findings for an install and a preview An install and a preview word the two findings about a PATH Python and a saved managed Python that cannot serve the install differently (the install says what it goes on to do, the preview states the finding only), but only the preview's wording was pinned, by the state matrix on Unix hosts. This pins both, on every platform, before the two parallel match blocks that print them are merged into one. Signed-off-by: Jussi Elo * refactor(install): print the unusable-Python findings from one place find_python_launcher_in printed the finding about a PATH Python and the one about a saved managed Python from two parallel Install/Preview match blocks that differed only in a trailing clause. LookupPurpose::unusable_python_line now builds both lines: both purposes state the finding, an install adds what it goes on to do. The output is byte-identical, as the test added in the previous commit pins for both purposes. Signed-off-by: Jussi Elo * docs: say which lane covers the Windows preview and that the previewed uv venv line is not verbatim docs/testing.md and the runtime-23 scenario comment now name the unit test that runs the preview's Python decision on every platform, the Windows CI job included, and say what stays uncovered (the Windows search order and any Windows run of the preview). docs/testing.md and docs/manual-testing.md also say that the plan's command line prints uv venv --python 3.12, a version rather than a path, so it is not copy-pasteable verbatim: uv resolves 3.12 itself and may choose a different Python 3.12 than the managed one a real install prepares. Signed-off-by: Jussi Elo * docs: wrap two over-long lines in the install sdk dry-run notes Re-wrap the manual-testing bullet and the testing.md paragraph on the dry-run Python preview to the surrounding 80-column style. No words change. Signed-off-by: Jussi Elo * ROCMAI-872: keep the update check from preparing a managed Python Reading the index for a wheel runtime needs the interpreter's wheel tag, and a runtime that records no usable Python made the check go and get one: on a host with none it downloaded a managed uv, had uv install Python and saved that Python's path in the configuration. That ran for `rocm update`, for `rocm update --json` (the dashboard's check), for the check the CLI runs ahead of other commands, and for the plan behind `rocm update --apply --dry-run`, so a read-only look at the runtimes changed the host. These callers now go through `runtime_update_check`, which finds an interpreter with the search-only entry point the install preview already uses and, when there is none, takes the wheel tag from the interpreter the layout requires. The text report adds a `python_note:` line saying an update would use uv to install the managed Python, which the check did not do; the JSON keeps its shape. `runtime_update_plan`, which `rocm update --apply` works from, still prepares the Python, since the install that follows needs it. A unit test drives the report, the JSON, the startup check and the check plan on a host with no Python and a recording uv, with the apply plan as the control that shows the stand-in is reachable. Scenarios update-09 and update-10 drive the report and the dry-run preview through the built binary and check the note against the state it left. Signed-off-by: Jussi Elo * ROCMAI-872: state what the update and SDK preview tools do and do not prepare The side-effects note on the read-only ROCm tools still said that `update_check` can download a managed `uv` and Python and record that Python in the saved config, which is the behaviour the update check no longer has: it runs `rocm update`, which now only reports the Python a real update would prepare, in a `python_note:` line of its text report. The note now says that neither `update_check` nor `install_sdk_dry_run` prepares a Python or saves a Python path, that only a real apply or install does, and keeps the caveat that a ROCm 10 or newer check or preview can still fetch the managed `uv`. No other file repeated the old claim. Signed-off-by: Jussi Elo * ROCMAI-872: pin on every platform that a --dry-run takes the check and an apply the plan apply_runtime_update chose between the two planners through a pass-through helper whose unit test proved only the `if`, so the dry_run argument at the call site was caught only by a test that runs on Unix and by update-10 on Linux. The flow now takes the two planners as arguments (apply_runtime_update_with), and the unit test drives it with planners that report nothing newer, so it needs no host without a Python and runs on Windows too: a --dry-run must call only the check and an apply only the plan. Which therock function is the check stays pinned only on Unix and by update-10, and the comments now say so. Signed-off-by: Jussi Elo * ROCMAI-872: say why a --apply --dry-run does not print the plan's python_note The plan an apply --dry-run works from carries a python_note, and apply_runtime_update never printed it. The preview that follows prints its own python_note from the decision a real install takes, which scenario update-10 asserts, so printing the plan's as well would say the same thing twice. A comment now records that, and that a dry run which finds nothing newer returns before the preview and prints neither note. Signed-off-by: Jussi Elo * ROCMAI-872: describe plan_python_launcher and the update note in the python.rs entry The architecture entry for therock/python.rs named neither the planned-Python lookup that previews and update checks use nor the python_note lines it writes, and did not list what therock/update.rs reads from it. It now says what plan_python_launcher, python_for_install and PlannedPython are and which items are pub(super) for the update module. Signed-off-by: Jussi Elo * ROCMAI-872: list RestoredEnv in the therock/test_support.rs architecture entry Signed-off-by: Jussi Elo * ROCMAI-872: describe the update check and its python_note in the update.rs module doc Signed-off-by: Jussi Elo * ROCMAI-872: say what the update check does with uv and Python now that the install preview fetches no uv The merged notes still said the update check prepares a managed Python, which this change made false, and that the install preview can fetch the managed uv, which the preview change made false. The check on a ROCm 10 or newer runtime still resolves the package versions with uv and can fetch it; it prepares no Python, and the install preview fetches no uv. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 227968dcb30edd574f0405c523536e91f14d4524) Signed-off-by: juhovainio --- apps/rocm/src/therock.rs | 4 +- apps/rocm/src/therock/python.rs | 155 ++++++++--- apps/rocm/src/therock/test_support.rs | 40 ++- apps/rocm/src/therock/update.rs | 245 +++++++++++++++++- apps/rocm/src/update_cmd.rs | 119 ++++++++- crates/rocm-dash-tui/src/agent/tools.rs | 19 +- docs/architecture.md | 8 +- docs/manual-testing.md | 10 + docs/testing.md | 39 ++- tests/e2e-cucumber/features/update.feature | 46 ++++ tests/e2e-cucumber/tests/e2e/therock_steps.rs | 94 +++++++ 11 files changed, 705 insertions(+), 74 deletions(-) diff --git a/apps/rocm/src/therock.rs b/apps/rocm/src/therock.rs index 2de4ecb64a..75bccb49b5 100644 --- a/apps/rocm/src/therock.rs +++ b/apps/rocm/src/therock.rs @@ -56,8 +56,8 @@ use self::source::{ requested_source_layout, select_source_layout, }; pub(crate) use self::update::{ - maybe_refresh_startup_update_check, render_update_json, render_update_report, - runtime_update_plan, + RuntimeUpdatePlan, maybe_refresh_startup_update_check, render_update_json, + render_update_report, runtime_update_check, runtime_update_plan, }; pub(crate) use self::version::runtime_version_display; use self::version::{ diff --git a/apps/rocm/src/therock/python.rs b/apps/rocm/src/therock/python.rs index 43b38f0261..3ca1ff1a55 100644 --- a/apps/rocm/src/therock/python.rs +++ b/apps/rocm/src/therock/python.rs @@ -525,8 +525,8 @@ pub(super) fn python_for_install( Ok(python) } -/// The interpreter a *preview* of the install reports, found without preparing -/// one. +/// The interpreter a *preview* of the install, or an update check, reports, +/// found without preparing one. /// /// [`resolve_python_launcher`] ends by downloading a managed `uv`, having it /// install Python, and saving that Python's path in the configuration when no @@ -538,7 +538,10 @@ pub(super) fn python_for_install( /// resolution raises before that step (an unusable `ROCM_CLI_PYTHON`, a /// disabled managed bootstrap) is raised here too, so a preview never succeeds /// on a host where the install would refuse. -fn plan_python_launcher(paths: &AppPaths, requirement: PythonRequirement) -> Result { +pub(super) fn plan_python_launcher( + paths: &AppPaths, + requirement: PythonRequirement, +) -> Result { let env = python_resolver_env_override().unwrap_or_else(PythonResolverEnv::from_process_env); plan_python_launcher_in(paths, &env, requirement) } @@ -655,6 +658,28 @@ impl PlannedPython { } } + /// The `python_note:` line of an update check, in the same terms as + /// [`Self::note`] but for the step an update would take. An update check + /// needs the interpreter's wheel tag to read the index, and a host with no + /// interpreter gets it from the one the layout requires, so this line is how + /// the check says it did not go and prepare one. Built from the decision, as + /// the preview's is. + pub(super) fn update_note(&self) -> Option { + match self { + Self::Ready(_) => None, + Self::WouldPrepareManaged { + version, + requirement, + tag_check, + } => { + let unchecked = unchecked_tag_caveat(*tag_check, *requirement); + Some(format!( + "no usable Python was found; updating would use uv to install managed Python {version}{unchecked} and save its path in the configuration, which this check did not do" + )) + } + } + } + /// The progress line that names the chosen interpreter: the one an install /// goes on with, or, for a preview on a host with no usable Python, what a /// real install would prepare. It is the only line that predicts the @@ -988,7 +1013,7 @@ fn python_venv_probe_temp_root() -> Result { mod tests { use super::*; use crate::test_support::PROCESS_ENV_TEST_LOCK; - use crate::therock::test_support::test_paths; + use crate::therock::test_support::{RestoredEnv, test_paths}; #[cfg(unix)] fn write_fake_python_with_venv(dir: &Path, name: &str) -> Result { @@ -1410,43 +1435,6 @@ exit 1 // ---- a dry run prepares no Python ---------------------------------------- - /// Process environment variables set for one test and restored on drop. - /// Every caller holds `PROCESS_ENV_TEST_LOCK` for as long as it holds this. - struct RestoredEnv(Vec<(&'static str, Option)>); - - impl RestoredEnv { - #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 - fn set(vars: &[(&'static str, Option<&Path>)]) -> Self { - let saved = vars - .iter() - .map(|(key, _)| (*key, std::env::var_os(key))) - .collect(); - for (key, value) in vars { - unsafe { - match value { - Some(value) => std::env::set_var(key, value), - None => std::env::remove_var(key), - } - } - } - Self(saved) - } - } - - impl Drop for RestoredEnv { - #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 - fn drop(&mut self) { - for (key, value) in self.0.drain(..) { - unsafe { - match value { - Some(value) => std::env::set_var(key, value), - None => std::env::remove_var(key), - } - } - } - } - } - /// A `uv` that records its arguments in `marker` and then fails. A host that /// reaches it has gone on to prepare a managed Python; failing keeps a test /// that does reach it from going on to the network. @@ -1541,6 +1529,16 @@ exit 1 ) } + /// What an update check goes on with, searching only this host's folder. + fn plan_for_update_check(&self) -> Result { + let _resolver = OverridePythonResolverEnv::new(PythonResolverEnv { + python_override: None, + search_dirs: vec![self.search_dir.clone()], + ..PythonResolverEnv::from_process_env() + }); + plan_python_launcher(&self.paths, python_requirement(SourceLayout::Canonical)) + } + fn uv_was_run(&self) -> bool { self.uv_marker.exists() } @@ -2300,6 +2298,81 @@ exit 1 Ok(()) } + /// An update check reads the index by the interpreter's wheel tag. On a host + /// with no interpreter that tag is the one the layout requires, which a real + /// update would prepare an interpreter to match, so the check needs nothing + /// prepared. The note it prints says so, and the folders agree; the same host + /// DOES reach `uv` for the install, so an unrun stand-in means the check + /// stayed out of it. The Gherkin scenario `update-09` drives the same case + /// through the built binary. + #[cfg(unix)] + #[test] + fn an_update_check_with_no_usable_python_reads_the_required_tag_and_prepares_nothing() + -> Result<()> { + if current_platform_wheel_tags().is_err() { + return Ok(()); + } + let host = PythonHost::new("python-update-check-none")?; + let requirement = python_requirement(SourceLayout::Canonical); + + let planned = host.plan_for_update_check()?; + + let compatibility = planned.wheel_compatibility()?; + assert_eq!(compatibility.python_tag, requirement.tag); + assert!(!compatibility.platform_tags.is_empty()); + assert_eq!( + planned.update_note().as_deref(), + Some( + "no usable Python was found; updating would use uv to install managed Python 3.12 and save its path in the configuration, which this check did not do" + ) + ); + assert!(!host.uv_was_run(), "the update check ran uv"); + assert_eq!( + snapshot(&host.paths), + Vec::::new(), + "the update check created files or folders under the config, data or cache folder" + ); + + // Control: the install on the same host does reach `uv`. + assert!(host.python_for_install(false).is_err()); + assert!( + host.uv_was_run(), + "the stand-in uv is not reachable, so the checks above prove nothing" + ); + Ok(()) + } + + /// A usable interpreter needs no note, and a managed Python whose tag cannot + /// be read off its version says so in the update note as in the preview's. + #[cfg(unix)] + #[test] + fn an_update_note_is_absent_for_a_usable_python_and_flags_an_unreadable_tag() -> Result<()> { + if current_platform_wheel_tags().is_err() { + return Ok(()); + } + let host = PythonHost::new("python-update-note")?; + { + let _unreadable = RestoredEnv::set(&[( + "ROCM_CLI_MANAGED_PYTHON_VERSION", + Some(Path::new("pypy@3.10")), + )]); + let planned = host.plan_for_update_check()?; + let note = planned.update_note().expect("nothing was prepared"); + assert!( + note.contains("managed Python pypy@3.10 (its wheel tag cannot be checked until it is installed; cp312 is required)") + && note.ends_with("which this check did not do"), + "{note}" + ); + } + + write_fake_python_with_venv(&host.search_dir, "python3")?; + let planned = host.plan_for_update_check()?; + assert!(matches!(planned, PlannedPython::Ready(_)), "{planned:?}"); + assert_eq!(planned.update_note(), None); + assert!(!host.uv_was_run()); + Ok(()) + } + /// The version-to-tag reading the preview's refusal rests on, and that it is /// the same rule `python_launcher_install_ready` applies (same message). #[test] diff --git a/apps/rocm/src/therock/test_support.rs b/apps/rocm/src/therock/test_support.rs index e0b41a46d0..f27c50dd8e 100644 --- a/apps/rocm/src/therock/test_support.rs +++ b/apps/rocm/src/therock/test_support.rs @@ -11,7 +11,45 @@ use crate::runtime_registry::{InstalledRuntimeManifest, runtime_manifest_path}; use anyhow::Result; use rocm_core::AppPaths; use std::fs; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; + +/// Process environment variables set for one test and restored on drop. Every +/// caller holds `PROCESS_ENV_TEST_LOCK` for as long as it holds this. +pub(super) struct RestoredEnv(Vec<(&'static str, Option)>); + +impl RestoredEnv { + /// Sets (`Some`) or removes (`None`) each variable, remembering what it was. + #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 + pub(super) fn set(vars: &[(&'static str, Option<&Path>)]) -> Self { + let saved = vars + .iter() + .map(|(key, _)| (*key, std::env::var_os(key))) + .collect(); + for (key, value) in vars { + unsafe { + match value { + Some(value) => std::env::set_var(key, value), + None => std::env::remove_var(key), + } + } + } + Self(saved) + } +} + +impl Drop for RestoredEnv { + #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 + fn drop(&mut self) { + for (key, value) in self.0.drain(..) { + unsafe { + match value { + Some(value) => std::env::set_var(key, value), + None => std::env::remove_var(key), + } + } + } + } +} pub(super) fn family_for_test(family: &str, raw_arch: Option<&str>) -> FamilyResolution { FamilyResolution { diff --git a/apps/rocm/src/therock/update.rs b/apps/rocm/src/therock/update.rs index ccfea9278a..a006fece71 100644 --- a/apps/rocm/src/therock/update.rs +++ b/apps/rocm/src/therock/update.rs @@ -4,14 +4,20 @@ //! Whether an installed runtime has something newer to offer, and the bounded //! startup check that asks. `runtime_update_plan` resolves the latest source for -//! one runtime (`resolve_latest_for_manifest`, which re-resolves a wheel -//! runtime through `resolve_pip_runtime_with_timeout` and a tarball one through +//! one runtime, and `runtime_update_check` does the same for a caller that +//! changes nothing (the report, the JSON, the startup check and an apply's +//! preview). They differ in `PythonAccess`: given a wheel runtime that records +//! no usable Python, the plan prepares a managed Python (an apply needs one), +//! while the check works the wheel tag out from the interpreter the layout +//! requires and carries a `python_note` saying the preparation was left undone +//! (`resolve_latest_for_manifest` re-resolves a wheel runtime through +//! `resolve_pip_runtime_with_timeout` and a tarball one through //! `resolve_tarball_artifact_with_timeout`, on the layout the runtime was //! installed from) and `runtime_freshness` classifies it (`RuntimeFreshness`: //! up to date, update available, repair available, or ahead of the index) into //! a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` show //! the plans for `rocm update` (the JSON form is what the dashboard's update -//! check parses), and `maybe_refresh_startup_update_check` keeps one +//! check parses and carries no `python_note`), and `maybe_refresh_startup_update_check` keeps one //! `StartupUpdateCheckRecord` in the cache so the check runs at most once an //! interval and is bounded by a short download timeout. //! @@ -25,8 +31,8 @@ use super::family::family_override_or_recovered_arch; use super::process::SuppressProgressOutput; use super::python::{ - PythonLauncher, PythonSource, python_launcher_install_ready, python_requirement, - resolve_python_launcher, wheel_compatibility_for_python, + PythonLauncher, PythonSource, plan_python_launcher, python_launcher_install_ready, + python_requirement, resolve_python_launcher, wheel_compatibility_for_python, }; use super::resolve::{ AggregateDeviceTarget, resolve_pip_runtime_with_timeout, resolve_tarball_artifact_with_timeout, @@ -84,6 +90,22 @@ pub(crate) struct RuntimeUpdatePlan { pub source_layout_generation: Option, pub repair_required: bool, pub update_available: bool, + /// Set when the runtime records no usable Python and the check left one + /// unprepared: what an update would do about it. + pub python_note: Option, +} + +/// Whether resolving a wheel runtime's latest version may prepare a Python. +/// +/// Reading the index needs the interpreter's wheel tag. A runtime that records +/// a usable Python supplies it. For one that does not, an apply goes on to +/// prepare a managed Python (it needs one for the install that follows), but a +/// check changes nothing, so it works the tag out from the interpreter the +/// layout requires and says it left the preparation undone. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +enum PythonAccess { + MayPrepare, + SearchOnly, } #[derive(Debug, Clone, Copy, Eq, PartialEq)] @@ -117,6 +139,7 @@ struct ResolvedRuntimeUpdate { format: String, source_layout_generation: String, wheel_composition: Option, + python_note: Option, } pub(crate) fn render_update_report(paths: &AppPaths) -> Result { @@ -145,7 +168,7 @@ pub(crate) fn render_update_report(paths: &AppPaths) -> Result { } for manifest in &manifests { - let plan = match runtime_update_plan(paths, manifest, &manifests, None) { + let plan = match runtime_update_check(paths, manifest, &manifests, None) { Ok(plan) => Some(plan), Err(error) => { let _ = writeln!( @@ -182,6 +205,9 @@ pub(crate) fn render_update_report(paths: &AppPaths) -> Result { manifest.install_root.display() ); let _ = writeln!(output, " source: {}", plan.latest_source); + if let Some(note) = &plan.python_note { + let _ = writeln!(output, " python_note: {note}"); + } if plan.update_available { let next_step = if plan.repair_required { format!( @@ -248,7 +274,7 @@ pub(crate) fn render_update_json( let manifests = load_runtime_manifests(paths)?; let mut runtimes = Vec::with_capacity(manifests.len()); for manifest in &manifests { - match runtime_update_plan(paths, manifest, &manifests, download_timeout_secs) { + match runtime_update_check(paths, manifest, &manifests, download_timeout_secs) { Ok(plan) => runtimes.push(UpdateJsonRuntime { runtime_key: manifest.runtime_key.clone(), channel: manifest.channel.clone(), @@ -357,13 +383,49 @@ fn runtime_freshness_with_manifests( } } +/// The plan an apply works from. A wheel runtime that records no usable Python +/// has one prepared here, since the install that follows needs it. pub(crate) fn runtime_update_plan( paths: &AppPaths, manifest: &InstalledRuntimeManifest, manifests: &[InstalledRuntimeManifest], download_timeout_secs: Option, ) -> Result { - let latest = resolve_latest_for_manifest(paths, manifest, download_timeout_secs)?; + plan_runtime_update( + paths, + manifest, + manifests, + download_timeout_secs, + PythonAccess::MayPrepare, + ) +} + +/// The same plan for a caller that changes nothing: the update report, the JSON +/// the dashboard reads, and the preview of an apply. It never prepares a Python. +pub(crate) fn runtime_update_check( + paths: &AppPaths, + manifest: &InstalledRuntimeManifest, + manifests: &[InstalledRuntimeManifest], + download_timeout_secs: Option, +) -> Result { + plan_runtime_update( + paths, + manifest, + manifests, + download_timeout_secs, + PythonAccess::SearchOnly, + ) +} + +fn plan_runtime_update( + paths: &AppPaths, + manifest: &InstalledRuntimeManifest, + manifests: &[InstalledRuntimeManifest], + download_timeout_secs: Option, + python_access: PythonAccess, +) -> Result { + let latest = + resolve_latest_for_manifest(paths, manifest, download_timeout_secs, python_access)?; let freshness = runtime_freshness_with_manifests( manifests, manifest, @@ -386,6 +448,7 @@ pub(crate) fn runtime_update_plan( source_layout_generation: Some(latest.source_layout_generation), repair_required: freshness == RuntimeFreshness::RepairAvailable, update_available: freshness.update_available(), + python_note: latest.python_note, }) } @@ -417,6 +480,7 @@ fn resolve_latest_for_manifest( paths: &AppPaths, manifest: &InstalledRuntimeManifest, download_timeout_secs: Option, + python_access: PythonAccess, ) -> Result { let channel = TheRockChannel::parse(&manifest.channel)?; let layout = manifest_source_layout(manifest)?; @@ -437,12 +501,17 @@ fn resolve_latest_for_manifest( executable, source: PythonSource::Manifest, }); - let python_executable = match manifest_python { - Some(python) => python, - None => resolve_python_launcher(paths, requirement)?, + let (wheel_compatibility, python_note) = match (manifest_python, python_access) { + (Some(python), _) => (wheel_compatibility_for_python(&python.executable)?, None), + (None, PythonAccess::MayPrepare) => { + let python = resolve_python_launcher(paths, requirement)?; + (wheel_compatibility_for_python(&python.executable)?, None) + } + (None, PythonAccess::SearchOnly) => { + let planned = plan_python_launcher(paths, requirement)?; + (planned.wheel_compatibility()?, planned.update_note()) + } }; - let wheel_compatibility = - wheel_compatibility_for_python(&python_executable.executable)?; let family_override = manifest_wheel_family_override(manifest); let resolution = resolve_pip_runtime_with_timeout( paths, @@ -488,6 +557,7 @@ fn resolve_latest_for_manifest( format: "wheel".to_owned(), source_layout_generation: resolution.layout.generation().to_owned(), wheel_composition, + python_note, }) } "tarball" => { @@ -512,6 +582,7 @@ fn resolve_latest_for_manifest( format: "tarball".to_owned(), source_layout_generation: artifact.layout.generation().to_owned(), wheel_composition: None, + python_note: None, }) } other => bail!("unknown manifest format `{other}`"), @@ -585,7 +656,12 @@ fn build_startup_update_check_record( now_unix_ms: u128, download_timeout_secs: Option, ) -> StartupUpdateCheckRecord { - match resolve_latest_for_manifest(paths, manifest, download_timeout_secs) { + match resolve_latest_for_manifest( + paths, + manifest, + download_timeout_secs, + PythonAccess::SearchOnly, + ) { Ok(latest) => { let freshness = runtime_freshness_with_manifests( manifests, @@ -654,11 +730,15 @@ mod tests { use super::super::process::{PROGRESS_LINE_SINK, progress_output_suppressed}; use super::super::python::{OverridePythonResolverEnv, PythonResolverEnv}; use super::super::source::THEROCK_SOURCE_LAYOUT_GENERATION; + #[cfg(unix)] + use super::super::test_support::RestoredEnv; use super::super::test_support::{ test_paths, test_runtime_manifest, write_test_runtime_manifest, }; use super::*; use crate::test_support::PROCESS_ENV_TEST_LOCK; + #[cfg(unix)] + use std::path::Path; /// Updating a next-layout manifest whose family is a group label (as /// `gfx125X-dcgpu` always is) must recover the exact arch this runtime was @@ -1242,4 +1322,141 @@ mod tests { let _ = fs::remove_dir_all(root); Ok(()) } + + /// The bug: a read-only look at a wheel runtime that records no usable Python, + /// on a host with none, went on to prepare one (download a managed `uv`, have + /// it install Python, save the path in the configuration). Driven here, each + /// against the same host: the update report, the JSON the dashboard reads, + /// the startup check, `runtime_update_check` itself, and + /// `apply_runtime_update` with `dry_run` (the preview of an apply, which takes + /// the check). Each must leave `uv` unrun and the configuration and managed + /// tools absent, and each must still reach the index (the connection to the + /// unreachable index is what each error names), so the absences are not a + /// check that stopped early. `runtime_update_plan` and a real + /// `apply_runtime_update` do prepare one on the same host, which is also what + /// shows the stand-in `uv` is reachable. That a `--dry-run` takes the check and + /// an apply the plan is pinned on every platform by + /// `a_dry_run_takes_the_check_and_an_apply_takes_the_plan` in `update_cmd.rs`; + /// which function is the check is pinned only here and by scenario `update-10` + /// on Linux, and no test on Windows pins it. The Gherkin scenarios `update-09` + /// and `update-10` drive the report and the preview through the built binary. + #[cfg(unix)] + #[test] + fn the_update_checks_prepare_no_python_and_the_apply_plan_does() -> Result<()> { + use rocm_core::fake_executable::write_fake_executable; + + let _env_lock = PROCESS_ENV_TEST_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let (root, paths) = test_paths("update-checks-prepare-no-python"); + + let marker = root.join("uv-was-run"); + let tools = root.join("tools"); + fs::create_dir_all(&tools)?; + let uv = tools.join("uv"); + write_fake_executable( + &uv, + format!("#!/bin/sh\necho \"$@\" >> '{}'\nexit 1\n", marker.display()), + )?; + let search = root.join("search"); + fs::create_dir_all(&search)?; + // Port 9 refuses the connection at once, so a check that gets past + // Python fails at the index without touching the network. + let unreachable_index = "http://127.0.0.1:9/"; + let _env = RestoredEnv::set(&[ + ("ROCM_CLI_UV_BINARY", Some(uv.as_path())), + ("ROCM_CLI_PYTHON", None), + ("ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP", None), + ("ROCM_CLI_MANAGED_PYTHON_VERSION", None), + ("ROCM_CLI_DISABLE_STARTUP_UPDATE_CHECK", None), + ("ROCM_CLI_THEROCK_ALLOW_BASE_OVERRIDE", Some(Path::new("1"))), + ( + "ROCM_CLI_THEROCK_RELEASE_PIP_BASE", + Some(Path::new(unreachable_index)), + ), + ]); + let _resolver = OverridePythonResolverEnv::new(PythonResolverEnv { + python_override: None, + search_dirs: vec![search], + ..PythonResolverEnv::from_process_env() + }); + + // The fixture's recorded interpreter does not exist, so the runtime + // records no usable Python. + let manifest = test_runtime_manifest("wheel-no-python", "therock-release:gfx120X-all", 1); + write_test_runtime_manifest(&paths, &manifest)?; + let manifests = load_runtime_manifests(&paths)?; + + let prepared_nothing = |what: &str| { + assert!(!marker.exists(), "{what} ran uv"); + assert!( + !paths.config_path().exists(), + "{what} saved a Python path in the configuration" + ); + assert!( + !rocm_core::managed_tools_dir(&paths.data_dir).exists(), + "{what} created the managed tools folder" + ); + }; + + let report = render_update_report(&paths)?; + assert!(report.contains("127.0.0.1:9"), "{report}"); + prepared_nothing("the update report"); + + let document = render_update_json(&paths, Some(1))?; + assert_eq!(document.runtimes[0].status, "error"); + assert!( + document.runtimes[0] + .message + .as_deref() + .is_some_and(|message| message.contains("127.0.0.1:9")), + "{document:?}" + ); + prepared_nothing("the update JSON"); + + let record = maybe_refresh_startup_update_check_at(&paths, None, 0)? + .expect("a registered runtime is checked at startup"); + assert!( + record + .message + .as_deref() + .is_some_and(|message| message.contains("127.0.0.1:9")), + "{record:?}" + ); + prepared_nothing("the startup check"); + + let error = runtime_update_check(&paths, &manifest, &manifests, Some(1)) + .expect_err("the index is unreachable") + .to_string(); + assert!(error.contains("127.0.0.1:9"), "{error}"); + prepared_nothing("the update check"); + + let mut config = rocm_core::RocmCliConfig::default(); + let preview = + crate::update_cmd::apply_runtime_update(&paths, &mut config, None, false, true) + .expect_err("the index is unreachable") + .to_string(); + assert!(preview.contains("127.0.0.1:9"), "{preview}"); + prepared_nothing("the --dry-run preview"); + + // Control: the plan an apply works from does reach `uv` on this host, and + // so does a real apply, which takes that plan. + assert!(runtime_update_plan(&paths, &manifest, &manifests, Some(1)).is_err()); + assert!( + marker.exists(), + "the stand-in uv is not reachable, so the checks above prove nothing" + ); + fs::remove_file(&marker)?; + assert!( + crate::update_cmd::apply_runtime_update(&paths, &mut config, None, false, false) + .is_err() + ); + assert!( + marker.exists(), + "a real apply did not take the plan that prepares a Python" + ); + + fs::remove_dir_all(&root).ok(); + Ok(()) + } } diff --git a/apps/rocm/src/update_cmd.rs b/apps/rocm/src/update_cmd.rs index 1b64a253c8..c329c90d9f 100644 --- a/apps/rocm/src/update_cmd.rs +++ b/apps/rocm/src/update_cmd.rs @@ -254,16 +254,61 @@ pub(crate) const fn update_should_preview_or_apply(apply: bool, dry_run: bool) - apply || dry_run } +/// Which planner each path takes is pinned in two places. The `--dry-run` choice, +/// the check and not the plan, is pinned on every platform by +/// `a_dry_run_takes_the_check_and_an_apply_takes_the_plan` below, which runs +/// `apply_runtime_update_with`. That `runtime_update_check` is the check and +/// `runtime_update_plan` the plan, which the two arguments below cannot tell +/// apart by type, is pinned only on Unix: by +/// `the_update_checks_prepare_no_python_and_the_apply_plan_does` in +/// `therock/update.rs` and by scenario `update-10` on Linux. No test on Windows +/// pins that pairing. pub(crate) fn apply_runtime_update( paths: &AppPaths, config: &mut RocmCliConfig, runtime_selector: Option<&str>, activate: bool, dry_run: bool, +) -> Result { + apply_runtime_update_with( + paths, + config, + runtime_selector, + activate, + dry_run, + |source, manifests| therock::runtime_update_check(paths, source, manifests, None), + |source, manifests| therock::runtime_update_plan(paths, source, manifests, None), + ) +} + +/// The update flow with its two planners passed in. A `--dry-run` only previews, +/// so it takes `check`, which prepares nothing; a real apply takes `apply_plan`, +/// which prepares the Python the install that follows needs. Taking them as +/// arguments lets a test run the choice on every platform without a host that +/// has no Python. +fn apply_runtime_update_with( + paths: &AppPaths, + config: &mut RocmCliConfig, + runtime_selector: Option<&str>, + activate: bool, + dry_run: bool, + check: impl FnOnce( + &therock::InstalledRuntimeManifest, + &[therock::InstalledRuntimeManifest], + ) -> Result, + apply_plan: impl FnOnce( + &therock::InstalledRuntimeManifest, + &[therock::InstalledRuntimeManifest], + ) -> Result, ) -> Result { let manifests = therock::load_runtime_manifests(paths)?; let source = select_runtime_update_source(&manifests, config, runtime_selector)?; - let plan = therock::runtime_update_plan(paths, source, &manifests, None)?; + let planned = if dry_run { + check(source, &manifests) + } else { + apply_plan(source, &manifests) + }; + let plan = planned?; let mut output = String::new(); let _ = writeln!(output, "runtime update"); let _ = writeln!(output, " source_runtime_key: {}", source.runtime_key); @@ -289,6 +334,11 @@ pub(crate) fn apply_runtime_update( return Ok(output); } + // The plan's `python_note` is not printed here: the preview below prints its + // own `python_note:` from the decision a real install takes (scenario + // `update-10` asserts it), so a second line would say the same thing twice. + // A dry run that finds nothing newer returned above and prints neither; the + // plain `rocm update` report is where the check's note shows. if dry_run { let _ = writeln!(output, " mode: dry-run"); let install_plan = therock::install_sdk_for_update( @@ -440,6 +490,73 @@ mod tests { use crate::test_support::write_test_pip_runtime; use crate::tests::{test_paths, test_runtime_manifest_for_update}; + /// A `--dry-run` takes its plan from the check and an apply from the plan, + /// and calls only that one: the check must never be the one that prepares a + /// Python, and an apply must still get the plan that does. Both planners + /// report nothing newer, so the flow returns before it installs or previews + /// anything and the test needs no host without a Python, which runs it on + /// every platform. Which `therock` function is the check is pinned only on + /// Unix, by `the_update_checks_prepare_no_python_and_the_apply_plan_does` in + /// `therock/update.rs`, and by scenario `update-10`. + #[test] + fn a_dry_run_takes_the_check_and_an_apply_takes_the_plan() -> Result<()> { + let (root, paths) = test_paths("update-takes-check-or-plan"); + write_test_pip_runtime( + &paths, + "release-pip-gfx120x-all", + "therock-release:gfx120X-all", + "7.13.0a20260416", + 1, + )?; + let calls = std::cell::RefCell::new(Vec::new()); + let planner = |name: &'static str| { + let calls = &calls; + move |source: &therock::InstalledRuntimeManifest, + _: &[therock::InstalledRuntimeManifest]| { + calls.borrow_mut().push(name); + Ok(therock::RuntimeUpdatePlan { + latest_version: source.version.clone(), + latest_source: String::new(), + format: source.format.clone(), + status: "up_to_date".to_owned(), + target_runtime_key: source.runtime_key.clone(), + device_target: None, + source_layout_generation: None, + repair_required: false, + update_available: false, + python_note: None, + }) + } + }; + let take = |dry_run: bool| -> Result<(String, Vec<&'static str>)> { + calls.borrow_mut().clear(); + let output = apply_runtime_update_with( + &paths, + &mut RocmCliConfig::default(), + None, + false, + dry_run, + planner("check"), + planner("plan"), + )?; + Ok((output, calls.borrow().clone())) + }; + let (output, taken) = take(true)?; + assert_eq!(taken, ["check"], "a --dry-run took the plan: {output}"); + assert!( + output.contains("result: no newer runtime found"), + "{output}" + ); + let (output, taken) = take(false)?; + assert_eq!(taken, ["plan"], "an apply took the check: {output}"); + assert!( + output.contains("result: no newer runtime found"), + "{output}" + ); + fs::remove_dir_all(root).ok(); + Ok(()) + } + #[test] fn render_update_text_reports_all_update_surfaces() -> Result<()> { let (root, paths) = test_paths("update-surfaces"); diff --git a/crates/rocm-dash-tui/src/agent/tools.rs b/crates/rocm-dash-tui/src/agent/tools.rs index 0cf7d08d56..6e9e2f6ac9 100644 --- a/crates/rocm-dash-tui/src/agent/tools.rs +++ b/crates/rocm-dash-tui/src/agent/tools.rs @@ -208,15 +208,16 @@ pub const TELEMETRY_TOOL_NAMES: [&str; 4] = [ // stop has finished). `doctor`/`examine` can re-register the configured setup // runtime in the runtime registry when its entry is missing. `update_check` and // `install_sdk_dry_run` cache package-index metadata under the cache folder. -// `update_check` can also, on a host with no usable Python on PATH, download a -// managed `uv` and Python into the data folder and record that Python in the -// saved config; `install_sdk_dry_run` prepares no Python and saves no Python -// path: on such a host it reports the Python a real install would prepare. It -// fetches no `uv` either: a ROCm 10 preview runs a `uv` that is already -// installed to resolve the package versions and, with none, prints the lines -// that depend on them as not resolved, so it creates no `tools` folder in the -// data folder. (`update_check` still resolves with `uv` and can fetch it.) The -// boundary type +// Neither prepares a Python or saves a Python path: on a host with no usable +// Python on PATH they report the Python a real update or install would prepare +// (`update_check` runs `rocm update`, whose text report then carries a +// `python_note:` line saying so), and only a real apply or install, which these +// tools never run, prepares it. `install_sdk_dry_run` fetches no `uv` either: a +// ROCm 10 preview runs a `uv` that is already installed to resolve the package +// versions and, with none, prints the lines that depend on them as not resolved, +// so it creates no `tools` folder in the data folder. (`update_check` on a ROCm +// 10 or newer runtime still resolves with `uv` and can fetch the managed `uv` +// into the data folder, though never a Python.) The boundary type // (`SharedRocmToolExecutor`) is plain data — importing it does NOT pull // `rocm-core` into this crate; the `agent` module stays the sole `rig` namer. // --------------------------------------------------------------------------- diff --git a/docs/architecture.md b/docs/architecture.md index 34a7b4ff3e..2c9377e771 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -108,7 +108,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/fetch.rs`](../apps/rocm/src/therock/fetch.rs) — full domain extraction, a child of `therock.rs` that depends on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for atomic writes and on `slugify` in the parent for cache file names (split out of `therock.rs`): HTTP fetching for installs. `download_text_cached` is the metadata GET with its on-disk etag cache (`CachedHttp*`) and detached-signature verification (`MetadataSignaturePolicy`, `PINNED_METADATA_PUBLIC_KEY_PEM`, the unset trust root that `cargo xtask verify-pinned-keys` checks against the canonical published key, so its path is pinned in `xtask`); `download_file` is the streaming artifact download; `head_content_length` and `preflight_tarball_space` size the download and check free space first; `http_get` is the shared request. `truthy_env` is `pub(crate)` and re-exported by `therock.rs` for [`driver_plan/rocdxg.rs`](../apps/rocm/src/driver_plan/rocdxg.rs); `download_text_cached`, `download_file`, `preflight_tarball_space` and the `CachedHttpText` struct (with its `text` field) that `download_text_cached` returns are `pub(super)` for the install and resolution code that stays in `therock.rs`; everything else is private. - [`therock/source.rs`](../apps/rocm/src/therock/source.rs) — full domain extraction, a child of `therock.rs` that depends only on `therock/version.rs` (split out of `therock.rs`): where an install reads from. `TheRockChannel` (release or nightly) and `SourceLayout` (the canonical stream, or ROCm 10's separately hosted next layout) are the two axes; `resolve_source` maps a pair to the `ResolvedSource` wheel-index and tarball-catalog bases, honouring a `ROCM_CLI_THEROCK_*_BASE` override only behind the explicit opt-in (`therock_base_override_allowed`, `select_base_override`, `env_override_base`); `select_source_layout`, `requested_source_layout` and `next_layout_requested` decide the layout from a pin; `RuntimeVersionSelector` is the version or build date a user pins (`pub(crate)`, re-exported by `therock.rs`). `FamilyResolution` lives here rather than in `therock/family.rs` because choosing the layout reads it; that keeps the dependency one-way, with `family.rs` building on this file. `TheRockChannel`, `SourceLayout`, `ResolvedSource` and `FamilyResolution` (with their methods and fields), `RuntimeVersionSelector`'s `describe` and `matches_version`, `resolve_source`, the three layout functions, and the `THEROCK_SOURCE_LAYOUT_GENERATION` and `THEROCK_NEXT_MIN_MAJOR` constants are `pub(super)` for `therock.rs` and its siblings; the base-override helpers and the other constants (the per-channel and next-layout base URLs, and the next layout's generation tag) are private to this file. - [`therock/family.rs`](../apps/rocm/src/therock/family.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` for `TheRockChannel` and `FamilyResolution` (split out of `therock.rs`): which package family a request targets. `resolve_family` picks it from `--family`, `ROCM_CLI_THEROCK_FAMILY`, a managed runtime or the host GPU, in that order, and refuses a GPU with no ROCm build (`unsupported_host_gpu_message`, `unsupported_requested_target_message`) instead of substituting another family; `family_override_or_recovered_arch` and `raw_arch_agreeing_with_family` recover the exact arch behind a grouped family; `family_resolution_hint` is the recovery text appended when no runtime is found. `resolve_family`, `family_override_or_recovered_arch` and `family_resolution_hint` are `pub(super)`; the rest is private. -- [`therock/python.rs`](../apps/rocm/src/therock/python.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/io.rs`, `therock/process.rs` and `therock/source.rs` (split out of `therock.rs`): the Python an install runs on. `python_requirement` maps a `SourceLayout` to the interpreter the layout needs (`PythonRequirement`: wheel tag and `uv` version), which is why `source.rs` had to move first; `resolve_python_launcher` finds one in order of an explicit `ROCM_CLI_PYTHON`, `PATH`, the saved managed Python, then `ensure_managed_python`, which installs one through `uv` and records it in the managed-Python manifest; the resolver reads its environment through `PythonResolverEnv`, so tests steer it with a thread-local override (`OverridePythonResolverEnv`) rather than rewriting `PATH`; `wheel_compatibility_for_python` and `current_platform_wheel_tags` report the wheel tags an interpreter and platform accept (`WheelCompatibility`); `ensure_uv_venv` creates, or recreates when the interpreter tag no longer matches, the runtime's virtual environment. `venv_python_path` stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The items `therock.rs` reaches, and `split_runtime_path`, which `therock/probe.rs` reads, are `pub(super)`; the rest is private, including `ensure_managed_python` and `resolve_python_launcher_in` (the resolver's entry point that takes its environment as an argument). Its Unix-only tests drive fake interpreters and `uv` scripts. +- [`therock/python.rs`](../apps/rocm/src/therock/python.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/io.rs`, `therock/process.rs` and `therock/source.rs` (split out of `therock.rs`): the Python an install runs on. `python_requirement` maps a `SourceLayout` to the interpreter the layout needs (`PythonRequirement`: wheel tag and `uv` version), which is why `source.rs` had to move first; `resolve_python_launcher` finds one in order of an explicit `ROCM_CLI_PYTHON`, `PATH`, the saved managed Python, then `ensure_managed_python`, which installs one through `uv` and records it in the managed-Python manifest; the resolver reads its environment through `PythonResolverEnv`, so tests steer it with a thread-local override (`OverridePythonResolverEnv`) rather than rewriting `PATH`; `wheel_compatibility_for_python` and `current_platform_wheel_tags` report the wheel tags an interpreter and platform accept (`WheelCompatibility`); `ensure_uv_venv` creates, or recreates when the interpreter tag no longer matches, the runtime's virtual environment. `plan_python_launcher` is the same lookup for a caller that must change nothing, an install preview or an update check: it returns a `PlannedPython`, either `Ready` with the interpreter a real install would pick or `WouldPrepareManaged` with the managed Python a real install would prepare and this caller did not, and it raises the refusals the real resolution raises before that step. `python_for_install` is the install's one choice between that and `resolve_python_launcher`, and prints the progress line that names the interpreter; `PlannedPython` also writes the `python_note:` line from the decision itself, `note` for an install preview and `update_note` for an update check, so a note cannot name a version the install will not prepare. `venv_python_path` stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. The items `therock.rs` reaches, `plan_python_launcher` with `PlannedPython`'s `wheel_compatibility` and `update_note`, which `therock/update.rs` reads, and `split_runtime_path`, which `therock/probe.rs` reads, are `pub(super)`; the rest is private, including `ensure_managed_python` and `resolve_python_launcher_in` (the resolver's entry point that takes its environment as an argument). Its Unix-only tests drive fake interpreters and `uv` scripts. - [`therock/probe.rs`](../apps/rocm/src/therock/probe.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/process.rs` and `therock/python.rs` (split out of `therock.rs`): asking an installed runtime about itself through its own interpreter. `probe_rocm_sdk_runtime` runs `ROCM_SDK_PROBE_SCRIPT` and parses what `rocm_sdk` reports into the `RocmSdkPythonProbe` that [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) owns and validates; `probe_runtime_devices` runs `RUNTIME_DEVICE_PROBE_SCRIPT` to count the devices the runtime's torch can open and whether one can run a kernel (`RuntimeDeviceProbe`); `probe_torch_alignment` reads the installed torch and the one an engine pins from distribution metadata without importing torch (`TorchAlignmentProbe`); and `split_local_version`, `requirement_pinned_version` and `install_pinned_package` are the torch-pin helpers [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) uses to put the SDK's torch back after an engine install. The `pub(crate)` items are re-exported by `therock.rs` for [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs); `probe_rocm_sdk_runtime_for_target`, which `install_wheel_runtime` calls with the device payload it just chose, is `pub(super)`. - [`therock/resolve.rs`](../apps/rocm/src/therock/resolve.rs) — a child of `therock.rs` that holds no logic (split out of `therock.rs`): only the `mod` declarations for the artifact-resolution modules under `therock/resolve/` and the `pub(super) use` lines through which `therock.rs` reads them. It is the parent the `resolve/` files hang from, so a module there is `pub(in crate::therock)` for the items `therock.rs` reaches and private otherwise. - [`therock/resolve/tarball.rs`](../apps/rocm/src/therock/resolve/tarball.rs) — full domain extraction, a child of `therock/resolve.rs` that builds on `therock/fetch.rs`, `therock/family.rs`, `therock/source.rs` and `therock/version.rs` (split out of `therock.rs`): resolving the SDK tarball a request installs. `resolve_tarball_artifact` (and `resolve_tarball_artifact_with_timeout`, which takes the metadata download timeout) resolves the family and source layout, downloads the catalog through the cached metadata fetch, and returns the `TarballArtifact` it chose; `parse_tarball_index_html` reads the catalog's embedded file list into `TarballIndexFile`s and `validate_tarball_file_name` refuses an entry that is not a plain file name; `select_tarball_candidate` picks the newest archive the request can install, with `tarball_family_token` spelling the family the way the next layout's catalog does. `platform_tarball_token`, which names the archives' platform, stays in `therock.rs` for good, with the other keys and roots helpers, and this module reads it from there. Only `resolve_tarball_artifact*` and `TarballArtifact` are visible beyond this file. @@ -116,12 +116,12 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`therock/resolve/uv_access.rs`](../apps/rocm/src/therock/resolve/uv_access.rs) — full domain extraction, a leaf child of `therock/resolve.rs` that builds only on `therock/source.rs` for `SourceLayout` (split out of `therock/resolve/wheel.rs` to keep it within the module-size limit): who may fetch `uv` while a wheel stack is resolved. `UvAccess` says how far a resolution may go (`Acquire`, what a real install and `rocm update` pass, or `PresentOnly`, a preview's), `plan_version_resolution` turns that, the layout and whether a `uv` is present into a `VersionResolution` (scrape the indexes, compile with a `UvChoice`, or skip) without touching anything, and `uv_for_compile` hands the compile step the `uv` it runs, acquiring it only on `Acquire`, with the context a failed fetch has always carried. `UvPolicy` carries the access and the two real lookups (`find_uv_binary`, `ensure_uv_binary`) as closures so a test can count what a resolution asks for. Everything is `pub(super)` and read by `wheel.rs`; the state matrix over layout, access and presence lives in this module's tests. - [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/source.rs` and `therock/version.rs` and reaches the registry through [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) (split out of `therock.rs`): the consent gate for an install that would displace the active default runtime. `active_default_runtime_relation` says what the install would replace (an upgrade, downgrade or reinstall when it is the same family and channel, otherwise the different runtime it displaces) or `None` when nothing is the active default, and it fails closed, with a relation naming what could not be resolved, when the config names an active default the registry cannot resolve; `SdkInstallConsent` and `SdkInstallApprovalSource` say how consent is obtained (asked for, or already granted by `--yes`, `--approve-replacing-active-default` or `rocm update --apply`); `sdk_install_approval` turns those and whether a terminal is attached into proceed, prompt or refuse (`SdkInstallApproval`); `preapproved_install_line`, `fresh_install_line`, `refuse_non_interactive_message` and `confirm_overwrite_existing_sdk` are the wording of each outcome, and the prompt reads its answer through `prompt_yes_no` and the shared [`confirm.rs`](../crates/rocm-core/src/confirm.rs). `SdkInstallConsent`, `SdkInstallApprovalSource` and `preapproved_install_line` are re-exported by `therock.rs`; the other items `therock.rs` calls are `pub(super)`, as is `active_default_relation_text`, which the version property tests call, and the rest is private. - [`therock/wheel_plan.rs`](../apps/rocm/src/therock/wheel_plan.rs) — full domain extraction, a leaf child of `therock.rs` (split out of `install_wheel_runtime`) that builds on `therock/python.rs` and `therock/resolve.rs` for the Python decision and the resolution it prints: the `sdk install` plan a wheel install prints, key by key. `WheelPlan` carries the facts, `write_wheel_plan` writes the lines a preview and a real install share, and `write_dry_run_tail` the lines only a preview adds (the `uv` commands it would run and the manifest path). `PlanStack` is the one place the plan differs: `Resolved` carries the package specs, runtime key, install folder and manifest path, and `NeedsUv` (a ROCm 10 preview on a host with no `uv`, see `UnresolvedPipRuntime` in `therock/resolve/wheel.rs`) prints those as `not resolved by this preview` and adds the `resolution_note:` line, so one writer serves both and a line the plan gains appears in both. `unresolved_wheel_preview` renders the second kind whole. `therock.rs` reads it through `pub(super)` items; it reads the provenance, host-note and `venv_python_path` helpers back from `therock.rs`. -- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests and on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for the startup-check record (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan` and `maybe_refresh_startup_update_check` are `pub(crate)` and re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. -- [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. +- [`therock/update.rs`](../apps/rocm/src/therock/update.rs) — full domain extraction, a child of `therock.rs` that builds on `therock/family.rs`, `therock/process.rs`, `therock/python.rs`, `therock/resolve.rs`, `therock/source.rs` and `therock/version.rs`, on [`runtime_registry.rs`](../apps/rocm/src/runtime_registry.rs) for the manifests and on rocm-core's [`atomic_write.rs`](../crates/rocm-core/src/atomic_write.rs) for the startup-check record (split out of `therock.rs`): whether an installed runtime has something newer, and the bounded startup check that asks. `runtime_update_plan` resolves the latest source for one runtime, and `runtime_update_check` does the same for a caller that changes nothing (the report, the JSON, the startup check and an apply's preview): given a wheel runtime that records no usable Python, the plan prepares a managed Python (an apply needs one) while the check works the wheel tag out from the interpreter the layout requires and carries a `python_note` instead (`resolve_latest_for_manifest` re-resolves a wheel runtime through `resolve_pip_runtime_with_timeout` and a tarball one through `resolve_tarball_artifact_with_timeout`, on the layout `manifest_source_layout` reads back from the manifest) and `runtime_freshness` classifies it (`RuntimeFreshness`: up to date, update available, repair available when the installed composition is not the one a fresh install would record, or ahead of the index) into a `RuntimeUpdatePlan`. `render_update_report` and `render_update_json` (the form the dashboard's update check parses) present the plans, and `maybe_refresh_startup_update_check` keeps one `StartupUpdateCheckRecord` in the cache so the check runs at most once every twelve hours, under a two-second download timeout. `runtime_key` and `wheel_runtime_key`, which name the install a plan targets, stay in `therock.rs`; whether a replacement's root is installed is `rocm_core::install_root_has_payload`. `render_update_report`, `render_update_json`, `runtime_update_plan`, `runtime_update_check` and `maybe_refresh_startup_update_check` are `pub(crate)` and, with `RuntimeUpdatePlan`, which [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) reads its verdict from, re-exported by `therock.rs`; `RuntimeFreshness`, `runtime_freshness` and `select_startup_update_manifest` are `pub(super)` for the version property tests, and the rest is private. +- [`therock/test_support.rs`](../apps/rocm/src/therock/test_support.rs) — test-only (`#[cfg(test)]`), a child of `therock.rs`: fixtures shared by the `therock` child modules' tests and the tests left in `therock.rs` (`test_paths`, `test_runtime_manifest`, `write_test_runtime_manifest`, `family_for_test`, and `RestoredEnv`, which sets or removes process environment variables for one test and puts them back on drop, so a caller holds `PROCESS_ENV_TEST_LOCK` for as long as it holds one). Separate from the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs), which holds the fixtures every module's tests share, such as `PROCESS_ENV_TEST_LOCK` and the background-helper fixtures `runtime_state` and `set_helper_facts`. - [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) — the chat assistant's tool-call policy: which `rocm` commands and bridge tools a model may request, which run immediately and which need the user's approval first. It owns `ChatRocmCommandAction` (read-only, or approval with the titles the approval prompt shows) and `ChatToolApprovalRequest`, the validators (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `validate_required_chat_string`, `validate_optional_chat_integer`, `reject_control_characters_in_tool_arguments` (which refuses a model-supplied tool name or argument carrying a terminal control character, for every tool except `rocm_command`, whose argv `normalized_chat_rocm_command_args` screens entry by entry), `validate_serve_command_safety`, the four rules a model-proposed `serve` must satisfy (no public bind, no CPU device, a loopback host, `--managed` without `--foreground`), and `validate_chat_rocm_command_safety`, which applies them to a `serve` argv and also refuses a system install folder (`chat_install_prefix_is_system`)), the classifiers (`chat_rocm_command_action`, `chat_rocm_command_action_from_args`, `chat_install_driver_action`, `canonicalize_chat_rocm_command`, `normalized_chat_rocm_command_args`, `chat_tool_call_is_read_only`), `chat_tool_approval_request`, and the readers that turn a tool call into its argv and one-line command (`rocm_chat_tool_requested_args`, `rocm_chat_tool_requested_command`, with `push_optional_json_cli_arg` and `json_string`). It moved verbatim from `chat.rs` with the tests that exercise only these items. It depends on [`rocm_argv.rs`](../apps/rocm/src/rocm_argv.rs), `providers::ChatToolCall`, `service_launch::is_loopback_host`, `services_text::validate_service_id`, `shell_quote::has_terminal_control_character`, `therock::RuntimeVersionSelector` and `driver_install::host_install_driver_only_reports`, none of which imports `chat`, so `freeform_planner.rs` calls `validate_serve_command_safety` and `validate_chat_rocm_command_safety` here instead of through `chat`, and keeps no copy of the serve rules. `chat.rs`'s child modules call into it and `chat.rs` re-exports `validate_chat_tool_call`, `chat_tool_call_is_read_only` and `chat_tool_approval_request` (read by `dash_seam.rs`), and `ChatRocmCommandAction`, `chat_install_driver_action`, `chat_rocm_command_action_from_args` and `rocm_chat_tool_requested_args`, which only other modules' tests read as `crate::chat::` (`driver_plan.rs`, `system_packages.rs`). - [`uninstall.rs`](../apps/rocm/src/uninstall.rs) — mechanical relocation, owns types: `uninstall()`, the refusal of the hidden `--force` (`reject_force`, which `main.rs` calls from the `uninstall` dispatch arm), plus the installer `PATH` undo and running-executable deletion. The shared plan type and its file-removal helpers live in [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) below instead, because [`storage.rs`](../apps/rocm/src/storage.rs) reuses them too. The two modules depend on each other: `uninstall_plan.rs` reads this module's `pub(crate)` `PathEdit`, `PATH_EDIT_RECORD`, `plan_path_edits`, `plan_is_empty`, `removal_count` and `render_extra_plan`. - [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) — mechanical relocation, shares crate-visible types (Phase 6f, ROCMAI-212): the uninstall-plan types (`UninstallOptions`, `UninstallPlan`, `UninstallPlanEntry`), the `PlanAudience` a rendered plan is written for (a person at a terminal, a captured caller, or the assistant's preview), which decides what the dry-run footer says comes next, building the plan (`build_uninstall_plan`, `plan_installed_binaries`, and the installed-binary/shared-cache discovery helpers behind them), rendering it (`render_uninstall_dry_run`, which renders for the assistant, and `render_uninstall_plan`), the `[y/N]` confirmation prompt (`confirm_uninstall`), and the filesystem removal primitive both [`uninstall.rs`](../apps/rocm/src/uninstall.rs) and [`storage.rs`](../apps/rocm/src/storage.rs) call to carry a plan out (`remove_path`, and the entry-resolution helpers behind it). Kept separate from `uninstall.rs` rather than folded into it, since `uninstall.rs`'s own doc comment and name both describe `uninstall()`'s exclusive logic (the installer `PATH` undo and running-executable deletion), while this module's plan/render/removal helpers are shared with `storage.rs`. `main.rs` (`UninstallOptions`), `uninstall.rs` (which also imports `PlanAudience` and picks it from `interactive_terminal`), `storage.rs`, `runtimes_cmd.rs`, `chat/mcp_bridge.rs` (which calls `render_uninstall_dry_run` and does not import `PlanAudience`), and [`deletion_properties.rs`](../apps/rocm/src/deletion_properties.rs) (Unix-only property tests, which render as `PlanAudience::Captured`) each import the subset of these `pub(crate)` types and helpers they need; see this module's own doc comment for which symbol each one uses. -- [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types (Phase 6f, ROCMAI-210): `rocm update`'s command handler (`update()`, the body of the `update` arm of `dispatch()`) and its rendering/activation logic — the read-only status report (`render_update_text`, `append_update_surfaces`), the predicate deciding whether `update` should route into the mutating path instead of that report (`update_should_preview_or_apply`), applying a runtime update (`apply_runtime_update`, `append_update_activate_summary`), and the update-source/installed-runtime pickers behind it (`select_runtime_update_source`, `select_installed_update_runtime`). `const ROLLBACK_RECOVERY_HINT` stays in `main.rs` as a shared crate-root constant, because callers outside this module also use it ([`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimesCommand::Activate` handler and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `render_sdk_install_success`); this module reaches it via its `use crate::{...};` import, the same way `gpu_policy.rs`'s `device_policy_name` stays in `main.rs`. `update()` is `pub(crate)` and called from [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` through a re-import (`use crate::update_cmd::update;`), and calls `update_should_preview_or_apply`, `apply_runtime_update` and `render_update_text` itself; `render_update_text` is also `pub(crate)` and read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the read-only `rocm update` handling), while `apply_runtime_update` and `update_should_preview_or_apply` keep the `pub(crate)` they had when `dispatch()` called them directly, though only `update()` calls them now; `select_runtime_update_source` is also `pub(crate)`, read from [`therock_version_proptests.rs`](../apps/rocm/src/therock_version_proptests.rs)'s property tests via `crate::update_cmd::select_runtime_update_source`. `append_update_surfaces`, `append_update_activate_summary`, and `select_installed_update_runtime` have no callers outside this module and stay private. The cluster's regression tests live in `update_cmd.rs`'s own `mod tests`, reaching shared fixtures (`test_paths`, `test_runtime_manifest_for_update`, `write_test_pip_runtime`) through `crate::tests::`/`crate::test_support::` the same way other mechanically relocated modules do. +- [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types (Phase 6f, ROCMAI-210): `rocm update`'s command handler (`update()`, the body of the `update` arm of `dispatch()`) and its rendering/activation logic — the read-only status report (`render_update_text`, `append_update_surfaces`), the predicate deciding whether `update` should route into the mutating path instead of that report (`update_should_preview_or_apply`), applying a runtime update (`apply_runtime_update`, a thin wrapper that hands `apply_runtime_update_with` the check and the plan from `therock/update.rs`, so a unit test can run the choice between them on any platform; `append_update_activate_summary`), and the update-source/installed-runtime pickers behind it (`select_runtime_update_source`, `select_installed_update_runtime`). `const ROLLBACK_RECOVERY_HINT` stays in `main.rs` as a shared crate-root constant, because callers outside this module also use it ([`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimesCommand::Activate` handler and [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `render_sdk_install_success`); this module reaches it via its `use crate::{...};` import, the same way `gpu_policy.rs`'s `device_policy_name` stays in `main.rs`. `update()` is `pub(crate)` and called from [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` through a re-import (`use crate::update_cmd::update;`), and calls `update_should_preview_or_apply`, `apply_runtime_update` and `render_update_text` itself; `render_update_text` is also `pub(crate)` and read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the read-only `rocm update` handling), while `apply_runtime_update` and `update_should_preview_or_apply` keep the `pub(crate)` they had when `dispatch()` called them directly, though only `update()` calls them now; `select_runtime_update_source` is also `pub(crate)`, read from [`therock_version_proptests.rs`](../apps/rocm/src/therock_version_proptests.rs)'s property tests via `crate::update_cmd::select_runtime_update_source`. `append_update_surfaces`, `append_update_activate_summary`, and `select_installed_update_runtime` have no callers outside this module and stay private. The cluster's regression tests live in `update_cmd.rs`'s own `mod tests`, reaching shared fixtures (`test_paths`, `test_runtime_manifest_for_update`, `write_test_pip_runtime`) through `crate::tests::`/`crate::test_support::` the same way other mechanically relocated modules do. Further `apps/rocm` modules, not part of the `main.rs` split: diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 062ec8d8cc..f61b855ce8 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -178,6 +178,16 @@ Expected result: preview`. Run the real install (or any command that fetches `uv`), then the preview again, and it prints the complete plan; so does a preview with `ROCM_CLI_UV_BINARY` naming a `uv`. +- On the same host, with a wheel runtime registered whose manifest records no + usable Python, `rocm update` (also `--json`, `--dry-run`, and the check the + CLI runs at startup) still reports. The text report gives the runtime's + verdict and a `python_note:` line saying an update would use `uv` to install + the managed Python, which the check did not do. Afterwards the data folder has + no `tools` folder and the config folder has no `config.json`. That holds for a + canonical-layout runtime; for a ROCm 10 or newer runtime the check still + resolves the package versions with `uv` and can fetch the managed `uv` into the + data folder, though never a Python. `rocm update --apply` on the same host does + prepare the Python, since the install needs it. - With `ROCM_CLI_DISABLE_MANAGED_PYTHON_BOOTSTRAP=1`, or a `ROCM_CLI_MANAGED_PYTHON_VERSION` whose Python the layout does not accept (such as `3.13` for the canonical layout), on the same host the preview fails diff --git a/docs/testing.md b/docs/testing.md index 48eccb31c8..f04f56c40d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -903,8 +903,8 @@ progress output and that `uv` was not run and no file was written. Beside it are `a_dry_run_reads_every_spelling_of_a_managed_python_version_the_same_way`. The preview does still write its package-index cache under the cache folder. The `update_check` tool and the plain `rocm update` check (no `--apply`) are not -previews in this sense: they resolve with `uv` and can fetch it, and prepare a -managed Python too. +previews in this sense: on a ROCm 10 or newer runtime they resolve with `uv` and +can fetch it. They prepare no Python, though (see `update-09` and `update-10`). Then it verifies: @@ -2487,6 +2487,34 @@ cargo test -p rocm --bin rocm render_update_text_reports_all_update_surfaces surfaces honestly. `rocm update --apply` remains runtime-only until production metadata feeds exist for the other surfaces. +A check changes nothing, and that includes Python. Reading the index for a wheel +runtime needs the interpreter's wheel tag. A runtime that records a usable Python +supplies it. For one that does not, on a host with no usable Python, the check +used to download a managed `uv`, have it install Python and save that Python's +path in the config, only to ask it for the tag. The text report, `--json`, the +startup check that runs ahead of other commands, and the plan behind +`rocm update --apply --dry-run` now take the tag from the interpreter the layout +requires instead and prepare nothing; the text report adds a `python_note:` line +saying an update would use `uv` to install the managed Python, which the check did +not do. `--json` carries no such line, since its shape is the dashboard's +contract. `rocm update --apply` still prepares the Python, because the install +that follows needs it. Scenarios `update-09` (the report) and `update-10` +(`--dry-run`) check the note against the state it left, as `runtime-23` does for +the install preview: the recording `uv` was never run, and there is no +managed-tools folder and no saved config. They run on Linux; the entry points the +scenarios do not drive are covered by +`the_update_checks_prepare_no_python_and_the_apply_plan_does` in +[update.rs](../apps/rocm/src/therock/update.rs), which drives the report, the +JSON, the startup check, the check plan and `apply_runtime_update` with +`--dry-run`, with the plan an apply works from, and a real apply, as the controls +that show the stand-in `uv` is reachable. Which of the two plans a `--dry-run` and +an apply take is also pinned on every platform, Windows included, by +`a_dry_run_takes_the_check_and_an_apply_takes_the_plan` in +[update_cmd.rs](../apps/rocm/src/update_cmd.rs). Unlike the install preview, which fetches no +`uv`, the check on a ROCm 10 or newer runtime still resolves the package versions +with `uv pip compile` and can fetch the managed `uv`, though never a Python; the +scenarios above use the canonical layout, which needs no `uv`. + Runtime version-ordering tests: ```bash @@ -2494,6 +2522,13 @@ cargo test -p rocm --bin rocm version_proptests cargo xtask e2e -- -n 'update-0[78]' ``` +Update checks that must not prepare Python: + +```bash +cargo test -p rocm --bin rocm the_update_checks_prepare_no_python +cargo xtask e2e -- -n 'update-(09|10)' +``` + Where `rocm install sdk` and `rocm update` compare runtime versions, they compare them in one PEP 440 order over published version strings (`7.9` < `7.10.0`, `7.0.0b1` < `7.0.0rc1` < `7.0.0` < `7.0.0.post1`, `7.2.4.70204` above diff --git a/tests/e2e-cucumber/features/update.feature b/tests/e2e-cucumber/features/update.feature index 21ad0c1a35..078a27cfa2 100644 --- a/tests/e2e-cucumber/features/update.feature +++ b/tests/e2e-cucumber/features/update.feature @@ -92,3 +92,49 @@ Feature: Update report When the user checks for updates against that catalog Then the report calls the runtime the catalog spells differently up to date And the report offers no update for the four-component runtime that is ahead of the catalog + + # `rocm update` is a check: it reads the installed runtimes and the index and + # changes nothing. To tell which wheels fit a wheel runtime it needs the + # interpreter's tag, and on a host with no usable Python it used to go and get + # one: it downloaded a managed `uv`, had `uv` install Python and saved that + # Python's path in the configuration, all before printing the report. The same + # resolution runs for `rocm update --json` (the dashboard's check) and for the + # startup check that runs ahead of other commands, so a read-only look at the + # runtimes could prepare a Python. Now the check works the tag out from the + # interpreter the install would prepare, says it would prepare one, and + # prepares nothing. + # + # The report's note says the check prepared nothing, so the Then checks that + # note and the state together, as runtime-23 does for the install preview: the + # recording `uv` is reached only by going on to prepare a Python, so its log not + # existing means no preparation was attempted, and the managed tools folder and + # the configuration must be absent too. The first Then shows the check still got + # as far as a verdict, so the absences are not just a command that stopped + # early. `--json` and the startup check are pinned by the unit tests in + # `apps/rocm/src/therock/update.rs`. + # + # Linux only, for the same empty-`PATH` host as runtime-23. The canonical + # release index is served from a loopback fixture, so no network is needed. + @id:update-check-prepares-no-python @requires-os:linux + Scenario: update-09 - Checking for updates on a host with no usable Python prepares no Python + Given a canonical release pip index fixture + And a host with no usable Python and a recording uv + And a registered wheel runtime older than the index with no recorded Python + When the user checks for updates against the pip index + Then the report offers the newer runtime and says it would prepare a managed Python + And the update check prepared no Python, as it says + + # The sibling for `--dry-run`, which promises to show what `--apply` would do + # and change nothing. Its plan step resolved the same way and prepared the + # Python before the preview was printed; the apply itself still prepares what it + # needs, which the unit tests keep apart from the check. + # + # Linux only, for the same empty-`PATH` host as update-09. + @id:update-dry-run-prepares-no-python @requires-os:linux + Scenario: update-10 - Previewing an update on a host with no usable Python prepares no Python + Given a canonical release pip index fixture + And a host with no usable Python and a recording uv + And a registered wheel runtime older than the index with no recorded Python + When the user previews applying the update to that wheel runtime + Then the preview says a real install would prepare a managed Python + And the preview prepared no Python, as it says diff --git a/tests/e2e-cucumber/tests/e2e/therock_steps.rs b/tests/e2e-cucumber/tests/e2e/therock_steps.rs index 9a3ee87d27..ed2d54a65d 100644 --- a/tests/e2e-cucumber/tests/e2e/therock_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/therock_steps.rs @@ -502,6 +502,100 @@ async fn registered_next_wheel_runtime_with_grouped_family(world: &mut E2eWorld) .expect("failed to write runtime manifest"); } +/// Registry key for the planted canonical wheel runtime below. +const OLDER_WHEEL_RUNTIME_KEY: &str = "release-wheel-gfx1200-7-9-0"; + +/// Plants a canonical-layout wheel runtime older than the release index the +/// fixture serves, whose manifest records no Python (a runtime registered before +/// the field was written, or whose recorded interpreter is gone). The update +/// check then has to find an interpreter for itself, which on the host the +/// recording-uv Given builds means preparing one. No `source_layout_generation` +/// is recorded, so the manifest reads as the canonical stream, as an old one does. +#[given("a registered wheel runtime older than the index with no recorded Python")] +async fn registered_older_wheel_runtime_without_python(world: &mut E2eWorld) { + let install_root = root(world).join("runtime-older-wheel"); + std::fs::create_dir_all(&install_root).expect("failed to create install root"); + let registry = root(world).join("data").join("runtimes").join("registry"); + std::fs::create_dir_all(®istry).expect("failed to create registry dir"); + let manifest = serde_json::to_string_pretty(&serde_json::json!({ + "runtime_key": OLDER_WHEEL_RUNTIME_KEY, + "runtime_id": format!("therock-release:{RAW_ARCH}"), + "channel": "release", + "format": "wheel", + "family": RAW_ARCH, + "family_source": "manual", + "version": "7.9.0", + "install_root": install_root, + "selected_artifact_url": "https://example.invalid/rocm", + "read_only": false, + "installed_at_unix_ms": 1_700_000_000_000u64, + })) + .expect("failed to serialize runtime manifest"); + std::fs::write( + registry.join(format!("{OLDER_WHEEL_RUNTIME_KEY}.json")), + manifest, + ) + .expect("failed to write runtime manifest"); +} + +#[when("the user checks for updates against the pip index")] +async fn check_updates_against_pip_index(world: &mut E2eWorld) { + preview_ok(world, &["update"]); +} + +#[when("the user previews applying the update to that wheel runtime")] +async fn preview_apply_update_to_older_wheel_runtime(world: &mut E2eWorld) { + preview_ok( + world, + &[ + "update", + "--apply", + "--dry-run", + "--runtime", + OLDER_WHEEL_RUNTIME_KEY, + ], + ); +} + +/// The report reached a verdict (the runtime is behind the index) and says what +/// the update would do about Python, without a path that would be a guess. +#[then("the report offers the newer runtime and says it would prepare a managed Python")] +async fn report_offers_update_and_names_the_python_it_would_prepare(world: &mut E2eWorld) { + assert_contains( + world, + &format!("runtime {OLDER_WHEEL_RUNTIME_KEY} format=wheel"), + "the runtime line", + ); + assert_contains(world, "status=update_available", "the runtime's verdict"); + assert_contains( + world, + &format!("latest={CURRENT_ROCM_VERSION}"), + "the version the index publishes", + ); + assert_contains( + world, + &format!( + "python_note: no usable Python was found; updating would use uv to install managed \ + Python {CANONICAL_MANAGED_PYTHON}" + ), + "the note about the Python an update would prepare", + ); +} + +/// The check says it prepared no Python, and the state it left agrees. Claim and +/// filesystem are checked together: the note alone would pass with the bug +/// present, and so would empty folders alone if the check had stopped before it +/// got to Python. +#[then("the update check prepared no Python, as it says")] +async fn update_check_prepared_no_python(world: &mut E2eWorld) { + assert_contains( + world, + "which this check did not do", + "the note about what the check skipped", + ); + assert_no_python_prepared(world); +} + /// Makes the planted ROCm 10 runtime the active default, so a real install /// without consent reaches the non-interactive refusal: past every point where /// the plan is chosen and announced, but before `uv`, any download or any From e34c94bff3ce2699c7e689225345c56988ba0a05 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 17:48:11 +0000 Subject: [PATCH 030/102] fix(confirm): say when a [y/N] answer is not understood, and end the line on EOF (#982) * fix(confirm): say when a [y/N] answer is not understood, and end the line on EOF An answer other than y/yes was declined exactly like n, with nothing to tell the user it had not been understood, and Ctrl-D left the caller's cancel message on the prompt line because no newline was echoed. read_answer now takes the output stream: an unrecognized answer is still a no, but a line says so; and input that ended without a newline gets one before the caller prints. Enter, n and no stay silent, since Enter is the documented default. The writes are best effort so a closed stdout cannot turn a decline into an error. rocm uninstall, rocm fix and the SDK-overwrite prompt all share the helper and so all change. Covered by unit tests over each input shape, and by uninstall-07 to -09 under a pseudo-terminal, which assert the message together with the config folder being left in place. Fixes #790 Signed-off-by: Jussi Elo * test(confirm): pin that a no with no final newline declines and ends the prompt line, and say piped input does too read_answer writes a newline when the line it read did not end in one. The doc named only EOF from Ctrl-D, but piped input such as printf no | has no final newline either and the cursor is on the prompt line in both cases, so the newline is right for it too. The doc now says so, and a test pins the decline and the lone newline for a no without a newline. Signed-off-by: Jussi Elo * fix(confirm): cut the answer echoed in the not-understood line to 40 characters The line quoted the whole trimmed answer, so a long pasted line was printed in full. It now shows at most 40 characters, counted as characters so a multi-byte one is never split, and marks a cut with an ellipsis after the closing quote. The answer is still quoted with control characters escaped, and the limit counts the typed characters, so an escaped line stays bounded too. Signed-off-by: Jussi Elo * docs(uninstall): say what a cancelled uninstall prints and returns, and that a long answer is shown cut The README said Enter, n and no cancel quietly, which reads as printing nothing while uninstall cancelled is printed. It now says what happens, as the uninstall scenarios assert: that line, nothing removed, exit code 0. One overlong line left by the reflow is rewrapped. The manual testing steps also say that an answer longer than 40 characters is shown cut. Signed-off-by: Jussi Elo * docs(architecture): say read_answer also ends the line for piped input and cuts a long echoed answer The confirm.rs entry named only Ctrl-D for the newline and said nothing about the length of the echoed answer. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit ca23a1c7de9653064c8f8ce4c1293f00fd3a49da) Signed-off-by: juhovainio --- AGENTS.md | 2 +- README.md | 7 +- apps/rocm/src/therock/consent.rs | 2 +- apps/rocm/src/uninstall_plan.rs | 2 +- crates/rocm-core/src/confirm.rs | 270 ++++++++++++++++-- crates/rocm-core/src/fix/consent.rs | 4 +- docs/architecture.md | 2 +- docs/manual-testing.md | 8 +- docs/testing.md | 7 + tests/e2e-cucumber/features/uninstall.feature | 31 ++ .../e2e-cucumber/tests/e2e/uninstall_steps.rs | 139 +++++++++ 11 files changed, 444 insertions(+), 30 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 17113230a6..d4cea5e832 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -250,7 +250,7 @@ Shared UI components — reuse rather than hand-rolling new ones. The CLI (`apps Shared (`crates/rocm-core`): -- [`confirm.rs`](crates/rocm-core/src/confirm.rs)'s **`print_prompt`**/**`read_answer`** — the shared `[y/N]` confirmation prompt: `print_prompt` writes and flushes a prompt (callers include their own trailing `[y/N]: `), `read_answer` reads a line and parses it as yes/no. Two fallible calls rather than one, so a caller can choose whether a failed write is fatal: [`fix/consent.rs`](crates/rocm-core/src/fix/consent.rs)'s `confirm()` attempts the read anyway, while [`uninstall_plan.rs`](apps/rocm/src/uninstall_plan.rs)'s `confirm_uninstall` and [`consent.rs`](apps/rocm/src/therock/consent.rs)'s `prompt_yes_no` treat it as fatal. [`uninstall.rs`](apps/rocm/src/uninstall.rs), [`storage.rs`](apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](apps/rocm/src/runtimes_cmd.rs) reach the prompt only through `confirm_uninstall`, each behind `require_yes_or_interactive` (below); [`fix/consent.rs`](crates/rocm-core/src/fix/consent.rs)'s `confirm()` additionally layers its own non-interactive short-circuit and refusal wording on top +- [`confirm.rs`](crates/rocm-core/src/confirm.rs)'s **`print_prompt`**/**`read_answer`** — the shared `[y/N]` confirmation prompt: `print_prompt` writes and flushes a prompt (callers include their own trailing `[y/N]: `), `read_answer` reads a line and parses it as yes/no, and writes to the stream it is given a newline when the input ended without one and a line saying so when the answer was neither yes nor no (declined either way). Two fallible calls rather than one, so a caller can choose whether a failed write is fatal: [`fix/consent.rs`](crates/rocm-core/src/fix/consent.rs)'s `confirm()` attempts the read anyway, while [`uninstall_plan.rs`](apps/rocm/src/uninstall_plan.rs)'s `confirm_uninstall` and [`consent.rs`](apps/rocm/src/therock/consent.rs)'s `prompt_yes_no` treat it as fatal. [`uninstall.rs`](apps/rocm/src/uninstall.rs), [`storage.rs`](apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](apps/rocm/src/runtimes_cmd.rs) reach the prompt only through `confirm_uninstall`, each behind `require_yes_or_interactive` (below); [`fix/consent.rs`](crates/rocm-core/src/fix/consent.rs)'s `confirm()` additionally layers its own non-interactive short-circuit and refusal wording on top - [`confirm.rs`](crates/rocm-core/src/confirm.rs)'s **`require_yes_or_interactive(yes, action)`** — the guard in front of a mutating command that confirms with that `[y/N]` prompt. `Ok(true)` means `--yes` already approved it (skip the prompt), `Ok(false)` means there is no `--yes` but a terminal, so the caller still asks, and otherwise it bails with `"{action} requires --yes outside an interactive terminal"`. A new `--yes`-gated command reuses it instead of re-deriving the `yes` / `interactive_terminal()` / `bail!` sequence. Not for a refusal that deliberately differs: [`therock/consent.rs`](apps/rocm/src/therock/consent.rs)'s `sdk_install_approval` names two consents that are not interchangeable, and [`fix/consent.rs`](crates/rocm-core/src/fix/consent.rs)'s `confirm()` has its own exit code and checks stdin only CLI: diff --git a/README.md b/README.md index 469757d321..07fcad7a1e 100644 --- a/README.md +++ b/README.md @@ -1787,8 +1787,11 @@ download's progress: `rocm uninstall` removes four categories of files that ROCm CLI manages on this computer: its binaries, configuration, data, and caches. It lists what it will -do and asks `[y/N]` (default: no) before doing it. Use `--dry-run` to only list, -and `--yes` to skip the confirmation. There is no `--force`; `--force-dev-binaries` +do and asks `[y/N]` (default: no) before doing it. Only `y` or `yes` proceeds. +Enter, `n` and `no` cancel: it prints `uninstall cancelled`, removes nothing and +exits with code 0. Any other answer cancels the same way, after a line saying +that the answer was not understood and is treated as no. Use `--dry-run` to only +list, and `--yes` to skip the confirmation. There is no `--force`; `--force-dev-binaries` is a separate opt-in for development builds. At a terminal, a dry run that finds something to remove ends by saying to run the same command again without `--dry-run`; with nothing to remove, or when stdin or stdout is not a terminal diff --git a/apps/rocm/src/therock/consent.rs b/apps/rocm/src/therock/consent.rs index 60ef0f9c9b..165bddca9b 100644 --- a/apps/rocm/src/therock/consent.rs +++ b/apps/rocm/src/therock/consent.rs @@ -440,7 +440,7 @@ pub(super) fn confirm_overwrite_existing_sdk( fn prompt_yes_no(prompt: &str) -> Result { rocm_core::confirm::print_prompt(&mut std::io::stdout(), prompt) .context("failed to write confirmation prompt")?; - rocm_core::confirm::read_answer(&mut std::io::stdin().lock()) + rocm_core::confirm::read_answer(&mut std::io::stdin().lock(), &mut std::io::stdout()) .context("failed to read confirmation response") } diff --git a/apps/rocm/src/uninstall_plan.rs b/apps/rocm/src/uninstall_plan.rs index e553dc9e47..1ad1f979a4 100644 --- a/apps/rocm/src/uninstall_plan.rs +++ b/apps/rocm/src/uninstall_plan.rs @@ -438,7 +438,7 @@ fn env_path(name: &str) -> Option { pub(crate) fn confirm_uninstall() -> Result { rocm_core::confirm::print_prompt(&mut io::stdout(), "Proceed with uninstall? [y/N]: ") .context("failed to write uninstall prompt")?; - rocm_core::confirm::read_answer(&mut io::stdin().lock()) + rocm_core::confirm::read_answer(&mut io::stdin().lock(), &mut io::stdout()) .context("failed to read uninstall confirmation") } diff --git a/crates/rocm-core/src/confirm.rs b/crates/rocm-core/src/confirm.rs index 20ee2587cc..e5fa9f9a03 100644 --- a/crates/rocm-core/src/confirm.rs +++ b/crates/rocm-core/src/confirm.rs @@ -23,6 +23,13 @@ //! without a controlled terminal. Real callers pass `std::io::stdout()` and //! `std::io::stdin().lock()`. //! +//! [`read_answer`] also tells the user when their answer was not understood, and +//! ends the prompt line when the input ended without a newline. Both go to the +//! `out` stream the caller passes, so every caller gets them and none can forget. +//! `y`/`yes` approve; `n`/`no`/an empty line decline silently (an empty line is +//! the documented default); anything else declines with a line saying so, rather +//! than being mistaken for a deliberate "no". +//! //! [`interactive_terminal`] answers the question callers ask before prompting at //! all: whether this process has a terminal on both streams. //! [`require_yes_or_interactive`] is the guard built on it for a mutating @@ -32,9 +39,24 @@ use anyhow::{Result, bail}; use std::io::{BufRead, IsTerminal, Write, stdin, stdout}; -/// Parse a user's typed response to a `[y/N]` prompt. -pub fn is_affirmative_answer(line: &str) -> bool { - matches!(line.trim().to_lowercase().as_str(), "y" | "yes") +/// How a typed response to a `[y/N]` prompt reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Answer { + /// `y` or `yes`, in any case. + Yes, + /// `n` or `no`, or an empty line: the prompt's default. + No, + /// Anything else. Declined like [`Answer::No`], but worth saying so. + Unrecognized, +} + +/// Classify a user's typed response to a `[y/N]` prompt. +pub fn parse_answer(line: &str) -> Answer { + match line.trim().to_lowercase().as_str() { + "y" | "yes" => Answer::Yes, + "" | "n" | "no" => Answer::No, + _ => Answer::Unrecognized, + } } /// Write `prompt` to `out` and flush it, so it's visible before a blocking @@ -54,10 +76,48 @@ pub fn print_prompt(out: &mut impl Write, prompt: &str) -> std::io::Result<()> { /// Read one line from `input` and parse it as yes/no. Only errors if the /// line can't be read; a declined answer — including an empty read at EOF — /// is `Ok(false)`, not an error. -pub fn read_answer(input: &mut impl BufRead) -> std::io::Result { +/// +/// Two things are written to `out` so the caller's own message lands where the +/// user can read it: +/// - when the line did not end in a newline (EOF, as from Ctrl-D, or piped input +/// with no final newline such as `printf no |`), a newline, ending the prompt +/// line: either way nothing echoed a line end, so the cursor is still on the +/// prompt line and the caller's next line would run into it; +/// - when the answer is [`Answer::Unrecognized`], a line saying it is being +/// treated as no, with the answer shown back quoted and escaped and cut to +/// [`MAX_ECHOED_ANSWER_CHARS`] characters, so a long pasted line does not fill +/// the screen. +/// +/// Both writes are best effort: a failure to explain a decline must not turn +/// into an error that replaces the decline, which `fix`'s `confirm()` relies on. +pub fn read_answer(input: &mut impl BufRead, out: &mut impl Write) -> std::io::Result { let mut line = String::new(); input.read_line(&mut line)?; - Ok(is_affirmative_answer(&line)) + if !line.ends_with('\n') { + let _ = writeln!(out); + } + let answer = parse_answer(&line); + if answer == Answer::Unrecognized { + let _ = writeln!( + out, + "{} is not yes or no; treating it as no.", + echoed_answer(line.trim()) + ); + } + Ok(answer == Answer::Yes) +} + +/// The most characters of a typed answer that the "not understood" line shows. +const MAX_ECHOED_ANSWER_CHARS: usize = 40; + +/// `answer` as shown back to the user: quoted with control characters escaped, and +/// cut to [`MAX_ECHOED_ANSWER_CHARS`] characters (never in the middle of one), with +/// `...` after the closing quote when there was more. +fn echoed_answer(answer: &str) -> String { + let mut chars = answer.chars(); + let shown: String = chars.by_ref().take(MAX_ECHOED_ANSWER_CHARS).collect(); + let more = if chars.next().is_some() { "..." } else { "" }; + format!("{shown:?}{more}") } /// Whether this process can hold an interactive exchange with a user. @@ -112,21 +172,34 @@ mod tests { use std::io::{Error, Result}; #[test] - fn is_affirmative_answer_accepts_only_y_and_yes() { + fn parse_answer_accepts_only_y_and_yes() { for accepted in ["y", "Y", "yes", "YES", "Yes", " y ", " yes\n"] { - assert!( - is_affirmative_answer(accepted), + assert_eq!( + parse_answer(accepted), + Answer::Yes, "expected {accepted:?} to be treated as a yes" ); } } #[test] - fn is_affirmative_answer_rejects_everything_else() { - for declined in ["n", "no", "", "\n", "yep", "ye"] { - assert!( - !is_affirmative_answer(declined), - "expected {declined:?} to be treated as a decline" + fn parse_answer_reads_n_no_and_an_empty_line_as_the_default_no() { + for declined in ["n", "N", "no", "NO", " no ", "", "\n", " \n"] { + assert_eq!( + parse_answer(declined), + Answer::No, + "expected {declined:?} to be the default no" + ); + } + } + + #[test] + fn parse_answer_flags_everything_else_as_unrecognized() { + for odd in ["maybe", "yep", "ye", "yes please", "nope", "1", "y n"] { + assert_eq!( + parse_answer(odd), + Answer::Unrecognized, + "expected {odd:?} to be unrecognized" ); } } @@ -189,19 +262,170 @@ mod tests { ); } + /// Run `read_answer` over `typed` and return the verdict and everything it + /// wrote to the output stream. + fn answer_and_output(typed: &[u8]) -> (bool, String) { + let mut input = typed; + let mut out = Vec::new(); + let verdict = + read_answer(&mut input, &mut out).expect("read from a byte slice cannot fail"); + ( + verdict, + String::from_utf8(out).expect("the helper writes UTF-8"), + ) + } + + #[test] + fn read_answer_approves_y_and_yes_without_writing_anything() { + for typed in [&b"y\n"[..], b"yes\n", b"YES\n"] { + let (verdict, written) = answer_and_output(typed); + assert!(verdict, "{typed:?} must approve"); + assert_eq!(written, "", "an approval needs no comment"); + } + } + + #[test] + fn read_answer_declines_n_no_and_enter_without_writing_anything() { + for typed in [&b"n\n"[..], b"no\n", b"\n"] { + let (verdict, written) = answer_and_output(typed); + assert!(!verdict, "{typed:?} must decline"); + assert_eq!( + written, "", + "a deliberate no (or the default) must not be commented on" + ); + } + } + + #[test] + fn read_answer_declines_an_unrecognized_answer_and_says_so() { + let (verdict, written) = answer_and_output(b"maybe\n"); + assert!(!verdict, "an unrecognized answer must decline"); + assert_eq!(written, "\"maybe\" is not yes or no; treating it as no.\n"); + } + + #[test] + fn read_answer_at_eof_declines_and_ends_the_prompt_line() { + let (verdict, written) = answer_and_output(b""); + assert!(!verdict, "EOF is Ok(false), not an error"); + assert_eq!(written, "\n", "EOF leaves the cursor on the prompt line"); + } + + #[test] + fn read_answer_for_a_partial_line_ends_the_prompt_line_then_says_so() { + // Typed text and then EOF, with no newline: the terminal echoed the text + // but no newline, so the line must be ended before the feedback line. + let (verdict, written) = answer_and_output(b"maybe"); + assert!(!verdict); + assert_eq!( + written, + "\n\"maybe\" is not yes or no; treating it as no.\n" + ); + } + + #[test] + fn read_answer_for_a_yes_without_a_newline_still_approves() { + let (verdict, written) = answer_and_output(b"yes"); + assert!(verdict, "a yes is a yes with or without the final newline"); + assert_eq!(written, "\n", "the unfinished prompt line is still ended"); + } + + #[test] + fn read_answer_cuts_a_long_answer_short_and_marks_the_cut() { + let typed = format!("{}\n", "a".repeat(100)); + let (verdict, written) = answer_and_output(typed.as_bytes()); + assert!(!verdict); + assert_eq!( + written, + format!( + "\"{}\"... is not yes or no; treating it as no.\n", + "a".repeat(MAX_ECHOED_ANSWER_CHARS) + ) + ); + } + + #[test] + fn read_answer_shows_an_answer_of_exactly_the_limit_whole() { + let answer = "b".repeat(MAX_ECHOED_ANSWER_CHARS); + let (_, written) = answer_and_output(format!("{answer}\n").as_bytes()); + assert_eq!( + written, + format!("\"{answer}\" is not yes or no; treating it as no.\n"), + "an answer that fits is not marked as cut" + ); + let (_, written) = answer_and_output(format!("{answer}c\n").as_bytes()); + assert_eq!( + written, + format!("\"{answer}\"... is not yes or no; treating it as no.\n"), + "one character over the limit is cut" + ); + } + + #[test] + fn read_answer_cuts_between_characters_not_inside_one() { + // 2-byte and 4-byte characters: a cut by bytes would split one and panic + // or write invalid UTF-8; the limit counts characters. + for ch in ['é', '🦀'] { + let typed = format!("{}\n", ch.to_string().repeat(MAX_ECHOED_ANSWER_CHARS + 5)); + let (_, written) = answer_and_output(typed.as_bytes()); + assert_eq!( + written, + format!( + "\"{}\"... is not yes or no; treating it as no.\n", + ch.to_string().repeat(MAX_ECHOED_ANSWER_CHARS) + ), + "answer of {ch:?}" + ); + } + } + + #[test] + fn read_answer_bounds_the_line_even_when_every_character_is_escaped() { + // An escaped control character is longer than itself, so the limit is on + // the typed characters and the line stays bounded by it. + let typed = format!("{}\n", "\x1b".repeat(100)); + let (_, written) = answer_and_output(typed.as_bytes()); + assert_eq!( + written, + format!( + "\"{}\"... is not yes or no; treating it as no.\n", + "\\u{1b}".repeat(MAX_ECHOED_ANSWER_CHARS) + ) + ); + assert!( + !written.contains('\x1b'), + "no raw escape reaches the terminal" + ); + } + #[test] - fn read_answer_parses_a_typed_line() { - let mut input: &[u8] = b"yes\n"; - assert!(read_answer(&mut input).expect("read from a byte slice cannot fail")); + fn read_answer_for_a_no_without_a_newline_declines_and_only_ends_the_prompt_line() { + // Piped input such as `printf no |` has no final newline either. The + // verdict is the decline `no\n` gives and nothing explains it, but the + // prompt line is still ended: no newline was echoed after the prompt. + let (verdict, written) = answer_and_output(b"no"); + assert!(!verdict, "a no is a no with or without the final newline"); + assert_eq!(written, "\n", "only the unfinished prompt line is ended"); + } - let mut input: &[u8] = b"n\n"; - assert!(!read_answer(&mut input).expect("read from a byte slice cannot fail")); + #[test] + fn read_answer_quotes_an_answer_that_holds_control_characters() { + // The answer is echoed back, so it is escaped rather than written raw. + let (_, written) = answer_and_output(b"\x1b[31mred\n"); + assert_eq!( + written, + "\"\\u{1b}[31mred\" is not yes or no; treating it as no.\n" + ); } #[test] - fn read_answer_at_eof_declines_without_erroring() { - let mut input: &[u8] = b""; - assert!(!read_answer(&mut input).expect("EOF is Ok(false), not an error")); + fn read_answer_keeps_the_verdict_when_the_feedback_cannot_be_written() { + // The explanation is best effort: a closed stdout must not turn a + // decline into an error, which `fix`'s `confirm()` relies on. + let mut input: &[u8] = b"maybe\n"; + let mut out = FailingWriter; + let verdict = read_answer(&mut input, &mut out) + .expect("a failed feedback write must not replace the verdict"); + assert!(!verdict); } /// A reader whose `read_line` always fails, so a real I/O failure on the @@ -224,7 +448,9 @@ mod tests { #[test] fn read_answer_surfaces_a_read_failure_as_an_error() { let mut input = FailingReader; - read_answer(&mut input).expect_err("a failing reader must return Err"); + let mut out = Vec::new(); + read_answer(&mut input, &mut out).expect_err("a failing reader must return Err"); + assert!(out.is_empty(), "nothing is written for a failed read"); } #[test] diff --git a/crates/rocm-core/src/fix/consent.rs b/crates/rocm-core/src/fix/consent.rs index ee3b0afbc3..95b9d8931f 100644 --- a/crates/rocm-core/src/fix/consent.rs +++ b/crates/rocm-core/src/fix/consent.rs @@ -75,7 +75,9 @@ pub(super) fn confirm(prompt: &str, assume_yes: bool) -> bool { return verdict; } let _ = crate::confirm::print_prompt(&mut std::io::stdout(), &format!("{prompt} [y/N]: ")); - let confirmed = crate::confirm::read_answer(&mut std::io::stdin().lock()).unwrap_or(false); + let confirmed = + crate::confirm::read_answer(&mut std::io::stdin().lock(), &mut std::io::stdout()) + .unwrap_or(false); if !confirmed { fail!("Not confirmed; refusing to apply."); } diff --git a/docs/architecture.md b/docs/architecture.md index 2c9377e771..3b8e4ae96d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -183,7 +183,7 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`codex_bridge.rs`](../crates/rocm-core/src/codex_bridge.rs) — (ROCMAI-662) the `CodexBridgeSnapshot`/`CodexBridgeGpuSnapshot`/`CodexBridgeEngine` host-snapshot DTOs. Plain serde data: `rocm` and `rocmd` each build their own snapshot of this shape, so the types live here rather than in either binary. - [`command_capture.rs`](../crates/rocm-core/src/command_capture.rs) — (ROCMAI-720) running the `rocm` binary as a captured child: `CommandCapture`, `run_rocm_capture_with_env` and the `command_capture_text` formatter, shared by `rocmd`'s sandbox/MCP clusters and `rocm`'s chat tool bridge. Each caller passes the environment the child should get rather than this module choosing it; the bounded-execution half is [`process.rs`](../crates/rocm-core/src/process.rs)'s `run_with_timeout`. - [`config.rs`](../crates/rocm-core/src/config.rs) — (ROCMAI-151) the unified `RocmCliConfig` and its nested sections, including the persisted `WatcherMode` enum, the built-in watcher catalog (`BuiltinWatcherSpec`, `builtin_watchers`, `builtin_watcher`) and the rocm-dash dashboard sub-config — see the module's own doc comment for what it owns. -- [`confirm.rs`](../crates/rocm-core/src/confirm.rs) — (issue #769) the shared `[y/N]` prompt primitives: `is_affirmative_answer`, plus `print_prompt` and `read_answer` (each generic over the `Write`/`BufRead` stream so the sequence is unit-testable without a controlled terminal). Deduplicates the print-prompt/read-line/parse sequence that [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()`, [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs)'s `confirm_uninstall` and [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs)'s `prompt_yes_no` each used to hand-roll. [`uninstall.rs`](../apps/rocm/src/uninstall.rs), [`storage.rs`](../apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) reach the prompt only through `confirm_uninstall`. `print_prompt` and `read_answer` stay separate rather than one bundled call because [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()` attempts the read even when the prompt couldn't be written, while the other callers treat a failed write as fatal; [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()` also keeps its own `--yes`/non-interactive short-circuit and refusal wording on top of these, since those are specific to the `fix` recipe flow. The module also holds `interactive_terminal`, whether stdin and stdout are both terminals, which callers check before they prompt, and `require_yes_or_interactive`, the `--yes`/terminal guard that [`uninstall.rs`](../apps/rocm/src/uninstall.rs), [`storage.rs`](../apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) put in front of that prompt; `interactive_terminal` is re-exported from `lib.rs` as `rocm_core::interactive_terminal` and from `host_gpu.rs` as `rocm_core::host_gpu::interactive_terminal`, the path it had before it moved here. +- [`confirm.rs`](../crates/rocm-core/src/confirm.rs) — (issue #769) the shared `[y/N]` prompt primitives: `parse_answer`, which sorts a typed line into `Answer::Yes`, `Answer::No` (`n`, `no` or an empty line) or `Answer::Unrecognized`, plus `print_prompt` and `read_answer` (each generic over the `Write`/`BufRead` stream so the sequence is unit-testable without a controlled terminal). `read_answer` takes the output stream as well: it ends the prompt line when the input ended without a newline (Ctrl-D, or piped input with no final newline) and says so when an answer is unrecognized and treated as no, showing at most the first 40 characters of the answer, best effort, so a failed write never replaces the verdict. Deduplicates the print-prompt/read-line/parse sequence that [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()`, [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs)'s `confirm_uninstall` and [`therock/consent.rs`](../apps/rocm/src/therock/consent.rs)'s `prompt_yes_no` each used to hand-roll. [`uninstall.rs`](../apps/rocm/src/uninstall.rs), [`storage.rs`](../apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) reach the prompt only through `confirm_uninstall`. `print_prompt` and `read_answer` stay separate rather than one bundled call because [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()` attempts the read even when the prompt couldn't be written, while the other callers treat a failed write as fatal; [`fix/consent.rs`](../crates/rocm-core/src/fix/consent.rs)'s `confirm()` also keeps its own `--yes`/non-interactive short-circuit and refusal wording on top of these, since those are specific to the `fix` recipe flow. The module also holds `interactive_terminal`, whether stdin and stdout are both terminals, which callers check before they prompt, and `require_yes_or_interactive`, the `--yes`/terminal guard that [`uninstall.rs`](../apps/rocm/src/uninstall.rs), [`storage.rs`](../apps/rocm/src/storage.rs) and [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs) put in front of that prompt; `interactive_terminal` is re-exported from `lib.rs` as `rocm_core::interactive_terminal` and from `host_gpu.rs` as `rocm_core::host_gpu::interactive_terminal`, the path it had before it moved here. - [`crypto.rs`](../crates/rocm-core/src/crypto.rs) — (ROCMAI-152, Phase 7c) RSASSA-PKCS#1 v1.5/SHA-256 sign/verify, generic — not model-catalog-specific. - [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs) — `rocm diagnose`: matches an `Examination` and optional symptom text against the closed failure catalog and returns ranked findings. The module root keeps the public types (`Fix`, `Diagnosis`, `Route`, `DiagnoseReport`), the two score thresholds, `diagnose` itself, the two `vllm_*_symptom_is_diagnosable` classifiers the vLLM engine calls (they run the whole catalog, so they cannot live below it) and the re-exports that keep every `rocm_core::diagnose::` path resolving; everything else lives in child modules under `diagnose/`, with `diagnose.rs` kept as the module root so those paths and the links to it stay valid. The platform children do not import each other, and `catalog.rs` is the only module that names every checker. [`catalog.rs`](../crates/rocm-core/src/diagnose/catalog.rs) (the `CHECKERS` table that tags each check with the platform families it applies to, in catalog order, plus the platform-family resolution, applicability and the out-of-scope message), [`scoring.rs`](../crates/rocm-core/src/diagnose/scoring.rs) (every keyword table, the scoring helpers over them, and the small helpers that read an `Examination` or build a `Diagnosis`), [`report.rs`](../crates/rocm-core/src/diagnose/report.rs) (where an unmatched report is routed upstream, `upstream_route`, `route_when_no_match` and the tracker table behind them, and `render_report_text`, the human-facing view), [`linux.rs`](../crates/rocm-core/src/diagnose/linux.rs) (the checks that apply on Linux only: kernel and module, device-node access, package repositories and the repo-native install, plus the package-manager helpers and `dkms_driver_plan_is_supported`, which the driver planner calls), [`windows.rs`](../crates/rocm-core/src/diagnose/windows.rs) (the checks that apply on Windows only: the HIP SDK, the Adrenalin driver, the Visual C++ runtime and the display driver date, with the version comparison and scores only they use), [`wsl.rs`](../crates/rocm-core/src/diagnose/wsl.rs) (the checks that apply on WSL2 only: the DXCore handoff, the ROCDXG userspace, the distro floor, the Windows host driver and WSL 1), [`host.rs`](../crates/rocm-core/src/diagnose/host.rs) (the checks on the host's configuration that run on more than one platform: wheel architecture, the HSA override, `PATH`, the wheel's ROCm against the system's, an integrated GPU beside a discrete one, and GPUs ROCm does not support), [`engine.rs`](../crates/rocm-core/src/diagnose/engine.rs) (the checks on the engine runtime that run on Linux and WSL2: the vLLM out-of-memory and import failures, a second copy of the code object manager, and a `/dev/shm` too small for a serving workload, plus the two canonical symptom strings the vLLM engine falls back to) and [`test_support.rs`](../crates/rocm-core/src/diagnose/test_support.rs) (test-only: the `Examination` builders that the tests of more than one `diagnose` module share). - [`download.rs`](../crates/rocm-core/src/download.rs) — streaming file downloads with resume and retry: `DownloadRequest`/`DownloadOutcome`, `download_file_streaming(_with_progress)` and the simpler `download_file_to_path(_with_progress)`, `Backoff`, and the retry policy (`DOWNLOAD_MAX_ATTEMPTS`). A download streams to a per-process `.part-` sibling, resumes with a `Range` request, and enforces the caller's size limit, expected length and SHA-256 digest before publishing the file. It uses `ureq`, not the loopback client in [`net.rs`](../crates/rocm-core/src/net.rs), and depends only on [`disk_space.rs`](../crates/rocm-core/src/disk_space.rs). The module is private and [`lib.rs`](../crates/rocm-core/src/lib.rs) re-exports its public names. diff --git a/docs/manual-testing.md b/docs/manual-testing.md index f61b855ce8..e9dac3f5a3 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1210,7 +1210,13 @@ Expected result: HOME=$(mktemp -d) rocm uninstall --keep-binaries < /dev/null # expect the refusal; exit 1 ``` - At a terminal, the same command asks `[y/N]` and treats Enter as no. + At a terminal, the same command asks `[y/N]` and treats Enter as no. Type + `maybe` and it cancels after printing `"maybe" is not yes or no; treating it + as no.`; an answer longer than 40 characters is shown cut to its first 40, + with `...` after the closing quote. Press Ctrl-D and `uninstall cancelled` + appears on a line of its own, not after the prompt. Nothing is removed in + either case. The same prompt is shared by `rocm fix` and the SDK-overwrite + prompt, so the same answers behave the same way there. ## 11. AIM (Docker) Engine Verification diff --git a/docs/testing.md b/docs/testing.md index f04f56c40d..705a024f16 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1563,6 +1563,13 @@ is not answered with `--force-dev-binaries` (`uninstall-10`); the help starts its examples with a dry run (`uninstall-11`). The assistant's uninstall preview ends differently, since it has no flag to offer: it says removal needs approval, which `lifecycle_read_mutate_split_is_honest` in [tool_policy.rs](../apps/rocm/src/tool_policy.rs) backs. +`uninstall-13` to `uninstall-15` drive the confirmation prompt under a +pseudo-terminal, which is the only way to reach it, and pin the shared `[y/N]` +helper's behaviour for an answer that is neither yes nor no, for ending the input +with Ctrl-D, and for a plain no, each together with the config folder being left +in place. They run on the default mock lane; the helper's other callers (`rocm fix`, +the SDK-overwrite prompt) share it and are covered by its unit tests in +[confirm.rs](../crates/rocm-core/src/confirm.rs). The `uninstall` command does not start the file logger, which would otherwise create the data folder's `logs` folder before the plan is built: diff --git a/tests/e2e-cucumber/features/uninstall.feature b/tests/e2e-cucumber/features/uninstall.feature index 63e2568f33..147f0455e3 100644 --- a/tests/e2e-cucumber/features/uninstall.feature +++ b/tests/e2e-cucumber/features/uninstall.feature @@ -156,3 +156,34 @@ Feature: Uninstall And the preview lists the config folder And the preview says removal needs approval And the config folder still holds the user's files + + # The `[y/N]` prompt is shared with `rocm fix` and the SDK-overwrite prompt; + # `rocm uninstall` is the one that is cheap to isolate, so the shared helper's + # behaviour is pinned here. A prompt is only reached on a real terminal (without + # one the command is refused, see uninstall-04), so these run under a + # pseudo-terminal. An answer that is neither yes nor no used to be declined with + # no sign it was not understood, and ending the input (Ctrl-D) left the + # cancellation on the prompt line because no newline had been echoed. + @id:uninstall-unrecognized-answer-is-declined-and-said-so @requires-os:linux + Scenario: uninstall-13 - An answer that is neither yes nor no is declined, and the user is told so + Given the config folder holds the user's files + When the user is asked to confirm an uninstall on a real terminal and types an answer that is neither yes nor no + Then the user is told the answer was not understood and is treated as no + And the uninstall is reported as cancelled + And the config folder still holds the user's files + + @id:uninstall-end-of-input-cancels-on-a-fresh-line @requires-os:linux + Scenario: uninstall-14 - Ending the input at the confirmation prompt cancels on a fresh line + Given the config folder holds the user's files + When the user is asked to confirm an uninstall on a real terminal and ends the input instead of answering + Then the cancellation is reported on its own line, not on the prompt line + And the config folder still holds the user's files + + # The other side of uninstall-13: a deliberate no is not an error to explain. + @id:uninstall-plain-no-is-declined-without-complaint @requires-os:linux + Scenario: uninstall-15 - Answering no is declined without any complaint + Given the config folder holds the user's files + When the user is asked to confirm an uninstall on a real terminal and types no + Then the uninstall is reported as cancelled + And the user is not told the answer was misunderstood + And the config folder still holds the user's files diff --git a/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs b/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs index 0dbf34f51c..49a16b9436 100644 --- a/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/uninstall_steps.rs @@ -172,6 +172,145 @@ async fn uninstall_config_unconfirmed(world: &mut E2eWorld) { ); } +/// Run `rocm uninstall` of the config folder only, without `--yes`, under a real +/// terminal, wait for the confirmation prompt and send `keys` as the answer. +/// +/// The piped harness can never reach the prompt (no terminal, so the command is +/// refused first); a pseudo-terminal is the only way to answer it. The screen +/// the user would be looking at is kept as the scenario's output. +async fn answer_uninstall_prompt(world: &mut E2eWorld, keys: &str) { + let root = root(world).to_path_buf(); + let home = root.join("home"); + std::fs::create_dir_all(&home).expect("failed to create isolated HOME"); + let home = home.display().to_string(); + let uv = root.join("uv-cache").display().to_string(); + let hf = root.join("hf-home").display().to_string(); + let config = config_dir(world).display().to_string(); + let mut session = TuiSession::spawn_with_env( + world, + &[ + "uninstall", + "--keep-binaries", + "--keep-data", + "--keep-cache", + ], + &[ + ("HOME", home.as_str()), + ("UV_CACHE_DIR", uv.as_str()), + ("HF_HOME", hf.as_str()), + ("ROCM_CLI_CONFIG_DIR", config.as_str()), + ], + ) + .unwrap_or_else(|e| panic!("failed to open the uninstall prompt: {e}")); + session + .wait_for_screen("Proceed with uninstall? [y/N]:", default_timeout()) + .await + .unwrap_or_else(|e| panic!("the confirmation prompt never appeared: {e}")); + session + .send(keys) + .unwrap_or_else(|e| panic!("failed to answer the prompt: {e}")); + let rc = session + .wait_for_exit_code(default_timeout()) + .await + .unwrap_or_else(|e| panic!("the CLI never exited after the answer: {e}")); + world.cli_output = Some(session.screen_text()); + world.cli_rc = Some(rc); + world.tui = Some(session); +} + +#[when( + "the user is asked to confirm an uninstall on a real terminal and types an answer that is neither yes nor no" +)] +async fn uninstall_prompt_unrecognized(world: &mut E2eWorld) { + answer_uninstall_prompt(world, "maybe\r").await; +} + +#[when( + "the user is asked to confirm an uninstall on a real terminal and ends the input instead of answering" +)] +async fn uninstall_prompt_end_of_input(world: &mut E2eWorld) { + // Ctrl-D on an empty line: the terminal reports end of input and echoes + // nothing, which is what left the cursor on the prompt line. + answer_uninstall_prompt(world, "\x04").await; +} + +#[when("the user is asked to confirm an uninstall on a real terminal and types no")] +async fn uninstall_prompt_no(world: &mut E2eWorld) { + answer_uninstall_prompt(world, "no\r").await; +} + +/// The screen's lines with trailing padding removed. +fn screen_lines(world: &E2eWorld) -> Vec { + output(world) + .lines() + .map(|l| l.trim_end().to_owned()) + .collect() +} + +#[then("the user is told the answer was not understood and is treated as no")] +async fn told_answer_not_understood(world: &mut E2eWorld) { + let lines = screen_lines(world); + let expected = "\"maybe\" is not yes or no; treating it as no."; + assert!( + lines.iter().any(|l| l == expected), + "expected a `{expected}` line on the terminal:\n{}", + output(world) + ); +} + +#[then("the user is not told the answer was misunderstood")] +async fn not_told_answer_misunderstood(world: &mut E2eWorld) { + assert!( + !output(world).contains("is not yes or no"), + "a plain no must not be commented on:\n{}", + output(world) + ); +} + +#[then("the uninstall is reported as cancelled")] +async fn uninstall_reported_cancelled(world: &mut E2eWorld) { + let rc = world.cli_rc.expect("no exit code captured"); + assert_eq!( + rc, + 0, + "declining is not a failure, got exit {rc}:\n{}", + output(world) + ); + assert!( + screen_lines(world) + .iter() + .any(|l| l == "uninstall cancelled"), + "expected an `uninstall cancelled` line on its own:\n{}", + output(world) + ); +} + +#[then("the cancellation is reported on its own line, not on the prompt line")] +async fn cancellation_on_its_own_line(world: &mut E2eWorld) { + let lines = screen_lines(world); + let prompt = lines + .iter() + .find(|l| l.starts_with("Proceed with uninstall? [y/N]:")) + .unwrap_or_else(|| panic!("no prompt line on the terminal:\n{}", output(world))); + assert_eq!( + prompt, + "Proceed with uninstall? [y/N]:", + "nothing may follow the prompt after the input ended:\n{}", + output(world) + ); + assert!( + lines.iter().any(|l| l == "uninstall cancelled"), + "expected an `uninstall cancelled` line on its own:\n{}", + output(world) + ); + assert_eq!( + world.cli_rc, + Some(0), + "ending the input declines, it does not fail:\n{}", + output(world) + ); +} + /// The scenario's own home folder, standing in for the user's. fn home_dir(world: &E2eWorld) -> PathBuf { root(world).join("home") From f095a14cb1b6cceef949135db92998e751177157 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 17:56:50 +0000 Subject: [PATCH 031/102] refactor(rocm): split service_launch.rs into a directory of child modules to get under the module-size limit (#966) * refactor(rocm): split service_launch.rs into a directory of child modules service_launch.rs held 2,172 production lines, over the 1,500-line limit that cargo xtask check-module-size enforces, plus a test module of about 2,750 lines. Move the code into service_launch/ along its seams: endpoint_auth (bind-host validation and the endpoint API key policy), launch_transaction (the locked port allocation decision, failed-launch cleanup and the reuse refusals), spawn (the detached engine child, its process and stdio plumbing, ManagedLaunchReport), readiness (waiting on a service and the OOM fault-injection seam), start (the background launch path and its plain output), helper_autostart (starting the background automation helper) and attach (the attached run and its log streaming). service_launch.rs now only declares the children and re-exports, with pub(crate) use, the items other modules call, so no caller changes. Items that siblings share are pub(super); everything else keeps the visibility it had. Each test moves with the code it covers, and the fixtures two or more children share go to a test-only test_support module. This is a pure move: the code is unchanged apart from visibility keywords, use and mod lines, the cfg gates those imports need so that each platform and feature combination still compiles without unused imports, four intra-doc link targets, and the module path in the three places the cleanup tests re-run themselves by test name. docs/architecture.md describes the new layout, and comments and docs that pointed at moved functions now name the file they live in. Signed-off-by: Jussi Elo * fix(rocm): gate the imports the service_launch split left unused on Windows Splitting service_launch.rs gave readiness.rs its own import list, and three of its imports are used only under a platform or feature gate, so they were unused on a Windows build and failed under -D warnings. The single file had hidden this because other code in it used the same names. In readiness.rs, gate std::fs on any(target_os = "linux", feature = "e2e-oom-fault-injection", test): the zombie probe and the OOM simulation use it in production, and the tests use it on every platform. Gate the two imports of the unreaped-child test, rocm_core::process_is_running and std::thread, on target_os = "linux", like the test itself. Every other import in the endpoint_auth, launch_transaction, spawn, start, helper_autostart and attach children and in the test_support module was checked against the same combinations (Windows and non-Windows, Linux and non-Linux, test on and off, e2e-oom-fault-injection on and off) and needs no change. Only imports and cfg gates change. Signed-off-by: Jussi Elo * docs(rocm): wrap the comments the service_launch split rewrote at 100 columns The pointers the split rewrote to name the new files, and the intra-doc links it added in spawn.rs and start.rs, ran past 100 columns. Rewrap them without changing a word. Comment lines the split only moved are left as they were, so that a move stays a move. Signed-off-by: Jussi Elo * test(rocm): derive the sleeper helper's test name instead of spelling it three times Three cleanup tests re-run the test binary with --exact and the module path of an ignored helper test, spelled out as a string each time. Renaming or moving the module would leave all three naming a test that no longer exists, and --exact selects nothing without saying so. Move the helper to the test-only test_support module next to the fixtures the tests already share, and give it sleeper_test_name, which reads the name off the helper function itself with type_name_of_val and drops the crate segment libtest leaves out. The function also asserts that the result matches the module path plus the function name and that it starts with service_launch::, so a helper that moves or is renamed without that function being updated fails loudly in the tests that use it. Signed-off-by: Jussi Elo * refactor(rocm): drop the cfg-gated std::fs import from readiness.rs The import was gated on a hand-derived union of its use sites (the Linux zombie probe, the OOM simulation and the tests), which drifts as soon as a use site changes. Name std::fs by its full path at the three production uses, give the tests module its own unconditional use std::fs, and do the same for the two Linux-only imports of the unreaped-child test (rocm_core::process_is_running and std::thread), which become full paths at their single use. No gate is left to keep in step, and no import is unused in any platform, test or feature combination. Signed-off-by: Jussi Elo * docs(rocm): fix the cross-file wording in the service_launch comments A few doc comments still said "above", "below" or "this file" about code that now lives in a sibling child module. Point them at the right place instead: a qualified link to spawn_managed_engine_child from the no-usable-GPU refusal, the start.rs location of print_managed_launch_plain from the attach.rs comment, "each test that uses it" for the reusable_record fixture, the service_launch module instead of this module for the stop-path import note, and "what was then a single file" in the parent module doc. Also say at the top of readiness.rs why it has no use for std::fs, std::thread or rocm_core::process_is_running: their use sites sit under different cfg combinations, so the full paths stay and no gate has to be kept in step with them. Signed-off-by: Jussi Elo * docs: correct the service_launch entries for the types they own The launch_transaction.rs entry said it owns private types, but ManagedPortDecision is pub(super) and ManagedReuseRefusal is pub(crate). Say it owns types, give ManagedReuseRefusal's visibility and the reason for it (the pub(crate) function that returns it), and list it in the general paragraph on the visibility of the types in the service_launch children. Signed-off-by: Jussi Elo * docs: point the model_serving comments at the service_launch child files Four comments in model_serving.feature named service_launch.rs for functions that now live in spawn.rs, endpoint_auth.rs, readiness.rs and launch_transaction.rs. Name the file each one lives in. Signed-off-by: Jussi Elo * docs: cite the moved service_launch files as relative links The citation of the launch_transaction.rs tests in docs/testing.md, and the mentions of service_launch.rs, start.rs and main.rs in the service_launch entries of docs/architecture.md that the split added or rewrote, were code-formatted paths that lychee and the docs-links job cannot check. Make them relative Markdown links, as AGENTS.md asks. Citations the split did not add or rewrite are untouched. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit b2d86a1406fa7e1d75ef841b1c258c4bb593d282) Signed-off-by: juhovainio --- apps/rocm/Cargo.toml | 2 +- apps/rocm/src/endpoint_keys.rs | 2 +- apps/rocm/src/serve_cmd.rs | 3 +- apps/rocm/src/serve_port.rs | 6 +- apps/rocm/src/service_launch.rs | 4947 +---------------- apps/rocm/src/service_launch/attach.rs | 768 +++ apps/rocm/src/service_launch/endpoint_auth.rs | 377 ++ .../src/service_launch/helper_autostart.rs | 477 ++ .../src/service_launch/launch_transaction.rs | 1002 ++++ apps/rocm/src/service_launch/readiness.rs | 634 +++ apps/rocm/src/service_launch/spawn.rs | 1306 +++++ apps/rocm/src/service_launch/start.rs | 487 ++ apps/rocm/src/service_launch/test_support.rs | 103 + apps/rocm/src/service_records.rs | 8 +- apps/rocm/src/storage.rs | 2 +- apps/rocm/src/test_support.rs | 2 +- apps/rocmd/src/supervise.rs | 2 +- apps/rocmd/src/watchers/server_recover.rs | 4 +- docs/architecture.md | 13 +- docs/testing.md | 2 +- .../features/model_serving.feature | 14 +- .../tests/e2e/automations_steps.rs | 2 +- tests/e2e-cucumber/tests/e2e/serving_steps.rs | 2 +- 23 files changed, 5236 insertions(+), 4929 deletions(-) create mode 100644 apps/rocm/src/service_launch/attach.rs create mode 100644 apps/rocm/src/service_launch/endpoint_auth.rs create mode 100644 apps/rocm/src/service_launch/helper_autostart.rs create mode 100644 apps/rocm/src/service_launch/launch_transaction.rs create mode 100644 apps/rocm/src/service_launch/readiness.rs create mode 100644 apps/rocm/src/service_launch/spawn.rs create mode 100644 apps/rocm/src/service_launch/start.rs create mode 100644 apps/rocm/src/service_launch/test_support.rs diff --git a/apps/rocm/Cargo.toml b/apps/rocm/Cargo.toml index 7d53b8c7d4..b9635c3c9f 100644 --- a/apps/rocm/Cargo.toml +++ b/apps/rocm/Cargo.toml @@ -22,7 +22,7 @@ e2e-test-hooks = [ # hook that simulates a managed vLLM launch which OOM'd and never became ready, # so the positive OOM-guidance summary can be verified on a GPU-less CI host. # See `e2e_simulate_oom_launch` / `simulate_oom_managed_launch` in -# `service_launch.rs`. +# `service_launch/readiness.rs`. e2e-oom-fault-injection = [] [dependencies] diff --git a/apps/rocm/src/endpoint_keys.rs b/apps/rocm/src/endpoint_keys.rs index a40ee68ab6..e85fc3d216 100644 --- a/apps/rocm/src/endpoint_keys.rs +++ b/apps/rocm/src/endpoint_keys.rs @@ -5,7 +5,7 @@ //! Secure storage for per-service inference-endpoint API keys. //! //! When `rocm serve` binds a public (non-loopback) interface it protects the -//! endpoint with an API key (see `resolve_endpoint_auth` in `service_launch.rs`). +//! endpoint with an API key (see `resolve_endpoint_auth` in `service_launch/endpoint_auth.rs`). //! The key is stored in a 0600 file under the services directory, keyed by //! service id. //! diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 95c22d6905..711e169b6c 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -1318,7 +1318,8 @@ pub(crate) fn serve(args: ServeArgs) -> Result<()> { notes, }; // Same intentional one-time key display as `print_managed_launch_plain` - // in service_launch.rs; see its rationale for why the tag below is currently inert. + // in service_launch/start.rs; see its rationale for why the tag below is + // currently inert. // codeql[rust/cleartext-logging] print!("{}", serve_summary::render_summary(&summary)); } else { diff --git a/apps/rocm/src/serve_port.rs b/apps/rocm/src/serve_port.rs index 2f501506fc..0b4a509d70 100644 --- a/apps/rocm/src/serve_port.rs +++ b/apps/rocm/src/serve_port.rs @@ -233,9 +233,9 @@ pub(crate) fn resolve_serve_port( if let Some(owner) = reserved.iter().find(|entry| entry.port == port) { // The "stop it" advice below is proven, not just worded: see // `stopping_the_reserving_record_frees_its_explicit_port_for_reuse` - // in `service_launch.rs`, which recreates the exact on-disk status - // transition `rocm services stop` performs and confirms the same - // explicit port then resolves. A full CLI-level E2E for this needs + // in `service_launch/launch_transaction.rs`, which recreates the exact + // on-disk status transition `rocm services stop` performs and confirms + // the same explicit port then resolves. A full CLI-level E2E for this needs // a real GPU, since reaching this branch requires a completed // model/engine resolution first. bail!("{}", reserved_message(DEFAULT_LOCAL_HOST, port, owner)); diff --git a/apps/rocm/src/service_launch.rs b/apps/rocm/src/service_launch.rs index 53f9b5c41d..c9b553b531 100644 --- a/apps/rocm/src/service_launch.rs +++ b/apps/rocm/src/service_launch.rs @@ -10,7 +10,7 @@ //! is the managed-service-spawning tail that was deliberately deferred out of //! the Phase 6a (ROCMAI-82) split: `start_managed_service`, //! `run_attached_service`, `spawn_managed_engine_child`, `stream_attached_logs`, -//! and the rest of this file. It was entangled with crate-root launch helpers +//! and the rest of what was then a single file. It was entangled with crate-root launch helpers //! — `record_cli_audit_event`, `existing_live_managed_service_in`, and the stop //! action of the service-record-management cluster (this module reaches only //! stop, via `AttachOutcome::Stop`; restart/remove/prune are not called here) @@ -28,4907 +28,52 @@ //! here uses) to `managed_service_status.rs`. Several functions here are `pub(crate)` //! because other modules call into this cluster across the module boundary; //! `main.rs` only declares the module. +//! +//! # Layout +//! +//! This file only declares the submodules and re-exports the items that other +//! modules call, so callers keep reaching them as `crate::service_launch::name`. +//! Items a submodule shares with its siblings are `pub(super)` and reached through +//! `super::`; nothing is wider than that unless a module outside this +//! one calls it. +//! +//! - `endpoint_auth`: bind-host validation and the endpoint API key checks. +//! - `launch_transaction`: the locked port allocation decision, the cleanup of a +//! failed launch, and the reuse refusals. +//! - `spawn`: the detached engine child, its process and stdio plumbing, and +//! `ManagedLaunchReport`. +//! - `readiness`: waiting on a service, its liveness, and the OOM fault-injection +//! seam. +//! - `start`: the background launch path and its plain output. +//! - `helper_autostart`: starting the background automation helper. +//! - `attach`: the attached run and its log streaming. + +mod attach; +mod endpoint_auth; +mod helper_autostart; +mod launch_transaction; +mod readiness; +mod spawn; +mod start; +#[cfg(test)] +mod test_support; -use std::fmt::Write as _; -use std::fs; -use std::io::{self, BufRead, Write}; -use std::path::{Path, PathBuf}; -#[cfg(not(windows))] -use std::process::{Command as ProcessCommand, Stdio}; -use std::thread; -use std::time::Duration; - -use anyhow::{Context, Result, anyhow, bail}; -use rocm_core::{AppPaths, EndpointReadiness, ManagedServiceRecord}; -use rocm_core::{format_http_base_url, process_is_running}; -use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse}; - -use crate::background_helper::background_helper_already_running; -use crate::endpoint_keys; -use crate::engine_ipc::builtin_engine_serve_http_args; -#[cfg(not(windows))] -use crate::engine_runtime_select::apply_app_path_env; -#[cfg(windows)] -use crate::engine_runtime_select::{app_path_env_var_refs, app_path_env_var_values}; -use crate::engine_runtime_select::{ - managed_service_launcher_path, wait_for_service_http_ready_with_progress, +pub(crate) use attach::run_attached_service; +pub(crate) use endpoint_auth::{ + drop_orphaned_endpoint_key_on_already_running, ensure_public_bind_engine_supported, + ensure_public_service_has_endpoint_key, is_loopback_host, resolve_endpoint_auth, + validate_bind_host, }; -use crate::engines_cmd::env_root_for_service; -use crate::managed_service_status::{ - existing_live_managed_service_in, load_managed_services, managed_service_is_live, +pub(crate) use helper_autostart::{ + ensure_background_helper_running, ensure_background_helper_running_quiet, }; -use crate::serve_port; -use crate::service_hints; -use crate::services_text::read_optional_tail_lines; -use crate::services_text::{ - render_service_action_result, render_service_action_result_with_helper_note, - service_stop_unconfirmed, -}; -use crate::{device_policy_name, record_cli_audit_event, status_for_readiness}; - -pub(crate) fn validate_bind_host(host: &str, allow_public_bind: bool) -> Result<()> { - if !is_loopback_host(host) && !allow_public_bind { - bail!( - "`rocm serve --host {host}` is not loopback; pass `--allow-public-bind` before binding a non-local interface" - ); - } - Ok(()) -} - -/// Inverse of [`rocm_engine_protocol::is_public_bind_host`], which owns the -/// policy so `rocmd` classifies a recorded `host` identically when it respawns -/// the service. -pub(crate) fn is_loopback_host(host: &str) -> bool { - !rocm_engine_protocol::is_public_bind_host(host) -} - -/// Resolve the API key that will guard this endpoint, applying the -/// loopback-vs-public policy for `rocm serve`. -/// -/// - **Loopback host** → `None`: local serving stays credential-free (the -/// unchanged default). Any key supplied for a loopback bind is ignored and -/// nothing is persisted — loopback needs no auth. -/// - **Public host** → `Some(key)`: use the user-supplied key when present, -/// otherwise generate a strong random one so a public endpoint can never come -/// up anonymous. An empty/whitespace supplied key is rejected rather than -/// silently treated as "no auth". -/// - **`required`** → treat a loopback bind as public for this purpose. -/// -/// That last case exists because "loopback" is a statement about the bind -/// address, not about who can reach the port. Publishing the port onto a -/// tailnet, proxying it, or mapping it out of a container all leave the bind -/// loopback while widening the audience — and the policy above would then hand -/// out an unauthenticated endpoint. Whoever widens the reach is responsible for -/// asking for the credential, so this is an explicit flag rather than a guess. -pub(crate) fn resolve_endpoint_auth( - host: &str, - supplied: Option<&str>, - required: bool, -) -> Result> { - if is_loopback_host(host) && !required { - return Ok(None); - } - match supplied { - Some(raw) => { - let trimmed = raw.trim(); - if trimmed.is_empty() { - bail!( - "`rocm serve --api-key` (or ROCM_SERVE_API_KEY) was empty; a public \ - endpoint must be protected by a non-empty API key" - ); - } - // The key is later interpolated verbatim into raw `Authorization: - // Bearer {key}\r\n` header lines; reject a control character (e.g. an - // embedded CR/LF) here so a crafted key cannot inject extra headers. - if rocm_core::endpoint_api_key_has_forbidden_chars(trimmed) { - bail!( - "`rocm serve --api-key` (or ROCM_SERVE_API_KEY) contained a control \ - character such as a carriage return or newline; an endpoint API key \ - must be a single line of printable characters" - ); - } - Ok(Some(trimmed.to_owned())) - } - None => Ok(Some(rocm_core::generate_endpoint_api_key())), - } -} - -/// When an equivalent managed service is already running, or the launch fails -/// before any record exists, the endpoint key serve() freshly stored for this -/// attempt is unused — drop it so it is not orphaned in storage. The already-running service keeps its own key. -/// Best-effort and idempotent; a loopback attempt (`freshly_stored == None`) -/// is a no-op. -pub(crate) fn drop_orphaned_endpoint_key_on_already_running( - paths: &AppPaths, - service_id: &str, - freshly_stored: Option<&str>, -) { - if freshly_stored.is_some() { - endpoint_keys::clear_endpoint_api_key(paths, service_id); - } -} - -/// Reject engine/platform combinations that cannot enforce a public endpoint's -/// API key, so a public bind fails closed instead of coming up unauthenticated. -/// -/// The one such case today: Windows managed Lemonade. Its server reads the -/// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn -/// primitive only carries path-valued env overrides, so the key never reaches it. -/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds -/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter -/// so both branches are unit-testable off-Windows. -pub(crate) fn ensure_public_bind_engine_supported( - engine: &str, - public_bind: bool, - is_windows: bool, -) -> Result<()> { - if public_bind && is_windows && engine == "lemonade" { - bail!( - "public binding with the lemonade engine is not supported on Windows: the endpoint \ - API key cannot be enforced there. Use `--engine vllm`, or bind a loopback host \ - (the default 127.0.0.1)." - ); - } - Ok(()) -} - -/// Refuse to (re)spawn a managed service recorded on a public host when its -/// endpoint API key is gone, so a respawn cannot reopen the endpoint anonymously. -/// -/// `serve()` applies the loopback-vs-public policy once, via -/// [`resolve_endpoint_auth`], and persists the resulting key. Every later spawn -/// — `rocm services restart`, and `rocmd`'s recovery supervisor — reads that key -/// file back and would otherwise treat "no key file" as "no auth wanted", -/// silently downgrading a protected public endpoint to an open one. The real -/// invariant is a property of the *host*, not of the file: a non-loopback bind -/// must always be authenticated. -/// -/// The gap is reachable through ordinary commands, because a stop deletes the -/// key file and `rocm services restart` accepts a stopped service id (its help -/// points at `rocm services list --all`). -/// -/// `key_present` is a plain `bool` rather than a path so both branches are -/// unit-testable without touching the filesystem, mirroring `is_windows` in -/// [`ensure_public_bind_engine_supported`]. -pub(crate) fn ensure_public_service_has_endpoint_key( - host: &str, - key_present: bool, - requires_api_key: bool, -) -> Result<()> { - // Two ways a service can need a key. A public bind is the obvious one. The - // other is a service that asked for auth on a loopback bind, because - // something outside this process republishes the port — a tailnet publish - // survives a reboot, let alone a restart, so "loopback" stops meaning - // "only this machine" and the bind address can no longer be trusted to - // answer the question on its own. - if requires_api_key && !key_present { - bail!( - "managed service was launched with `--require-api-key` but has no endpoint API key, \ - so restarting it would reopen it without authentication. Something outside this \ - machine may still be publishing its port. The key is dropped when a service stops \ - and cannot be recovered. Launch it again with \ - `rocm serve --require-api-key` (add `--api-key `, or set ROCM_SERVE_API_KEY, \ - to choose the key instead of generating one)." - ); - } - if rocm_engine_protocol::is_public_bind_host(host) && !key_present { - bail!( - "managed service is bound to the public host `{host}` but has no endpoint API key, \ - so restarting it would reopen it without authentication. The key is dropped when a \ - service stops and cannot be recovered. Launch it again with \ - `rocm serve --host {host} --port --allow-public-bind` (add `--api-key `, or set \ - ROCM_SERVE_API_KEY, to choose the key instead of generating one)." - ); - } - Ok(()) -} - -#[cfg(not(windows))] -pub(crate) fn detach_background_command(command: &mut ProcessCommand) { - rocm_core::detach_command_session(command); -} - -#[cfg(not(windows))] -pub(crate) fn attach_background_stdio( - command: &mut ProcessCommand, - log_path: Option<&Path>, -) -> Result<()> { - if let Some(log_path) = log_path { - let log = fs::OpenOptions::new() - .create(true) - .append(true) - .open(log_path) - .with_context(|| format!("failed to open {}", log_path.display()))?; - command - .stdout(Stdio::from(log.try_clone()?)) - .stderr(Stdio::from(log)); - } else { - command.stdout(Stdio::null()).stderr(Stdio::null()); - } - Ok(()) -} - -/// Windows spawn-liveness check: wait out the startup window, then fail with the -/// startup log when the child is already gone. The wait and the liveness probe -/// are injected so a unit test runs the exit-detected path on every platform, -/// including the Windows lane that actually calls it. -/// -/// # Errors -/// Fails with the startup-failure detail when `is_running` reports the child dead. -#[cfg_attr(not(windows), allow(dead_code))] -fn ensure_spawned_child_survives_startup( - child_pid: u32, - log_path: &Path, - settle: impl FnOnce(), - is_running: impl FnOnce(u32) -> bool, -) -> Result<()> { - settle(); - if is_running(child_pid) { - Ok(()) - } else { - bail!("{}", managed_engine_startup_failure_detail("", log_path)) - } -} - -/// `status_clause` is a pre-formatted fragment inserted right after "exited -/// immediately" (e.g. `" with status exit status: 1"` on Unix, where a real -/// `ExitStatus` is available; empty on Windows, where the liveness check -/// below only has a yes/no answer, not an exit code). -pub(crate) fn managed_engine_startup_failure_detail( - status_clause: &str, - log_path: &Path, -) -> String { - let mut recent_lines = read_optional_tail_lines(log_path, 80, "service log"); - if recent_lines.is_empty() { - for _ in 0..5 { - thread::sleep(Duration::from_millis(120)); - recent_lines = read_optional_tail_lines(log_path, 80, "service log"); - if !recent_lines.is_empty() { - break; - } - } - } - if recent_lines.is_empty() { - return format!( - "managed engine exited immediately{status_clause}; inspect {}", - log_path.display() - ); - } - format!( - "managed engine exited immediately{status_clause}; inspect {}\n\nrecent startup log output:\n{}", - log_path.display(), - recent_lines.join("\n") - ) -} -#[cfg(not(windows))] -pub(crate) fn managed_service_process_command(program: &Path, args: &[String]) -> ProcessCommand { - let mut command = ProcessCommand::new(program); - command.args(args); - command -} - -/// Engine-neutral result of a managed launch. Returned rather than printed so the -/// caller can render it either as the rich deployment summary (interactive TTY) or -/// as the plain line-by-line form (piped output, chat assistant), from one code path. -pub(crate) struct ManagedLaunchReport { - pub(crate) service_id: String, - /// `http://host:port/v1`. - pub(crate) endpoint_url: String, - /// `"ready"` (inference confirmed), `"running"` (model listed but not serving - /// yet), `"starting"` (endpoint not answering), or the existing service's - /// status when nothing was spawned. - pub(crate) status: String, - /// True when an equivalent service was already live and nothing was spawned. - pub(crate) already_running: bool, - pub(crate) child_pid: Option, - pub(crate) log_path: Option, - pub(crate) manifest_path: Option, -} - -/// Either an already-live service (nothing spawned) or a freshly spawned engine -/// child that is `running` but not yet HTTP-ready. -/// -/// Split out of [`start_managed_service`] so the attached (`--verbose` / -/// `--foreground`) serve path can spawn the very same detached child and stream -/// its log live from the first line — including startup — instead of blocking on -/// the readiness wait before any output appears. -enum ManagedSpawn { - AlreadyRunning(ManagedLaunchReport), - // `ManagedServiceRecord` is large; box it so the two variants stay a similar - // size (clippy::large_enum_variant). - Spawned { - record: Box, - child_pid: u32, - }, -} - -/// Outcome of the locked allocation step. -/// -/// `Debug` so tests can assert on `Result::expect_err`. -#[derive(Debug)] -enum ManagedPortDecision { - /// An equivalent managed service is already live; nothing may be spawned. - /// Boxed to keep the variants a similar size (clippy::large_enum_variant). - AlreadyLive(Box), - /// The concrete port this launch owns until it hands it to the child. - Lease(rocm_core::LoopbackPortLease), -} - -/// Live managed services, with liveness refreshed by [`load_managed_services`] -/// (dead PIDs demote to "stopped" and therefore neither satisfy the duplicate -/// check nor reserve a port). -/// -/// A read failure degrades to "nothing is live": for an already-`ready` -/// service the OS preflight still guards its port (the engine holds it), and -/// refusing to serve because the services directory could not be listed would -/// be a worse failure than launching. That guard does not cover a `starting` -/// record: its port is released back to the OS in -/// `launch_managed_engine_child_locked` (see `lease.release()`), right before the -/// spawn, and stays unbound until the new engine child has actually bound it, so a concurrent launch that hits this -/// read failure in that narrow window can probe the same port successfully. -/// Accepted as a known gap — narrower than the general services-dir read -/// failure case, and only reachable by a second, near-simultaneous I/O -/// failure on top of one already in flight. -fn live_managed_services(paths: &AppPaths) -> Vec { - load_managed_services(paths) - .unwrap_or_default() - .into_iter() - .filter(managed_service_is_live) - .collect() -} - -/// Ports that live managed services still own. -/// -/// Conservative by design: on the default loopback host a record reserves its -/// port regardless of the host it recorded, because a `0.0.0.0` (or `localhost`) -/// listener also blocks the loopback address a new local server would bind. -/// Over-reserving costs one candidate in a 101-port scan; under-reserving costs -/// a collision. On a custom host the recorded host is compared (see -/// `serve_port::hosts_overlap`). -fn reserved_service_ports(live: &[ManagedServiceRecord]) -> Vec { - live.iter() - .map(|record| serve_port::ReservedPort { - port: record.port, - host: record.host.clone(), - service_id: record.service_id.clone(), - status: record.status.clone(), - }) - .collect() -} - -/// The decision half of the launch transaction: refresh liveness, apply the -/// `(engine, canonical_model_id)` idempotency guard, then lease a concrete port -/// against both live records and the real OS. -/// -/// Callers MUST hold [`rocm_core::AppPaths::managed_launch_lock_path`] across -/// this call *and* the record publication that follows, or two launches can -/// pick the same candidate. -/// `probe` is injected so tests can drive deterministic bind outcomes without a -/// GPU or a spawn. -fn resolve_managed_port_in_transaction( - paths: &AppPaths, - engine: &str, - canonical_model_id: &str, - host: &str, - request: serve_port::PortRequest, - probe: &mut dyn FnMut(u16) -> std::io::Result, -) -> Result { - let live = live_managed_services(paths); - // Idempotency first: an already-live equivalent service means no port is - // needed at all, so it must be answered before any candidate is leased. - if let Some(existing) = existing_live_managed_service_in(&live, engine, canonical_model_id) { - return Ok(ManagedPortDecision::AlreadyLive(Box::new(existing.clone()))); - } - let reserved = reserved_service_ports(&live); - serve_port::resolve_serve_port(host, request, &reserved, probe).map(ManagedPortDecision::Lease) -} - -/// Grace given to a child that must be terminated because its record could not -/// be published. Short: the child is milliseconds old and has nothing to flush. -const MANAGED_LAUNCH_CLEANUP_GRACE: Duration = Duration::from_secs(2); - -/// Undo a failed managed launch attempt before the launch lock is released, -/// so no failed transaction leaves a live reservation or a stray endpoint key. -/// -/// Order matters: a spawned child is terminated *first*, so it is not still -/// holding the port or writing to the log by the time cleanup runs. The -/// termination outcome is audited but not acted on: the record is marked -/// abandoned (`stopped`, releasing the port) even when death was not confirmed. If no child was ever spawned (`record.engine_pid` is still `None`), -/// every artifact this attempt created is removed: there is nothing for -/// `rocm services`/`rocm logs` to show. If a child did run, its record and log -/// are kept instead (marked `status = "stopped"`, see -/// [`ManagedServiceRecord::mark_launch_abandoned`]), since the error that led here -/// already told the user to go inspect that log; only the endpoint key, which a -/// failed attempt has no use for, is cleared. -/// Every step is best-effort and idempotent; cleanup must never mask the -/// original failure with a second one. -fn abandon_failed_managed_launch(paths: &AppPaths, record: &ManagedServiceRecord) { - if let Some(pid) = record.engine_pid.filter(|pid| *pid != 0) { - // The token captured at spawn, not a fresh `capture`: when the child - // already exited, a fresh one would describe a recycled pid's new owner - // and the forced tree kill would hit it. Where start tokens exist - // (Linux), a missing one means the child was gone before it could be - // captured, so the pid is unverifiable and nothing is signalled - // (`terminate_verified` alone would "best-effort proceed"). Windows - // records no token at all, so there the kill proceeds as before. - let outcome = if record.engine_start_ticks.is_none() && cfg!(target_os = "linux") { - rocm_core::TerminationOutcome::Unverified - } else { - rocm_core::terminate_verified( - &rocm_core::ProcessIdentity::new(pid, record.engine_start_ticks), - rocm_core::KillScope::Tree, - MANAGED_LAUNCH_CLEANUP_GRACE, - true, - ) - }; - record_cli_audit_event( - paths, - "service", - "managed_service_launch_cleanup", - "warn", - format!( - "terminated engine child pid={pid} after a failed launch publication outcome={}", - outcome.as_str() - ), - Some(&record.service_id), - ); - let mut abandoned = record.clone(); - abandoned.mark_launch_abandoned(); - let _ = abandoned.write(); - endpoint_keys::clear_endpoint_api_key(paths, &record.service_id); - return; - } - for path in [ - record.manifest_path.as_path(), - record.log_path.as_path(), - record.engine_state_path.as_path(), - ] { - let _ = fs::remove_file(path); - } - endpoint_keys::clear_endpoint_api_key(paths, &record.service_id); -} - -/// The ways a live managed service that already matches on liveness, engine and -/// canonical model id can still be unusable by this invocation. -/// -/// Those three terms are necessary but not sufficient: three further terms can -/// still refuse, one variant each. [`managed_service_reuse_refusal`] is the -/// single decision procedure both callers ask, so the two cannot answer -/// differently — which they previously did; see that function for what the -/// disagreement cost. -pub(crate) enum ManagedReuseRefusal { - /// The live service was started with different serve options. - RecipeMismatch, - /// `--require-api-key` was asked for and the live service has no key. - /// - /// The engine reads its key once, at launch, from the environment the spawn - /// builds — so a server started without one keeps serving anonymously no - /// matter what is written afterwards. Upgrading the record would be worse - /// than doing nothing: it would claim auth the live process does not - /// enforce, and `ensure_public_service_has_endpoint_key` would pass on the - /// strength of a key file nothing reads. `rocm remote serve` relies on this: - /// it publishes a loopback port onto the tailnet and prints "the API key - /// above is what stops anyone else calling it", which reusing an - /// unauthenticated service silently would make false about an endpoint the - /// whole tailnet can reach. Refusing is the only answer that fails closed. - MissingApiKey, - /// The live service runs under a device policy that does not satisfy this - /// invocation's. Reusing a `cpu_only` service to answer a `gpu_required` - /// serve is the silent CPU fallback AGENTS.md section 6 forbids. - DevicePolicyTooWeak, -} - -/// Why this invocation cannot reuse `existing` — or `None` when it genuinely -/// can. See [`ManagedReuseRefusal`] for the terms this answers on. -/// -/// This exists because `serve()`'s reuse pre-gate lets a matching live service -/// skip the no-usable-AMD-GPU refusal. Deciding that bypass on a *weaker* test -/// than the spawn then uses to accept reuse produced a dead end: on a GPU-less -/// host (`HIP_VISIBLE_DEVICES=` with a server live from another shell) a serve -/// with different options bypassed the refusal, reached the spawn, and was told -/// to stop the server and run the command again — and the rerun, with nothing -/// left to reuse, hit the no-GPU refusal. The advice named a command that could -/// not clear the condition, which AGENTS.md section 3 forbids. Sharing the -/// predicate means a service that cannot actually be reused never buys the -/// bypass, so the user gets the accurate refusal first and keeps their server. -/// -/// `requested_device_policy` is taken by value, not as an `Option`, on purpose. -/// It was optional so the spawn could opt out of the device-policy term, and -/// that left two holes at once: the spawn's `DevicePolicyTooWeak` arm was -/// unreachable (an untestable bail), and `serve()` — the caller that *does* -/// gate the GPU refusal on this answer — could silently drop the term by -/// passing `None`, with the whole suite still green. Both callers have a -/// `DevicePolicy` in hand, so requiring it costs nothing and makes that -/// mutation a compile error. It widens nothing observable today either: the -/// only policy a record can carry is the one `parse_device_policy` admits -/// (see below), so `existing.device_policy == Some("cpu_only")` is not -/// reachable from any `rocm serve` this CLI can run. -/// The `rocm serve` no-usable-GPU refusal, worded once. -/// -/// Two places raise it and they must say the same thing: `serve()`'s fail-fast -/// pre-flight, and the spawn below when the reuse that waived that pre-flight -/// turns out not to be available after all. -pub(crate) fn no_usable_gpu_refusal(device_policy: &DevicePolicy) -> String { - // Keyed on the same `is_wsl_host()` the usable-GPU probe branches on, so the - // advice follows the platform whose answer produced the refusal. - no_usable_gpu_refusal_text(device_policy_name(device_policy), rocm_core::is_wsl_host()) -} - -/// The refusal text for `policy`, split on whether the probe that refused was -/// the WSL one. -/// -/// WSL reaches the GPU through `/dev/dxg` and ROCDXG, and has no `/dev/kfd`, so -/// telling a WSL user to confirm that device is present names something that can -/// never clear the condition. There the advice points at the status `rocm -/// examine` reports for the path the probe actually asked about -/// ([`rocm_core::WSL_ROCDXG_READY`]), and at `rocm diagnose`, which names what is -/// missing. -fn no_usable_gpu_refusal_text(policy: &str, on_wsl: bool) -> String { - let remedy = if on_wsl { - format!( - "WSL reaches the GPU through /dev/dxg and ROCDXG rather than /dev/kfd: run \ - `rocm examine` and check that `driver_status` is `{ready}` (`rocm diagnose` \ - names what is missing; a ROCDXG library the loader reaches only through \ - LD_LIBRARY_PATH needs that variable exported in this shell, see docs/wsl.md), and ensure HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES \ - are not masking every device.", - ready = rocm_core::WSL_ROCDXG_READY - ) - } else { - "Check the driver with `rocm examine`, confirm /dev/kfd is present, and ensure \ - HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES are not masking every device." - .to_owned() - }; - format!( - "no usable AMD GPU detected; `rocm serve` requires a GPU under the {policy} policy \ - and does not fall back to CPU. {remedy}" - ) -} - -pub(crate) fn managed_service_reuse_refusal( - existing: &ManagedServiceRecord, - requested_recipe_json: Option<&str>, - require_api_key: bool, - requested_device_policy: &DevicePolicy, -) -> Option { - if existing.engine_recipe_json.as_deref() != requested_recipe_json { - return Some(ManagedReuseRefusal::RecipeMismatch); - } - if require_api_key && !existing.requires_api_key { - return Some(ManagedReuseRefusal::MissingApiKey); - } - // Only `gpu_required` can be let down by the record's policy, and the - // enumeration is exhaustive rather than narrow: `parse_device_policy` - // (`main.rs`) maps `auto`, `gpu`, `gpu_required` *and* `gpu_preferred` onto - // `GpuRequired`, and rejects `cpu_only` with an error, so `GpuRequired` is - // the only policy `serve()` can hand this predicate — pinned by - // `parse_device_policy_defaults_to_gpu_required_without_cpu_fallback`. There - // is deliberately no wider match on "anything but `CpuOnly`": it would be - // identical at runtime and would read as a guard against a `gpu_preferred` - // bypass that no CLI invocation can reach. A `cpu_only` invocation would be - // satisfied by anything anyway and never reaches the bail. README.md's - // "that is the only policy pair refused" is the contract, and this matches - // it exactly. - if matches!(requested_device_policy, DevicePolicy::GpuRequired) - && existing.device_policy.as_deref() == Some("cpu_only") - { - return Some(ManagedReuseRefusal::DevicePolicyTooWeak); - } - None -} - -/// Spawn the detached engine child shared by the managed (background) and -/// attached (`--verbose`/`--foreground`) serve paths. Returns before the HTTP -/// readiness wait; callers decide whether to block on readiness -/// ([`start_managed_service`]) or start tailing the log immediately -/// ([`run_attached_service`]). -/// -/// The whole body is one serialized transaction under the shared allocation -/// lock: duplicate detection, live-reservation refresh, candidate leasing, -/// manifest/log creation, argv preparation, the bounded immediate-exit check, -/// and record publication. Readiness deliberately runs after the lock is -/// released. -#[allow(clippy::too_many_arguments)] -fn spawn_managed_engine_child( - paths: &AppPaths, - engine: &str, - service_id: &str, - requested_model: &str, - resolve: &ResolveModelResponse, - host: &str, - port_request: serve_port::PortRequest, - device_policy: &DevicePolicy, - gpu_indices: &[u32], - runtime_id: Option<&str>, - env_id: Option<&str>, - engine_recipe: Option<&EngineRecipeHint>, - require_api_key: bool, - gpu_refusal_waived_for_reuse: bool, -) -> Result { - paths.ensure()?; - fs::create_dir_all(paths.services_dir())?; - - // No lock is acquired here: every production caller - // ([`start_managed_service`], [`run_attached_service`]) already holds - // [`rocm_core::AppPaths::managed_launch_lock_path`] for the whole of this - // call, so no two transactions can choose the same port or publish - // competing records for one engine+model. A second acquisition in this - // frame would self-deadlock, since the lock is exclusive per open file - // description, not per process. - // - // Idempotency guard: if a managed service for this engine+model is already - // alive, surface it and spawn nothing. A second `serve --managed` (e.g. the - // chat assistant re-issuing the same request) is treated as satisfied, not - // an error. Keyed on engine+canonical model — the freshly generated - // `service_id` is timestamp-unique and would never match an existing one. - // Stale/dead services fall through and relaunch normally. - let requested_recipe_json = engine_recipe - .map(serde_json::to_string) - .transpose() - .context("failed to encode engine recipe hint")?; - let lease = match resolve_managed_port_in_transaction( - paths, - engine, - &resolve.canonical_model_id, - host, - port_request, - &mut serve_port::loopback_bind_probe, - )? { - ManagedPortDecision::AlreadyLive(existing) => { - let existing = *existing; - // Asked through the shared predicate rather than re-tested here, so this - // refusal and `serve()`'s reuse pre-gate cannot disagree about whether a - // service is reusable. When they disagreed, the pre-gate let a - // non-reusable service bypass the no-GPU refusal and the user was sent - // here to be told to stop a working server for nothing. - // - // The same device policy the serve path passes. This side used to pass - // `None` to keep the spawn's acceptance untouched, but that opted the - // spawn out of a term it has the input for, left its own - // `DevicePolicyTooWeak` bail unreachable, and made the serve path's - // argument droppable. Nothing observable widens: `parse_device_policy` - // is the only producer of a `DevicePolicy` on every path that reaches - // here and it rejects `cpu_only`, so no record this CLI writes can - // carry the policy this term refuses. - match managed_service_reuse_refusal( - &existing, - requested_recipe_json.as_deref(), - require_api_key, - device_policy, - ) { - Some(ManagedReuseRefusal::RecipeMismatch) => bail!( - "managed service `{}` is already running for engine `{engine}` and model `{}` with different serve options (recipe hint, tool-call parser, or generation defaults); stop it and run `rocm serve` again to apply the requested options", - existing.service_id, - resolve.canonical_model_id - ), - Some(ManagedReuseRefusal::MissingApiKey) => bail!( - "managed service `{}` is already running for engine `{engine}` and model `{}` \ - without authentication, and a running server cannot be given a key it did not \ - start with; stop it with `{}` and run the command again to \ - serve it with `--require-api-key`", - existing.service_id, - resolve.canonical_model_id, - service_hints::service_stop_hint(&existing.service_id) - ), - // Reachable only from a record whose `device_policy` says - // `cpu_only`, which no `rocm serve` can write today - // (`parse_device_policy` refuses that policy outright) — so this is - // a guard against a record written by some future path, not a - // refusal users see now. The advice still names a command that - // clears the condition: stopping the service removes the only thing - // this term can object to, and the rerun then launches normally - // (AGENTS.md section 3). - Some(ManagedReuseRefusal::DevicePolicyTooWeak) => bail!( - "managed service `{}` is already running under a device policy that cannot \ - satisfy this request; stop it with `{}` and run the command again", - existing.service_id, - service_hints::service_stop_hint(&existing.service_id) - ), - None => {} - } - // An explicit port is a demand, not a hint: reporting success on a - // different port than the one asked for would silently drop it. - if let Some(port) = port_request.explicit() - && port != existing.port - { - bail!( - "managed service `{}` is already serving engine `{engine}` and model `{}` on port {}, so port {port} would not be used; stop it with `{}` and run the command again to serve on that port", - existing.service_id, - resolve.canonical_model_id, - existing.port, - service_hints::service_stop_hint(&existing.service_id) - ); - } - record_cli_audit_event( - paths, - "service", - "managed_service_launch_skipped", - "info", - format!( - "skipped duplicate managed launch engine={engine} model={} existing_service_id={} status={}", - resolve.canonical_model_id, existing.service_id, existing.status - ), - Some(&existing.service_id), - ); - return Ok(ManagedSpawn::AlreadyRunning(ManagedLaunchReport { - service_id: existing.service_id, - endpoint_url: existing.endpoint_url, - status: existing.status, - already_running: true, - child_pid: None, - log_path: None, - manifest_path: None, - })); - } - ManagedPortDecision::Lease(lease) => lease, - }; - - // Nothing live to reuse — and if the only reason this invocation got past - // `serve()`'s no-usable-GPU pre-flight was a reusable service, that premise - // has just been falsified. `serve()` decided it from a snapshot taken before - // GPU selection and the launch lock; the check above is the authoritative - // one, so a service that died in between (an OOM, say) would otherwise fall - // through to a real spawn on a host with no usable GPU, and the user would - // get whatever the engine says about having no device instead of the - // actionable refusal the pre-flight exists to give. Raise it here, where the - // premise is known to be false. - if gpu_refusal_waived_for_reuse { - bail!("{}", no_usable_gpu_refusal(device_policy)); - } - - let mut record = ManagedServiceRecord::new( - paths, - service_id, - engine, - requested_model, - resolve.canonical_model_id.clone(), - host, - lease.port(), - "managed", - 0, - runtime_id.map(str::to_owned), - env_id.map(str::to_owned), - Some(device_policy_name(device_policy).to_owned()), - ); - record.gpu_indices = gpu_indices.to_vec(); - record.served_model_id = resolve.served_model_id.clone(); - record.engine_recipe_json = requested_recipe_json; - // The flag the user actually passed, carried through rather than re-derived. - // - // Deriving it from key-file presence looked equivalent and was not: - // `resolve_endpoint_auth` mints a key for *every* non-loopback bind whether or - // not auth was demanded, so a plain `--host 0.0.0.0 --allow-public-bind` - // recorded `true` here. The guard below tests this field before the bind - // address, so that service was then refused with a message naming a flag it - // never used and a relaunch command that drops `--allow-public-bind` — the - // public-bind branch, which carries the right command, became unreachable. - // - // This field means "the user demanded auth on a bind that would not otherwise - // require it". A public bind needs no such record; its address still says so. - record.requires_api_key = require_api_key; - // Publishing the record inside the lock is what reserves the leased port for - // every other transaction. From here on, any failure must remove it again. - record.write()?; - - match launch_managed_engine_child_locked( - paths, - &mut record, - lease, - engine, - service_id, - resolve, - host, - device_policy, - gpu_indices, - runtime_id, - env_id, - engine_recipe, - #[cfg(not(windows))] - None, - ) { - Ok(child_pid) => Ok(ManagedSpawn::Spawned { - record: Box::new(record), - child_pid, - }), - Err(error) => { - // The single cleanup choke point for every failure path below the - // record's first write: `record.engine_pid` tells it whether a child - // ever ran, so a pre-spawn failure deletes everything and a post-spawn - // one keeps the record and log (marked `status = "stopped"`) instead. - abandon_failed_managed_launch(paths, &record); - Err(error) - } - } -} - -// Test-only, thread-scoped spawn interceptor for callers that cannot take a -// `spawn_override` argument (`start_managed_service`, `run_attached_service`): -// lets a test observe state at the instant production would spawn the child. -#[cfg(all(test, not(windows)))] -thread_local! { - static TEST_SPAWN_HOOK: std::cell::RefCell>>> = - const { std::cell::RefCell::new(None) }; -} - -#[cfg(all(test, not(windows)))] -fn test_spawn_hook() -> Option>> { - TEST_SPAWN_HOOK.with(|hook| hook.borrow().clone()) -} - +pub(crate) use launch_transaction::{managed_service_reuse_refusal, no_usable_gpu_refusal}; +pub(crate) use readiness::e2e_simulate_oom_launch; #[cfg(not(windows))] -type SpawnOverride<'a> = dyn Fn(&mut ProcessCommand) -> io::Result + 'a; - -/// The side-effecting half of the launch transaction: create the log, prepare -/// concrete argv, hand the leased port to the child, spawn it, reject a child -/// that has already exited, and publish the running record. -/// -/// Split out of [`spawn_managed_engine_child`] so that *every* failure below the -/// record's first write is funnelled through one cleanup path before the -/// launch lock is released. Returns the child pid. -#[allow(clippy::too_many_arguments)] -fn launch_managed_engine_child_locked( - paths: &AppPaths, - record: &mut ManagedServiceRecord, - lease: rocm_core::LoopbackPortLease, - engine: &str, - service_id: &str, - resolve: &ResolveModelResponse, - host: &str, - device_policy: &DevicePolicy, - gpu_indices: &[u32], - runtime_id: Option<&str>, - env_id: Option<&str>, - engine_recipe: Option<&EngineRecipeHint>, - // Test-only seam: lets a test assert the leased port is actually bindable - // (i.e. released) at the exact point production would spawn the child, - // without needing to fake or intercept the real spawn mechanics. - #[cfg(not(windows))] spawn_override: Option<&SpawnOverride<'_>>, -) -> Result { - let port = record.port; - if let Some(parent) = record.engine_state_path.parent() { - fs::create_dir_all(parent) - .with_context(|| format!("failed to create {}", parent.display()))?; - } - fs::File::create(&record.log_path) - .with_context(|| format!("failed to create {}", record.log_path.display()))?; - let current_exe = managed_service_launcher_path() - .context("failed to resolve current rocm executable path")?; - let serve_args = builtin_engine_serve_http_args( - engine, - service_id, - &resolve.canonical_model_id, - host, - port, - device_policy, - gpu_indices, - runtime_id, - env_id, - engine_recipe, - &record.engine_state_path, - Some(&record.log_path), - )?; - let engine_envs_root = env_root_for_service(paths, engine, runtime_id, env_id)?; - // Hand the child the *path* to the endpoint key file (public bind only) via the - // environment. A path — not the secret value — is what the detached-spawn - // primitives accept as an env override, and it keeps the key off both the argv - // and the environment block. `serve()` wrote the file before spawning. - // Validity, not mere existence: the engine adapters resolve the key with - // `endpoint_api_key_from_file` and enforce nothing when it yields `None`, so - // an empty or malformed key file would otherwise satisfy the guard below and - // still produce an unauthenticated public listener. - let endpoint_key_file = endpoint_keys::endpoint_key_file_if_present(paths, service_id) - .filter(|path| rocm_engine_protocol::endpoint_api_key_from_file(path).is_some()); - // `serve()` already resolved and stored the key for a public bind, so the - // public-bind branch cannot fire on the fresh-launch path today. It is the - // shared choke point for managed spawns, so enforce the invariant here too - // rather than relying on every future caller having done so. - // - // `record.requires_api_key` is passed, not a literal, and the two arguments - // are deliberately different things: that field is the `--require-api-key` - // flag the caller passed, `endpoint_key_file` is whether a *usable* key is on - // disk. A present but empty or malformed key file is where they disagree, and - // is exactly what the `requires_api_key` branch exists to refuse. - // - // The field is threaded, never derived from the key file. Deriving it marked - // every public bind as having demanded auth — see the assignment above. - ensure_public_service_has_endpoint_key( - host, - endpoint_key_file.is_some(), - record.requires_api_key, - )?; - // Hand the port over: close the probe socket immediately before the spawn, - // still under the caller's launch lock, so no other ROCm transaction can - // take it in the meantime. On a custom host the lease never held a socket - // and this is a no-op — the engine owns that bind. - debug_assert_eq!(lease.port(), port); - lease.release(); - #[cfg(windows)] - let child_pid = { - let env_values = app_path_env_var_values(paths, engine_envs_root.as_deref()); - let mut env_refs = app_path_env_var_refs(&env_values); - if let Some(key_file) = endpoint_key_file.as_deref() { - env_refs.push((rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV, key_file)); - } - let child_pid = rocm_core::spawn_detached_no_inherit(¤t_exe, &serve_args, &env_refs) - .context("failed to launch managed engine process")?; - // Set before the immediate-exit check below, so a bail! here still leaves - // `record.engine_pid` telling cleanup that a child was spawned. The - // start-time token is captured here too, while the child is certainly - // ours: cleanup binds its kill to it rather than to whatever owns the - // pid by then. - record.record_engine_identity(child_pid); - ensure_spawned_child_survives_startup( - child_pid, - &record.log_path, - || thread::sleep(Duration::from_millis(200)), - process_is_running, - )?; - child_pid - }; - #[cfg(not(windows))] - let child_pid = { - let mut command = managed_service_process_command(¤t_exe, &serve_args); - command.stdin(Stdio::null()); - attach_background_stdio(&mut command, Some(&record.log_path))?; - detach_background_command(&mut command); - apply_app_path_env(&mut command, paths); - if let Some(engine_envs_root) = engine_envs_root.as_deref() { - command.env("ROCM_CLI_ENGINE_ENVS_ROOT", engine_envs_root); - } - if let Some(key_file) = endpoint_key_file.as_deref() { - command.env(rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV, key_file); - } - #[cfg(all(test, not(windows)))] - let hook = test_spawn_hook(); - #[cfg(all(test, not(windows)))] - let spawn_override = spawn_override.or(hook.as_deref()); - let mut child = match spawn_override { - Some(spawn) => spawn(&mut command), - None => command.spawn(), - } - .context("failed to launch managed engine process")?; - let child_pid = child.id(); - // Set before the immediate-exit check below, so a bail! here still leaves - // `record.engine_pid` telling cleanup that a child was spawned. The - // start-time token is captured here too, while the child is certainly - // ours: cleanup binds its kill to it rather than to whatever owns the - // pid by then. - record.record_engine_identity(child_pid); - thread::sleep(Duration::from_millis(200)); - if let Some(status) = child - .try_wait() - .context("failed to check managed engine startup state")? - { - bail!( - "{}", - managed_engine_startup_failure_detail( - &format!(" with status {status}"), - &record.log_path - ) - ); - } - child_pid - }; - // The child is both roles until the engine state file names a separate - // server process. Each role gets its own token, captured while the child is - // alive: a stop pairs every recorded PID with its own token, and the two - // PIDs diverge as soon as `rocmd` restarts the service — at which point an - // engine PID recorded without one could only be signalled blind. - // The engine's token was already captured right after the spawn (so cleanup - // can bind to it); re-capturing here could replace it with a later owner's. - record.record_supervisor_identity(child_pid); - record.status = "running".to_owned(); - if let Err(error) = record.write() { - // Cleanup is the caller's job now: `record.engine_pid` (set above) is - // already enough for `abandon_failed_managed_launch` to terminate this - // child, so there is nothing left for this inner path to do beyond - // surfacing the error. - return Err(error.context("failed to publish the managed service record")); - } - - Ok(child_pid) -} - -/// How long one readiness wait polls before `rocm serve` reports `starting`. -const READINESS_WAIT: Duration = Duration::from_secs(45); - -/// `rocm serve --wait`: keep waiting past [`READINESS_WAIT`] (a first start -/// downloads the model, which can take far longer) until the server answers or -/// stops being live. A server that exited ends the wait with the last reading -/// instead of blocking forever. -fn wait_past_readiness_budget( - mut probe: impl FnMut() -> EndpointReadiness, - mut still_live: impl FnMut() -> bool, -) -> EndpointReadiness { - loop { - let readiness = probe(); - if readiness == EndpointReadiness::Serving || !still_live() { - return readiness; - } - } -} - -/// Whether `pid` is a running process that is not a zombie. The serve -/// supervisor is an unreaped child of this process, so after it dies -/// `process_is_running` alone keeps reporting it alive. -fn service_pid_is_alive(pid: u32) -> bool { - rocm_core::process_is_running(pid) && !process_is_zombie(pid) -} - -#[cfg(target_os = "linux")] -fn process_is_zombie(pid: u32) -> bool { - let Ok(stat) = fs::read_to_string(format!("/proc/{pid}/stat")) else { - return false; - }; - // `comm` may contain spaces and parentheses; the state follows the last `)`. - stat.rfind(')') - .and_then(|end| stat.get(end + 1..)) - .and_then(|rest| rest.split_whitespace().next()) - == Some("Z") -} - -#[cfg(not(target_os = "linux"))] -const fn process_is_zombie(_pid: u32) -> bool { - false -} - -/// Whether the record still describes a server that can finish starting: its -/// status is a live one *and* a tracked process is alive. The status alone is -/// not enough, since a killed engine never writes a terminal state. -fn record_still_live(record: &ManagedServiceRecord, pid_alive: impl Fn(u32) -> bool) -> bool { - if !crate::managed_service_status::managed_service_is_live(record) { - return false; - } - let pids = crate::managed_service_status::recorded_service_pids(record); - pids.is_empty() || pids.into_iter().any(pid_alive) -} - -/// [`record_still_live`] over the manifest on disk, so a `rocm services stop` -/// from another terminal (which writes only the manifest) ends the wait too. -fn service_still_live(paths: &AppPaths, service_id: &str, pid_alive: impl Fn(u32) -> bool) -> bool { - let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { - return false; - }; - let _ = latest.refresh_from_engine_state(); - record_still_live(&latest, pid_alive) -} - -/// Record a server that died mid-start as stopped, leaving a manifest that a -/// `services stop` already settled untouched. A `failed` state the engine -/// reported is persisted rather than overwritten, so the reason survives. -fn settle_exited_service(paths: &AppPaths, service_id: &str) { - let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { - return; - }; - let on_disk = latest.status.clone(); - let _ = latest.refresh_from_engine_state(); - if crate::managed_service_status::managed_service_is_live(&latest) { - latest.status = "stopped".to_owned(); - } else if latest.status == on_disk { - return; - } - let _ = latest.write(); -} - -/// The failure a `--wait` ends with when the server died before it became -/// ready, after recording the death in the manifest. `None` while the wait did -/// not apply, the endpoint is serving, or the server is still alive. -fn died_during_wait( - paths: &AppPaths, - service_id: &str, - wait_until_ready: bool, - readiness: EndpointReadiness, - pid_alive: impl Fn(u32) -> bool, -) -> Option { - if !wait_until_ready - || readiness == EndpointReadiness::Serving - || service_still_live(paths, service_id, pid_alive) - { - return None; - } - settle_exited_service(paths, service_id); - let mut message = format!( - "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `rocm logs --service {service_id}` for its output" - ); - // A plain `serve` adds the memory guidance for a vLLM launch that ran out of - // VRAM; the wait's failure is the same launch, so it gets the same note. - // "starting" is the status the note keys on for "never became ready", which is - // what a `--wait` death is, whatever the settled record now says. - if let Ok(record) = ManagedServiceRecord::read_manifest(paths, service_id) { - for note in crate::serve_summary::append_oom_serve_note( - Vec::new(), - &record.engine, - false, - "starting", - Some(&record.log_path), - ) { - let _ = write!(message, "\n note: {note}"); - } - } - Some(anyhow!(message)) -} - -/// Record the outcome of a finished wait on the manifest as it is now, changing -/// only what the wait learned. The copy read at spawn time can be long stale: -/// `rocmd` may have recorded a restart or a new supervisor pid, and `rocm logs` -/// refreshes the record, so writing it back whole would revert those. -fn record_wait_outcome( - paths: &AppPaths, - held: &mut ManagedServiceRecord, - status: &str, - verified: bool, -) -> Result<()> { - let mut latest = ManagedServiceRecord::read_manifest(paths, &held.service_id) - .unwrap_or_else(|_| held.clone()); - status.clone_into(&mut latest.status); - if verified { - // Latch the verification the wait just performed, so the readiness checks - // behind `services list` and chat read it instead of re-probing. - latest.inference_verified_at_unix_ms = Some(rocm_core::unix_time_millis() as u64); - } - latest.write()?; - *held = latest; - Ok(()) -} - -/// Test-only fault-injection switch, off (`const false`) in shipped binaries: -/// the `e2e-oom-fault-injection` feature is enabled only by the `xtask e2e` -/// mock-lane build, never by the release build. When enabled and -/// `ROCM_E2E_SIMULATE_OOM_LAUNCH=1` is set on the process, `rocm serve` fabricates -/// the on-disk state of a managed vLLM launch that spawned, wrote an -/// out-of-memory traceback to the log it owns, and never became ready — so the -/// positive OOM-guidance summary path can be verified on a GPU-less CI host, -/// where a real launch cannot run. See [`simulate_oom_managed_launch`]. -#[cfg(feature = "e2e-oom-fault-injection")] -pub(crate) fn e2e_simulate_oom_launch() -> bool { - std::env::var_os("ROCM_E2E_SIMULATE_OOM_LAUNCH").is_some_and(|value| value == "1") -} - -/// Release builds carry no fault-injection switch: the seam compiles out -/// entirely, so `rocm serve` behaves identically to a build without the feature. -#[cfg(not(feature = "e2e-oom-fault-injection"))] -pub(crate) const fn e2e_simulate_oom_launch() -> bool { - false -} - -/// Test-only (see [`e2e_simulate_oom_launch`]): fabricate the on-disk state of a -/// managed vLLM launch that spawned, OOM'd, and never became ready, then return -/// the launch report the summary path would see for it — `already_running: false` -/// (this invocation launched it), `status: "starting"` (it failed to become -/// ready), and a `log_path` whose contents carry a real allocator OOM signature. -/// [`crate::serve_summary::append_oom_serve_note`] then renders the memory-knob guidance. No process is -/// spawned. Compiled out of shipped binaries. -#[cfg(feature = "e2e-oom-fault-injection")] -#[allow(clippy::too_many_arguments)] -fn simulate_oom_managed_launch( - paths: &AppPaths, - engine: &str, - service_id: &str, - requested_model: &str, - resolve: &ResolveModelResponse, - host: &str, - port: u16, - device_policy: &DevicePolicy, - gpu_indices: &[u32], - runtime_id: Option<&str>, - env_id: Option<&str>, -) -> Result { - paths.ensure()?; - fs::create_dir_all(paths.services_dir())?; - let mut record = ManagedServiceRecord::new( - paths, - service_id, - engine, - requested_model, - resolve.canonical_model_id.clone(), - host, - port, - "managed", - std::process::id(), - runtime_id.map(str::to_owned), - env_id.map(str::to_owned), - Some(device_policy_name(device_policy).to_owned()), - ); - record.gpu_indices = gpu_indices.to_vec(); - record.status = "starting".to_owned(); - record.write()?; - // A real allocator OOM signature in the log this invocation owns, so the - // summary renders memory guidance for a launch that actually OOM'd. - // - // The engine's own error block is reproduced below it, because that is what - // a real managed launch leaves on disk, and leaving it out hid a bug this - // scenario should have caught: the supervisor and the engine share one log, - // so when vLLM OOMs the engine bails with its startup-log context — which - // embeds rocm-cli's own OOM hint — and `exit_code_for` writes that whole - // error back into the same file, *after* the real line. A fixture carrying - // only the raw OOM line can never see the newest-first scan land on - // rocm-cli's own sentence instead of the user's error. - let real_line = "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB."; - fs::write( - &record.log_path, - format!( - "{real_line}\n\ - Error: vLLM server process exited before becoming ready (status: exit status: 1)\n\ - \n\ - Startup log, last {} lines as the terminal drew them:\n\ - {real_line}\n\ - \n\ - {}{real_line}). {}\n\ - For conditional remediation, run `{} 'vllm: {real_line}'`.\n", - rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES, - rocm_core::VLLM_OOM_DETECTED_PREFIX, - rocm_core::VLLM_GPU_MEMORY_UTILIZATION_HINT, - rocm_core::VLLM_OOM_DIAGNOSE_COMMAND, - ), - ) - .with_context(|| format!("failed to write {}", record.log_path.display()))?; - Ok(ManagedLaunchReport { - service_id: service_id.to_owned(), - endpoint_url: format!("{}/v1", format_http_base_url(host, port)), - status: "starting".to_owned(), - already_running: false, - child_pid: None, - log_path: Some(record.log_path), - manifest_path: Some(record.manifest_path), - }) -} - -/// Probe `record`'s endpoint until it answers: once within the readiness -/// budget, or (with `wait_until_ready`) for as long as the server stays alive. -fn wait_for_record_ready( - paths: &AppPaths, - record: &ManagedServiceRecord, - wait_until_ready: bool, - endpoint_api_key: Option<&str>, - on_wait_tick: &mut dyn FnMut(Duration), -) -> EndpointReadiness { - let probe = |on_tick: &mut dyn FnMut(Duration)| { - wait_for_service_http_ready_with_progress( - &record.engine, - &record.host, - record.port, - &|| { - let mut latest = record.clone(); - let _ = latest.refresh_from_engine_state(); - latest.served_model_name().to_owned() - }, - endpoint_api_key, - READINESS_WAIT, - on_tick, - ) - }; - if wait_until_ready { - wait_past_readiness_budget( - || probe(&mut *on_wait_tick), - || service_still_live(paths, &record.service_id, service_pid_is_alive), - ) - } else { - probe(on_wait_tick) - } -} - -/// An equivalent service is already live. Without `--wait`, or once it is -/// serving, report it as is; otherwise wait for it like a fresh launch would. -fn wait_on_existing_service( - paths: &AppPaths, - mut report: ManagedLaunchReport, - wait_until_ready: bool, - on_wait_begin: &mut dyn FnMut(&str, u16), - on_wait_tick: &mut dyn FnMut(Duration), -) -> Result { - if !wait_until_ready || report.status == "ready" { - return Ok(report); - } - let Ok(mut record) = ManagedServiceRecord::read_manifest(paths, &report.service_id) else { - return Ok(report); - }; - // The id of the server being waited on, which is not the caller's freshly - // generated one when it attached to an existing server. - on_wait_begin(&report.service_id, record.port); - // The key the caller generated belongs to a launch that did not happen; the - // existing service answers to its own. - let stored_key = endpoint_keys::endpoint_api_key(paths, &report.service_id); - let readiness = - wait_for_record_ready(paths, &record, true, stored_key.as_deref(), on_wait_tick); - if let Some(error) = died_during_wait( - paths, - &report.service_id, - true, - readiness, - service_pid_is_alive, - ) { - return Err(error); - } - report.status = status_for_readiness(readiness).to_owned(); - record_wait_outcome( - paths, - &mut record, - &report.status, - readiness == EndpointReadiness::Serving, - )?; - Ok(report) -} - -/// Launch a managed service in the background: run the locked allocation and -/// spawn transaction, release the launch lock once the record is published, -/// then wait for HTTP readiness. -/// -/// Drops the endpoint key `serve()` freshly staged for this attempt if the -/// launch fails before any `ManagedServiceRecord` exists to own it (port -/// reserved/in use, range exhausted, other bind error) — such a failure -/// returns before `abandon_failed_managed_launch` ever runs, so nothing else -/// would clear it. Scoped to the spawn call only, mirroring -/// [`run_attached_service`]'s identical guard: a later failure (the readiness -/// wait, `record.write()`) leaves a record a live child actually owns, so it -/// must not also wipe that key. -#[allow(clippy::too_many_arguments)] -pub(crate) fn start_managed_service( - paths: &AppPaths, - engine: &str, - service_id: &str, - requested_model: &str, - resolve: &ResolveModelResponse, - host: &str, - port_request: serve_port::PortRequest, - device_policy: &DevicePolicy, - gpu_indices: &[u32], - runtime_id: Option<&str>, - env_id: Option<&str>, - engine_recipe: Option<&EngineRecipeHint>, - endpoint_api_key: Option<&str>, - launch_lock: rocm_core::FileLock, - require_api_key: bool, - gpu_refusal_waived_for_reuse: bool, - wait_until_ready: bool, - on_wait_begin: &mut dyn FnMut(&str, u16), - on_wait_tick: &mut dyn FnMut(Duration), -) -> Result { - #[cfg(feature = "e2e-oom-fault-injection")] - if e2e_simulate_oom_launch() { - return simulate_oom_managed_launch( - paths, - engine, - service_id, - requested_model, - resolve, - host, - port_request - .explicit() - .unwrap_or(rocm_core::DEFAULT_LOCAL_PORT), - device_policy, - gpu_indices, - runtime_id, - env_id, - ); - } - let (mut record, child_pid) = match spawn_managed_engine_child( - paths, - engine, - service_id, - requested_model, - resolve, - host, - port_request, - device_policy, - gpu_indices, - runtime_id, - env_id, - engine_recipe, - require_api_key, - gpu_refusal_waived_for_reuse, - ) - .inspect_err(|_| { - drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); - })? { - ManagedSpawn::AlreadyRunning(report) => { - // `--wait` applies to a service that is still starting too: the - // natural retry after `rocm serve m` printed `starting`. - drop(launch_lock); - return wait_on_existing_service( - paths, - report, - wait_until_ready, - on_wait_begin, - on_wait_tick, - ); - } - ManagedSpawn::Spawned { record, child_pid } => (*record, child_pid), - }; - // The transaction resolved the request to one concrete port; everything - // below — readiness, the endpoint URL, the audit line — uses that. - let port = record.port; - // The claiming service record is now persisted, so the selected GPU is - // visible to any concurrent auto-selection. Release the launch lock before - // the readiness wait below, which can block for many seconds — holding it - // that long would needlessly serialize unrelated serves. - drop(launch_lock); - - #[cfg(windows)] - thread::sleep(Duration::from_millis(200)); - - if wait_until_ready { - on_wait_begin(service_id, port); - } - let readiness = wait_for_record_ready( - paths, - &record, - wait_until_ready, - endpoint_api_key, - on_wait_tick, - ); - if let Some(error) = died_during_wait( - paths, - service_id, - wait_until_ready, - readiness, - service_pid_is_alive, - ) { - return Err(error); - } - let launch_status = status_for_readiness(readiness); - record_wait_outcome( - paths, - &mut record, - launch_status, - readiness == EndpointReadiness::Serving, - )?; - let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); - record_cli_audit_event( - paths, - "service", - "managed_service_launch", - "info", - format!( - "launched managed service engine={} model={} endpoint={} readiness={}", - engine, resolve.canonical_model_id, endpoint_url, launch_status - ), - Some(service_id), - ); - Ok(ManagedLaunchReport { - service_id: service_id.to_owned(), - endpoint_url, - status: launch_status.to_owned(), - already_running: false, - child_pid: Some(child_pid), - log_path: Some(record.log_path), - manifest_path: Some(record.manifest_path), - }) -} - -/// Reproduce the original plain, line-by-line managed-launch output. Used for -/// non-interactive output (piped, CI, the chat assistant's `serve --managed`), -/// where the animated summary is inappropriate. The interactive path renders the -/// summary table via [`crate::serve_summary`] instead. -pub(crate) fn print_managed_launch_plain( - report: &ManagedLaunchReport, - endpoint_api_key: Option<&str>, -) { - if report.already_running { - println!("managed service already running"); - println!(" service_id: {}", report.service_id); - println!(" endpoint: {}", report.endpoint_url); - println!(" status: {}", report.status); - println!(" note: existing service detected; no second process spawned"); - return; - } - println!("managed service launched"); - println!(" service_id: {}", report.service_id); - if let Some(child_pid) = report.child_pid { - println!(" process_pid: {child_pid}"); - } - println!(" endpoint: {}", report.endpoint_url); - if let Some(key) = endpoint_api_key { - // Intentional one-time display of a freshly generated API key to the - // terminal so the user can copy it — the designed delivery channel - // documented on `render_endpoint_client_config`, not a log. The tag below - // is currently inert (Rust's CodeQL pack has no AlertSuppression.ql yet — - // github/codeql#21637) but will start working once that lands, since the - // tag must be the single line immediately before the flagged code. - print!( - // codeql[rust/cleartext-logging] - "{}", - render_endpoint_client_config(&report.endpoint_url, key) - ); - } - if let Some(log_path) = report.log_path.as_deref() { - println!(" log_path: {}", log_path.display()); - } - if let Some(manifest_path) = report.manifest_path.as_deref() { - println!(" manifest_path: {}", manifest_path.display()); - } - println!(" readiness: {}", report.status); -} - -/// The client configuration to show when a `--wait` begins, so a key generated for -/// a public bind reaches the user even if the wait is interrupted or fails. -/// `None` unless the wait is on the launch that key belongs to: an attach to an -/// already-starting server (`waiting_on` differs from `service_id`) answers to -/// its own key, which is not shown here. -pub(crate) fn wait_begin_client_config( - waiting_on: &str, - service_id: &str, - host: &str, - port: u16, - api_key: Option<&str>, -) -> Option { - let key = api_key.filter(|_| waiting_on == service_id)?; - let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); - Some(render_endpoint_client_config(&endpoint_url, key)) -} - -/// Render the one-time secure client configuration for a public, authenticated -/// endpoint. This is the *intended* channel for delivering the key to the user -/// (unlike logs/status, which must never contain it) — it prints the key once at -/// launch alongside a ready-to-use example. Callers only invoke this for a -/// non-loopback bind that generated/received a key. -fn render_endpoint_client_config(endpoint_url: &str, api_key: &str) -> String { - let mut out = String::new(); - let _ = writeln!(out, " api key: {api_key}"); - let _ = writeln!( - out, - " note: this key is shown only now - clients must send `Authorization: Bearer `" - ); - let _ = writeln!( - out, - " example: curl -H \"Authorization: Bearer {api_key}\" {endpoint_url}/models" - ); - out -} - -/// Shared daemon-lifecycle entrypoint: ensures the background automation helper -/// (`rocm daemon`) is running when an automation watcher needs it, spawning it -/// detached if not. Liveness is read from the file-based automation runtime -/// state. Intentionally `pub(crate)` — reused by both the `serve --managed` path -/// and `automations enable`. Only the spawn result itself (`command.spawn()` / -/// `spawn_detached_no_inherit`) is logged rather than propagated; setup errors -/// (path discovery, config load, stdio attach) still return `Err`. -/// -/// The daemon's only work is running enabled automation watchers (server -/// recovery included — `server-recover` is one of them and is off by default), -/// and it exits on its own once none is enabled. So with no watcher enabled -/// nothing is spawned: a `serve` would otherwise leave a no-op daemon running -/// long after its service was stopped (ROCMAI-827). -pub(crate) fn ensure_background_helper_running() -> Result<()> { - ensure_background_helper_running_quiet(false) -} - -/// As [`ensure_background_helper_running`], but suppresses the stdout status line -/// when `quiet` is set. The interactive `rocm serve` summary path uses `quiet` so -/// the daemon-spawn note does not appear above the deployment summary table. -pub(crate) fn ensure_background_helper_running_quiet(quiet: bool) -> Result<()> { - let paths = AppPaths::discover()?; - ensure_background_helper_running_with(&paths, quiet, spawn_background_helper, &|| {}) -} - -/// The body of [`ensure_background_helper_running_quiet`], with the actual -/// process spawn injected so the decision around it is testable without -/// launching a daemon. `spawn` returns `Err` for a setup failure (propagated) -/// and `Ok(Err(..))` for a failed spawn (reported, not fatal). `on_lock_busy` is -/// called each time the autostart lock is found held by another process, which -/// lets a test wait for this call to be genuinely queued behind it. -fn ensure_background_helper_running_with( - paths: &AppPaths, - quiet: bool, - spawn: impl FnOnce(&AppPaths) -> Result>, - on_lock_busy: &dyn Fn(), -) -> Result<()> { - // An unreadable config is "no watcher": this also runs after `serve` has - // already launched its child, where failing would report a launch that - // happened as an error. - if !rocm_core::RocmCliConfig::load(paths).is_ok_and(|config| config.automation_daemon_enabled()) - { - return Ok(()); - } - - // Two concurrent callers (e.g. two `rocm serve`) can both read "not running" - // and each spawn a daemon. Serialize the decision on a lock file — the first - // holder spawns, later holders observe the now-running daemon and return - // without spawning. - // - // The "already running" read is deliberately made only under the lock, with - // no lock-free fast path. The daemon exits by itself once the last watcher - // is disabled, and it decides that under this same lock and publishes - // `running = false` before releasing it. Caller order is: save the enabling - // config, then come here. A lock-free read could still see the daemon that is - // about to exit as running, spawn nothing, and leave a watcher enabled with - // no daemon; under the lock the daemon has either seen the new config and - // stays, or has already published that it stopped. - let _autostart_lock = loop { - if let Some(lock) = - rocm_core::FileLock::try_acquire(paths.automation_autostart_lock_path())? - { - break lock; - } - on_lock_busy(); - std::thread::sleep(std::time::Duration::from_millis(10)); - }; - if background_helper_already_running(paths)? { - return Ok(()); - } - - // The lock alone does not close the window: the spawned daemon does not - // publish its `running` runtime state until well after `spawn()` (clap parse, - // runtime build, config load, banner flush). A second caller that acquires - // this lock during that gap still sees "not running" and would spawn a - // duplicate. Bridge the gap with a short-lived claim recording the child PID - // and spawn time: a holder that finds a live, recent claim defers instead. - let claim_path = paths.automation_autostart_claim_path(); - if autostart_spawn_in_flight( - read_autostart_claim(&claim_path), - now_unix_millis(), - AUTOSTART_CLAIM_TTL_MS, - rocm_core::process_is_running, - ) { - return Ok(()); - } - - match spawn(paths)? { - Ok(daemon_pid) => { - // Record the claim before returning (and thus releasing the lock) so a - // concurrent holder in the spawn→publish window defers. Best-effort: a - // failed write only reopens the original, already-tolerated race. - let _ = write_autostart_claim( - &claim_path, - AutostartClaim { - daemon_pid, - spawned_at_ms: now_unix_millis(), - }, - ); - if !quiet { - println!(" helper: started background automation daemon"); - } - } - Err(error) if !quiet => { - println!(" helper: could not start background automation daemon: {error}"); - } - Err(_) => {} - } - Ok(()) -} - -/// Spawn `rocm daemon` detached from this process. The outer `Result` is a setup -/// failure (resolving the executable, attaching stdio); the inner one is the -/// spawn itself. -fn spawn_background_helper(paths: &AppPaths) -> Result> { - let exe = managed_service_launcher_path() - .context("failed to resolve current rocm executable path")?; - let args = vec!["daemon".to_owned()]; - #[cfg(windows)] - let spawn_result = { - let env_values = app_path_env_var_values(paths, None); - let env_refs = app_path_env_var_refs(&env_values); - rocm_core::spawn_detached_no_inherit(&exe, &args, &env_refs) - }; - #[cfg(not(windows))] - let spawn_result = { - let mut command = managed_service_process_command(&exe, &args); - command.stdin(Stdio::null()); - attach_background_stdio(&mut command, None)?; - detach_background_command(&mut command); - apply_app_path_env(&mut command, paths); - command - .spawn() - .map(|child| child.id()) - .map_err(anyhow::Error::from) - }; - Ok(spawn_result) -} - -/// How long an autostart claim is honoured before it is treated as stale even if -/// its recorded PID is still alive. Comfortably longer than a cold daemon boot -/// (clap parse → runtime build → config load → state publish) yet short enough -/// that a crashed spawn cannot suppress autostart for long. -const AUTOSTART_CLAIM_TTL_MS: u128 = 30_000; - -/// A just-spawned daemon's autostart claim: the child PID and the wall-clock time -/// (milliseconds since the Unix epoch) the spawn was recorded. It lets a -/// concurrent autostart holder distinguish a live, in-flight spawn from a stale -/// leftover. Serialized as a single `" "` line — no dependency and -/// trivially forward-compatible. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -struct AutostartClaim { - daemon_pid: u32, - spawned_at_ms: u128, -} - -/// Milliseconds since the Unix epoch, or `0` if the clock is before the epoch -/// (which only makes a fresh claim look old — safe, it just permits a respawn). -fn now_unix_millis() -> u128 { - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0, |elapsed| elapsed.as_millis()) -} - -/// Read an autostart claim, returning `None` when the file is absent or -/// unparseable (either is treated as "no claim", so a respawn is permitted). -fn read_autostart_claim(path: &Path) -> Option { - let text = std::fs::read_to_string(path).ok()?; - let mut parts = text.split_whitespace(); - let daemon_pid = parts.next()?.parse().ok()?; - let spawned_at_ms = parts.next()?.parse().ok()?; - Some(AutostartClaim { - daemon_pid, - spawned_at_ms, - }) -} - -/// Write an autostart claim as `" "`. Best-effort at the call site. -fn write_autostart_claim(path: &Path, claim: AutostartClaim) -> std::io::Result<()> { - std::fs::write( - path, - format!("{} {}", claim.daemon_pid, claim.spawned_at_ms), - ) -} - -/// Whether an existing autostart `claim` means a daemon spawn is still in flight, -/// so the current lock holder should defer rather than spawn a duplicate. A claim -/// counts as in-flight only while its child PID is alive *and* it is younger than -/// `ttl_ms` — the TTL bounds how long a crashed spawn (or a PID later reused by an -/// unrelated process) can suppress autostart. `pid_alive` is injected so the -/// decision is unit-testable without a live process. -fn autostart_spawn_in_flight( - claim: Option, - now_ms: u128, - ttl_ms: u128, - pid_alive: impl Fn(u32) -> bool, -) -> bool { - claim.is_some_and(|claim| { - now_ms.saturating_sub(claim.spawned_at_ms) < ttl_ms && pid_alive(claim.daemon_pid) - }) -} - -/// What ended an attached (`--verbose`/`--foreground`) streaming session. Kept -/// as a plain enum, separate from any terminal I/O, so the follow-up action -/// (detach note vs. stop the server) is unit-testable without a TTY. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum AttachOutcome { - /// Ctrl-D: leave the server running and hand the terminal back. - Detach, - /// Ctrl-C: stop the server, then hand the terminal back. - Stop, - /// The engine process exited on its own while we were streaming its log. - ServerExited, -} - -/// Attached serve path for `--verbose`/`--foreground`: spawn the engine as a -/// detached managed child (the same child the background path spawns) and stream -/// its log in this terminal. Unlike the old in-process foreground, the server -/// survives the session — Ctrl-D detaches and leaves it running, Ctrl-C stops it. -/// -/// `stop_service` is how Ctrl-C stops the server: the caller passes the managed -/// stop, so this module does not import the stop path, which itself imports this -/// module's spawn helpers. It returns the same `stop_server` answer -/// [`finish_attached_stop`] reads. -#[allow(clippy::too_many_arguments)] -pub(crate) fn run_attached_service( - paths: &AppPaths, - engine: &str, - service_id: &str, - requested_model: &str, - resolve: &ResolveModelResponse, - host: &str, - port_request: serve_port::PortRequest, - gpu_indices: &[u32], - runtime_id: Option<&str>, - env_id: Option<&str>, - endpoint_api_key: Option<&str>, - launch_lock: rocm_core::FileLock, - require_api_key: bool, - gpu_refusal_waived_for_reuse: bool, - stop_service: impl FnOnce(&AppPaths, &str) -> Result, -) -> Result<()> { - let spawn = spawn_managed_engine_child( - paths, - engine, - service_id, - requested_model, - resolve, - host, - port_request, - &resolve.device_policy, - gpu_indices, - runtime_id, - env_id, - resolve.engine_recipe.as_ref(), - require_api_key, - gpu_refusal_waived_for_reuse, - ) - // Mirrors `start_managed_service`'s cleanup: a pre-spawn - // failure (port reserved/in use, range exhausted, other bind error) returns - // before any service record exists, so nothing else clears the key just - // stored for this attempt. - .inspect_err(|_| { - drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); - })?; - // The claiming record is persisted (or an existing service was found), so the - // selected GPU is now visible to concurrent auto-selection. Release the launch - // lock before streaming logs, which blocks for the whole attached session. - drop(launch_lock); - - // Attached mode is a managed record too, so it shares the transaction's - // resolved concrete port. - let (service_id, log_path, port, child_pid) = match spawn { - // A server for this engine+model is already live. Don't fight it for the - // port — point the user at the existing one instead of tailing a log we - // did not start. - ManagedSpawn::AlreadyRunning(report) => { - println!("model already being served"); - println!(" service_id: {}", report.service_id); - println!(" endpoint: {}", report.endpoint_url); - println!(" status: {}", report.status); - println!(" logs: rocm logs {}", report.service_id); - println!( - " stop: {}", - service_hints::service_stop_hint(&report.service_id) - ); - drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); - return Ok(()); - } - ManagedSpawn::Spawned { record, child_pid } => ( - service_id.to_owned(), - record.log_path.clone(), - record.port, - child_pid, - ), - }; - - // The child is a managed service that outlives this session once detached. - // The background helper adds one thing for it: with the `server-recover` - // watcher enabled, it health-checks managed servers and restarts failed ones. - // Every duty of the helper is watcher-gated, so with no watcher enabled no - // helper is started. Nothing else depends on it: a self-exited server's - // record is settled by its own supervisor (`supervise_service` writes the - // exit via `record_supervised_exit` in `apps/rocmd/src/supervise.rs`) and on - // every read (`refresh_managed_service_runtime_liveness`), and the dashboard - // has its own embedded daemon. Match the background ordering (spawn, then - // ensure the helper) and keep it quiet so no status line breaks into the - // log stream. - ensure_background_helper_running_quiet(true)?; - - // The resolution detail (model, engine, runtime, GPU, warnings) was already - // printed as the "serve plan" block in `serve()`; extend it with the launch - // coordinates and the streaming hint rather than repeating it. - let endpoint = format!("{}/v1", format_http_base_url(host, port)); - println!(" service_id: {service_id}"); - println!(" endpoint: {endpoint}"); - if let Some(key) = endpoint_api_key { - // Same intentional one-time key display as `print_managed_launch_plain` - // above; see its rationale for why the tag below is currently inert. - // codeql[rust/cleartext-logging] - print!("{}", render_endpoint_client_config(&endpoint, key)); - } - println!(" streaming engine logs - Ctrl-D detaches (leaves it running), Ctrl-C stops it"); - println!(); - - let outcome = stream_attached_logs(&log_path, child_pid)?; - println!(); - - match outcome { - AttachOutcome::Detach => { - // Audit before printing: a closed stdout makes `println!` panic, - // which must not lose the event. - record_cli_audit_event( - paths, - "service", - "serve_detach", - "info", - format!("detached from attached serve service_id={service_id} endpoint={endpoint}"), - Some(&service_id), - ); - println!("detached - server still running"); - println!(" service_id: {service_id}"); - println!(" endpoint: {endpoint}"); - println!(" list: rocm services"); - println!(" logs: rocm logs {service_id}"); - println!(" stop: {}", service_hints::service_stop_hint(&service_id)); - Ok(()) - } - AttachOutcome::Stop => { - println!("stopping server..."); - let stop = stop_service(paths, &service_id); - finish_attached_stop( - paths, - &service_id, - child_pid, - stop, - rocm_core::terminate_process_tree, - &mut io::stdout().lock(), - &mut io::stderr().lock(), - ) - } - AttachOutcome::ServerExited => { - println!("server process exited"); - println!(" service_id: {service_id}"); - println!(" recent logs: rocm logs {service_id}"); - Ok(()) - } - } -} - -/// Restores cooked terminal mode when dropped, so [`stream_attached_logs`] leaves -/// the terminal usable on every exit path (normal return, `?` error, or panic). -struct RawModeGuard; - -impl Drop for RawModeGuard { - fn drop(&mut self) { - let _ = crossterm::terminal::disable_raw_mode(); - } -} - -/// Map a key press (control modifier + lowercased character) to the attach -/// action it triggers, if any. Factored out of the raw-mode reader loop so the -/// Ctrl-D/Ctrl-C mapping is unit-testable without a terminal. -const fn detach_key_outcome(ctrl: bool, ch: char) -> Option { - if !ctrl { - return None; - } - match ch { - 'c' => Some(AttachOutcome::Stop), - 'd' => Some(AttachOutcome::Detach), - _ => None, - } -} - -/// Follow `log_path` in the terminal until the user presses Ctrl-D (detach) or -/// Ctrl-C (stop), or the engine process exits. Uses crossterm raw mode to -/// capture the keys directly (in raw mode Ctrl-C does not raise SIGINT, so we see -/// it as a key event). When stdin is not a TTY (piped/CI), keystroke capture is -/// impossible, so we follow the log until the process exits instead. -fn stream_attached_logs(log_path: &Path, child_pid: u32) -> Result { - use std::io::IsTerminal as _; - use std::sync::Arc; - use std::sync::atomic::{AtomicBool, Ordering}; - use std::sync::mpsc; - - if !std::io::stdin().is_terminal() { - return stream_attached_logs_no_tty(log_path, child_pid); - } - - // Enter raw mode *before* spawning the key reader. In raw mode Ctrl-C arrives - // as a key event instead of SIGINT; if the reader started first, a Ctrl-C in - // that window would kill the CLI outright (leaving the detached child alive - // but printing no detach/stop message). The guard restores cooked mode on - // every exit path (normal return, `?` error, panic). - crossterm::terminal::enable_raw_mode().context("failed to enter raw terminal mode")?; - let _raw_guard = RawModeGuard; - - let stop = Arc::new(AtomicBool::new(false)); - let (tx, rx) = mpsc::channel::(); - - let reader_stop = Arc::clone(&stop); - let reader = thread::spawn(move || { - use crossterm::event::{self, Event, KeyCode, KeyEventKind, KeyModifiers}; - while !reader_stop.load(Ordering::Relaxed) { - match event::poll(Duration::from_millis(100)) { - Ok(true) => match event::read() { - Ok(Event::Key(key)) if key.kind != KeyEventKind::Release => { - let outcome = match key.code { - KeyCode::Char(ch) => detach_key_outcome( - key.modifiers.contains(KeyModifiers::CONTROL), - ch.to_ascii_lowercase(), - ), - _ => None, - }; - if let Some(outcome) = outcome { - let _ = tx.send(outcome); - break; - } - } - Ok(_) => {} - Err(_) => break, - }, - Ok(false) => {} - Err(_) => break, - } - } - }); - - let mut stdout = io::stdout(); - let mut log_reader: Option> = None; - let mut line = String::new(); - let outcome = loop { - if log_reader.is_none() { - log_reader = fs::File::open(log_path).ok().map(io::BufReader::new); - } - if let Some(reader) = log_reader.as_mut() { - loop { - line.clear(); - match reader.read_line(&mut line) { - Ok(0) => break, - Ok(_) => { - // Raw mode disables the terminal's own \n -> \r\n - // translation, so emit an explicit carriage return to - // keep the log left-aligned instead of stair-stepping. - let _ = write!(stdout, "{}\r\n", line.trim_end_matches('\n')); - let _ = stdout.flush(); - } - Err(_) => break, - } - } - } - - if let Ok(signal) = rx.try_recv() { - break signal; - } - if !process_is_running(child_pid) { - break AttachOutcome::ServerExited; - } - thread::sleep(Duration::from_millis(150)); - }; - - stop.store(true, Ordering::Relaxed); - let _ = reader.join(); - Ok(outcome) -} - -/// Non-interactive fallback for [`stream_attached_logs`]: no keystroke capture, -/// so just follow the log until the (detached) engine process exits. A Ctrl-C -/// here delivers SIGINT to this process and leaves the managed server running. -fn stream_attached_logs_no_tty(log_path: &Path, child_pid: u32) -> Result { - let mut stdout = io::stdout(); - let mut log_reader: Option> = None; - let mut line = String::new(); - loop { - if log_reader.is_none() { - log_reader = fs::File::open(log_path).ok().map(io::BufReader::new); - } - if let Some(reader) = log_reader.as_mut() { - loop { - line.clear(); - match reader.read_line(&mut line) { - Ok(0) => break, - Ok(_) => { - let _ = write!(stdout, "{line}"); - let _ = stdout.flush(); - } - Err(_) => break, - } - } - } - if !process_is_running(child_pid) { - return Ok(AttachOutcome::ServerExited); - } - thread::sleep(Duration::from_millis(200)); - } -} - -/// Finish the attached `rocm serve` Ctrl-C stop once the managed stop has -/// answered: what is printed where, what is signalled, what is audited, and -/// how the command exits. Everything but the stop itself and the real -/// signal is here, so a test can drive it. -/// -/// - Confirmed: the report, with the background-helper note `rocm services -/// stop` also prints, goes to `out` and the command succeeds. -/// - Unconfirmed: the same report `rocm services stop` gives, check lines -/// included, becomes the error, so it reaches stderr and the command exits -/// non-zero. Nothing is signalled on the stop's behalf: the stop has already -/// done what it safely could. -/// - The stop itself failed: `signal_child` gets the child this session -/// launched, so Ctrl-C never leaves the model orphaned when the stop path -/// breaks. The stop's error, and the signal's if that fails too, are noted on -/// `err`, and the audit event records whether the signal worked. The command -/// still exits 0, as it always has here. -fn finish_attached_stop( - paths: &AppPaths, - service_id: &str, - child_pid: u32, - stop: Result, - signal_child: impl FnOnce(u32) -> Result<()>, - out: &mut dyn Write, - err: &mut dyn Write, -) -> Result<()> { - match stop { - Ok(result) if service_stop_unconfirmed(&result) => { - record_cli_audit_event( - paths, - "service", - "serve_stop", - "error", - format!("stop of attached serve not confirmed service_id={service_id}"), - Some(service_id), - ); - bail!( - "{}", - render_service_action_result("stop_server", &result).trim_end() - ) - } - Ok(result) => { - record_cli_audit_event( - paths, - "service", - "serve_stop", - "info", - format!("stopped attached serve service_id={service_id}"), - Some(service_id), - ); - write!( - out, - "{}", - render_service_action_result_with_helper_note(paths, "stop_server", &result) - )?; - Ok(()) - } - Err(error) => { - let signal = signal_child(child_pid); - // Record the audit event before writing to stderr, so a failed - // write cannot skip it. - let message = match &signal { - Ok(()) => format!( - "stop failed; signaled attached serve child {child_pid} service_id={service_id}" - ), - Err(signal_error) => format!( - "stop failed; signaling attached serve child {child_pid} also failed: {signal_error:#} service_id={service_id}" - ), - }; - record_cli_audit_event( - paths, - "service", - "serve_stop", - if signal.is_ok() { "warn" } else { "error" }, - message, - Some(service_id), - ); - writeln!(err, " note: {error:#}")?; - if let Err(signal_error) = &signal { - writeln!( - err, - " note: signaling this session's server process {child_pid} directly also failed: {signal_error:#}" - )?; - } - Ok(()) - } - } -} - -#[cfg(test)] -mod tests { - #[test] - fn the_no_gpu_refusal_sends_a_bare_metal_user_to_dev_kfd() { - let text = super::no_usable_gpu_refusal_text("gpu_required", false); - assert!(text.contains("confirm /dev/kfd is present"), "{text}"); - assert!(!text.contains("ROCDXG"), "{text}"); - } - - #[test] - fn the_no_gpu_refusal_does_not_send_a_wsl_user_to_dev_kfd() { - // WSL has no /dev/kfd, so advice to confirm it can never clear the - // condition. The text names the status `examine` reports for the path - // the probe asked about, taken from the constant the producer uses, so - // the two cannot drift. That the status really is the one a refused - // host shows is asserted end to end by serve-42. - let text = super::no_usable_gpu_refusal_text("gpu_required", true); - assert!(!text.contains("/dev/kfd is present"), "{text}"); - assert!(text.contains("rocm examine"), "{text}"); - assert!(text.contains("rocm diagnose"), "{text}"); - // The advice is true only because `rocdxg_ready` counts a library on - // LD_LIBRARY_PATH; `a_rocdxg_only_on_the_loader_path_is_ready` in - // `rocm-core`'s host_gpu/wsl.rs proves that behavior. - assert!(text.contains("LD_LIBRARY_PATH"), "{text}"); - assert!( - text.contains(&format!("`{}`", rocm_core::WSL_ROCDXG_READY)), - "{text}" - ); - } - - #[test] - fn the_no_gpu_refusal_names_the_policy_on_both_platforms() { - for on_wsl in [false, true] { - let text = super::no_usable_gpu_refusal_text("gpu_required", on_wsl); - assert!( - text.starts_with("no usable AMD GPU detected; `rocm serve` requires a GPU under the gpu_required policy"), - "{text}" - ); - } - } - use super::*; - use crate::test_support::{runtime_state, set_helper_facts}; - - #[test] - fn a_child_that_exited_during_the_startup_window_is_reported_with_its_log() -> Result<()> { - let dir = tempfile::tempdir()?; - let log = dir.path().join("engine.log"); - fs::write(&log, "boom: engine could not start\n")?; - let settled = std::cell::Cell::new(false); - let error = ensure_spawned_child_survives_startup( - 4242, - &log, - || settled.set(true), - |pid| { - assert_eq!(pid, 4242); - assert!(settled.get(), "liveness must be probed after the wait"); - false - }, - ) - .expect_err("a dead child must fail the launch"); - assert!(settled.get(), "the startup window must elapse first"); - assert!(format!("{error:#}").contains("boom"), "{error:#}"); - ensure_spawned_child_survives_startup(4242, &log, || {}, |_| true)?; - Ok(()) - } - use crate::tests::test_paths; - use rocm_core::DEFAULT_LOCAL_HOST; - - #[test] - fn wait_past_readiness_budget_keeps_probing_a_live_starting_server() { - let mut readings = vec![ - EndpointReadiness::Unreachable, - EndpointReadiness::Unreachable, - EndpointReadiness::Serving, - ] - .into_iter(); - let mut probes = 0; - let result = wait_past_readiness_budget( - || { - probes += 1; - readings.next().expect("no probe after Serving") - }, - || true, - ); - assert_eq!(result, EndpointReadiness::Serving); - assert_eq!(probes, 3, "must outlast the first (45s) budget"); - } - - #[test] - fn wait_past_readiness_budget_stops_when_the_server_exits() { - let mut probes = 0; - let result = wait_past_readiness_budget( - || { - probes += 1; - EndpointReadiness::Unreachable - }, - || false, - ); - assert_eq!(result, EndpointReadiness::Unreachable); - assert_eq!(probes, 1); - } - - #[test] - fn wait_past_readiness_budget_returns_at_once_when_already_serving() { - let result = wait_past_readiness_budget( - || EndpointReadiness::Serving, - || panic!("liveness is irrelevant once serving"), - ); - assert_eq!(result, EndpointReadiness::Serving); - } - - fn record_tracking_pid(paths: &AppPaths, pid: u32, status: &str) -> ManagedServiceRecord { - let mut record = reusable_record(paths); - record.supervisor_pid = pid; - record.status = status.to_owned(); - record - } - - #[cfg(unix)] - fn exited_child_pid() -> u32 { - let mut child = std::process::Command::new("true").spawn().expect("spawn"); - let pid = child.id(); - child.wait().expect("reap"); - pid - } - - #[test] - #[cfg(unix)] - fn record_still_live_ends_the_wait_for_a_dead_pid_with_a_live_status() { - // The engine was killed before writing any terminal state: the status - // still says `starting`, but nothing tracked is running. - let (root, paths) = test_paths("wait-dead-pid"); - paths.ensure().expect("paths"); - let dead = record_tracking_pid(&paths, exited_child_pid(), "starting"); - assert!(!record_still_live(&dead, service_pid_is_alive)); - - let alive = record_tracking_pid(&paths, std::process::id(), "starting"); - assert!(record_still_live(&alive, service_pid_is_alive)); - - // A settled status ends the wait even while the pid is still around. - let stopped = record_tracking_pid(&paths, std::process::id(), "stopped"); - assert!(!record_still_live(&stopped, service_pid_is_alive)); - let _ = fs::remove_dir_all(root); - } - - #[test] - #[cfg(target_os = "linux")] - fn service_pid_is_alive_treats_an_unreaped_child_as_dead() { - // The serve supervisor is our own unreaped child; once it dies it is a - // zombie that `kill(pid, 0)` still reports as running. - let mut child = std::process::Command::new("true").spawn().expect("spawn"); - let pid = child.id(); - let deadline = std::time::Instant::now() + Duration::from_secs(10); - while !process_is_zombie(pid) { - assert!(std::time::Instant::now() < deadline, "child never exited"); - thread::sleep(Duration::from_millis(10)); - } - assert!( - process_is_running(pid), - "a zombie still answers kill(pid, 0)" - ); - assert!(!service_pid_is_alive(pid)); - child.wait().expect("reap"); - } - - #[test] - fn service_still_live_follows_the_manifest_on_disk() { - // `rocm services stop` from another terminal writes only the manifest. - let (root, paths) = test_paths("wait-manifest"); - paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, std::process::id(), "starting"); - record.write().expect("write"); - assert!(service_still_live(&paths, &record.service_id, |_| true)); - - record.status = "stopped".to_owned(); - record.write().expect("write"); - assert!(!service_still_live(&paths, &record.service_id, |_| true)); - assert!(!service_still_live(&paths, "no-such-service", |_| true)); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn settle_exited_service_demotes_a_live_manifest_and_spares_a_settled_one() { - let (root, paths) = test_paths("wait-settle"); - paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, 0, "starting"); - record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); - let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); - assert_eq!(after.status, "stopped"); - - // A record that already failed keeps its reason: the guard, not the - // overwrite, is what spares it (a `stopped` -> `stopped` rewrite would - // be invisible). - record.status = "failed".to_owned(); - record.write().expect("write"); - settle_exited_service(&paths, &record.service_id); - let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); - assert_eq!(after.status, "failed"); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn wait_on_existing_service_only_waits_for_one_that_is_not_serving_yet() { - // The probing path itself blocks for the readiness budget, so only the - // cases that must return at once are driven here; a manifest planted - // as `starting` would hang a wrong guard for 45 seconds and fail it. - let (root, paths) = test_paths("wait-existing"); - paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, 0, "starting"); - record.write().expect("write"); - let report = |status: &str| ManagedLaunchReport { - service_id: record.service_id.clone(), - endpoint_url: record.endpoint_url.clone(), - status: status.to_owned(), - already_running: true, - child_pid: None, - log_path: None, - manifest_path: None, - }; - let mut no_tick = |_: Duration| panic!("must not probe"); - - let untouched = wait_on_existing_service( - &paths, - report("starting"), - false, - &mut |_, _| panic!("no --wait, no banner"), - &mut no_tick, - ) - .expect("no --wait"); - assert_eq!(untouched.status, "starting"); - let ready = wait_on_existing_service( - &paths, - report("ready"), - true, - &mut |_, _| panic!("already ready, no banner"), - &mut no_tick, - ) - .expect("already ready"); - assert_eq!(ready.status, "ready"); - record.status = "starting".to_owned(); - assert_eq!( - ManagedServiceRecord::read_manifest(&paths, &record.service_id) - .unwrap() - .status, - "starting" - ); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn died_during_wait_fails_with_a_logs_pointer_and_records_the_death() { - let (root, paths) = test_paths("wait-died"); - paths.ensure().expect("paths"); - let record = record_tracking_pid(&paths, std::process::id(), "starting"); - record.write().expect("write"); - let id = record.service_id; - let status = || { - ManagedServiceRecord::read_manifest(&paths, &id) - .unwrap() - .status - }; - - // Not waiting, already serving, or still alive: no failure, no change. - let pending = EndpointReadiness::Unreachable; - assert!(died_during_wait(&paths, &id, false, pending, |_| false).is_none()); - assert!( - died_during_wait(&paths, &id, true, EndpointReadiness::Serving, |_| false).is_none() - ); - assert!(died_during_wait(&paths, &id, true, pending, |_| true).is_none()); - assert_eq!(status(), "starting"); - - // Dead server: the error names the logs command and the record is settled. - let error = died_during_wait(&paths, &id, true, pending, |_| false).expect("died"); - let message = error.to_string(); - assert!( - message.contains("exited before it became ready"), - "{message}" - ); - assert!( - message.contains(&format!("rocm logs --service {id}")), - "{message}" - ); - assert_eq!(status(), "stopped"); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn died_during_wait_adds_the_oom_guidance_for_a_vllm_launch_only() { - let (root, paths) = test_paths("wait-died-oom"); - paths.ensure().expect("paths"); - let mut record = record_tracking_pid(&paths, std::process::id(), "starting"); - record.engine = "vllm".to_owned(); - record.write().expect("write"); - fs::write( - &record.log_path, - "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB.\n", - ) - .expect("log"); - let id = record.service_id; - let pending = EndpointReadiness::Unreachable; - let message = died_during_wait(&paths, &id, true, pending, |_| false) - .expect("died") - .to_string(); - assert!(message.contains("--gpu-memory-utilization"), "{message}"); - - // Same log, another engine: that engine has no such flag to suggest. - let mut record = ManagedServiceRecord::read_manifest(&paths, &id).unwrap(); - record.engine = "lemonade".to_owned(); - "starting".clone_into(&mut record.status); - record.write().expect("write"); - let message = died_during_wait(&paths, &id, true, pending, |_| false) - .expect("died") - .to_string(); - assert!(!message.contains("--gpu-memory-utilization"), "{message}"); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn record_wait_outcome_keeps_what_changed_on_disk_during_the_wait() { - let (root, paths) = test_paths("wait-outcome"); - paths.ensure().expect("paths"); - let mut held = record_tracking_pid(&paths, 111, "starting"); - held.write().expect("write"); - // While the wait ran, `rocmd` recorded a restart and a new supervisor. - let mut elsewhere = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); - elsewhere.restart_count = 3; - elsewhere.supervisor_pid = 222; - elsewhere.write().expect("write"); - - record_wait_outcome(&paths, &mut held, "ready", true).expect("outcome"); - let after = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); - assert_eq!(after.status, "ready"); - assert!(after.inference_verified_at_unix_ms.is_some()); - assert_eq!(after.restart_count, 3, "the restart count was reverted"); - assert_eq!(after.supervisor_pid, 222, "the supervisor pid was reverted"); - - record_wait_outcome(&paths, &mut held, "starting", false).expect("outcome"); - let after = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); - assert_eq!(after.status, "starting"); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn wait_begin_shows_the_key_only_for_the_launch_it_belongs_to() { - let own = wait_begin_client_config("svc-new", "svc-new", "0.0.0.0", 8000, Some("k-secret")) - .expect("own launch shows its key"); - // No `{own}` in the failure messages: the text carries the dummy key, - // and formatting it into an assert is what CodeQL flags as logging it. - assert!(own.contains("api key: k-secret"), "key line missing"); - assert!( - own.contains("http://0.0.0.0:8000/v1/models"), - "models url missing" - ); - // Attached to an already-starting server: its key is not this one. - assert!( - wait_begin_client_config("svc-old", "svc-new", "0.0.0.0", 8000, Some("k-secret")) - .is_none() - ); - // Loopback binds have no key to show. - assert!(wait_begin_client_config("svc-new", "svc-new", "127.0.0.1", 8000, None).is_none()); - } - - #[test] - fn serve_bind_validation_requires_public_ack() { - validate_bind_host("127.0.0.1", false).unwrap(); - validate_bind_host("localhost", false).unwrap(); - validate_bind_host("::1", false).unwrap(); - let error = validate_bind_host("0.0.0.0", false).unwrap_err(); - assert!( - error.to_string().contains("--allow-public-bind"), - "{error:#}" - ); - validate_bind_host("0.0.0.0", true).unwrap(); - } - - /// A `stop_server` answer whose one recorded process ended with `outcome`. - fn stop_answer(confirmed: bool, outcome: &str) -> serde_json::Value { - crate::services_text::internal_stop_server_value(serde_json::json!({ - "service_id": "svc-attached", - "status": "ready", - "signaled_pids": if outcome == "unverified" { vec![] } else { vec![4242] }, - "pid_outcomes": [{ "pid": 4242, "role": "supervisor_and_engine", "outcome": outcome }], - // An attached serve on a public bind stores a key; an unconfirmed - // stop keeps it. - "endpoint_key_kept": !confirmed, - "stopped": confirmed, - })) - } - - /// Run [`finish_attached_stop`] for `stop`, returning its result, what it - /// wrote to stdout, the PID it signalled (if any) and the audit event. - fn finish( - name: &str, - stop: Result, - ) -> (Result<()>, String, Option, serde_json::Value) { - finish_signalling(name, stop, Ok(())) - } - - /// [`finish`], with the direct signal answering `signal_result`. - fn finish_signalling( - name: &str, - stop: Result, - signal_result: Result<()>, - ) -> (Result<()>, String, Option, serde_json::Value) { - let (result, out, _err, signalled, audit) = finish_capturing(name, stop, signal_result); - (result, out, signalled, audit) - } - - /// [`finish_signalling`], also returning what was written to stderr. - fn finish_capturing( - name: &str, - stop: Result, - signal_result: Result<()>, - ) -> (Result<()>, String, String, Option, serde_json::Value) { - let (root, paths) = test_paths(name); - paths.ensure().unwrap(); - let mut out = Vec::new(); - let mut err = Vec::new(); - let mut signalled = None; - let result = finish_attached_stop( - &paths, - "svc-attached", - 777, - stop, - |pid| { - signalled = Some(pid); - signal_result - }, - &mut out, - &mut err, - ); - let audit_text = fs::read_to_string(paths.audit_events_path()).expect("audit log"); - let audit = serde_json::from_str(audit_text.lines().last().expect("an audit event")) - .expect("audit json"); - let _ = fs::remove_dir_all(root); - ( - result, - String::from_utf8(out).unwrap(), - String::from_utf8(err).unwrap(), - signalled, - audit, - ) - } - - #[test] - fn attached_ctrl_c_reports_a_confirmed_stop_on_stdout() { - let (result, out, signalled, audit) = - finish("attached-confirmed", Ok(stop_answer(true, "graceful"))); - result.expect("a confirmed stop succeeds"); - assert!(out.contains("Local server stopped"), "{out}"); - assert_eq!(signalled, None); - assert_eq!(audit["level"], "info", "{audit}"); - assert_eq!( - audit["message"], - "stopped attached serve service_id=svc-attached" - ); - } - - #[test] - fn attached_ctrl_c_fails_an_unconfirmed_stop_like_services_stop_and_signals_nothing() { - // The same report `rocm services stop` gives, as the error (stderr, - // non-zero exit), naming no retry; nothing is signalled on the stop's - // behalf and nothing reaches stdout. - let answer = stop_answer(false, "timed_out"); - let (result, out, signalled, audit) = finish("attached-timed-out", Ok(answer.clone())); - let error = format!("{:#}", result.expect_err("an unconfirmed stop fails")); - assert_eq!( - error, - render_service_action_result("stop_server", &answer).trim_end(), - "the report is `rocm services stop`'s own" - ); - assert!( - error.contains( - "process 4242 (supervisor_and_engine) or a process under it did not exit" - ), - "{error}" - ); - assert!( - error.lines().any(|line| line - == " kept: the server record, which still names its processes, and its endpoint key, which the server may still be enforcing"), - "{error}" - ); - crate::services_text::tests::assert_no_retry_line(&error); - assert!(out.is_empty(), "nothing on stdout: {out}"); - assert_eq!(signalled, None, "an unconfirmed stop signals nothing more"); - assert_eq!(audit["level"], "error", "{audit}"); - assert_eq!( - audit["message"], - "stop of attached serve not confirmed service_id=svc-attached" - ); - } - - #[test] - fn attached_ctrl_c_names_a_check_for_an_unverified_process() { - let (result, out, signalled, audit) = - finish("attached-unverified", Ok(stop_answer(false, "unverified"))); - let error = format!("{:#}", result.expect_err("an unverified stop fails")); - assert!(error.contains("not signaled: process 4242"), "{error}"); - assert!(error.contains("check: `ps -p 4242 -o pid,args`"), "{error}"); - crate::services_text::tests::assert_no_retry_line(&error); - assert!(out.is_empty(), "{out}"); - assert_eq!(signalled, None); - assert_eq!(audit["level"], "error", "{audit}"); - } - - #[test] - fn attached_ctrl_c_signals_its_own_child_when_the_stop_itself_fails() { - let (result, out, err, signalled, audit) = finish_capturing( - "attached-stop-error", - Err(anyhow::anyhow!("managed service `svc-attached` not found") - .context("failed to stop svc-attached")), - Ok(()), - ); - result.expect("a failed stop falls back and still exits cleanly, as before"); - assert_eq!(signalled, Some(777)); - assert_eq!( - err, " note: failed to stop svc-attached: managed service `svc-attached` not found\n", - "the stop's error goes to stderr, and a signal that worked adds nothing" - ); - assert!(out.is_empty(), "nothing on stdout: {out}"); - assert_eq!(audit["level"], "warn", "{audit}"); - assert_eq!( - audit["message"], - "stop failed; signaled attached serve child 777 service_id=svc-attached" - ); - - // The direct signal failing too is printed on stderr and recorded, with - // its error. - let (result, _, err, signalled, audit) = finish_capturing( - "attached-stop-error-signal-fails", - Err(anyhow::anyhow!("managed service `svc-attached` not found")), - Err(anyhow::anyhow!("no such process")), - ); - result.expect("still exits 0, as on main"); - assert_eq!( - err, - " note: managed service `svc-attached` not found\n \ - note: signaling this session's server process 777 directly also failed: no such process\n", - "both notes go to stderr" - ); - assert_eq!(signalled, Some(777)); - assert_eq!(audit["level"], "error", "{audit}"); - assert_eq!( - audit["message"], - "stop failed; signaling attached serve child 777 also failed: no such process service_id=svc-attached" - ); - } - - /// A writer whose every write fails, standing in for a closed stdout/stderr. - struct BrokenWriter; - - impl Write for BrokenWriter { - fn write(&mut self, _: &[u8]) -> std::io::Result { - Err(std::io::Error::from(std::io::ErrorKind::BrokenPipe)) - } - fn flush(&mut self) -> std::io::Result<()> { - Ok(()) - } - } - - #[test] - fn attached_ctrl_c_records_the_audit_event_even_when_writing_the_outcome_fails() { - // One case per branch that writes after deciding: a confirmed stop - // (stdout) and a stop that itself failed (stderr notes). The write - // error is returned, but the audit event is already on disk. - let cases: [(&str, Result, &str); 2] = [ - ( - "attached-broken-out", - Ok(stop_answer(true, "graceful")), - "stopped attached serve service_id=svc-attached", - ), - ( - "attached-broken-err", - Err(anyhow::anyhow!("managed service `svc-attached` not found")), - "stop failed; signaled attached serve child 777 service_id=svc-attached", - ), - ]; - for (name, stop, expected_message) in cases { - let (root, paths) = test_paths(name); - paths.ensure().unwrap(); - let result = finish_attached_stop( - &paths, - "svc-attached", - 777, - stop, - |_| Ok(()), - &mut BrokenWriter, - &mut BrokenWriter, - ); - let audit_text = fs::read_to_string(paths.audit_events_path()).unwrap_or_default(); - let _ = fs::remove_dir_all(root); - assert!(result.is_err(), "{name}: the write error is surfaced"); - let audit: serde_json::Value = serde_json::from_str( - audit_text - .lines() - .last() - .unwrap_or_else(|| panic!("{name}: no audit event recorded")), - ) - .expect("audit json"); - assert_eq!(audit["message"], expected_message, "{name}: {audit}"); - } - } - - #[test] - fn detach_key_ctrl_d_detaches_ctrl_c_stops() { - assert_eq!(detach_key_outcome(true, 'd'), Some(AttachOutcome::Detach)); - assert_eq!(detach_key_outcome(true, 'c'), Some(AttachOutcome::Stop)); - } - - #[test] - fn detach_key_ignores_plain_and_unrelated_keys() { - // Without the control modifier, `d`/`c` are ordinary log-scroll input. - assert_eq!(detach_key_outcome(false, 'd'), None); - assert_eq!(detach_key_outcome(false, 'c'), None); - // Other control combos are not detach/stop triggers. - assert_eq!(detach_key_outcome(true, 'q'), None); - assert_eq!(detach_key_outcome(true, 'z'), None); - } - - /// A record with every reuse-relevant field set to the "reusable" value, so - /// each test below breaks exactly one term and nothing else. - fn reusable_record(paths: &AppPaths) -> ManagedServiceRecord { - let mut record = ManagedServiceRecord::new( - paths, - "vllm-qwen-1000", - "vllm", - "qwen", - "qwen-canonical".to_owned(), - "127.0.0.1", - 8000, - "managed", - 0, - None, - None, - Some("gpu_required".to_owned()), - ); - record.engine_recipe_json = None; - record.requires_api_key = true; - record - } - - #[test] - fn reuse_is_refused_on_every_term_the_spawn_and_the_gpu_bail_depend_on() { - let (root, paths) = test_paths("managed-reuse-predicate"); - paths.ensure().expect("paths"); - let base = reusable_record(&paths); - - // The baseline must be reusable, or the negative cases below could pass - // for the wrong reason. - assert!( - managed_service_reuse_refusal(&base, None, true, &DevicePolicy::GpuRequired).is_none(), - "a matching service with no recipe, a key, and a gpu_required policy is reusable" - ); - - // Recipe mismatch — the spawn's first refusal. - let mut mismatched = base.clone(); - mismatched.engine_recipe_json = Some("{\"engine\":\"vllm\"}".to_owned()); - assert!(matches!( - managed_service_reuse_refusal(&mismatched, None, true, &DevicePolicy::GpuRequired), - Some(ManagedReuseRefusal::RecipeMismatch) - )); - - // Unmet `--require-api-key` — the spawn's second refusal. Also pins that - // it is *only* refused when the key is actually demanded. - let mut anonymous = base.clone(); - anonymous.requires_api_key = false; - assert!(matches!( - managed_service_reuse_refusal(&anonymous, None, true, &DevicePolicy::GpuRequired), - Some(ManagedReuseRefusal::MissingApiKey) - )); - assert!( - managed_service_reuse_refusal(&anonymous, None, false, &DevicePolicy::GpuRequired) - .is_none(), - "an anonymous service is fine when no key was demanded" - ); - - // Device policy — the term `serve()` depends on, because it is the one - // gating the no-usable-GPU bail on the answer. Reusing a `cpu_only` - // service to answer a `gpu_required` serve is the silent CPU fallback. - let mut cpu_only = base; - cpu_only.device_policy = Some("cpu_only".to_owned()); - assert!(matches!( - managed_service_reuse_refusal(&cpu_only, None, true, &DevicePolicy::GpuRequired), - Some(ManagedReuseRefusal::DevicePolicyTooWeak) - )); - // `GpuRequired` is the only policy the CLI can produce — `--device - // gpu_preferred`/`auto`/`gpu` all parse to it and `cpu_only` is rejected - // (`parse_device_policy_defaults_to_gpu_required_without_cpu_fallback`) — - // so there is no second policy arm to discriminate here, and asserting - // one would pin an input `serve()` never produces. Both callers pass - // the same policy; the parameter is not optional, so neither can drop - // this term without a compile error. - // - // A cpu_only *invocation* never reaches the bail, so nothing to refuse. - assert!( - managed_service_reuse_refusal(&cpu_only, None, true, &DevicePolicy::CpuOnly).is_none() - ); - - let _ = std::fs::remove_dir_all(&root); - } - - #[test] - fn spawn_managed_engine_child_refuses_an_explicit_port_a_live_equivalent_does_not_serve() - -> Result<()> { - let (root, paths) = test_paths("dup-managed-explicit-port-differs"); - paths.ensure()?; - let mut existing = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-1000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11510, - "managed", - std::process::id(), - None, - None, - None, - ); - existing.status = "ready".to_owned(); - existing.engine_pid = Some(std::process::id()); - existing.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - let launch = |port_request| { - spawn_managed_engine_child( - &paths, - "lemonade", - "lemonade-qwen-2000", - "qwen", - &resolve, - "127.0.0.1", - port_request, - &resolve.device_policy, - &[], - None, - None, - None, - false, - false, - ) - }; - - let different = launch(serve_port::PortRequest::Explicit(11511)); - // The same port, and no port at all, still reuse the live service. - let same = launch(serve_port::PortRequest::Explicit(11510)); - let auto = launch(serve_port::PortRequest::Auto); - let _ = fs::remove_dir_all(root); - - let Err(error) = different else { - panic!("a port the live service does not serve must be refused") - }; - let message = error.to_string(); - assert!( - message.contains("on port 11510") && message.contains("port 11511 would not be used"), - "{message}" - ); - assert!(matches!(same?, ManagedSpawn::AlreadyRunning(_))); - assert!(matches!(auto?, ManagedSpawn::AlreadyRunning(_))); - Ok(()) - } - - #[test] - fn spawn_managed_engine_child_blocks_reuse_with_mismatched_recipe() -> Result<()> { - // A live service recorded with one recipe (e.g. a tool-call parser flag) - // must reject a relaunch requesting a different recipe rather than - // silently reusing the old server, and the error must not claim the - // mismatch is specifically about generation defaults when it could stem - // from any recipe field. - let (root, paths) = test_paths("dup-managed-recipe-mismatch"); - paths.ensure()?; - let mut existing = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-1000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11510, - "managed", - std::process::id(), - None, - None, - None, - ); - existing.status = "ready".to_owned(); - existing.engine_pid = Some(std::process::id()); - existing.engine_recipe_json = Some(serde_json::to_string(&EngineRecipeHint { - contract_version: rocm_engine_protocol::ENGINE_RECIPE_CONTRACT_VERSION.to_owned(), - engine: "lemonade".to_owned(), - required_flags: vec!["--tool-call-parser".to_owned(), "hermes".to_owned()], - ..EngineRecipeHint::default() - })?); - existing.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - let requested_recipe = EngineRecipeHint { - contract_version: rocm_engine_protocol::ENGINE_RECIPE_CONTRACT_VERSION.to_owned(), - engine: "lemonade".to_owned(), - required_flags: vec!["--temperature".to_owned(), "0.5".to_owned()], - ..EngineRecipeHint::default() - }; - - let result = spawn_managed_engine_child( - &paths, - "lemonade", - "lemonade-qwen-2000", - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Explicit(11511), - &resolve.device_policy, - &[], - None, - None, - Some(&requested_recipe), - false, - false, - ); - let _ = fs::remove_dir_all(root); - - let Err(error) = result else { - panic!("mismatched recipe on a live service must be rejected") - }; - let message = error.to_string(); - assert!( - message.contains("different serve options"), - "message should describe the mismatch generically: {message}" - ); - assert!( - message.contains("recipe hint, tool-call parser, or generation defaults"), - "message should not single out generation defaults as the sole cause: {message}" - ); - Ok(()) - } - - #[test] - fn spawn_managed_engine_child_refuses_to_reuse_an_unauthenticated_service() -> Result<()> { - // `--require-api-key` used to be accepted and dropped on this path. The - // reuse branch returns before the flag is recorded and before the - // endpoint key is checked, so a caller demanding auth got a server that - // never had it, with no error. `rocm remote serve` then published that - // endpoint onto the tailnet and printed a freshly minted key under "the - // API key above is what stops anyone else calling it" — a false - // assurance about an endpoint the whole tailnet can reach. - // - // Asserted through `spawn_managed_engine_child` rather than against the - // guard's own arguments: the defect was the early return, so only the - // real call site can fail for it. - let (root, paths) = test_paths("dup-managed-unauthenticated-reuse"); - paths.ensure()?; - let mut existing = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-3000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11520, - "managed", - std::process::id(), - None, - None, - None, - ); - existing.status = "ready".to_owned(); - existing.engine_pid = Some(std::process::id()); - // The state that matters: live, matching, and serving without auth. - existing.requires_api_key = false; - existing.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let result = spawn_managed_engine_child( - &paths, - "lemonade", - "lemonade-qwen-3001", - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Explicit(11520), - &resolve.device_policy, - &[], - None, - None, - None, - // The demand that used to be silently discarded. - true, - false, - ); - let _ = fs::remove_dir_all(root); - - let Err(error) = result else { - panic!( - "reusing an unauthenticated service must not satisfy `--require-api-key`; a \ - satisfied reuse leaves the endpoint open while the caller is told it is not" - ) - }; - let message = error.to_string(); - assert!( - message.contains("without authentication"), - "the refusal must say why it refused: {message}" - ); - // The printed hint must be the exact, directly-runnable command: - // `main.rs`'s `service_actions_require_yes_and_render_sandbox_result` - // proves this exact form (`rocm services stop --yes`) is the one - // the approval gate itself recommends. That running this form actually - // clears *this* refusal — a service reused without the key it is now - // being asked to require — is proven by - // `@id:serve-require-api-key-refuses-unauthenticated-reuse` - // (`tests/e2e-cucumber/features/model_serving.feature`, serve-24), - // which reproduces this exact condition, runs the printed hint, and - // asserts the record is left `stopped` — not by `stop_managed_services` - // in `tests/e2e-cucumber/tests/e2e.rs`, which runs this same - // invocation as best-effort teardown and never asserts success. - assert!( - message.contains("rocm services stop lemonade-qwen-3000 --yes"), - "the refusal must name the way out, with the service to stop, directly runnable with --yes: {message}" - ); - Ok(()) - } - - #[test] - fn a_managed_spawn_refuses_an_invalid_key_file_on_a_service_that_requires_one() -> Result<()> { - // Drives the real call site, not the guard's own arguments. The service - // was launched with `--require-api-key`, and its key file is present but - // invalid — so `requires_api_key` is true while `key_present` is false, - // which is the only way to reach the first branch. A test calling - // `ensure_public_service_has_endpoint_key` directly cannot catch a - // mis-wired call site, which is the defect that has occurred here twice. - // - // The flag is passed explicitly rather than inferred from the key file. - // Inferring it marked every public bind as having demanded auth, because - // a public bind always has a key file whether or not it asked for one. - let (root, paths) = test_paths("managed-spawn-invalid-key"); - paths.ensure()?; - fs::create_dir_all(paths.services_dir())?; - - // Non-empty but invalid (embedded CR/LF), so the file exists and is not - // merely empty (requires_api_key = true) but yields no usable key - // (key_present = false) — `endpoint_api_key_from_file` rejects embedded - // control characters. An empty seed here would let the post-condition - // below pass even if the real cleanup call never ran, since - // `endpoint_api_key` already reads back `None` for an empty file whether - // or not it was ever deleted. - endpoint_keys::store_endpoint_api_key( - &paths, - "lemonade-qwen-3000", - "bad-key\r\nX-Injected: value", - )?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let result = spawn_managed_engine_child( - &paths, - "lemonade", - "lemonade-qwen-3000", - "qwen", - &resolve, - // Loopback on purpose: the public-bind branch must not be what - // refuses this, or the test would pass with the guard disabled. - "127.0.0.1", - serve_port::PortRequest::Explicit(11512), - &resolve.device_policy, - &[], - None, - None, - None, - true, - false, - ); - - // This guard fires before any child is spawned, so `record.engine_pid` - // is still `None` when `spawn_managed_engine_child`'s cleanup arm calls - // `abandon_failed_managed_launch` — proving its pre-spawn branch - // (delete everything) runs from the real call site, not just from the - // `abandoned_launch_leaves_no_record_log_or_endpoint_key` test that - // calls the helper directly. - let manifest_path = paths.service_manifest_path("lemonade-qwen-3000"); - let log_path = paths.service_log_path("lemonade-qwen-3000"); - let engine_state_path = paths.service_engine_state_path("lemonade", "lemonade-qwen-3000"); - let key_file_path = endpoint_keys::endpoint_key_file_path(&paths, "lemonade-qwen-3000"); - let live = live_managed_services(&paths); - let _ = fs::remove_dir_all(root); - - let Err(error) = result else { - panic!("a service requiring a key must not spawn with an unusable key file") - }; - let message = error.to_string(); - assert!( - message.contains("--require-api-key"), - "the refusal must name the flag the service was launched with: {message}" - ); - assert!( - message.contains("without authentication"), - "the refusal must say what the risk is: {message}" - ); - assert!( - !manifest_path.exists(), - "a pre-spawn failure must not leave a record" - ); - assert!( - !log_path.exists(), - "a pre-spawn failure must not leave a log" - ); - assert!( - !engine_state_path.exists(), - "a pre-spawn failure must not leave engine state" - ); - assert!( - !key_file_path.exists(), - "a pre-spawn failure must delete the unusable key file from disk, not merely \ - leave it unreadable as a key" - ); - assert!( - live.is_empty(), - "a pre-spawn failure must leave no reservation" - ); - Ok(()) - } - - #[test] - fn a_failed_spawn_past_the_first_write_terminates_the_child_and_keeps_an_abandoned_record() - -> Result<()> { - // Drives the real call site's *post*-spawn cleanup branch, which - // neither `abandoned_launch_*` test reaches (they call - // `abandon_failed_managed_launch` directly). An engine name this - // binary does not build in makes the real spawned child (a second - // `rocm` process) exit almost immediately via - // `run_builtin_engine_serve_http`'s `other => bail!(...)` arm, after - // `launch_managed_engine_child_locked` has already set - // `record.engine_pid` — so cleanup must terminate (already-dead) the - // child, keep the record and log for inspection, mark the record - // `stopped` (a `failed` one would be respawned by rocmd recovery), and - // clear only the endpoint key. - // - // The child always dies near-instantly, but `spawn_managed_engine_child` - // only reports that within a fixed startup-check window. On a loaded or - // antivirus-scanning Windows CI box, a fresh debug `rocm.exe` can - // occasionally be slow enough to start that the window is missed even - // though the child is already doomed — so poll by retrying a fresh - // attempt (its own service id/port, so no stale manifest collides) up - // to a deadline instead of asserting on one race-prone attempt. - let deadline = std::time::Instant::now() + Duration::from_secs(10); - let mut attempt: u16 = 0; - let (root, paths, service_id, result) = loop { - attempt += 1; - let (root, paths) = - test_paths(&format!("managed-spawn-unknown-engine-cleanup-{attempt}")); - paths.ensure()?; - let service_id = format!("no-such-engine-qwen-{}", 4000 + attempt); - endpoint_keys::store_endpoint_api_key(&paths, &service_id, "test-key")?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let result = spawn_managed_engine_child( - &paths, - "no-such-engine", - &service_id, - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Explicit(11_514 + attempt), - &resolve.device_policy, - &[], - None, - None, - None, - false, - false, - ); - - if result.is_ok() && std::time::Instant::now() < deadline { - let _ = fs::remove_dir_all(&root); - continue; - } - break (root, paths, service_id, result); - }; - - let manifest_path = paths.service_manifest_path(&service_id); - let log_path = paths.service_log_path(&service_id); - let persisted = load_managed_services(&paths) - .expect("re-read manifests") - .into_iter() - .find(|candidate| candidate.service_id == service_id); - let key = endpoint_keys::endpoint_api_key(&paths, &service_id); - let live = live_managed_services(&paths); - - let Err(error) = result else { - panic!("an engine this binary does not build in must not report success") - }; - let message = error.to_string(); - assert!( - message.contains("managed engine exited immediately"), - "the failure must come from the spawned child dying, not an earlier guard: {message}" - ); - assert!( - manifest_path.exists(), - "a post-spawn failure must keep the record for inspection" - ); - assert!( - log_path.exists(), - "a post-spawn failure must keep the log for inspection" - ); - let persisted = - persisted.expect("a post-spawn failure must leave the record on disk, not delete it"); - // `stopped`, not `failed`: rocmd's recovery restarts a `failed` record - // without a stop marker, which would respawn a launch the CLI just - // reported as failed (pinned on the rocmd side by - // `abandoned_launch_record_is_not_a_recovery_candidate`). - assert_eq!( - persisted.status, "stopped", - "a post-spawn failure must mark the record abandoned, not leave it stuck" - ); - assert_eq!( - key, None, - "a post-spawn failure must still clear the endpoint key" - ); - assert!( - live.is_empty(), - "an abandoned record must not still reserve its port" - ); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn failed_port_reservation_leaves_no_orphaned_endpoint_key() -> Result<()> { - // serve() stores the new attempt's endpoint key before calling - // start_managed_service. If spawn_managed_engine_child fails inside - // resolve_managed_port_in_transaction (explicit port already reserved - // by another live service), it returns before any - // ManagedServiceRecord exists for this attempt, so - // abandon_failed_managed_launch never runs. Driving the real function - // end to end (not a simulated call to - // drop_orphaned_endpoint_key_on_already_running) proves - // `start_managed_service`'s own `.inspect_err` actually wires the cleanup, mirroring how - // attached_launch_failing_before_spawn_leaves_no_orphaned_endpoint_key - // drives run_attached_service for the foreground path. - let (root, paths) = test_paths("orphaned-key-on-reserved-port"); - paths.ensure()?; - let mut other = ManagedServiceRecord::new( - &paths, - "lemonade-llama-1000", - "lemonade", - "llama", - "llama-canonical", - "127.0.0.1", - 11513, - "managed", - std::process::id(), - None, - None, - None, - ); - other.status = "ready".to_owned(); - other.engine_pid = Some(std::process::id()); - other.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let service_id = "lemonade-qwen-2000"; - endpoint_keys::store_endpoint_api_key(&paths, service_id, "test-key")?; - let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; - - let result = start_managed_service( - &paths, - "lemonade", - service_id, - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Explicit(11513), - &resolve.device_policy, - &[], - None, - None, - None, - Some("test-key"), - launch_lock, - false, - false, - false, - &mut |_, _| {}, - &mut |_elapsed| {}, - ); - let Err(error) = result else { - let _ = fs::remove_dir_all(root); - panic!("explicit port already reserved by a live service must be rejected") - }; - assert!( - error - .to_string() - .contains("already reserved by managed service"), - "unexpected error: {error}" - ); - - assert!( - endpoint_keys::endpoint_api_key(&paths, service_id).is_none(), - "cleanup must drop the key this failed attempt staged before launching" - ); - - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - fn attached_launch_failing_before_spawn_leaves_no_orphaned_endpoint_key() -> Result<()> { - // The background path's cleanup is the `.inspect_err` in - // start_managed_service. run_attached_service is the other caller of - // spawn_managed_engine_child and needs the identical .inspect_err() - // cleanup, not a simulated one: drive the real function end to end. - let (root, paths) = test_paths("orphaned-key-attached-reserved-port"); - paths.ensure()?; - fs::create_dir_all(paths.services_dir())?; - let mut other = ManagedServiceRecord::new( - &paths, - "lemonade-llama-1000", - "lemonade", - "llama", - "llama-canonical", - "127.0.0.1", - 11514, - "managed", - std::process::id(), - None, - None, - None, - ); - other.status = "ready".to_owned(); - other.engine_pid = Some(std::process::id()); - other.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let service_id = "lemonade-qwen-attached-2000"; - endpoint_keys::store_endpoint_api_key(&paths, service_id, "test-key")?; - let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; - - let result = run_attached_service( - &paths, - "lemonade", - service_id, - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Explicit(11514), - &[], - None, - None, - Some("test-key"), - launch_lock, - false, - false, - |_, _| unreachable!("the launch fails before a Ctrl-C could stop it"), - ); - let Err(error) = result else { - let _ = fs::remove_dir_all(root); - panic!("explicit port already reserved by a live service must be rejected") - }; - assert!( - error - .to_string() - .contains("already reserved by managed service"), - "unexpected error: {error}" - ); - assert!( - endpoint_keys::endpoint_api_key(&paths, service_id).is_none(), - "a pre-spawn failure must drop the key this attempt staged before launching" - ); - - let _ = fs::remove_dir_all(root); - Ok(()) - } - - /// The launch lock must still be held when the child is spawned: it is what - /// serializes allocation against publication. Observed through the spawn - /// hook, since releasing it early changes nothing else a test can see. - #[cfg(not(windows))] - #[test] - fn start_and_attached_launches_hold_the_launch_lock_until_the_spawn() -> Result<()> { - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - for attached in [false, true] { - let (root, paths) = test_paths("launch-lock-held-until-spawn"); - paths.ensure()?; - let lock_path = paths.managed_launch_lock_path(); - let lock_was_held = std::rc::Rc::new(std::cell::Cell::new(false)); - let observed = lock_was_held.clone(); - TEST_SPAWN_HOOK.with(|hook| { - *hook.borrow_mut() = Some(std::rc::Rc::new(move |_command| { - observed.set( - rocm_core::FileLock::acquire_bounded(&lock_path, Duration::ZERO).is_err(), - ); - Err(io::Error::other("spawn intercepted by test")) - })); - }); - let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; - let result = if attached { - run_attached_service( - &paths, - "lemonade", - "lemonade-lock-attached", - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Auto, - &[], - None, - None, - None, - launch_lock, - false, - false, - |_, _| unreachable!("the spawn fails before a Ctrl-C could stop it"), - ) - } else { - start_managed_service( - &paths, - "lemonade", - "lemonade-lock-start", - "qwen", - &resolve, - "127.0.0.1", - serve_port::PortRequest::Auto, - &resolve.device_policy, - &[], - None, - None, - None, - None, - launch_lock, - false, - false, - false, - &mut |_, _| {}, - &mut |_elapsed| {}, - ) - .map(|_| ()) - }; - TEST_SPAWN_HOOK.with(|hook| *hook.borrow_mut() = None); - let _ = fs::remove_dir_all(root); - assert!( - format!("{:#}", result.expect_err("the intercepted spawn fails")) - .contains("spawn intercepted by test") - ); - assert!( - lock_was_held.get(), - "the launch lock must be held at spawn (attached: {attached})" - ); - } - Ok(()) - } - - #[test] - fn resolve_endpoint_auth_loopback_stays_credential_free() { - // Loopback binds never require auth, even if a key is supplied. - for host in ["127.0.0.1", "localhost", "::1"] { - assert_eq!(resolve_endpoint_auth(host, None, false).unwrap(), None); - assert_eq!( - resolve_endpoint_auth(host, Some("ignored"), false).unwrap(), - None - ); - } - } - - #[test] - fn resolve_endpoint_auth_loopback_can_be_required_when_something_republishes_it() { - // "Loopback" describes the bind address, not who can reach the port. A - // tailnet publish, a proxy, or a container port map all leave the bind - // loopback while widening the audience, and the default policy would - // hand out an unauthenticated endpoint. Whoever widens the reach asks - // for the credential explicitly. - for host in ["127.0.0.1", "localhost", "::1"] { - let generated = resolve_endpoint_auth(host, None, true) - .unwrap() - .expect("a required key must be generated, not skipped"); - assert!(!generated.trim().is_empty()); - - assert_eq!( - resolve_endpoint_auth(host, Some("supplied-key"), true).unwrap(), - Some("supplied-key".to_owned()), - "a supplied key must be honoured rather than ignored as it is by default" - ); - } - - // The same validation a public bind gets: an empty key is a refusal, not - // a silent downgrade to no auth. - assert!(resolve_endpoint_auth("127.0.0.1", Some(" "), true).is_err()); - } - - #[test] - fn resolve_endpoint_auth_public_uses_supplied_key_trimmed() { - let key = resolve_endpoint_auth("0.0.0.0", Some(" my-key "), false) - .unwrap() - .expect("public bind must have a key"); - assert_eq!(key, "my-key"); - } - - #[test] - fn resolve_endpoint_auth_public_generates_key_when_absent() { - let key = resolve_endpoint_auth("0.0.0.0", None, false) - .unwrap() - .expect("public bind must generate a key"); - assert_eq!(key.len(), 48); - assert!(key.chars().all(|c| c.is_ascii_alphanumeric())); - } - - #[test] - fn resolve_endpoint_auth_public_rejects_empty_supplied_key() { - let error = resolve_endpoint_auth("0.0.0.0", Some(" "), false).unwrap_err(); - assert!(error.to_string().contains("non-empty"), "{error:#}"); - } - - #[test] - fn resolve_endpoint_auth_public_rejects_embedded_crlf() { - // A supplied key survives `trim()` with embedded CR/LF intact and would - // otherwise be interpolated into a raw `Authorization: Bearer` header, - // injecting an extra header line. It must be rejected at input validation. - for supplied in [ - "good-key\r\nX-Injected: value", - "good-key\nmore", - "line\rreturn", - ] { - let error = resolve_endpoint_auth("0.0.0.0", Some(supplied), false).unwrap_err(); - assert!(error.to_string().contains("control character"), "{error:#}"); - } - } - - #[test] - fn drop_orphaned_endpoint_key_on_already_running_clears_stored_key() { - let (root, paths) = test_paths("drop-orphaned-key-stored"); - let service_id = "svc-orphaned"; - endpoint_keys::store_endpoint_api_key(&paths, service_id, "secret-key").unwrap(); - - drop_orphaned_endpoint_key_on_already_running(&paths, service_id, Some("secret-key")); - - assert_eq!(endpoint_keys::endpoint_api_key(&paths, service_id), None); - assert!(!endpoint_keys::endpoint_key_file_path(&paths, service_id).exists()); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn drop_orphaned_endpoint_key_on_already_running_is_noop_for_loopback() { - // A loopback attempt never stores a key (`freshly_stored == None`), so the - // helper must not panic or error, and no file must appear. - let (root, paths) = test_paths("drop-orphaned-key-loopback"); - let service_id = "svc-loopback"; - - drop_orphaned_endpoint_key_on_already_running(&paths, service_id, None); - - assert_eq!(endpoint_keys::endpoint_api_key(&paths, service_id), None); - assert!(!endpoint_keys::endpoint_key_file_path(&paths, service_id).exists()); - let _ = fs::remove_dir_all(root); - } - - #[test] - fn public_bind_fails_closed_for_windows_lemonade_only() { - // Windows + Lemonade + public bind: refuse (cannot enforce the key). - let error = ensure_public_bind_engine_supported("lemonade", true, true).unwrap_err(); - assert!(error.to_string().contains("lemonade"), "{error:#}"); - // Every other combination is allowed: - ensure_public_bind_engine_supported("vllm", true, true).unwrap(); // vLLM enforces auth on Windows - ensure_public_bind_engine_supported("lemonade", true, false).unwrap(); // non-Windows - ensure_public_bind_engine_supported("lemonade", false, true).unwrap(); // loopback needs no key - } - - #[test] - fn a_public_bind_is_refused_with_the_command_that_restores_it() { - // `resolve_endpoint_auth` mints a key for every non-loopback bind whether - // or not auth was demanded, so deriving `requires_api_key` from key-file - // presence marked ordinary public binds as having asked for it. The guard - // tests that field first, so those services were refused with a message - // naming a flag they never passed and a relaunch command that drops - // `--allow-public-bind` — coming back on loopback instead. - // - // A plain `--host 0.0.0.0 --allow-public-bind` launch: key file present, - // `--require-api-key` never passed. - let error = ensure_public_service_has_endpoint_key("0.0.0.0", false, false) - .expect_err("a public bind with no key must be refused"); - let rendered = format!("{error:#}"); - assert!( - rendered.contains("--allow-public-bind"), - "the refusal must name the command that restores the public bind: {rendered}" - ); - assert!( - !rendered.contains("--require-api-key"), - "a service that never passed the flag must not be told it did: {rendered}" - ); - - // And the loopback-with-forced-auth case still reports its own reason. - let error = ensure_public_service_has_endpoint_key("127.0.0.1", false, true) - .expect_err("a service that demanded auth must not come back without it"); - assert!( - format!("{error:#}").contains("--require-api-key"), - "{error:#}" - ); - } - - #[test] - fn respawn_fails_closed_for_a_public_service_whose_key_is_gone() { - // A stop deletes the key file, so a later restart of a public service - // would otherwise respawn it with no auth at all. - let error = ensure_public_service_has_endpoint_key("0.0.0.0", false, false).unwrap_err(); - let message = error.to_string(); - assert!(message.contains("0.0.0.0"), "{error:#}"); - assert!(message.contains("without authentication"), "{error:#}"); - // Actionable: name the command that mints a fresh key. - assert!(message.contains("--allow-public-bind"), "{error:#}"); - - // A public service that still has its key restarts normally. - ensure_public_service_has_endpoint_key("0.0.0.0", true, false).unwrap(); - } - - #[test] - fn respawn_allows_loopback_services_without_an_endpoint_key() { - // Loopback stays credential-free, so every accepted spelling must pass - // the guard with no key present — when nothing asked for auth. - for host in ["127.0.0.1", "localhost", "::1"] { - ensure_public_service_has_endpoint_key(host, false, false) - .unwrap_or_else(|error| panic!("{host} must not require a key: {error:#}")); - } - } - - #[test] - fn respawn_refuses_a_loopback_service_that_was_launched_with_a_key() { - // The hole this closes: a loopback bind that something else republishes - // — a tailnet publish, a proxy, a container port map. The publish - // outlives the process, so a restart after the key was dropped would - // reopen a reachable endpoint with no authentication, and the bind - // address gives the guard no way to notice. - for host in ["127.0.0.1", "localhost", "::1"] { - let error = ensure_public_service_has_endpoint_key(host, false, true) - .expect_err("a service launched with a key must not restart without one"); - let message = format!("{error:#}"); - assert!(message.contains("without authentication"), "{message}"); - assert!(message.contains("--require-api-key"), "{message}"); - } - - // With its key still present it restarts normally. - ensure_public_service_has_endpoint_key("127.0.0.1", true, true).unwrap(); - } - - #[test] - fn endpoint_client_config_shows_key_once_with_bearer_guidance() { - let rendered = render_endpoint_client_config("http://0.0.0.0:11435/v1", "secret-123"); - // Dummy literal exercising the intentional one-time key display documented - // on `render_endpoint_client_config`; not a real credential or a log write. - // codeql[rust/cleartext-logging] - assert!(rendered.contains("secret-123"), "{rendered}"); - // codeql[rust/cleartext-logging]: see rationale above. - assert!(rendered.contains("Authorization: Bearer"), "{rendered}"); - // codeql[rust/cleartext-logging]: see rationale above. - assert!(rendered.contains("shown only now"), "{rendered}"); - } - - /// ROCMAI-827: `rocm serve` started the background daemon on every launch, - /// and with no automation watcher enabled that daemon had nothing to do yet - /// kept running after every service was stopped. With the default config - /// (no watcher enabled) the helper must not be spawned at all. - #[test] - fn background_helper_not_spawned_when_no_automation_is_enabled() { - let (root, paths) = test_paths("helper-no-automation"); - paths.ensure().expect("create app dirs"); - // The stored flag alone is not "a watcher is enabled": nothing would run. - let mut config = rocm_core::RocmCliConfig::default(); - config.automations.daemon_enabled = true; - config.save(&paths).expect("save default config"); - - let mut spawned = false; - ensure_background_helper_running_with( - &paths, - true, - |_| { - spawned = true; - Ok(Ok(4242)) - }, - &|| {}, - ) - .expect("ensure helper"); - - let claim_written = paths.automation_autostart_claim_path().exists(); - let _ = fs::remove_dir_all(&root); - assert!( - !spawned, - "no watcher enabled ⇒ the daemon has no work and must not be started" - ); - assert!(!claim_written, "no spawn ⇒ no autostart claim"); - } - - /// The other half of the gate: once a watcher is enabled the helper is still - /// started (exactly once) and its autostart claim recorded. - #[test] - fn background_helper_spawned_when_an_automation_is_enabled() { - let (root, paths) = test_paths("helper-automation-enabled"); - paths.ensure().expect("create app dirs"); - let mut config = rocm_core::RocmCliConfig::default(); - config.watcher_config_mut("server-recover").enabled = true; - config.save(&paths).expect("save config"); - - let mut spawns = 0; - ensure_background_helper_running_with( - &paths, - true, - |_| { - spawns += 1; - Ok(Ok(4242)) - }, - &|| {}, - ) - .expect("ensure helper"); - - let claim = read_autostart_claim(&paths.automation_autostart_claim_path()); - let _ = fs::remove_dir_all(&root); - assert_eq!(spawns, 1, "an enabled watcher needs the daemon"); - assert_eq!(claim.map(|claim| claim.daemon_pid), Some(4242)); - } - - /// ROCMAI-827 race, enable side: the daemon that is about to exit still has - /// `running = true` and a live pid in its state when `automations enable` - /// (config already saved) asks for a helper. It decides under the autostart - /// lock and publishes "stopped" before releasing it, so the enable must wait - /// on that lock and then start a new helper. A lock-free "already running" - /// read would return here without spawning, leaving a watcher with no daemon. - #[test] - fn background_helper_is_started_when_the_exiting_daemon_publishes_stopped() { - let (root, paths) = test_paths("helper-exit-window"); - paths.ensure().expect("create app dirs"); - let mut config = rocm_core::RocmCliConfig::default(); - config.watcher_config_mut("server-recover").enabled = true; - config.save(&paths).expect("save config"); - // The old daemon: alive (this process stands in) and still `running`. - runtime_state(true, std::process::id()) - .write(&paths) - .expect("write runtime state"); - - // The exiting daemon holds the autostart lock while it decides. - let exiting_daemon_lock = - rocm_core::FileLock::acquire(paths.automation_autostart_lock_path()) - .expect("hold the lock"); - let enable_paths = paths.clone(); - let queued_flag = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); - let queued = queued_flag.clone(); - let enable = std::thread::spawn(move || { - let mut spawns = 0; - ensure_background_helper_running_with( - &enable_paths, - true, - |_| { - spawns += 1; - Ok(Ok(4242)) - }, - &|| queued.store(true, std::sync::atomic::Ordering::SeqCst), - ) - .expect("ensure helper"); - spawns - }); - // Wait until the enable path has actually found the lock held and queued - // behind it, rather than assuming how long that takes. - let deadline = std::time::Instant::now() + std::time::Duration::from_mins(1); - // A caller that skipped the lock finishes without ever queuing. - while !queued_flag.load(std::sync::atomic::Ordering::SeqCst) && !enable.is_finished() { - assert!(std::time::Instant::now() < deadline, "enable never queued"); - std::thread::yield_now(); - } - runtime_state(false, std::process::id()) - .write(&paths) - .expect("daemon publishes stopped"); - drop(exiting_daemon_lock); - - let spawns = enable.join().expect("enable thread"); - let _ = fs::remove_dir_all(&root); - assert_eq!( - spawns, 1, - "the enable must start a helper once the old one has published it stopped" - ); - } - - fn stop_result() -> serde_json::Value { - serde_json::json!({"output": {"result": { - "service": {"service_id": "svc-a", "status": "stopped"}, - "signaled_pids": [1] - }}}) - } - - /// `rocm services stop` prints the note for a stop only, and only the variant - /// that matches the state. - #[test] - fn stop_output_carries_the_helper_note_that_matches_the_state() { - use crate::services_text::render_service_action_result_with_helper_note as render; - - let (root, paths) = test_paths("stop-helper-note"); - paths.ensure().expect("create app dirs"); - set_helper_facts(&paths, false, false); - let quiet = render(&paths, "stop_server", &stop_result()); - set_helper_facts(&paths, true, true); - let outlives = render(&paths, "stop_server", &stop_result()); - let restart = render(&paths, "restart_server", &stop_result()); - set_helper_facts(&paths, true, false); - let not_running = render(&paths, "stop_server", &stop_result()); - let _ = fs::remove_dir_all(&root); - - assert!(!quiet.contains("background automation helper"), "{quiet}"); - assert!( - outlives.contains("stopped processes: 1") - && outlives.contains("keeps running while an automation is enabled") - && outlives.contains("`rocm automations disable server-recover`"), - "{outlives}" - ); - assert!( - not_running.contains("is not running") && !not_running.contains("keeps running"), - "{not_running}" - ); - assert!( - !restart.contains("background automation helper"), - "{restart}" - ); - } - - /// `serve` calls the helper autostart after its child is already running. An - /// unreadable config must not turn that launch into an error: it reads as "no - /// watcher enabled" and nothing is spawned. - #[test] - fn background_helper_autostart_treats_an_unreadable_config_as_no_watcher() { - let (root, paths) = test_paths("helper-bad-config"); - paths.ensure().expect("create app dirs"); - fs::write(paths.config_path(), b"{ this is not json").expect("write bad config"); - assert!(rocm_core::RocmCliConfig::load(&paths).is_err()); - - let mut spawned = false; - let result = ensure_background_helper_running_with( - &paths, - true, - |_| { - spawned = true; - Ok(Ok(4242)) - }, - &|| {}, - ); - let _ = fs::remove_dir_all(&root); - - assert!(result.is_ok(), "{result:?}"); - assert!(!spawned); - } - - #[test] - fn autostart_spawn_in_flight_defers_only_for_a_live_recent_claim() { - let claim = AutostartClaim { - daemon_pid: 4242, - spawned_at_ms: 1_000, - }; - // A recent claim whose PID is still alive ⇒ a spawn is in flight, defer. - assert!(autostart_spawn_in_flight( - Some(claim), - 1_000 + 5_000, - AUTOSTART_CLAIM_TTL_MS, - |_| true, - )); - // Same claim but the child PID is gone (crashed spawn) ⇒ respawn. - assert!(!autostart_spawn_in_flight( - Some(claim), - 1_000 + 5_000, - AUTOSTART_CLAIM_TTL_MS, - |_| false, - )); - // Older than the TTL, even with a live PID (possible PID reuse) ⇒ respawn. - assert!(!autostart_spawn_in_flight( - Some(claim), - 1_000 + AUTOSTART_CLAIM_TTL_MS, - AUTOSTART_CLAIM_TTL_MS, - |_| true, - )); - // No claim at all ⇒ nothing in flight, spawn. - assert!(!autostart_spawn_in_flight( - None, - 1_000, - AUTOSTART_CLAIM_TTL_MS, - |_| true, - )); - } - - #[test] - fn autostart_claim_round_trips_and_absent_or_garbage_reads_as_none() { - let (root, paths) = test_paths("autostart-claim-roundtrip"); - let path = paths.automation_autostart_claim_path(); - fs::create_dir_all(path.parent().expect("claim parent")).expect("mkdir claim dir"); - // Absent file ⇒ no claim. - assert_eq!(read_autostart_claim(&path), None); - // Round-trip a written claim. - let claim = AutostartClaim { - daemon_pid: 9987, - spawned_at_ms: 1_726_000_000_123, - }; - write_autostart_claim(&path, claim).expect("write claim"); - assert_eq!(read_autostart_claim(&path), Some(claim)); - // Garbage / partial content ⇒ no claim (permits a respawn, never panics). - fs::write(&path, "not-a-pid").expect("write garbage"); - assert_eq!(read_autostart_claim(&path), None); - let _ = fs::remove_dir_all(&root); - } - - #[test] - fn duplicate_managed_launch_detected_across_distinct_service_ids() -> Result<()> { - // `generate_service_id` embeds a timestamp, so a second launch for the - // same engine+model has a DIFFERENT service_id. The guard must still - // detect the live service by (engine, canonical_model_id), and return - // the newest live match. `starting` skips the endpoint probe; the - // current process id is a guaranteed-live PID. - let (root, paths) = test_paths("dup-managed-distinct-ids"); - paths.ensure()?; - - // Older, dead manifest for the same engine+model (distinct service_id). - let mut dead = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-1000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11500, - "managed", - 999_999_999, - None, - None, - None, - ); - dead.status = "ready".to_owned(); - dead.engine_pid = Some(999_999_999); - dead.created_at_unix_ms = 1000; - dead.write()?; - - // Newer, live manifest for the same engine+model (distinct service_id). - let mut live = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-2000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11501, - "managed", - std::process::id(), - None, - None, - None, - ); - live.status = "starting".to_owned(); - live.engine_pid = Some(std::process::id()); - live.created_at_unix_ms = 2000; - live.write()?; - - let live_services = live_managed_services(&paths); - let found = - existing_live_managed_service_in(&live_services, "lemonade", "qwen-canonical").cloned(); - let _ = fs::remove_dir_all(root); - - let found = found.expect("a live managed service should be detected by engine+model"); - assert_eq!( - found.service_id, "lemonade-qwen-2000", - "should return the newest live match" - ); - assert!(managed_service_is_live(&found)); - Ok(()) - } - - #[test] - fn dead_managed_service_allows_relaunch() -> Result<()> { - // A stale manifest with dead PIDs must NOT block a relaunch: liveness - // refresh demotes it to "stopped", so the guard returns None. - let (root, paths) = test_paths("dup-managed-dead"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "lemonade-qwen-3000", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - 11502, - "managed", - 999_999_999, - None, - None, - None, - ); - record.status = "ready".to_owned(); - record.engine_pid = Some(999_999_999); - record.write()?; - - let live = live_managed_services(&paths); - let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); - let _ = fs::remove_dir_all(root); - - assert!( - found.is_none(), - "a dead managed service must not block relaunch" - ); - Ok(()) - } - - #[test] - fn live_service_for_other_model_does_not_block() -> Result<()> { - // A live service for a DIFFERENT model must not match — the guard keys - // on the model, not just the engine. - let (root, paths) = test_paths("dup-managed-other-model"); - paths.ensure()?; - let mut record = ManagedServiceRecord::new( - &paths, - "lemonade-other-1", - "lemonade", - "other", - "other-canonical", - "127.0.0.1", - 11503, - "managed", - std::process::id(), - None, - None, - None, - ); - record.status = "starting".to_owned(); - record.engine_pid = Some(std::process::id()); - record.write()?; - - let live = live_managed_services(&paths); - let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); - let _ = fs::remove_dir_all(root); - - assert!( - found.is_none(), - "a live service for a different model must not match" - ); - Ok(()) - } - - #[test] - fn missing_manifest_allows_launch() { - // No services dir / manifests → nothing to detect, launch proceeds. - let (root, paths) = test_paths("dup-managed-missing"); - let live = live_managed_services(&paths); - let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); - let _ = fs::remove_dir_all(root); - assert!(found.is_none()); - } - - fn write_live_service_record( - paths: &AppPaths, - service_id: &str, - canonical_model_id: &str, - port: u16, - status: &str, - ) -> Result { - let mut record = ManagedServiceRecord::new( - paths, - service_id, - "lemonade", - canonical_model_id, - canonical_model_id.to_owned(), - rocm_core::DEFAULT_LOCAL_HOST, - port, - "managed", - std::process::id(), - None, - None, - None, - ); - record.status = status.to_owned(); - record.engine_pid = Some(std::process::id()); - record.write()?; - Ok(record) - } - - fn never_probe(_port: u16) -> std::io::Result { - panic!("no port may be probed on this path") - } - - fn always_free(port: u16) -> std::io::Result { - Ok(rocm_core::LoopbackPortLease::engine_owned(port)) - } - - #[test] - fn allocation_transaction_answers_an_equivalent_live_service_before_leasing() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-already-live"); - paths.ensure()?; - let existing = write_live_service_record( - &paths, - "lemonade-qwen-9000", - "qwen-canonical", - 11501, - "starting", - )?; - - let decision = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "qwen-canonical", - "127.0.0.1", - serve_port::PortRequest::Auto, - &mut never_probe, - )?; - let _ = fs::remove_dir_all(root); - - match decision { - ManagedPortDecision::AlreadyLive(record) => { - assert_eq!(record.service_id, existing.service_id); - assert_eq!(record.port, existing.port); - } - ManagedPortDecision::Lease(_) => { - panic!("an equivalent live service must be answered, not leased a new port") - } - } - Ok(()) - } - - #[test] - fn allocation_transaction_skips_ports_reserved_by_other_live_services() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-skips-reserved"); - paths.ensure()?; - write_live_service_record(&paths, "lemonade-a", "model-a", 11_435, "running")?; - write_live_service_record(&paths, "lemonade-b", "model-b", 11_436, "recovering")?; - - let decision = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "model-c", - "127.0.0.1", - serve_port::PortRequest::Auto, - &mut always_free, - )?; - let _ = fs::remove_dir_all(root); - - match decision { - ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_437), - ManagedPortDecision::AlreadyLive(record) => { - panic!("model-c has no live service, expected a lease, got {record:?}") - } - } - Ok(()) - } - - #[test] - fn allocation_transaction_ignores_stopped_records() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-ignores-stopped"); - paths.ensure()?; - let mut stale = - write_live_service_record(&paths, "lemonade-stale", "model-stale", 11_435, "running")?; - // A dead pid demotes the record to stopped when liveness is refreshed, so - // it must not reserve its port. - stale.engine_pid = Some(999_999_999); - stale.supervisor_pid = 999_999_999; - stale.write()?; - - let decision = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "model-fresh", - "127.0.0.1", - serve_port::PortRequest::Auto, - &mut always_free, - )?; - let _ = fs::remove_dir_all(root); - - match decision { - ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_435), - ManagedPortDecision::AlreadyLive(record) => { - panic!("a stopped record must not block a fresh launch, got {record:?}") - } - } - Ok(()) - } - - #[test] - fn stopping_the_reserving_record_frees_its_explicit_port_for_reuse() -> Result<()> { - // Proves the mechanism `rocm services stop` relies on: it never touches - // the port directly, it only flips the stopped record's `status` (see - // the `all_stopped` branch in `main.rs`'s stop handler). This recreates - // that exact disk state and checks the same explicit port a live - // competitor refused now resolves, closing the loop the E2E - // `assert_port_reserved_message` step only checks by wording. - let (root, paths) = test_paths("alloc-explicit-port-freed-by-stop"); - paths.ensure()?; - let mut holder = ManagedServiceRecord::new( - &paths, - "lemonade-a-1", - "lemonade", - "model-a", - "model-a".to_owned(), - DEFAULT_LOCAL_HOST, - 11_435, - "managed", - std::process::id(), - None, - None, - None, - ); - holder.status = "running".to_owned(); - holder.engine_pid = Some(std::process::id()); - holder.write()?; - - let still_reserved = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "model-b", - DEFAULT_LOCAL_HOST, - serve_port::PortRequest::Explicit(11_435), - &mut always_free, - ); - // The refusal names the service AND the command that frees the port; - // the stop below is that command's own implementation, and the grant - // after it is the outcome, so advice and result are pinned together. - let stop_hint = service_hints::service_stop_hint("lemonade-a-1"); - assert!( - matches!(still_reserved, Err(ref error) if error.to_string().contains("already reserved by managed service lemonade-a-1") - && error.to_string().contains(&stop_hint)), - "unexpected result while still running: {still_reserved:?}" - ); - - // Go through the real stop path (`rocm services stop --yes`'s own - // implementation) rather than writing `status = "stopped"` to disk - // directly, so this test actually proves the advice serve-45 pins: that - // running the stop command frees the port. The termination closure is - // the same seam production injects a real signal through; here it just - // confirms every recorded process gone without touching the test - // process itself. - crate::service_control::stop_internal_managed_service_with(&paths, "lemonade-a-1", |_| { - crate::service_control::RecordedStopTermination::confirmed_stopped() - })?; - - let decision = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "model-b", - DEFAULT_LOCAL_HOST, - serve_port::PortRequest::Explicit(11_435), - &mut always_free, - ); - let _ = fs::remove_dir_all(root); - - match decision? { - ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_435), - ManagedPortDecision::AlreadyLive(existing) => { - panic!( - "a stopped record must not block reuse: {}", - existing.service_id - ) - } - } - Ok(()) - } - - #[test] - fn allocation_transaction_rejects_an_explicit_port_a_recovering_service_holds() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-rejects-explicit-collision"); - paths.ensure()?; - let existing = write_live_service_record( - &paths, - "lemonade-recovering", - "model-recovering", - 11_435, - "recovering", - )?; - - let result = resolve_managed_port_in_transaction( - &paths, - "lemonade", - "model-other", - "127.0.0.1", - serve_port::PortRequest::Explicit(11_435), - &mut never_probe, - ); - let _ = fs::remove_dir_all(root); - - let Err(error) = result else { - panic!("an explicit port a recovering service holds must be refused") - }; - let message = error.to_string(); - assert!( - message.contains(&existing.service_id), - "the refusal must name the service holding the port: {message}" - ); - assert!( - message.contains("recovering"), - "the refusal must say why the port is unavailable: {message}" - ); - Ok(()) - } - - #[test] - fn concurrent_auto_transactions_publish_distinct_ports() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-concurrent-distinct"); - paths.ensure()?; - let paths = std::sync::Arc::new(paths); - - let spawn_one = |model: &'static str| { - let paths = std::sync::Arc::clone(&paths); - thread::spawn(move || -> Result { - let _launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; - let lease = match resolve_managed_port_in_transaction( - &paths, - "lemonade", - model, - "127.0.0.1", - serve_port::PortRequest::Auto, - &mut serve_port::loopback_bind_probe, - )? { - ManagedPortDecision::Lease(lease) => lease, - ManagedPortDecision::AlreadyLive(record) => { - panic!("fresh model must not already be live: {record:?}") - } - }; - let port = lease.port(); - write_live_service_record(&paths, model, model, port, "running")?; - lease.release(); - Ok(port) - }) - }; - - let first = spawn_one("model-one"); - let second = spawn_one("model-two"); - let first_port = first.join().expect("thread one panicked")?; - let second_port = second.join().expect("thread two panicked")?; - let _ = fs::remove_dir_all(root); - - assert_ne!(first_port, second_port); - for port in [first_port, second_port] { - assert!( - (serve_port::AUTO_PORT_FIRST..=serve_port::AUTO_PORT_LAST).contains(&port), - "leased port {port} must fall within the auto-select range" - ); - } - Ok(()) - } - - #[test] - fn concurrent_equivalent_transactions_converge_on_one_record() -> Result<()> { - let (root, paths) = test_paths("alloc-txn-concurrent-converge"); - paths.ensure()?; - let paths = std::sync::Arc::new(paths); - - let race_once = || { - let paths = std::sync::Arc::clone(&paths); - thread::spawn(move || -> Result { - let _launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; - match resolve_managed_port_in_transaction( - &paths, - "lemonade", - "shared-model", - "127.0.0.1", - serve_port::PortRequest::Auto, - &mut serve_port::loopback_bind_probe, - )? { - ManagedPortDecision::Lease(lease) => { - let port = lease.port(); - write_live_service_record( - &paths, - "lemonade-shared", - "shared-model", - port, - "running", - )?; - lease.release(); - Ok(true) - } - ManagedPortDecision::AlreadyLive(_) => Ok(false), - } - }) - }; - - let first = race_once(); - let second = race_once(); - let first_launched = first.join().expect("thread one panicked")?; - let second_launched = second.join().expect("thread two panicked")?; - - let live_count = live_managed_services(&paths).len(); - let _ = fs::remove_dir_all(root); - - assert_ne!( - first_launched, second_launched, - "exactly one racing transaction must launch" - ); - assert_eq!(live_count, 1, "only one record may end up live"); - Ok(()) - } - - #[test] - fn select_gpu_indices_under_launch_lock_blocks_until_the_holder_releases() -> Result<()> { - // Exercises the real lock-acquisition function production callers use - // (`select_gpu_indices_under_launch_lock`) against a lock another thread - // actually holds, so a refactor that dropped the lock early would fail - // here. `spawn_managed_engine_child` is then run for real, but only via - // its `AlreadyLive` short-circuit (a pre-seeded duplicate record), so no - // child process is ever launched. The holder writes nothing: this does - // not show a write made under the lock being observed. - use std::sync::{Arc, Barrier}; - use std::time::Instant; - - let (root, paths) = test_paths("launch-lock-holds-across-spawn"); - paths.ensure()?; - write_live_service_record(&paths, "lemonade-held-1", "held-canonical", 11_520, "ready")?; - - let barrier = Arc::new(Barrier::new(2)); - let hold_for = Duration::from_millis(250); - - let holder_paths = paths.clone(); - let holder_barrier = Arc::clone(&barrier); - let holder = thread::spawn(move || -> Result<()> { - let _lock = rocm_core::FileLock::acquire(holder_paths.managed_launch_lock_path())?; - holder_barrier.wait(); - thread::sleep(hold_for); - Ok(()) - }); - - barrier.wait(); - let started = Instant::now(); - let (gpu_indices, launch_lock) = crate::gpu_policy::select_gpu_indices_under_launch_lock( - &paths, - true, - None, - || None, - None, - None, - )?; - let waited = started.elapsed(); - - let resolve = ResolveModelResponse { - canonical_model_id: "held-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - let spawn = spawn_managed_engine_child( - &paths, - "lemonade", - "lemonade-held-2", - "held-model", - &resolve, - DEFAULT_LOCAL_HOST, - serve_port::PortRequest::Auto, - &resolve.device_policy, - &gpu_indices, - None, - None, - None, - false, - false, - ); - drop(launch_lock); - holder.join().expect("holder thread")?; - let _ = fs::remove_dir_all(root); - - // `thread::sleep` only guarantees *at least* `hold_for`, but under a - // loaded host (e.g. a full parallel `cargo test` run on WSL2) timer - // coalescing can make the measured wait a hair short of the nominal - // hold. Allow a small margin so the test still fails loudly if the - // call returns early, without being flaky on actual block duration. - let margin = Duration::from_millis(20); - assert!( - waited + margin >= hold_for, - "select_gpu_indices_under_launch_lock returned before the holder released the lock: waited {waited:?}, held {hold_for:?}" - ); - match spawn? { - ManagedSpawn::AlreadyRunning(report) => { - assert_eq!(report.service_id, "lemonade-held-1"); - } - ManagedSpawn::Spawned { .. } => { - panic!( - "expected the pre-seeded AlreadyLive record to short-circuit, not a real spawn" - ) - } - } - Ok(()) - } - - #[test] - fn abandoned_launch_leaves_no_record_log_or_endpoint_key() -> Result<()> { - let (root, paths) = test_paths("abandon-launch-cleanup"); - paths.ensure()?; - let mut record = write_live_service_record( - &paths, - "lemonade-abandon", - "abandon-model", - 11_490, - "running", - )?; - fs::create_dir_all( - record - .engine_state_path - .parent() - .expect("engine state parent"), - )?; - fs::write(&record.engine_state_path, "{}")?; - endpoint_keys::store_endpoint_api_key(&paths, &record.service_id, "secret-key")?; - // Simulate a pre-spawn failure: nothing ever ran. - record.engine_pid = None; - - abandon_failed_managed_launch(&paths, &record); - - assert!(!record.manifest_path.exists()); - assert!(!record.log_path.exists()); - assert!(!record.engine_state_path.exists()); - assert_eq!( - endpoint_keys::endpoint_api_key(&paths, &record.service_id), - None - ); - assert!(live_managed_services(&paths).is_empty()); - let _ = fs::remove_dir_all(root); - Ok(()) - } - - #[test] - #[cfg(not(windows))] - #[ignore = "spawned as a child process by abandoned_launch_terminates_the_spawned_child"] - fn managed_launch_cleanup_sleeper_child() { - if std::env::var_os("ROCM_TEST_SLEEPER_CHILD").is_some() { - thread::sleep(Duration::from_mins(2)); - } - } - - #[test] - #[cfg(target_os = "linux")] - fn abandoned_launch_never_signals_a_pid_that_no_longer_matches_the_spawn_token() -> Result<()> { - // The child exited and its pid now belongs to an unrelated live - // process: the start token recorded at spawn no longer matches, so the - // forced tree kill must not touch it. - let (root, paths) = test_paths("abandon-launch-recycled-pid"); - paths.ensure()?; - let mut record = write_live_service_record( - &paths, - "lemonade-abandon-recycled", - "abandon-recycled-model", - 11_493, - "running", - )?; - let current_exe = std::env::current_exe().expect("current test exe"); - let mut bystander = ProcessCommand::new(current_exe) - .arg("--exact") - .arg("service_launch::tests::managed_launch_cleanup_sleeper_child") - .arg("--ignored") - .arg("--test-threads=1") - .env("ROCM_TEST_SLEEPER_CHILD", "1") - .spawn() - .expect("spawn bystander"); - record.record_engine_identity(bystander.id()); - record.engine_start_ticks = record.engine_start_ticks.map(|ticks| ticks + 1); - - abandon_failed_managed_launch(&paths, &record); - - let mismatched_left_alone = bystander.try_wait().expect("poll bystander").is_none(); - // A child that was gone before its token could be captured leaves none: - // that must not fall back to "best-effort" signalling either. - record.engine_start_ticks = None; - abandon_failed_managed_launch(&paths, &record); - let still_running = - mismatched_left_alone && bystander.try_wait().expect("poll bystander").is_none(); - let _ = bystander.kill(); - let _ = bystander.wait(); - let _ = fs::remove_dir_all(root); - assert!( - still_running, - "a pid with a mismatched token must be left alone" - ); - Ok(()) - } - - #[test] - #[cfg(not(windows))] - fn abandoned_launch_terminates_the_spawned_child() -> Result<()> { - let (root, paths) = test_paths("abandon-launch-terminates-child"); - paths.ensure()?; - let mut record = write_live_service_record( - &paths, - "lemonade-abandon-child", - "abandon-child-model", - 11_491, - "running", - )?; - - let current_exe = std::env::current_exe().expect("current test exe"); - let mut child = ProcessCommand::new(current_exe) - .arg("--exact") - .arg("service_launch::tests::managed_launch_cleanup_sleeper_child") - .arg("--ignored") - .arg("--test-threads=1") - .env("ROCM_TEST_SLEEPER_CHILD", "1") - .spawn() - .expect("spawn sleeper child"); - let child_pid = child.id(); - record.record_engine_identity(child_pid); - - abandon_failed_managed_launch(&paths, &record); - - let wait_result = child.wait(); - - match wait_result { - Ok(status) => assert!( - !status.success(), - "the sleeper child must be terminated, not exit cleanly" - ), - Err(error) => panic!("failed to wait for sleeper child: {error}"), - } - let persisted = load_managed_services(&paths) - .expect("re-read manifests") - .into_iter() - .find(|candidate| candidate.service_id == record.service_id); - let _ = fs::remove_dir_all(root); - let persisted = persisted.expect( - "a launch that spawned a child must keep its record so `rocm services`/`rocm logs` can show the attempt", - ); - // `stopped`, not `failed`: `rocmd`'s recovery restarts a `failed` record - // that carries no stop marker, which would respawn a launch the CLI just - // reported as failed. The rocmd side of the pair is - // `abandoned_launch_record_is_not_a_recovery_candidate` in - // `apps/rocmd/src/watchers.rs`. - assert_eq!( - persisted.status, "stopped", - "a spawned child's record must be kept, but in a status recovery ignores" - ); - Ok(()) - } - - #[test] - #[cfg(not(windows))] - fn launch_releases_the_leased_port_before_spawning_the_child() -> Result<()> { - // `lease.release()` must run before the child is spawned, not after, - // or the engine is handed a port the CLI is still holding open. - // Proven via the injected `spawn_override` seam: it asserts a fresh - // bind on the leased port succeeds at the exact instant production - // would spawn, which only holds if the lease was already released. - let (root, paths) = test_paths("launch-releases-port-before-spawn"); - paths.ensure()?; - fs::create_dir_all(paths.services_dir())?; - - // Ephemeral, not a fixed number in the automatic range, which a real - // `rocm serve` on the developer's machine could be holding. - let port = std::net::TcpListener::bind(("127.0.0.1", 0))? - .local_addr()? - .port(); - let lease = rocm_core::lease_loopback_port(port)?; - let mut record = ManagedServiceRecord::new( - &paths, - "lemonade-release-before-spawn", - "lemonade", - "qwen", - "qwen-canonical", - "127.0.0.1", - port, - "managed", - 0, - None, - None, - None, - ); - record.write()?; - - let resolve = ResolveModelResponse { - canonical_model_id: "qwen-canonical".to_owned(), - served_model_id: None, - task: "chat".to_owned(), - source: "hf".to_owned(), - revision: "main".to_owned(), - loader: "llama.cpp".to_owned(), - trust_remote_code: false, - chat_template_mode: "auto".to_owned(), - dtype: "auto".to_owned(), - device_policy: DevicePolicy::GpuPreferred, - estimated_memory: "unknown".to_owned(), - launch_defaults: serde_json::json!({}), - engine_recipe: None, - warnings: Vec::new(), - }; - - let bind_succeeded = std::cell::Cell::new(false); - let spawn_override: &dyn Fn(&mut ProcessCommand) -> io::Result = - &|_command| { - // Retry briefly: a sibling test's concurrent fork can hold a copy - // of the just-released socket until its exec closes it. - let deadline = std::time::Instant::now() + Duration::from_secs(2); - while !bind_succeeded.get() && std::time::Instant::now() < deadline { - bind_succeeded.set(std::net::TcpListener::bind(("127.0.0.1", port)).is_ok()); - } - // A long-lived no-op child stands in for the engine: `_command` - // carries real serve args this test binary cannot execute as - // itself (see `managed_launch_cleanup_sleeper_child`). - ProcessCommand::new(std::env::current_exe().expect("test binary")) - .arg("service_launch::tests::managed_launch_cleanup_sleeper_child") - .arg("--exact") - .arg("--ignored") - .env("ROCM_TEST_SLEEPER_CHILD", "1") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()) - .spawn() - }; - - let result = launch_managed_engine_child_locked( - &paths, - &mut record, - lease, - "lemonade", - "lemonade-release-before-spawn", - &resolve, - "127.0.0.1", - &resolve.device_policy, - &[], - None, - None, - None, - Some(spawn_override), - ); - if let Ok(child_pid) = result.as_ref() { - let _ = rocm_core::terminate_process(*child_pid); - } - let _ = fs::remove_dir_all(root); - - assert!( - bind_succeeded.get(), - "the port must already be bindable (released) by the time the child is spawned" - ); - result?; - Ok(()) - } -} +pub(crate) use spawn::{ + attach_background_stdio, detach_background_command, managed_engine_startup_failure_detail, + managed_service_process_command, +}; +pub(crate) use start::{ + print_managed_launch_plain, start_managed_service, wait_begin_client_config, +}; diff --git a/apps/rocm/src/service_launch/attach.rs b/apps/rocm/src/service_launch/attach.rs new file mode 100644 index 0000000000..cd1bfe7700 --- /dev/null +++ b/apps/rocm/src/service_launch/attach.rs @@ -0,0 +1,768 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The attached serve path (`--verbose`/`--foreground`): spawn the managed child +//! and stream its log in this terminal until the user detaches or stops it. + +use super::endpoint_auth::drop_orphaned_endpoint_key_on_already_running; +use super::helper_autostart::ensure_background_helper_running_quiet; +use super::spawn::{ManagedSpawn, spawn_managed_engine_child}; +use super::start::render_endpoint_client_config; +use crate::record_cli_audit_event; +use crate::serve_port; +use crate::service_hints; +use crate::services_text::{ + render_service_action_result, render_service_action_result_with_helper_note, + service_stop_unconfirmed, +}; +use anyhow::{Context, Result, bail}; +use rocm_core::AppPaths; +use rocm_core::{format_http_base_url, process_is_running}; +use rocm_engine_protocol::ResolveModelResponse; +use std::fs; +use std::io::{self, BufRead, Write}; +use std::path::Path; +use std::thread; +use std::time::Duration; + +/// What ended an attached (`--verbose`/`--foreground`) streaming session. Kept +/// as a plain enum, separate from any terminal I/O, so the follow-up action +/// (detach note vs. stop the server) is unit-testable without a TTY. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum AttachOutcome { + /// Ctrl-D: leave the server running and hand the terminal back. + Detach, + /// Ctrl-C: stop the server, then hand the terminal back. + Stop, + /// The engine process exited on its own while we were streaming its log. + ServerExited, +} + +/// Attached serve path for `--verbose`/`--foreground`: spawn the engine as a +/// detached managed child (the same child the background path spawns) and stream +/// its log in this terminal. Unlike the old in-process foreground, the server +/// survives the session — Ctrl-D detaches and leaves it running, Ctrl-C stops it. +/// +/// `stop_service` is how Ctrl-C stops the server: the caller passes the managed +/// stop, so the `service_launch` module does not import the stop path, which +/// itself imports its spawn helpers. It returns the same `stop_server` answer +/// [`finish_attached_stop`] reads. +#[allow(clippy::too_many_arguments)] +pub(crate) fn run_attached_service( + paths: &AppPaths, + engine: &str, + service_id: &str, + requested_model: &str, + resolve: &ResolveModelResponse, + host: &str, + port_request: serve_port::PortRequest, + gpu_indices: &[u32], + runtime_id: Option<&str>, + env_id: Option<&str>, + endpoint_api_key: Option<&str>, + launch_lock: rocm_core::FileLock, + require_api_key: bool, + gpu_refusal_waived_for_reuse: bool, + stop_service: impl FnOnce(&AppPaths, &str) -> Result, +) -> Result<()> { + let spawn = spawn_managed_engine_child( + paths, + engine, + service_id, + requested_model, + resolve, + host, + port_request, + &resolve.device_policy, + gpu_indices, + runtime_id, + env_id, + resolve.engine_recipe.as_ref(), + require_api_key, + gpu_refusal_waived_for_reuse, + ) + // Mirrors `start_managed_service`'s cleanup: a pre-spawn + // failure (port reserved/in use, range exhausted, other bind error) returns + // before any service record exists, so nothing else clears the key just + // stored for this attempt. + .inspect_err(|_| { + drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); + })?; + // The claiming record is persisted (or an existing service was found), so the + // selected GPU is now visible to concurrent auto-selection. Release the launch + // lock before streaming logs, which blocks for the whole attached session. + drop(launch_lock); + + // Attached mode is a managed record too, so it shares the transaction's + // resolved concrete port. + let (service_id, log_path, port, child_pid) = match spawn { + // A server for this engine+model is already live. Don't fight it for the + // port — point the user at the existing one instead of tailing a log we + // did not start. + ManagedSpawn::AlreadyRunning(report) => { + println!("model already being served"); + println!(" service_id: {}", report.service_id); + println!(" endpoint: {}", report.endpoint_url); + println!(" status: {}", report.status); + println!(" logs: rocm logs {}", report.service_id); + println!( + " stop: {}", + service_hints::service_stop_hint(&report.service_id) + ); + drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); + return Ok(()); + } + ManagedSpawn::Spawned { record, child_pid } => ( + service_id.to_owned(), + record.log_path.clone(), + record.port, + child_pid, + ), + }; + + // The child is a managed service that outlives this session once detached. + // The background helper adds one thing for it: with the `server-recover` + // watcher enabled, it health-checks managed servers and restarts failed ones. + // Every duty of the helper is watcher-gated, so with no watcher enabled no + // helper is started. Nothing else depends on it: a self-exited server's + // record is settled by its own supervisor (`supervise_service` writes the + // exit via `record_supervised_exit` in `apps/rocmd/src/supervise.rs`) and on + // every read (`refresh_managed_service_runtime_liveness`), and the dashboard + // has its own embedded daemon. Match the background ordering (spawn, then + // ensure the helper) and keep it quiet so no status line breaks into the + // log stream. + ensure_background_helper_running_quiet(true)?; + + // The resolution detail (model, engine, runtime, GPU, warnings) was already + // printed as the "serve plan" block in `serve()`; extend it with the launch + // coordinates and the streaming hint rather than repeating it. + let endpoint = format!("{}/v1", format_http_base_url(host, port)); + println!(" service_id: {service_id}"); + println!(" endpoint: {endpoint}"); + if let Some(key) = endpoint_api_key { + // Same intentional one-time key display as `print_managed_launch_plain` + // in `service_launch/start.rs`; see its rationale for why the tag below is + // currently inert. + // codeql[rust/cleartext-logging] + print!("{}", render_endpoint_client_config(&endpoint, key)); + } + println!(" streaming engine logs - Ctrl-D detaches (leaves it running), Ctrl-C stops it"); + println!(); + + let outcome = stream_attached_logs(&log_path, child_pid)?; + println!(); + + match outcome { + AttachOutcome::Detach => { + // Audit before printing: a closed stdout makes `println!` panic, + // which must not lose the event. + record_cli_audit_event( + paths, + "service", + "serve_detach", + "info", + format!("detached from attached serve service_id={service_id} endpoint={endpoint}"), + Some(&service_id), + ); + println!("detached - server still running"); + println!(" service_id: {service_id}"); + println!(" endpoint: {endpoint}"); + println!(" list: rocm services"); + println!(" logs: rocm logs {service_id}"); + println!(" stop: {}", service_hints::service_stop_hint(&service_id)); + Ok(()) + } + AttachOutcome::Stop => { + println!("stopping server..."); + let stop = stop_service(paths, &service_id); + finish_attached_stop( + paths, + &service_id, + child_pid, + stop, + rocm_core::terminate_process_tree, + &mut io::stdout().lock(), + &mut io::stderr().lock(), + ) + } + AttachOutcome::ServerExited => { + println!("server process exited"); + println!(" service_id: {service_id}"); + println!(" recent logs: rocm logs {service_id}"); + Ok(()) + } + } +} + +/// Restores cooked terminal mode when dropped, so [`stream_attached_logs`] leaves +/// the terminal usable on every exit path (normal return, `?` error, or panic). +struct RawModeGuard; + +impl Drop for RawModeGuard { + fn drop(&mut self) { + let _ = crossterm::terminal::disable_raw_mode(); + } +} + +/// Map a key press (control modifier + lowercased character) to the attach +/// action it triggers, if any. Factored out of the raw-mode reader loop so the +/// Ctrl-D/Ctrl-C mapping is unit-testable without a terminal. +const fn detach_key_outcome(ctrl: bool, ch: char) -> Option { + if !ctrl { + return None; + } + match ch { + 'c' => Some(AttachOutcome::Stop), + 'd' => Some(AttachOutcome::Detach), + _ => None, + } +} + +/// Follow `log_path` in the terminal until the user presses Ctrl-D (detach) or +/// Ctrl-C (stop), or the engine process exits. Uses crossterm raw mode to +/// capture the keys directly (in raw mode Ctrl-C does not raise SIGINT, so we see +/// it as a key event). When stdin is not a TTY (piped/CI), keystroke capture is +/// impossible, so we follow the log until the process exits instead. +fn stream_attached_logs(log_path: &Path, child_pid: u32) -> Result { + use std::io::IsTerminal as _; + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::mpsc; + + if !std::io::stdin().is_terminal() { + return stream_attached_logs_no_tty(log_path, child_pid); + } + + // Enter raw mode *before* spawning the key reader. In raw mode Ctrl-C arrives + // as a key event instead of SIGINT; if the reader started first, a Ctrl-C in + // that window would kill the CLI outright (leaving the detached child alive + // but printing no detach/stop message). The guard restores cooked mode on + // every exit path (normal return, `?` error, panic). + crossterm::terminal::enable_raw_mode().context("failed to enter raw terminal mode")?; + let _raw_guard = RawModeGuard; + + let stop = Arc::new(AtomicBool::new(false)); + let (tx, rx) = mpsc::channel::(); + + let reader_stop = Arc::clone(&stop); + let reader = thread::spawn(move || { + use crossterm::event::{self, Event, KeyCode, KeyEventKind, KeyModifiers}; + while !reader_stop.load(Ordering::Relaxed) { + match event::poll(Duration::from_millis(100)) { + Ok(true) => match event::read() { + Ok(Event::Key(key)) if key.kind != KeyEventKind::Release => { + let outcome = match key.code { + KeyCode::Char(ch) => detach_key_outcome( + key.modifiers.contains(KeyModifiers::CONTROL), + ch.to_ascii_lowercase(), + ), + _ => None, + }; + if let Some(outcome) = outcome { + let _ = tx.send(outcome); + break; + } + } + Ok(_) => {} + Err(_) => break, + }, + Ok(false) => {} + Err(_) => break, + } + } + }); + + let mut stdout = io::stdout(); + let mut log_reader: Option> = None; + let mut line = String::new(); + let outcome = loop { + if log_reader.is_none() { + log_reader = fs::File::open(log_path).ok().map(io::BufReader::new); + } + if let Some(reader) = log_reader.as_mut() { + loop { + line.clear(); + match reader.read_line(&mut line) { + Ok(0) => break, + Ok(_) => { + // Raw mode disables the terminal's own \n -> \r\n + // translation, so emit an explicit carriage return to + // keep the log left-aligned instead of stair-stepping. + let _ = write!(stdout, "{}\r\n", line.trim_end_matches('\n')); + let _ = stdout.flush(); + } + Err(_) => break, + } + } + } + + if let Ok(signal) = rx.try_recv() { + break signal; + } + if !process_is_running(child_pid) { + break AttachOutcome::ServerExited; + } + thread::sleep(Duration::from_millis(150)); + }; + + stop.store(true, Ordering::Relaxed); + let _ = reader.join(); + Ok(outcome) +} + +/// Non-interactive fallback for [`stream_attached_logs`]: no keystroke capture, +/// so just follow the log until the (detached) engine process exits. A Ctrl-C +/// here delivers SIGINT to this process and leaves the managed server running. +fn stream_attached_logs_no_tty(log_path: &Path, child_pid: u32) -> Result { + let mut stdout = io::stdout(); + let mut log_reader: Option> = None; + let mut line = String::new(); + loop { + if log_reader.is_none() { + log_reader = fs::File::open(log_path).ok().map(io::BufReader::new); + } + if let Some(reader) = log_reader.as_mut() { + loop { + line.clear(); + match reader.read_line(&mut line) { + Ok(0) => break, + Ok(_) => { + let _ = write!(stdout, "{line}"); + let _ = stdout.flush(); + } + Err(_) => break, + } + } + } + if !process_is_running(child_pid) { + return Ok(AttachOutcome::ServerExited); + } + thread::sleep(Duration::from_millis(200)); + } +} + +/// Finish the attached `rocm serve` Ctrl-C stop once the managed stop has +/// answered: what is printed where, what is signalled, what is audited, and +/// how the command exits. Everything but the stop itself and the real +/// signal is here, so a test can drive it. +/// +/// - Confirmed: the report, with the background-helper note `rocm services +/// stop` also prints, goes to `out` and the command succeeds. +/// - Unconfirmed: the same report `rocm services stop` gives, check lines +/// included, becomes the error, so it reaches stderr and the command exits +/// non-zero. Nothing is signalled on the stop's behalf: the stop has already +/// done what it safely could. +/// - The stop itself failed: `signal_child` gets the child this session +/// launched, so Ctrl-C never leaves the model orphaned when the stop path +/// breaks. The stop's error, and the signal's if that fails too, are noted on +/// `err`, and the audit event records whether the signal worked. The command +/// still exits 0, as it always has here. +fn finish_attached_stop( + paths: &AppPaths, + service_id: &str, + child_pid: u32, + stop: Result, + signal_child: impl FnOnce(u32) -> Result<()>, + out: &mut dyn Write, + err: &mut dyn Write, +) -> Result<()> { + match stop { + Ok(result) if service_stop_unconfirmed(&result) => { + record_cli_audit_event( + paths, + "service", + "serve_stop", + "error", + format!("stop of attached serve not confirmed service_id={service_id}"), + Some(service_id), + ); + bail!( + "{}", + render_service_action_result("stop_server", &result).trim_end() + ) + } + Ok(result) => { + record_cli_audit_event( + paths, + "service", + "serve_stop", + "info", + format!("stopped attached serve service_id={service_id}"), + Some(service_id), + ); + write!( + out, + "{}", + render_service_action_result_with_helper_note(paths, "stop_server", &result) + )?; + Ok(()) + } + Err(error) => { + let signal = signal_child(child_pid); + // Record the audit event before writing to stderr, so a failed + // write cannot skip it. + let message = match &signal { + Ok(()) => format!( + "stop failed; signaled attached serve child {child_pid} service_id={service_id}" + ), + Err(signal_error) => format!( + "stop failed; signaling attached serve child {child_pid} also failed: {signal_error:#} service_id={service_id}" + ), + }; + record_cli_audit_event( + paths, + "service", + "serve_stop", + if signal.is_ok() { "warn" } else { "error" }, + message, + Some(service_id), + ); + writeln!(err, " note: {error:#}")?; + if let Err(signal_error) = &signal { + writeln!( + err, + " note: signaling this session's server process {child_pid} directly also failed: {signal_error:#}" + )?; + } + Ok(()) + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::endpoint_keys; + use crate::tests::test_paths; + use rocm_core::ManagedServiceRecord; + use rocm_engine_protocol::DevicePolicy; + + /// A `stop_server` answer whose one recorded process ended with `outcome`. + fn stop_answer(confirmed: bool, outcome: &str) -> serde_json::Value { + crate::services_text::internal_stop_server_value(serde_json::json!({ + "service_id": "svc-attached", + "status": "ready", + "signaled_pids": if outcome == "unverified" { vec![] } else { vec![4242] }, + "pid_outcomes": [{ "pid": 4242, "role": "supervisor_and_engine", "outcome": outcome }], + // An attached serve on a public bind stores a key; an unconfirmed + // stop keeps it. + "endpoint_key_kept": !confirmed, + "stopped": confirmed, + })) + } + + /// Run [`finish_attached_stop`] for `stop`, returning its result, what it + /// wrote to stdout, the PID it signalled (if any) and the audit event. + fn finish( + name: &str, + stop: Result, + ) -> (Result<()>, String, Option, serde_json::Value) { + finish_signalling(name, stop, Ok(())) + } + + /// [`finish`], with the direct signal answering `signal_result`. + fn finish_signalling( + name: &str, + stop: Result, + signal_result: Result<()>, + ) -> (Result<()>, String, Option, serde_json::Value) { + let (result, out, _err, signalled, audit) = finish_capturing(name, stop, signal_result); + (result, out, signalled, audit) + } + + /// [`finish_signalling`], also returning what was written to stderr. + fn finish_capturing( + name: &str, + stop: Result, + signal_result: Result<()>, + ) -> (Result<()>, String, String, Option, serde_json::Value) { + let (root, paths) = test_paths(name); + paths.ensure().unwrap(); + let mut out = Vec::new(); + let mut err = Vec::new(); + let mut signalled = None; + let result = finish_attached_stop( + &paths, + "svc-attached", + 777, + stop, + |pid| { + signalled = Some(pid); + signal_result + }, + &mut out, + &mut err, + ); + let audit_text = fs::read_to_string(paths.audit_events_path()).expect("audit log"); + let audit = serde_json::from_str(audit_text.lines().last().expect("an audit event")) + .expect("audit json"); + let _ = fs::remove_dir_all(root); + ( + result, + String::from_utf8(out).unwrap(), + String::from_utf8(err).unwrap(), + signalled, + audit, + ) + } + + #[test] + fn attached_ctrl_c_reports_a_confirmed_stop_on_stdout() { + let (result, out, signalled, audit) = + finish("attached-confirmed", Ok(stop_answer(true, "graceful"))); + result.expect("a confirmed stop succeeds"); + assert!(out.contains("Local server stopped"), "{out}"); + assert_eq!(signalled, None); + assert_eq!(audit["level"], "info", "{audit}"); + assert_eq!( + audit["message"], + "stopped attached serve service_id=svc-attached" + ); + } + + #[test] + fn attached_ctrl_c_fails_an_unconfirmed_stop_like_services_stop_and_signals_nothing() { + // The same report `rocm services stop` gives, as the error (stderr, + // non-zero exit), naming no retry; nothing is signalled on the stop's + // behalf and nothing reaches stdout. + let answer = stop_answer(false, "timed_out"); + let (result, out, signalled, audit) = finish("attached-timed-out", Ok(answer.clone())); + let error = format!("{:#}", result.expect_err("an unconfirmed stop fails")); + assert_eq!( + error, + render_service_action_result("stop_server", &answer).trim_end(), + "the report is `rocm services stop`'s own" + ); + assert!( + error.contains( + "process 4242 (supervisor_and_engine) or a process under it did not exit" + ), + "{error}" + ); + assert!( + error.lines().any(|line| line + == " kept: the server record, which still names its processes, and its endpoint key, which the server may still be enforcing"), + "{error}" + ); + crate::services_text::tests::assert_no_retry_line(&error); + assert!(out.is_empty(), "nothing on stdout: {out}"); + assert_eq!(signalled, None, "an unconfirmed stop signals nothing more"); + assert_eq!(audit["level"], "error", "{audit}"); + assert_eq!( + audit["message"], + "stop of attached serve not confirmed service_id=svc-attached" + ); + } + + #[test] + fn attached_ctrl_c_names_a_check_for_an_unverified_process() { + let (result, out, signalled, audit) = + finish("attached-unverified", Ok(stop_answer(false, "unverified"))); + let error = format!("{:#}", result.expect_err("an unverified stop fails")); + assert!(error.contains("not signaled: process 4242"), "{error}"); + assert!(error.contains("check: `ps -p 4242 -o pid,args`"), "{error}"); + crate::services_text::tests::assert_no_retry_line(&error); + assert!(out.is_empty(), "{out}"); + assert_eq!(signalled, None); + assert_eq!(audit["level"], "error", "{audit}"); + } + + #[test] + fn attached_ctrl_c_signals_its_own_child_when_the_stop_itself_fails() { + let (result, out, err, signalled, audit) = finish_capturing( + "attached-stop-error", + Err(anyhow::anyhow!("managed service `svc-attached` not found") + .context("failed to stop svc-attached")), + Ok(()), + ); + result.expect("a failed stop falls back and still exits cleanly, as before"); + assert_eq!(signalled, Some(777)); + assert_eq!( + err, " note: failed to stop svc-attached: managed service `svc-attached` not found\n", + "the stop's error goes to stderr, and a signal that worked adds nothing" + ); + assert!(out.is_empty(), "nothing on stdout: {out}"); + assert_eq!(audit["level"], "warn", "{audit}"); + assert_eq!( + audit["message"], + "stop failed; signaled attached serve child 777 service_id=svc-attached" + ); + + // The direct signal failing too is printed on stderr and recorded, with + // its error. + let (result, _, err, signalled, audit) = finish_capturing( + "attached-stop-error-signal-fails", + Err(anyhow::anyhow!("managed service `svc-attached` not found")), + Err(anyhow::anyhow!("no such process")), + ); + result.expect("still exits 0, as on main"); + assert_eq!( + err, + " note: managed service `svc-attached` not found\n \ + note: signaling this session's server process 777 directly also failed: no such process\n", + "both notes go to stderr" + ); + assert_eq!(signalled, Some(777)); + assert_eq!(audit["level"], "error", "{audit}"); + assert_eq!( + audit["message"], + "stop failed; signaling attached serve child 777 also failed: no such process service_id=svc-attached" + ); + } + + /// A writer whose every write fails, standing in for a closed stdout/stderr. + struct BrokenWriter; + + impl Write for BrokenWriter { + fn write(&mut self, _: &[u8]) -> std::io::Result { + Err(std::io::Error::from(std::io::ErrorKind::BrokenPipe)) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + #[test] + fn attached_ctrl_c_records_the_audit_event_even_when_writing_the_outcome_fails() { + // One case per branch that writes after deciding: a confirmed stop + // (stdout) and a stop that itself failed (stderr notes). The write + // error is returned, but the audit event is already on disk. + let cases: [(&str, Result, &str); 2] = [ + ( + "attached-broken-out", + Ok(stop_answer(true, "graceful")), + "stopped attached serve service_id=svc-attached", + ), + ( + "attached-broken-err", + Err(anyhow::anyhow!("managed service `svc-attached` not found")), + "stop failed; signaled attached serve child 777 service_id=svc-attached", + ), + ]; + for (name, stop, expected_message) in cases { + let (root, paths) = test_paths(name); + paths.ensure().unwrap(); + let result = finish_attached_stop( + &paths, + "svc-attached", + 777, + stop, + |_| Ok(()), + &mut BrokenWriter, + &mut BrokenWriter, + ); + let audit_text = fs::read_to_string(paths.audit_events_path()).unwrap_or_default(); + let _ = fs::remove_dir_all(root); + assert!(result.is_err(), "{name}: the write error is surfaced"); + let audit: serde_json::Value = serde_json::from_str( + audit_text + .lines() + .last() + .unwrap_or_else(|| panic!("{name}: no audit event recorded")), + ) + .expect("audit json"); + assert_eq!(audit["message"], expected_message, "{name}: {audit}"); + } + } + + #[test] + fn detach_key_ctrl_d_detaches_ctrl_c_stops() { + assert_eq!(detach_key_outcome(true, 'd'), Some(AttachOutcome::Detach)); + assert_eq!(detach_key_outcome(true, 'c'), Some(AttachOutcome::Stop)); + } + + #[test] + fn detach_key_ignores_plain_and_unrelated_keys() { + // Without the control modifier, `d`/`c` are ordinary log-scroll input. + assert_eq!(detach_key_outcome(false, 'd'), None); + assert_eq!(detach_key_outcome(false, 'c'), None); + // Other control combos are not detach/stop triggers. + assert_eq!(detach_key_outcome(true, 'q'), None); + assert_eq!(detach_key_outcome(true, 'z'), None); + } + + #[test] + fn attached_launch_failing_before_spawn_leaves_no_orphaned_endpoint_key() -> Result<()> { + // The background path's cleanup is the `.inspect_err` in + // start_managed_service. run_attached_service is the other caller of + // spawn_managed_engine_child and needs the identical .inspect_err() + // cleanup, not a simulated one: drive the real function end to end. + let (root, paths) = test_paths("orphaned-key-attached-reserved-port"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + let mut other = ManagedServiceRecord::new( + &paths, + "lemonade-llama-1000", + "lemonade", + "llama", + "llama-canonical", + "127.0.0.1", + 11514, + "managed", + std::process::id(), + None, + None, + None, + ); + other.status = "ready".to_owned(); + other.engine_pid = Some(std::process::id()); + other.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let service_id = "lemonade-qwen-attached-2000"; + endpoint_keys::store_endpoint_api_key(&paths, service_id, "test-key")?; + let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + + let result = run_attached_service( + &paths, + "lemonade", + service_id, + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Explicit(11514), + &[], + None, + None, + Some("test-key"), + launch_lock, + false, + false, + |_, _| unreachable!("the launch fails before a Ctrl-C could stop it"), + ); + let Err(error) = result else { + let _ = fs::remove_dir_all(root); + panic!("explicit port already reserved by a live service must be rejected") + }; + assert!( + error + .to_string() + .contains("already reserved by managed service"), + "unexpected error: {error}" + ); + assert!( + endpoint_keys::endpoint_api_key(&paths, service_id).is_none(), + "a pre-spawn failure must drop the key this attempt staged before launching" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } +} diff --git a/apps/rocm/src/service_launch/endpoint_auth.rs b/apps/rocm/src/service_launch/endpoint_auth.rs new file mode 100644 index 0000000000..6a647314bc --- /dev/null +++ b/apps/rocm/src/service_launch/endpoint_auth.rs @@ -0,0 +1,377 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Bind-host validation and the endpoint API key checks that gate a public +//! bind: [`validate_bind_host`], [`resolve_endpoint_auth`] and the guards that +//! keep a respawn from reopening a service without its key. + +use crate::endpoint_keys; +use anyhow::{Result, bail}; +use rocm_core::AppPaths; + +pub(crate) fn validate_bind_host(host: &str, allow_public_bind: bool) -> Result<()> { + if !is_loopback_host(host) && !allow_public_bind { + bail!( + "`rocm serve --host {host}` is not loopback; pass `--allow-public-bind` before binding a non-local interface" + ); + } + Ok(()) +} + +/// Inverse of [`rocm_engine_protocol::is_public_bind_host`], which owns the +/// policy so `rocmd` classifies a recorded `host` identically when it respawns +/// the service. +pub(crate) fn is_loopback_host(host: &str) -> bool { + !rocm_engine_protocol::is_public_bind_host(host) +} + +/// Resolve the API key that will guard this endpoint, applying the +/// loopback-vs-public policy for `rocm serve`. +/// +/// - **Loopback host** → `None`: local serving stays credential-free (the +/// unchanged default). Any key supplied for a loopback bind is ignored and +/// nothing is persisted — loopback needs no auth. +/// - **Public host** → `Some(key)`: use the user-supplied key when present, +/// otherwise generate a strong random one so a public endpoint can never come +/// up anonymous. An empty/whitespace supplied key is rejected rather than +/// silently treated as "no auth". +/// - **`required`** → treat a loopback bind as public for this purpose. +/// +/// That last case exists because "loopback" is a statement about the bind +/// address, not about who can reach the port. Publishing the port onto a +/// tailnet, proxying it, or mapping it out of a container all leave the bind +/// loopback while widening the audience — and the policy above would then hand +/// out an unauthenticated endpoint. Whoever widens the reach is responsible for +/// asking for the credential, so this is an explicit flag rather than a guess. +pub(crate) fn resolve_endpoint_auth( + host: &str, + supplied: Option<&str>, + required: bool, +) -> Result> { + if is_loopback_host(host) && !required { + return Ok(None); + } + match supplied { + Some(raw) => { + let trimmed = raw.trim(); + if trimmed.is_empty() { + bail!( + "`rocm serve --api-key` (or ROCM_SERVE_API_KEY) was empty; a public \ + endpoint must be protected by a non-empty API key" + ); + } + // The key is later interpolated verbatim into raw `Authorization: + // Bearer {key}\r\n` header lines; reject a control character (e.g. an + // embedded CR/LF) here so a crafted key cannot inject extra headers. + if rocm_core::endpoint_api_key_has_forbidden_chars(trimmed) { + bail!( + "`rocm serve --api-key` (or ROCM_SERVE_API_KEY) contained a control \ + character such as a carriage return or newline; an endpoint API key \ + must be a single line of printable characters" + ); + } + Ok(Some(trimmed.to_owned())) + } + None => Ok(Some(rocm_core::generate_endpoint_api_key())), + } +} + +/// When an equivalent managed service is already running, or the launch fails +/// before any record exists, the endpoint key serve() freshly stored for this +/// attempt is unused — drop it so it is not orphaned in storage. The already-running service keeps its own key. +/// Best-effort and idempotent; a loopback attempt (`freshly_stored == None`) +/// is a no-op. +pub(crate) fn drop_orphaned_endpoint_key_on_already_running( + paths: &AppPaths, + service_id: &str, + freshly_stored: Option<&str>, +) { + if freshly_stored.is_some() { + endpoint_keys::clear_endpoint_api_key(paths, service_id); + } +} + +/// Reject engine/platform combinations that cannot enforce a public endpoint's +/// API key, so a public bind fails closed instead of coming up unauthenticated. +/// +/// The one such case today: Windows managed Lemonade. Its server reads the +/// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn +/// primitive only carries path-valued env overrides, so the key never reaches it. +/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds +/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter +/// so both branches are unit-testable off-Windows. +pub(crate) fn ensure_public_bind_engine_supported( + engine: &str, + public_bind: bool, + is_windows: bool, +) -> Result<()> { + if public_bind && is_windows && engine == "lemonade" { + bail!( + "public binding with the lemonade engine is not supported on Windows: the endpoint \ + API key cannot be enforced there. Use `--engine vllm`, or bind a loopback host \ + (the default 127.0.0.1)." + ); + } + Ok(()) +} + +/// Refuse to (re)spawn a managed service recorded on a public host when its +/// endpoint API key is gone, so a respawn cannot reopen the endpoint anonymously. +/// +/// `serve()` applies the loopback-vs-public policy once, via +/// [`resolve_endpoint_auth`], and persists the resulting key. Every later spawn +/// — `rocm services restart`, and `rocmd`'s recovery supervisor — reads that key +/// file back and would otherwise treat "no key file" as "no auth wanted", +/// silently downgrading a protected public endpoint to an open one. The real +/// invariant is a property of the *host*, not of the file: a non-loopback bind +/// must always be authenticated. +/// +/// The gap is reachable through ordinary commands, because a stop deletes the +/// key file and `rocm services restart` accepts a stopped service id (its help +/// points at `rocm services list --all`). +/// +/// `key_present` is a plain `bool` rather than a path so both branches are +/// unit-testable without touching the filesystem, mirroring `is_windows` in +/// [`ensure_public_bind_engine_supported`]. +pub(crate) fn ensure_public_service_has_endpoint_key( + host: &str, + key_present: bool, + requires_api_key: bool, +) -> Result<()> { + // Two ways a service can need a key. A public bind is the obvious one. The + // other is a service that asked for auth on a loopback bind, because + // something outside this process republishes the port — a tailnet publish + // survives a reboot, let alone a restart, so "loopback" stops meaning + // "only this machine" and the bind address can no longer be trusted to + // answer the question on its own. + if requires_api_key && !key_present { + bail!( + "managed service was launched with `--require-api-key` but has no endpoint API key, \ + so restarting it would reopen it without authentication. Something outside this \ + machine may still be publishing its port. The key is dropped when a service stops \ + and cannot be recovered. Launch it again with \ + `rocm serve --require-api-key` (add `--api-key `, or set ROCM_SERVE_API_KEY, \ + to choose the key instead of generating one)." + ); + } + if rocm_engine_protocol::is_public_bind_host(host) && !key_present { + bail!( + "managed service is bound to the public host `{host}` but has no endpoint API key, \ + so restarting it would reopen it without authentication. The key is dropped when a \ + service stops and cannot be recovered. Launch it again with \ + `rocm serve --host {host} --port --allow-public-bind` (add `--api-key `, or set \ + ROCM_SERVE_API_KEY, to choose the key instead of generating one)." + ); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::tests::test_paths; + use std::fs; + + #[test] + fn serve_bind_validation_requires_public_ack() { + validate_bind_host("127.0.0.1", false).unwrap(); + validate_bind_host("localhost", false).unwrap(); + validate_bind_host("::1", false).unwrap(); + let error = validate_bind_host("0.0.0.0", false).unwrap_err(); + assert!( + error.to_string().contains("--allow-public-bind"), + "{error:#}" + ); + validate_bind_host("0.0.0.0", true).unwrap(); + } + + #[test] + fn resolve_endpoint_auth_loopback_stays_credential_free() { + // Loopback binds never require auth, even if a key is supplied. + for host in ["127.0.0.1", "localhost", "::1"] { + assert_eq!(resolve_endpoint_auth(host, None, false).unwrap(), None); + assert_eq!( + resolve_endpoint_auth(host, Some("ignored"), false).unwrap(), + None + ); + } + } + + #[test] + fn resolve_endpoint_auth_loopback_can_be_required_when_something_republishes_it() { + // "Loopback" describes the bind address, not who can reach the port. A + // tailnet publish, a proxy, or a container port map all leave the bind + // loopback while widening the audience, and the default policy would + // hand out an unauthenticated endpoint. Whoever widens the reach asks + // for the credential explicitly. + for host in ["127.0.0.1", "localhost", "::1"] { + let generated = resolve_endpoint_auth(host, None, true) + .unwrap() + .expect("a required key must be generated, not skipped"); + assert!(!generated.trim().is_empty()); + + assert_eq!( + resolve_endpoint_auth(host, Some("supplied-key"), true).unwrap(), + Some("supplied-key".to_owned()), + "a supplied key must be honoured rather than ignored as it is by default" + ); + } + + // The same validation a public bind gets: an empty key is a refusal, not + // a silent downgrade to no auth. + assert!(resolve_endpoint_auth("127.0.0.1", Some(" "), true).is_err()); + } + + #[test] + fn resolve_endpoint_auth_public_uses_supplied_key_trimmed() { + let key = resolve_endpoint_auth("0.0.0.0", Some(" my-key "), false) + .unwrap() + .expect("public bind must have a key"); + assert_eq!(key, "my-key"); + } + + #[test] + fn resolve_endpoint_auth_public_generates_key_when_absent() { + let key = resolve_endpoint_auth("0.0.0.0", None, false) + .unwrap() + .expect("public bind must generate a key"); + assert_eq!(key.len(), 48); + assert!(key.chars().all(|c| c.is_ascii_alphanumeric())); + } + + #[test] + fn resolve_endpoint_auth_public_rejects_empty_supplied_key() { + let error = resolve_endpoint_auth("0.0.0.0", Some(" "), false).unwrap_err(); + assert!(error.to_string().contains("non-empty"), "{error:#}"); + } + + #[test] + fn resolve_endpoint_auth_public_rejects_embedded_crlf() { + // A supplied key survives `trim()` with embedded CR/LF intact and would + // otherwise be interpolated into a raw `Authorization: Bearer` header, + // injecting an extra header line. It must be rejected at input validation. + for supplied in [ + "good-key\r\nX-Injected: value", + "good-key\nmore", + "line\rreturn", + ] { + let error = resolve_endpoint_auth("0.0.0.0", Some(supplied), false).unwrap_err(); + assert!(error.to_string().contains("control character"), "{error:#}"); + } + } + + #[test] + fn drop_orphaned_endpoint_key_on_already_running_clears_stored_key() { + let (root, paths) = test_paths("drop-orphaned-key-stored"); + let service_id = "svc-orphaned"; + endpoint_keys::store_endpoint_api_key(&paths, service_id, "secret-key").unwrap(); + + drop_orphaned_endpoint_key_on_already_running(&paths, service_id, Some("secret-key")); + + assert_eq!(endpoint_keys::endpoint_api_key(&paths, service_id), None); + assert!(!endpoint_keys::endpoint_key_file_path(&paths, service_id).exists()); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn drop_orphaned_endpoint_key_on_already_running_is_noop_for_loopback() { + // A loopback attempt never stores a key (`freshly_stored == None`), so the + // helper must not panic or error, and no file must appear. + let (root, paths) = test_paths("drop-orphaned-key-loopback"); + let service_id = "svc-loopback"; + + drop_orphaned_endpoint_key_on_already_running(&paths, service_id, None); + + assert_eq!(endpoint_keys::endpoint_api_key(&paths, service_id), None); + assert!(!endpoint_keys::endpoint_key_file_path(&paths, service_id).exists()); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn public_bind_fails_closed_for_windows_lemonade_only() { + // Windows + Lemonade + public bind: refuse (cannot enforce the key). + let error = ensure_public_bind_engine_supported("lemonade", true, true).unwrap_err(); + assert!(error.to_string().contains("lemonade"), "{error:#}"); + // Every other combination is allowed: + ensure_public_bind_engine_supported("vllm", true, true).unwrap(); // vLLM enforces auth on Windows + ensure_public_bind_engine_supported("lemonade", true, false).unwrap(); // non-Windows + ensure_public_bind_engine_supported("lemonade", false, true).unwrap(); // loopback needs no key + } + + #[test] + fn a_public_bind_is_refused_with_the_command_that_restores_it() { + // `resolve_endpoint_auth` mints a key for every non-loopback bind whether + // or not auth was demanded, so deriving `requires_api_key` from key-file + // presence marked ordinary public binds as having asked for it. The guard + // tests that field first, so those services were refused with a message + // naming a flag they never passed and a relaunch command that drops + // `--allow-public-bind` — coming back on loopback instead. + // + // A plain `--host 0.0.0.0 --allow-public-bind` launch: key file present, + // `--require-api-key` never passed. + let error = ensure_public_service_has_endpoint_key("0.0.0.0", false, false) + .expect_err("a public bind with no key must be refused"); + let rendered = format!("{error:#}"); + assert!( + rendered.contains("--allow-public-bind"), + "the refusal must name the command that restores the public bind: {rendered}" + ); + assert!( + !rendered.contains("--require-api-key"), + "a service that never passed the flag must not be told it did: {rendered}" + ); + + // And the loopback-with-forced-auth case still reports its own reason. + let error = ensure_public_service_has_endpoint_key("127.0.0.1", false, true) + .expect_err("a service that demanded auth must not come back without it"); + assert!( + format!("{error:#}").contains("--require-api-key"), + "{error:#}" + ); + } + + #[test] + fn respawn_fails_closed_for_a_public_service_whose_key_is_gone() { + // A stop deletes the key file, so a later restart of a public service + // would otherwise respawn it with no auth at all. + let error = ensure_public_service_has_endpoint_key("0.0.0.0", false, false).unwrap_err(); + let message = error.to_string(); + assert!(message.contains("0.0.0.0"), "{error:#}"); + assert!(message.contains("without authentication"), "{error:#}"); + // Actionable: name the command that mints a fresh key. + assert!(message.contains("--allow-public-bind"), "{error:#}"); + + // A public service that still has its key restarts normally. + ensure_public_service_has_endpoint_key("0.0.0.0", true, false).unwrap(); + } + + #[test] + fn respawn_allows_loopback_services_without_an_endpoint_key() { + // Loopback stays credential-free, so every accepted spelling must pass + // the guard with no key present — when nothing asked for auth. + for host in ["127.0.0.1", "localhost", "::1"] { + ensure_public_service_has_endpoint_key(host, false, false) + .unwrap_or_else(|error| panic!("{host} must not require a key: {error:#}")); + } + } + + #[test] + fn respawn_refuses_a_loopback_service_that_was_launched_with_a_key() { + // The hole this closes: a loopback bind that something else republishes + // — a tailnet publish, a proxy, a container port map. The publish + // outlives the process, so a restart after the key was dropped would + // reopen a reachable endpoint with no authentication, and the bind + // address gives the guard no way to notice. + for host in ["127.0.0.1", "localhost", "::1"] { + let error = ensure_public_service_has_endpoint_key(host, false, true) + .expect_err("a service launched with a key must not restart without one"); + let message = format!("{error:#}"); + assert!(message.contains("without authentication"), "{message}"); + assert!(message.contains("--require-api-key"), "{message}"); + } + + // With its key still present it restarts normally. + ensure_public_service_has_endpoint_key("127.0.0.1", true, true).unwrap(); + } +} diff --git a/apps/rocm/src/service_launch/helper_autostart.rs b/apps/rocm/src/service_launch/helper_autostart.rs new file mode 100644 index 0000000000..7a54cb5134 --- /dev/null +++ b/apps/rocm/src/service_launch/helper_autostart.rs @@ -0,0 +1,477 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Starting the background automation helper (`rocm daemon`) when a watcher needs +//! it, with the autostart claim that keeps concurrent callers from spawning two. + +#[cfg(not(windows))] +use super::spawn::{ + attach_background_stdio, detach_background_command, managed_service_process_command, +}; +use crate::background_helper::background_helper_already_running; +#[cfg(not(windows))] +use crate::engine_runtime_select::apply_app_path_env; +use crate::engine_runtime_select::managed_service_launcher_path; +#[cfg(windows)] +use crate::engine_runtime_select::{app_path_env_var_refs, app_path_env_var_values}; +use anyhow::{Context, Result}; +use rocm_core::AppPaths; +use std::path::Path; +#[cfg(not(windows))] +use std::process::Stdio; + +/// Shared daemon-lifecycle entrypoint: ensures the background automation helper +/// (`rocm daemon`) is running when an automation watcher needs it, spawning it +/// detached if not. Liveness is read from the file-based automation runtime +/// state. Intentionally `pub(crate)` — reused by both the `serve --managed` path +/// and `automations enable`. Only the spawn result itself (`command.spawn()` / +/// `spawn_detached_no_inherit`) is logged rather than propagated; setup errors +/// (path discovery, config load, stdio attach) still return `Err`. +/// +/// The daemon's only work is running enabled automation watchers (server +/// recovery included — `server-recover` is one of them and is off by default), +/// and it exits on its own once none is enabled. So with no watcher enabled +/// nothing is spawned: a `serve` would otherwise leave a no-op daemon running +/// long after its service was stopped (ROCMAI-827). +pub(crate) fn ensure_background_helper_running() -> Result<()> { + ensure_background_helper_running_quiet(false) +} + +/// As [`ensure_background_helper_running`], but suppresses the stdout status line +/// when `quiet` is set. The interactive `rocm serve` summary path uses `quiet` so +/// the daemon-spawn note does not appear above the deployment summary table. +pub(crate) fn ensure_background_helper_running_quiet(quiet: bool) -> Result<()> { + let paths = AppPaths::discover()?; + ensure_background_helper_running_with(&paths, quiet, spawn_background_helper, &|| {}) +} + +/// The body of [`ensure_background_helper_running_quiet`], with the actual +/// process spawn injected so the decision around it is testable without +/// launching a daemon. `spawn` returns `Err` for a setup failure (propagated) +/// and `Ok(Err(..))` for a failed spawn (reported, not fatal). `on_lock_busy` is +/// called each time the autostart lock is found held by another process, which +/// lets a test wait for this call to be genuinely queued behind it. +fn ensure_background_helper_running_with( + paths: &AppPaths, + quiet: bool, + spawn: impl FnOnce(&AppPaths) -> Result>, + on_lock_busy: &dyn Fn(), +) -> Result<()> { + // An unreadable config is "no watcher": this also runs after `serve` has + // already launched its child, where failing would report a launch that + // happened as an error. + if !rocm_core::RocmCliConfig::load(paths).is_ok_and(|config| config.automation_daemon_enabled()) + { + return Ok(()); + } + + // Two concurrent callers (e.g. two `rocm serve`) can both read "not running" + // and each spawn a daemon. Serialize the decision on a lock file — the first + // holder spawns, later holders observe the now-running daemon and return + // without spawning. + // + // The "already running" read is deliberately made only under the lock, with + // no lock-free fast path. The daemon exits by itself once the last watcher + // is disabled, and it decides that under this same lock and publishes + // `running = false` before releasing it. Caller order is: save the enabling + // config, then come here. A lock-free read could still see the daemon that is + // about to exit as running, spawn nothing, and leave a watcher enabled with + // no daemon; under the lock the daemon has either seen the new config and + // stays, or has already published that it stopped. + let _autostart_lock = loop { + if let Some(lock) = + rocm_core::FileLock::try_acquire(paths.automation_autostart_lock_path())? + { + break lock; + } + on_lock_busy(); + std::thread::sleep(std::time::Duration::from_millis(10)); + }; + if background_helper_already_running(paths)? { + return Ok(()); + } + + // The lock alone does not close the window: the spawned daemon does not + // publish its `running` runtime state until well after `spawn()` (clap parse, + // runtime build, config load, banner flush). A second caller that acquires + // this lock during that gap still sees "not running" and would spawn a + // duplicate. Bridge the gap with a short-lived claim recording the child PID + // and spawn time: a holder that finds a live, recent claim defers instead. + let claim_path = paths.automation_autostart_claim_path(); + if autostart_spawn_in_flight( + read_autostart_claim(&claim_path), + now_unix_millis(), + AUTOSTART_CLAIM_TTL_MS, + rocm_core::process_is_running, + ) { + return Ok(()); + } + + match spawn(paths)? { + Ok(daemon_pid) => { + // Record the claim before returning (and thus releasing the lock) so a + // concurrent holder in the spawn→publish window defers. Best-effort: a + // failed write only reopens the original, already-tolerated race. + let _ = write_autostart_claim( + &claim_path, + AutostartClaim { + daemon_pid, + spawned_at_ms: now_unix_millis(), + }, + ); + if !quiet { + println!(" helper: started background automation daemon"); + } + } + Err(error) if !quiet => { + println!(" helper: could not start background automation daemon: {error}"); + } + Err(_) => {} + } + Ok(()) +} + +/// Spawn `rocm daemon` detached from this process. The outer `Result` is a setup +/// failure (resolving the executable, attaching stdio); the inner one is the +/// spawn itself. +fn spawn_background_helper(paths: &AppPaths) -> Result> { + let exe = managed_service_launcher_path() + .context("failed to resolve current rocm executable path")?; + let args = vec!["daemon".to_owned()]; + #[cfg(windows)] + let spawn_result = { + let env_values = app_path_env_var_values(paths, None); + let env_refs = app_path_env_var_refs(&env_values); + rocm_core::spawn_detached_no_inherit(&exe, &args, &env_refs) + }; + #[cfg(not(windows))] + let spawn_result = { + let mut command = managed_service_process_command(&exe, &args); + command.stdin(Stdio::null()); + attach_background_stdio(&mut command, None)?; + detach_background_command(&mut command); + apply_app_path_env(&mut command, paths); + command + .spawn() + .map(|child| child.id()) + .map_err(anyhow::Error::from) + }; + Ok(spawn_result) +} + +/// How long an autostart claim is honoured before it is treated as stale even if +/// its recorded PID is still alive. Comfortably longer than a cold daemon boot +/// (clap parse → runtime build → config load → state publish) yet short enough +/// that a crashed spawn cannot suppress autostart for long. +const AUTOSTART_CLAIM_TTL_MS: u128 = 30_000; + +/// A just-spawned daemon's autostart claim: the child PID and the wall-clock time +/// (milliseconds since the Unix epoch) the spawn was recorded. It lets a +/// concurrent autostart holder distinguish a live, in-flight spawn from a stale +/// leftover. Serialized as a single `" "` line — no dependency and +/// trivially forward-compatible. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct AutostartClaim { + daemon_pid: u32, + spawned_at_ms: u128, +} + +/// Milliseconds since the Unix epoch, or `0` if the clock is before the epoch +/// (which only makes a fresh claim look old — safe, it just permits a respawn). +fn now_unix_millis() -> u128 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |elapsed| elapsed.as_millis()) +} + +/// Read an autostart claim, returning `None` when the file is absent or +/// unparseable (either is treated as "no claim", so a respawn is permitted). +fn read_autostart_claim(path: &Path) -> Option { + let text = std::fs::read_to_string(path).ok()?; + let mut parts = text.split_whitespace(); + let daemon_pid = parts.next()?.parse().ok()?; + let spawned_at_ms = parts.next()?.parse().ok()?; + Some(AutostartClaim { + daemon_pid, + spawned_at_ms, + }) +} + +/// Write an autostart claim as `" "`. Best-effort at the call site. +fn write_autostart_claim(path: &Path, claim: AutostartClaim) -> std::io::Result<()> { + std::fs::write( + path, + format!("{} {}", claim.daemon_pid, claim.spawned_at_ms), + ) +} + +/// Whether an existing autostart `claim` means a daemon spawn is still in flight, +/// so the current lock holder should defer rather than spawn a duplicate. A claim +/// counts as in-flight only while its child PID is alive *and* it is younger than +/// `ttl_ms` — the TTL bounds how long a crashed spawn (or a PID later reused by an +/// unrelated process) can suppress autostart. `pid_alive` is injected so the +/// decision is unit-testable without a live process. +fn autostart_spawn_in_flight( + claim: Option, + now_ms: u128, + ttl_ms: u128, + pid_alive: impl Fn(u32) -> bool, +) -> bool { + claim.is_some_and(|claim| { + now_ms.saturating_sub(claim.spawned_at_ms) < ttl_ms && pid_alive(claim.daemon_pid) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_support::{runtime_state, set_helper_facts}; + use crate::tests::test_paths; + use std::fs; + + /// ROCMAI-827: `rocm serve` started the background daemon on every launch, + /// and with no automation watcher enabled that daemon had nothing to do yet + /// kept running after every service was stopped. With the default config + /// (no watcher enabled) the helper must not be spawned at all. + #[test] + fn background_helper_not_spawned_when_no_automation_is_enabled() { + let (root, paths) = test_paths("helper-no-automation"); + paths.ensure().expect("create app dirs"); + // The stored flag alone is not "a watcher is enabled": nothing would run. + let mut config = rocm_core::RocmCliConfig::default(); + config.automations.daemon_enabled = true; + config.save(&paths).expect("save default config"); + + let mut spawned = false; + ensure_background_helper_running_with( + &paths, + true, + |_| { + spawned = true; + Ok(Ok(4242)) + }, + &|| {}, + ) + .expect("ensure helper"); + + let claim_written = paths.automation_autostart_claim_path().exists(); + let _ = fs::remove_dir_all(&root); + assert!( + !spawned, + "no watcher enabled ⇒ the daemon has no work and must not be started" + ); + assert!(!claim_written, "no spawn ⇒ no autostart claim"); + } + + /// The other half of the gate: once a watcher is enabled the helper is still + /// started (exactly once) and its autostart claim recorded. + #[test] + fn background_helper_spawned_when_an_automation_is_enabled() { + let (root, paths) = test_paths("helper-automation-enabled"); + paths.ensure().expect("create app dirs"); + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = true; + config.save(&paths).expect("save config"); + + let mut spawns = 0; + ensure_background_helper_running_with( + &paths, + true, + |_| { + spawns += 1; + Ok(Ok(4242)) + }, + &|| {}, + ) + .expect("ensure helper"); + + let claim = read_autostart_claim(&paths.automation_autostart_claim_path()); + let _ = fs::remove_dir_all(&root); + assert_eq!(spawns, 1, "an enabled watcher needs the daemon"); + assert_eq!(claim.map(|claim| claim.daemon_pid), Some(4242)); + } + + /// ROCMAI-827 race, enable side: the daemon that is about to exit still has + /// `running = true` and a live pid in its state when `automations enable` + /// (config already saved) asks for a helper. It decides under the autostart + /// lock and publishes "stopped" before releasing it, so the enable must wait + /// on that lock and then start a new helper. A lock-free "already running" + /// read would return here without spawning, leaving a watcher with no daemon. + #[test] + fn background_helper_is_started_when_the_exiting_daemon_publishes_stopped() { + let (root, paths) = test_paths("helper-exit-window"); + paths.ensure().expect("create app dirs"); + let mut config = rocm_core::RocmCliConfig::default(); + config.watcher_config_mut("server-recover").enabled = true; + config.save(&paths).expect("save config"); + // The old daemon: alive (this process stands in) and still `running`. + runtime_state(true, std::process::id()) + .write(&paths) + .expect("write runtime state"); + + // The exiting daemon holds the autostart lock while it decides. + let exiting_daemon_lock = + rocm_core::FileLock::acquire(paths.automation_autostart_lock_path()) + .expect("hold the lock"); + let enable_paths = paths.clone(); + let queued_flag = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let queued = queued_flag.clone(); + let enable = std::thread::spawn(move || { + let mut spawns = 0; + ensure_background_helper_running_with( + &enable_paths, + true, + |_| { + spawns += 1; + Ok(Ok(4242)) + }, + &|| queued.store(true, std::sync::atomic::Ordering::SeqCst), + ) + .expect("ensure helper"); + spawns + }); + // Wait until the enable path has actually found the lock held and queued + // behind it, rather than assuming how long that takes. + let deadline = std::time::Instant::now() + std::time::Duration::from_mins(1); + // A caller that skipped the lock finishes without ever queuing. + while !queued_flag.load(std::sync::atomic::Ordering::SeqCst) && !enable.is_finished() { + assert!(std::time::Instant::now() < deadline, "enable never queued"); + std::thread::yield_now(); + } + runtime_state(false, std::process::id()) + .write(&paths) + .expect("daemon publishes stopped"); + drop(exiting_daemon_lock); + + let spawns = enable.join().expect("enable thread"); + let _ = fs::remove_dir_all(&root); + assert_eq!( + spawns, 1, + "the enable must start a helper once the old one has published it stopped" + ); + } + + fn stop_result() -> serde_json::Value { + serde_json::json!({"output": {"result": { + "service": {"service_id": "svc-a", "status": "stopped"}, + "signaled_pids": [1] + }}}) + } + + /// `rocm services stop` prints the note for a stop only, and only the variant + /// that matches the state. + #[test] + fn stop_output_carries_the_helper_note_that_matches_the_state() { + use crate::services_text::render_service_action_result_with_helper_note as render; + + let (root, paths) = test_paths("stop-helper-note"); + paths.ensure().expect("create app dirs"); + set_helper_facts(&paths, false, false); + let quiet = render(&paths, "stop_server", &stop_result()); + set_helper_facts(&paths, true, true); + let outlives = render(&paths, "stop_server", &stop_result()); + let restart = render(&paths, "restart_server", &stop_result()); + set_helper_facts(&paths, true, false); + let not_running = render(&paths, "stop_server", &stop_result()); + let _ = fs::remove_dir_all(&root); + + assert!(!quiet.contains("background automation helper"), "{quiet}"); + assert!( + outlives.contains("stopped processes: 1") + && outlives.contains("keeps running while an automation is enabled") + && outlives.contains("`rocm automations disable server-recover`"), + "{outlives}" + ); + assert!( + not_running.contains("is not running") && !not_running.contains("keeps running"), + "{not_running}" + ); + assert!( + !restart.contains("background automation helper"), + "{restart}" + ); + } + + /// `serve` calls the helper autostart after its child is already running. An + /// unreadable config must not turn that launch into an error: it reads as "no + /// watcher enabled" and nothing is spawned. + #[test] + fn background_helper_autostart_treats_an_unreadable_config_as_no_watcher() { + let (root, paths) = test_paths("helper-bad-config"); + paths.ensure().expect("create app dirs"); + fs::write(paths.config_path(), b"{ this is not json").expect("write bad config"); + assert!(rocm_core::RocmCliConfig::load(&paths).is_err()); + + let mut spawned = false; + let result = ensure_background_helper_running_with( + &paths, + true, + |_| { + spawned = true; + Ok(Ok(4242)) + }, + &|| {}, + ); + let _ = fs::remove_dir_all(&root); + + assert!(result.is_ok(), "{result:?}"); + assert!(!spawned); + } + + #[test] + fn autostart_spawn_in_flight_defers_only_for_a_live_recent_claim() { + let claim = AutostartClaim { + daemon_pid: 4242, + spawned_at_ms: 1_000, + }; + // A recent claim whose PID is still alive ⇒ a spawn is in flight, defer. + assert!(autostart_spawn_in_flight( + Some(claim), + 1_000 + 5_000, + AUTOSTART_CLAIM_TTL_MS, + |_| true, + )); + // Same claim but the child PID is gone (crashed spawn) ⇒ respawn. + assert!(!autostart_spawn_in_flight( + Some(claim), + 1_000 + 5_000, + AUTOSTART_CLAIM_TTL_MS, + |_| false, + )); + // Older than the TTL, even with a live PID (possible PID reuse) ⇒ respawn. + assert!(!autostart_spawn_in_flight( + Some(claim), + 1_000 + AUTOSTART_CLAIM_TTL_MS, + AUTOSTART_CLAIM_TTL_MS, + |_| true, + )); + // No claim at all ⇒ nothing in flight, spawn. + assert!(!autostart_spawn_in_flight( + None, + 1_000, + AUTOSTART_CLAIM_TTL_MS, + |_| true, + )); + } + + #[test] + fn autostart_claim_round_trips_and_absent_or_garbage_reads_as_none() { + let (root, paths) = test_paths("autostart-claim-roundtrip"); + let path = paths.automation_autostart_claim_path(); + fs::create_dir_all(path.parent().expect("claim parent")).expect("mkdir claim dir"); + // Absent file ⇒ no claim. + assert_eq!(read_autostart_claim(&path), None); + // Round-trip a written claim. + let claim = AutostartClaim { + daemon_pid: 9987, + spawned_at_ms: 1_726_000_000_123, + }; + write_autostart_claim(&path, claim).expect("write claim"); + assert_eq!(read_autostart_claim(&path), Some(claim)); + // Garbage / partial content ⇒ no claim (permits a respawn, never panics). + fs::write(&path, "not-a-pid").expect("write garbage"); + assert_eq!(read_autostart_claim(&path), None); + let _ = fs::remove_dir_all(&root); + } +} diff --git a/apps/rocm/src/service_launch/launch_transaction.rs b/apps/rocm/src/service_launch/launch_transaction.rs new file mode 100644 index 0000000000..13fe90ed1b --- /dev/null +++ b/apps/rocm/src/service_launch/launch_transaction.rs @@ -0,0 +1,1002 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The locked launch transaction around a managed service: the port allocation +//! decision against live records and the OS, the cleanup of a failed launch +//! attempt, and the reuse refusals (including the no-usable-GPU wording) that +//! decide whether a live service may answer a `serve` request. + +use crate::endpoint_keys; +use crate::managed_service_status::{ + existing_live_managed_service_in, load_managed_services, managed_service_is_live, +}; +use crate::serve_port; +use crate::{device_policy_name, record_cli_audit_event}; +use anyhow::Result; +use rocm_core::{AppPaths, ManagedServiceRecord}; +use rocm_engine_protocol::DevicePolicy; +use std::fs; +use std::time::Duration; + +/// Outcome of the locked allocation step. +/// +/// `Debug` so tests can assert on `Result::expect_err`. +#[derive(Debug)] +pub(super) enum ManagedPortDecision { + /// An equivalent managed service is already live; nothing may be spawned. + /// Boxed to keep the variants a similar size (clippy::large_enum_variant). + AlreadyLive(Box), + /// The concrete port this launch owns until it hands it to the child. + Lease(rocm_core::LoopbackPortLease), +} + +/// Live managed services, with liveness refreshed by [`load_managed_services`] +/// (dead PIDs demote to "stopped" and therefore neither satisfy the duplicate +/// check nor reserve a port). +/// +/// A read failure degrades to "nothing is live": for an already-`ready` +/// service the OS preflight still guards its port (the engine holds it), and +/// refusing to serve because the services directory could not be listed would +/// be a worse failure than launching. That guard does not cover a `starting` +/// record: its port is released back to the OS in +/// `launch_managed_engine_child_locked` (see `lease.release()`), right before the +/// spawn, and stays unbound until the new engine child has actually bound it, so a concurrent launch that hits this +/// read failure in that narrow window can probe the same port successfully. +/// Accepted as a known gap — narrower than the general services-dir read +/// failure case, and only reachable by a second, near-simultaneous I/O +/// failure on top of one already in flight. +pub(super) fn live_managed_services(paths: &AppPaths) -> Vec { + load_managed_services(paths) + .unwrap_or_default() + .into_iter() + .filter(managed_service_is_live) + .collect() +} + +/// Ports that live managed services still own. +/// +/// Conservative by design: on the default loopback host a record reserves its +/// port regardless of the host it recorded, because a `0.0.0.0` (or `localhost`) +/// listener also blocks the loopback address a new local server would bind. +/// Over-reserving costs one candidate in a 101-port scan; under-reserving costs +/// a collision. On a custom host the recorded host is compared (see +/// `serve_port::hosts_overlap`). +fn reserved_service_ports(live: &[ManagedServiceRecord]) -> Vec { + live.iter() + .map(|record| serve_port::ReservedPort { + port: record.port, + host: record.host.clone(), + service_id: record.service_id.clone(), + status: record.status.clone(), + }) + .collect() +} + +/// The decision half of the launch transaction: refresh liveness, apply the +/// `(engine, canonical_model_id)` idempotency guard, then lease a concrete port +/// against both live records and the real OS. +/// +/// Callers MUST hold [`rocm_core::AppPaths::managed_launch_lock_path`] across +/// this call *and* the record publication that follows, or two launches can +/// pick the same candidate. +/// `probe` is injected so tests can drive deterministic bind outcomes without a +/// GPU or a spawn. +pub(super) fn resolve_managed_port_in_transaction( + paths: &AppPaths, + engine: &str, + canonical_model_id: &str, + host: &str, + request: serve_port::PortRequest, + probe: &mut dyn FnMut(u16) -> std::io::Result, +) -> Result { + let live = live_managed_services(paths); + // Idempotency first: an already-live equivalent service means no port is + // needed at all, so it must be answered before any candidate is leased. + if let Some(existing) = existing_live_managed_service_in(&live, engine, canonical_model_id) { + return Ok(ManagedPortDecision::AlreadyLive(Box::new(existing.clone()))); + } + let reserved = reserved_service_ports(&live); + serve_port::resolve_serve_port(host, request, &reserved, probe).map(ManagedPortDecision::Lease) +} + +/// Grace given to a child that must be terminated because its record could not +/// be published. Short: the child is milliseconds old and has nothing to flush. +const MANAGED_LAUNCH_CLEANUP_GRACE: Duration = Duration::from_secs(2); + +/// Undo a failed managed launch attempt before the launch lock is released, +/// so no failed transaction leaves a live reservation or a stray endpoint key. +/// +/// Order matters: a spawned child is terminated *first*, so it is not still +/// holding the port or writing to the log by the time cleanup runs. The +/// termination outcome is audited but not acted on: the record is marked +/// abandoned (`stopped`, releasing the port) even when death was not confirmed. If no child was ever spawned (`record.engine_pid` is still `None`), +/// every artifact this attempt created is removed: there is nothing for +/// `rocm services`/`rocm logs` to show. If a child did run, its record and log +/// are kept instead (marked `status = "stopped"`, see +/// [`ManagedServiceRecord::mark_launch_abandoned`]), since the error that led here +/// already told the user to go inspect that log; only the endpoint key, which a +/// failed attempt has no use for, is cleared. +/// Every step is best-effort and idempotent; cleanup must never mask the +/// original failure with a second one. +pub(super) fn abandon_failed_managed_launch(paths: &AppPaths, record: &ManagedServiceRecord) { + if let Some(pid) = record.engine_pid.filter(|pid| *pid != 0) { + // The token captured at spawn, not a fresh `capture`: when the child + // already exited, a fresh one would describe a recycled pid's new owner + // and the forced tree kill would hit it. Where start tokens exist + // (Linux), a missing one means the child was gone before it could be + // captured, so the pid is unverifiable and nothing is signalled + // (`terminate_verified` alone would "best-effort proceed"). Windows + // records no token at all, so there the kill proceeds as before. + let outcome = if record.engine_start_ticks.is_none() && cfg!(target_os = "linux") { + rocm_core::TerminationOutcome::Unverified + } else { + rocm_core::terminate_verified( + &rocm_core::ProcessIdentity::new(pid, record.engine_start_ticks), + rocm_core::KillScope::Tree, + MANAGED_LAUNCH_CLEANUP_GRACE, + true, + ) + }; + record_cli_audit_event( + paths, + "service", + "managed_service_launch_cleanup", + "warn", + format!( + "terminated engine child pid={pid} after a failed launch publication outcome={}", + outcome.as_str() + ), + Some(&record.service_id), + ); + let mut abandoned = record.clone(); + abandoned.mark_launch_abandoned(); + let _ = abandoned.write(); + endpoint_keys::clear_endpoint_api_key(paths, &record.service_id); + return; + } + for path in [ + record.manifest_path.as_path(), + record.log_path.as_path(), + record.engine_state_path.as_path(), + ] { + let _ = fs::remove_file(path); + } + endpoint_keys::clear_endpoint_api_key(paths, &record.service_id); +} + +/// The ways a live managed service that already matches on liveness, engine and +/// canonical model id can still be unusable by this invocation. +/// +/// Those three terms are necessary but not sufficient: three further terms can +/// still refuse, one variant each. [`managed_service_reuse_refusal`] is the +/// single decision procedure both callers ask, so the two cannot answer +/// differently — which they previously did; see that function for what the +/// disagreement cost. +pub(crate) enum ManagedReuseRefusal { + /// The live service was started with different serve options. + RecipeMismatch, + /// `--require-api-key` was asked for and the live service has no key. + /// + /// The engine reads its key once, at launch, from the environment the spawn + /// builds — so a server started without one keeps serving anonymously no + /// matter what is written afterwards. Upgrading the record would be worse + /// than doing nothing: it would claim auth the live process does not + /// enforce, and `ensure_public_service_has_endpoint_key` would pass on the + /// strength of a key file nothing reads. `rocm remote serve` relies on this: + /// it publishes a loopback port onto the tailnet and prints "the API key + /// above is what stops anyone else calling it", which reusing an + /// unauthenticated service silently would make false about an endpoint the + /// whole tailnet can reach. Refusing is the only answer that fails closed. + MissingApiKey, + /// The live service runs under a device policy that does not satisfy this + /// invocation's. Reusing a `cpu_only` service to answer a `gpu_required` + /// serve is the silent CPU fallback AGENTS.md section 6 forbids. + DevicePolicyTooWeak, +} + +/// Why this invocation cannot reuse `existing` — or `None` when it genuinely +/// can. See [`ManagedReuseRefusal`] for the terms this answers on. +/// +/// This exists because `serve()`'s reuse pre-gate lets a matching live service +/// skip the no-usable-AMD-GPU refusal. Deciding that bypass on a *weaker* test +/// than the spawn then uses to accept reuse produced a dead end: on a GPU-less +/// host (`HIP_VISIBLE_DEVICES=` with a server live from another shell) a serve +/// with different options bypassed the refusal, reached the spawn, and was told +/// to stop the server and run the command again — and the rerun, with nothing +/// left to reuse, hit the no-GPU refusal. The advice named a command that could +/// not clear the condition, which AGENTS.md section 3 forbids. Sharing the +/// predicate means a service that cannot actually be reused never buys the +/// bypass, so the user gets the accurate refusal first and keeps their server. +/// +/// `requested_device_policy` is taken by value, not as an `Option`, on purpose. +/// It was optional so the spawn could opt out of the device-policy term, and +/// that left two holes at once: the spawn's `DevicePolicyTooWeak` arm was +/// unreachable (an untestable bail), and `serve()` — the caller that *does* +/// gate the GPU refusal on this answer — could silently drop the term by +/// passing `None`, with the whole suite still green. Both callers have a +/// `DevicePolicy` in hand, so requiring it costs nothing and makes that +/// mutation a compile error. It widens nothing observable today either: the +/// only policy a record can carry is the one `parse_device_policy` admits +/// (see below), so `existing.device_policy == Some("cpu_only")` is not +/// reachable from any `rocm serve` this CLI can run. +/// The `rocm serve` no-usable-GPU refusal, worded once. +/// +/// Two places raise it and they must say the same thing: `serve()`'s fail-fast +/// pre-flight, and [`spawn_managed_engine_child`](super::spawn::spawn_managed_engine_child) +/// when the reuse that waived that pre-flight turns out not to be available after +/// all. +pub(crate) fn no_usable_gpu_refusal(device_policy: &DevicePolicy) -> String { + // Keyed on the same `is_wsl_host()` the usable-GPU probe branches on, so the + // advice follows the platform whose answer produced the refusal. + no_usable_gpu_refusal_text(device_policy_name(device_policy), rocm_core::is_wsl_host()) +} + +/// The refusal text for `policy`, split on whether the probe that refused was +/// the WSL one. +/// +/// WSL reaches the GPU through `/dev/dxg` and ROCDXG, and has no `/dev/kfd`, so +/// telling a WSL user to confirm that device is present names something that can +/// never clear the condition. There the advice points at the status `rocm +/// examine` reports for the path the probe actually asked about +/// ([`rocm_core::WSL_ROCDXG_READY`]), and at `rocm diagnose`, which names what is +/// missing. +fn no_usable_gpu_refusal_text(policy: &str, on_wsl: bool) -> String { + let remedy = if on_wsl { + format!( + "WSL reaches the GPU through /dev/dxg and ROCDXG rather than /dev/kfd: run \ + `rocm examine` and check that `driver_status` is `{ready}` (`rocm diagnose` \ + names what is missing; a ROCDXG library the loader reaches only through \ + LD_LIBRARY_PATH needs that variable exported in this shell, see docs/wsl.md), and ensure HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES \ + are not masking every device.", + ready = rocm_core::WSL_ROCDXG_READY + ) + } else { + "Check the driver with `rocm examine`, confirm /dev/kfd is present, and ensure \ + HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES are not masking every device." + .to_owned() + }; + format!( + "no usable AMD GPU detected; `rocm serve` requires a GPU under the {policy} policy \ + and does not fall back to CPU. {remedy}" + ) +} + +pub(crate) fn managed_service_reuse_refusal( + existing: &ManagedServiceRecord, + requested_recipe_json: Option<&str>, + require_api_key: bool, + requested_device_policy: &DevicePolicy, +) -> Option { + if existing.engine_recipe_json.as_deref() != requested_recipe_json { + return Some(ManagedReuseRefusal::RecipeMismatch); + } + if require_api_key && !existing.requires_api_key { + return Some(ManagedReuseRefusal::MissingApiKey); + } + // Only `gpu_required` can be let down by the record's policy, and the + // enumeration is exhaustive rather than narrow: `parse_device_policy` + // (`main.rs`) maps `auto`, `gpu`, `gpu_required` *and* `gpu_preferred` onto + // `GpuRequired`, and rejects `cpu_only` with an error, so `GpuRequired` is + // the only policy `serve()` can hand this predicate — pinned by + // `parse_device_policy_defaults_to_gpu_required_without_cpu_fallback`. There + // is deliberately no wider match on "anything but `CpuOnly`": it would be + // identical at runtime and would read as a guard against a `gpu_preferred` + // bypass that no CLI invocation can reach. A `cpu_only` invocation would be + // satisfied by anything anyway and never reaches the bail. README.md's + // "that is the only policy pair refused" is the contract, and this matches + // it exactly. + if matches!(requested_device_policy, DevicePolicy::GpuRequired) + && existing.device_policy.as_deref() == Some("cpu_only") + { + return Some(ManagedReuseRefusal::DevicePolicyTooWeak); + } + None +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::service_hints; + #[cfg(not(windows))] + use crate::service_launch::test_support::sleeper_test_name; + use crate::service_launch::test_support::{reusable_record, write_live_service_record}; + use crate::tests::test_paths; + use rocm_core::DEFAULT_LOCAL_HOST; + #[cfg(not(windows))] + use std::process::Command as ProcessCommand; + use std::thread; + + #[test] + fn the_no_gpu_refusal_sends_a_bare_metal_user_to_dev_kfd() { + let text = super::no_usable_gpu_refusal_text("gpu_required", false); + assert!(text.contains("confirm /dev/kfd is present"), "{text}"); + assert!(!text.contains("ROCDXG"), "{text}"); + } + + #[test] + fn the_no_gpu_refusal_does_not_send_a_wsl_user_to_dev_kfd() { + // WSL has no /dev/kfd, so advice to confirm it can never clear the + // condition. The text names the status `examine` reports for the path + // the probe asked about, taken from the constant the producer uses, so + // the two cannot drift. That the status really is the one a refused + // host shows is asserted end to end by serve-42. + let text = super::no_usable_gpu_refusal_text("gpu_required", true); + assert!(!text.contains("/dev/kfd is present"), "{text}"); + assert!(text.contains("rocm examine"), "{text}"); + assert!(text.contains("rocm diagnose"), "{text}"); + // The advice is true only because `rocdxg_ready` counts a library on + // LD_LIBRARY_PATH; `a_rocdxg_only_on_the_loader_path_is_ready` in + // `rocm-core`'s host_gpu/wsl.rs proves that behavior. + assert!(text.contains("LD_LIBRARY_PATH"), "{text}"); + assert!( + text.contains(&format!("`{}`", rocm_core::WSL_ROCDXG_READY)), + "{text}" + ); + } + + #[test] + fn the_no_gpu_refusal_names_the_policy_on_both_platforms() { + for on_wsl in [false, true] { + let text = super::no_usable_gpu_refusal_text("gpu_required", on_wsl); + assert!( + text.starts_with("no usable AMD GPU detected; `rocm serve` requires a GPU under the gpu_required policy"), + "{text}" + ); + } + } + + #[test] + fn reuse_is_refused_on_every_term_the_spawn_and_the_gpu_bail_depend_on() { + let (root, paths) = test_paths("managed-reuse-predicate"); + paths.ensure().expect("paths"); + let base = reusable_record(&paths); + + // The baseline must be reusable, or the negative cases below could pass + // for the wrong reason. + assert!( + managed_service_reuse_refusal(&base, None, true, &DevicePolicy::GpuRequired).is_none(), + "a matching service with no recipe, a key, and a gpu_required policy is reusable" + ); + + // Recipe mismatch — the spawn's first refusal. + let mut mismatched = base.clone(); + mismatched.engine_recipe_json = Some("{\"engine\":\"vllm\"}".to_owned()); + assert!(matches!( + managed_service_reuse_refusal(&mismatched, None, true, &DevicePolicy::GpuRequired), + Some(ManagedReuseRefusal::RecipeMismatch) + )); + + // Unmet `--require-api-key` — the spawn's second refusal. Also pins that + // it is *only* refused when the key is actually demanded. + let mut anonymous = base.clone(); + anonymous.requires_api_key = false; + assert!(matches!( + managed_service_reuse_refusal(&anonymous, None, true, &DevicePolicy::GpuRequired), + Some(ManagedReuseRefusal::MissingApiKey) + )); + assert!( + managed_service_reuse_refusal(&anonymous, None, false, &DevicePolicy::GpuRequired) + .is_none(), + "an anonymous service is fine when no key was demanded" + ); + + // Device policy — the term `serve()` depends on, because it is the one + // gating the no-usable-GPU bail on the answer. Reusing a `cpu_only` + // service to answer a `gpu_required` serve is the silent CPU fallback. + let mut cpu_only = base; + cpu_only.device_policy = Some("cpu_only".to_owned()); + assert!(matches!( + managed_service_reuse_refusal(&cpu_only, None, true, &DevicePolicy::GpuRequired), + Some(ManagedReuseRefusal::DevicePolicyTooWeak) + )); + // `GpuRequired` is the only policy the CLI can produce — `--device + // gpu_preferred`/`auto`/`gpu` all parse to it and `cpu_only` is rejected + // (`parse_device_policy_defaults_to_gpu_required_without_cpu_fallback`) — + // so there is no second policy arm to discriminate here, and asserting + // one would pin an input `serve()` never produces. Both callers pass + // the same policy; the parameter is not optional, so neither can drop + // this term without a compile error. + // + // A cpu_only *invocation* never reaches the bail, so nothing to refuse. + assert!( + managed_service_reuse_refusal(&cpu_only, None, true, &DevicePolicy::CpuOnly).is_none() + ); + + let _ = std::fs::remove_dir_all(&root); + } + + #[test] + fn duplicate_managed_launch_detected_across_distinct_service_ids() -> Result<()> { + // `generate_service_id` embeds a timestamp, so a second launch for the + // same engine+model has a DIFFERENT service_id. The guard must still + // detect the live service by (engine, canonical_model_id), and return + // the newest live match. `starting` skips the endpoint probe; the + // current process id is a guaranteed-live PID. + let (root, paths) = test_paths("dup-managed-distinct-ids"); + paths.ensure()?; + + // Older, dead manifest for the same engine+model (distinct service_id). + let mut dead = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-1000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11500, + "managed", + 999_999_999, + None, + None, + None, + ); + dead.status = "ready".to_owned(); + dead.engine_pid = Some(999_999_999); + dead.created_at_unix_ms = 1000; + dead.write()?; + + // Newer, live manifest for the same engine+model (distinct service_id). + let mut live = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-2000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11501, + "managed", + std::process::id(), + None, + None, + None, + ); + live.status = "starting".to_owned(); + live.engine_pid = Some(std::process::id()); + live.created_at_unix_ms = 2000; + live.write()?; + + let live_services = live_managed_services(&paths); + let found = + existing_live_managed_service_in(&live_services, "lemonade", "qwen-canonical").cloned(); + let _ = fs::remove_dir_all(root); + + let found = found.expect("a live managed service should be detected by engine+model"); + assert_eq!( + found.service_id, "lemonade-qwen-2000", + "should return the newest live match" + ); + assert!(managed_service_is_live(&found)); + Ok(()) + } + + #[test] + fn dead_managed_service_allows_relaunch() -> Result<()> { + // A stale manifest with dead PIDs must NOT block a relaunch: liveness + // refresh demotes it to "stopped", so the guard returns None. + let (root, paths) = test_paths("dup-managed-dead"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-3000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11502, + "managed", + 999_999_999, + None, + None, + None, + ); + record.status = "ready".to_owned(); + record.engine_pid = Some(999_999_999); + record.write()?; + + let live = live_managed_services(&paths); + let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); + let _ = fs::remove_dir_all(root); + + assert!( + found.is_none(), + "a dead managed service must not block relaunch" + ); + Ok(()) + } + + #[test] + fn live_service_for_other_model_does_not_block() -> Result<()> { + // A live service for a DIFFERENT model must not match — the guard keys + // on the model, not just the engine. + let (root, paths) = test_paths("dup-managed-other-model"); + paths.ensure()?; + let mut record = ManagedServiceRecord::new( + &paths, + "lemonade-other-1", + "lemonade", + "other", + "other-canonical", + "127.0.0.1", + 11503, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = "starting".to_owned(); + record.engine_pid = Some(std::process::id()); + record.write()?; + + let live = live_managed_services(&paths); + let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); + let _ = fs::remove_dir_all(root); + + assert!( + found.is_none(), + "a live service for a different model must not match" + ); + Ok(()) + } + + #[test] + fn missing_manifest_allows_launch() { + // No services dir / manifests → nothing to detect, launch proceeds. + let (root, paths) = test_paths("dup-managed-missing"); + let live = live_managed_services(&paths); + let found = existing_live_managed_service_in(&live, "lemonade", "qwen-canonical"); + let _ = fs::remove_dir_all(root); + assert!(found.is_none()); + } + + fn never_probe(_port: u16) -> std::io::Result { + panic!("no port may be probed on this path") + } + + fn always_free(port: u16) -> std::io::Result { + Ok(rocm_core::LoopbackPortLease::engine_owned(port)) + } + + #[test] + fn allocation_transaction_answers_an_equivalent_live_service_before_leasing() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-already-live"); + paths.ensure()?; + let existing = write_live_service_record( + &paths, + "lemonade-qwen-9000", + "qwen-canonical", + 11501, + "starting", + )?; + + let decision = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "qwen-canonical", + "127.0.0.1", + serve_port::PortRequest::Auto, + &mut never_probe, + )?; + let _ = fs::remove_dir_all(root); + + match decision { + ManagedPortDecision::AlreadyLive(record) => { + assert_eq!(record.service_id, existing.service_id); + assert_eq!(record.port, existing.port); + } + ManagedPortDecision::Lease(_) => { + panic!("an equivalent live service must be answered, not leased a new port") + } + } + Ok(()) + } + + #[test] + fn allocation_transaction_skips_ports_reserved_by_other_live_services() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-skips-reserved"); + paths.ensure()?; + write_live_service_record(&paths, "lemonade-a", "model-a", 11_435, "running")?; + write_live_service_record(&paths, "lemonade-b", "model-b", 11_436, "recovering")?; + + let decision = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "model-c", + "127.0.0.1", + serve_port::PortRequest::Auto, + &mut always_free, + )?; + let _ = fs::remove_dir_all(root); + + match decision { + ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_437), + ManagedPortDecision::AlreadyLive(record) => { + panic!("model-c has no live service, expected a lease, got {record:?}") + } + } + Ok(()) + } + + #[test] + fn allocation_transaction_ignores_stopped_records() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-ignores-stopped"); + paths.ensure()?; + let mut stale = + write_live_service_record(&paths, "lemonade-stale", "model-stale", 11_435, "running")?; + // A dead pid demotes the record to stopped when liveness is refreshed, so + // it must not reserve its port. + stale.engine_pid = Some(999_999_999); + stale.supervisor_pid = 999_999_999; + stale.write()?; + + let decision = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "model-fresh", + "127.0.0.1", + serve_port::PortRequest::Auto, + &mut always_free, + )?; + let _ = fs::remove_dir_all(root); + + match decision { + ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_435), + ManagedPortDecision::AlreadyLive(record) => { + panic!("a stopped record must not block a fresh launch, got {record:?}") + } + } + Ok(()) + } + + #[test] + fn stopping_the_reserving_record_frees_its_explicit_port_for_reuse() -> Result<()> { + // Proves the mechanism `rocm services stop` relies on: it never touches + // the port directly, it only flips the stopped record's `status` (see + // the `all_stopped` branch in `main.rs`'s stop handler). This recreates + // that exact disk state and checks the same explicit port a live + // competitor refused now resolves, closing the loop the E2E + // `assert_port_reserved_message` step only checks by wording. + let (root, paths) = test_paths("alloc-explicit-port-freed-by-stop"); + paths.ensure()?; + let mut holder = ManagedServiceRecord::new( + &paths, + "lemonade-a-1", + "lemonade", + "model-a", + "model-a".to_owned(), + DEFAULT_LOCAL_HOST, + 11_435, + "managed", + std::process::id(), + None, + None, + None, + ); + holder.status = "running".to_owned(); + holder.engine_pid = Some(std::process::id()); + holder.write()?; + + let still_reserved = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "model-b", + DEFAULT_LOCAL_HOST, + serve_port::PortRequest::Explicit(11_435), + &mut always_free, + ); + // The refusal names the service AND the command that frees the port; + // the stop below is that command's own implementation, and the grant + // after it is the outcome, so advice and result are pinned together. + let stop_hint = service_hints::service_stop_hint("lemonade-a-1"); + assert!( + matches!(still_reserved, Err(ref error) if error.to_string().contains("already reserved by managed service lemonade-a-1") + && error.to_string().contains(&stop_hint)), + "unexpected result while still running: {still_reserved:?}" + ); + + // Go through the real stop path (`rocm services stop --yes`'s own + // implementation) rather than writing `status = "stopped"` to disk + // directly, so this test actually proves the advice serve-45 pins: that + // running the stop command frees the port. The termination closure is + // the same seam production injects a real signal through; here it just + // confirms every recorded process gone without touching the test + // process itself. + crate::service_control::stop_internal_managed_service_with(&paths, "lemonade-a-1", |_| { + crate::service_control::RecordedStopTermination::confirmed_stopped() + })?; + + let decision = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "model-b", + DEFAULT_LOCAL_HOST, + serve_port::PortRequest::Explicit(11_435), + &mut always_free, + ); + let _ = fs::remove_dir_all(root); + + match decision? { + ManagedPortDecision::Lease(lease) => assert_eq!(lease.port(), 11_435), + ManagedPortDecision::AlreadyLive(existing) => { + panic!( + "a stopped record must not block reuse: {}", + existing.service_id + ) + } + } + Ok(()) + } + + #[test] + fn allocation_transaction_rejects_an_explicit_port_a_recovering_service_holds() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-rejects-explicit-collision"); + paths.ensure()?; + let existing = write_live_service_record( + &paths, + "lemonade-recovering", + "model-recovering", + 11_435, + "recovering", + )?; + + let result = resolve_managed_port_in_transaction( + &paths, + "lemonade", + "model-other", + "127.0.0.1", + serve_port::PortRequest::Explicit(11_435), + &mut never_probe, + ); + let _ = fs::remove_dir_all(root); + + let Err(error) = result else { + panic!("an explicit port a recovering service holds must be refused") + }; + let message = error.to_string(); + assert!( + message.contains(&existing.service_id), + "the refusal must name the service holding the port: {message}" + ); + assert!( + message.contains("recovering"), + "the refusal must say why the port is unavailable: {message}" + ); + Ok(()) + } + + #[test] + fn concurrent_auto_transactions_publish_distinct_ports() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-concurrent-distinct"); + paths.ensure()?; + let paths = std::sync::Arc::new(paths); + + let spawn_one = |model: &'static str| { + let paths = std::sync::Arc::clone(&paths); + thread::spawn(move || -> Result { + let _launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + let lease = match resolve_managed_port_in_transaction( + &paths, + "lemonade", + model, + "127.0.0.1", + serve_port::PortRequest::Auto, + &mut serve_port::loopback_bind_probe, + )? { + ManagedPortDecision::Lease(lease) => lease, + ManagedPortDecision::AlreadyLive(record) => { + panic!("fresh model must not already be live: {record:?}") + } + }; + let port = lease.port(); + write_live_service_record(&paths, model, model, port, "running")?; + lease.release(); + Ok(port) + }) + }; + + let first = spawn_one("model-one"); + let second = spawn_one("model-two"); + let first_port = first.join().expect("thread one panicked")?; + let second_port = second.join().expect("thread two panicked")?; + let _ = fs::remove_dir_all(root); + + assert_ne!(first_port, second_port); + for port in [first_port, second_port] { + assert!( + (serve_port::AUTO_PORT_FIRST..=serve_port::AUTO_PORT_LAST).contains(&port), + "leased port {port} must fall within the auto-select range" + ); + } + Ok(()) + } + + #[test] + fn concurrent_equivalent_transactions_converge_on_one_record() -> Result<()> { + let (root, paths) = test_paths("alloc-txn-concurrent-converge"); + paths.ensure()?; + let paths = std::sync::Arc::new(paths); + + let race_once = || { + let paths = std::sync::Arc::clone(&paths); + thread::spawn(move || -> Result { + let _launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + match resolve_managed_port_in_transaction( + &paths, + "lemonade", + "shared-model", + "127.0.0.1", + serve_port::PortRequest::Auto, + &mut serve_port::loopback_bind_probe, + )? { + ManagedPortDecision::Lease(lease) => { + let port = lease.port(); + write_live_service_record( + &paths, + "lemonade-shared", + "shared-model", + port, + "running", + )?; + lease.release(); + Ok(true) + } + ManagedPortDecision::AlreadyLive(_) => Ok(false), + } + }) + }; + + let first = race_once(); + let second = race_once(); + let first_launched = first.join().expect("thread one panicked")?; + let second_launched = second.join().expect("thread two panicked")?; + + let live_count = live_managed_services(&paths).len(); + let _ = fs::remove_dir_all(root); + + assert_ne!( + first_launched, second_launched, + "exactly one racing transaction must launch" + ); + assert_eq!(live_count, 1, "only one record may end up live"); + Ok(()) + } + + #[test] + fn abandoned_launch_leaves_no_record_log_or_endpoint_key() -> Result<()> { + let (root, paths) = test_paths("abandon-launch-cleanup"); + paths.ensure()?; + let mut record = write_live_service_record( + &paths, + "lemonade-abandon", + "abandon-model", + 11_490, + "running", + )?; + fs::create_dir_all( + record + .engine_state_path + .parent() + .expect("engine state parent"), + )?; + fs::write(&record.engine_state_path, "{}")?; + endpoint_keys::store_endpoint_api_key(&paths, &record.service_id, "secret-key")?; + // Simulate a pre-spawn failure: nothing ever ran. + record.engine_pid = None; + + abandon_failed_managed_launch(&paths, &record); + + assert!(!record.manifest_path.exists()); + assert!(!record.log_path.exists()); + assert!(!record.engine_state_path.exists()); + assert_eq!( + endpoint_keys::endpoint_api_key(&paths, &record.service_id), + None + ); + assert!(live_managed_services(&paths).is_empty()); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + #[cfg(target_os = "linux")] + fn abandoned_launch_never_signals_a_pid_that_no_longer_matches_the_spawn_token() -> Result<()> { + // The child exited and its pid now belongs to an unrelated live + // process: the start token recorded at spawn no longer matches, so the + // forced tree kill must not touch it. + let (root, paths) = test_paths("abandon-launch-recycled-pid"); + paths.ensure()?; + let mut record = write_live_service_record( + &paths, + "lemonade-abandon-recycled", + "abandon-recycled-model", + 11_493, + "running", + )?; + let current_exe = std::env::current_exe().expect("current test exe"); + let mut bystander = ProcessCommand::new(current_exe) + .arg("--exact") + .arg(sleeper_test_name()) + .arg("--ignored") + .arg("--test-threads=1") + .env("ROCM_TEST_SLEEPER_CHILD", "1") + .spawn() + .expect("spawn bystander"); + record.record_engine_identity(bystander.id()); + record.engine_start_ticks = record.engine_start_ticks.map(|ticks| ticks + 1); + + abandon_failed_managed_launch(&paths, &record); + + let mismatched_left_alone = bystander.try_wait().expect("poll bystander").is_none(); + // A child that was gone before its token could be captured leaves none: + // that must not fall back to "best-effort" signalling either. + record.engine_start_ticks = None; + abandon_failed_managed_launch(&paths, &record); + let still_running = + mismatched_left_alone && bystander.try_wait().expect("poll bystander").is_none(); + let _ = bystander.kill(); + let _ = bystander.wait(); + let _ = fs::remove_dir_all(root); + assert!( + still_running, + "a pid with a mismatched token must be left alone" + ); + Ok(()) + } + + #[test] + #[cfg(not(windows))] + fn abandoned_launch_terminates_the_spawned_child() -> Result<()> { + let (root, paths) = test_paths("abandon-launch-terminates-child"); + paths.ensure()?; + let mut record = write_live_service_record( + &paths, + "lemonade-abandon-child", + "abandon-child-model", + 11_491, + "running", + )?; + + let current_exe = std::env::current_exe().expect("current test exe"); + let mut child = ProcessCommand::new(current_exe) + .arg("--exact") + .arg(sleeper_test_name()) + .arg("--ignored") + .arg("--test-threads=1") + .env("ROCM_TEST_SLEEPER_CHILD", "1") + .spawn() + .expect("spawn sleeper child"); + let child_pid = child.id(); + record.record_engine_identity(child_pid); + + abandon_failed_managed_launch(&paths, &record); + + let wait_result = child.wait(); + + match wait_result { + Ok(status) => assert!( + !status.success(), + "the sleeper child must be terminated, not exit cleanly" + ), + Err(error) => panic!("failed to wait for sleeper child: {error}"), + } + let persisted = load_managed_services(&paths) + .expect("re-read manifests") + .into_iter() + .find(|candidate| candidate.service_id == record.service_id); + let _ = fs::remove_dir_all(root); + let persisted = persisted.expect( + "a launch that spawned a child must keep its record so `rocm services`/`rocm logs` can show the attempt", + ); + // `stopped`, not `failed`: `rocmd`'s recovery restarts a `failed` record + // that carries no stop marker, which would respawn a launch the CLI just + // reported as failed. The rocmd side of the pair is + // `abandoned_launch_record_is_not_a_recovery_candidate` in + // `apps/rocmd/src/watchers.rs`. + assert_eq!( + persisted.status, "stopped", + "a spawned child's record must be kept, but in a status recovery ignores" + ); + Ok(()) + } +} diff --git a/apps/rocm/src/service_launch/readiness.rs b/apps/rocm/src/service_launch/readiness.rs new file mode 100644 index 0000000000..1e14c85599 --- /dev/null +++ b/apps/rocm/src/service_launch/readiness.rs @@ -0,0 +1,634 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Waiting on a launched or already-live managed service: the readiness budget, +//! liveness of the tracked process, how a wait ends, and the end-to-end OOM +//! fault-injection seam that fabricates a launch that never became ready. + +// There is deliberately no `use std::fs`, `use std::thread` or +// `use rocm_core::process_is_running` here: this file names them by full path. +// Their use sites sit under different `cfg` combinations (the Linux-only zombie +// probe, the OOM fault-injection seam, one Linux-only test), so any single import +// would be unused in some build, and a gate on it would have to be kept in step +// with every use site. Keep the full paths. The tests module imports `fs` for +// itself because its tests use it on every platform. +use super::spawn::ManagedLaunchReport; +#[cfg(feature = "e2e-oom-fault-injection")] +use crate::device_policy_name; +use crate::endpoint_keys; +use crate::engine_runtime_select::wait_for_service_http_ready_with_progress; +use crate::status_for_readiness; +#[cfg(feature = "e2e-oom-fault-injection")] +use anyhow::Context; +use anyhow::{Result, anyhow}; +#[cfg(feature = "e2e-oom-fault-injection")] +use rocm_core::format_http_base_url; +use rocm_core::{AppPaths, EndpointReadiness, ManagedServiceRecord}; +#[cfg(feature = "e2e-oom-fault-injection")] +use rocm_engine_protocol::{DevicePolicy, ResolveModelResponse}; +use std::fmt::Write as _; +use std::time::Duration; + +/// How long one readiness wait polls before `rocm serve` reports `starting`. +const READINESS_WAIT: Duration = Duration::from_secs(45); + +/// `rocm serve --wait`: keep waiting past [`READINESS_WAIT`] (a first start +/// downloads the model, which can take far longer) until the server answers or +/// stops being live. A server that exited ends the wait with the last reading +/// instead of blocking forever. +fn wait_past_readiness_budget( + mut probe: impl FnMut() -> EndpointReadiness, + mut still_live: impl FnMut() -> bool, +) -> EndpointReadiness { + loop { + let readiness = probe(); + if readiness == EndpointReadiness::Serving || !still_live() { + return readiness; + } + } +} + +/// Whether `pid` is a running process that is not a zombie. The serve +/// supervisor is an unreaped child of this process, so after it dies +/// `process_is_running` alone keeps reporting it alive. +pub(super) fn service_pid_is_alive(pid: u32) -> bool { + rocm_core::process_is_running(pid) && !process_is_zombie(pid) +} + +#[cfg(target_os = "linux")] +fn process_is_zombie(pid: u32) -> bool { + let Ok(stat) = std::fs::read_to_string(format!("/proc/{pid}/stat")) else { + return false; + }; + // `comm` may contain spaces and parentheses; the state follows the last `)`. + stat.rfind(')') + .and_then(|end| stat.get(end + 1..)) + .and_then(|rest| rest.split_whitespace().next()) + == Some("Z") +} + +#[cfg(not(target_os = "linux"))] +const fn process_is_zombie(_pid: u32) -> bool { + false +} + +/// Whether the record still describes a server that can finish starting: its +/// status is a live one *and* a tracked process is alive. The status alone is +/// not enough, since a killed engine never writes a terminal state. +fn record_still_live(record: &ManagedServiceRecord, pid_alive: impl Fn(u32) -> bool) -> bool { + if !crate::managed_service_status::managed_service_is_live(record) { + return false; + } + let pids = crate::managed_service_status::recorded_service_pids(record); + pids.is_empty() || pids.into_iter().any(pid_alive) +} + +/// [`record_still_live`] over the manifest on disk, so a `rocm services stop` +/// from another terminal (which writes only the manifest) ends the wait too. +fn service_still_live(paths: &AppPaths, service_id: &str, pid_alive: impl Fn(u32) -> bool) -> bool { + let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { + return false; + }; + let _ = latest.refresh_from_engine_state(); + record_still_live(&latest, pid_alive) +} + +/// Record a server that died mid-start as stopped, leaving a manifest that a +/// `services stop` already settled untouched. A `failed` state the engine +/// reported is persisted rather than overwritten, so the reason survives. +fn settle_exited_service(paths: &AppPaths, service_id: &str) { + let Ok(mut latest) = ManagedServiceRecord::read_manifest(paths, service_id) else { + return; + }; + let on_disk = latest.status.clone(); + let _ = latest.refresh_from_engine_state(); + if crate::managed_service_status::managed_service_is_live(&latest) { + latest.status = "stopped".to_owned(); + } else if latest.status == on_disk { + return; + } + let _ = latest.write(); +} + +/// The failure a `--wait` ends with when the server died before it became +/// ready, after recording the death in the manifest. `None` while the wait did +/// not apply, the endpoint is serving, or the server is still alive. +pub(super) fn died_during_wait( + paths: &AppPaths, + service_id: &str, + wait_until_ready: bool, + readiness: EndpointReadiness, + pid_alive: impl Fn(u32) -> bool, +) -> Option { + if !wait_until_ready + || readiness == EndpointReadiness::Serving + || service_still_live(paths, service_id, pid_alive) + { + return None; + } + settle_exited_service(paths, service_id); + let mut message = format!( + "the server exited before it became ready (vLLM and AIM give up after 5 minutes by default; `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS` allow longer); see `rocm logs --service {service_id}` for its output" + ); + // A plain `serve` adds the memory guidance for a vLLM launch that ran out of + // VRAM; the wait's failure is the same launch, so it gets the same note. + // "starting" is the status the note keys on for "never became ready", which is + // what a `--wait` death is, whatever the settled record now says. + if let Ok(record) = ManagedServiceRecord::read_manifest(paths, service_id) { + for note in crate::serve_summary::append_oom_serve_note( + Vec::new(), + &record.engine, + false, + "starting", + Some(&record.log_path), + ) { + let _ = write!(message, "\n note: {note}"); + } + } + Some(anyhow!(message)) +} + +/// Record the outcome of a finished wait on the manifest as it is now, changing +/// only what the wait learned. The copy read at spawn time can be long stale: +/// `rocmd` may have recorded a restart or a new supervisor pid, and `rocm logs` +/// refreshes the record, so writing it back whole would revert those. +pub(super) fn record_wait_outcome( + paths: &AppPaths, + held: &mut ManagedServiceRecord, + status: &str, + verified: bool, +) -> Result<()> { + let mut latest = ManagedServiceRecord::read_manifest(paths, &held.service_id) + .unwrap_or_else(|_| held.clone()); + status.clone_into(&mut latest.status); + if verified { + // Latch the verification the wait just performed, so the readiness checks + // behind `services list` and chat read it instead of re-probing. + latest.inference_verified_at_unix_ms = Some(rocm_core::unix_time_millis() as u64); + } + latest.write()?; + *held = latest; + Ok(()) +} + +/// Test-only fault-injection switch, off (`const false`) in shipped binaries: +/// the `e2e-oom-fault-injection` feature is enabled only by the `xtask e2e` +/// mock-lane build, never by the release build. When enabled and +/// `ROCM_E2E_SIMULATE_OOM_LAUNCH=1` is set on the process, `rocm serve` fabricates +/// the on-disk state of a managed vLLM launch that spawned, wrote an +/// out-of-memory traceback to the log it owns, and never became ready — so the +/// positive OOM-guidance summary path can be verified on a GPU-less CI host, +/// where a real launch cannot run. See [`simulate_oom_managed_launch`]. +#[cfg(feature = "e2e-oom-fault-injection")] +pub(crate) fn e2e_simulate_oom_launch() -> bool { + std::env::var_os("ROCM_E2E_SIMULATE_OOM_LAUNCH").is_some_and(|value| value == "1") +} + +/// Release builds carry no fault-injection switch: the seam compiles out +/// entirely, so `rocm serve` behaves identically to a build without the feature. +#[cfg(not(feature = "e2e-oom-fault-injection"))] +pub(crate) const fn e2e_simulate_oom_launch() -> bool { + false +} + +/// Test-only (see [`e2e_simulate_oom_launch`]): fabricate the on-disk state of a +/// managed vLLM launch that spawned, OOM'd, and never became ready, then return +/// the launch report the summary path would see for it — `already_running: false` +/// (this invocation launched it), `status: "starting"` (it failed to become +/// ready), and a `log_path` whose contents carry a real allocator OOM signature. +/// [`crate::serve_summary::append_oom_serve_note`] then renders the memory-knob guidance. No process is +/// spawned. Compiled out of shipped binaries. +#[cfg(feature = "e2e-oom-fault-injection")] +#[allow(clippy::too_many_arguments)] +pub(super) fn simulate_oom_managed_launch( + paths: &AppPaths, + engine: &str, + service_id: &str, + requested_model: &str, + resolve: &ResolveModelResponse, + host: &str, + port: u16, + device_policy: &DevicePolicy, + gpu_indices: &[u32], + runtime_id: Option<&str>, + env_id: Option<&str>, +) -> Result { + paths.ensure()?; + std::fs::create_dir_all(paths.services_dir())?; + let mut record = ManagedServiceRecord::new( + paths, + service_id, + engine, + requested_model, + resolve.canonical_model_id.clone(), + host, + port, + "managed", + std::process::id(), + runtime_id.map(str::to_owned), + env_id.map(str::to_owned), + Some(device_policy_name(device_policy).to_owned()), + ); + record.gpu_indices = gpu_indices.to_vec(); + record.status = "starting".to_owned(); + record.write()?; + // A real allocator OOM signature in the log this invocation owns, so the + // summary renders memory guidance for a launch that actually OOM'd. + // + // The engine's own error block is reproduced below it, because that is what + // a real managed launch leaves on disk, and leaving it out hid a bug this + // scenario should have caught: the supervisor and the engine share one log, + // so when vLLM OOMs the engine bails with its startup-log context — which + // embeds rocm-cli's own OOM hint — and `exit_code_for` writes that whole + // error back into the same file, *after* the real line. A fixture carrying + // only the raw OOM line can never see the newest-first scan land on + // rocm-cli's own sentence instead of the user's error. + let real_line = "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB."; + std::fs::write( + &record.log_path, + format!( + "{real_line}\n\ + Error: vLLM server process exited before becoming ready (status: exit status: 1)\n\ + \n\ + Startup log, last {} lines as the terminal drew them:\n\ + {real_line}\n\ + \n\ + {}{real_line}). {}\n\ + For conditional remediation, run `{} 'vllm: {real_line}'`.\n", + rocm_engine_protocol::DEFAULT_LOG_TAIL_LINES, + rocm_core::VLLM_OOM_DETECTED_PREFIX, + rocm_core::VLLM_GPU_MEMORY_UTILIZATION_HINT, + rocm_core::VLLM_OOM_DIAGNOSE_COMMAND, + ), + ) + .with_context(|| format!("failed to write {}", record.log_path.display()))?; + Ok(ManagedLaunchReport { + service_id: service_id.to_owned(), + endpoint_url: format!("{}/v1", format_http_base_url(host, port)), + status: "starting".to_owned(), + already_running: false, + child_pid: None, + log_path: Some(record.log_path), + manifest_path: Some(record.manifest_path), + }) +} + +/// Probe `record`'s endpoint until it answers: once within the readiness +/// budget, or (with `wait_until_ready`) for as long as the server stays alive. +pub(super) fn wait_for_record_ready( + paths: &AppPaths, + record: &ManagedServiceRecord, + wait_until_ready: bool, + endpoint_api_key: Option<&str>, + on_wait_tick: &mut dyn FnMut(Duration), +) -> EndpointReadiness { + let probe = |on_tick: &mut dyn FnMut(Duration)| { + wait_for_service_http_ready_with_progress( + &record.engine, + &record.host, + record.port, + &|| { + let mut latest = record.clone(); + let _ = latest.refresh_from_engine_state(); + latest.served_model_name().to_owned() + }, + endpoint_api_key, + READINESS_WAIT, + on_tick, + ) + }; + if wait_until_ready { + wait_past_readiness_budget( + || probe(&mut *on_wait_tick), + || service_still_live(paths, &record.service_id, service_pid_is_alive), + ) + } else { + probe(on_wait_tick) + } +} + +/// An equivalent service is already live. Without `--wait`, or once it is +/// serving, report it as is; otherwise wait for it like a fresh launch would. +pub(super) fn wait_on_existing_service( + paths: &AppPaths, + mut report: ManagedLaunchReport, + wait_until_ready: bool, + on_wait_begin: &mut dyn FnMut(&str, u16), + on_wait_tick: &mut dyn FnMut(Duration), +) -> Result { + if !wait_until_ready || report.status == "ready" { + return Ok(report); + } + let Ok(mut record) = ManagedServiceRecord::read_manifest(paths, &report.service_id) else { + return Ok(report); + }; + // The id of the server being waited on, which is not the caller's freshly + // generated one when it attached to an existing server. + on_wait_begin(&report.service_id, record.port); + // The key the caller generated belongs to a launch that did not happen; the + // existing service answers to its own. + let stored_key = endpoint_keys::endpoint_api_key(paths, &report.service_id); + let readiness = + wait_for_record_ready(paths, &record, true, stored_key.as_deref(), on_wait_tick); + if let Some(error) = died_during_wait( + paths, + &report.service_id, + true, + readiness, + service_pid_is_alive, + ) { + return Err(error); + } + report.status = status_for_readiness(readiness).to_owned(); + record_wait_outcome( + paths, + &mut record, + &report.status, + readiness == EndpointReadiness::Serving, + )?; + Ok(report) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::service_launch::test_support::reusable_record; + use crate::tests::test_paths; + use std::fs; + + #[test] + fn wait_past_readiness_budget_keeps_probing_a_live_starting_server() { + let mut readings = vec![ + EndpointReadiness::Unreachable, + EndpointReadiness::Unreachable, + EndpointReadiness::Serving, + ] + .into_iter(); + let mut probes = 0; + let result = wait_past_readiness_budget( + || { + probes += 1; + readings.next().expect("no probe after Serving") + }, + || true, + ); + assert_eq!(result, EndpointReadiness::Serving); + assert_eq!(probes, 3, "must outlast the first (45s) budget"); + } + + #[test] + fn wait_past_readiness_budget_stops_when_the_server_exits() { + let mut probes = 0; + let result = wait_past_readiness_budget( + || { + probes += 1; + EndpointReadiness::Unreachable + }, + || false, + ); + assert_eq!(result, EndpointReadiness::Unreachable); + assert_eq!(probes, 1); + } + + #[test] + fn wait_past_readiness_budget_returns_at_once_when_already_serving() { + let result = wait_past_readiness_budget( + || EndpointReadiness::Serving, + || panic!("liveness is irrelevant once serving"), + ); + assert_eq!(result, EndpointReadiness::Serving); + } + + fn record_tracking_pid(paths: &AppPaths, pid: u32, status: &str) -> ManagedServiceRecord { + let mut record = reusable_record(paths); + record.supervisor_pid = pid; + record.status = status.to_owned(); + record + } + + #[cfg(unix)] + fn exited_child_pid() -> u32 { + let mut child = std::process::Command::new("true").spawn().expect("spawn"); + let pid = child.id(); + child.wait().expect("reap"); + pid + } + + #[test] + #[cfg(unix)] + fn record_still_live_ends_the_wait_for_a_dead_pid_with_a_live_status() { + // The engine was killed before writing any terminal state: the status + // still says `starting`, but nothing tracked is running. + let (root, paths) = test_paths("wait-dead-pid"); + paths.ensure().expect("paths"); + let dead = record_tracking_pid(&paths, exited_child_pid(), "starting"); + assert!(!record_still_live(&dead, service_pid_is_alive)); + + let alive = record_tracking_pid(&paths, std::process::id(), "starting"); + assert!(record_still_live(&alive, service_pid_is_alive)); + + // A settled status ends the wait even while the pid is still around. + let stopped = record_tracking_pid(&paths, std::process::id(), "stopped"); + assert!(!record_still_live(&stopped, service_pid_is_alive)); + let _ = fs::remove_dir_all(root); + } + + #[test] + #[cfg(target_os = "linux")] + fn service_pid_is_alive_treats_an_unreaped_child_as_dead() { + // The serve supervisor is our own unreaped child; once it dies it is a + // zombie that `kill(pid, 0)` still reports as running. + let mut child = std::process::Command::new("true").spawn().expect("spawn"); + let pid = child.id(); + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while !process_is_zombie(pid) { + assert!(std::time::Instant::now() < deadline, "child never exited"); + std::thread::sleep(Duration::from_millis(10)); + } + assert!( + rocm_core::process_is_running(pid), + "a zombie still answers kill(pid, 0)" + ); + assert!(!service_pid_is_alive(pid)); + child.wait().expect("reap"); + } + + #[test] + fn service_still_live_follows_the_manifest_on_disk() { + // `rocm services stop` from another terminal writes only the manifest. + let (root, paths) = test_paths("wait-manifest"); + paths.ensure().expect("paths"); + let mut record = record_tracking_pid(&paths, std::process::id(), "starting"); + record.write().expect("write"); + assert!(service_still_live(&paths, &record.service_id, |_| true)); + + record.status = "stopped".to_owned(); + record.write().expect("write"); + assert!(!service_still_live(&paths, &record.service_id, |_| true)); + assert!(!service_still_live(&paths, "no-such-service", |_| true)); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn settle_exited_service_demotes_a_live_manifest_and_spares_a_settled_one() { + let (root, paths) = test_paths("wait-settle"); + paths.ensure().expect("paths"); + let mut record = record_tracking_pid(&paths, 0, "starting"); + record.write().expect("write"); + settle_exited_service(&paths, &record.service_id); + let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); + assert_eq!(after.status, "stopped"); + + // A record that already failed keeps its reason: the guard, not the + // overwrite, is what spares it (a `stopped` -> `stopped` rewrite would + // be invisible). + record.status = "failed".to_owned(); + record.write().expect("write"); + settle_exited_service(&paths, &record.service_id); + let after = ManagedServiceRecord::read_manifest(&paths, &record.service_id).unwrap(); + assert_eq!(after.status, "failed"); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn wait_on_existing_service_only_waits_for_one_that_is_not_serving_yet() { + // The probing path itself blocks for the readiness budget, so only the + // cases that must return at once are driven here; a manifest planted + // as `starting` would hang a wrong guard for 45 seconds and fail it. + let (root, paths) = test_paths("wait-existing"); + paths.ensure().expect("paths"); + let mut record = record_tracking_pid(&paths, 0, "starting"); + record.write().expect("write"); + let report = |status: &str| ManagedLaunchReport { + service_id: record.service_id.clone(), + endpoint_url: record.endpoint_url.clone(), + status: status.to_owned(), + already_running: true, + child_pid: None, + log_path: None, + manifest_path: None, + }; + let mut no_tick = |_: Duration| panic!("must not probe"); + + let untouched = wait_on_existing_service( + &paths, + report("starting"), + false, + &mut |_, _| panic!("no --wait, no banner"), + &mut no_tick, + ) + .expect("no --wait"); + assert_eq!(untouched.status, "starting"); + let ready = wait_on_existing_service( + &paths, + report("ready"), + true, + &mut |_, _| panic!("already ready, no banner"), + &mut no_tick, + ) + .expect("already ready"); + assert_eq!(ready.status, "ready"); + record.status = "starting".to_owned(); + assert_eq!( + ManagedServiceRecord::read_manifest(&paths, &record.service_id) + .unwrap() + .status, + "starting" + ); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn died_during_wait_fails_with_a_logs_pointer_and_records_the_death() { + let (root, paths) = test_paths("wait-died"); + paths.ensure().expect("paths"); + let record = record_tracking_pid(&paths, std::process::id(), "starting"); + record.write().expect("write"); + let id = record.service_id; + let status = || { + ManagedServiceRecord::read_manifest(&paths, &id) + .unwrap() + .status + }; + + // Not waiting, already serving, or still alive: no failure, no change. + let pending = EndpointReadiness::Unreachable; + assert!(died_during_wait(&paths, &id, false, pending, |_| false).is_none()); + assert!( + died_during_wait(&paths, &id, true, EndpointReadiness::Serving, |_| false).is_none() + ); + assert!(died_during_wait(&paths, &id, true, pending, |_| true).is_none()); + assert_eq!(status(), "starting"); + + // Dead server: the error names the logs command and the record is settled. + let error = died_during_wait(&paths, &id, true, pending, |_| false).expect("died"); + let message = error.to_string(); + assert!( + message.contains("exited before it became ready"), + "{message}" + ); + assert!( + message.contains(&format!("rocm logs --service {id}")), + "{message}" + ); + assert_eq!(status(), "stopped"); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn died_during_wait_adds_the_oom_guidance_for_a_vllm_launch_only() { + let (root, paths) = test_paths("wait-died-oom"); + paths.ensure().expect("paths"); + let mut record = record_tracking_pid(&paths, std::process::id(), "starting"); + record.engine = "vllm".to_owned(); + record.write().expect("write"); + fs::write( + &record.log_path, + "torch.OutOfMemoryError: HIP out of memory. Tried to allocate 7.21 GiB.\n", + ) + .expect("log"); + let id = record.service_id; + let pending = EndpointReadiness::Unreachable; + let message = died_during_wait(&paths, &id, true, pending, |_| false) + .expect("died") + .to_string(); + assert!(message.contains("--gpu-memory-utilization"), "{message}"); + + // Same log, another engine: that engine has no such flag to suggest. + let mut record = ManagedServiceRecord::read_manifest(&paths, &id).unwrap(); + record.engine = "lemonade".to_owned(); + "starting".clone_into(&mut record.status); + record.write().expect("write"); + let message = died_during_wait(&paths, &id, true, pending, |_| false) + .expect("died") + .to_string(); + assert!(!message.contains("--gpu-memory-utilization"), "{message}"); + let _ = fs::remove_dir_all(root); + } + + #[test] + fn record_wait_outcome_keeps_what_changed_on_disk_during_the_wait() { + let (root, paths) = test_paths("wait-outcome"); + paths.ensure().expect("paths"); + let mut held = record_tracking_pid(&paths, 111, "starting"); + held.write().expect("write"); + // While the wait ran, `rocmd` recorded a restart and a new supervisor. + let mut elsewhere = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); + elsewhere.restart_count = 3; + elsewhere.supervisor_pid = 222; + elsewhere.write().expect("write"); + + record_wait_outcome(&paths, &mut held, "ready", true).expect("outcome"); + let after = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); + assert_eq!(after.status, "ready"); + assert!(after.inference_verified_at_unix_ms.is_some()); + assert_eq!(after.restart_count, 3, "the restart count was reverted"); + assert_eq!(after.supervisor_pid, 222, "the supervisor pid was reverted"); + + record_wait_outcome(&paths, &mut held, "starting", false).expect("outcome"); + let after = ManagedServiceRecord::read_manifest(&paths, &held.service_id).unwrap(); + assert_eq!(after.status, "starting"); + let _ = fs::remove_dir_all(root); + } +} diff --git a/apps/rocm/src/service_launch/spawn.rs b/apps/rocm/src/service_launch/spawn.rs new file mode 100644 index 0000000000..0bc3d02a34 --- /dev/null +++ b/apps/rocm/src/service_launch/spawn.rs @@ -0,0 +1,1306 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Spawning the detached managed engine child shared by the background and +//! attached serve paths: process and stdio plumbing, [`ManagedLaunchReport`] and +//! [`ManagedSpawn`], and the serialized spawn transaction itself. + +use super::endpoint_auth::ensure_public_service_has_endpoint_key; +use super::launch_transaction::{ + ManagedPortDecision, ManagedReuseRefusal, abandon_failed_managed_launch, + managed_service_reuse_refusal, no_usable_gpu_refusal, resolve_managed_port_in_transaction, +}; +use crate::endpoint_keys; +use crate::engine_ipc::builtin_engine_serve_http_args; +#[cfg(not(windows))] +use crate::engine_runtime_select::apply_app_path_env; +use crate::engine_runtime_select::managed_service_launcher_path; +#[cfg(windows)] +use crate::engine_runtime_select::{app_path_env_var_refs, app_path_env_var_values}; +use crate::engines_cmd::env_root_for_service; +use crate::serve_port; +use crate::service_hints; +use crate::services_text::read_optional_tail_lines; +use crate::{device_policy_name, record_cli_audit_event}; +use anyhow::{Context, Result, bail}; +#[cfg(windows)] +use rocm_core::process_is_running; +use rocm_core::{AppPaths, ManagedServiceRecord}; +use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse}; +use std::fs; +#[cfg(not(windows))] +use std::io; +use std::path::{Path, PathBuf}; +#[cfg(not(windows))] +use std::process::{Command as ProcessCommand, Stdio}; +use std::thread; +use std::time::Duration; + +#[cfg(not(windows))] +pub(crate) fn detach_background_command(command: &mut ProcessCommand) { + rocm_core::detach_command_session(command); +} + +#[cfg(not(windows))] +pub(crate) fn attach_background_stdio( + command: &mut ProcessCommand, + log_path: Option<&Path>, +) -> Result<()> { + if let Some(log_path) = log_path { + let log = fs::OpenOptions::new() + .create(true) + .append(true) + .open(log_path) + .with_context(|| format!("failed to open {}", log_path.display()))?; + command + .stdout(Stdio::from(log.try_clone()?)) + .stderr(Stdio::from(log)); + } else { + command.stdout(Stdio::null()).stderr(Stdio::null()); + } + Ok(()) +} + +/// Windows spawn-liveness check: wait out the startup window, then fail with the +/// startup log when the child is already gone. The wait and the liveness probe +/// are injected so a unit test runs the exit-detected path on every platform, +/// including the Windows lane that actually calls it. +/// +/// # Errors +/// Fails with the startup-failure detail when `is_running` reports the child dead. +#[cfg_attr(not(windows), allow(dead_code))] +fn ensure_spawned_child_survives_startup( + child_pid: u32, + log_path: &Path, + settle: impl FnOnce(), + is_running: impl FnOnce(u32) -> bool, +) -> Result<()> { + settle(); + if is_running(child_pid) { + Ok(()) + } else { + bail!("{}", managed_engine_startup_failure_detail("", log_path)) + } +} + +/// `status_clause` is a pre-formatted fragment inserted right after "exited +/// immediately" (e.g. `" with status exit status: 1"` on Unix, where a real +/// `ExitStatus` is available; empty on Windows, where the liveness check +/// below only has a yes/no answer, not an exit code). +pub(crate) fn managed_engine_startup_failure_detail( + status_clause: &str, + log_path: &Path, +) -> String { + let mut recent_lines = read_optional_tail_lines(log_path, 80, "service log"); + if recent_lines.is_empty() { + for _ in 0..5 { + thread::sleep(Duration::from_millis(120)); + recent_lines = read_optional_tail_lines(log_path, 80, "service log"); + if !recent_lines.is_empty() { + break; + } + } + } + if recent_lines.is_empty() { + return format!( + "managed engine exited immediately{status_clause}; inspect {}", + log_path.display() + ); + } + format!( + "managed engine exited immediately{status_clause}; inspect {}\n\nrecent startup log output:\n{}", + log_path.display(), + recent_lines.join("\n") + ) +} +#[cfg(not(windows))] +pub(crate) fn managed_service_process_command(program: &Path, args: &[String]) -> ProcessCommand { + let mut command = ProcessCommand::new(program); + command.args(args); + command +} + +/// Engine-neutral result of a managed launch. Returned rather than printed so the +/// caller can render it either as the rich deployment summary (interactive TTY) or +/// as the plain line-by-line form (piped output, chat assistant), from one code path. +pub(crate) struct ManagedLaunchReport { + pub(crate) service_id: String, + /// `http://host:port/v1`. + pub(crate) endpoint_url: String, + /// `"ready"` (inference confirmed), `"running"` (model listed but not serving + /// yet), `"starting"` (endpoint not answering), or the existing service's + /// status when nothing was spawned. + pub(crate) status: String, + /// True when an equivalent service was already live and nothing was spawned. + pub(crate) already_running: bool, + pub(crate) child_pid: Option, + pub(crate) log_path: Option, + pub(crate) manifest_path: Option, +} + +/// Either an already-live service (nothing spawned) or a freshly spawned engine +/// child that is `running` but not yet HTTP-ready. +/// +/// Split out of [`start_managed_service`](super::start_managed_service) so the +/// attached (`--verbose` / `--foreground`) serve path can spawn the very same +/// detached child and stream its log live from the first line — including +/// startup — instead of blocking on the readiness wait before any output appears. +pub(super) enum ManagedSpawn { + AlreadyRunning(ManagedLaunchReport), + // `ManagedServiceRecord` is large; box it so the two variants stay a similar + // size (clippy::large_enum_variant). + Spawned { + record: Box, + child_pid: u32, + }, +} + +/// Spawn the detached engine child shared by the managed (background) and +/// attached (`--verbose`/`--foreground`) serve paths. Returns before the HTTP +/// readiness wait; callers decide whether to block on readiness +/// ([`start_managed_service`](super::start_managed_service)) or start tailing the +/// log immediately ([`run_attached_service`](super::run_attached_service)). +/// +/// The whole body is one serialized transaction under the shared allocation +/// lock: duplicate detection, live-reservation refresh, candidate leasing, +/// manifest/log creation, argv preparation, the bounded immediate-exit check, +/// and record publication. Readiness deliberately runs after the lock is +/// released. +#[allow(clippy::too_many_arguments)] +pub(super) fn spawn_managed_engine_child( + paths: &AppPaths, + engine: &str, + service_id: &str, + requested_model: &str, + resolve: &ResolveModelResponse, + host: &str, + port_request: serve_port::PortRequest, + device_policy: &DevicePolicy, + gpu_indices: &[u32], + runtime_id: Option<&str>, + env_id: Option<&str>, + engine_recipe: Option<&EngineRecipeHint>, + require_api_key: bool, + gpu_refusal_waived_for_reuse: bool, +) -> Result { + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + + // No lock is acquired here: every production caller + // ([`start_managed_service`], [`run_attached_service`]) already holds + // [`rocm_core::AppPaths::managed_launch_lock_path`] for the whole of this + // call, so no two transactions can choose the same port or publish + // competing records for one engine+model. A second acquisition in this + // frame would self-deadlock, since the lock is exclusive per open file + // description, not per process. + // + // Idempotency guard: if a managed service for this engine+model is already + // alive, surface it and spawn nothing. A second `serve --managed` (e.g. the + // chat assistant re-issuing the same request) is treated as satisfied, not + // an error. Keyed on engine+canonical model — the freshly generated + // `service_id` is timestamp-unique and would never match an existing one. + // Stale/dead services fall through and relaunch normally. + let requested_recipe_json = engine_recipe + .map(serde_json::to_string) + .transpose() + .context("failed to encode engine recipe hint")?; + let lease = match resolve_managed_port_in_transaction( + paths, + engine, + &resolve.canonical_model_id, + host, + port_request, + &mut serve_port::loopback_bind_probe, + )? { + ManagedPortDecision::AlreadyLive(existing) => { + let existing = *existing; + // Asked through the shared predicate rather than re-tested here, so this + // refusal and `serve()`'s reuse pre-gate cannot disagree about whether a + // service is reusable. When they disagreed, the pre-gate let a + // non-reusable service bypass the no-GPU refusal and the user was sent + // here to be told to stop a working server for nothing. + // + // The same device policy the serve path passes. This side used to pass + // `None` to keep the spawn's acceptance untouched, but that opted the + // spawn out of a term it has the input for, left its own + // `DevicePolicyTooWeak` bail unreachable, and made the serve path's + // argument droppable. Nothing observable widens: `parse_device_policy` + // is the only producer of a `DevicePolicy` on every path that reaches + // here and it rejects `cpu_only`, so no record this CLI writes can + // carry the policy this term refuses. + match managed_service_reuse_refusal( + &existing, + requested_recipe_json.as_deref(), + require_api_key, + device_policy, + ) { + Some(ManagedReuseRefusal::RecipeMismatch) => bail!( + "managed service `{}` is already running for engine `{engine}` and model `{}` with different serve options (recipe hint, tool-call parser, or generation defaults); stop it and run `rocm serve` again to apply the requested options", + existing.service_id, + resolve.canonical_model_id + ), + Some(ManagedReuseRefusal::MissingApiKey) => bail!( + "managed service `{}` is already running for engine `{engine}` and model `{}` \ + without authentication, and a running server cannot be given a key it did not \ + start with; stop it with `{}` and run the command again to \ + serve it with `--require-api-key`", + existing.service_id, + resolve.canonical_model_id, + service_hints::service_stop_hint(&existing.service_id) + ), + // Reachable only from a record whose `device_policy` says + // `cpu_only`, which no `rocm serve` can write today + // (`parse_device_policy` refuses that policy outright) — so this is + // a guard against a record written by some future path, not a + // refusal users see now. The advice still names a command that + // clears the condition: stopping the service removes the only thing + // this term can object to, and the rerun then launches normally + // (AGENTS.md section 3). + Some(ManagedReuseRefusal::DevicePolicyTooWeak) => bail!( + "managed service `{}` is already running under a device policy that cannot \ + satisfy this request; stop it with `{}` and run the command again", + existing.service_id, + service_hints::service_stop_hint(&existing.service_id) + ), + None => {} + } + // An explicit port is a demand, not a hint: reporting success on a + // different port than the one asked for would silently drop it. + if let Some(port) = port_request.explicit() + && port != existing.port + { + bail!( + "managed service `{}` is already serving engine `{engine}` and model `{}` on port {}, so port {port} would not be used; stop it with `{}` and run the command again to serve on that port", + existing.service_id, + resolve.canonical_model_id, + existing.port, + service_hints::service_stop_hint(&existing.service_id) + ); + } + record_cli_audit_event( + paths, + "service", + "managed_service_launch_skipped", + "info", + format!( + "skipped duplicate managed launch engine={engine} model={} existing_service_id={} status={}", + resolve.canonical_model_id, existing.service_id, existing.status + ), + Some(&existing.service_id), + ); + return Ok(ManagedSpawn::AlreadyRunning(ManagedLaunchReport { + service_id: existing.service_id, + endpoint_url: existing.endpoint_url, + status: existing.status, + already_running: true, + child_pid: None, + log_path: None, + manifest_path: None, + })); + } + ManagedPortDecision::Lease(lease) => lease, + }; + + // Nothing live to reuse — and if the only reason this invocation got past + // `serve()`'s no-usable-GPU pre-flight was a reusable service, that premise + // has just been falsified. `serve()` decided it from a snapshot taken before + // GPU selection and the launch lock; the check above is the authoritative + // one, so a service that died in between (an OOM, say) would otherwise fall + // through to a real spawn on a host with no usable GPU, and the user would + // get whatever the engine says about having no device instead of the + // actionable refusal the pre-flight exists to give. Raise it here, where the + // premise is known to be false. + if gpu_refusal_waived_for_reuse { + bail!("{}", no_usable_gpu_refusal(device_policy)); + } + + let mut record = ManagedServiceRecord::new( + paths, + service_id, + engine, + requested_model, + resolve.canonical_model_id.clone(), + host, + lease.port(), + "managed", + 0, + runtime_id.map(str::to_owned), + env_id.map(str::to_owned), + Some(device_policy_name(device_policy).to_owned()), + ); + record.gpu_indices = gpu_indices.to_vec(); + record.served_model_id = resolve.served_model_id.clone(); + record.engine_recipe_json = requested_recipe_json; + // The flag the user actually passed, carried through rather than re-derived. + // + // Deriving it from key-file presence looked equivalent and was not: + // `resolve_endpoint_auth` mints a key for *every* non-loopback bind whether or + // not auth was demanded, so a plain `--host 0.0.0.0 --allow-public-bind` + // recorded `true` here. The guard below tests this field before the bind + // address, so that service was then refused with a message naming a flag it + // never used and a relaunch command that drops `--allow-public-bind` — the + // public-bind branch, which carries the right command, became unreachable. + // + // This field means "the user demanded auth on a bind that would not otherwise + // require it". A public bind needs no such record; its address still says so. + record.requires_api_key = require_api_key; + // Publishing the record inside the lock is what reserves the leased port for + // every other transaction. From here on, any failure must remove it again. + record.write()?; + + match launch_managed_engine_child_locked( + paths, + &mut record, + lease, + engine, + service_id, + resolve, + host, + device_policy, + gpu_indices, + runtime_id, + env_id, + engine_recipe, + #[cfg(not(windows))] + None, + ) { + Ok(child_pid) => Ok(ManagedSpawn::Spawned { + record: Box::new(record), + child_pid, + }), + Err(error) => { + // The single cleanup choke point for every failure path below the + // record's first write: `record.engine_pid` tells it whether a child + // ever ran, so a pre-spawn failure deletes everything and a post-spawn + // one keeps the record and log (marked `status = "stopped"`) instead. + abandon_failed_managed_launch(paths, &record); + Err(error) + } + } +} + +// Test-only, thread-scoped spawn interceptor for callers that cannot take a +// `spawn_override` argument (`start_managed_service`, `run_attached_service`): +// lets a test observe state at the instant production would spawn the child. +#[cfg(all(test, not(windows)))] +thread_local! { + pub(super) static TEST_SPAWN_HOOK: std::cell::RefCell>>> = + const { std::cell::RefCell::new(None) }; +} + +#[cfg(all(test, not(windows)))] +fn test_spawn_hook() -> Option>> { + TEST_SPAWN_HOOK.with(|hook| hook.borrow().clone()) +} + +#[cfg(not(windows))] +type SpawnOverride<'a> = dyn Fn(&mut ProcessCommand) -> io::Result + 'a; + +/// The side-effecting half of the launch transaction: create the log, prepare +/// concrete argv, hand the leased port to the child, spawn it, reject a child +/// that has already exited, and publish the running record. +/// +/// Split out of [`spawn_managed_engine_child`] so that *every* failure below the +/// record's first write is funnelled through one cleanup path before the +/// launch lock is released. Returns the child pid. +#[allow(clippy::too_many_arguments)] +fn launch_managed_engine_child_locked( + paths: &AppPaths, + record: &mut ManagedServiceRecord, + lease: rocm_core::LoopbackPortLease, + engine: &str, + service_id: &str, + resolve: &ResolveModelResponse, + host: &str, + device_policy: &DevicePolicy, + gpu_indices: &[u32], + runtime_id: Option<&str>, + env_id: Option<&str>, + engine_recipe: Option<&EngineRecipeHint>, + // Test-only seam: lets a test assert the leased port is actually bindable + // (i.e. released) at the exact point production would spawn the child, + // without needing to fake or intercept the real spawn mechanics. + #[cfg(not(windows))] spawn_override: Option<&SpawnOverride<'_>>, +) -> Result { + let port = record.port; + if let Some(parent) = record.engine_state_path.parent() { + fs::create_dir_all(parent) + .with_context(|| format!("failed to create {}", parent.display()))?; + } + fs::File::create(&record.log_path) + .with_context(|| format!("failed to create {}", record.log_path.display()))?; + let current_exe = managed_service_launcher_path() + .context("failed to resolve current rocm executable path")?; + let serve_args = builtin_engine_serve_http_args( + engine, + service_id, + &resolve.canonical_model_id, + host, + port, + device_policy, + gpu_indices, + runtime_id, + env_id, + engine_recipe, + &record.engine_state_path, + Some(&record.log_path), + )?; + let engine_envs_root = env_root_for_service(paths, engine, runtime_id, env_id)?; + // Hand the child the *path* to the endpoint key file (public bind only) via the + // environment. A path — not the secret value — is what the detached-spawn + // primitives accept as an env override, and it keeps the key off both the argv + // and the environment block. `serve()` wrote the file before spawning. + // Validity, not mere existence: the engine adapters resolve the key with + // `endpoint_api_key_from_file` and enforce nothing when it yields `None`, so + // an empty or malformed key file would otherwise satisfy the guard below and + // still produce an unauthenticated public listener. + let endpoint_key_file = endpoint_keys::endpoint_key_file_if_present(paths, service_id) + .filter(|path| rocm_engine_protocol::endpoint_api_key_from_file(path).is_some()); + // `serve()` already resolved and stored the key for a public bind, so the + // public-bind branch cannot fire on the fresh-launch path today. It is the + // shared choke point for managed spawns, so enforce the invariant here too + // rather than relying on every future caller having done so. + // + // `record.requires_api_key` is passed, not a literal, and the two arguments + // are deliberately different things: that field is the `--require-api-key` + // flag the caller passed, `endpoint_key_file` is whether a *usable* key is on + // disk. A present but empty or malformed key file is where they disagree, and + // is exactly what the `requires_api_key` branch exists to refuse. + // + // The field is threaded, never derived from the key file. Deriving it marked + // every public bind as having demanded auth — see the assignment above. + ensure_public_service_has_endpoint_key( + host, + endpoint_key_file.is_some(), + record.requires_api_key, + )?; + // Hand the port over: close the probe socket immediately before the spawn, + // still under the caller's launch lock, so no other ROCm transaction can + // take it in the meantime. On a custom host the lease never held a socket + // and this is a no-op — the engine owns that bind. + debug_assert_eq!(lease.port(), port); + lease.release(); + #[cfg(windows)] + let child_pid = { + let env_values = app_path_env_var_values(paths, engine_envs_root.as_deref()); + let mut env_refs = app_path_env_var_refs(&env_values); + if let Some(key_file) = endpoint_key_file.as_deref() { + env_refs.push((rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV, key_file)); + } + let child_pid = rocm_core::spawn_detached_no_inherit(¤t_exe, &serve_args, &env_refs) + .context("failed to launch managed engine process")?; + // Set before the immediate-exit check below, so a bail! here still leaves + // `record.engine_pid` telling cleanup that a child was spawned. The + // start-time token is captured here too, while the child is certainly + // ours: cleanup binds its kill to it rather than to whatever owns the + // pid by then. + record.record_engine_identity(child_pid); + ensure_spawned_child_survives_startup( + child_pid, + &record.log_path, + || thread::sleep(Duration::from_millis(200)), + process_is_running, + )?; + child_pid + }; + #[cfg(not(windows))] + let child_pid = { + let mut command = managed_service_process_command(¤t_exe, &serve_args); + command.stdin(Stdio::null()); + attach_background_stdio(&mut command, Some(&record.log_path))?; + detach_background_command(&mut command); + apply_app_path_env(&mut command, paths); + if let Some(engine_envs_root) = engine_envs_root.as_deref() { + command.env("ROCM_CLI_ENGINE_ENVS_ROOT", engine_envs_root); + } + if let Some(key_file) = endpoint_key_file.as_deref() { + command.env(rocm_engine_protocol::ENDPOINT_API_KEY_FILE_ENV, key_file); + } + #[cfg(all(test, not(windows)))] + let hook = test_spawn_hook(); + #[cfg(all(test, not(windows)))] + let spawn_override = spawn_override.or(hook.as_deref()); + let mut child = match spawn_override { + Some(spawn) => spawn(&mut command), + None => command.spawn(), + } + .context("failed to launch managed engine process")?; + let child_pid = child.id(); + // Set before the immediate-exit check below, so a bail! here still leaves + // `record.engine_pid` telling cleanup that a child was spawned. The + // start-time token is captured here too, while the child is certainly + // ours: cleanup binds its kill to it rather than to whatever owns the + // pid by then. + record.record_engine_identity(child_pid); + thread::sleep(Duration::from_millis(200)); + if let Some(status) = child + .try_wait() + .context("failed to check managed engine startup state")? + { + bail!( + "{}", + managed_engine_startup_failure_detail( + &format!(" with status {status}"), + &record.log_path + ) + ); + } + child_pid + }; + // The child is both roles until the engine state file names a separate + // server process. Each role gets its own token, captured while the child is + // alive: a stop pairs every recorded PID with its own token, and the two + // PIDs diverge as soon as `rocmd` restarts the service — at which point an + // engine PID recorded without one could only be signalled blind. + // The engine's token was already captured right after the spawn (so cleanup + // can bind to it); re-capturing here could replace it with a later owner's. + record.record_supervisor_identity(child_pid); + record.status = "running".to_owned(); + if let Err(error) = record.write() { + // Cleanup is the caller's job now: `record.engine_pid` (set above) is + // already enough for `abandon_failed_managed_launch` to terminate this + // child, so there is nothing left for this inner path to do beyond + // surfacing the error. + return Err(error.context("failed to publish the managed service record")); + } + + Ok(child_pid) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::managed_service_status::load_managed_services; + use crate::service_launch::launch_transaction::live_managed_services; + #[cfg(not(windows))] + use crate::service_launch::test_support::sleeper_test_name; + use crate::service_launch::test_support::write_live_service_record; + use crate::tests::test_paths; + use rocm_core::DEFAULT_LOCAL_HOST; + + #[test] + fn a_child_that_exited_during_the_startup_window_is_reported_with_its_log() -> Result<()> { + let dir = tempfile::tempdir()?; + let log = dir.path().join("engine.log"); + fs::write(&log, "boom: engine could not start\n")?; + let settled = std::cell::Cell::new(false); + let error = ensure_spawned_child_survives_startup( + 4242, + &log, + || settled.set(true), + |pid| { + assert_eq!(pid, 4242); + assert!(settled.get(), "liveness must be probed after the wait"); + false + }, + ) + .expect_err("a dead child must fail the launch"); + assert!(settled.get(), "the startup window must elapse first"); + assert!(format!("{error:#}").contains("boom"), "{error:#}"); + ensure_spawned_child_survives_startup(4242, &log, || {}, |_| true)?; + Ok(()) + } + + #[test] + fn spawn_managed_engine_child_refuses_an_explicit_port_a_live_equivalent_does_not_serve() + -> Result<()> { + let (root, paths) = test_paths("dup-managed-explicit-port-differs"); + paths.ensure()?; + let mut existing = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-1000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11510, + "managed", + std::process::id(), + None, + None, + None, + ); + existing.status = "ready".to_owned(); + existing.engine_pid = Some(std::process::id()); + existing.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + let launch = |port_request| { + spawn_managed_engine_child( + &paths, + "lemonade", + "lemonade-qwen-2000", + "qwen", + &resolve, + "127.0.0.1", + port_request, + &resolve.device_policy, + &[], + None, + None, + None, + false, + false, + ) + }; + + let different = launch(serve_port::PortRequest::Explicit(11511)); + // The same port, and no port at all, still reuse the live service. + let same = launch(serve_port::PortRequest::Explicit(11510)); + let auto = launch(serve_port::PortRequest::Auto); + let _ = fs::remove_dir_all(root); + + let Err(error) = different else { + panic!("a port the live service does not serve must be refused") + }; + let message = error.to_string(); + assert!( + message.contains("on port 11510") && message.contains("port 11511 would not be used"), + "{message}" + ); + assert!(matches!(same?, ManagedSpawn::AlreadyRunning(_))); + assert!(matches!(auto?, ManagedSpawn::AlreadyRunning(_))); + Ok(()) + } + + #[test] + fn spawn_managed_engine_child_blocks_reuse_with_mismatched_recipe() -> Result<()> { + // A live service recorded with one recipe (e.g. a tool-call parser flag) + // must reject a relaunch requesting a different recipe rather than + // silently reusing the old server, and the error must not claim the + // mismatch is specifically about generation defaults when it could stem + // from any recipe field. + let (root, paths) = test_paths("dup-managed-recipe-mismatch"); + paths.ensure()?; + let mut existing = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-1000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11510, + "managed", + std::process::id(), + None, + None, + None, + ); + existing.status = "ready".to_owned(); + existing.engine_pid = Some(std::process::id()); + existing.engine_recipe_json = Some(serde_json::to_string(&EngineRecipeHint { + contract_version: rocm_engine_protocol::ENGINE_RECIPE_CONTRACT_VERSION.to_owned(), + engine: "lemonade".to_owned(), + required_flags: vec!["--tool-call-parser".to_owned(), "hermes".to_owned()], + ..EngineRecipeHint::default() + })?); + existing.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + let requested_recipe = EngineRecipeHint { + contract_version: rocm_engine_protocol::ENGINE_RECIPE_CONTRACT_VERSION.to_owned(), + engine: "lemonade".to_owned(), + required_flags: vec!["--temperature".to_owned(), "0.5".to_owned()], + ..EngineRecipeHint::default() + }; + + let result = spawn_managed_engine_child( + &paths, + "lemonade", + "lemonade-qwen-2000", + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Explicit(11511), + &resolve.device_policy, + &[], + None, + None, + Some(&requested_recipe), + false, + false, + ); + let _ = fs::remove_dir_all(root); + + let Err(error) = result else { + panic!("mismatched recipe on a live service must be rejected") + }; + let message = error.to_string(); + assert!( + message.contains("different serve options"), + "message should describe the mismatch generically: {message}" + ); + assert!( + message.contains("recipe hint, tool-call parser, or generation defaults"), + "message should not single out generation defaults as the sole cause: {message}" + ); + Ok(()) + } + + #[test] + fn spawn_managed_engine_child_refuses_to_reuse_an_unauthenticated_service() -> Result<()> { + // `--require-api-key` used to be accepted and dropped on this path. The + // reuse branch returns before the flag is recorded and before the + // endpoint key is checked, so a caller demanding auth got a server that + // never had it, with no error. `rocm remote serve` then published that + // endpoint onto the tailnet and printed a freshly minted key under "the + // API key above is what stops anyone else calling it" — a false + // assurance about an endpoint the whole tailnet can reach. + // + // Asserted through `spawn_managed_engine_child` rather than against the + // guard's own arguments: the defect was the early return, so only the + // real call site can fail for it. + let (root, paths) = test_paths("dup-managed-unauthenticated-reuse"); + paths.ensure()?; + let mut existing = ManagedServiceRecord::new( + &paths, + "lemonade-qwen-3000", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + 11520, + "managed", + std::process::id(), + None, + None, + None, + ); + existing.status = "ready".to_owned(); + existing.engine_pid = Some(std::process::id()); + // The state that matters: live, matching, and serving without auth. + existing.requires_api_key = false; + existing.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let result = spawn_managed_engine_child( + &paths, + "lemonade", + "lemonade-qwen-3001", + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Explicit(11520), + &resolve.device_policy, + &[], + None, + None, + None, + // The demand that used to be silently discarded. + true, + false, + ); + let _ = fs::remove_dir_all(root); + + let Err(error) = result else { + panic!( + "reusing an unauthenticated service must not satisfy `--require-api-key`; a \ + satisfied reuse leaves the endpoint open while the caller is told it is not" + ) + }; + let message = error.to_string(); + assert!( + message.contains("without authentication"), + "the refusal must say why it refused: {message}" + ); + // The printed hint must be the exact, directly-runnable command: + // `main.rs`'s `service_actions_require_yes_and_render_sandbox_result` + // proves this exact form (`rocm services stop --yes`) is the one + // the approval gate itself recommends. That running this form actually + // clears *this* refusal — a service reused without the key it is now + // being asked to require — is proven by + // `@id:serve-require-api-key-refuses-unauthenticated-reuse` + // (`tests/e2e-cucumber/features/model_serving.feature`, serve-24), + // which reproduces this exact condition, runs the printed hint, and + // asserts the record is left `stopped` — not by `stop_managed_services` + // in `tests/e2e-cucumber/tests/e2e.rs`, which runs this same + // invocation as best-effort teardown and never asserts success. + assert!( + message.contains("rocm services stop lemonade-qwen-3000 --yes"), + "the refusal must name the way out, with the service to stop, directly runnable with --yes: {message}" + ); + Ok(()) + } + + #[test] + fn a_managed_spawn_refuses_an_invalid_key_file_on_a_service_that_requires_one() -> Result<()> { + // Drives the real call site, not the guard's own arguments. The service + // was launched with `--require-api-key`, and its key file is present but + // invalid — so `requires_api_key` is true while `key_present` is false, + // which is the only way to reach the first branch. A test calling + // `ensure_public_service_has_endpoint_key` directly cannot catch a + // mis-wired call site, which is the defect that has occurred here twice. + // + // The flag is passed explicitly rather than inferred from the key file. + // Inferring it marked every public bind as having demanded auth, because + // a public bind always has a key file whether or not it asked for one. + let (root, paths) = test_paths("managed-spawn-invalid-key"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + + // Non-empty but invalid (embedded CR/LF), so the file exists and is not + // merely empty (requires_api_key = true) but yields no usable key + // (key_present = false) — `endpoint_api_key_from_file` rejects embedded + // control characters. An empty seed here would let the post-condition + // below pass even if the real cleanup call never ran, since + // `endpoint_api_key` already reads back `None` for an empty file whether + // or not it was ever deleted. + endpoint_keys::store_endpoint_api_key( + &paths, + "lemonade-qwen-3000", + "bad-key\r\nX-Injected: value", + )?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let result = spawn_managed_engine_child( + &paths, + "lemonade", + "lemonade-qwen-3000", + "qwen", + &resolve, + // Loopback on purpose: the public-bind branch must not be what + // refuses this, or the test would pass with the guard disabled. + "127.0.0.1", + serve_port::PortRequest::Explicit(11512), + &resolve.device_policy, + &[], + None, + None, + None, + true, + false, + ); + + // This guard fires before any child is spawned, so `record.engine_pid` + // is still `None` when `spawn_managed_engine_child`'s cleanup arm calls + // `abandon_failed_managed_launch` — proving its pre-spawn branch + // (delete everything) runs from the real call site, not just from the + // `abandoned_launch_leaves_no_record_log_or_endpoint_key` test that + // calls the helper directly. + let manifest_path = paths.service_manifest_path("lemonade-qwen-3000"); + let log_path = paths.service_log_path("lemonade-qwen-3000"); + let engine_state_path = paths.service_engine_state_path("lemonade", "lemonade-qwen-3000"); + let key_file_path = endpoint_keys::endpoint_key_file_path(&paths, "lemonade-qwen-3000"); + let live = live_managed_services(&paths); + let _ = fs::remove_dir_all(root); + + let Err(error) = result else { + panic!("a service requiring a key must not spawn with an unusable key file") + }; + let message = error.to_string(); + assert!( + message.contains("--require-api-key"), + "the refusal must name the flag the service was launched with: {message}" + ); + assert!( + message.contains("without authentication"), + "the refusal must say what the risk is: {message}" + ); + assert!( + !manifest_path.exists(), + "a pre-spawn failure must not leave a record" + ); + assert!( + !log_path.exists(), + "a pre-spawn failure must not leave a log" + ); + assert!( + !engine_state_path.exists(), + "a pre-spawn failure must not leave engine state" + ); + assert!( + !key_file_path.exists(), + "a pre-spawn failure must delete the unusable key file from disk, not merely \ + leave it unreadable as a key" + ); + assert!( + live.is_empty(), + "a pre-spawn failure must leave no reservation" + ); + Ok(()) + } + + #[test] + fn a_failed_spawn_past_the_first_write_terminates_the_child_and_keeps_an_abandoned_record() + -> Result<()> { + // Drives the real call site's *post*-spawn cleanup branch, which + // neither `abandoned_launch_*` test reaches (they call + // `abandon_failed_managed_launch` directly). An engine name this + // binary does not build in makes the real spawned child (a second + // `rocm` process) exit almost immediately via + // `run_builtin_engine_serve_http`'s `other => bail!(...)` arm, after + // `launch_managed_engine_child_locked` has already set + // `record.engine_pid` — so cleanup must terminate (already-dead) the + // child, keep the record and log for inspection, mark the record + // `stopped` (a `failed` one would be respawned by rocmd recovery), and + // clear only the endpoint key. + // + // The child always dies near-instantly, but `spawn_managed_engine_child` + // only reports that within a fixed startup-check window. On a loaded or + // antivirus-scanning Windows CI box, a fresh debug `rocm.exe` can + // occasionally be slow enough to start that the window is missed even + // though the child is already doomed — so poll by retrying a fresh + // attempt (its own service id/port, so no stale manifest collides) up + // to a deadline instead of asserting on one race-prone attempt. + let deadline = std::time::Instant::now() + Duration::from_secs(10); + let mut attempt: u16 = 0; + let (root, paths, service_id, result) = loop { + attempt += 1; + let (root, paths) = + test_paths(&format!("managed-spawn-unknown-engine-cleanup-{attempt}")); + paths.ensure()?; + let service_id = format!("no-such-engine-qwen-{}", 4000 + attempt); + endpoint_keys::store_endpoint_api_key(&paths, &service_id, "test-key")?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let result = spawn_managed_engine_child( + &paths, + "no-such-engine", + &service_id, + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Explicit(11_514 + attempt), + &resolve.device_policy, + &[], + None, + None, + None, + false, + false, + ); + + if result.is_ok() && std::time::Instant::now() < deadline { + let _ = fs::remove_dir_all(&root); + continue; + } + break (root, paths, service_id, result); + }; + + let manifest_path = paths.service_manifest_path(&service_id); + let log_path = paths.service_log_path(&service_id); + let persisted = load_managed_services(&paths) + .expect("re-read manifests") + .into_iter() + .find(|candidate| candidate.service_id == service_id); + let key = endpoint_keys::endpoint_api_key(&paths, &service_id); + let live = live_managed_services(&paths); + + let Err(error) = result else { + panic!("an engine this binary does not build in must not report success") + }; + let message = error.to_string(); + assert!( + message.contains("managed engine exited immediately"), + "the failure must come from the spawned child dying, not an earlier guard: {message}" + ); + assert!( + manifest_path.exists(), + "a post-spawn failure must keep the record for inspection" + ); + assert!( + log_path.exists(), + "a post-spawn failure must keep the log for inspection" + ); + let persisted = + persisted.expect("a post-spawn failure must leave the record on disk, not delete it"); + // `stopped`, not `failed`: rocmd's recovery restarts a `failed` record + // without a stop marker, which would respawn a launch the CLI just + // reported as failed (pinned on the rocmd side by + // `abandoned_launch_record_is_not_a_recovery_candidate`). + assert_eq!( + persisted.status, "stopped", + "a post-spawn failure must mark the record abandoned, not leave it stuck" + ); + assert_eq!( + key, None, + "a post-spawn failure must still clear the endpoint key" + ); + assert!( + live.is_empty(), + "an abandoned record must not still reserve its port" + ); + let _ = fs::remove_dir_all(root); + Ok(()) + } + + #[test] + fn select_gpu_indices_under_launch_lock_blocks_until_the_holder_releases() -> Result<()> { + // Exercises the real lock-acquisition function production callers use + // (`select_gpu_indices_under_launch_lock`) against a lock another thread + // actually holds, so a refactor that dropped the lock early would fail + // here. `spawn_managed_engine_child` is then run for real, but only via + // its `AlreadyLive` short-circuit (a pre-seeded duplicate record), so no + // child process is ever launched. The holder writes nothing: this does + // not show a write made under the lock being observed. + use std::sync::{Arc, Barrier}; + use std::time::Instant; + + let (root, paths) = test_paths("launch-lock-holds-across-spawn"); + paths.ensure()?; + write_live_service_record(&paths, "lemonade-held-1", "held-canonical", 11_520, "ready")?; + + let barrier = Arc::new(Barrier::new(2)); + let hold_for = Duration::from_millis(250); + + let holder_paths = paths.clone(); + let holder_barrier = Arc::clone(&barrier); + let holder = thread::spawn(move || -> Result<()> { + let _lock = rocm_core::FileLock::acquire(holder_paths.managed_launch_lock_path())?; + holder_barrier.wait(); + thread::sleep(hold_for); + Ok(()) + }); + + barrier.wait(); + let started = Instant::now(); + let (gpu_indices, launch_lock) = crate::gpu_policy::select_gpu_indices_under_launch_lock( + &paths, + true, + None, + || None, + None, + None, + )?; + let waited = started.elapsed(); + + let resolve = ResolveModelResponse { + canonical_model_id: "held-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + let spawn = spawn_managed_engine_child( + &paths, + "lemonade", + "lemonade-held-2", + "held-model", + &resolve, + DEFAULT_LOCAL_HOST, + serve_port::PortRequest::Auto, + &resolve.device_policy, + &gpu_indices, + None, + None, + None, + false, + false, + ); + drop(launch_lock); + holder.join().expect("holder thread")?; + let _ = fs::remove_dir_all(root); + + // `thread::sleep` only guarantees *at least* `hold_for`, but under a + // loaded host (e.g. a full parallel `cargo test` run on WSL2) timer + // coalescing can make the measured wait a hair short of the nominal + // hold. Allow a small margin so the test still fails loudly if the + // call returns early, without being flaky on actual block duration. + let margin = Duration::from_millis(20); + assert!( + waited + margin >= hold_for, + "select_gpu_indices_under_launch_lock returned before the holder released the lock: waited {waited:?}, held {hold_for:?}" + ); + match spawn? { + ManagedSpawn::AlreadyRunning(report) => { + assert_eq!(report.service_id, "lemonade-held-1"); + } + ManagedSpawn::Spawned { .. } => { + panic!( + "expected the pre-seeded AlreadyLive record to short-circuit, not a real spawn" + ) + } + } + Ok(()) + } + + #[test] + #[cfg(not(windows))] + fn launch_releases_the_leased_port_before_spawning_the_child() -> Result<()> { + // `lease.release()` must run before the child is spawned, not after, + // or the engine is handed a port the CLI is still holding open. + // Proven via the injected `spawn_override` seam: it asserts a fresh + // bind on the leased port succeeds at the exact instant production + // would spawn, which only holds if the lease was already released. + let (root, paths) = test_paths("launch-releases-port-before-spawn"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + + // Ephemeral, not a fixed number in the automatic range, which a real + // `rocm serve` on the developer's machine could be holding. + let port = std::net::TcpListener::bind(("127.0.0.1", 0))? + .local_addr()? + .port(); + let lease = rocm_core::lease_loopback_port(port)?; + let mut record = ManagedServiceRecord::new( + &paths, + "lemonade-release-before-spawn", + "lemonade", + "qwen", + "qwen-canonical", + "127.0.0.1", + port, + "managed", + 0, + None, + None, + None, + ); + record.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let bind_succeeded = std::cell::Cell::new(false); + let spawn_override: &dyn Fn(&mut ProcessCommand) -> io::Result = + &|_command| { + // Retry briefly: a sibling test's concurrent fork can hold a copy + // of the just-released socket until its exec closes it. + let deadline = std::time::Instant::now() + Duration::from_secs(2); + while !bind_succeeded.get() && std::time::Instant::now() < deadline { + bind_succeeded.set(std::net::TcpListener::bind(("127.0.0.1", port)).is_ok()); + } + // A long-lived no-op child stands in for the engine: `_command` + // carries real serve args this test binary cannot execute as + // itself (see `managed_launch_cleanup_sleeper_child`). + ProcessCommand::new(std::env::current_exe().expect("test binary")) + .arg(sleeper_test_name()) + .arg("--exact") + .arg("--ignored") + .env("ROCM_TEST_SLEEPER_CHILD", "1") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + }; + + let result = launch_managed_engine_child_locked( + &paths, + &mut record, + lease, + "lemonade", + "lemonade-release-before-spawn", + &resolve, + "127.0.0.1", + &resolve.device_policy, + &[], + None, + None, + None, + Some(spawn_override), + ); + if let Ok(child_pid) = result.as_ref() { + let _ = rocm_core::terminate_process(*child_pid); + } + let _ = fs::remove_dir_all(root); + + assert!( + bind_succeeded.get(), + "the port must already be bindable (released) by the time the child is spawned" + ); + result?; + Ok(()) + } +} diff --git a/apps/rocm/src/service_launch/start.rs b/apps/rocm/src/service_launch/start.rs new file mode 100644 index 0000000000..06cb93697e --- /dev/null +++ b/apps/rocm/src/service_launch/start.rs @@ -0,0 +1,487 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! The background launch path: [`start_managed_service`] and the plain, +//! line-by-line rendering of its [`ManagedLaunchReport`]. + +use super::endpoint_auth::drop_orphaned_endpoint_key_on_already_running; +use super::readiness::{ + died_during_wait, record_wait_outcome, service_pid_is_alive, wait_for_record_ready, + wait_on_existing_service, +}; +#[cfg(feature = "e2e-oom-fault-injection")] +use super::readiness::{e2e_simulate_oom_launch, simulate_oom_managed_launch}; +use super::spawn::{ManagedLaunchReport, ManagedSpawn, spawn_managed_engine_child}; +use crate::serve_port; +use crate::{record_cli_audit_event, status_for_readiness}; +use anyhow::Result; +use rocm_core::format_http_base_url; +use rocm_core::{AppPaths, EndpointReadiness}; +use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse}; +use std::fmt::Write as _; +#[cfg(windows)] +use std::thread; +use std::time::Duration; + +/// Launch a managed service in the background: run the locked allocation and +/// spawn transaction, release the launch lock once the record is published, +/// then wait for HTTP readiness. +/// +/// Drops the endpoint key `serve()` freshly staged for this attempt if the +/// launch fails before any `ManagedServiceRecord` exists to own it (port +/// reserved/in use, range exhausted, other bind error) — such a failure +/// returns before `abandon_failed_managed_launch` ever runs, so nothing else +/// would clear it. Scoped to the spawn call only, mirroring +/// [`run_attached_service`](super::run_attached_service)'s identical guard: a later +/// failure (the readiness wait, `record.write()`) leaves a record a live child actually owns, so it +/// must not also wipe that key. +#[allow(clippy::too_many_arguments)] +pub(crate) fn start_managed_service( + paths: &AppPaths, + engine: &str, + service_id: &str, + requested_model: &str, + resolve: &ResolveModelResponse, + host: &str, + port_request: serve_port::PortRequest, + device_policy: &DevicePolicy, + gpu_indices: &[u32], + runtime_id: Option<&str>, + env_id: Option<&str>, + engine_recipe: Option<&EngineRecipeHint>, + endpoint_api_key: Option<&str>, + launch_lock: rocm_core::FileLock, + require_api_key: bool, + gpu_refusal_waived_for_reuse: bool, + wait_until_ready: bool, + on_wait_begin: &mut dyn FnMut(&str, u16), + on_wait_tick: &mut dyn FnMut(Duration), +) -> Result { + #[cfg(feature = "e2e-oom-fault-injection")] + if e2e_simulate_oom_launch() { + return simulate_oom_managed_launch( + paths, + engine, + service_id, + requested_model, + resolve, + host, + port_request + .explicit() + .unwrap_or(rocm_core::DEFAULT_LOCAL_PORT), + device_policy, + gpu_indices, + runtime_id, + env_id, + ); + } + let (mut record, child_pid) = match spawn_managed_engine_child( + paths, + engine, + service_id, + requested_model, + resolve, + host, + port_request, + device_policy, + gpu_indices, + runtime_id, + env_id, + engine_recipe, + require_api_key, + gpu_refusal_waived_for_reuse, + ) + .inspect_err(|_| { + drop_orphaned_endpoint_key_on_already_running(paths, service_id, endpoint_api_key); + })? { + ManagedSpawn::AlreadyRunning(report) => { + // `--wait` applies to a service that is still starting too: the + // natural retry after `rocm serve m` printed `starting`. + drop(launch_lock); + return wait_on_existing_service( + paths, + report, + wait_until_ready, + on_wait_begin, + on_wait_tick, + ); + } + ManagedSpawn::Spawned { record, child_pid } => (*record, child_pid), + }; + // The transaction resolved the request to one concrete port; everything + // below — readiness, the endpoint URL, the audit line — uses that. + let port = record.port; + // The claiming service record is now persisted, so the selected GPU is + // visible to any concurrent auto-selection. Release the launch lock before + // the readiness wait below, which can block for many seconds — holding it + // that long would needlessly serialize unrelated serves. + drop(launch_lock); + + #[cfg(windows)] + thread::sleep(Duration::from_millis(200)); + + if wait_until_ready { + on_wait_begin(service_id, port); + } + let readiness = wait_for_record_ready( + paths, + &record, + wait_until_ready, + endpoint_api_key, + on_wait_tick, + ); + if let Some(error) = died_during_wait( + paths, + service_id, + wait_until_ready, + readiness, + service_pid_is_alive, + ) { + return Err(error); + } + let launch_status = status_for_readiness(readiness); + record_wait_outcome( + paths, + &mut record, + launch_status, + readiness == EndpointReadiness::Serving, + )?; + let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); + record_cli_audit_event( + paths, + "service", + "managed_service_launch", + "info", + format!( + "launched managed service engine={} model={} endpoint={} readiness={}", + engine, resolve.canonical_model_id, endpoint_url, launch_status + ), + Some(service_id), + ); + Ok(ManagedLaunchReport { + service_id: service_id.to_owned(), + endpoint_url, + status: launch_status.to_owned(), + already_running: false, + child_pid: Some(child_pid), + log_path: Some(record.log_path), + manifest_path: Some(record.manifest_path), + }) +} + +/// Reproduce the original plain, line-by-line managed-launch output. Used for +/// non-interactive output (piped, CI, the chat assistant's `serve --managed`), +/// where the animated summary is inappropriate. The interactive path renders the +/// summary table via [`crate::serve_summary`] instead. +pub(crate) fn print_managed_launch_plain( + report: &ManagedLaunchReport, + endpoint_api_key: Option<&str>, +) { + if report.already_running { + println!("managed service already running"); + println!(" service_id: {}", report.service_id); + println!(" endpoint: {}", report.endpoint_url); + println!(" status: {}", report.status); + println!(" note: existing service detected; no second process spawned"); + return; + } + println!("managed service launched"); + println!(" service_id: {}", report.service_id); + if let Some(child_pid) = report.child_pid { + println!(" process_pid: {child_pid}"); + } + println!(" endpoint: {}", report.endpoint_url); + if let Some(key) = endpoint_api_key { + // Intentional one-time display of a freshly generated API key to the + // terminal so the user can copy it — the designed delivery channel + // documented on `render_endpoint_client_config`, not a log. The tag below + // is currently inert (Rust's CodeQL pack has no AlertSuppression.ql yet — + // github/codeql#21637) but will start working once that lands, since the + // tag must be the single line immediately before the flagged code. + print!( + // codeql[rust/cleartext-logging] + "{}", + render_endpoint_client_config(&report.endpoint_url, key) + ); + } + if let Some(log_path) = report.log_path.as_deref() { + println!(" log_path: {}", log_path.display()); + } + if let Some(manifest_path) = report.manifest_path.as_deref() { + println!(" manifest_path: {}", manifest_path.display()); + } + println!(" readiness: {}", report.status); +} + +/// The client configuration to show when a `--wait` begins, so a key generated for +/// a public bind reaches the user even if the wait is interrupted or fails. +/// `None` unless the wait is on the launch that key belongs to: an attach to an +/// already-starting server (`waiting_on` differs from `service_id`) answers to +/// its own key, which is not shown here. +pub(crate) fn wait_begin_client_config( + waiting_on: &str, + service_id: &str, + host: &str, + port: u16, + api_key: Option<&str>, +) -> Option { + let key = api_key.filter(|_| waiting_on == service_id)?; + let endpoint_url = format!("{}/v1", format_http_base_url(host, port)); + Some(render_endpoint_client_config(&endpoint_url, key)) +} + +/// Render the one-time secure client configuration for a public, authenticated +/// endpoint. This is the *intended* channel for delivering the key to the user +/// (unlike logs/status, which must never contain it) — it prints the key once at +/// launch alongside a ready-to-use example. Callers only invoke this for a +/// non-loopback bind that generated/received a key. +pub(super) fn render_endpoint_client_config(endpoint_url: &str, api_key: &str) -> String { + let mut out = String::new(); + let _ = writeln!(out, " api key: {api_key}"); + let _ = writeln!( + out, + " note: this key is shown only now - clients must send `Authorization: Bearer `" + ); + let _ = writeln!( + out, + " example: curl -H \"Authorization: Bearer {api_key}\" {endpoint_url}/models" + ); + out +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::endpoint_keys; + #[cfg(not(windows))] + use crate::service_launch::run_attached_service; + #[cfg(not(windows))] + use crate::service_launch::spawn::TEST_SPAWN_HOOK; + use crate::tests::test_paths; + use rocm_core::ManagedServiceRecord; + use std::fs; + #[cfg(not(windows))] + use std::io; + + #[test] + fn wait_begin_shows_the_key_only_for_the_launch_it_belongs_to() { + let own = wait_begin_client_config("svc-new", "svc-new", "0.0.0.0", 8000, Some("k-secret")) + .expect("own launch shows its key"); + // No `{own}` in the failure messages: the text carries the dummy key, + // and formatting it into an assert is what CodeQL flags as logging it. + assert!(own.contains("api key: k-secret"), "key line missing"); + assert!( + own.contains("http://0.0.0.0:8000/v1/models"), + "models url missing" + ); + // Attached to an already-starting server: its key is not this one. + assert!( + wait_begin_client_config("svc-old", "svc-new", "0.0.0.0", 8000, Some("k-secret")) + .is_none() + ); + // Loopback binds have no key to show. + assert!(wait_begin_client_config("svc-new", "svc-new", "127.0.0.1", 8000, None).is_none()); + } + + #[test] + fn endpoint_client_config_shows_key_once_with_bearer_guidance() { + let rendered = render_endpoint_client_config("http://0.0.0.0:11435/v1", "secret-123"); + // Dummy literal exercising the intentional one-time key display documented + // on `render_endpoint_client_config`; not a real credential or a log write. + // codeql[rust/cleartext-logging] + assert!(rendered.contains("secret-123"), "{rendered}"); + // codeql[rust/cleartext-logging]: see rationale above. + assert!(rendered.contains("Authorization: Bearer"), "{rendered}"); + // codeql[rust/cleartext-logging]: see rationale above. + assert!(rendered.contains("shown only now"), "{rendered}"); + } + + #[test] + fn failed_port_reservation_leaves_no_orphaned_endpoint_key() -> Result<()> { + // serve() stores the new attempt's endpoint key before calling + // start_managed_service. If spawn_managed_engine_child fails inside + // resolve_managed_port_in_transaction (explicit port already reserved + // by another live service), it returns before any + // ManagedServiceRecord exists for this attempt, so + // abandon_failed_managed_launch never runs. Driving the real function + // end to end (not a simulated call to + // drop_orphaned_endpoint_key_on_already_running) proves + // `start_managed_service`'s own `.inspect_err` actually wires the cleanup, mirroring how + // attached_launch_failing_before_spawn_leaves_no_orphaned_endpoint_key + // drives run_attached_service for the foreground path. + let (root, paths) = test_paths("orphaned-key-on-reserved-port"); + paths.ensure()?; + let mut other = ManagedServiceRecord::new( + &paths, + "lemonade-llama-1000", + "lemonade", + "llama", + "llama-canonical", + "127.0.0.1", + 11513, + "managed", + std::process::id(), + None, + None, + None, + ); + other.status = "ready".to_owned(); + other.engine_pid = Some(std::process::id()); + other.write()?; + + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + + let service_id = "lemonade-qwen-2000"; + endpoint_keys::store_endpoint_api_key(&paths, service_id, "test-key")?; + let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + + let result = start_managed_service( + &paths, + "lemonade", + service_id, + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Explicit(11513), + &resolve.device_policy, + &[], + None, + None, + None, + Some("test-key"), + launch_lock, + false, + false, + false, + &mut |_, _| {}, + &mut |_elapsed| {}, + ); + let Err(error) = result else { + let _ = fs::remove_dir_all(root); + panic!("explicit port already reserved by a live service must be rejected") + }; + assert!( + error + .to_string() + .contains("already reserved by managed service"), + "unexpected error: {error}" + ); + + assert!( + endpoint_keys::endpoint_api_key(&paths, service_id).is_none(), + "cleanup must drop the key this failed attempt staged before launching" + ); + + let _ = fs::remove_dir_all(root); + Ok(()) + } + + /// The launch lock must still be held when the child is spawned: it is what + /// serializes allocation against publication. Observed through the spawn + /// hook, since releasing it early changes nothing else a test can see. + #[cfg(not(windows))] + #[test] + fn start_and_attached_launches_hold_the_launch_lock_until_the_spawn() -> Result<()> { + let resolve = ResolveModelResponse { + canonical_model_id: "qwen-canonical".to_owned(), + served_model_id: None, + task: "chat".to_owned(), + source: "hf".to_owned(), + revision: "main".to_owned(), + loader: "llama.cpp".to_owned(), + trust_remote_code: false, + chat_template_mode: "auto".to_owned(), + dtype: "auto".to_owned(), + device_policy: DevicePolicy::GpuPreferred, + estimated_memory: "unknown".to_owned(), + launch_defaults: serde_json::json!({}), + engine_recipe: None, + warnings: Vec::new(), + }; + for attached in [false, true] { + let (root, paths) = test_paths("launch-lock-held-until-spawn"); + paths.ensure()?; + let lock_path = paths.managed_launch_lock_path(); + let lock_was_held = std::rc::Rc::new(std::cell::Cell::new(false)); + let observed = lock_was_held.clone(); + TEST_SPAWN_HOOK.with(|hook| { + *hook.borrow_mut() = Some(std::rc::Rc::new(move |_command| { + observed.set( + rocm_core::FileLock::acquire_bounded(&lock_path, Duration::ZERO).is_err(), + ); + Err(io::Error::other("spawn intercepted by test")) + })); + }); + let launch_lock = rocm_core::FileLock::acquire(paths.managed_launch_lock_path())?; + let result = if attached { + run_attached_service( + &paths, + "lemonade", + "lemonade-lock-attached", + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Auto, + &[], + None, + None, + None, + launch_lock, + false, + false, + |_, _| unreachable!("the spawn fails before a Ctrl-C could stop it"), + ) + } else { + start_managed_service( + &paths, + "lemonade", + "lemonade-lock-start", + "qwen", + &resolve, + "127.0.0.1", + serve_port::PortRequest::Auto, + &resolve.device_policy, + &[], + None, + None, + None, + None, + launch_lock, + false, + false, + false, + &mut |_, _| {}, + &mut |_elapsed| {}, + ) + .map(|_| ()) + }; + TEST_SPAWN_HOOK.with(|hook| *hook.borrow_mut() = None); + let _ = fs::remove_dir_all(root); + assert!( + format!("{:#}", result.expect_err("the intercepted spawn fails")) + .contains("spawn intercepted by test") + ); + assert!( + lock_was_held.get(), + "the launch lock must be held at spawn (attached: {attached})" + ); + } + Ok(()) + } +} diff --git a/apps/rocm/src/service_launch/test_support.rs b/apps/rocm/src/service_launch/test_support.rs new file mode 100644 index 0000000000..70a5504e09 --- /dev/null +++ b/apps/rocm/src/service_launch/test_support.rs @@ -0,0 +1,103 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Fixtures shared by the tests of more than one launch submodule. + +use anyhow::Result; +use rocm_core::{AppPaths, ManagedServiceRecord}; +#[cfg(not(windows))] +use std::thread; +#[cfg(not(windows))] +use std::time::Duration; + +/// A record with every reuse-relevant field set to the "reusable" value, so +/// each test that uses it breaks exactly one term and nothing else. +pub(super) fn reusable_record(paths: &AppPaths) -> ManagedServiceRecord { + let mut record = ManagedServiceRecord::new( + paths, + "vllm-qwen-1000", + "vllm", + "qwen", + "qwen-canonical".to_owned(), + "127.0.0.1", + 8000, + "managed", + 0, + None, + None, + Some("gpu_required".to_owned()), + ); + record.engine_recipe_json = None; + record.requires_api_key = true; + record +} + +pub(super) fn write_live_service_record( + paths: &AppPaths, + service_id: &str, + canonical_model_id: &str, + port: u16, + status: &str, +) -> Result { + let mut record = ManagedServiceRecord::new( + paths, + service_id, + "lemonade", + canonical_model_id, + canonical_model_id.to_owned(), + rocm_core::DEFAULT_LOCAL_HOST, + port, + "managed", + std::process::id(), + None, + None, + None, + ); + record.status = status.to_owned(); + record.engine_pid = Some(std::process::id()); + record.write()?; + Ok(record) +} + +/// The helper the cleanup tests re-run this test binary as, to get a live child +/// process that outlives the spawn they are checking. It is `#[ignore]`d so a +/// normal run skips it, and it only sleeps when the tests that spawn it set +/// `ROCM_TEST_SLEEPER_CHILD`. They select it by [`sleeper_test_name`]. +#[test] +#[cfg(not(windows))] +#[ignore = "spawned as a child process by abandoned_launch_terminates_the_spawned_child"] +fn managed_launch_cleanup_sleeper_child() { + if std::env::var_os("ROCM_TEST_SLEEPER_CHILD").is_some() { + thread::sleep(Duration::from_mins(2)); + } +} + +/// The name libtest lists [`managed_launch_cleanup_sleeper_child`] under, which +/// is what `--exact` has to be given to run it alone. +/// +/// It is read off the function item itself, so renaming the function or moving +/// this module changes it with them and the three tests that spawn the helper +/// cannot be left naming a test that no longer exists. libtest names a test by +/// its path inside the crate, while the item path starts with the crate name, +/// which is dropped. The assertions check that against the module path as well. +#[cfg(not(windows))] +pub(super) fn sleeper_test_name() -> String { + let item_path = std::any::type_name_of_val(&managed_launch_cleanup_sleeper_child); + let name = item_path + .split_once("::") + .map_or(item_path, |(_krate, rest)| rest); + let module = module_path!() + .split_once("::") + .map_or("", |(_krate, rest)| rest); + assert_eq!( + name, + format!("{module}::managed_launch_cleanup_sleeper_child"), + "the sleeper helper is no longer where its name is derived from" + ); + assert!( + name.starts_with("service_launch::"), + "libtest names tests by their path inside the crate: {name}" + ); + name.to_owned() +} diff --git a/apps/rocm/src/service_records.rs b/apps/rocm/src/service_records.rs index 7be31d9579..5fe013a611 100644 --- a/apps/rocm/src/service_records.rs +++ b/apps/rocm/src/service_records.rs @@ -350,7 +350,7 @@ fn describe_hours(hours: u64) -> String { /// /// "No `.json` beside it" is also exactly what a *launch in progress* looks /// like: `serve` writes the 0600 `.endpoint-key` before -/// `spawn_managed_engine_child` (in `service_launch.rs`) writes the first +/// `spawn_managed_engine_child` (in `service_launch/spawn.rs`) writes the first /// `.json`. Nothing about /// the file distinguishes the two cases — under `--any-age` there is no age gate /// left to ask — so this function does not try. It is only ever reached with the @@ -633,9 +633,9 @@ fn apply_service_prune_plan( /// leftover, and that is also what a launch looks like mid-flight: `serve` /// writes the 0600 `.endpoint-key` at the top of the managed path, and the /// first `.json` only lands at `record.write()` inside -/// `spawn_managed_engine_child` (in `service_launch.rs`). Between those two writes a live server's -/// secret is indistinguishable from a leftover, and `--any-age` removes the age -/// gate that used to hide the window, so a concurrent +/// `spawn_managed_engine_child` (in `service_launch/spawn.rs`). Between those two +/// writes a live server's secret is indistinguishable from a leftover, and +/// `--any-age` removes the age gate that used to hide the window, so a concurrent /// `rocm services prune --any-age --yes` deleted it. /// /// `serve` already holds `managed_launch_lock_path` across the whole of that diff --git a/apps/rocm/src/storage.rs b/apps/rocm/src/storage.rs index b930874699..d1c08eaca6 100644 --- a/apps/rocm/src/storage.rs +++ b/apps/rocm/src/storage.rs @@ -643,7 +643,7 @@ pub(crate) fn build_report(paths: &AppPaths, config: &RocmCliConfig) -> Result.log` beside the manifest. `ManagedServiceRecord::new` // only *computes* that path (via `AppPaths::service_log_path`) - it // writes nothing. The file is created by the launch site, - // `spawn_managed_engine_child` (in `service_launch.rs`) for + // `spawn_managed_engine_child` (in `service_launch/spawn.rs`) for // `rocm serve --managed` and `supervise_service` in `rocmd` on the // supervised/recovery path. What *fills* it differs by platform, so the // redirect cannot be stated unqualified: `supervise_service` redirects diff --git a/apps/rocm/src/test_support.rs b/apps/rocm/src/test_support.rs index caaa736810..88de384287 100644 --- a/apps/rocm/src/test_support.rs +++ b/apps/rocm/src/test_support.rs @@ -389,7 +389,7 @@ pub(crate) fn owned_argv(args: &[&str]) -> Vec { /// Build an `AutomationRuntimeState` for the background-helper tests (the /// no-double-spawn guard in `background_helper.rs` and the autostart tests in -/// `service_launch.rs`). +/// `service_launch/helper_autostart.rs`). pub(crate) fn runtime_state(running: bool, daemon_pid: u32) -> AutomationRuntimeState { AutomationRuntimeState { running, diff --git a/apps/rocmd/src/supervise.rs b/apps/rocmd/src/supervise.rs index 023a00d313..0076e641a6 100644 --- a/apps/rocmd/src/supervise.rs +++ b/apps/rocmd/src/supervise.rs @@ -2210,7 +2210,7 @@ mod tests { } /// Mirrors `launch_releases_the_leased_port_before_spawning_the_child` in - /// `apps/rocm/src/service_launch.rs`: the probe lease on the recorded port + /// `apps/rocm/src/service_launch/spawn.rs`: the probe lease on the recorded port /// must already be released by the time the recovery supervisor is /// spawned, or the supervisor's own bind of that port would fail. #[test] diff --git a/apps/rocmd/src/watchers/server_recover.rs b/apps/rocmd/src/watchers/server_recover.rs index f43402a4e1..aac0a9e9d5 100644 --- a/apps/rocmd/src/watchers/server_recover.rs +++ b/apps/rocmd/src/watchers/server_recover.rs @@ -798,8 +798,8 @@ mod tests { /// The record `rocm serve` keeps when it abandons a launch whose child /// already ran (`abandon_failed_managed_launch` in - /// `apps/rocm/src/service_launch.rs`) must not be respawned: the CLI has just - /// reported that launch as failed. The producing side is + /// `apps/rocm/src/service_launch/launch_transaction.rs`) must not be respawned: + /// the CLI has just reported that launch as failed. The producing side is /// `abandoned_launch_terminates_the_spawned_child` in that file; both go /// through `ManagedServiceRecord::mark_launch_abandoned`. #[test] diff --git a/docs/architecture.md b/docs/architecture.md index 3b8e4ae96d..7d2e972e65 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -17,7 +17,7 @@ A second check runs the other way: `cargo xtask check-architecture-doc` ([`archi New subcommands and subsystems default to their own file from day one — they should not grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. Two extraction patterns already exist in the codebase; use whichever fits: - **Full domain extraction** — a subsystem's domain implementation moves into its own file that owns its own types (structs/enums), not just relocated functions; that ownership is usually what distinguishes this pattern from mechanical relocation below, though the Phase 6a variant described next shows ownership alone doesn't fully tell them apart — every `apps/rocm` module is a private `mod x;` regardless of pattern, so module privacy alone doesn't tell the two apart either. Full-domain-extraction modules are typically reached via qualified paths (e.g. `comfyui::render_status(...)`); mechanically relocated modules with no owned types are reached the same way (e.g. [`endpoint_keys.rs`](../apps/rocm/src/endpoint_keys.rs), [`logging.rs`](../apps/rocm/src/logging.rs)). The exception — reached unqualified through `use crate::x::{...}` re-imports, matching their dispatch call sites' pre-extraction form — is limited to the clusters that moved as a dispatch-adjacent group; the block of `use crate::x::{...}` re-imports near the top of [`main.rs`](../apps/rocm/src/main.rs) is the current list of the modules it reaches that way (a few of them re-imported for reasons other than dispatch, as the comment above the block explains). [`service_launch.rs`](../apps/rocm/src/service_launch.rs) (as of Phase 6b, ROCMAI-91) uses the same unqualified re-import form for its cross-module callers, but it is not a dispatch-adjacent cluster in the same sense — one of its callers, `automations.rs`, is not a dispatch call site, and its pre-extraction form was a qualified `crate::` call rather than an unqualified import, so the re-import is new rather than kept. This call-site form, not type ownership or dispatch-fn placement, is what actually separates the Phase 6a variant below from full domain extraction. Where a subsystem has a dedicated clap subcommand, its command *enum* usually lives in [`cli.rs`](../apps/rocm/src/cli.rs) as part of the `Command`/`Cli` definition (e.g. `ComfyuiCommand`, `RuntimesCommand`), the way [`apps/rocmd`](../apps/rocmd/src/cli.rs) keeps its command tree in a `cli.rs` of its own — but the dispatch function, the match over that tree, does not stay in `main.rs` with it: `comfyui::run` lives in `comfyui.rs`, reached by the qualified path full domain extraction uses, while `engines()` and `runtimes()` moved out to `engines_cmd.rs`/`runtimes_cmd.rs` respectively, following the mechanical-relocation re-import form below. Only the top-level `dispatch()`, the match that sends each `Command` variant to its handler, stays in `main.rs`. Not every domain-extracted module has a dedicated command enum at all (`providers.rs` has none; it's invoked from the existing chat/config command flows). In library crates (`crates/rocm-core`) the module is `pub mod x;` plus a `pub use x::{...};` re-export, since it's part of the crate's public API. This is the default for new subsystems. Examples: [`apps/rocm/src/therock.rs`](../apps/rocm/src/therock.rs) (TheRock SDK install orchestration, with its update planning and other clusters in `therock/` child modules and no command enum of its own), [`comfyui.rs`](../apps/rocm/src/comfyui.rs) (`ComfyuiCommand`), [`providers.rs`](../apps/rocm/src/providers.rs) (no dedicated command enum); `crates/rocm-core`'s [`diagnose.rs`](../crates/rocm-core/src/diagnose.rs)/[`examine.rs`](../crates/rocm-core/src/examine.rs). -- **Mechanical relocation** — a `pub(crate) fn` moves out verbatim, with shared types/config staying at the crate root and reached via `crate::`. Used for dispatch-adjacent clusters where a minimal, easy-to-review diff matters more than full extraction. Example: [`apps/rocm/src/update_cmd.rs`](../apps/rocm/src/update_cmd.rs). Variant: when the cluster is large enough to own types (part of `main.rs`'s Phase 6a split, ROCMAI-82, `main.rs`'s Phase 6b split, ROCMAI-91, `main.rs`'s Phase 6c split, ROCMAI-89, and Phase 6e's follow-up, ROCMAI-726), those types move with it instead of staying in `main.rs` — [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s types are used nowhere else, but [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s `DriverInstallResult`/`DriverInstallError` are `pub(crate)` and read from [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs)'s `install()` (`result.output`/`result.executed`, `error.source`/`error.executed`); [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs)'s `FreeformExecution` is `pub(crate)` and read from `main.rs`'s `execute_freeform_next_action()` (`execution.action`) — it was already `pub(crate)` before the split, unlike the two functions that return/consume it, which widened from module-private; [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `ServeEngineSelection`/`ServeArgs` both stay `pub(crate)` and are used from `main.rs` (`ServeArgs`) and [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`assess_model_for_host` calls `select_serve_engine` and reads the resulting `ServeEngineSelection`'s `.engine`/`.source` fields) — and `ServeGenerationDefaults` moved out with the engine-recipe overrides to [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs), where it and its fields are `pub(crate)` because `serve()` builds it; [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are private, and its `ManagedLaunchReport` is `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) for the interactive summary; [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimeActivationResult` and `ActiveRuntimeMarker` are `pub(crate)` and read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary` and from [`storage.rs`](../apps/rocm/src/storage.rs) respectively; its sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`. The pattern is still "mechanical relocation" rather than full domain extraction because call sites keep their pre-extraction unqualified-re-import form — the exception named above — rather than switching to the qualified paths full-domain-extraction modules use. Where a subcommand's dispatch fn lives is not what makes it so: a full-domain-extraction module can host its own dispatch fn too (see `comfyui::run` above). +- **Mechanical relocation** — a `pub(crate) fn` moves out verbatim, with shared types/config staying at the crate root and reached via `crate::`. Used for dispatch-adjacent clusters where a minimal, easy-to-review diff matters more than full extraction. Example: [`apps/rocm/src/update_cmd.rs`](../apps/rocm/src/update_cmd.rs). Variant: when the cluster is large enough to own types (part of `main.rs`'s Phase 6a split, ROCMAI-82, `main.rs`'s Phase 6b split, ROCMAI-91, `main.rs`'s Phase 6c split, ROCMAI-89, and Phase 6e's follow-up, ROCMAI-726), those types move with it instead of staying in `main.rs` — [`engines_cmd.rs`](../apps/rocm/src/engines_cmd.rs)'s types are used nowhere else, but [`driver_install.rs`](../apps/rocm/src/driver_install.rs)'s `DriverInstallResult`/`DriverInstallError` are `pub(crate)` and read from [`install_cmd.rs`](../apps/rocm/src/install_cmd.rs)'s `install()` (`result.output`/`result.executed`, `error.source`/`error.executed`); [`freeform_execution.rs`](../apps/rocm/src/freeform_execution.rs)'s `FreeformExecution` is `pub(crate)` and read from `main.rs`'s `execute_freeform_next_action()` (`execution.action`) — it was already `pub(crate)` before the split, unlike the two functions that return/consume it, which widened from module-private; [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs)'s `ServeEngineSelection`/`ServeArgs` both stay `pub(crate)` and are used from `main.rs` (`ServeArgs`) and [`model_fit.rs`](../apps/rocm/src/model_fit.rs) (`assess_model_for_host` calls `select_serve_engine` and reads the resulting `ServeEngineSelection`'s `.engine`/`.source` fields) — and `ServeGenerationDefaults` moved out with the engine-recipe overrides to [`serve_recipe.rs`](../apps/rocm/src/serve_recipe.rs), where it and its fields are `pub(crate)` because `serve()` builds it; the types in the child modules of [`service_launch.rs`](../apps/rocm/src/service_launch.rs) are private to the child that owns them (`AttachOutcome`, `RawModeGuard`, `AutostartClaim`) or `pub(super)` where a sibling child reads them (`ManagedSpawn`, `ManagedPortDecision`), except `ManagedLaunchReport`, which is `pub(crate)` and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) for the interactive summary, and `ManagedReuseRefusal`, which is `pub(crate)` because `managed_service_reuse_refusal`, a function `serve_cmd.rs` calls, returns it; [`runtimes_cmd.rs`](../apps/rocm/src/runtimes_cmd.rs)'s `RuntimeActivationResult` and `ActiveRuntimeMarker` are `pub(crate)` and read from [`update_cmd.rs`](../apps/rocm/src/update_cmd.rs)'s `apply_runtime_update`/`append_update_activate_summary` and from [`storage.rs`](../apps/rocm/src/storage.rs) respectively; its sibling [`runtime_install_finalize.rs`](../apps/rocm/src/runtime_install_finalize.rs)'s `SdkInstallConsents` is `pub(crate)` and read from `install_cmd.rs`'s `install()`. The pattern is still "mechanical relocation" rather than full domain extraction because call sites keep their pre-extraction unqualified-re-import form — the exception named above — rather than switching to the qualified paths full-domain-extraction modules use. Where a subcommand's dispatch fn lives is not what makes it so: a full-domain-extraction module can host its own dispatch fn too (see `comfyui::run` above). Moving a `proptest!` block to another file leaves its saved failure cases behind unless they are carried over: proptest keys them by the test's source file (`/src/.rs` reads `/proptest-regressions/.txt`, where `src/` is the directory holding `main.rs` or `lib.rs`) and does not warn when the file stops matching. Carry them in the same commit, and how depends on how much moved. When every property test of the file moves and the new file has no regression file yet, `git mv` the old one to the name the new file needs. When only some move, the tests left behind still read the old file, so keep it, and copy the saved cases to the new file's regression file (appending to it when it already exists, so its own cases stay). An `xtask` test fails the build on a regression file no test reads; it cannot notice a regression file that was moved away from tests that still need it, so check a split's diff for that. See [Property tests](testing.md#property-tests). @@ -34,7 +34,7 @@ Each per-crate module inventory lists one file per line. Flat inventories (every [`main.rs`](../apps/rocm/src/main.rs) is the crate root. Besides smaller helpers, it holds `main()`, `run()` and `dispatch()`, the match over the clap `Cli`/`Command` tree that [`cli.rs`](../apps/rocm/src/cli.rs) defines; the builders that turn that tree into a `clap::Command` (`cli_command()`, `completion_command()`); the command handlers still defined here (`version()` and `launch_default()`, the no-subcommand arm); freeform-invocation routing (`parse_freeform_invocation`, `should_treat_as_freeform`, `run_freeform`, `execute_freeform_next_action`, `treat_as_natural_language`); a few crate-root items several modules share (`status_for_readiness`, `device_policy_name`, `engine_inventory`, `ROLLBACK_RECOVERY_HINT`); the `use crate::x::{...}` re-import block described above, which also re-imports the `cli.rs` types so other modules keep reaching them as `crate::Command`, `crate::Provider` and so on; and the crate's shared `mod tests` block. - [`automations.rs`](../apps/rocm/src/automations.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-210): the `rocm automations` dispatch handler (`automations()`) plus the `rocm automations` status report it prints (`render_automations_text`), the `ProposalKind` classification that report renders through (`proposal_kind`, `fallback_tool_for_automation_action`, `proposal_plain_summary`, `proposal_plain_reason`), the small plain-language label helpers (`proposal_status_label`, `proposal_bool_argument`, `automation_event_plain_summary`, `audit_event_plain_summary`, `watcher_mode_plain_label`, `watcher_last_check`, `watcher_plain_name`, `watcher_plain_trigger`, `watcher_plain_action`, `watcher_policy_note`), and the unrelated `rocm daemon --status` summary (`render_daemon_text`). `render_automations_text` and `render_daemon_text` are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `render_daemon_text` directly (re-imported alongside `automations` via `use crate::automations::{automations, render_daemon_text};`; `render_automations_text` has no `dispatch()` call site of its own, reached only through `automations()` above, now local to this file), and [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner calls both. Every other item in this list, including `watcher_policy_note` (called from `automations()`'s `Enable` arm and from `render_automations_text`, both in this same file), has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do. -- [`background_helper.rs`](../apps/rocm/src/background_helper.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), owns one `pub(crate)` type: what the CLI can tell about the background automation helper (`rocm daemon`). It holds `background_helper_already_running` (the file-based liveness check behind the no-double-spawn decision), the `HelperNote` type and `background_helper_note` with its `helper_note_for` wording, which [`services_text.rs`](../apps/rocm/src/services_text.rs) prints after a stop and [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) turns into an uninstall warning. Spawning the helper (`ensure_background_helper_running` and its autostart lock) stays in [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which reads `background_helper_already_running` from here. The module is a leaf: it imports nothing from the service modules, so the note can be read from [`services_text.rs`](../apps/rocm/src/services_text.rs) without that module importing the one that spawns services. The cluster's regression tests live in this module's own `mod tests`, which stays private: the `runtime_state` and `set_helper_facts` fixtures they share with the helper-autostart and stop-output tests in [`service_launch.rs`](../apps/rocm/src/service_launch.rs) live in the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs). +- [`background_helper.rs`](../apps/rocm/src/background_helper.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), owns one `pub(crate)` type: what the CLI can tell about the background automation helper (`rocm daemon`). It holds `background_helper_already_running` (the file-based liveness check behind the no-double-spawn decision), the `HelperNote` type and `background_helper_note` with its `helper_note_for` wording, which [`services_text.rs`](../apps/rocm/src/services_text.rs) prints after a stop and [`uninstall_plan.rs`](../apps/rocm/src/uninstall_plan.rs) turns into an uninstall warning. Spawning the helper (`ensure_background_helper_running` and its autostart lock) stays in [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs), a child of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which reads `background_helper_already_running` from here. The module is a leaf: it imports nothing from the service modules, so the note can be read from [`services_text.rs`](../apps/rocm/src/services_text.rs) without that module importing the one that spawns services. The cluster's regression tests live in this module's own `mod tests`, which stays private: the `runtime_state` and `set_helper_facts` fixtures they share with the helper-autostart and stop-output tests in [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) live in the crate-level [`test_support.rs`](../apps/rocm/src/test_support.rs). - [`bootstrap.rs`](../apps/rocm/src/bootstrap.rs) — a further-extracted variant of full domain extraction: it owns its clap command enum (`BootstrapCommand`) and dispatch function too, rather than leaving the enum in [`cli.rs`](../apps/rocm/src/cli.rs) and the dispatch in [`main.rs`](../apps/rocm/src/main.rs). - [`bridge_snapshot.rs`](../apps/rocm/src/bridge_snapshot.rs) — mechanical relocation, no owned types: the host snapshot behind the internal bridge surface (`build_codex_bridge_snapshot`, `build_codex_bridge_gpu_snapshot`, `builtin_codex_bridge_engine_inventory`), whose engine list is derived from `main.rs`'s `engine_inventory`, the single table of built-in engine ids that `rocm examine`, `rocm update` and the `--engine` help test read as well. All three are `pub(crate)`: [`main.rs`](../apps/rocm/src/main.rs)'s `dispatch()` calls `build_codex_bridge_snapshot` through a re-import, [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s internal MCP tools reach them as `crate::` through the same re-import. It has no tests of its own. - [`chat.rs`](../apps/rocm/src/chat.rs) — the chat assistant's entry points and the module that ties its pieces together, a file beside a `chat/` directory of child modules (as [`providers.rs`](../apps/rocm/src/providers.rs) is beside `providers/`). It keeps `read_piped_prompt` and `piped_prompt_from_input` (the `echo "…" | rocm chat` path), `render_chat_text` (the status screen), `render_chat_prompt_text` (the one-shot reply text, which calls into `chat/turn.rs`), `plain_provider_auth_status` and the clap value parsers for the sampling flags (`parse_non_negative_temperature`, `parse_unit_interval`, `parse_positive_u32`), plus the re-exports below. The work behind them lives in four leaf children: [`chat/turn.rs`](../apps/rocm/src/chat/turn.rs) (the turn loop, the system prompt and the transcript), [`chat/intent.rs`](../apps/rocm/src/chat/intent.rs) (the prompt-intent heuristics), [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs) (the MCP tool bridge and in-process command execution) and [`chat/summaries.rs`](../apps/rocm/src/chat/summaries.rs) (the deterministic tool-output summaries); `turn` is the top of that call graph and calls the other three, none of which calls back. The tool-call validation and approval policy (`validate_chat_tool_call`, the `validate_chat_*_tool_call` family, `ChatRocmCommandAction`, `ChatToolApprovalRequest`) lives in [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs), which the children call into. Callers in other modules, production and test-only, reach the moved items through the `chat::` module boundary, so `chat.rs` re-exports each one another module reads: `run_internal_mcp_call` (`main.rs`, `dash_seam.rs`), `rocm_chat_tool_system_prompt_for_host` (`dash.rs`), `chat_tool_approval_request`, `chat_tool_call_is_read_only` and `validate_chat_tool_call` (`dash_seam.rs`), `any_substring` and `ensure_flag` (from `prompt_text.rs` and `rocm_argv.rs`, for `freeform_answer.rs` and `freeform_execution.rs`), and, only under `#[cfg(test)]`, `deterministic_comfyui_tool_summary` (`comfyui.rs`'s tests, unix only), `ChatRocmCommandAction`, `rocm_chat_tool_requested_args`, `chat_install_driver_action` and `chat_rocm_command_action_from_args` (`driver_plan.rs`'s and `system_packages.rs`'s tests). Each child's tests live in its own `mod tests`, with their original names; the two piped-prompt tests and the command-preview property tests (which drive the tool-call guards, the planner's guards and the rendered command lines across the policy, the bridge and the turn loop) stay in `chat.rs`. `serve_negative_sampling_space_form_reaches_range_validator`, a shared clap value-parser wiring test living alongside its sibling command-parsing tests, has moved with them to the test module in [`cli.rs`](../apps/rocm/src/cli.rs); the four structural checks that read [`chat_cmd.rs`](../apps/rocm/src/chat_cmd.rs)'s source for the `Command::Chat` handler's wiring (`command_chat_interactive_routes_to_dash_run_chat_not_tui`, `command_chat_interactive_notes_dropped_provider_flag`, `command_chat_honors_chat_mock_and_keeps_prompt_passthrough`, `command_chat_reads_prompt_from_piped_stdin`) live in that module's own `mod tests`. @@ -93,7 +93,14 @@ Each per-crate module inventory lists one file per line. Flat inventories (every - [`service_control.rs`](../apps/rocm/src/service_control.rs) — mechanical relocation, owns private types: stopping and restarting a managed local server, the process control behind `rocm services stop|restart`, the chat/LLM sandbox tool-calling path and `runtime_services.rs`'s moves onto a new runtime. It holds the sandbox-tool runner (`run_internal_sandbox_tool`), the stop path (`terminate_recorded_service_pids`, the `MANAGED_STOP_GRACE` constant, the `RecordedPidOutcome` and `RecordedStopTermination` types, `scripted_stop_outcome`, the end-to-end hook behind the `e2e-test-hooks` feature, `stop_internal_managed_service`, `stop_internal_managed_service_with`, `stop_internal_managed_service_confirmed_with`, `unload_lemonade_service_model`) and the restart path (`apply_restart_runtime_pin`, `load_service_for_restart`, `restart_child_argv`, `restart_internal_managed_service`, `restart_internal_managed_service_with`, `update_restarting_record`, `record_restarted_child`, `record_restart_readiness`). Only `run_internal_sandbox_tool` and `restart_internal_managed_service` are `pub(crate)`: the first is read from `dispatch()` in [`main.rs`](../apps/rocm/src/main.rs), from [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `run_approved_service_action` and from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), which passes it to [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s `run_attached_service` as the stop for a Ctrl-C, the second from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs); each importer reaches them through `crate::service_control` rather than a re-export. Every other item has no caller outside this module and stays private — see [`engine_runtime_select.rs`](../apps/rocm/src/engine_runtime_select.rs)'s and [`gpu_policy.rs`](../apps/rocm/src/gpu_policy.rs)'s entries above for how those two modules' own doc comments describe this module as the caller of their helpers. The module reads managed-service records through [`services_text.rs`](../apps/rocm/src/services_text.rs) (`load_managed_service`, `read_managed_service_manifest`) and shapes the `stop_server` answer with its `internal_stop_server_value`; `services_text.rs` does not call back into it. This module takes `ensure_public_service_has_endpoint_key` and the spawn helpers from [`service_launch.rs`](../apps/rocm/src/service_launch.rs), which does not import it back. The cluster's regression tests live in this module's own `mod tests`, reaching shared fixtures (`test_paths`, `ScopedTestEnv`, `plant_service_record_on_runtime`, `OLD_RUNTIME_KEY`, `NEW_RUNTIME_KEY`) through `crate::tests::` the same way other mechanically relocated modules do, and `runtime_services::` through its own `use crate::runtime_services;` for the handful of tests that drive a restart through `runtime_services`'s public entry points rather than calling it directly. - [`serve_port.rs`](../apps/rocm/src/serve_port.rs) — full domain extraction: the central `rocm serve` port policy (automatic vs explicit request, reservation by live managed services, OS preflight lease), used by [`service_launch.rs`](../apps/rocm/src/service_launch.rs); the CLI restart path shares the `rocm-core` lease primitives (`lease_loopback_port_or_bind_error`, `is_canonical_loopback_host`) rather than this module. - [`service_hints.rs`](../apps/rocm/src/service_hints.rs) — shared CLI-output component: the `rocm services stop|restart|remove --yes` hint text, shared by [`remote/mod.rs`](../apps/rocm/src/remote/mod.rs), [`serve_summary.rs`](../apps/rocm/src/serve_summary.rs), [`service_launch.rs`](../apps/rocm/src/service_launch.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs), and [`services_text.rs`](../apps/rocm/src/services_text.rs) so they cannot go out of sync with each other, as `serve_summary.rs`'s stop row did from the day it was added — #417. It also holds the helpers that map a service tool name to the command word, verb and past tense those hints and messages use (`service_action_command`, `service_action_verb`, `service_action_past_tense`) and to its `SandboxToolArg` (`sandbox_tool_arg_from_service_tool`); [`services_text.rs`](../apps/rocm/src/services_text.rs) reads `service_action_past_tense` from here rather than through the crate root. It quotes the service id with `shell_quote`, so a hint names exactly the argv it runs. -- [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, owns types (Phase 6b, ROCMAI-91): spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind — `AttachOutcome`/`RawModeGuard`/`ManagedSpawn`/`AutostartClaim` are its private types; `ManagedLaunchReport` is `pub(crate)`, since [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads its fields directly for the interactive summary. Most of this module's functions are `pub(crate)` for the same reason — other modules, including [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs) and [`service_control.rs`](../apps/rocm/src/service_control.rs), call into it; `main.rs` does not. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path, out of scope for Phase 6b. This module does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it: `run_attached_service` takes the stop for a Ctrl-C as a parameter, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) passes `run_internal_sandbox_tool`. +- [`service_launch.rs`](../apps/rocm/src/service_launch.rs) — mechanical relocation, since split into a directory of child modules, owns no code of its own beyond its declarations: spawning/attaching to a managed service process and streaming its log, the port allocation transaction and failed-launch cleanup (under the shared launch lock), plus the endpoint-key/auth checks that gate a public bind, divided among the children listed below. The file declares the children, a `test_support` module compiled only for tests (the fixtures more than one child's tests share), and `pub(crate) use` re-exports of the items other modules call, so [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), [`automations.rs`](../apps/rocm/src/automations.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs) and [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) still reach them as `crate::service_launch::name`; [`main.rs`](../apps/rocm/src/main.rs) does not. An item that sibling children share is `pub(super)` and reached through `super::`; everything else is private to the child that owns it. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs) rather than moving alongside, and `services()` lives in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs); the record removal and pruning it dispatches to now lives in [`service_records.rs`](../apps/rocm/src/service_records.rs), and only stop/restart are also reachable from the chat/LLM sandbox tool-calling path. This module does not import [`service_control.rs`](../apps/rocm/src/service_control.rs), which imports it: `run_attached_service` takes the stop for a Ctrl-C as a parameter, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) passes `run_internal_sandbox_tool`. +- [`service_launch/endpoint_auth.rs`](../apps/rocm/src/service_launch/endpoint_auth.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: bind-host validation and the endpoint API key policy. `validate_bind_host` refuses a non-loopback host without `--allow-public-bind` (`is_loopback_host` is the inverse of `rocm_engine_protocol::is_public_bind_host`, which `rocmd` shares); `resolve_endpoint_auth` decides the key a launch uses (none for a loopback bind unless a key is required, otherwise the supplied key trimmed or a generated one, refusing an empty key or one with a control character); `drop_orphaned_endpoint_key_on_already_running` clears a key staged for an attempt that spawned nothing; `ensure_public_bind_engine_supported` refuses a public bind with Windows managed Lemonade, whose key cannot be enforced; and `ensure_public_service_has_endpoint_key` is the guard a respawn runs so a public (or `--require-api-key`) service whose key file is gone is not reopened without authentication. All six are `pub(crate)` and re-exported by the parent: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads `ensure_public_service_has_endpoint_key`, [`tool_policy.rs`](../apps/rocm/src/tool_policy.rs) reads `is_loopback_host`, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) reads the rest. +- [`service_launch/launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the part of a managed launch that decides before anything spawns. `resolve_managed_port_in_transaction` refreshes liveness, answers an equivalent live service before leasing any port (`ManagedPortDecision`), and otherwise leases a concrete port against both the ports live records still reserve (`live_managed_services`, `reserved_service_ports`) and the real OS through [`serve_port.rs`](../apps/rocm/src/serve_port.rs); a caller must hold the launch lock across it and the record write that follows. `abandon_failed_managed_launch` undoes a failed attempt before that lock is released: a spawned child is terminated first and its record kept, marked stopped, while an attempt that never spawned has every artifact removed. `managed_service_reuse_refusal` (with `ManagedReuseRefusal`) is the one answer to whether a live matching service can serve a request (recipe, `--require-api-key`, device policy), and `no_usable_gpu_refusal` words the no-usable-GPU refusal once for `serve()`'s pre-flight and for the spawn. `managed_service_reuse_refusal` and `no_usable_gpu_refusal` are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs), and `ManagedReuseRefusal` is `pub(crate)` too because the first returns it (only [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) matches on it); `ManagedPortDecision`, `live_managed_services`, `resolve_managed_port_in_transaction` and `abandon_failed_managed_launch` are `pub(super)` for [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs). +- [`service_launch/spawn.rs`](../apps/rocm/src/service_launch/spawn.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns types: the detached engine child that the background and attached serve paths share. The process and stdio plumbing (`detach_background_command`, `attach_background_stdio`, `managed_service_process_command`, `managed_engine_startup_failure_detail`) is `pub(crate)` and, off Windows, re-exported: [`service_control.rs`](../apps/rocm/src/service_control.rs) reads all four for a restart and [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) reads the first three. `ManagedLaunchReport` (`pub(crate)`) is the engine-neutral result of a launch, and `ManagedSpawn` (`pub(super)`) says whether the launch found a live service or spawned a child. `spawn_managed_engine_child` is the whole serialized transaction under the launch lock (duplicate detection, port lease, manifest and log creation, argv preparation, the bounded immediate-exit check and record publication) and returns before any readiness wait; `launch_managed_engine_child_locked` is its side-effecting half, split out so that every failure after the record's first write goes through `abandon_failed_managed_launch`. The tests reach the spawn through a test-only, thread-scoped hook (`TEST_SPAWN_HOOK`, `pub(super)` so the tests in [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) can observe the state at the instant of the spawn). +- [`service_launch/readiness.rs`](../apps/rocm/src/service_launch/readiness.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns private items: waiting on a service after it is launched or found live. `wait_for_record_ready` and `wait_on_existing_service` run the wait (one wait is bounded by `READINESS_WAIT`, and `wait_past_readiness_budget` is what `rocm serve --wait` uses to keep polling past it until the server answers or stops being live), `service_pid_is_alive`, `record_still_live` and `service_still_live` decide whether the tracked process is still there, `settle_exited_service` demotes the manifest of one that is not, and `died_during_wait` and `record_wait_outcome` end a wait, writing its outcome back without discarding what changed on disk meanwhile. `e2e_simulate_oom_launch` and `simulate_oom_managed_launch` are the end-to-end fault-injection seam behind the `e2e-oom-fault-injection` feature: the first is a `const fn` returning false without the feature, is `pub(crate)` and re-exported, and is read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs); the second fabricates the on-disk state of a launch that ran out of memory. `wait_for_record_ready`, `wait_on_existing_service`, `service_pid_is_alive`, `died_during_wait`, `record_wait_outcome` and `simulate_oom_managed_launch` are `pub(super)` for [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs); the rest are private. +- [`service_launch/start.rs`](../apps/rocm/src/service_launch/start.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, no owned types: the background launch path. `start_managed_service` runs the spawn transaction, releases the launch lock once the record is published, waits for HTTP readiness and returns a `ManagedLaunchReport`; if the launch fails before a record exists it drops the endpoint key `serve()` staged for the attempt. `print_managed_launch_plain` renders the report line by line for output that is not interactive, and `wait_begin_client_config` returns the one-time client configuration for a generated key when a wait begins, built by `render_endpoint_client_config` (`pub(super)`, since [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) prints it too). The three functions other modules call are `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs). +- [`service_launch/helper_autostart.rs`](../apps/rocm/src/service_launch/helper_autostart.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns a private type: starting the background automation helper (`rocm daemon`) when a watcher needs it. `ensure_background_helper_running` and `ensure_background_helper_running_quiet` do nothing unless an automation is enabled and [`background_helper.rs`](../apps/rocm/src/background_helper.rs)'s `background_helper_already_running` says no helper is running; `ensure_background_helper_running_with` takes the spawn as a parameter so the decision is testable, and `spawn_background_helper` does the detached spawn. Concurrent callers queue on an autostart lock, and `AutostartClaim` (with `AUTOSTART_CLAIM_TTL_MS`, `read_autostart_claim`, `write_autostart_claim` and `autostart_spawn_in_flight`) records a spawn in flight so the next holder defers instead of starting a second helper. Both entry points are `pub(crate)` and re-exported: [`automations.rs`](../apps/rocm/src/automations.rs) calls the first, and [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs) and [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) call the quiet one. +- [`service_launch/attach.rs`](../apps/rocm/src/service_launch/attach.rs) — mechanical relocation out of [`service_launch.rs`](../apps/rocm/src/service_launch.rs), a leaf child of it, owns private types: the attached serve path (`--verbose`/`--foreground`). `run_attached_service` spawns the same detached child the background path does and follows its log in the terminal until the user detaches with Ctrl-D (the server keeps running) or stops it with Ctrl-C, or the engine exits; `AttachOutcome` is that result and `detach_key_outcome` maps a key press to it. `stream_attached_logs` reads keys in crossterm raw mode (`RawModeGuard` restores the terminal on every exit path) and `stream_attached_logs_no_tty` follows the log until the process exits when stdin is not a terminal. `finish_attached_stop` turns the answer of the stop that Ctrl-C ran into output: a confirmed stop is reported on stdout, an unconfirmed one becomes the error, and a failed stop signals the child this session launched. `run_attached_service` is `pub(crate)`, re-exported, and read from [`serve_cmd.rs`](../apps/rocm/src/serve_cmd.rs). - [`service_records.rs`](../apps/rocm/src/service_records.rs) — mechanical relocation, owns types: removing one managed-service record and every file it owns (`remove_managed_service_record`, `service_record_artifacts`) and the `rocm services prune` plan/render/apply steps (`build_service_prune_plan`, `render_service_prune_plan`, `apply_service_prune_plan`, `prune_managed_service_records`) with the age gate behind them (`DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS`, `service_prune_min_age_hours`). It reads record state through [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs) and [`services_text.rs`](../apps/rocm/src/services_text.rs) and builds its advice text with [`service_hints.rs`](../apps/rocm/src/service_hints.rs); none of the three depends on it. `remove_managed_service_record`, `prune_managed_service_records`, `service_prune_min_age_hours` and `DEFAULT_SERVICE_PRUNE_MIN_AGE_HOURS` are `pub(crate)` (`services()` in [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) calls the first three; the `ServicesCommand::Prune` clap definition in [`cli.rs`](../apps/rocm/src/cli.rs) uses the constant), and `ServicePruneOutcome`, the return type of `prune_managed_service_records`, is `pub(crate)` with `pub(crate)` `text`, `removed_records`, `removed_files` and `failures` fields that `services()` reads. Every other item is private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths`, `plant_service_record` and `plant_service_record_without_its_key` fixtures through `crate::tests::` (the last two widened to `pub(crate)` for that). - [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs) — mechanical relocation, dispatch-adjacent, no owned types: `rocm services`' command handler (`services()`, which dispatches `list`, `logs`, `stop`, `restart`, `remove` and `prune`) and `run_approved_service_action`, the approval-gated stop/restart path. `ServicesCommand` lives in [`cli.rs`](../apps/rocm/src/cli.rs). Both functions are `pub(crate)`: `dispatch()` calls `services()` through a re-import, and [`advised_commands.rs`](../apps/rocm/src/advised_commands.rs)'s test reaches `run_approved_service_action` through a test-only re-import in `main.rs`. The module reads from [`services_text.rs`](../apps/rocm/src/services_text.rs), [`service_control.rs`](../apps/rocm/src/service_control.rs), [`service_records.rs`](../apps/rocm/src/service_records.rs), [`service_hints.rs`](../apps/rocm/src/service_hints.rs) and [`cli_audit.rs`](../apps/rocm/src/cli_audit.rs); none of them calls back into it. The cluster's one regression test lives in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::`. - [`services_text.rs`](../apps/rocm/src/services_text.rs) — mechanical relocation, owns private types (Phase 6f, ROCMAI-209): the CLI/TUI logs browser (`render_logs_text`, `render_logs_browser_text`, `render_logs_browser_page_text`, `render_logs_browser_page_text_with_options`, `logs_browser_total_pages`, the `LogBrowserEntry` type, `collect_log_browser_entries`, `filter_log_browser_entries`, `log_browser_source_label`, `write_past_attempts_hint`), the `rocm services`/`rocm services logs` status surfaces (`render_services_text`, `render_services_json`, `render_service_logs_text`, `render_service_logs_text_with_options`, `render_service_action_result`, `read_managed_service_manifest`, `load_managed_service`), the `--follow` tail behind `rocm logs --service --follow` and `rocm services logs --follow` (`follow_service_logs`, `follow_service_logs_with`, the `LogTail`, `LogSegment` and `FollowFooter` types, `write_log_segments`, `newly_ready`; new code written in this module, not relocated), the internal `rocmd`-compatible status surface (`render_internal_status_text`), the `stop_server` answer shape and its renderer (`internal_stop_server_value`, `service_stop_unconfirmed`, `render_unconfirmed_service_stop`, `process_check_command`), plus `validate_service_id`, `list_log_files`, `read_optional_tail_lines`, and the CLI-lifecycle-log tail formatting helpers (`format_cli_lifecycle_tail_line`, `lifecycle_event_label`, `humanize_log_label`, `lifecycle_field`, `read_tail_lines`). The managed-service stop and restart implementation that used to live here is in [`service_control.rs`](../apps/rocm/src/service_control.rs). The `LogBrowserEntry` struct moved here from `main.rs` with the logs browser and stays private. `dispatch()` calls directly into this module at several points — `render_internal_status_text`, `render_service_logs_text`, `render_logs_browser_text`, and `render_logs_text` — all re-imported via `use crate::services_text::{...};`; [`services_cmd.rs`](../apps/rocm/src/services_cmd.rs)'s `services()` calls `render_services_text`, `render_services_json` and `render_service_logs_text` through its own imports. `render_service_action_result`, `service_stop_unconfirmed` and `validate_service_id` are also `pub(crate)` and read from `services_cmd.rs`'s `run_approved_service_action`; `validate_service_id` is also read from [`service_records.rs`](../apps/rocm/src/service_records.rs)'s `remove_managed_service_record`/`service_record_artifacts`, and `load_managed_service` from its `remove_managed_service_record`/`apply_service_prune_plan`. `load_managed_service` is `pub(crate)`, read from [`runtime_services.rs`](../apps/rocm/src/runtime_services.rs) (reconciling live servers against a runtime activation), from [`service_control.rs`](../apps/rocm/src/service_control.rs) and from [`managed_service_status.rs`](../apps/rocm/src/managed_service_status.rs)'s own test module; `read_managed_service_manifest` and `internal_stop_server_value` are `pub(crate)` for `service_control.rs`, and `process_check_command` for its tests; `read_tail_lines`, `render_logs_text`, `render_logs_browser_text`, `render_services_text`, `render_service_logs_text`, and `validate_service_id` are also `pub(crate)`, read from [`chat/mcp_bridge.rs`](../apps/rocm/src/chat/mcp_bridge.rs)'s read-only command runner; `read_optional_tail_lines` and `render_service_action_result` are also `pub(crate)`, read from [`service_launch.rs`](../apps/rocm/src/service_launch.rs)'s managed-spawn/attach path. Every other item in this list has no caller outside this module and stays private. The cluster's regression tests live in this module's own `mod tests`, reaching the shared `test_paths` fixture through `crate::tests::` the same way other mechanically relocated modules do; `assert_no_retry_line` stays here as a `pub(crate)` test helper because it checks the unconfirmed-stop rendering and `service_launch.rs`'s tests read it too. Two tests that call this module's `load_managed_service` only incidentally — their primary subject is `runtime_services::pin_service_record_to_runtime`/`restore_service_record_pin` and `runtime_services::restart_service_onto_runtime_with`'s outcome classification respectively — stay in `main.rs`'s shared `mod tests` instead. diff --git a/docs/testing.md b/docs/testing.md index 705a024f16..94ea97343d 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2960,7 +2960,7 @@ the per-PR mock lane. The host predicate itself is the shared than restated in either, so the test cannot drift from the production rule. The bypass decision's other arguments are in the same position — the unit tests -in `apps/rocm/src/service_launch.rs` pin `managed_service_reuse_refusal` itself, +in [`launch_transaction.rs`](../apps/rocm/src/service_launch/launch_transaction.rs) pin `managed_service_reuse_refusal` itself, not which values `serve()` hands it — so each one has a scenario instead: serve-30 for `require_api_key`, serve-32 for the requested recipe. The device policy needs neither: the predicate takes it by value rather than as an diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 2259478525..3df5db80b7 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -307,7 +307,7 @@ Feature: Model serving # `--require-api-key` reuse refusal (#604). A running server's key is read once # at launch; reuse cannot retroactively satisfy a demand for auth the server # never got, so the idempotency guard refuses instead of upgrading the record - # in place (see `spawn_managed_engine_child` in `service_launch.rs`). This is the + # in place (see `spawn_managed_engine_child` in `service_launch/spawn.rs`). This is the # local-side proof that the printed stop hint is not just present but correct # and directly runnable: the scenario stops the service with it and checks the # record is left `stopped`, the same proof `@id:service-stop-yes-stops-a- @@ -318,10 +318,10 @@ Feature: Model serving # this guard without a live GPU, same as every other `@requires-gpu` scenario # on this lane. Linux-only: `--require-api-key` forces auth even on a loopback # bind, and the serve path's `ensure_public_bind_engine_supported` guard - # (defined in `service_launch.rs`, called from `serve_cmd.rs`) refuses that combination - # for Lemonade on Windows (its server can't receive the key through the - # Windows detached-spawn primitive), so this scenario cannot reach the - # refusal it exercises there. + # (defined in `service_launch/endpoint_auth.rs`, called from `serve_cmd.rs`) + # refuses that combination for Lemonade on Windows (its server can't receive the + # key through the Windows detached-spawn primitive), so this scenario cannot + # reach the refusal it exercises there. @id:serve-require-api-key-refuses-unauthenticated-reuse @requires-gpu @requires-engine:lemonade @requires-os:linux Scenario: serve-24 - Reusing a running server without a key refuses a --require-api-key request Given a managed runtime is active @@ -725,7 +725,7 @@ Feature: Model serving # so it cannot be combined with the attached modes, which already stay on the # terminal. The waiting itself (a live server that becomes ready, one that # exits first) is covered by `wait_past_readiness_budget_*` in - # `apps/rocm/src/service_launch.rs`, since a scenario would need a slow engine. + # `apps/rocm/src/service_launch/readiness.rs`, since a scenario would need a slow engine. @id:serve-wait-conflicts-with-attached-modes Scenario: serve-43 - Waiting for readiness cannot be combined with attached mode When the user serves a model with both --wait and --verbose @@ -750,7 +750,7 @@ Feature: Model serving # "stop the service holding that port" advice is checked here by its wording # only; the advice is actually proven by # `stopping_the_reserving_record_frees_its_explicit_port_for_reuse` in - # apps/rocm/src/service_launch.rs, per AGENTS.md section 3. + # apps/rocm/src/service_launch/launch_transaction.rs, per AGENTS.md section 3. # Runs in the merge queue, like its automatic-port sibling serve-auto-port-skips-occupied. @id:serve-explicit-port-reserved @requires-gpu @merge-queue Scenario: serve-45 - Serving on a port another managed service already holds is refused diff --git a/tests/e2e-cucumber/tests/e2e/automations_steps.rs b/tests/e2e-cucumber/tests/e2e/automations_steps.rs index 95972ef56e..82176a92d7 100644 --- a/tests/e2e-cucumber/tests/e2e/automations_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/automations_steps.rs @@ -101,7 +101,7 @@ fn write_harness_autostart_claim(path: &std::path::Path) { /// Re-stamp a claim `plant_inflight_autostart_claim` planted, right before the /// CLI runs. The CLI honours a claim for only 30 seconds -/// (`AUTOSTART_CLAIM_TTL_MS` in `apps/rocm/src/service_launch.rs`). The mock +/// (`AUTOSTART_CLAIM_TTL_MS` in `apps/rocm/src/service_launch/helper_autostart.rs`). The mock /// lane runs up to 64 scenarios at once, and every step blocks a runtime /// worker while its `rocm` child runs, so this scenario's When can start more /// than 30 seconds after its Given. The CLI then rightly reads the claim as diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 079f588e7c..714aa6dab0 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -2527,7 +2527,7 @@ async fn plant_oom_launch(world: &mut E2eWorld) { // in the `When`, which left this step a no-op whose deletion changed nothing. // // The consumer of this variable is `e2e_simulate_oom_launch` in - // `apps/rocm/src/service_launch.rs`. Unlike `OOM_FAULT_INJECTION_ENV` (shared + // `apps/rocm/src/service_launch/readiness.rs`. Unlike `OOM_FAULT_INJECTION_ENV` (shared // through `e2e-report`, whose producer and consumer are both test-side), the // consumer here is the shipped binary's crate, which must not depend on the // e2e harness — so the name is spelled out on both sides. A typo fails loudly From 516df40da3247b309ca49d309d80f762c637b849 Mon Sep 17 00:00:00 2001 From: Tomas Saaristola <144015745+tomastola@users.noreply.github.com> Date: Mon, 9 Nov 2026 18:21:17 +0000 Subject: [PATCH 032/102] ci: run the nightly GPU E2E lanes on a shared GPU queue (#950) * ci: publish nightly E2E runs from a shared GPU queue Adds nightly-queue.yml, which publishes nightly GPU lanes run on a shared GPU queue, one lane per GPU model in its fleet. The run itself happens next to the queue; this workflow prints a finished run's lane matrix and inputs, uploads the lanes' files and passes or fails with its verdict. It is manual-only and gates nothing: the queue's runs do not yet use a lane image carrying the suite, and the shared runner does not yet make the published verdict trustworthy enough to gate on. The scheduled lanes in nightly.yml remain the nightly gate. The publishing job holds no secret and no queue credential, and asks for no permissions. Its runner is a persistent self-hosted machine that any job naming its label can run on, so the queue credential stays with the queue side. The runner carries the queue-report label alone, without GitHub's default self-hosted, OS and architecture labels, so a job reaches it only by naming it. A workflow_contract test holds the workflow to that: dispatch only, two allowed expressions, one action (upload-artifact, pinned to a SHA, no hidden files), no environment, runtime-token variable or permission, and no YAML shapes the check cannot read. It also refuses any other workflow on main that names the label or this workflow literally, so none lands on the runner or waits on this one by mistake. Workflows on other branches are out of its reach, which the doc states. docs/nightly-on-the-queue.md is the contract: what a lane must write to /out to be published, how each lane outcome is reported, what the workflow needs, and why it is built this way, including what the shared runner does not yet protect. ci-hardware-testing.md links to it. Signed-off-by: Tomas Saaristola * ci: run the E2E suite in each queue lane from a checkout of the commit Each lane of the queue nightly now checks out the commit its run resolved and runs a command from that checkout, so what a lane runs is defined here and changes through a pull request like any other test. scripts/run-e2e-lane.sh is that command: the self-hosted nightly job's build, pre-warm and suite, from a cold start, with nightly scenarios included. A lane has no toolchain or cache from an earlier run, so it installs the pinned Rust toolchain when cargo is missing. The suite's results are copied to /out on every way out, so a failing suite still reports which scenarios failed, and the lane keeps the suite's status. Signed-off-by: Tomas Saaristola * ci: start, wait for and publish a queue run from GitHub queue-run.yml runs on the queue-report runner, on dispatch with a branch to test or nightly on main. It asks the queue side for a run of that branch, waits for it, then publishes it as nightly-queue.yml does. It holds no secret and no permission: the queue side starts the run with its own credential and trusts the request because the runner runs only these two workflows from main. The credential contract test now covers both workflows; a schedule's cron line is the one place `*` is allowed. Signed-off-by: Tomas Saaristola * ci: publish a finished queue run only by its id queue-run.yml now starts, waits for and publishes every run, the nightly included, so nightly-queue.yml no longer publishes a default "newest scheduled run". It republishes one run by its required request_id, for instance one whose queue-run.yml job was cancelled while the run went on. With no default, an old pass can never stand in for a night that did not run; that night shows as a missing scheduled job instead. Signed-off-by: Tomas Saaristola * ci: retry apt-get update in a queue lane, and pass shellcheck A lane starts from a fresh container against the public Ubuntu archive, and while a mirror is mid-sync apt-get update fails for a few minutes; the lane then failed before building anything. Retry it as the WSL lane in e2e-selfhosted.yml does. The script also now passes the repository's shellcheck settings, which it did not. Signed-off-by: Tomas Saaristola * ci: pin what the queue workflows and the lane script do The contract tests checked that the queue workflows hold no credential, not that they do their job: a run step that ignored the dispatched branch, published some other run, or was allowed to fail quietly still passed. They now pin each publishing step and the input behind it, read as its own YAML block, the schedule, that a running job is never cancelled for a newer one, and that each workflow uploads, whatever the verdict, the artifact the doc names for it. The lane script is run against stubbed tools: it runs the preflight, build, pre-warm and full suite in that order with the environment the nightly job gives them, exits with the suite's status, keeps the suite's results and its status even when the copy fails, rides out a failing apt-get update, and on an image without cargo installs rustup and leaves the toolchain to the checkout. The doc named one artifact for both workflows; it now names each. It also says what replacing the existing nightly lanes would lose (the nightly channel, Windows and WSL), that GitHub keeps only one dispatch waiting, and that a signed report helps only if something other than the runner checks the signature. Signed-off-by: Tomas Saaristola * ci: give nightly-queue room to finish a run before publishing it The queue side no longer finishes a run on its own: a run is finished when the job following it asks. So a run whose queue-run.yml job was cancelled is finished by nightly-queue.yml, once its lanes are done, before it is published. Finishing fetches and scans every lane's files, which the publisher allows 30 minutes, so the job's timeout goes from 15 to 45 minutes. The contract doc says so, and that the runner's calls to the queue side are now three (start, status, finish), none carrying a file. The workflows' commands are unchanged. Signed-off-by: Tomas Saaristola * ci: run both channels in each queue lane, and drop the advisory framing A queue lane now runs the full suite on the release channel and then the nightly channel, as the GPU jobs in nightly.yml do: one build, each channel with its own runtimes and its own results under /results. Both channels always run, and the lane fails if either did. The queue's results are trusted the way the self-hosted runners' nightly results are: the same infrastructure and access, and neither signs what it reports. The doc no longer calls them advisory or caveats their integrity, and says that native Windows and WSL remain the gap: the queue runs Linux pods, so those lanes stay on their dedicated runners. Signed-off-by: Tomas Saaristola * ci: pin the rustup install, correct the wait bound, and say where the main-only restriction lives - The lane installs a pinned rustup-init checked against a recorded SHA-256 instead of piping an installer script into a shell. - The wait bound is stated once and from the real limit: lanes wait at most 15000 s (4 h 10 min), and the job limit leaves room to finish and publish. - The doc says the main-only restriction is a check on the runner, outside this repository, and keeps the runner treated as untrusted. - The doc gives one window for each of a lane's files and a finished run's results, and no longer shows the publish command's path. - The contract refuses an expression inside a run: script, so inputs reach the shell only through env:. Signed-off-by: Tomas Saaristola * ci: refuse a whitespace-only channel list, and say what a queue request can reach - run-e2e-lane.sh counts the words in E2E_CHANNELS, so a tab or newline alone is refused like an empty list; the test gains those cases. - queue-run.yml no longer cross-references a gating condition the doc dropped. - The doc says what the lane pods are given, and that the queue's results are an additional signal next to nightly.yml's report artifacts. - The contract no longer pins wording removed from the doc. Signed-off-by: Tomas Saaristola * ci: refuse a repeated channel, keep a failed results cleanup loud, and tie the queue timeout to the wait limit - run-e2e-lane.sh refuses a channel named twice, and returns a failed cleanup of the previous channel's results instead of running on with them. - A contract test ties queue-run.yml's timeout, its comment and the doc to the 15000 s wait limit. - The doc says the both-channel lane time is not measured on a queue lane. Signed-off-by: Tomas Saaristola --------- Signed-off-by: Tomas Saaristola (cherry picked from commit 07c93f323513a0d0b85af19303f67ddba54c90d9) Signed-off-by: juhovainio --- .github/workflows/nightly-queue.yml | 51 ++ .github/workflows/queue-run.yml | 56 ++ docs/ci-hardware-testing.md | 28 + docs/nightly-on-the-queue.md | 248 +++++++ scripts/run-e2e-lane.sh | 169 +++++ xtask/src/workflow_contract.rs | 959 +++++++++++++++++++++++++++- 6 files changed, 1510 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/nightly-queue.yml create mode 100644 .github/workflows/queue-run.yml create mode 100644 docs/nightly-on-the-queue.md create mode 100755 scripts/run-e2e-lane.sh diff --git a/.github/workflows/nightly-queue.yml b/.github/workflows/nightly-queue.yml new file mode 100644 index 0000000000..065dbb67de --- /dev/null +++ b/.github/workflows/nightly-queue.yml @@ -0,0 +1,51 @@ +# GPU E2E on the shared queue — publish a run that has already finished. +# +# queue-run.yml starts, waits for and publishes runs. This publishes one again +# by its id, for instance one whose queue-run.yml job was cancelled while the +# run went on: once the run's lanes are done, it has the queue side finish the +# run if no one has, prints its report and lane matrix, uploads the lanes' +# files, and passes or fails with the run's verdict. It needs no credential, so +# the shared self-hosted runner never holds the queue's. What that protects is +# in docs/nightly-on-the-queue.md. +name: Nightly E2E (queue) + +on: + # Manual only. + workflow_dispatch: + inputs: + request_id: + description: "The id of the finished run to publish" + type: string + required: true + +# Nothing here reads the repository. Uploading files uses the per-job runtime +# token GitHub issues to every job regardless of these permissions. +permissions: {} + +concurrency: + group: nightly-queue + cancel-in-progress: false + +jobs: + publish: + name: Publish nightly results + runs-on: [queue-report] + # Nothing here waits on GPUs, but finishing a run nobody finished fetches + # and scans every lane's files, which the publisher allows 30 minutes. + timeout-minutes: 45 + steps: + - name: Publish the run + env: + REQUEST_ID: ${{ inputs.request_id }} + # One line: the workflow contract forbids backslashes in this file. + run: /opt/queue/bin/queue-publish --out "${RUNNER_TEMP}/results" --request "${REQUEST_ID}" + + - name: Upload lane results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: nightly-queue-results + path: ${{ runner.temp }}/results + # The publisher's files only; a lane's hidden files are not published. + include-hidden-files: false + if-no-files-found: warn diff --git a/.github/workflows/queue-run.yml b/.github/workflows/queue-run.yml new file mode 100644 index 0000000000..f35bcd4cc0 --- /dev/null +++ b/.github/workflows/queue-run.yml @@ -0,0 +1,56 @@ +# GPU E2E on the shared queue — start a run, wait for it, publish it. +# +# This job runs on the queue-report runner, which sits next to the queue. It asks +# for a run of one branch of this repository (main when scheduled), waits for +# every lane to finish (up to 4 h 10 min), then prints the run's report and +# lane matrix, uploads the lanes' files, and passes or fails with the run's +# verdict. It needs no credential and no permission: the runner asks from where +# it stands. The runner is set up to run only this workflow and nightly-queue.yml +# as they are on main; that is enforced on the runner, outside this repository. +# What that protects is in docs/nightly-on-the-queue.md. +name: E2E on the queue + +on: + workflow_dispatch: + inputs: + branch: + description: "The branch of this repository to test" + type: string + default: main + # Whether this is a required check is a repository setting, not set here. + schedule: + - cron: "17 2 * * *" + +# Nothing here reads the repository. Uploading files uses the per-job runtime +# token GitHub issues to every job regardless of these permissions. +permissions: {} + +# One run at a time; the queue side refuses a second while one is unfinished. +concurrency: + group: queue-run + cancel-in-progress: false + +jobs: + run: + name: Run and publish + runs-on: [queue-report] + # The lanes' wait is at most 15000 s (4 h 10 min); finishing and publishing + # the run can take 30 min more, and the rest is margin. + timeout-minutes: 330 + steps: + - name: Run on the queue and publish + env: + # Empty when scheduled. + BRANCH: ${{ inputs.branch }} + # One line: the workflow contract forbids backslashes in this file. + run: /opt/queue/bin/queue-publish --out "${RUNNER_TEMP}/results" --start "${BRANCH:-main}" + + - name: Upload lane results + if: always() + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: queue-run-results + path: ${{ runner.temp }}/results + # The publisher's files only; a lane's hidden files are not published. + include-hidden-files: false + if-no-files-found: warn diff --git a/docs/ci-hardware-testing.md b/docs/ci-hardware-testing.md index cdc89135e3..b7ab118afa 100644 --- a/docs/ci-hardware-testing.md +++ b/docs/ci-hardware-testing.md @@ -44,6 +44,15 @@ R9700 and MI350P were dropped from the per-PR gate (ROCMAI-125): still covered, but off every PR's critical path, so they moved to the nightly-only rung instead of being removed outright. +[`queue-run.yml`](../.github/workflows/queue-run.yml) runs GPU lanes on a +shared GPU queue, one per GPU model, so no runner holds a GPU for the length +of a job, and publishes the result; +[`nightly-queue.yml`](../.github/workflows/nightly-queue.yml) publishes a +finished run again. They cover the Linux GPU nightly on both the release and +the nightly channel; native Windows and WSL stay on their dedicated runners, +since the queue runs Linux pods. See +[nightly-on-the-queue.md](nightly-on-the-queue.md). + The release-candidate rung's `release/**` trigger merged in PR #415, so `e2e-selfhosted.yml` on `main` already triggers on `push: branches: [main, "release/**"]`. On that trigger, every self-hosted GPU job additionally pins @@ -113,6 +122,18 @@ mixed-silicon pool. `every_self_hosted_lane_pins_a_hardware_label` in the lane simply passes on the wrong GPU and reports under the label it was named for. +No workflow other than [`queue-run.yml`](../.github/workflows/queue-run.yml) +and [`nightly-queue.yml`](../.github/workflows/nightly-queue.yml) may name the +`queue-report` label. +The shared-queue publishing runner carries that label alone, without GitHub's +default `self-hosted`, OS and architecture labels, so a job reaches it only by +naming that label, and `queue_nightly_publisher_holds_no_credential` in +[`workflow_contract.rs`](../xtask/src/workflow_contract.rs) fails if another workflow on `main` names it +literally. A label assembled from an expression is not seen; the rule guards +against mistakes, not evasion. The runner is registered to this repository +alone, so no runner group reaches it. See +[nightly-on-the-queue.md](nightly-on-the-queue.md). + `e2e` is the blocking, GitHub-hosted mock job: `@requires-gpu` scenarios resolve to skip here, and known bugs resolve to xfail from `expectations.toml`. It is a required check and must stay green. @@ -514,6 +535,13 @@ does not dispatch self-hosted-runner jobs from an external fork's against branches and PRs within the repository (and on `workflow_dispatch`, which requires write access to trigger). +That holds for every self-hosted runner here, and so does its limit: approval +does not change what an approved fork job may do, and labels do not restrict +which jobs reach a runner, so a job naming a runner's labels can run there. +Only the shared-queue publishing runner is built around that, by holding no +queue credential; see +[nightly-on-the-queue.md](nightly-on-the-queue.md#why-it-works-this-way). + ## Notes - The hardware jobs build and run **release** binaries: they assert diff --git a/docs/nightly-on-the-queue.md b/docs/nightly-on-the-queue.md new file mode 100644 index 0000000000..be49046878 --- /dev/null +++ b/docs/nightly-on-the-queue.md @@ -0,0 +1,248 @@ + + +# Nightly E2E on the shared GPU queue + +The nightly GPU lanes can run on a shared queue of GPU workers: one lane per +GPU model in the queue's fleet. The run itself happens next to the queue, which +submits the lanes, waits for them and collects their results. +[`queue-run.yml`](../.github/workflows/queue-run.yml) starts a run of one +branch, waits for it and publishes it here: its lane matrix in the job log and +summary, the lanes' files as the `queue-run-results` artifact, and a pass or +fail. [`nightly-queue.yml`](../.github/workflows/nightly-queue.yml) publishes +a run that has already finished. + +The queue's lanes cover the Linux GPU nightly: one lane per GPU model, each +running the full suite on both the `release` and the `nightly` channel, as the +GPU jobs of [`nightly.yml`](../.github/workflows/nightly.yml) do (see +[ci-hardware-testing.md](ci-hardware-testing.md)). Their results are trusted the +way those jobs' results are: the queue runs on the same infrastructure and under +the same access as the self-hosted runners, and neither signs what it reports. + +**The remaining gap is native Windows and WSL.** The queue runs Linux pods, so +the native Windows and WSL lanes stay on their dedicated runners, in `nightly.yml` +and [`e2e-selfhosted.yml`](../.github/workflows/e2e-selfhosted.yml); nothing +here replaces them. Which checks are required before merging is a repository +setting, outside this page. + +This page is the **contract** between this repository and the queue side: what +a lane must do to be reported, and how to read the report. The queue side — +how runs are started and executed, and the `queue-publish` command the +workflows call — is maintained privately with the queue, not in this +repository. + +## Starting a run + +Dispatch `queue-run.yml` with the branch to test (`main` by default); it also +runs every night at 02:17 UTC on `main`. The job asks the queue side for a run +of that branch, named for the job's run id and attempt, so re-running the job +starts a new run. It waits for every lane to finish, up to 4 h 10 min (15000 s, the run's +wait limit below), then publishes the run as below. The job log starts with the run's inputs +(below). Only one run goes at a time: a second +dispatch waits for the first, and the queue side refuses one while another, +however started, is unfinished. GitHub keeps at most one job waiting, so a +third dispatch, or the nightly schedule, replaces a dispatch still waiting +and that one never runs. The branch must exist in this repository; +otherwise the job fails at once. + +## Publishing a finished run + +Dispatch `nightly-queue.yml` with a run's `request_id` — the id its +`queue-run.yml` job printed — to publish that run again, for instance when that +job was cancelled while the run went on. Such a run was never finished, so if +its lanes are done the queue side finishes it first. Do that within a day of +its lanes finishing: the queue keeps a lane's files for about a day, and a run +not finished by then loses them. It never starts a run, +and there is no default: a run that did not happen cannot be published in its +place. It +fails, rather than reporting green, when: + +- the named run has not finished, or never existed +- the run failed, could not start, was cancelled before it finished, or + discovered no lanes at all + +A finished run's results are kept for two weeks: each new run removes older +ones, so a finished run's `request_id` can be published again until then. An +uploaded artifact follows this repository's own artifact retention. + +## What a run executes + +Every run is asked for: by `queue-run.yml`, or by an operator on the queue +side. A run has six inputs; the queue side sets their defaults, and any of them +can be overridden for a single run (`queue-run.yml` sets the branch): + +- the branch — the branch of this repository to test, `main` by default. The + run resolves it to a commit once, as it starts, and every lane checks out + that commit, so all lanes test the same code however long each waits for a + GPU. The report's first line names the commit. Only a branch of this + repository can be named, never a commit id. A lane that cannot check the + commit out fails, with the reason in its `lane.log`. +- the lane image — the container every lane runs +- the lane command — a single line run by `bash` under `set -euo pipefail` in + that image, from the root of the checkout, with the commit in + `ROCM_CLI_COMMIT`. The default is + [`scripts/run-e2e-lane.sh`](../scripts/run-e2e-lane.sh), which builds + once and runs the full suite from a cold start on each channel, so what a + lane runs is changed here, by a pull request, like any other test +- the blocking lanes — which lanes can fail the run; left empty, every lane can +- the excluded lanes — GPU models to leave out of this run +- the wait — how long the run waits for its slowest lane before cancelling + what is left, which is then reported as **incomplete**; at most 15000 + seconds, and a run given more does not start + +The lane set itself is discovered, not listed: one lane per GPU model available +in the queue's fleet, minus any excluded. New hardware appears without any +edit, and hardware that is away does not produce a red lane — though a +blocking lane that was not discovered is reported as a warning, since that run +did not test it. A run that discovers no lanes at all fails. + +A discovered lane that is not blocking is advisory: reported, but unable to +fail the run. Promoting it is a deliberate edit once it has run clean. + +Every lane runs the same image and command; there is no per-lane variant. The +channels are the command's, not the queue's: the default command runs the +`release` channel and then the `nightly` one (`E2E_CHANNELS`, default `release +nightly`), each with its own runtimes and its own suite run, so a lane takes +about twice as long as one channel would. That time has not been measured on a +queue lane from a cold start, so a lane that cannot finish both channels within +the run's wait (which also covers waiting for a GPU) is reported as +**incomplete**; the wait can be raised for a run, up to its limit. Both always run, so a failing +channel does not hide the other, and the lane fails if either did. + +The queue's results are an additional signal, not a replacement: the lanes +write only into the queue artifact, and the `e2e-gpu--report` +artifacts that `nightly.yml`'s consolidated report reads are not produced from +them. + +## Output + +**A lane must write everything it wants reported directly into `/out`.** The +default command writes each channel's suite results to `/out//results`. + +- Each lane's `/out` lands in the workflow's artifact, `queue-run-results` + from `queue-run.yml` or `nightly-queue-results` from `nightly-queue.yml`, + one directory per lane, next to that lane's `lane.log` (the command's combined + output). +- Files keep their paths under `/out`, subdirectories included. `lane.log` is + written by the harness; a lane must not use that name. Names ending in + `.quarantine`, hidden files and anything under a hidden directory are never + published. +- Anything a lane leaves elsewhere is not published. The queue records more + than the suite asked for, including details of the machine that ran it, and + the published results are public. +- Collected files are also scanned, and a file containing an identifier the + queue side treats as infrastructure — any UUID, a private overlay-network + address, a few internal names — in its content or its path is withheld + whole, a directory so named with everything in it, counted as one; the + report's `scanned … withheld …` line says how many. The scan is a backstop, not the + control: only writing to `/out`, and nothing identifying, is. + +Widening what is collected is not the fix for a missing file; writing the file +to `/out` is. + +A lane passes when its command exits zero. Capturing the command's output into +`lane.log` does not affect that status. + +### Reading the matrix + +The report has one row per lane and a summary. A lane is not simply pass or +fail, and the distinction matters when deciding whether to chase something: + +| Outcome | Meaning | Fails the run? | +|---|---|---| +| **passed** | the command exited zero | no | +| **failed** | the lane's command or its container failed; lanes are not retried, so one failure is final. `lane.log` shows a test failure; a lane whose container never started, for example because its image could not be pulled, has no `lane.log` | yes | +| **incomplete** | no result: cancelled at the run's deadline, or no worker in the fleet could take it | yes | +| **missing** | discovered but never submitted | yes | +| **advisory** | any outcome but passed, on a lane that is not blocking | no | + +The matrix is drawn before the lanes' files are collected, so a lane that +passed but whose files did not all come back, or that came back with nothing +from `/out` (every lane that runs writes `lane.log` there), is still counted +as passed in it. A `no result` line after the matrix names each such lane, and +that line is what counts: a blocking lane without a result fails the run. + +**incomplete** and **missing** fail the run because a blocking lane without a +result is not a pass, but they point at queue capacity or infrastructure +rather than at a regression. + +## What the workflow requires + +- a self-hosted Linux runner registered with the `queue-report` label alone, + without GitHub's default `self-hosted`, OS and architecture labels, so only a + job naming `queue-report` lands on it, providing + the queue side's publish command and read access to the queue side's results +- a registered runner name and host name that do not identify internal + infrastructure, since GitHub prints both into every job log + +Both workflows require **no secret** and no permission, and contract tests in +[workflow_contract.rs](../xtask/src/workflow_contract.rs) +(`queue_nightly_publisher_holds_no_credential`, `queue_run_holds_no_credential`) +keep it that way. + +That test reads the workflow files on `main`. A workflow on another branch of +this repository can still name the runner's label, which is one reason the +runner holds no credential, below. On `main`, the tests also refuse any +other workflow that names the `queue-report` label or these workflows, so none +sends a job to the runner or waits on them by mistake. The runner is set up to +run only these two workflows as they are on `main`, by a check on the runner +that runs before any step. That check lives outside this repository, so +nothing here can verify it; the runner is still treated as untrusted, as above +and below. Whether this +workflow's check is required before merging is a repository setting the test +cannot see. It must not be: a manual-only workflow never reports on a pull +request, so requiring it would block every merge. + +## Why it works this way + +**The run happens next to the queue, not in a GitHub job.** Submitting work to +the queue needs a credential, and a GitHub job is the wrong place to hold one +here. The runner is a persistent self-hosted machine on a public repository, +and runner labels are not an access control: any job that names the label can +run on it, including one from a branch or, after approval, a fork. Such a job +could read whatever the runner holds, or tamper with it to capture what the +next job receives. Approval gates on environments only govern jobs that name +the environment, not jobs that run next to one. Restricting a runner to a single +workflow, or making it ephemeral, needs organisation-level configuration or a +token-minting credential. So the queue credential stays with the queue side, +and the runner has no credential for the queue or the cluster, and no sudo. + +**What the report is trusted for.** The runner keeps its own registration with +GitHub between jobs, and a job on it can read every retained run's results, +including runs started by hand that were never published. That is the same trust +the self-hosted runners already carry: their nightly results are not signed +either, and the queue runs on the same infrastructure under the same access. A +job on this runner still cannot reach the queue or its credential. Each job also +receives GitHub's own short-lived runtime token, as on any runner. + +**Results move by a shared volume, not a network call.** The queue side writes +each finished run's publishable files; the runner reads them read-only. No +credential is needed between the two, and the runner's only calls ask the +queue side to start a run, how it is going, and to finish it; none carries a +file. + +**A run is asked for, not submitted, from here.** `queue-run.yml` holds no +credential and sends nothing to the queue: the runner, which sits next to the +queue side, asks it for a run of a branch, and the queue side starts it with +its own credential. It believes the request because of where it comes from, +which is why the runner is set up to run only these two workflows from `main` +(a setting outside this repository). A request can start a run of a branch +already in this repository, and that branch's lane command then runs in the +lane pods. The lane spec gives those pods no queue or cluster credential and +mounts neither the results volume nor the runner's files, so the command can +publish only what it writes to `/out`, through the scan above. What else a pod +can reach is set by the queue's own isolation, which this repository cannot +verify. + +**Only `/out` is published, and a missing run is never green.** Publishing by +origin is the main control on what reaches public results, with the scan +behind it. An unfinished run fails the job, and a job only ever publishes the +run it started or was given, so a nightly that stops running cannot keep +reporting an old pass: the missing night shows as a missing scheduled job. + +**Nothing public names the queue's infrastructure.** Runner names, labels, +paths, branches and this page use neutral names, per +[AGENTS.md](../AGENTS.md)'s rules for public content. diff --git a/scripts/run-e2e-lane.sh b/scripts/run-e2e-lane.sh new file mode 100755 index 0000000000..eca0423be4 --- /dev/null +++ b/scripts/run-e2e-lane.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# Copyright © Advanced Micro Devices, Inc., or its affiliates. +# +# SPDX-License-Identifier: MIT + +# One lane of the nightly on the shared GPU queue: the full E2E suite, nightly +# scenarios included, on the one GPU the lane was given, once for each ROCm +# channel. +# +# It mirrors the self-hosted nightly jobs (`e2e-gpu-nightly` in nightly.yml, a +# matrix over the release and nightly channels) from a cold start: the lane is a +# fresh container holding a checkout of the commit under test, with no +# toolchain, build or runtime cached from an earlier run. The build is shared; +# each channel gets its own runtimes, its own suite run, and its own directory +# under the lane's out, `/results`. Both channels always run, so one +# failing does not hide the other; the lane fails if either did. The queue side +# runs this from the checkout's root. What a lane must and must not write is in +# docs/nightly-on-the-queue.md. +# +# Inputs, all optional: +# E2E_CHANNELS the ROCm channels to run, in order: release, nightly, or +# both (default "release nightly") +# E2E_LANE_WORK where the build and runtimes go (default /var/tmp/e2e-lane) +# LANE_OUT where reported files go (default /out) + +set -euo pipefail + +cd "$(dirname "$0")/.." +out="${LANE_OUT:-/out}" +work="${E2E_LANE_WORK:-/var/tmp/e2e-lane}" +channels="${E2E_CHANNELS:-release nightly}" + +export E2E_INCLUDE_NIGHTLY="${E2E_INCLUDE_NIGHTLY-1}" +export E2E_SERVE_TIMEOUT_SECS="${E2E_SERVE_TIMEOUT_SECS:-300}" + +# The channels are checked before anything is built, so a typo costs nothing. +# Each at most once: a repeat would rerun into the same pre-warm tree and +# overwrite that channel's results. +seen=" " +for channel in ${channels}; do + case "${channel}" in + release | nightly) ;; + *) + echo "E2E_CHANNELS: unknown channel '${channel}' (release or nightly)" >&2 + exit 2 + ;; + esac + if [[ "${seen}" == *" ${channel} "* ]]; then + echo "E2E_CHANNELS: channel '${channel}' is named twice" >&2 + exit 2 + fi + seen+="${channel} " +done +# Words, not characters: a tab or a newline alone is no channel either. +# shellcheck disable=SC2086 +set -- ${channels} +if [[ "$#" -eq 0 ]]; then + echo "E2E_CHANNELS names no channel" >&2 + exit 2 +fi + +# After each channel's run, so a failing suite still reports which scenarios +# failed. The lane's status stays the suite's: a failed copy only warns. +save_results() { + local channel="$1" + if [[ -d tests/e2e-cucumber/results ]]; then + mkdir -p "${out}/${channel}/results" && + cp -R tests/e2e-cucumber/results/. "${out}/${channel}/results/" || + echo "warning: could not copy the E2E results to ${out}/${channel}" >&2 + fi +} + +# One tree on one filesystem: uv only hardlinks out of its cache into a runtime +# on the same mount, and copies every file otherwise (see e2e-selfhosted.yml). +mkdir -p "${work}" +export CARGO_TARGET_DIR="${work}/target" +export E2E_SHARED_CACHE_DIR="${work}/shared" +export E2E_SHARED_UV_CACHE_DIR="${work}/uv-cache" +# What a cold Ubuntu build needs, as the WSL lane in e2e-selfhosted.yml +# installs it; the ROCm dev images carry a compiler but not these. The update is +# retried as there: a mirror mid-sync fails it for a few minutes. +if ! command -v pkg-config >/dev/null || [[ ! -e /usr/include/sys/capability.h ]]; then + for attempt in 1 2 3 4; do + if apt-get update -qq; then + break + fi + if [[ "${attempt}" -eq 4 ]]; then + echo "apt-get update failed after ${attempt} attempts" >&2 + exit 1 + fi + echo "apt-get update failed (attempt ${attempt}); retrying in 60s" >&2 + sleep 60 + done + DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \ + pkg-config build-essential libcap-dev >/dev/null +fi + +if ! command -v cargo >/dev/null; then + # A pinned rustup-init, checked against the SHA-256 recorded here, not a + # script piped into a shell. To move to a newer rustup, replace the version + # and both sums with those published beside its rustup-init. + rustup_version="1.29.3" + case "$(uname -m)" in + x86_64) + rustup_triple="x86_64-unknown-linux-gnu" + rustup_sha256="b1c8cb14e30889e7e2999e6b69e62d9e9b9ffc0f4df8180564d306d9680e6c55" + ;; + aarch64 | arm64) + rustup_triple="aarch64-unknown-linux-gnu" + rustup_sha256="9f608c142e0e38a78957d0661ba1dfaf0b6fb37184cfaa4b9a0aa57af6184282" + ;; + *) + echo "no pinned rustup-init for $(uname -m)" >&2 + exit 1 + ;; + esac + rustup_init="${work}/rustup-init" + curl --proto '=https' --tlsv1.2 -sSf -o "${rustup_init}" \ + "https://static.rust-lang.org/rustup/archive/${rustup_version}/${rustup_triple}/rustup-init" + if ! echo "${rustup_sha256} ${rustup_init}" | sha256sum -c --quiet -; then + echo "rustup-init ${rustup_version} does not match its recorded SHA-256" >&2 + exit 1 + fi + chmod +x "${rustup_init}" + # No default toolchain: rust-toolchain.toml pins it, and the first cargo + # call in this checkout installs that one. + "${rustup_init}" -y --profile minimal --default-toolchain none + # shellcheck source=/dev/null + . "${HOME}/.cargo/env" +fi + +bash .github/scripts/gpu-preflight.sh + +# The e2e-test-hooks feature must match what `cargo xtask e2e` builds for +# itself; see the e2e-gpu job in e2e-selfhosted.yml. +cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks +export ROCM_CLI_BINARY="${CARGO_TARGET_DIR}/release/rocm" +export ROCM_CLI_ROCMD_BINARY="${CARGO_TARGET_DIR}/release/rocmd" + +# One channel: its runtimes, installed in place, never moved (install sdk bakes +# absolute paths into the runtime manifest; see `xtask e2e-prewarm`), in a tree +# of its own so the two channels' runtimes never meet; then the full suite on +# the build. A failure is returned, not fatal, so the next channel still runs. +run_channel() { + local channel="$1" + local prewarm="${work}/prewarm-${channel}" + export E2E_SHARED_RUNTIMES_DIR="${prewarm}/data/runtimes" + # The suite writes here, and a channel must not report the last one's files. + # Returned, not ignored: the caller's `||` turns `set -e` off in here. + rm -rf tests/e2e-cucumber/results || return $? + HF_HOME="${E2E_SHARED_CACHE_DIR}/huggingface" \ + UV_CACHE_DIR="${E2E_SHARED_UV_CACHE_DIR}" \ + cargo xtask e2e-prewarm --channel "${channel}" --prewarm-dir "${prewarm}" || + return $? + cargo xtask e2e +} + +status=0 +for channel in ${channels}; do + echo "== channel ${channel}" + rc=0 + run_channel "${channel}" || rc=$? + save_results "${channel}" + if [[ "${rc}" -ne 0 ]]; then + echo "channel ${channel} failed (status ${rc})" >&2 + [[ "${status}" -ne 0 ]] || status="${rc}" + fi +done +exit "${status}" diff --git a/xtask/src/workflow_contract.rs b/xtask/src/workflow_contract.rs index f3d8ce9a96..7d3f824700 100644 --- a/xtask/src/workflow_contract.rs +++ b/xtask/src/workflow_contract.rs @@ -1654,7 +1654,9 @@ jobs: } } - /// Both self-hosted workflows, so a lane added to either is covered. + /// Both self-hosted workflows that run GPU lanes, so a lane added to either is + /// covered. `nightly-queue.yml` and `queue-run.yml` also run self-hosted but + /// run no GPU lane; `assert_queue_workflow_holds_no_credential` covers them. fn self_hosted_workflows() -> [(&'static str, String); 2] { [ ("e2e-selfhosted.yml", read_workflow("e2e-selfhosted.yml")), @@ -8778,4 +8780,959 @@ permissions: ["cargo build"] ); } + + /// The workflows that may run on the queue-report runner. + const QUEUE_WORKFLOWS: [&str; 2] = ["nightly-queue.yml", "queue-run.yml"]; + + #[test] + fn queue_nightly_publisher_holds_no_credential() { + assert_queue_workflow_holds_no_credential( + "nightly-queue.yml", + &["${{inputs.request_id}}", "${{runner.temp}}"], + &["workflow_dispatch:"], + ); + } + + #[test] + fn queue_run_holds_no_credential() { + assert_queue_workflow_holds_no_credential( + "queue-run.yml", + &["${{inputs.branch}}", "${{runner.temp}}"], + &["workflow_dispatch:", "schedule:"], + ); + } + + /// Whether `line` is a schedule entry, `- cron: "..."`, holding nothing + /// but a cron expression's characters: the one place a queue workflow + /// needs `*`, which no anchor or alias can hide in. + fn is_cron_line(line: &str) -> bool { + line.trim() + .strip_prefix("- cron: \"") + .and_then(|rest| rest.strip_suffix('"')) + .is_some_and(|cron| { + !cron.is_empty() + && cron + .chars() + .all(|c| c.is_ascii_digit() || matches!(c, ' ' | '*' | '/' | ',' | '-')) + }) + } + + /// The checks every workflow on the queue-report runner must pass: `file` + /// may evaluate only `expressions` and be triggered only by `triggers`. + fn assert_queue_workflow_holds_no_credential( + file: &str, + expressions: &[&str], + triggers: &[&str], + ) { + // The queue-report runner is persistent, and any job that names its label + // can run on it, so the publishing job must not bring a credential onto + // it. A credential reaches a job through an expression, an action that + // takes the job token by default, an environment, a permission, or a + // runtime variable, so this is an allowlist over all of those rather than + // a list of known spellings where one is possible: expressions and actions + // are allowlisted, permissions must be empty, and environments and + // runtime-token variables are refused outright. + let text = read_workflow(file); + + // The checks below read text, not parsed YAML, so first rule out the YAML + // that decodes to something other than it reads: escape sequences (only + // double-quoted scalars have them), explicit `? key` entries, and anchors + // and aliases (`&a` / `*a`), which can stand in for a key. A plain + // workflow needs none of them. + assert!( + !text.contains('\\'), + "{file} contains a backslash; a YAML escape could spell a forbidden \ + key or expression past these checks" + ); + assert!( + !text.lines().any(|line| { + let line = line.trim_start().trim_start_matches('-').trim_start(); + line == "?" || line.starts_with("? ") + }), + "{file} uses an explicit `? key` entry, which these checks cannot read" + ); + let uncommented = without_comments(&text); + assert!( + !uncommented + .lines() + .filter(|line| !is_cron_line(line)) + .any(|line| line.contains(['&', '*'])), + "{file} contains `&` or `*`, banned outright here because a YAML anchor \ + or alias could stand in for a key these checks read" + ); + + // Expressions and keys can be written in any case, spacing or quoting. + // Lines keep their breaks; comments are kept too, because GitHub + // substitutes expressions inside a `run:` script, shell comments included. + let flatten = |text: &str| -> String { + text.to_lowercase() + .chars() + .filter(|c| *c == '\n' || !(c.is_whitespace() || *c == '"' || *c == '\'')) + .collect() + }; + let flat = flatten(&text); + let found_expressions: Vec<&str> = flat + .match_indices("${{") + .map(|(start, _)| { + let end = flat[start..] + .find("}}") + .unwrap_or_else(|| panic!("every ${{{{ in {file} is closed")); + &flat[start..start + end + 2] + }) + .collect(); + for expression in &found_expressions { + assert!( + expressions.contains(expression), + "{file} evaluates `{expression}`: only {expressions:?} may be read, so no \ + secret or token can reach the shared runner (see docs/nightly-on-the-queue.md)" + ); + } + + // Inputs reach a shell only through `env:`: an expression inside a + // `run:` script is substituted into the script text before the shell + // reads it, so a dispatched branch name could then be code. A script's + // comments count, as above. + let lines: Vec<&str> = text.lines().collect(); + for (at, line) in lines.iter().enumerate() { + let key = line.trim_start().trim_start_matches('-').trim_start(); + // A step's key sits at least six deep; a job may itself be called `run`. + let Some(inline) = key.strip_prefix("run:").filter(|_| indent_of(line) >= 6) else { + continue; + }; + let mut body = vec![inline]; + body.extend( + lines[at + 1..] + .iter() + .take_while(|next| next.trim().is_empty() || indent_of(next) > indent_of(line)) + .copied(), + ); + assert!( + !body.iter().any(|l| l.contains("${{")), + "{file} evaluates an expression inside a `run:` script: pass it \ + through `env:` and read the variable instead" + ); + } + + // Keys, by contrast, are only keys outside comments. + let keys = flatten(&without_comments(&text)); + for (start, _) in keys.match_indices("uses:") { + let value: String = keys[start + "uses:".len()..] + .chars() + .take_while(|c| { + c.is_ascii_alphanumeric() || matches!(c, '/' | '@' | '.' | '_' | '-') + }) + .collect(); + let pinned = value + .strip_prefix("actions/upload-artifact@") + .is_some_and(|sha| sha.len() == 40 && sha.chars().all(|c| c.is_ascii_hexdigit())); + assert!( + pinned, + "{file} uses `{value}`: only upload-artifact, pinned to a full commit \ + SHA, may run there, since another action could take the job token by default" + ); + } + for forbidden in ["environment:", "actions_runtime_token", "actions_id_token"] { + assert!( + !keys.contains(forbidden), + "{file} mentions `{forbidden}`, which would bring a credential to \ + the shared runner" + ); + } + assert_eq!( + keys.matches("permissions:").count(), + 1, + "{file} must declare permissions once, at the top, so no job widens them" + ); + assert_eq!( + top_level_block(&without_comments(&text), "permissions").trim(), + "{}", + "{file} must grant no permissions: the job only uploads files" + ); + + // The workflow itself: dispatched by hand only, routed to the reporting + // runner, uploading only the results it was given, and never hidden files. + let plain = without_comments(&text); + let on = top_level_block(&plain, "on"); + let found_triggers: Vec<&str> = on + .lines() + .filter(|line| indent_of(line) == 2) + .map(str::trim) + .collect(); + assert_eq!( + found_triggers, triggers, + "{file} may be triggered by {triggers:?} only" + ); + assert_eq!( + runs_on_values(&plain), + ["[queue-report]"], + "{file} must run on the queue-report runner, which carries that label \ + alone" + ); + let upload: Vec<&str> = plain + .lines() + .map(str::trim) + .filter(|line| line.starts_with("path:") || line.starts_with("include-hidden-files:")) + .collect(); + assert_eq!( + upload, + [ + "path: ${{ runner.temp }}/results", + "include-hidden-files: false" + ], + "{file} must upload only the published results, and no hidden files" + ); + + // The runner is registered with `queue-report` alone, without GitHub's + // default `self-hosted`, OS and architecture labels, so only a job naming + // that literal label lands on it. No other workflow on `main` may name it, + // nor name this workflow, which is how another one would wait on its + // result. This catches a mistake on `main`, not an + // attempt to hide one: a workflow on any other branch can name the label + // anyway, which is why the runner holds no credential. + let workflow_name = strip_comment(&top_level_block(&plain, "name")).to_owned(); + let dir = repo_root().join(".github/workflows"); + for entry in std::fs::read_dir(&dir).expect("reading .github/workflows") { + let name = entry.expect("workflow dir entry").file_name(); + let name = name.to_string_lossy(); + if QUEUE_WORKFLOWS.contains(&name.as_ref()) + || !(name.ends_with(".yml") || name.ends_with(".yaml")) + { + continue; + } + let found = queue_runner_references(&read_workflow(&name), &workflow_name); + assert!( + found.is_empty(), + "{name} names {found:?}: only {QUEUE_WORKFLOWS:?} may use the queue-report \ + runner, and no workflow may depend on them (see \ + docs/nightly-on-the-queue.md)" + ); + } + } + + /// What in `workflow` would route a job to the queue-report runner or make + /// it depend on the workflow named `queue_workflow_name`: the runner's + /// label, the workflow's file stem, or its name. Compared with case, quotes + /// and every whitespace character (line breaks included) removed, so a name + /// split across lines still counts; comments do not. + fn queue_runner_references(workflow: &str, queue_workflow_name: &str) -> Vec { + let squeeze = |text: &str| -> String { + text.to_lowercase() + .chars() + .filter(|c| !(c.is_whitespace() || *c == '"' || *c == '\'')) + .collect() + }; + let text = squeeze(&without_comments(workflow)); + [ + "queue-report".to_owned(), + "nightly-queue".to_owned(), + "queue-run".to_owned(), + squeeze(queue_workflow_name), + ] + .into_iter() + .filter(|reference| text.contains(reference.as_str())) + .collect() + } + + #[test] + fn queue_runner_references_catches_each_way_of_naming_the_queue() { + let name = r#""Nightly E2E (queue)""#; + let job = |body: &str| format!("jobs:\n build:\n{body}\n"); + for (workflow, expected) in [ + (job(" runs-on: [queue-report]"), vec!["queue-report"]), + ( + job(" runs-on: [self-hosted, Queue-Report]"), + vec!["queue-report"], + ), + (job(" runs-on: \"queue-report\""), vec!["queue-report"]), + ( + "on:\n workflow_run:\n workflows: [\"Nightly E2E (queue)\"]\n".to_owned(), + vec!["nightlye2e(queue)"], + ), + ( + "on:\n workflow_run:\n workflows:\n - Nightly E2E\n (queue)\n" + .to_owned(), + vec!["nightlye2e(queue)"], + ), + ( + job(" uses: ./.github/workflows/nightly-queue.yml"), + vec!["nightly-queue"], + ), + ( + job(" uses: ./.github/workflows/queue-run.yml"), + vec!["queue-run"], + ), + (job(" runs-on: ubuntu-latest"), vec![]), + (job(" runs-on: [self-hosted, linux, mi300x]"), vec![]), + ( + job(" runs-on: ubuntu-latest\n # moved off queue-report and nightly-queue"), + vec![], + ), + ] { + assert_eq!( + queue_runner_references(&workflow, name), + expected, + "references found in:\n{workflow}" + ); + } + } + + /// The block a line of `file` opens: that line, and every line after it + /// indented deeper, blank lines included, up to the first that is not. So a + /// step's or an input's own keys, and nothing of the next one. + fn yaml_block(file: &str, header: &str) -> String { + let text = read_workflow(file); + let indent = |line: &str| line.len() - line.trim_start().len(); + let mut lines = text.lines().skip_while(|line| *line != header); + let first = lines + .next() + .unwrap_or_else(|| panic!("{file} has a line `{header}`")); + let mut block = vec![first]; + block.extend( + lines.take_while(|line| line.trim().is_empty() || indent(line) > indent(first)), + ); + block.join("\n").trim_end().to_owned() + "\n" + } + + /// What every queue workflow must keep, beside holding no credential: a + /// failed run fails the job, a run is never cancelled for a newer one, and + /// the lanes' files are uploaded whatever the verdict. + #[test] + fn each_queue_workflow_fails_when_its_run_does_and_cancels_nothing() { + for (file, group) in [ + ("queue-run.yml", "queue-run"), + ("nightly-queue.yml", "nightly-queue"), + ] { + let text = read_workflow(file); + // A publisher that may fail without failing the job reports a + // failed run as green. + assert!( + !without_comments(&text).contains("continue-on-error"), + "{file} must not let a step fail quietly" + ); + // Cancelling a waiting job for a newer one is GitHub's to do; a + // running one holds the runner for the run it started. + assert_eq!( + yaml_block(file, "concurrency:"), + format!("concurrency:\n group: {group}\n cancel-in-progress: false\n"), + "{file}" + ); + } + } + + #[test] + // `${BRANCH:-main}` is the step's shell, not a format argument. + #[allow(clippy::literal_string_with_formatting_args)] + fn queue_run_starts_the_dispatched_branch_and_publishes_it() { + let step = yaml_block( + "queue-run.yml", + " - name: Run on the queue and publish", + ); + // Empty when scheduled, hence the default. + assert_eq!( + step, + " - name: Run on the queue and publish\n\ + \x20 env:\n\ + \x20 # Empty when scheduled.\n\ + \x20 BRANCH: ${{ inputs.branch }}\n\ + \x20 # One line: the workflow contract forbids backslashes in this file.\n\ + \x20 run: /opt/queue/bin/queue-publish --out \"${RUNNER_TEMP}/results\" \ + --start \"${BRANCH:-main}\"\n", + "queue-run.yml must start a run of the dispatched branch and publish it" + ); + assert_eq!( + yaml_block("queue-run.yml", " schedule:"), + " schedule:\n - cron: \"17 2 * * *\"\n" + ); + assert!( + docs_nightly_on_the_queue().contains("every night at 02:17 UTC"), + "docs/nightly-on-the-queue.md must give the schedule queue-run.yml has" + ); + } + + #[test] + fn nightly_queue_publishes_only_the_named_run() { + let step = yaml_block("nightly-queue.yml", " - name: Publish the run"); + assert_eq!( + step, + " - name: Publish the run\n\ + \x20 env:\n\ + \x20 REQUEST_ID: ${{ inputs.request_id }}\n\ + \x20 # One line: the workflow contract forbids backslashes in this file.\n\ + \x20 run: /opt/queue/bin/queue-publish --out \"${RUNNER_TEMP}/results\" \ + --request \"${REQUEST_ID}\"\n", + "nightly-queue.yml must publish the run it was given" + ); + // No default: a run that did not happen cannot be published in its place. + assert_eq!( + yaml_block("nightly-queue.yml", " request_id:"), + " request_id:\n\ + \x20 description: \"The id of the finished run to publish\"\n\ + \x20 type: string\n\ + \x20 required: true\n" + ); + } + + fn docs_nightly_on_the_queue() -> String { + let p = repo_root().join("docs/nightly-on-the-queue.md"); + std::fs::read_to_string(&p).unwrap_or_else(|e| panic!("reading {}: {e}", p.display())) + } + + /// The queue covers the Linux GPU nightly on both channels and not native + /// Windows or WSL; the doc must say both, plainly, and say no more than the + /// lane script does about the channels. + #[test] + fn the_queue_doc_states_its_channels_and_its_remaining_gap() { + let doc = docs_nightly_on_the_queue(); + let flat = doc.split_whitespace().collect::>().join(" "); + for needed in [ + "both the `release` and the `nightly` channel", + "The remaining gap is native Windows and WSL.", + "the native Windows and WSL lanes stay on their dedicated runners", + "`E2E_CHANNELS`", + "`/out//results`", + ] { + assert!( + flat.contains(needed), + "docs/nightly-on-the-queue.md must say: {needed}" + ); + } + let script = std::fs::read_to_string(repo_root().join("scripts/run-e2e-lane.sh")) + .expect("reading the lane script"); + assert!( + script.contains("E2E_CHANNELS:-release nightly"), + "the lane script must default to both channels, as the doc says" + ); + } + + /// The job's time limit, the lanes' wait limit and what the doc and the + /// workflow comment say of them move together: the limit must leave the + /// 30 minutes finishing and publishing a run can take after the longest wait. + #[test] + fn the_queue_run_timeout_covers_the_wait_limit_the_doc_gives() { + let doc = docs_nightly_on_the_queue(); + let flat = doc.split_whitespace().collect::>().join(" "); + // The limit as the run inputs give it: "at most N seconds". + let wait_secs: u64 = flat + .split("at most ") + .find_map(|rest| rest.split(" seconds").next()?.parse().ok()) + .expect("the doc's run inputs give the wait limit as `at most N seconds`"); + assert!( + flat.contains(&format!("({wait_secs} s")), + "the doc's start section must give the same wait limit" + ); + let wait_text = format!("{} h {} min", wait_secs / 3600, wait_secs % 3600 / 60); + assert!(flat.contains(&wait_text), "the doc must say {wait_text}"); + let text = read_workflow("queue-run.yml"); + assert!( + text.contains(&format!("{wait_secs} s ({wait_text})")), + "queue-run.yml's comment must give the wait as {wait_secs} s ({wait_text})" + ); + let timeout_minutes: u64 = text + .lines() + .find_map(|line| line.strip_prefix(" timeout-minutes: ")) + .expect("queue-run.yml's job has timeout-minutes") + .trim() + .parse() + .expect("timeout-minutes is a number"); + assert!( + timeout_minutes * 60 >= wait_secs + 30 * 60, + "timeout-minutes: {timeout_minutes} leaves less than 30 minutes after a {wait_secs} s wait" + ); + } + + #[test] + fn each_queue_workflow_uploads_the_artifact_the_doc_names_for_it() { + let doc = docs_nightly_on_the_queue(); + for (file, artifact) in [ + ("queue-run.yml", "queue-run-results"), + ("nightly-queue.yml", "nightly-queue-results"), + ] { + let upload = yaml_block(file, " - name: Upload lane results"); + // Uploaded whatever the run's verdict: a failed run's files are + // what someone needs. + assert!( + upload.contains("\n if: always()\n"), + "{file}:\n{upload}" + ); + assert!( + upload.contains(&format!("\n name: {artifact}\n")), + "{file} must upload {artifact}:\n{upload}" + ); + assert!( + doc.contains(&format!("`{artifact}`\n from `{file}`")) + || doc.contains(&format!("`{artifact}` from `{file}`")), + "docs/nightly-on-the-queue.md must say {file} uploads `{artifact}`" + ); + } + } + + /// What `scripts/run-e2e-lane.sh` needs from a lane image, linked into a + /// directory that becomes the whole `PATH`, so a test decides what is + /// missing. + #[cfg(unix)] + const LANE_TOOLS: [&str; 7] = ["bash", "dirname", "mkdir", "cp", "cat", "rm", "chmod"]; + + /// One run of the lane script against stubbed tools: how often + /// `apt-get update` fails first, what the suite exits with (on the nightly + /// channel too, unless `nightly_rc` says otherwise), the channels asked for, + /// whether cargo is already installed, and whether the out directory cannot + /// be written. + #[cfg(unix)] + struct LaneCase { + apt_failures: u32, + suite_rc: i32, + nightly_rc: Option, + channels: Option<&'static str>, + /// What `uname -m` says, and whether the downloaded rustup-init + /// matches the sum the script records. + machine: &'static str, + checksum_ok: bool, + cargo_installed: bool, + out_blocked: bool, + } + + #[cfg(unix)] + impl Default for LaneCase { + fn default() -> Self { + Self { + apt_failures: 0, + suite_rc: 0, + nightly_rc: None, + channels: None, + machine: "x86_64", + checksum_ok: true, + cargo_installed: true, + out_blocked: false, + } + } + } + + /// What a run of the lane script left: its exit code, its log, the out + /// directory, and the scratch directory holding the stubs' records. + #[cfg(unix)] + struct LaneRun { + rc: i32, + log: String, + out: PathBuf, + dir: tempfile::TempDir, + } + + /// Run a copy of the lane script in a scratch checkout with every tool it + /// calls stubbed. Each stub logs how it was called: the preflight, `sleep`, + /// `curl` (whose output is a stand-in for rustup's installer), `apt-get` (to + /// a file), and cargo, which also logs the environment the pre-warm and the + /// suite get; the suite leaves a result file and exits `suite_rc`. Without + /// `cargo_installed`, cargo is reachable only once that installer has run. + #[cfg(unix)] + fn run_lane(case: &LaneCase) -> LaneRun { + use rocm_core::fake_executable::write_fake_executable; + + let dir = tempfile::tempdir().expect("temp dir"); + let root = dir.path().join("checkout"); + let tools = dir.path().join("tools"); + let out = dir.path().join("out"); + for sub in ["scripts", ".github/scripts"] { + std::fs::create_dir_all(root.join(sub)).expect("scratch checkout"); + } + std::fs::create_dir(&tools).expect("tools dir"); + std::fs::copy( + repo_root().join("scripts/run-e2e-lane.sh"), + root.join("scripts/run-e2e-lane.sh"), + ) + .expect("copy the lane script"); + std::fs::write( + root.join(".github/scripts/gpu-preflight.sh"), + "echo \"preflight ran\"\n", + ) + .expect("stub the preflight"); + if case.out_blocked { + std::fs::write(&out, "a file where the out directory goes\n").expect("block out"); + } + let host_tool = |tool: &str| { + ["/usr/bin", "/bin"] + .iter() + .map(|d| Path::new(d).join(tool)) + .find(|p| p.exists()) + .unwrap_or_else(|| panic!("{tool} on this host")) + }; + for tool in LANE_TOOLS.iter().chain(&["sh"]) { + std::os::unix::fs::symlink(host_tool(tool), tools.join(tool)).expect("link a tool"); + } + let calls = dir.path().join("apt-calls"); + let updates = dir.path().join("apt-updates"); + write_fake_executable( + &tools.join("apt-get"), + format!( + "#!/bin/bash\necho \"apt-get $*\" >> {calls}\n\ + [ \"$1\" = update ] || exit 0\n\ + n=0; [ -f {updates} ] && read -r n < {updates}\n\ + n=$((n + 1)); echo \"$n\" > {updates}\n\ + [ \"$n\" -gt {failures} ]\n", + calls = calls.display(), + updates = updates.display(), + failures = case.apt_failures, + ), + ) + .expect("stub apt-get"); + write_fake_executable(&tools.join("sleep"), "#!/bin/sh\necho \"sleep $*\"\n") + .expect("stub sleep"); + // Where cargo is: on PATH, or where the rustup installer puts it. + let home = dir.path().join("home"); + let cargo_bin = if case.cargo_installed { + tools.clone() + } else { + home.join(".cargo/bin") + }; + std::fs::create_dir_all(&cargo_bin).expect("cargo's directory"); + // What rustup's installer would do: install cargo, and write the env + // file the script sources. It logs the arguments the script gave it. + write_fake_executable( + &tools.join("curl"), + format!( + "#!/bin/bash\necho \"curl $*\" >&2\n\ + while [ $# -gt 0 ]; do [ \"$1\" = -o ] && {{ dest=$2; shift; }}; shift; done\n\ + printf '%s\\n' '#!/bin/bash' \ + 'echo \"rustup-init $*\"; mkdir -p \"$HOME/.cargo\"; \ + echo \"export PATH={bin}:\\$PATH\" > \"$HOME/.cargo/env\"' > \"$dest\"\n", + bin = home.join(".cargo/bin").display() + ), + ) + .expect("stub curl"); + write_fake_executable( + &tools.join("uname"), + format!("#!/bin/sh\necho {}\n", case.machine), + ) + .expect("stub uname"); + // Logs the line it is asked to check, which holds the recorded sum. + write_fake_executable( + &tools.join("sha256sum"), + format!( + "#!/bin/bash\nread -r line\necho \"sha256sum $* <<< $line\" >&2\n{}\n", + if case.checksum_ok { "exit 0" } else { "exit 1" } + ), + ) + .expect("stub sha256sum"); + write_fake_executable( + &cargo_bin.join("cargo"), + format!( + "#!/bin/bash\necho \"cargo $*\"\n\ + case \"$*\" in\n\ + \"xtask e2e-prewarm\"*) echo \"prewarm HF_HOME=$HF_HOME UV_CACHE_DIR=$UV_CACHE_DIR\";;\n\ + \"xtask e2e\")\n\ + echo \"suite E2E_INCLUDE_NIGHTLY=${{E2E_INCLUDE_NIGHTLY-unset}} \ + ROCM_CLI_BINARY=$ROCM_CLI_BINARY\"\n\ + echo \"suite runtimes=$E2E_SHARED_RUNTIMES_DIR\"\n\ + [ -d tests/e2e-cucumber/results ] && echo \"stale results\"\n\ + mkdir -p tests/e2e-cucumber/results\n\ + echo \"suite $E2E_SHARED_RUNTIMES_DIR\" > tests/e2e-cucumber/results/junit.xml\n\ + case \"$E2E_SHARED_RUNTIMES_DIR\" in\n\ + *prewarm-nightly*) exit {nightly};;\n\ + *) exit {rc};;\n\ + esac;;\n\ + esac\n", + rc = case.suite_rc, + nightly = case.nightly_rc.unwrap_or(case.suite_rc) + ), + ) + .expect("stub cargo"); + + let mut lane = std::process::Command::new(tools.join("bash")); + lane.arg(root.join("scripts/run-e2e-lane.sh")) + .env_clear() + .env("PATH", &tools) + .env("HOME", &home) + .env("LANE_OUT", &out) + .env("E2E_LANE_WORK", dir.path().join("work")); + if let Some(channels) = case.channels { + lane.env("E2E_CHANNELS", channels); + } + let output = lane.output().expect("running the lane script"); + let mut log = String::from_utf8_lossy(&output.stdout).into_owned(); + log.push_str(&String::from_utf8_lossy(&output.stderr)); + LaneRun { + rc: exit_code(output.status, "lane"), + log, + out, + dir, + } + } + + #[cfg(unix)] + #[test] + fn a_lane_runs_the_nightly_job_and_exits_with_the_suites_status() { + for suite_rc in [0, 1, 3] { + let LaneRun { rc, log, out, dir } = run_lane(&LaneCase { + suite_rc, + ..LaneCase::default() + }); + let work = dir.path().join("work"); + assert_eq!(rc, suite_rc, "{log}"); + // Each channel's results in a directory of its own, and each the + // run of its own runtimes: nothing of one in the other. + for channel in ["release", "nightly"] { + assert_eq!( + std::fs::read_to_string(out.join(channel).join("results/junit.xml")) + .ok() + .as_deref(), + Some( + format!("suite {}/prewarm-{channel}/data/runtimes\n", work.display()) + .as_str() + ), + "the {channel} suite's results must reach the lane's out directory:\n{log}" + ); + } + assert!( + !out.join("results").exists(), + "results sit under their channel:\n{log}" + ); + assert!(!log.contains("stale results"), "{log}"); + // In this order, as the nightly jobs run them: one build, then + // each channel's pre-warm and full suite, nightly scenarios + // included. + let mut steps = vec![ + "preflight ran\n".to_owned(), + "cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks\n" + .to_owned(), + ]; + for channel in ["release", "nightly"] { + steps.extend([ + format!( + "cargo xtask e2e-prewarm --channel {channel} --prewarm-dir {}/prewarm-{channel}\n", + work.display() + ), + // The pre-warm's caches on the same mount as the runtimes. + format!( + "prewarm HF_HOME={w}/shared/huggingface UV_CACHE_DIR={w}/uv-cache\n", + w = work.display() + ), + format!( + "suite E2E_INCLUDE_NIGHTLY=1 ROCM_CLI_BINARY={}/target/release/rocm\n", + work.display() + ), + ]); + } + let mut from = 0; + for step in &steps { + let at = log[from..] + .find(step.as_str()) + .unwrap_or_else(|| panic!("`{step}` missing or out of order in:\n{log}")); + from += at + step.len(); + } + assert_eq!(log.matches("cargo build").count(), 1, "{log}"); + assert!(!log.contains("curl"), "cargo was there already:\n{log}"); + } + } + + #[cfg(unix)] + #[test] + fn a_failing_channel_does_not_hide_the_other() { + // Nightly fails, release passes: both ran and reported, and the lane + // is red. + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + nightly_rc: Some(3), + ..LaneCase::default() + }); + assert_eq!(rc, 3, "{log}"); + assert!(out.join("release/results/junit.xml").is_file(), "{log}"); + assert!(out.join("nightly/results/junit.xml").is_file(), "{log}"); + assert!(log.contains("channel nightly failed (status 3)"), "{log}"); + assert!(!log.contains("channel release failed"), "{log}"); + + // Release fails, nightly passes: the nightly channel still runs, and + // the lane's status is the first failure's. + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + suite_rc: 1, + nightly_rc: Some(0), + ..LaneCase::default() + }); + assert_eq!(rc, 1, "{log}"); + assert!(out.join("nightly/results/junit.xml").is_file(), "{log}"); + assert!(log.contains("channel release failed (status 1)"), "{log}"); + } + + #[cfg(unix)] + #[test] + fn a_lane_runs_only_the_channels_it_is_asked_for_and_refuses_others() { + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + channels: Some("nightly"), + ..LaneCase::default() + }); + assert_eq!(rc, 0, "{log}"); + assert!(out.join("nightly/results/junit.xml").is_file(), "{log}"); + assert!(!out.join("release").exists(), "{log}"); + assert!(!log.contains("--channel release"), "{log}"); + + for channels in [ + "beta", + "release stable", + " ", + "\t", + "\n", + " \t\n ", + "release release", + "nightly release\tnightly", + ] { + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + channels: Some(channels), + ..LaneCase::default() + }); + assert_eq!(rc, 2, "{channels:?}: {log}"); + assert!( + !log.contains("cargo"), + "nothing is built for {channels:?}:\n{log}" + ); + assert!(!out.exists(), "{channels:?}: {log}"); + } + } + + #[cfg(unix)] + #[test] + fn a_lane_that_cannot_copy_its_results_keeps_the_suites_status() { + for suite_rc in [0, 1] { + let LaneRun { rc, log, .. } = run_lane(&LaneCase { + suite_rc, + out_blocked: true, + ..LaneCase::default() + }); + assert_eq!(rc, suite_rc, "{log}"); + assert!( + log.contains("warning: could not copy the E2E results to"), + "{log}" + ); + } + } + + #[cfg(unix)] + #[test] + fn a_lane_without_cargo_installs_rustup_and_leaves_the_toolchain_to_the_checkout() { + // Bound, not `..`: the scratch directory goes when `_dir` does. + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + cargo_installed: false, + ..LaneCase::default() + }); + assert_eq!(rc, 0, "{log}"); + // A pinned rustup-init over TLS, not a script piped into a shell, and + // checked against the sum recorded for this machine before it runs. + assert!( + log.contains( + "curl --proto =https --tlsv1.2 -sSf -o " + ) && log.contains( + "https://static.rust-lang.org/rustup/archive/1.29.3/x86_64-unknown-linux-gnu/rustup-init" + ), + "{log}" + ); + assert!(!log.contains("sh.rustup.rs"), "{log}"); + assert!( + log.contains("b1c8cb14e30889e7e2999e6b69e62d9e9b9ffc0f4df8180564d306d9680e6c55 "), + "{log}" + ); + // No default toolchain: rust-toolchain.toml picks it. + assert!( + log.contains("rustup-init -y --profile minimal --default-toolchain none"), + "{log}" + ); + assert!(log.contains("cargo xtask e2e\n"), "{log}"); + assert!(out.join("release/results/junit.xml").is_file(), "{log}"); + } + + #[cfg(unix)] + #[test] + fn a_rustup_init_that_does_not_match_its_sum_is_never_run() { + let LaneRun { rc, log, .. } = run_lane(&LaneCase { + cargo_installed: false, + checksum_ok: false, + ..LaneCase::default() + }); + assert_eq!(rc, 1, "{log}"); + assert!(log.contains("does not match its recorded SHA-256"), "{log}"); + assert!(!log.contains("rustup-init -y"), "{log}"); + assert!(!log.contains("cargo"), "nothing is built:\n{log}"); + } + + #[cfg(unix)] + #[test] + fn the_rustup_pin_follows_the_machine() { + let LaneRun { rc, log, .. } = run_lane(&LaneCase { + cargo_installed: false, + machine: "aarch64", + ..LaneCase::default() + }); + assert_eq!(rc, 0, "{log}"); + assert!( + log.contains("1.29.3/aarch64-unknown-linux-gnu/rustup-init"), + "{log}" + ); + assert!( + log.contains("9f608c142e0e38a78957d0661ba1dfaf0b6fb37184cfaa4b9a0aa57af6184282 "), + "{log}" + ); + + let LaneRun { rc, log, .. } = run_lane(&LaneCase { + cargo_installed: false, + machine: "riscv64", + ..LaneCase::default() + }); + assert_eq!(rc, 1, "{log}"); + assert!(log.contains("no pinned rustup-init for riscv64"), "{log}"); + } + + #[cfg(unix)] + #[test] + fn a_lane_rides_out_a_mirror_mid_sync() { + let LaneRun { rc, log, dir, .. } = run_lane(&LaneCase { + apt_failures: 3, + ..LaneCase::default() + }); + assert_eq!(rc, 0, "{log}"); + let calls = std::fs::read_to_string(dir.path().join("apt-calls")).expect("apt-get ran"); + assert_eq!(calls.matches("apt-get update").count(), 4, "{calls}"); + assert!(calls.contains("install"), "{calls}"); + // Spaced out: a mirror's sync takes minutes. + assert_eq!(log.matches("sleep 60\n").count(), 3, "{log}"); + + let LaneRun { + rc, + log, + out, + dir: _dir, + } = run_lane(&LaneCase { + apt_failures: 4, + ..LaneCase::default() + }); + assert_eq!(rc, 1, "{log}"); + assert!( + log.contains("apt-get update failed after 4 attempts"), + "{log}" + ); + assert!( + !log.contains("cargo"), + "nothing is built after apt-get gave up:\n{log}" + ); + assert!(!out.exists()); + } } From 3214bcc6d11e39c781254bc88bffbe5311bae2e1 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 19:21:54 +0000 Subject: [PATCH 033/102] docs(dash): correct stale module comments in the dashboard crates (#978) * docs(dash): correct stale module comments in the dashboard crates Module-level comments in the dashboard crates described things that are not true of the code or the repository. None affects behavior. - Drop the pointers to a wiki/ directory the repository does not have (14 files), keeping the one sentence in each that still carries information. - demo.rs and the two examples named rocm gen-demo, rocm demo, rocm replay and make demo, none of which exist, and default paths under marketing/ and app/. They now name rocm dash --demo / --replay and the session file format. - bench_rollup, rocm-deps, the collectors Cargo.toml, parallel.rs, lib.rs, model_picker, chat, pane and home described a key, a consumer set, a dependency edge, a call path, a tab default or a phase that no longer matches the code. - Remove "Phase N Wave N" labels from the titles of the dash TUI source module headers, together with the sentences that leaned on them, and the pointer to a working-agreements document that is not in the repository. Comment-only change. Fixes #972 Signed-off-by: Jussi Elo * docs(dash): rewrap the serve wizard module header The rewritten header line ran to about 120 columns; rewrap it to match the surrounding `//!` lines. Signed-off-by: Jussi Elo --------- Signed-off-by: Jussi Elo (cherry picked from commit 7da776cb404f7cf20037c39676a3ef27f58092ce) Signed-off-by: juhovainio --- crates/rocm-dash-collectors/Cargo.toml | 4 ++-- crates/rocm-dash-collectors/src/amd_smi.rs | 2 +- crates/rocm-dash-collectors/src/docker.rs | 1 - crates/rocm-dash-collectors/src/parallel.rs | 7 +++---- crates/rocm-dash-collectors/src/vllm_prom.rs | 2 +- crates/rocm-dash-core/src/bench_rollup.rs | 8 ++++---- crates/rocm-dash-core/src/bench_schema.rs | 1 - crates/rocm-dash-core/src/efficiency.rs | 2 +- crates/rocm-dash-core/src/lib.rs | 3 ++- crates/rocm-dash-core/src/metrics.rs | 1 - crates/rocm-dash-core/src/partition.rs | 2 +- crates/rocm-dash-core/src/protocol.rs | 1 - crates/rocm-dash-core/src/state.rs | 1 - crates/rocm-dash-core/src/vram.rs | 2 +- crates/rocm-dash-daemon/src/demo.rs | 9 +++++---- crates/rocm-dash-tui/examples/gen_cast.rs | 18 +++++++++++------- .../rocm-dash-tui/examples/gen_screenshots.rs | 19 +++++++++++-------- crates/rocm-dash-tui/src/jobs.rs | 2 +- crates/rocm-dash-tui/src/llm.rs | 2 +- crates/rocm-dash-tui/src/tool_exec.rs | 4 ++-- crates/rocm-dash-tui/src/ui/approval.rs | 5 ++--- .../src/ui/automations_manager.rs | 2 +- crates/rocm-dash-tui/src/ui/command_screen.rs | 2 +- crates/rocm-dash-tui/src/ui/config_manager.rs | 2 +- crates/rocm-dash-tui/src/ui/dock.rs | 2 +- crates/rocm-dash-tui/src/ui/engine_manager.rs | 7 +++---- .../rocm-dash-tui/src/ui/examine_manager.rs | 2 +- crates/rocm-dash-tui/src/ui/exec.rs | 2 +- crates/rocm-dash-tui/src/ui/folder_browser.rs | 2 +- .../rocm-dash-tui/src/ui/install_manager.rs | 4 ++-- crates/rocm-dash-tui/src/ui/job_console.rs | 2 +- crates/rocm-dash-tui/src/ui/logs_view.rs | 2 +- crates/rocm-dash-tui/src/ui/model_picker.rs | 7 ++++--- crates/rocm-dash-tui/src/ui/onboarding.rs | 11 +++++------ .../rocm-dash-tui/src/ui/runtime_manager.rs | 4 ++-- crates/rocm-dash-tui/src/ui/serve_wizard.rs | 14 +++++++------- .../rocm-dash-tui/src/ui/services_manager.rs | 11 +++++------ crates/rocm-dash-tui/src/ui/tabs/chat.rs | 8 +++++--- crates/rocm-dash-tui/src/ui/tabs/home.rs | 4 +--- crates/rocm-dash-tui/src/ui/tabs/pane.rs | 7 ++++--- crates/rocm-dash-tui/src/ui/theme.rs | 2 +- crates/rocm-dash-tui/src/ui/update_manager.rs | 2 +- crates/rocm-deps/src/lib.rs | 6 +++--- 43 files changed, 101 insertions(+), 100 deletions(-) diff --git a/crates/rocm-dash-collectors/Cargo.toml b/crates/rocm-dash-collectors/Cargo.toml index 91add936f4..440eef13fd 100644 --- a/crates/rocm-dash-collectors/Cargo.toml +++ b/crates/rocm-dash-collectors/Cargo.toml @@ -29,8 +29,8 @@ reqwest = { version = "0.12", default-features = false } [dev-dependencies] # Test-only: `write_fake_executable` plants the fake `amd-smi` the timeout tests # run without the intermittent "Text file busy" failure that `std::fs::write` -# followed by a chmod hits. Dev-dependencies are exempt from the layering -# allowlist, so the collectors keep no normal dependency on `rocm-core`. +# followed by a chmod hits. `rocm-core` is also a normal dependency above +# (`amd_smi.rs` uses `device_readable_bounded`), so this entry only restates it. rocm-core = { path = "../rocm-core" } tokio = { version = "1", features = ["full"] } tempfile = "3" diff --git a/crates/rocm-dash-collectors/src/amd_smi.rs b/crates/rocm-dash-collectors/src/amd_smi.rs index 0d258952e1..6370f19860 100644 --- a/crates/rocm-dash-collectors/src/amd_smi.rs +++ b/crates/rocm-dash-collectors/src/amd_smi.rs @@ -5,7 +5,7 @@ //! amd-smi subprocess + JSON parse. //! //! Field paths and the GPU-device pre-flight are vendored from the TypeScript -//! `AmdSmiProvider` in instinct-dash. See `../wiki/entities/amd-smi.md`. +//! `AmdSmiProvider` in instinct-dash. use std::ffi::OsString; use std::io; diff --git a/crates/rocm-dash-collectors/src/docker.rs b/crates/rocm-dash-collectors/src/docker.rs index 1727995052..8e573dcbfd 100644 --- a/crates/rocm-dash-collectors/src/docker.rs +++ b/crates/rocm-dash-collectors/src/docker.rs @@ -6,7 +6,6 @@ //! //! Field-extraction logic is vendored from instinct-dash `DockerService.ts` //! (HIP_VISIBLE_DEVICES → gpu_ids, `--tensor-parallel-size`/`-tp`, etc.). -//! See `../../wiki/entities/dockerode.md`. //! //! The async inherent methods are the public API. The `ServiceDiscovery` trait //! impl is sync-only and returns `Unsupported`: bollard requires a tokio diff --git a/crates/rocm-dash-collectors/src/parallel.rs b/crates/rocm-dash-collectors/src/parallel.rs index 0ad1d68f9a..cfe79b221f 100644 --- a/crates/rocm-dash-collectors/src/parallel.rs +++ b/crates/rocm-dash-collectors/src/parallel.rs @@ -4,10 +4,9 @@ //! Shared primitives for async collectors that scrape N targets concurrently. //! -//! Today: [[VllmPrometheusCollector]] uses this pattern hand-rolled in the -//! daemon's runner. As future per-instance scrapers come online, they'll reuse -//! `parallel_scrape` and the `WarningBus` instead of re-implementing the -//! JoinSet glue. +//! Today: the daemon's runner calls `parallel_scrape` for the vLLM Prometheus +//! scrape. Future per-instance scrapers should reuse `parallel_scrape` instead +//! of re-implementing the JoinSet glue. use std::future::Future; use std::sync::Arc; diff --git a/crates/rocm-dash-collectors/src/vllm_prom.rs b/crates/rocm-dash-collectors/src/vllm_prom.rs index 0fb6e2eb76..1aa184d151 100644 --- a/crates/rocm-dash-collectors/src/vllm_prom.rs +++ b/crates/rocm-dash-collectors/src/vllm_prom.rs @@ -5,7 +5,7 @@ //! vLLM Prometheus `/metrics` scraper. //! //! Field paths and the kv-cache 0..1 → 0..100 scaling are vendored from -//! instinct-dash `VllmMetricsService.ts`. See `../../wiki/entities/vllm.md`. +//! instinct-dash `VllmMetricsService.ts`. //! //! Sync `InstanceMetrics::fetch` returns `Unsupported` (the underlying client //! is async); call `fetch_async` from the runner. diff --git a/crates/rocm-dash-core/src/bench_rollup.rs b/crates/rocm-dash-core/src/bench_rollup.rs index f134cfac43..8acbc2a157 100644 --- a/crates/rocm-dash-core/src/bench_rollup.rs +++ b/crates/rocm-dash-core/src/bench_rollup.rs @@ -8,10 +8,10 @@ //! it whenever the row set changes rather than relying on the upstream //! `pass_n_of_n` / `pass_at_n` CSV columns. //! -//! See `../wiki/concepts/benchmark-result-schema.md` and -//! `../wiki/entities/normalize-results.md`: rows are grouped by -//! `(cell, model, backend, concurrency)` and each group of N trials yields -//! two verdicts — strict (all N passed) and lenient (at least one passed). +//! Rows are grouped by `(cell, model, engine, tp, dtype, concurrency)` (the +//! private `RollupKey`) and each group of N trials yields two verdicts — +//! strict (all N passed) and lenient (at least one passed). `engine` is part +//! of the key, so a row with no `engine` does not group with one that has it. use std::collections::BTreeMap; diff --git a/crates/rocm-dash-core/src/bench_schema.rs b/crates/rocm-dash-core/src/bench_schema.rs index 4090b6a4de..ce5aa70359 100644 --- a/crates/rocm-dash-core/src/bench_schema.rs +++ b/crates/rocm-dash-core/src/bench_schema.rs @@ -3,7 +3,6 @@ // SPDX-License-Identifier: MIT //! Benchmark row schema, vendored from instinct-agent-bench. -//! See `../wiki/concepts/benchmark-result-schema.md` and `../wiki/entities/csv-emitter.md`. use serde::{Deserialize, Serialize}; diff --git a/crates/rocm-dash-core/src/efficiency.rs b/crates/rocm-dash-core/src/efficiency.rs index 6781c86024..4083f82f0d 100644 --- a/crates/rocm-dash-core/src/efficiency.rs +++ b/crates/rocm-dash-core/src/efficiency.rs @@ -5,7 +5,7 @@ //! Pure efficiency derivations (tokens-per-watt). No I/O, no rendering. //! //! This is the join that turns vLLM throughput + amd-smi power into a -//! per-instance capacity number. See `../wiki/concepts/metric-registry.md`. +//! per-instance capacity number. use std::collections::HashMap; use std::hash::BuildHasher; diff --git a/crates/rocm-dash-core/src/lib.rs b/crates/rocm-dash-core/src/lib.rs index 03bd4fb6e7..fbaa4bc20e 100644 --- a/crates/rocm-dash-core/src/lib.rs +++ b/crates/rocm-dash-core/src/lib.rs @@ -7,7 +7,8 @@ //! Pure types, traits, schemas, and the reducer for rocm-dash. //! No rendering deps. No async deps at the type boundary. //! -//! See `../wiki/concepts/tea-reducer-pattern.md` for the architectural pattern. +//! `State::apply` in `state` is the pure reducer, in the style of The Elm +//! Architecture. pub mod bench_rollup; pub mod bench_schema; diff --git a/crates/rocm-dash-core/src/metrics.rs b/crates/rocm-dash-core/src/metrics.rs index 77d14afbe3..c474b1f9eb 100644 --- a/crates/rocm-dash-core/src/metrics.rs +++ b/crates/rocm-dash-core/src/metrics.rs @@ -3,7 +3,6 @@ // SPDX-License-Identifier: MIT //! Live metric types. snake_case + units encoded in field names. -//! See `../wiki/concepts/metric-registry.md` and `../wiki/data/metric-field-index.md`. use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; diff --git a/crates/rocm-dash-core/src/partition.rs b/crates/rocm-dash-core/src/partition.rs index 6f7706623a..3aa94bb6e7 100644 --- a/crates/rocm-dash-core/src/partition.rs +++ b/crates/rocm-dash-core/src/partition.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! GPU partition mode enums. See `../wiki/concepts/gpu-partition-modes.md`. +//! GPU partition mode enums. use serde::{Deserialize, Serialize}; diff --git a/crates/rocm-dash-core/src/protocol.rs b/crates/rocm-dash-core/src/protocol.rs index 01e9888b3c..6dd2a5cbc6 100644 --- a/crates/rocm-dash-core/src/protocol.rs +++ b/crates/rocm-dash-core/src/protocol.rs @@ -3,7 +3,6 @@ // SPDX-License-Identifier: MIT //! NDJSON command/event protocol between rocmdashd and rocmdash. -//! See `../wiki/comparisons/ctux-vs-rocm-dash.md` (resolved decisions). use serde::{Deserialize, Serialize}; diff --git a/crates/rocm-dash-core/src/state.rs b/crates/rocm-dash-core/src/state.rs index ec80d1ffc5..1f50df725f 100644 --- a/crates/rocm-dash-core/src/state.rs +++ b/crates/rocm-dash-core/src/state.rs @@ -3,7 +3,6 @@ // SPDX-License-Identifier: MIT //! Pure reducer. `State::apply(StateEvent) -> Vec`. -//! See `../wiki/concepts/tea-reducer-pattern.md`. use std::collections::{HashMap, VecDeque}; use std::sync::Arc; diff --git a/crates/rocm-dash-core/src/vram.rs b/crates/rocm-dash-core/src/vram.rs index 5752b45901..98fd294809 100644 --- a/crates/rocm-dash-core/src/vram.rs +++ b/crates/rocm-dash-core/src/vram.rs @@ -17,7 +17,7 @@ //! //! Graceful degradation: an instance that matches nothing (empty `gpu_ids`, //! no container entry — e.g. Lemonade) resolves to `(0, 0)`, never a panic and -//! never a confidently-wrong number. See `../wiki/concepts/metric-registry.md`. +//! never a confidently-wrong number. use std::collections::HashMap; diff --git a/crates/rocm-dash-daemon/src/demo.rs b/crates/rocm-dash-daemon/src/demo.rs index 50ede75a7e..acb7b6565c 100644 --- a/crates/rocm-dash-daemon/src/demo.rs +++ b/crates/rocm-dash-daemon/src/demo.rs @@ -6,13 +6,14 @@ //! //! Simulates a single 8× AMD Instinct MI355X node serving inference traffic //! through three vLLM containers. Output is the same `PersistedEntry` NDJSON -//! the daemon writes when `--persist-dir` is set, so `rocm replay` (and -//! `rocm demo`) play it back unmodified. +//! the daemon's session writer produces, so `rocm dash --replay` plays it back +//! unmodified. //! //! Deterministic: the same [`DemoOptions`] always produce byte-identical //! output (seeded xorshift jitter + a fixed wallclock anchor), which the -//! tests below pin as a hard invariant. `rocm gen-demo`, `rocm demo`, and the -//! marketing screenshot/cast examples all consume this one generator. +//! tests below pin as a hard invariant. `rocm dash --demo` is the consumer: it +//! writes a session to a private file under the user data directory with +//! [`generate_file`] and replays it. use std::collections::BTreeMap; use std::f64::consts::PI; diff --git a/crates/rocm-dash-tui/examples/gen_cast.rs b/crates/rocm-dash-tui/examples/gen_cast.rs index 490480958c..6468caabf6 100644 --- a/crates/rocm-dash-tui/examples/gen_cast.rs +++ b/crates/rocm-dash-tui/examples/gen_cast.rs @@ -3,13 +3,17 @@ // SPDX-License-Identifier: MIT //! Render the TUI against a synthetic session NDJSON to an asciicast v2 -//! recording. The marketing page replays it as a real animated terminal -//! demo — no live terminal, no external capture tools, no drift. +//! recording — no live terminal and no external capture tools. //! //! Usage: //! cargo run --release --example gen_cast -p rocm-dash-tui -- \ -//! --input /tmp/mi355x-demo.ndjson \ -//! --output ../marketing/asciinema/rocm-dash.cast +//! --input session.ndjson \ +//! --output rocm-dash.cast +//! +//! `--input` is a `PersistedEntry` NDJSON session recording, the format +//! `rocm dash --replay` reads, such as the `session-*.ndjson` that +//! `rocm dash --demo` writes under the `demo/` folder of the user data +//! directory. Whatever the input, the cast is marked as simulated data. //! //! Each frame is the actual `ui::draw` framebuffer for a point in a //! scripted storyboard (tab + modal walk), converted to a truecolor ANSI @@ -45,12 +49,12 @@ const DEFAULT_BG: (u8, u8, u8) = (13, 15, 18); about = "Render TUI storyboard to an asciicast v2 recording" )] struct Args { - /// Synthetic session file (produced by `make demo`). + /// Session NDJSON recording to render (`PersistedEntry` lines). #[arg(long, default_value = "/tmp/mi355x-demo.ndjson")] input: PathBuf, - /// Output cast file. Parent dir created if absent. Default targets the - /// repo-root `marketing/asciinema/` directory when run from `app/`. + /// Output cast file. Parent dir created if absent. The default is relative + /// to the working directory. #[arg(long, default_value = "../marketing/asciinema/rocm-dash.cast")] output: PathBuf, diff --git a/crates/rocm-dash-tui/examples/gen_screenshots.rs b/crates/rocm-dash-tui/examples/gen_screenshots.rs index 77bd598741..9677b39984 100644 --- a/crates/rocm-dash-tui/examples/gen_screenshots.rs +++ b/crates/rocm-dash-tui/examples/gen_screenshots.rs @@ -3,13 +3,17 @@ // SPDX-License-Identifier: MIT //! Render the TUI against a synthetic session NDJSON and dump each view to -//! a standalone SVG. The screenshots embed cleanly in the marketing page -//! and demo decks — no live terminal, no manual capture, no drift. +//! a standalone SVG — no live terminal and no manual capture. //! //! Usage: //! cargo run --release --example gen_screenshots -p rocm-dash-tui -- \ -//! --input /tmp/mi355x-demo.ndjson \ -//! --output-dir marketing/screenshots +//! --input session.ndjson \ +//! --output-dir screenshots +//! +//! `--input` is a `PersistedEntry` NDJSON session recording, the format +//! `rocm dash --replay` reads, such as the `session-*.ndjson` that +//! `rocm dash --demo` writes under the `demo/` folder of the user data +//! directory. Whatever the input, the views are marked as simulated data. //! //! Each output is the actual `ui::draw` framebuffer for a specific //! AppState configuration (tab + modal + theme). Cells get run-length @@ -35,13 +39,12 @@ use rocm_dash_tui::ui; about = "Render TUI views to SVG for marketing" )] struct Args { - /// Synthetic session file (produced by `make demo`). + /// Session NDJSON recording to render (`PersistedEntry` lines). #[arg(long, default_value = "/tmp/mi355x-demo.ndjson")] input: PathBuf, - /// Output directory. Created if absent. SVGs are named per view. - /// Default targets the repo-root `marketing/screenshots/` directory - /// when run from the `app/` workspace. + /// Output directory. Created if absent. SVGs are named per view. The + /// default is relative to the working directory. #[arg(long, default_value = "../marketing/screenshots")] output_dir: PathBuf, diff --git a/crates/rocm-dash-tui/src/jobs.rs b/crates/rocm-dash-tui/src/jobs.rs index 8f964bc71d..a3a1b55169 100644 --- a/crates/rocm-dash-tui/src/jobs.rs +++ b/crates/rocm-dash-tui/src/jobs.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Async job-bridge runtime (Phase 3 Wave 0). +//! Async job-bridge runtime. //! //! Interprets [`SideEffect::SpawnJob`]: launches the child process, streams its //! stdout+stderr back as [`StateEvent::JobLine`], finishes with diff --git a/crates/rocm-dash-tui/src/llm.rs b/crates/rocm-dash-tui/src/llm.rs index f83f85e687..b5627f9616 100644 --- a/crates/rocm-dash-tui/src/llm.rs +++ b/crates/rocm-dash-tui/src/llm.rs @@ -4,7 +4,7 @@ //! Chat LLM configuration: a pure precedence resolver plus a std-only TCP liveness probe. //! -//! No HTTP client here — the Rig-built `AgentClient` (Phase 3) +//! No HTTP client here — the Rig-built `AgentClient` //! lives in `agent/clients.rs`. Keeping detection in the TUI crate preserves the //! core's render/async-free boundary. diff --git a/crates/rocm-dash-tui/src/tool_exec.rs b/crates/rocm-dash-tui/src/tool_exec.rs index 0ae43ff45f..8ad611e742 100644 --- a/crates/rocm-dash-tui/src/tool_exec.rs +++ b/crates/rocm-dash-tui/src/tool_exec.rs @@ -7,8 +7,8 @@ //! Plain-data signatures ONLY (serde_json / std / serde). The bin (`apps/rocm`, //! which owns `rocm-core` and the tool engine) implements [`RocmToolExecutor`]; //! the dash holds it as `Option>` and never depends on -//! `rocm-core`. Phase 2 stored the seam; Phase 3 used it for read-only tools; -//! Phase 4 adds the mutating "execute approved" path + the approval descriptor. +//! `rocm-core`. The trait has a read-only `execute` and a mutating +//! `execute_approved` path, plus the approval descriptor. use std::sync::Arc; diff --git a/crates/rocm-dash-tui/src/ui/approval.rs b/crates/rocm-dash-tui/src/ui/approval.rs index 66d080905b..6fa82ffef9 100644 --- a/crates/rocm-dash-tui/src/ui/approval.rs +++ b/crates/rocm-dash-tui/src/ui/approval.rs @@ -2,12 +2,11 @@ // // SPDX-License-Identifier: MIT -//! Approval-gate seam (Phase 3 Wave 0). +//! Approval-gate seam. //! //! This is a **render + event seam ONLY**. The decision *logic* — what an //! approval actually does (run a CLI command, enable full access, approve a -//! proposal) — stays CLI-side per the working agreement -//! (`rocm-cli-unification-working-agreements.md` §1). This module renders an +//! proposal) — stays CLI-side. This module renders an //! [`ApprovalRequest`] and reports the user's [`ApprovalVerdict`]; the caller //! maps the verdict onto a CLI-side action. //! diff --git a/crates/rocm-dash-tui/src/ui/automations_manager.rs b/crates/rocm-dash-tui/src/ui/automations_manager.rs index b54ee199ca..acf6e4ab1e 100644 --- a/crates/rocm-dash-tui/src/ui/automations_manager.rs +++ b/crates/rocm-dash-tui/src/ui/automations_manager.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Automations manager overlay (Phase 3 Wave 3). +//! Automations manager overlay. //! //! Lists the background checks ("watchers", plumbed in from the bin which owns //! `rocm-core`) and toggles them via the `rocm automations …` verbs: diff --git a/crates/rocm-dash-tui/src/ui/command_screen.rs b/crates/rocm-dash-tui/src/ui/command_screen.rs index dca012b8bc..e52936af09 100644 --- a/crates/rocm-dash-tui/src/ui/command_screen.rs +++ b/crates/rocm-dash-tui/src/ui/command_screen.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Command runner overlay (Phase 3 Wave 3). +//! Command runner overlay. //! //! A general escape hatch: type any `rocm …` subcommand and run it through the //! job-bridge. Because arbitrary subcommands can't be statically classified as diff --git a/crates/rocm-dash-tui/src/ui/config_manager.rs b/crates/rocm-dash-tui/src/ui/config_manager.rs index 278104ac4d..9f124dbc5a 100644 --- a/crates/rocm-dash-tui/src/ui/config_manager.rs +++ b/crates/rocm-dash-tui/src/ui/config_manager.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Config + provider manager overlay (Phase 3 Wave 3). +//! Config + provider manager overlay. //! //! Folds the frozen `config_manager` and `provider_manager` into one screen: //! diff --git a/crates/rocm-dash-tui/src/ui/dock.rs b/crates/rocm-dash-tui/src/ui/dock.rs index 3ba1ea738f..fc1c0e58c0 100644 --- a/crates/rocm-dash-tui/src/ui/dock.rs +++ b/crates/rocm-dash-tui/src/ui/dock.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Wide-layout triptych rails (Phase 6). +//! Wide-layout triptych rails. //! //! When the terminal is wide enough, `ui::draw` splits the body into a left GPU //! wall, the existing center tab body, and a persistent right dock. The right diff --git a/crates/rocm-dash-tui/src/ui/engine_manager.rs b/crates/rocm-dash-tui/src/ui/engine_manager.rs index 5a1cd82037..802a645d90 100644 --- a/crates/rocm-dash-tui/src/ui/engine_manager.rs +++ b/crates/rocm-dash-tui/src/ui/engine_manager.rs @@ -2,13 +2,12 @@ // // SPDX-License-Identifier: MIT -//! Engine manager overlay (Phase 3 Wave 1). +//! Engine manager overlay. //! //! Lists the serving engines ROCm CLI knows about and runs use/install/reinstall //! **through the approval gate and the job-bridge** — never inline. Installs -//! stream their output into the shared job console (the Wave-0 `running_job` -//! replacement). This is the third operational screen on the proven primitives; -//! it feeds the serve wizard (which engine to launch a model with). +//! stream their output into the shared job console. It feeds the serve wizard +//! (which engine to launch a model with). //! //! The approval *decision* is the user's, captured by the render+event seam; the //! CLI owns the actual mutation (`rocm config set-default-engine` / diff --git a/crates/rocm-dash-tui/src/ui/examine_manager.rs b/crates/rocm-dash-tui/src/ui/examine_manager.rs index 40138296b0..67c21878c4 100644 --- a/crates/rocm-dash-tui/src/ui/examine_manager.rs +++ b/crates/rocm-dash-tui/src/ui/examine_manager.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Examine overlay (Phase 3 Wave 2). +//! Examine overlay. //! //! Runs `rocm examine` — a read-only environment check — through the job-bridge //! and shows its streamed output in the shared job console. That run is diff --git a/crates/rocm-dash-tui/src/ui/exec.rs b/crates/rocm-dash-tui/src/ui/exec.rs index 5da3542ef0..48a9fda0a3 100644 --- a/crates/rocm-dash-tui/src/ui/exec.rs +++ b/crates/rocm-dash-tui/src/ui/exec.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Shared helpers for launching `rocm` sub-commands from operational screens (Phase 3 Wave 1). +//! Shared helpers for launching `rocm` sub-commands from operational screens. //! //! Every screen that routes a mutating action through the //! approval gate + job-bridge resolves the binary the same way, so the logic diff --git a/crates/rocm-dash-tui/src/ui/folder_browser.rs b/crates/rocm-dash-tui/src/ui/folder_browser.rs index b26c490c9a..b6e5adf618 100644 --- a/crates/rocm-dash-tui/src/ui/folder_browser.rs +++ b/crates/rocm-dash-tui/src/ui/folder_browser.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Reusable folder browser (Phase 3 Wave 0). +//! Reusable folder browser. //! //! A shared input primitive: pick (or create-under) a directory. Dependency of //! serve / install / runtime / onboarding. Models the frozen rocm-cli diff --git a/crates/rocm-dash-tui/src/ui/install_manager.rs b/crates/rocm-dash-tui/src/ui/install_manager.rs index 6a25a1a742..e2d4cdb6df 100644 --- a/crates/rocm-dash-tui/src/ui/install_manager.rs +++ b/crates/rocm-dash-tui/src/ui/install_manager.rs @@ -2,12 +2,12 @@ // // SPDX-License-Identifier: MIT -//! Install overlay (Phase 3 Wave 2). +//! Install overlay. //! //! A compact form that builds a `rocm install sdk …` invocation (TheRock ROCm //! wheels). Dry-run is read-only and spawns straight through the job-bridge; a //! real install is mutating and routes through the approval gate first. The -//! install folder is picked with the Wave-0 [`FolderBrowser`]. Streaming-install +//! install folder is picked with the [`FolderBrowser`]. Streaming-install //! archetype, zero thread::spawn/try_recv. use crossterm::event::{KeyCode, KeyEvent}; diff --git a/crates/rocm-dash-tui/src/ui/job_console.rs b/crates/rocm-dash-tui/src/ui/job_console.rs index 7df8480d25..15cb9d69cf 100644 --- a/crates/rocm-dash-tui/src/ui/job_console.rs +++ b/crates/rocm-dash-tui/src/ui/job_console.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Job console widget (Phase 3 Wave 0). +//! Job console widget. //! //! Renders a [`JobState`] from the reducer: a status header, the streamed //! output ring, and key hints. This is the shared "running job" surface every diff --git a/crates/rocm-dash-tui/src/ui/logs_view.rs b/crates/rocm-dash-tui/src/ui/logs_view.rs index ff5f40afc1..53f6e71530 100644 --- a/crates/rocm-dash-tui/src/ui/logs_view.rs +++ b/crates/rocm-dash-tui/src/ui/logs_view.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Logs overlay (Phase 3 Wave 3). +//! Logs overlay. //! //! Browses recent ROCm CLI logs via `rocm logs [--search WORDS]` — read-only, so //! no approval gate. An optional search box filters before running. Output diff --git a/crates/rocm-dash-tui/src/ui/model_picker.rs b/crates/rocm-dash-tui/src/ui/model_picker.rs index 0072970e3c..aa2c8db33e 100644 --- a/crates/rocm-dash-tui/src/ui/model_picker.rs +++ b/crates/rocm-dash-tui/src/ui/model_picker.rs @@ -2,12 +2,13 @@ // // SPDX-License-Identifier: MIT -//! Model picker (Phase 3 Wave 1). +//! Model picker. //! //! A reusable, filterable list of model recipes — the sub-step the serve wizard //! opens from its Model field so the user can pick a built-in recipe instead of -//! typing a model name. Selecting a recipe fills the model id and (when known) -//! pre-selects the recipe's preferred engine. +//! typing a model name. Selecting a recipe returns its summary; the serve +//! wizard keeps only the id and fills the Model field with it. The recipe's +//! preferred engine is shown in the list but never applied. //! //! Recipes are passed in as plain TUI-local [`ModelRecipeSummary`] values — the //! bin (`apps/rocm`, which has `rocm-core`) adapts the full `ModelRecipeRecord` diff --git a/crates/rocm-dash-tui/src/ui/onboarding.rs b/crates/rocm-dash-tui/src/ui/onboarding.rs index 0afcd56cf8..ea38f4d669 100644 --- a/crates/rocm-dash-tui/src/ui/onboarding.rs +++ b/crates/rocm-dash-tui/src/ui/onboarding.rs @@ -2,19 +2,18 @@ // // SPDX-License-Identifier: MIT -//! Onboarding wizard (Phase 3 Wave 2, minimal). +//! Onboarding wizard (minimal). //! -//! The first-run flow, rebuilt on the Wave-0 primitives. This is the **minimal** -//! version the phase-3 plan calls for: get a clean machine to a working ROCm -//! runtime two ways — +//! The first-run flow, kept deliberately **minimal**: get a clean machine to a +//! working ROCm runtime two ways — //! //! - **Install ROCm SDK** — a one-shot gated `rocm install sdk` with a Configure //! step to pick the channel (Release/Nightly), an optional version pin //! (`--build-date` / `--version`), and an optional install folder -//! (`--prefix`, via the same Wave-0 [`FolderBrowser`] the Adopt path uses); +//! (`--prefix`, via the same [`FolderBrowser`] the Adopt path uses); //! defaults to `--channel release --format wheel` installed into the //! default managed folder. -//! - **Adopt existing folder** — pick an existing ROCm env with the Wave-0 +//! - **Adopt existing folder** — pick an existing ROCm env with the //! [`FolderBrowser`], then approve `rocm runtimes adopt`. //! //! Reinstall / uninstall / show-log sub-modals (the full frozen onboarding), diff --git a/crates/rocm-dash-tui/src/ui/runtime_manager.rs b/crates/rocm-dash-tui/src/ui/runtime_manager.rs index 98d2e443da..dcbacb05c4 100644 --- a/crates/rocm-dash-tui/src/ui/runtime_manager.rs +++ b/crates/rocm-dash-tui/src/ui/runtime_manager.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Runtime manager overlay (Phase 3 Wave 2). +//! Runtime manager overlay. //! //! Lists the registered ROCm runtimes (plumbed in from the bin, which owns //! `rocm-core` — this crate stays core-free) and runs the @@ -12,7 +12,7 @@ //! job-bridge. //! - **activate / rollback / uninstall** — mutating, routed through the approval //! gate first. -//! - **adopt** — pick an existing ROCm env folder with the Wave-0 +//! - **adopt** — pick an existing ROCm env folder with the //! [`FolderBrowser`], then approve `rocm runtimes adopt`. //! - **import** — type a manifest path, then approve `rocm runtimes import`. //! diff --git a/crates/rocm-dash-tui/src/ui/serve_wizard.rs b/crates/rocm-dash-tui/src/ui/serve_wizard.rs index 2ea58fde01..172dbc8099 100644 --- a/crates/rocm-dash-tui/src/ui/serve_wizard.rs +++ b/crates/rocm-dash-tui/src/ui/serve_wizard.rs @@ -2,13 +2,13 @@ // // SPDX-License-Identifier: MIT -//! Serve wizard overlay (Phase 3 Wave 1). +//! Serve wizard overlay. //! -//! The headline operational screen rebuilt on the Wave-0 primitives: a compact -//! **model-first** form that builds a `rocm serve …` invocation and runs it -//! **through the approval gate and the job-bridge** — never inline, never with -//! a legacy `std::thread::spawn` + `try_recv`. A served model launched here -//! surfaces in the services manager and the dashboard's live `gen_tps`. +//! The headline operational screen: a compact **model-first** form that builds +//! a `rocm serve …` invocation and runs it **through the approval gate and the +//! job-bridge** — never inline, never with a legacy `std::thread::spawn` + +//! `try_recv`. A served model launched here surfaces in the services manager +//! and the dashboard's live `gen_tps`. //! //! Progressive disclosure: the default form is Model → `Advanced settings` → //! Launch. Everything else (engine, device policy, host, port, mode) is @@ -19,7 +19,7 @@ //! policy, the loopback host, the port, and managed mode. //! //! The model field can be typed directly (a recipe name, alias, or path) or -//! filled from the reusable Wave-0 [`FolderBrowser`] for a local model path +//! filled from the reusable [`FolderBrowser`] for a local model path //! (`Tab` on the Model field). The approval *decision* is the user's, captured //! by the render+event seam; the CLI owns the actual launch — the read-only //! chat invariant is untouched. diff --git a/crates/rocm-dash-tui/src/ui/services_manager.rs b/crates/rocm-dash-tui/src/ui/services_manager.rs index 24441b3b7d..1fe23796fe 100644 --- a/crates/rocm-dash-tui/src/ui/services_manager.rs +++ b/crates/rocm-dash-tui/src/ui/services_manager.rs @@ -2,13 +2,12 @@ // // SPDX-License-Identifier: MIT -//! Services manager overlay (Phase 3 Wave 1). +//! Services manager overlay. //! -//! The first operational screen rebuilt on the Wave-0 primitives: it lists the -//! managed inference services the daemon surfaces (model · port · status · live -//! `gen_tps`) and runs stop/restart **through the approval gate and the -//! job-bridge** — never inline. This is the pattern the remaining Wave-1 screens -//! (serve_wizard / model_picker / engine_manager) reuse. +//! It lists the managed inference services the daemon surfaces (model · port · +//! status · live `gen_tps`) and runs stop/restart **through the approval gate +//! and the job-bridge** — never inline. The serve wizard and the engine manager +//! follow the same pattern. //! //! Mutating actions invoke the CLI (`rocm services stop|restart --yes`) as //! a background job; the approval *decision* is the user's, captured by the diff --git a/crates/rocm-dash-tui/src/ui/tabs/chat.rs b/crates/rocm-dash-tui/src/ui/tabs/chat.rs index ffb32550a3..a3a1736fe8 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/chat.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/chat.rs @@ -4,9 +4,11 @@ //! Chat tab — a TUI-local conversation surface. //! -//! Phase 1 is render-only with a local echo backend; later phases wire a Rig-built agent behind the -//! `AgentClient` trait. The transcript and input buffer are plain TUI state on -//! `AppState` (`rocm-dash-core` carries no chat types). +//! Until the user accepts the detected endpoint (`ChatConsent`), the tab shows a +//! consent gate or an empty state; once accepted it draws the transcript and +//! the input line of the real chat, whose backend sits behind the `AgentClient` +//! trait. The transcript and input buffer are plain TUI state on `AppState` +//! (`rocm-dash-core` carries no chat types). use ratatui::Frame; use ratatui::layout::{Constraint, Direction, Layout, Rect}; diff --git a/crates/rocm-dash-tui/src/ui/tabs/home.rs b/crates/rocm-dash-tui/src/ui/tabs/home.rs index e49cdeb872..9ee54d7ec4 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/home.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/home.rs @@ -11,9 +11,7 @@ //! with a hint at the tab/key that would produce something, so no tile is a //! dead end. //! -//! ponytail: Home is added behind the existing default this phase (P2). It is -//! reachable by Tab / digit `1` but is NOT the default tab yet — P3 repoints -//! the default and folds the telemetry tabs into Observe. +//! Home is the default tab (`ActiveTab::Home`) and is reachable by Tab / digit `1`. use ratatui::Frame; use ratatui::layout::{Constraint, Direction, Layout, Rect}; diff --git a/crates/rocm-dash-tui/src/ui/tabs/pane.rs b/crates/rocm-dash-tui/src/ui/tabs/pane.rs index 0909fb74cb..be6373134c 100644 --- a/crates/rocm-dash-tui/src/ui/tabs/pane.rs +++ b/crates/rocm-dash-tui/src/ui/tabs/pane.rs @@ -6,9 +6,10 @@ //! //! Each domain tab is a left **Actions** bento (its locked verb list) plus a //! right **`Details: `** bento that previews the selected operation. The -//! verb rows open EXISTING managers through the EXISTING execution seam -//! (`KeyAction::Open*` → `apply_action` → `RocmToolExecutor` / `ui/approval.rs`) -//! — no second approval path, no reimplementation of the managers. +//! verb rows open EXISTING managers: a verb produces a `KeyAction::Open*`, and +//! `apply_action` (in `app/actions.rs`) opens the matching overlay. Nothing +//! here executes anything — no second approval path, no reimplementation of +//! the managers. //! //! `rocm.rs` and `serving.rs` only declare their verb tables; all rendering, //! hit-testing, and the list|detail geometry live here so the two tabs cannot diff --git a/crates/rocm-dash-tui/src/ui/theme.rs b/crates/rocm-dash-tui/src/ui/theme.rs index d875802791..b90827a168 100644 --- a/crates/rocm-dash-tui/src/ui/theme.rs +++ b/crates/rocm-dash-tui/src/ui/theme.rs @@ -11,7 +11,7 @@ //! sourced from each project's published palette. Theme catalogue //! inspired by [ansicolor.com](https://ansicolor.com/). //! -//! Pattern borrowed from ctux (see `../../../wiki/sources/ctux.md`). +//! Pattern borrowed from ctux. use ratatui::style::{Color, Modifier, Style}; use rocm_dash_core::metrics::InstanceStatus; diff --git a/crates/rocm-dash-tui/src/ui/update_manager.rs b/crates/rocm-dash-tui/src/ui/update_manager.rs index 49e9c80896..4b6c6ac2aa 100644 --- a/crates/rocm-dash-tui/src/ui/update_manager.rs +++ b/crates/rocm-dash-tui/src/ui/update_manager.rs @@ -2,7 +2,7 @@ // // SPDX-License-Identifier: MIT -//! Update overlay (Phase 3 Wave 2). +//! Update overlay. //! //! Wraps `rocm update`: a read-only check (and a `--apply --dry-run` preview) //! that run straight through the job-bridge, plus the mutating `--apply` / diff --git a/crates/rocm-deps/src/lib.rs b/crates/rocm-deps/src/lib.rs index e8044d7602..ee95efbf0d 100644 --- a/crates/rocm-deps/src/lib.rs +++ b/crates/rocm-deps/src/lib.rs @@ -6,9 +6,9 @@ //! helpers that derive artifact names and URLs from them. //! //! The pins live in the workspace-root `runtime-deps.toml` and are turned into -//! the constants below by `build.rs`. Every consumer — the engine adapters and -//! the dashboard TUI alike — derives what it needs from those constants, so a -//! runtime version is spelled exactly once in the repository and two crates +//! the constants below by `build.rs`. Every consumer — the `rocm` binary and +//! the Lemonade engine adapter — derives what it needs from those constants, so +//! a runtime version is spelled exactly once in the repository and two crates //! cannot disagree about it. // `pub const LEMONADE_VERSION: &str = "..."`, one per `runtime-deps.toml` field. From 752b613cb69f0c0f65b3ac8c7cd08c0a49d608d6 Mon Sep 17 00:00:00 2001 From: Jussi Elo Date: Mon, 9 Nov 2026 22:35:57 +0000 Subject: [PATCH 034/102] fix(logs): read every log tail through one bounded, lossy routine and stop reporting a log with content as empty (#961) * test(logs): pin how each log-tail reader behaves today Three modules read the last lines of a log their own way: the ComfyUI log view (whole file, sixteen-bit decode, ANSI strip, lossy text), the services text module (streamed, fails on invalid text) and the rocmd MCP service_logs tool (whole file into one string, fails on invalid text), while the seek-bounded routine in rocm-core serves only the engines. Add characterisation tests that pass on the unmodified code and document the differences before they are removed: a large file (the whole-file readers return a line far outside any tail window; the shared routine does not), a sixteen-bit little-endian log with ANSI colour codes, invalid eight-bit text, a missing file, a limit of zero, fewer lines than the limit, a final line without a trailing newline, and CRLF endings. Test-only; no behaviour change. Signed-off-by: Jussi Elo * fix(logs): read every log tail through one bounded, lossy routine The ComfyUI log view, the services text module and the rocmd MCP service_logs tool each read a log tail their own way, while the seek-bounded rocm_core::tail_lines served only the engines. Converge all of them on it. rocm_core::log_tail gains tail_lines_decoded, taking a LogDecode: PLAIN (UTF-8, what the engines and the service logs use) or CONSOLE (UTF-16 little-endian detected by byte order mark or zero-byte pattern, ANSI sequences stripped; what the ComfyUI log view used). The UTF-16 path keeps the seek-boundary handling: the window starts on a character boundary and a partial first line is dropped. strip_ansi_sequences is exported for the assistant-prose note in tool_policy. A shared LOG_VIEW_MAX_TAIL_READ budget (4 MiB) is what the three log views pass. The private readers are deleted; each caller keeps a thin adapter that picks a LogDecode and its own error label. User-visible differences: - A log larger than 4 MiB is read from its last 4 MiB, not whole (all three). - Invalid UTF-8 no longer fails the view: `rocm services logs` and `rocm logs` showed one "" placeholder line and the MCP service_logs tool returned an error; both now show the readable text with U+FFFD for the bad bytes. - Service logs and the MCP tool still read UTF-8 and keep ANSI sequences; a UTF-16 service log is now shown as lossy text rather than as an error. - ComfyUI logs: ANSI stripping is per line, so an unterminated escape sequence ends at its line instead of swallowing the lines after it. - Error wording: a missing log is "failed to read