diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a3091b821f..8605f6ee84 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,7 +20,7 @@ Bug reports and feature requests are tracked as [GitHub Issues](https://github.c Use [Conventional Commits](https://www.conventionalcommits.org/) for commit messages: -``` +```text feat: add support for vLLM multi-GPU serving fix: correct VRAM probe when amd-smi is absent docs: clarify GPU selection flag behavior @@ -29,7 +29,7 @@ chore: bump rust-toolchain to 1.96 Branch names should be short and descriptive: -``` +```text feat/vllm-multi-gpu fix/vram-probe-fallback ``` @@ -48,7 +48,20 @@ prek install -t pre-push # heavier checks on push (clippy + tests of the crat `prek` runs the same checks locally that CI enforces: `cargo fmt`, `clippy`, `cargo test`, `ruff` (Python), `shellcheck` (shell), PowerShell syntax, license headers (`hawkeye`), markdown links (`lychee`), and the generated manifests (`MANIFEST.md`, `THIRD_PARTY_NOTICES.txt`). -The pre-push hooks (`cargo clippy`, the e2e harness clippy and `cargo test`) check only the crates your push affects: the crates whose files changed plus every crate that depends on them, selected the way CI's `cargo xtask affected` does. They check the whole workspace when a change cannot be confined to specific crates (the lockfile, the root manifest, the toolchain file, workflow files, a doc an xtask test reads, or a path no crate owns), and when the affected crates cannot be determined at all, for example because the remote branch tip is not fetched locally or the pushed range lists no changed files. They also add the crates whose tests read files from outside their own crate: any Rust file adds `xtask`, whose contract tests scan every source file, and `README.md`, `docs/` and `skills/` add `rocm` and `xtask`. Each step prints the scope it chose and why. A push that changes no Rust file runs none of the cargo steps, whatever else it changes (a lockfile-only push included). A green hook is an early warning and not the gate: on pull requests CI lints the whole workspace, but its Linux test job runs only the affected crates, and the whole workspace's tests run in the Windows debug job, in the merge queue and on `main`. To see what a push would check without compiling anything, run `cargo xtask pre-push-check test --dry-run --base origin/main` (the steps are `clippy`, `clippy-e2e` and `test`). +The pre-push hooks (`cargo clippy`, the e2e harness clippy, and `cargo test`) check only the crates your push affects: the crates whose files changed plus every crate that depends on them, selected the way CI's `cargo xtask affected` does. Each step prints the scope it chose and why. + +The hooks check the whole workspace when: + +- A change cannot be confined to specific crates (the lockfile, the root manifest, the toolchain file, workflow files, a doc an xtask test reads, or a path no crate owns). +- The affected crates cannot be determined at all, for example because the remote branch tip is not fetched locally or the pushed range lists no changed files. + +The hooks also add the crates whose tests read files from outside their own crate. Any Rust file adds `xtask`, whose contract tests scan every source file. `README.md`, `docs/`, and `skills/` add `rocm` and `xtask`. A push that changes no Rust file runs none of the cargo steps, whatever else it changes (a lockfile-only push included). + +A green hook is an early warning, not the gate. On pull requests, CI lints the whole workspace, but its Linux test job runs only the affected crates. The whole workspace's tests run in the Windows debug job, in the merge queue, and on `main`. To see what a push would check without compiling anything, run the following command (the steps are `clippy`, `clippy-e2e`, and `test`): + +```bash +cargo xtask pre-push-check test --dry-run --base origin/main +``` The manifest hooks only run when you change the dependency graph. They *rewrite* the generated file instead of only reporting that it is stale. When that happens, the commit stops so you can re-stage the refreshed file. @@ -58,7 +71,9 @@ The `THIRD_PARTY_NOTICES.txt` hook also needs the pinned generator. Without it, cargo install cargo-about@0.9.1 --locked --features cli # optional, for THIRD_PARTY_NOTICES.txt ``` -The license-headers hook (`hawkeye`) runs on every commit and push, whatever is staged. It has no file-type filter, because `hawkeye` always scans the whole working tree per `licenserc.toml` regardless of which files changed or are staged, including untracked files that aren't gitignored, so an unrelated unheadered file (even one not yet committed) can block an otherwise-unrelated commit. A clean CI checkout never has untracked files, so this hook can be stricter locally than CI is. `prek` runs it like the others, but doesn't provision the `hawkeye` binary. Unlike the manifest hooks above, it fails hard (not just skips) when the binary is missing. It does not check that an installed binary is the pinned version; a mismatched version may run whatever that version's own license-header rules happen to be, which may disagree with CI, or may reject `licenserc.toml` outright. For example, `hawkeye` 6.x rejects this repository's config with `unknown field 'files'`, which blocks every commit and push, not just ones touching code: +The license-headers hook (`hawkeye`) runs on every commit and push, whatever is staged. It has no file-type filter, because `hawkeye` always scans the whole working tree per `licenserc.toml`, including untracked files that aren't gitignored. An unrelated unheadered file, even one not yet committed, can therefore block an otherwise unrelated commit. A clean CI checkout never has untracked files, so this hook can be stricter locally than CI is. + +`prek` runs the hook like the others but doesn't provision the `hawkeye` binary. Unlike the manifest hooks above, it fails hard (not just skips) when the binary is missing. It also doesn't check that an installed binary is the pinned version. A mismatched version might apply different license-header rules than CI, or might reject `licenserc.toml` outright. For example, `hawkeye` 6.x rejects this repository's config with `unknown field 'files'`, which blocks every commit and push, not just ones touching code. Install the pinned version: ```bash cargo install hawkeye@7.0.0 --locked # pinned to match the CI license-headers job @@ -66,7 +81,7 @@ cargo install hawkeye@7.0.0 --locked # pinned to match the CI license-headers To commit or push without it (for example, while iterating without the binary installed, or on a version it rejects), skip it explicitly: `SKIP=license-headers git commit ...` or `SKIP=license-headers git push`. CI's `license-headers` job still enforces the check either way. -The Markdown-links hook (`lychee`) runs on every commit, over the whole repository rather than just the changed files, since moving or deleting a file can break a link in a Markdown file you didn't touch. It checks your working tree, so a link to a new file you haven't `git add`ed passes locally and still fails in CI. The hook runs `cargo xtask lychee --if-available`. `prek` doesn't provision the binary; without it the hook prints a warning and passes, leaving the `docs-links` CI job as the gate, and a version other than the pinned one runs with a warning: +The Markdown-links hook (`lychee`) runs on every commit, over the whole repository instead of just the changed files, because moving or deleting a file can break a link in a Markdown file you didn't touch. It checks your working tree, so a link to a new file you haven't `git add`ed passes locally and still fails in CI. The hook runs `cargo xtask lychee --if-available`. `prek` doesn't provision the binary; without it the hook prints a warning and passes, leaving the `docs-links` CI job as the gate, and a version other than the pinned one runs with a warning: ```bash cargo install lychee@0.24.2 --locked # pinned to match the CI docs-links job (lycheeVersion) @@ -74,32 +89,37 @@ cargo install lychee@0.24.2 --locked # pinned to match the CI docs-links job ( ### Workspace layout +The workspace contains these top-level components: + | Path | Description | | --- | --- | | `apps/rocm` | Main CLI binary | | `apps/rocmd` | Background daemon | | `crates/rocm-core` | Core library | | `crates/rocm-dash-*` | Dashboard terminal user interface (TUI) libraries | -| `crates/rocm-engine-protocol` | Engine IPC protocol | +| `crates/rocm-engine-protocol` | Engine inter-process communication (IPC) protocol | | `engines/` | Inference engine adapters (lemonade, vllm, aim) | ### Module organization -New subcommands and subsystems default to their own file from day one. Don't let them grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. See `docs/architecture.md` for the two extraction patterns in use, the current module map, and the module-organization convention in full. Its file links, including every concrete file citation in the per-crate inventories, are checked for resolution (not prose accuracy) by the `docs-links` job below; its section headings and directory names stay bare and unchecked. +New subcommands and subsystems default to their own file from day one. Don't let them grow inside `main.rs`/`lib.rs` waiting for a future extraction pass. See [`docs/architecture.md`](https://github.com/ROCm/rocm-cli/blob/main/docs/architecture.md) for the two extraction patterns in use, the current module map, and the module-organization convention in full. + +The `docs-links` job below checks the file links in that document, including every concrete file citation in the per-crate inventories. It checks that they resolve, not that the prose is accurate. Its section headings and directory names stay bare and unchecked. Crate-layering invariants, such as `rocmd` never depending on `rocm`, are enforced by `cargo xtask check-crate-edges` (`xtask/src/crate_edges.rs`). File size is checked by `cargo xtask check-module-size` (`xtask/src/module_size.rs`): no source file may exceed 1,500 production lines, test code excluded, and a change that would cross the limit splits the file instead. Editing a file that is already over the limit does not by itself require a split; those files are split under their own tickets. For now the CI step and the `cargo-module-size` pre-commit hook report offenders without failing. Every local (relative-path) Markdown link and `#anchor` fragment in the repository (outside `docs/rocm-docs/`, which `docs-build` covers instead, and hidden directories such as `.github/`, which neither check walks) is checked by the `docs-links` CI job and by the `lychee` prek hook above, both configured by `lychee.toml` (which lists the current exclusions). Both run offline only, so they don't catch broken external `https://` links. -Since only links are checked, cite a specific repository file as a link relative to the citing file, `[fix.rs](../crates/rocm-core/src/fix.rs)` from `docs/`, rather than a bare backtick path, which nothing checks. The full rule, including the link forms that fail the check and the places it doesn't apply (`.github/`, links leaving a `skills/` folder, and `docs/rocm-docs/` along with the files it includes, which are this file and README.md), is in AGENTS.md section 5 ("Investigate rocm-cli Before Editing"); it applies to citations you add or edit. +Because only links are checked, cite a specific repository file as a link relative to the citing file, `[fix.rs](../crates/rocm-core/src/fix.rs)` from `docs/`, instead of a bare backtick path, which nothing checks. The full rule, including the link forms that fail the check and the places it doesn't apply (`.github/`, links leaving a `skills/` folder, and `docs/rocm-docs/` along with the files it includes, which are this file and README.md), is in [AGENTS.md](https://github.com/ROCm/rocm-cli/blob/main/AGENTS.md) section 5 ("Investigate rocm-cli Before Editing"); it applies to citations you add or edit. ### Updating the AIM catalog -AIM model entries are hand-maintained in `crates/rocm-core/src/model_catalog.json`; -see `docs/aim-catalog.md` for the refresh process and staleness warning. +AIM model entries are hand-maintained in `crates/rocm-core/src/model_catalog.json`; see [`docs/aim-catalog.md`](https://github.com/ROCm/rocm-cli/blob/main/docs/aim-catalog.md) for the refresh process and staleness warning. ### Test commands +Run the tests with these commands: + | Component | Command | | --- | --- | | Rust (all crates) | `cargo test` | @@ -109,6 +129,8 @@ See the [test guide](https://github.com/ROCm/rocm-cli/blob/main/docs/testing.md) ## Making changes +To contribute a change: + 1. **Fork** the repository and create a branch from `main`. 2. **Make your changes.** Keep commits focused, with one logical change per commit. 3. **Add or update tests** for any new behavior. @@ -131,6 +153,8 @@ See [Commit signatures](https://github.com/ROCm/rocm-cli/blob/main/docs/commit-s ### What reviewers look for +Reviewers look for the following: + - Tests cover the new behavior - No secrets, credentials, or internal hostnames in committed files - Third-party dependencies recorded in `Cargo.lock` diff --git a/README.md b/README.md index ff5d7f9020..7848990ec1 100644 --- a/README.md +++ b/README.md @@ -17,15 +17,15 @@ SPDX-License-Identifier: MIT ██║ ██║╚██████╔╝╚██████╗██║ ╚═╝ ██║ ╚██████╗███████╗██║ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝╚═╝ ╚═╝ ╚═════╝╚══════╝╚═╝ - Local AI on AMD GPUs — one binary, zero setup + Local AI on AMD GPUs — one install, zero setup ``` ROCm CLI is a command-line tool for setting up and running local AI on AMD GPUs, with a full-screen terminal user interface (TUI) dashboard for GPU telemetry, model serving, and chat. -It ships as a single prebuilt binary for Linux and Windows (x86_64), needs no -Python, Rust, or existing ROCm install, and includes inference engine adapters -for Lemonade and vLLM. +It ships as prebuilt binaries (`rocm` and `rocmd`) for Linux and Windows +(x86_64), needs no Python, Rust, or existing ROCm install, and includes +inference engine adapters for Lemonade and vLLM. | Platform | Prebuilt binary | Notes | @@ -38,17 +38,19 @@ for Lemonade and vLLM. For WSL2 setup and diagnosis, see [docs/wsl.md](docs/wsl.md). -Live dashboard telemetry works on Windows too, but shows only GPU/CPU/memory — -the serving-instance registry view needs the `rocmdashd` unix socket, so it's +Live dashboard telemetry works on Windows too, but shows only GPU/CPU/memory. +The serving-instance registry view needs the `rocmdashd` Unix socket, so it's Linux/WSL2 only (see [Interactive interfaces](#interactive-interfaces)). vLLM serving is Linux or WSL2 only (see [docs/vllm.md](docs/vllm.md)). The minimum supported Linux release, native or under WSL2, is Ubuntu 24.04. On -other distributions the equivalent requirement is glibc 2.38 with -`GLIBCXX_3.4.32`: that is what the Lemonade engine is linked against, and every -published build of it needs those versions, so there is no older release to fall -back to. Ubuntu 22.04 ships glibc 2.35 and cannot run it; Ubuntu 24.04 provides -glibc 2.39 and `GLIBCXX_3.4.33`. +other distributions, the equivalent requirement is glibc 2.38 with +`GLIBCXX_3.4.32`. + +The Lemonade engine is linked against these versions, and every published build +of it needs them, so there is no older release to fall back to. Ubuntu 22.04 +ships glibc 2.35 and cannot run the engine. Ubuntu 24.04 provides glibc 2.39 and +`GLIBCXX_3.4.33`. > [!IMPORTANT] > **Tech Preview:** This software is provided as-is, without warranty or @@ -126,10 +128,10 @@ version. A successful install prints the CLI version line. The SDK and driver lines read `not detected` when none is found, which is expected on a computer where you haven't installed ROCm or a driver yet. -If the shell reports `rocm: command not found`, confirm that the install -directory is on your `PATH`. That is `~/.local/bin` on Linux, -`%USERPROFILE%\.local\bin` on Windows, or the folder you set with -`ROCM_CLI_INSTALL_DIR`. +If your shell can't find the `rocm` command, for example with +`rocm: command not found`, confirm that the install directory is on your `PATH`. +The directory is `~/.local/bin` on Linux, `%USERPROFILE%\.local\bin` on Windows, +or the folder you set with `ROCM_CLI_INSTALL_DIR`. ## Build from source @@ -189,11 +191,18 @@ that job finishes, `Ctrl-C` quits there too. If a model is still being served, quitting asks for confirmation first. This covers `q`, the Esc menu's `Quit` row, and `/quit` or `/exit` in chat, from the -launcher or from a dashboard or chat session opened directly. The prompt names -the model and the real `rocm services stop --yes` command to stop it. From -the dashboard, pressing `s` instead jumps straight to the Services overlay to -stop it there. Declining (`n` or Esc) returns to where you were. Confirming -(`y` or Enter) quits anyway, and the model keeps running in the background. +launcher or from a dashboard or chat session opened directly. + +The prompt names the model and the `rocm services stop --yes` command to +stop it. From the dashboard, press `s` instead to jump straight to the Services +overlay and stop it there. + +At the prompt: + +- Decline (`n` or Esc) to return to where you were. +- Confirm (`y` or Enter) to quit anyway. The model keeps running in the + background. + Unlike `q`, `Ctrl-C` always quits immediately without asking, with the same running-job exception as above. See [Developer manual QA](https://github.com/ROCm/rocm-cli/blob/main/docs/manual-testing.md) @@ -227,7 +236,7 @@ five tabs (switch with `Tab`/`Shift+Tab` or number keys `1`–`5`): | **Observe** | Live GPU utilization, instances, and benchmark telemetry | | **Chat** | Assistant chat backed by a local server or configured provider | -Live mode works on Windows too, but shows only GPU/CPU/memory — the +Live mode works on Windows too, but shows only GPU/CPU/memory. The serving-instance registry needs the Unix domain socket telemetry, so that view is Linux/WSL2 only. Use `rocm dash --demo` for a synthetic session that runs anywhere without a GPU or daemon. @@ -256,7 +265,7 @@ legacy install. ROCm CLI supports ROCm 7.14, 10.0, and 10.1. Without `--version`, `rocm install sdk` currently installs ROCm 7.14. To install ROCm 10.0 or 10.1, -add `--version`. If ROCm CLI can't tell which exact GPU arch this computer has, +add `--version`. If ROCm CLI can't tell which exact GPU architecture this computer has, or you want a different one, also add `--family` with the raw `gfx` code from `rocm examine`. For example: `--version 10.1.0 --family gfx1200`. @@ -325,7 +334,7 @@ compares the driver date with the release of the first HIP SDK for Windows (27 July 2023). AMD publishes no minimum driver version for the HIP SDK or for the Lemonade Vulkan backend, so this is a hint to look at the driver, not a statement that the driver is broken. The date is only a floor: it cannot judge -newer GPUs such as gfx120X, gfx1150 and gfx1151, which may need newer drivers. +newer GPUs such as gfx120X, gfx1150, and gfx1151, which might need newer drivers. `rocm diagnose` lists the same finding as `fix-22-display-driver-old` with a WEAK score. Its advice is AMD's support page and a restart after the install. The entry is advice about the host, not a @@ -383,35 +392,43 @@ position. address even though the report does not, and the CLI says so each time it offers one. -The `--report` content is deliberately narrow (a schema version, the matched entry, whether -a fix was offered for it, the GPU architecture and which compatibility -matrix snapshot it was checked against, the OS family, distribution and -major version, the ROCm release, the inference engine and its release, the -CLI version), and it carries no host name, user name, file path, or error -text. - -The ROCm release and the inference engine's release are each cut -back to a release, so a build number that would narrow toward one machine -never appears there; the CLI's own version is the exception, since it names -the tool that wrote the report instead of something read off the machine. -The distribution is checked against a list of known names instead of -being repeated from the machine. - -Hardware that is not on AMD's published -compatibility matrix produces no report at all, and the CLI says why. So -does a WSL machine, for a different reason: this CLI does not inspect the -GPU on WSL yet, so it cannot confirm the hardware is on the compatibility -matrix and says that instead of claiming the architecture could not be -read. - -With `--send`, a mail client opens only when you asked and the machine looks like -a desktop you are at; over SSH, with no display, or with `ROCM_NO_BROWSER` -set, the address and the link are printed instead, which is also what -happens on a machine with no mail client. It is not combinable with -`--json`, which exists for scripts, and a script is not a person who can -read a mail before sending it. Note that a mail carries your address, which -the report itself does not. The dashboard offers the same path on its Diagnose -screen: `p` shows the report, and `s` asks before it runs `--report --send`. +The `--report` content is deliberately narrow. It consists of: + +- A schema version +- The matched entry +- Whether a fix was offered for it +- The GPU architecture +- The compatibility matrix snapshot it was checked against +- The OS family, distribution, and major version +- The ROCm release +- The inference engine and its release +- The CLI version + +It carries no host name, user name, file path, or error text. Two values are +generalized first: + +- The ROCm release and the inference engine's release are each cut back to a + release, so a build number that would narrow toward one machine never + appears there. The CLI's own version is the exception, because it names the + tool that wrote the report instead of something read off the machine. +- The distribution is checked against a list of known names instead of being + repeated from the machine. + +No report is produced in two cases, and the CLI says why each time: + +- The hardware is not on AMD's published compatibility matrix. +- The machine is WSL. The CLI does not inspect the GPU on WSL yet, so it + cannot confirm that the hardware is on the compatibility matrix, and it says + that instead of claiming the architecture could not be read. + +With `--send`, a mail client opens only when you asked and the machine looks +like a desktop you are at. Over SSH, with no display, or with `ROCM_NO_BROWSER` +set, the address and the link are printed instead, which is also what happens +on a machine with no mail client. `--send` is not combinable with `--json`, +which exists for scripts, and a script is not a person who can read a mail +before sending it. A mail carries your address, which the report itself does +not. The dashboard offers the same path on its Diagnose screen: `p` shows the +report, and `s` asks before it runs `--report --send`. `fix` applies a known fix by the `id:` that `diagnose` reported, not the ranking position noted above, which isn't a stable name. Run it with no id @@ -467,8 +484,8 @@ To remove ROCm CLI and what it manages, see #### GPUs with no ROCm build Some AMD GPUs are detected but have no ROCm SDK build at all. The Vega iGPU in -Renoir, Lucienne, Cezanne and Barcelo APUs — the Radeon Vega graphics in Ryzen -4000 and 5000 mobile and desktop parts, reported as `gfx90c` — is one of them. +Renoir, Lucienne, Cezanne, and Barcelo APUs (the Radeon Vega graphics in Ryzen +4000 and 5000 mobile and desktop parts, reported as `gfx90c`) is one of them. For these, `install sdk` stops and says no ROCm SDK build exists for that GPU, instead of listing package families to try. It does the same when you name such @@ -535,7 +552,7 @@ active default runtime, not about what a named prefix loses. ROCm 10 and newer ship from a different source layout. You opt in by pinning the version with `--version`. These versions select their device payload by exact -GPU arch, so when ROCm CLI can't resolve one for this computer (it detects only +GPU architecture, so when ROCm CLI can't resolve one for this computer (it detects only a family of GPUs, or none), or you want to target a different GPU, also pass `--family` with the raw `gfx` code instead of a family label: @@ -574,12 +591,17 @@ you what it couldn't find instead. #### Driver installation -`install driver` installs the AMD kernel driver on Linux, using DKMS or a native -package. On bare metal it defaults to the floating `latest` amdgpu release; -set `ROCM_CLI_AMDGPU_DRIVER_VERSION` to pin a specific one. This replaces the -older `ROCM_CLI_AMDGPU_VERSION`/`ROCM_CLI_AMDGPU_PACKAGE_RELEASE` pair, which -pinned a fixed default and are no longer read; the plan is refused while either is still set, -so an old pin is never silently turned into `latest`. +`install driver` previews or installs Linux AMD driver support. On bare metal, +it prints a non-mutating plan and changes nothing unless you pass `--dkms`, +which plans the DKMS install; add `--yes` to apply it. On WSL2 there is no +DKMS flow: the command plans the ROCDXG install instead, and `--yes` applies it +without `--dkms` (see [Install ROCDXG in WSL](https://github.com/ROCm/rocm-cli/blob/main/docs/wsl.md#install-rocdxg-in-wsl)). + +On bare metal it defaults to the floating `latest` amdgpu release; set +`ROCM_CLI_AMDGPU_DRIVER_VERSION` to pin a specific one. This replaces the older +`ROCM_CLI_AMDGPU_VERSION`/`ROCM_CLI_AMDGPU_PACKAGE_RELEASE` pair, which pinned a +fixed default and are no longer read; the plan is refused while either is still +set, so an old pin is never silently turned into `latest`. On Ubuntu, the plan uses the `repo.radeon.com/amdgpu` suite matching the release (`jammy` 22.04, `noble` 24.04, `resolute` 26.04). If `amdgpu-dkms` from @@ -589,6 +611,11 @@ never removes the old driver itself, and the install continues after the warning. This applies to every apt-based distro the DKMS flow covers (Ubuntu and Debian 12/13). +On dnf hosts (RHEL, Oracle Linux, and Rocky Linux) and zypper hosts (SLES), the +plan follows the same repository-based flow: it imports the AMD key, writes a +package-manager repository file, and installs `amdgpu-dkms` with `dnf` or +`zypper`. + Approving (`--yes`) a plan that is `supported: false` (an undocumented distro, or a refused version override) installs nothing and exits non-zero, after printing the plan. @@ -602,7 +629,7 @@ with or without `--yes`, and it starts no driver command. It prints the AMD display driver version and date, and whether the date is older than the first HIP SDK for Windows, 27 July 2023 (`outdated: yes`, `no` or `unknown`). Next to `outdated: no` it repeats that the date cannot judge newer GPUs such as -gfx120X, gfx1150 and gfx1151. For an +gfx120X, gfx1150, and gfx1151. For an old driver it also names the fix id `fix-22-display-driver-old`. Run `rocm diagnose` for the advice. AMD publishes no minimum driver version, so an old date is a hint to look at the driver, not a finding that it is broken. See @@ -659,19 +686,21 @@ When nothing is left behind the count is `services_on_previous_runtime: 0`. A running server whose record names no runtime is counted separately under `services_with_unrecorded_runtime:`, because what it loaded cannot be read back from the record. Only servers on an engine that brings its own runtime -(`lemonade`) are never counted; an `env_id` recorded in a service record is not -a pin and does not exempt the server. The same summary appears in the -`rocm install sdk` and `rocm update --apply --activate` reports, which activate -a runtime without taking `--restart-services` themselves — which is why the note -names the `rocm runtimes activate` invocation that does, rather than a flag to -add to whatever was just run. +(`lemonade`) are never counted. An `env_id` recorded in a service record is not +a pin and does not exempt the server. + +The same summary appears in the `rocm install sdk` and +`rocm update --apply --activate` reports. Those commands activate a runtime +without taking `--restart-services` themselves, so the note names the +`rocm runtimes activate` invocation that does, instead of a flag to add to +whatever was just run. The note's command re-activates the runtime that is already active, which moves the servers without moving the runtime. Re-activating the active runtime prints `changed_from_runtime_key: ` and keeps the existing rollback target. Add `--restart-services` to move those servers onto the newly active runtime -rather than leave them behind; each one is put on the new runtime and then +instead of leaving them behind; each one is put on the new runtime and then restarted. Moving a server also clears any `env_id` stored in its record; the engine writes its own back on the next launch. That restarts running servers, so it requires `--yes` and never prompts, the same as @@ -701,8 +730,8 @@ failure is named in the summary and its record is put back on the runtime it last ran on. After a failure the server is read back: one that is still live stays counted under `services_on_previous_runtime`, and one that is not is reported as stopped by the attempt. The error says which is which. Read -`rocm services logs ` first — a restart that just failed is likely -to fail the same way again — then start a stopped server with +`rocm services logs ` first (a restart that just failed is likely +to fail the same way again), then start a stopped server with `rocm services restart --yes` once the cause is fixed. `runtimes uninstall` prompts `[y/N]` (default: no) unless `--yes` is passed. @@ -744,9 +773,9 @@ two are separate runtimes serving different purposes instead of newer and older versions of the same thing. Anything it declines to remove is listed with the reason, and `--dry-run` shows the whole plan without changing anything. -`remove-downloads` clears cached archives that ROCm CLI can download again; a -cache folder that is a link to somewhere else is left alone instead of -being followed. The two archive rows the report prints are tagged +`remove-downloads` clears cached archives that ROCm CLI can download again. A +cache folder that is a link to somewhere else is left alone instead of being +followed. The two archive rows the report prints are tagged `note: can be downloaded again; safe to remove`, so it is visible which rows `remove-downloads` acts on. @@ -852,16 +881,22 @@ rocm serve [--engine lemonade|vllm|aim] `--engine aim` runs the model in a Docker container instead of a host process; it requires a reachable Docker daemon and is Linux/WSL2 only. +`ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the 5-minute readiness wait for a +cold-start model download. + `--engine-env` passes an extra environment variable through to the engine's -own process/container (repeatable); currently only `aim` consumes these (e.g. -`AIM_PROFILE_ID`). `KEY=VALUE` values are visible in the detached child's -argv and in `rocm services list --json`, so do not put secrets there; pass a -bare `KEY` (e.g. `--engine-env HF_TOKEN`) to forward the value from your -current environment without storing it. A service restarted by the daemon -after a crash (with the `server-recover` automation enabled) inherits the -daemon's environment, not yours, so a forwarded secret must also be set -there. `ROCM_CLI_AIM_READY_TIMEOUT_SECS` raises the 5-minute readiness wait -for a cold-start model download. +own process/container (repeatable); currently only `aim` consumes these (for +example, `AIM_PROFILE_ID`). Handle secrets as follows: + +- `KEY=VALUE` values are visible in the detached child's argv and in + `rocm services list --json`, so do not put secrets there. +- Pass a bare `KEY` (for example, `--engine-env HF_TOKEN`) to forward the value + from your current environment without storing it. +- A service restarted by the daemon after a crash (with the `server-recover` + automation enabled) inherits the daemon's environment, not yours, so a + forwarded secret must also be set there. + +#### Ports and hosts `--port` is optional on the default loopback host (`127.0.0.1`): omit it and `rocm serve` picks the first free port in `11435`-`11535`, checking both @@ -869,19 +904,21 @@ rocm-cli's own records and the OS. The non-interactive plan output (piped output, or the Dash approval card) shows `Port: automatic; endpoint shown after launch` while the launch is in flight and prints the concrete endpoint once the port is actually leased; the interactive deployment summary table -only ever shows the final resolved endpoint. Pass `--port` explicitly to pin -one; on `127.0.0.1`, an explicit loopback port already reserved by another -managed service is rejected up front, naming that service and how to free it -(`rocm services stop --yes`). A port held by an untracked process -is rejected too, but without naming the holder. `--host` to anything other -than the default -loopback address (`localhost`, `::1`, `0.0.0.0`, or any other address) -requires an explicit `--port`: automatic selection only probes `127.0.0.1` and -cannot promise a free port anywhere else. An explicit `--port` on such a host -is refused when a live managed service already holds that port on the same -address (`localhost` counts as both loopback addresses) or on a wildcard -listener that covers it (`0.0.0.0` for IPv4 addresses, `::` for any); the OS is not probed on such a -host, so any other clash is reported by the engine when it starts. +only ever shows the final resolved endpoint. + +Pass `--port` explicitly to pin one; on `127.0.0.1`, an explicit loopback port +already reserved by another managed service is rejected up front, naming that +service and how to free it (`rocm services stop --yes`). A port +held by an untracked process is rejected too, but without naming the holder. + +`--host` to anything other than the default loopback address (`localhost`, +`::1`, `0.0.0.0`, or any other address) requires an explicit `--port`: +automatic selection only probes `127.0.0.1` and cannot promise a free port +anywhere else. An explicit `--port` on such a host is refused when a live +managed service already holds that port on the same address (`localhost` counts +as both loopback addresses) or on a wildcard listener that covers it (`0.0.0.0` +for IPv4 addresses, `::` for any); the OS is not probed on such a host, so any +other clash is reported by the engine when it starts. `rocm services restart` and `rocmd`'s automatic recovery are deliberately stricter about hosts than that launch check: they refuse to bring a service @@ -889,14 +926,17 @@ back on a port that any other live managed service has recorded, whatever host that service is bound to, because a `0.0.0.0` listener also covers the loopback address. +#### Other serve options + `--gpu-memory-utilization` (a fraction above 0.0 and up to 1.0 of each GPU's -total VRAM) and `--tool-call-parser` (for example `hermes`, `llama3_json`, or -`mistral`, which also turns on automatic tool choice) apply to vLLM only. Lower +total video RAM, or VRAM) and `--tool-call-parser` (for example `hermes`, +`llama3_json`, or `mistral`, which also turns on automatic tool choice) apply to +vLLM only. Lower the memory fraction when a small model shouldn't reserve most of a large card. `--require-api-key` makes a loopback endpoint require an API key too, which matters when something else republishes the port. A public bind generates a key -automatically unless you supply one. Set `ROCM_SERVE_API_KEY` rather than +automatically unless you supply one. Set `ROCM_SERVE_API_KEY` instead of passing `--api-key`, so the secret stays out of your shell history and the process table. @@ -909,6 +949,8 @@ llama.cpp's `--temperature`, `--top-p`, and `--n-predict` flags. Each control is optional and independent, so omit any of them to keep the engine's own default. +#### Reusing a running service + `rocm serve` only reuses an already-running service for the same engine and model if its sampling controls (and other recipe settings) match the ones requested this time; otherwise it errors out instead of silently serving with @@ -918,9 +960,11 @@ flags, or with different ones, stop the existing service first (`rocm services stop --yes`) or match the original flags. Likewise, an explicit `--port` that differs from the port the running service -is already on is refused rather than silently ignored; omit `--port` to reuse +is already on is refused instead of silently ignored; omit `--port` to reuse the running service, or stop it first to serve on the new port. +#### Foreground, background, and waiting + By default the server runs in the background under ROCm CLI's supervision and prints a deployment summary: a progress indicator while it starts, then a table with the status, the full inference endpoint, the API-qualified model name, and a @@ -933,9 +977,9 @@ terminal and streams every engine log line. Use it to debug a startup problem. The server still runs as a managed background process, so while streaming you can: -- Press **Ctrl-D** to detach. The log stream stops, your shell comes back, and +- Press `Ctrl-D` to detach. The log stream stops, your shell comes back, and the server keeps running. Manage it afterward with `rocm services`. -- Press **Ctrl-C** to stop the server instead. If the stop answers that it +- Press `Ctrl-C` to stop the server instead. If the stop answers that it could not confirm the server's process exited, the command prints the same report as `rocm services stop` on stderr and exits with an error. If the stop itself fails, the command says why on stderr, signals the server process @@ -945,16 +989,23 @@ can: `--wait` keeps `rocm serve` waiting until the server answers requests or its engine gives up (vLLM and AIM after 5 minutes by default; raise it with `ROCM_CLI_VLLM_READY_TIMEOUT_SECS` / `ROCM_CLI_AIM_READY_TIMEOUT_SECS`), instead -of returning `starting` after 45 seconds; if the server exits first (crashes, is killed, or is stopped with -`rocm services stop`), it fails with an error that points at the server's logs -instead of waiting forever. Ctrl-C ends the wait and leaves the server starting -in the background; a generated API key (public bind) is printed when the wait -begins, so interrupting the wait does not lose it. It cannot be combined with `--foreground` or `--verbose`, and -when the model is already being served and still starting, it waits on that -server instead (an already-serving one is reported at once). Watch the download with -`rocm logs --service --follow` from another terminal. +of returning `starting` after 45 seconds. While it waits: + +- If the server exits first (crashes, is killed, or is stopped with + `rocm services stop`), the command fails with an error that points at the + server's logs instead of waiting forever. +- `Ctrl-C` ends the wait and leaves the server starting in the background. A + generated API key (public bind) is printed when the wait begins, so + interrupting the wait does not lose it. +- If the model is already being served and still starting, the command waits on + that server instead. An already-serving one is reported at once. + +`--wait` cannot be combined with `--foreground` or `--verbose`. Watch the +download with `rocm logs --service --follow` from another terminal. `--no-smoke-test` skips the post-startup inference probe. +#### Choosing the model form + Which model form to pass depends on the engine your GPU selects. The Lemonade engine (Ryzen AI or Radeon) serves llama.cpp **GGUF** models. Pass a GGUF repo with an explicit quantization variant, for example, @@ -972,25 +1023,36 @@ does not read your environment on its own: export `HF_TOKEN` and pass `--engine-env HF_TOKEN` (see above), or `rocm serve` refuses a gated AIM image before pulling anything. -`--gpu` selects which AMD GPU the server runs on. `auto` (the default) probes -per-GPU VRAM (through `amd-smi`, or the amdgpu DRM sysfs counters when `amd-smi` -is not installed) and picks the lowest-numbered GPU that is idle and not already -used by another ROCm CLI server (managed or foreground), falling back to the GPU -with the most free memory. Pass a single index (`--gpu 1`) to pin a specific -device. An index the machine does not have is refused before anything starts, -and the error lists the indices that exist. On Windows the index is checked -against the AMD display adapters Windows reports (and against the `amd-smi` -count when that works, whichever is lower). That list can include an adapter the -ROCm runtime does not support, so on Windows the check can still accept an index -the runtime cannot use; it is a guard against a mistyped index, not a guarantee -the device is usable. If neither the Windows list nor `amd-smi` can be read, the -index is used unchecked and `rocm serve` prints a warning on stderr saying so. -If a visibility mask is set but cannot be applied (on Windows `HIP_VISIBLE_DEVICES` -holding an empty value, a UUID, or an ordinal past the adapters Windows lists; on -Linux a mask when the GPU lists could not be read but `amd-smi` still gives a -count), the index is still refused when it is at or past the GPU count, and one -below the count is accepted with a warning that the mask was not applied. When -`amd-smi` gives a count and no mask is set, the Windows list is not queried. +#### Choosing a GPU + +`--gpu` selects which AMD GPU the server runs on: + +- **`auto` (the default):** probes per-GPU VRAM (through `amd-smi`, or the + amdgpu DRM sysfs counters when `amd-smi` is not installed) and picks the + lowest-numbered GPU that is idle and not already used by another ROCm CLI + server (managed or foreground), falling back to the GPU with the most free + memory. +- **A single index (`--gpu 1`):** pins a specific device. An index the machine + does not have is refused before anything starts, and the error lists the + indices that exist. + +On Windows, the index check has these details: + +- The index is checked against the AMD display adapters Windows reports (and + against the `amd-smi` count when that works, whichever is lower). That list + can include an adapter the ROCm runtime does not support, so the check can + still accept an index the runtime cannot use; it is a guard against a + mistyped index, not a guarantee the device is usable. +- If neither the Windows list nor `amd-smi` can be read, the index is used + unchecked and `rocm serve` prints a warning on stderr saying so. + +If a visibility mask is set but cannot be applied (on Windows +`HIP_VISIBLE_DEVICES` holding an empty value, a UUID, or an ordinal past the +adapters Windows lists; on Linux a mask when the GPU lists could not be read but +`amd-smi` still gives a count), the index is still refused when it is at or past +the GPU count, and one below the count is accepted with a warning that the mask +was not applied. When `amd-smi` gives a count and no mask is set, the Windows +list is not queried. The selected GPU is exposed to the engine through `HIP_VISIBLE_DEVICES`. Serving one model across multiple GPUs is not supported. Because selection uses the @@ -1003,7 +1065,7 @@ Under the default `gpu_required` policy, `rocm serve` on a host with no usable AMD GPU fails fast with `no usable AMD GPU detected`, before any engine is prepared or downloaded. A `rocm serve` that matches an **already-running managed service for the same engine and model** now skips that refusal and -prints the reuse summary instead. This is a real change to a refusal you may be +prints the reuse summary instead. This is a real change to a refusal you might be relying on: on a GPU-less host, `rocm serve ` can now succeed where it previously always failed. @@ -1011,25 +1073,30 @@ The exemption is limited to that case, and nothing else about `gpu_required` changes. The reused service was vetted against the policy at its own launch, and the reusing invocation launches no process and does no GPU work. A live service for a different engine does not soften the refusal, and neither does one -for a model whose name does not overlap — those still fail fast with the usual +for a model whose name does not overlap. Those still fail fast with the usual message and prepare nothing. Matching the engine and model is necessary but not sufficient. The exemption applies only when the service would *actually* be reused, and that is decided by -exactly the predicate the launch itself applies: the recipe settings must match -(the sampling-control rule under [Model serving](#model-serving) above), a -`--require-api-key` request must be satisfiable by a service that already has a -key, and the service's recorded device policy must satisfy this one — a -`gpu_required` serve is not reused from a service recorded as `cpu_only`. That -is the only policy pair refused; any other combination, including a record that -predates the field and so names no policy at all, is accepted. (No `rocm serve` -can record a `cpu_only` service today — `--device cpu_only` is refused outright, -since CPU is not a fallback path here — so that term is a guard, not a refusal -you will meet.) A live service that fails any of those is not reusable, so it -buys no bypass: `rocm serve` refuses on the GPU instead, which is the accurate -message and leaves your running server alone. (Telling you to stop it and re-run -would not help: the re-run would have nothing left to reuse and would hit the -same GPU refusal.) +exactly the predicate the launch itself applies. All of these must hold: + +- The recipe settings match (the sampling-control rule under + [Model serving](#model-serving) above). +- A `--require-api-key` request is satisfiable by a service that already has a + key. +- The service's recorded device policy satisfies this one: a `gpu_required` + serve is not reused from a service recorded as `cpu_only`. That is the only + policy pair refused; any other combination, including a record that predates + the field and so names no policy at all, is accepted. (No `rocm serve` can + record a `cpu_only` service today. `--device cpu_only` is refused outright + because CPU is not a fallback path here, so that term is a guard, not a + refusal you will meet.) + +A live service that fails any of those is not reusable, so it buys no bypass: +`rocm serve` refuses on the GPU instead, which is the accurate message and +leaves your running server alone. (Telling you to stop it and re-run would not +help: the re-run would have nothing left to reuse and would hit the same GPU +refusal.) Two limits are worth stating exactly, because they are easy to read as stronger than they are: @@ -1038,63 +1105,69 @@ than they are: even worth asking about uses the same two-way substring match the service listings use, so a live `qwen3-8b-instruct` service opens it for `rocm serve qwen`. On a GPU-less host that invocation then does engine - preparation — possibly printing `Preparing for GPU serving...` and - installing — *before* the refusal it ultimately gets, and if that preparation - fails you get its error rather than the GPU one. Leniency is deliberate: - tightening it would silently break legitimate reuse by short name. Only a - model name that does not overlap is guaranteed to refuse first with nothing - prepared. + preparation (possibly printing `Preparing for GPU serving...` and + installing) *before* the refusal it ultimately gets, and if that preparation + fails you get its error instead of the GPU one. + + Leniency is deliberate: tightening it would silently break legitimate reuse + by short name. Only a model name that does not overlap is guaranteed to + refuse first with nothing prepared. - **The exemption does not apply with an explicit `--gpu `.** It skips - the no-usable-GPU bail, not the pinned-index check that follows it — and that + the no-usable-GPU bail, not the pinned-index check that follows it. That check refuses *every* index on a host with no usable GPU, because there is no visible ordinal for one to name. So `rocm serve --gpu 0` is refused on a GPU-less host whether or not a reusable service is running, and so is - `--gpu 1`, and so on. The exemption applies under the default `--gpu auto`. + any other index, such as `--gpu 1`. The exemption applies under the default + `--gpu auto`. #### When a serve runs out of GPU memory When a vLLM serve that **this command launched** fails to become ready and its -engine log shows an out-of-memory failure, the deployment summary adds a note +engine log shows an out-of-memory (OOM) failure, the deployment summary adds a note confirming the attempt ran out of GPU memory, naming the two knobs that can fix -it — `--gpu-memory-utilization ` and -`--gpu ` — warning that lowering the reservation will not help if the -model simply does not fit, and printing a +it (`--gpu-memory-utilization ` and +`--gpu `), warning that lowering the reservation will not help if the +model does not fit, and printing a `rocm diagnose --symptom ''` command for the full busy-GPU-versus-model-too-large breakdown. A failing line that cannot be placed -in a shell-quoted argument intact — one containing an apostrophe, for instance — +in a shell-quoted argument intact (one containing an apostrophe, for instance) is replaced by a canonical vLLM OOM symptom, which reports the same cause. -A line carrying **terminal control bytes** that an *escape sequence* is part of -— vLLM's colourised logger, or a repaint that moves the cursor with something -like `ESC[1A` or clears the row with `ESC[2K` — is the one case where the two -surfaces can print different commands for the same failure, and both are sound. -(A bare carriage-return repaint is not that case: the log is split on `\r`, so -the allocator line comes back as a clean segment carrying no escape and the two -surfaces quote it identically.) The engine's own failure -message refuses the raw line and falls back to the canonical symptom. The serve -summary reads the log *after* the engine has echoed its tail into it as the rows -a terminal would have drawn, so the note quotes that rendered row — real text the -terminal printed, carrying no control bytes — instead. See -[docs/vllm.md](https://github.com/ROCm/rocm-cli/blob/main/docs/vllm.md) for the -flag itself and the quoting rule each printed command follows. +One case makes the deployment summary and the engine's own failure message print +different commands for the same failure, and both commands are sound: a failing +line that carries **terminal control bytes**. These are escape sequences, such +as vLLM's colorized logger output, a repaint that moves the cursor (`ESC[1A`), +or one that clears the row (`ESC[2K`). The engine's failure message refuses the +raw line and falls back to the canonical symptom. The serve summary reads the +log *after* the engine has echoed its tail into it as the rows a terminal would +have drawn, so the note quotes that rendered row (real text the terminal +printed, carrying no control bytes) instead. + +A bare carriage-return repaint is not that case. The log is split on `\r`, so +the allocator line comes back as a clean segment carrying no escape, and the two +surfaces quote it identically. + +For the `--gpu-memory-utilization` flag and the quoting rule each printed +command follows, see the +[vLLM adapter page](https://github.com/ROCm/rocm-cli/blob/main/docs/vllm.md). The note is deliberately narrow, so its absence is not evidence that memory was fine. It appears only when all of the following hold: - The engine is vLLM. It is the only engine with `--gpu-memory-utilization`. -- You are looking at the **interactive deployment summary** — the default +- You are looking at the **interactive deployment summary**, the default background managed launch with a terminal on stdout. A piped, redirected, or otherwise scripted run prints the plain machine-readable form - (`readiness: `), which carries no *post-launch* notes — it still + (`readiness: `), which carries no *post-launch* notes. It still prints the pre-launch planning lines, including the low-VRAM `note:` when - amd-smi reports a tight card, so a `note:` in piped output is never this one — - and `--verbose`/`--foreground` streams the engine log in your terminal instead, - where the traceback is already in front of you. Either way, the log is on + amd-smi reports a tight card, so a `note:` in piped output is never this one. + With `--verbose` or `--foreground`, the engine log streams in your terminal + instead, where the traceback is already in front of you. Either way, the log is on disk: `rocm services logs `. - The service **failed to become ready** (recorded status `starting`). A `ready` service, or a `running` one whose endpoint is up while the model is still loading, is healthy and is never given memory advice. -- The log tail carries a real allocator failure — `HIP out of memory`, +- The log tail carries a real allocator failure: `HIP out of memory`, `CUDA out of memory`, `hipErrorOutOfMemory`, or a `torch.OutOfMemoryError` corroborated by an allocator message. A bare "out of memory" phrase from the kernel OOM killer or an unrelated subprocess does not qualify, and a failure @@ -1106,7 +1179,7 @@ fine. It appears only when all of the following hold: `rocm services logs ` instead. - **The failing line is still near the end of the log.** The summary reads a fixed tail of the engine log, and it reads it *after* the engine has appended - its own failure message — which echoes that same tail back — to the file. A + its own failure message (which echoes that same tail back) to the file. A launch that OOMs and then logs another ~80 lines of shutdown can therefore push the allocator line out of the window the summary sees, and no note appears even though the engine's own message (in `rocm services logs `, or in @@ -1116,8 +1189,8 @@ fine. It appears only when all of the following hold: If the pre-launch low-VRAM warning already printed the `--gpu-memory-utilization` advice for the selected GPU, the OOM note drops that -one sentence rather than repeating it; the rest of the note — the confirmation, -the model-too-large caveat, and the `diagnose` command — still appears. +one sentence instead of repeating it. The rest of the note (the confirmation, +the model-too-large caveat, and the `diagnose` command) still appears. Manage background servers started with `--managed`: @@ -1133,7 +1206,7 @@ rocm services prune [--older-than-hours | --any-age] [--dry-run] [--yes] `stop` reports a server stopped only once every recorded process is confirmed gone. If it cannot confirm that, it prints `Local server stop not confirmed` on stderr and exits with an error. It keeps the record, which still names the -processes, and any endpoint key, which the server may still be enforcing. It +processes, and any endpoint key, which the server might still be enforcing. It names each process that, or a process under it, did not exit after it was signaled; for a process it could not verify as the server's, it signals nothing and names an operating-system command to check on it. It does not suggest @@ -1233,9 +1306,9 @@ For `rocm dash`: - `--demo` runs a deterministic synthetic session with no GPU or daemon needed. It works on all platforms. -- `--replay ` replays a recorded NDJSON session. -- Live mode works on Windows too, but shows only GPU/CPU/memory — there's no - `rocmdashd` unix socket to connect to there, so the serving-instance +- `--replay ` replays a recorded newline-delimited JSON (NDJSON) session. +- Live mode works on Windows too, but shows only GPU/CPU/memory. There is no + `rocmdashd` Unix socket to connect to there, so the serving-instance registry view stays empty (Linux and WSL2 get the full live view). ### Bench @@ -1257,7 +1330,7 @@ Flags and output for `rocm bench load`: - `--endpoint` is the OpenAI-compatible URL shown by `rocm services list` (a plain host address without `/v1` also works). Only `http://` is accepted; `https://` endpoints are rejected outright, because the load generator has no - TLS backend compiled in. + Transport Layer Security (TLS) backend compiled in. - `--concurrency` sweeps a comma-separated list of levels (default `1,8,32,64`, each 1-128); `--auto-ramp` ignores `--concurrency` and instead ramps `1,2,4,8,16,32,64,128` automatically, stopping early once generation @@ -1265,11 +1338,11 @@ Flags and output for `rocm bench load`: - `--isl` and `--osl` (input and output sequence length, default 1024 each) accept 1-32768, and `--requests` (default 128) accepts 1-10000. - Results are written to `--out` (default `/bench/results.csv`, - where `` is `~/.rocm` unless overridden), intended to match the - path the daemon tails to feed the dashboard's **Observe** tab. The CLI's - default output path and the daemon's tailed path are computed - independently, so if either the CLI's data dir or the daemon's - `bench_results_dir` config has been customized, confirm they still point + where `` is `~/.rocm` unless overridden). This is intended to + match the path the daemon tails to feed the dashboard's **Observe** tab. The + CLI's default output path and the daemon's tailed path are computed + independently. If either the CLI's data directory or the daemon's + `bench_results_dir` config has been customized, confirm that they still point at the same file. ### Chat @@ -1328,12 +1401,12 @@ folder that contains the `models path:` it prints): - **Kept:** `models/` (the folder `models-path` prints), `user/` (saved workflows and settings), `output/`, `input/`, `custom_nodes/`, `datasets/` and `extra_model_paths.yaml`. They are never moved, copied or deleted. Files - a new release adds inside them are added alongside yours; where both have a + a new release adds inside them are added alongside yours. Where both have a file of the same name, yours is kept, and a folder you replaced with a symlink is not written into. - **Set aside:** an entry of yours whose name the new release starts shipping (an `app/` of yours when a release adds `app/`, say) is renamed to - `.rocm-cli-kept-