Repository navigation
Backport main since v0.3.0-rc.2 to release/v0.3.0 - #1122
Merged
juhovainio merged 32 commits intoNov 17, 2026
Merged
Conversation
…rix Halo Ubuntu (#1058) (#1097) The row marked serve-aim-lifecycle-parity as an expected failure on native Linux Strix Halo. It passed on both channels of the 2026-11-15 and 2026-11-16 nightlies, which the harness reports as a stale XPASS and fails the lane on (exit 1), so the lane has been red for that alone. Signed-off-by: juhovainio <juho.vainio@amd.com> (cherry picked from commit 2980faf) Signed-off-by: juhovainio <juho.vainio@amd.com>
…1043) * fix(serve): keep endpoint API key files owner-only on every platform (ROCMAI-455) A served endpoint's API key file was created 0600 on Unix only when it did not exist yet: re-storing a key over an existing file truncated it in place, so a key file whose mode had been loosened (for example to 0644) stayed readable by others. On Windows the file was written with default permissions and inherited the data directory's ACL, so any account that could read the data directory could read the key. Add rocm_core::owner_only_file::open_new_owner_only, which creates a new file only the current user can open: mode 0600 on Unix, set again on the handle so the umask cannot leave it unreadable to its owner; on Windows a protected DACL granting only the process token's user, applied by CreateFileW itself so nothing is inherited. It never opens an existing file or symbolic link. Add atomic_write::write_owner_only_file_atomically for secrets: it stages the bytes in such a file and publishes by rename alone, so the old file's mode or DACL is never carried over and a symbolic link at the destination is replaced rather than written through. Endpoint keys are now stored with it, so a re-store repairs a loosened key file, and the remote session key write uses open_new_owner_only, gaining the Windows DACL. Docs and comments say "owner-only" instead of "0600", and scenario serve-59 checks the mode of a real serve's key file on the Linux GPU lanes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(serve): remove staged key copies, harden Windows open (ROCMAI-455) A store killed between staging the endpoint key and renaming it into place left a plaintext copy beside it (<id>.endpoint-key.tmp-<pid>-<millis>-<n>) that nothing ever removed. Clearing a key on stop, in both rocm and rocmd, now removes such copies too; `rocm services remove` and `prune` delete and count them with their record, and prune sweeps a copy whose record is gone under the usual age gate. A copy is recognised only by the exact suffix the atomic writer produces, so a service id crafted to look like one cannot get another service's key deleted. A store racing a clear for the same service now fails instead of bringing the cleared key back. On Windows the owner-only create passed neither FILE_FLAG_OPEN_REPARSE_POINT nor a long-path name, unlike std's create_new: CREATE_NEW could follow a dangling symbolic link and create the file at its target, and paths past MAX_PATH failed. It now does what std does on both counts. Add Windows tests for the symlink and long-path cases, a negative control for the DACL check, and DACL checks of the endpoint and remote session keys the apps store, through a test-support helper enabled only as a dev-dependency. Extend scenarios service-cleanup-01 and -03, and bring README, the services remove help and the docs in line, including that on Windows a key-protected AIM serve stores its key but cannot start. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(serve): pin the owner-only create mode and correct key-cleanup claims (ROCMAI-455) The owner-only file is meant to be unreadable to others from the moment it exists, not only once its mode is set again on the handle. Nothing checked that: a create that asked for a wider mode would still have passed every test, because the follow-up reset hid it. Split the create into its own function and test the mode it leaves under umask 0, in a re-executed child so the umask change cannot reach other tests. No behaviour changes. The cleanup comments overstated what removing a service's key does. The previous commit's message and the doc comments said a store racing a clear fails instead of bringing the cleared key back, fail-closed. That is not guaranteed: the cleanup removes the key and the staged copies present when the folder is listed, best-effort, and a concurrent store for the same service may still publish a key or may fail. The comments now say only that, and that an unlistable folder or an unremovable file is left in place without being reported. docs/testing.md now names the unit tests that fail without the repair of a loosened key file, and says serve-59 guards only the fresh-create path, which held before the repair too. docs/manual-testing.md states that what happens to a key-protected AIM serve's key file on native Windows when its server fails to start has not been verified, and keeps the remote session key as the check there. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(core): pin owner-only publish and staged-key paths (ROCMAI-455) Several properties the fix relies on held only by inspection. The owner-only writer must never publish through a step that can fall back to ReplaceFileW, which would carry the replaced file's DACL over to the secret; a test-only counter at every such step now shows it is never entered, and that the general writer is counted. The umask-0 "as created" check went through a helper that production no longer had to call; it now drives the real create through a test-only switch that skips only the handle mode reset. Staged-name recognition is pinned to reject empty digit runs. staged_siblings read the empty parent of a bare relative name and so found nothing; it now lists the current directory for it and returns names in the form it was given, tested in a scratch directory via a re-executed child so the working directory change stays contained. service-cleanup-03 now also plants a staged key copy for the record prune removes and checks it goes with the record, so that path has scenario coverage and not only a unit test. The serve-59 step uses the bounded serve runner, so a serve that never returns fails the scenario instead of stalling the lane, and its comments state the fixed readiness wait rather than claiming readiness is not awaited. Comments on the restart key handoff and the Windows long-path conversion are corrected to match the code. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 778c970) Signed-off-by: juhovainio <juho.vainio@amd.com>
… review fixes, skip tags and commit messages (#1076) * docs(contributing): check code-behavior claims in comments (ROCMAI-41) A comment or doc line that states how the code behaves is easy to accept, and no test reads it, yet a decision such as narrowing a test's scope or dropping a guard often rests on one such sentence. AGENTS.md section 3 now requires checking each such claim against the code it describes. Where the claim is the reason for a decision, it names the place as a file and a function, constant or test rather than a line number, which goes stale; a claim about timing or ordering names where the enforcing mechanism is defined. It also prefers an invariant to a count, and asks that a sentence rewritten for a review finding be checked the same way. Section 11 tells reviewers to treat such lines in a diff as claims to check, the pre-flight checklist gains a matching item, and CONTRIBUTING.md and the pull request template point at section 3. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(contributing): name a blocking objection's mechanism (ROCMAI-131) An objection resting on a timing, ordering, concurrency or "this cannot happen" argument is hard to answer when it does not say which mechanism it assumes. AGENTS.md section 11 now asks a reviewer to name that mechanism and where it is defined before marking such an objection blocking, and to post it as a question when the mechanism cannot be located. CONTRIBUTING.md's list of what reviewers look for points at the rule. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(contributing): review a review-fix commit as a whole (ROCMAI-44) A commit answering review feedback is written in a narrow window and gets the least scrutiny, so it is where new defects arrive. AGENTS.md section 11 gains a Review-fix commits group. The author diffs such a commit whole against the head the reviewer saw, and says whether it changes behavior outside the finding. Once the base has moved into the branch, the author uses git range-diff instead, and checks whether each commit it marks removed is the base's old history or a commit dropped since the review. A reviewed head a force-push removed is fetched by its full SHA. The commit does not widen what a documented user-facing option means or authorizes; a needed widening goes in its own commit with the option's help text and docs. A reviewer diffs such a revision whole as well, section 12's self-review reminder names these rules, the pre-flight checklist gains a matching item, and CONTRIBUTING.md and the pull request template point at section 11. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(contributing): treat e2e skip tags as deletions (ROCMAI-161) A tag that makes an existing scenario skip on the blocking mock e2e job removes its pre-merge coverage, yet the diff shows only a tag and no coverage floor catches it. AGENTS.md section 8 now treats moving a scenario off that job, in any change including CI plumbing, as a deletion: the PR text names, for each direction the behavior could be broken in, which blocking check still fails. A scenario that is the only pre-merge guard for a seam says so in a comment beside it, and a neighbor it slows or destabilizes is fixed at its own layer rather than by moving the guard. Section 3's gated-lane bullet points such moves at section 8, the pre-flight checklist gains a matching item, and CONTRIBUTING.md and the pull request template point at section 8. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(contributing): review commit messages for accuracy (ROCMAI-182) Pull requests here are normally squash-merged with the branch's commit messages as the default squash body (rebase-merge is also enabled; either way each commit message lands in public history), and that history is not rewritten. AGENTS.md section 5 now requires each commit message to describe what its own commit does, in words still true at the branch head. A message whose words a later commit made misleading is reworded before merge, and a commit whose whole subject a later commit removed is folded into that later commit or dropped; either needs an approved force-push. Section 11 tells reviewers to read the commit messages with the diff, the pre-flight checklist gains a matching item, CONTRIBUTING.md gains a short section on why commit messages are reviewed for accuracy, and the pull request template gains a checkbox for it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(contributing): scope the code-claim rule to the change (ROCMAI-41) The AGENTS.md section 3 rule, its pre-flight item and the CONTRIBUTING.md reviewer bullet said every comment or doc line stating how the code behaves is checked against that code, with no scope. Read on its own, that asks a change to re-verify every existing comment in a file it touches. They now cover the lines a change adds or edits, matching the section 11 reviewer bullet ("in the diff") and the pull request template ("this PR adds or changes"). Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 3a08d6b) Signed-off-by: juhovainio <juho.vainio@amd.com>
…t (ROCMAI-454) (#1039) * fix(rocmd): stop the engine child when the supervisor cannot record it (ROCMAI-454) std::process::Child does not kill on drop, so a failing engine_pid or ready write after the engine was spawned left the engine running with no supervisor, holding the GPU. When the failed write was the one recording the PID, the record did not name the engine either, so `rocm services stop` had nothing to stop and reported success. A guard now terminates the engine's process tree through rocm-core's terminate_recorded_pid_within and reaps it whenever the supervisor leaves before waiting for it, on an error return or a panic. A waited-for child is not signalled. terminate_recovery_supervisor shares the same terminate_and_reap helper. On Windows that adds a taskkill /T over the supervisor's descendants, which the previous root-only kill did not reach. The existing recovery-cleanup test re-executed itself with a filter naming a stale module path, so it matched zero tests and the child exited at once, which the test read as a kill. The filter is now derived from module_path!() and the spawner waits for a marker proving the child is sleeping. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(rocmd): report unconfirmed terminations and cover the engine's process tree (ROCMAI-454) The engine guard and the recovery-supervisor cleanup now report a termination they could not confirm instead of discarding the outcome, and the tests exercise the shapes the earlier change only asserted for a single process. The kill tests now terminate real process trees: the sleeper can start a grandchild, and the stand-in engine runs a shell with a child under it, so the tests fail if the termination stops reaching below the child. The termination step and the report sink are injectable, which lets the tests hand the reap a TimedOut or Unverified outcome a real child cannot produce: there the child is not reaped with a blocking wait and is otherwise left alone, and a confirmed stop is still waited for and reaped. A termination that could not be confirmed is now reported. The engine guard writes "rocmd: terminating the engine (pid N) ended <outcome>: ... may still be running" to its stderr, which is the service log, and the recovery path attaches the same line to the restart error the daemon records. Both failure arms of the restart go through one helper, abandon_recovery_supervisor, so neither can terminate without reporting. The line is written so that a failed write cannot panic inside Drop. A failed wait leaves the guard armed; the decision moved into settle_wait so a test can hand it a failed wait. The engine's log is emptied and then reopened in append mode, so a surviving engine cannot write over a line the supervisor appended to the same file. Doc comments now say the reap is conditional on the termination outcome. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(rocmd): pin the guard's default report and the per-run log truncation (ROCMAI-454) The engine guard's report goes to the supervisor's stderr when it is given no writer, but only injected writers were tested. A new test runs the guard in a child test process and reads that process's real stderr back, so the production default is exercised rather than assumed. Each supervised run empties the service log before the engine writes to it, and nothing pinned that. A test now seeds the log with a line from an earlier run and checks it is gone and this run's output is there. The PID write and the ready write now carry distinct error contexts, and each guard test asserts which write failed, so a test cannot pass by failing at a different record write than the one it targets. The stand-in engine waits in these tests are bounded, and the tests register the engine-spawn seam, so a run that times out kills its engine instead of leaving it sleeping. Comments and the architecture row are narrowed to what supervise.rs itself does. An external kill of the supervisor (a signal, the OOM killer) bypasses the guard; that is noted as not covered and tracked as ROCMAI-957. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(rocmd): pin the already-exited reap and publish the sleeper marker atomically (ROCMAI-454) A new Linux test hands TimedOut and Unverified to a child that has already exited and asserts the child is reaped, pinning the try_wait in terminate_and_reap_with. The test first waits for the exit with waitid(WNOWAIT), which observes it without reaping, so the child is certainly an unreaped zombie when the call is made and there is no timing race. The test sleeper now publishes its marker by writing a sibling file and renaming it into place, and the spawner waits until the marker holds a parseable grandchild PID. A half-written marker can no longer panic the test and leave a sleeper tree running. supervise_on_stand_in_rocm now tells a panic in the supervisor thread (its result channel disconnected) apart from a timeout, so the failure message points at the panic instead of claiming the supervisor hung. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(rocmd): say why the engine guard's termination cannot panic (ROCMAI-454) The guard terminates the engine from Drop, which can run during an unwind, where a panic aborts the process. Say at the call that the termination path under it does not panic, and how long it blocks on each platform. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit f95135c) Signed-off-by: juhovainio <juho.vainio@amd.com>
…er the default path in e2e (ROCMAI-480) (#1038) * docs(install): say the release signing key is pinned and enforced (ROCMAI-480) The installer comments and release-trust.md still described the release signing key as unpublished and signature verification as opt-in. The key is pinned in both installers, and every install verifies the detached signature against it, so a missing .sig sidecar fails the install. The docs now list only the trust roots that are still unpublished: the runtime metadata signing key and the model recipe index signing key. The same stale claim is fixed in testing.md and aim-catalog.md. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(install): cover default signature verification in the lifecycle lane (ROCMAI-480) Every existing lifecycle install sets ROCM_CLI_REQUIRE_SIGNATURE=1, so a regression that only bites when nobody asks for a signature would pass the whole lane. Two Linux scenarios now run as a user does: lifecycle-36 installs a specific version with the requirement turned off and still rejects a bundle signed by an untrusted key, and lifecycle-37 runs a release install with the variable unset and still fails on a missing .sig. The harness gains a per-scenario signature-requirement setting, takes the channel as a parameter, and no longer lets an inherited ROCM_CLI_SIGNING_PUBLIC_KEY_PATH/_PEM stand in for the pinned key. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(install): describe signature verification and trust-root slots exactly (ROCMAI-480) Review found several statements that were broader or vaguer than the code. Release and nightly CI sign every bundle archive the installers download, but the nightly release also carries unsigned convenience copies; the installers verify every install only while a release key is pinned, and the release-channel floor changes no outcome today; verify-pinned-keys also requires unpublished slots to stay empty and cross-checks a CI-configured signing key. Say each of those as it is, and drop a line-number reference to install.sh that had already drifted. The installer changes are comments only. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(install): address review on signature-verification wording (ROCMAI-480) Reflow the lines a partial edit left ragged, and say plainly what the new lifecycle scenarios do not cover: the missing-.sig case with nothing asked for runs on the release channel only, the Windows default path has no scenario, and lifecycle-36 cannot tell a dropped channel argument from a working one because the release-channel floor would force the same rejection. Name the follow-up ticket in the installer comments instead of calling the fail-closed change "planned". The installer changes are comments only. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 058fa6e) Signed-off-by: juhovainio <juho.vainio@amd.com>
…a removed rollback target from the active marker (#1106) * fix(runtimes): clear a removed rollback target from marker (ROCMAI-184) Uninstalling the runtime that is the rollback target cleared config.previous_runtime_key but left data/runtimes/active.json naming it, so storage retention kept holding a key for a runtime that no longer exists. Uninstall now clears it from the marker too. The marker writer moves next to ActiveRuntimeMarker so activation and uninstall share the one atomic write; the marker keeps naming the runtime in use, and a rollback target that names another runtime is left alone. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): cover ROCm 7 and ROCm 10 runtimes side by side (ROCMAI-184) Runtimes installed before the ROCm 10 layout and runtimes installed from it coexist on one machine, and the CLI has to keep reading the older record as it was written while it lists, switches between and removes the newer ones. Nothing exercised that. Mock-lane scenarios runtime-lifecycle-19/20/21 plant a ROCm 7 runtime installed before the ROCm 10 layout plus ROCm 10 release and nightly runtimes for one GPU family. They list, activate each in turn, uninstall one runtime and then the active one while the others' folders and records stay untouched, and refuse a runtime id shared by two runtimes with both keys named. The Windows release lane also runs them, so the Windows paths are covered on a required lane. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(runtimes): describe every active-runtime marker writer (ROCMAI-184) The comments called write_active_runtime_marker the marker's one writer, but a failed activation's restore writes or removes the marker directly, and uninstalling the runtime in use and `rocm config set-default-runtime`/`clear-default-runtime` remove it. The registry module doc also said uninstall only reads and clears the marker, when it now rewrites the rollback target too. Name each of them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 54df940) Signed-off-by: juhovainio <juho.vainio@amd.com>
…3) (#1045) * fix(lemonade): write service state atomically under a lock (ROCMAI-453) A Lemonade service's state file has several writers: the serve wrapper, a healthcheck latching that inference was verified, and rocmd publishing the Anthropic endpoint. Each merge read the file, patched it and wrote it back in place with no lock, and a file it could not parse was silently replaced by the patch alone. Two merges racing could drop each other's keys, and a reader catching a write half-done, or a merge into a torn file, could lose server_pid, so a later stop targeted the wrapper instead of the server. rocm-core now owns the state-file writers. merge_json_state_file and the new write_json_state_file hold one cross-process lock per state directory (.json-state.lock) across the whole read-modify-write and publish through the shared atomic writer, so a lock-free reader always sees a whole document. A merge into a file that is empty, not valid JSON or not a JSON object is refused with a typed JsonStateFileRefused and leaves the file byte for byte as it was; an absent file is still created. Every Lemonade state write goes through these writers. The serve paths need no change: their existing guards stop the server and try to mark the state failed when a merge fails. The router test that makes the ready merge hit a directory now expects the refusal the merge reports when it cannot read the file. Hardening what happens when a running server cannot be recorded or marked failed is left to a follow-up. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(rocmd): retry the endpoint publish on a torn state file (ROCMAI-453) rocmd records the Anthropic endpoint by merging into the engine's state file. The merge now refuses a file it cannot merge into rather than replace it, and the vLLM and AIM engines still rewrite their state files in place, so the publish can catch one empty or cut short for a moment and fail although the file is whole again an instant later. Retry the publish only for that refusal (empty, or not valid JSON), up to 5 attempts 200 ms apart. A whole document that is not an object, or a failure to lock, read or write the file, is returned at once. The error names the state file and says how many attempts were made. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(lemonade): prove the torn-state refusal and read lock (ROCMAI-453) The ready-merge failure test used a directory where the state file belongs, so it exercised a read error rather than the refusal of a torn file, and pinned the "left it untouched and recorded nothing" message without checking the file. It now tears the state file and asserts the typed NotJson refusal together with the bytes left on disk. Whether a merge reads the existing file under the state directory's lock was only caught by a timing-dependent concurrency test. The read now goes through an injectable seam, and a new test observes the lock held at the moment of reading, so moving the read before the lock fails deterministically. Document that AIM's write_state, like vLLM's, takes no lock and writes in place. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(lemonade): say a torn state file is left unmarked (ROCMAI-453) The router serve comment, the direct server's supervisor doc, the serve_http_resolving doc and the router_serve.rs architecture entry said a failure after the spawn always marks the state `failed`. Since merges refuse an empty, torn or non-object state file, that mark is refused too and the file is left as it was, which the router test already asserts. Say so in each place. No code changes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 2c28f39) Signed-off-by: juhovainio <juho.vainio@amd.com>
…called usable (#1092) * fix(vllm): make vLLM's pinned tensorizer import before an install is called usable vLLM's wheel pins tensorizer==2.10.1, which raises KeyError: 'crypto_stream_salsa20_NONCEBYTES' at import under Python 3.14. The ROCm 10.1.0 vLLM build is published for Python 3.14 only, so the pin cannot be avoided, and `rocm engines install vllm` reported its dependencies satisfied and the first `rocm serve` died in the engine core. After installing, and on every later install of a managed runtime, the engine imports tensorizer.serialization with the runtime's Python under the environment the server is launched with (including PYTHONPATH; not python -I, which would drop it) and with colour switched off, via the shared bounded subprocess executor (five-minute timeouts for the probe and the upgrade). A failure counts as tensorizer's own only when the last traceback frame is inside the tensorizer package and the exception is not a library-load or missing-module error; torch and shared-library failures are left to the device check. Colour escapes are stripped before the traceback is read. When the import is broken, tensorizer==2.12.1 is installed with --no-deps and imported again. When this run installed vLLM (a first install, --reinstall or a repair), a failed upgrade, a probe that cannot run, or a tensorizer that still cannot be imported fails the install with the exception. On a runtime this run did not install vLLM into the same three cases are warnings, so an install that used to be a no-op (offline, say) stays one. A tensorizer that imports keeps vLLM's pin. The upgraded release no longer matches vLLM's exact pin, so `uv pip check` reports it. The repair assessment and the CLI's dependency_check treat vLLM's own unmet tensorizer pin, with an installed release at or above the fixed one, as an intended divergence: no reinstall loop, and dependency_check is expected_divergence with a note naming the release the runtime holds. The note claims only what is checked; the rule does not look at the Python version. A reinstall re-checks and upgrades tensorizer again, so it stays the advice beside a genuine violation, unlike a torch divergence. Known limits: - the check runs inside the engine install, before the CLI realigns torch, so a torch that cannot load at that moment reads as not tensorizer's and the next install checks again; a reinstall's re-upgrade depends on that too; - a no-op install on an existing managed runtime now imports torch and tensorizer under the launch environment, which can write compatibility shims; - on the `rocm install sdk` auto-install path a refused tensorizer upgrade can still print `dependency_check: satisfied` beside the refusal error, because a refused upgrade keeps vLLM's own pin and the metadata check is clean; the refusal text carries the truth and that output is deliberately not changed here; - the upgrade is fetched from the default package index. Signed-off-by: juhovainio <juho.vainio@amd.com> * test(e2e): cover the vLLM tensorizer check through rocm engines install Fourteen Linux scenarios (ungated, no GPU or network) run the built CLI against a planted managed runtime whose Python and uv are shell stand-ins sharing state files, so each message is asserted together with the state it describes: - the broken pin is upgraded, the warning, expected_divergence and the note's release are printed, and the runtime holds 2.12.1 (a coloured traceback, and a PYTHONPATH only the server's launch environment provides, included); - a second install neither upgrades again nor reinstalls vLLM; - a missing library, or a non-library exception raised below tensorizer, is left alone, and the probe is shown to have looked; - an upgrade that cannot be fetched or does not help warns on a runtime this run did not install into, and fails a first install or a reinstall, saying so; - repairing a genuine violation beside the upgraded tensorizer clears it and upgrades tensorizer again, and the reinstall the CLI advises, taken from the printed advice line, does the same. docs/testing.md names how to run them (cargo xtask e2e -- -n 'vllm-tensorizer') and docs/manual-testing.md adds the hand step for the real Python 3.14 import, which no scenario covers. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): print an upgrade failure once, flatten the divergence note, link docs - A failed tensorizer upgrade on an install that just installed vLLM built its message with the cause already interpolated and then chained the same error as the source, so the alternate Display printed "network unreachable: network unreachable". The message now carries the cause once, and the test counts it. - The divergence note prints the installed release parsed from a `uv pip check` line, so it goes through sanitize_log_value like the other values in that report, and a test shows a newline in the version is flattened. - docs/testing.md cites install.rs, repair.rs and runtime_install_finalize.rs as relative links, so the link check can see a moved file. - The stand-in uv comment says why the repair scenario does not require --reinstall: the automatic repair does not pass it, and the scenario whose violation only a reinstall clears is the one that does. Signed-off-by: juhovainio <juho.vainio@amd.com> * test(vllm): pin the tensorizer probe's remaining branches, fail closed on an unreadable pin - The library-load rule has four independent markers; a table-driven test now raises an exception with tensorizer as the last frame and exactly one marker each (OSError, ImportError, ModuleNotFoundError, 'shared object file' and 'DLL load failed' wrapped in another type), so deleting any one turns it red. - The post-upgrade re-probe test asserts the second probe happened. - The tensorizer override rule requires the pin to read as a plain release strictly below the fix; an unreadable, empty or marker-qualified pin, or one at or above the fix, stays a violation instead of passing as the override. - The repair assessment names a torch and a tensorizer divergence in separate notes, each under the 200-character limit the neighbouring tests enforce. - The e2e stand-in Python loses PYTHONPATH under -E as well as -I (and combined short flags such as -sE), so scenario 07 goes red for any of them. - docs: the unit-test command selects by module and is counted, the scenarios are described as gated only on @requires-os:linux, the manual step covers hosts where rocm install sdk installs vLLM itself, and vllm.md says plainly that a retry after a failed install only warns. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(rocm-core): read a tensorizer version only in its plain shapes The release parser behind the tensorizer override rule accepted any text after a post segment and ignored everything after it, so a pin such as 2.10.1.postgarbage or 2.10.1.post1.foo counted as the plain release 2.10.1 and could pass as the deliberate override: fail-open, the opposite of the direction the rule is meant to fail in. A version now reads as a release only when it is dotted ASCII-digit components, optionally ending in one post<digits> component, optionally followed by a +local segment. A bare post, text after the post component, a pre-release, a marker, a dash or space, a non-ASCII digit, a component too large for u64 and the empty string have no release, so a pin of that shape stays a violation and an installed version of that shape is not fixed. Leading zeros read as the number and trailing zero components are kept (a longer release compares greater than its prefix, which already gives the right answer against the fix). Tests cover post<digits> with and without +local, the rejected post shapes on the pin and installed sides, trailing and leading zeros, and the unreadable shapes. This also corrects the count in the previous commit's message: the library-load rule has five markers, not four, and its table has five cases. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(rocm-core): require a well-formed local segment; pin the repair rule and the stand-in's flags - A tensorizer version's +local segment must now be non-empty ASCII letters, digits and dots with no empty component and no second +. Real locals such as +rocm10.1.0.g98dff2a81.d20261002 and the short +local, +x, +1 still read; +, +a+b, +a..b, +a-b and a non-ASCII digit do not, on the pin side (stays a violation) or the installed side (not fixed). Because of that, a version with control characters is a plain violation, and the CLI test that exercised the divergence note's flattening now checks the violation line instead. - The release parser's separate digits-only guard was redundant with u64 parsing (an equivalent mutant), so it is removed and a comment says that parsing is the guard: it rejects empty components, signs, non-ASCII digits, other text and overflow. - A repair-level test pins that a marker-qualified, unreadable or post-garbage pin forces a reinstall instead of passing as the deliberate override. - The 200-character assertion now covers only the two headlines; the expected-divergence lines carry the raw uv text and are not bounded by it. - The e2e stand-in Python no longer reads an option's value, such as -Wignore::ImportWarning, as -E or -I, so it drops PYTHONPATH only where a real interpreter would; -E, -I, -sE, -Es and -sI still turn scenario 07 red. Signed-off-by: juhovainio <juho.vainio@amd.com> * test(vllm): share the uv check frame, correct stale comments, pin the discover PYTHONPATH entry - The tensorizer override rule splits a uv check line on the same constant violation_subject uses (now crate-visible) instead of repeating the literal; a test builds a line from the constant, and changing the constant turns it red. - Two comments that still described only the torch case are corrected: the ExpectedDivergence doc now says a reinstall would undo a torch divergence but not a tensorizer one (the install upgrades tensorizer again), and the nothing-to-repair comment in repair_from_violations covers both intended divergences. - The e2e module doc lists every file the stand-ins share and the marker files that pick their behaviour; the refusal step says this run installed vLLM, which holds for a first install as well as a reinstall. - Scenario 07 now needs the exact entry the server's launch environment prepends to PYTHONPATH, and the run exports a decoy PYTHONPATH that replaces any host value, so dropping the prepend, dropping the launch environment or running python -I each turn it red, independent of the host. Signed-off-by: juhovainio <juho.vainio@amd.com> * test(vllm): describe the stand-ins' shared state files exactly The module doc said the scenario planter only reads tensorizer-version, but it also seeds it, as the already upgraded release, for the two scenarios that plant a violation beside the upgraded tensorizer, and the stand-in uv writes bin/vllm on every vLLM install, not only when the file was missing. Signed-off-by: juhovainio <juho.vainio@amd.com> --------- Signed-off-by: juhovainio <juho.vainio@amd.com> (cherry picked from commit cedbced) Signed-off-by: juhovainio <juho.vainio@amd.com>
…p the queue's GPUs in the wiki (#1063) * ci: retire the self-hosted nightly lanes that never get a runner The R9700, MI350P and W7900 nightly jobs have sat queued for hours and been cancelled since at least 11-09. Remove them from nightly.yml and from the consolidated report that waited on them. These lanes now run on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu, Windows, WSL2, all on the Strix pool since #1018) are unchanged. Update the workflow contract tests and the docs to match. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs,test: take main's wording where #1066 and this change touch the same lines Preparation for merging main into this branch without conflicts; the next commits restore this change's own wording on top of the merge. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * feat(xtask): add e2e-queue-import to turn a queue run into nightly report inputs Reads a finished run of the shared GPU queue (its per-lane results artifact and the run's job log) and writes each lane's report files where `cargo xtask e2e-report` reads them, named for the lane and the queue's gate, plus a status section for the top of the support matrix. Wired in by the following commits. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * feat(e2e-report): label queue lanes by lane and gate, and keep advisory ones out of the verdict The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X lane is not the self-hosted one) and marks one the queue does not block on as advisory. release-readiness leaves advisory rows out of its failure count and still counts a blocking lane that came back with no result. xtask gains the `e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly platform check goes, since #1066 removed the per-PR W7900 lane it exempted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: keep the queue's GPUs in the support matrix and release readiness Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear. e2e-report-nightly now fetches the newest queue-run-results artifact of a main run that finished in the last 24 hours (and its job log, for the tested branch and each lane's gate), imports it, and opens the matrix with a Queue lanes table. A missing, late or malformed queue run never fails the report: the page says so instead. The workflow contract pins the new steps and the native Linux preflight lane counts for this change plus #1066 (four). The docs describe the queue import, link the queue page instead of "(below; advisory)", and fix the "Of those" wording in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: say when the queue lanes are missing from the readiness page, and skip other-branch runs Review of the queue import found two gaps against "no GPU row goes missing": - When no queue run was imported, the Release-Readiness table still read "0 failure(s) across all platforms" and GO, and the absence was noted only on the E2E-Support-Matrix page. release-readiness now reads the matrix's absent note and marks the E2E row and adds a warning under the table. The verdict itself is unchanged: absence is still not a failure. - The fetch step took the single newest queue-run-results artifact and then dropped the whole import when its log said another branch, so a later dispatch of another branch hid a valid run of main. It now walks the five newest runs of the last 24 hours and takes the first that tested main. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: address review on the queue import and readiness doc - release_readiness: move queue_absent_reason above the total_e2e_failures doc block so each function keeps its own rustdoc. - nightly: import only a completed queue run (any conclusion), and scan every job of the run for the lane matrix instead of assuming the first. - docs: neutral pool wording, drop the leftover blank line. - workflow_contract: pin the new fetch-step guards. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: keep unnamed queue rows and directories apart; tidy the importer - An unnamed matrix row is now unnamed-row-N and an unnamed artifact directory unnamed-dir-N, so plan() can no longer merge two unrelated lanes into one. - Compile the three static regexes once (LazyLock), stop shadowing plan(), and share one make_dir helper in write_reports. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit eb90e41) Signed-off-by: juhovainio <juho.vainio@amd.com>
#1064) * fix(runtime): keep a second amd-smi library out of the runtime's torch On an MI300X with ROCm SDK 7.14.1 installed beside a system ROCm, torch.cuda.device_count() reported 0 while HIP itself saw the device, so `rocm install sdk` with the torch-alignment opt-out failed its device check ("imported but reports 0 devices") and vLLM would have stopped with "Failed to infer device type". On ROCm, torch counts devices through the amdsmi Python package. That package opens `libamd_smi.so` by its bare name, after a lookup relative to its own path that only fits the SDK's own tree and misses in a virtual environment. The SDK ships the library as `libamd_smi.so.26` only, so the bare name resolves to the host's amd-smi, or to the copy inside the amdsmi package. Torch has already loaded the SDK's own copy through librccl, and two amd-smi libraries in one process enumerate no devices. Link `libamd_smi.so` to the SDK's own library in the runtime's compat directory and put that directory on the loader path, so the package opens the library that is already loaded. The probe behind the install device check, `rocm examine` and the vLLM launch all take it from one helper in rocm-core. Nothing is written when the SDK already ships the bare name. Measured on the MI300X: the same runtime reports no_devices before and usable (1 device) after. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * test(runtime): cover the ComfyUI amd-smi link; fix a doc comment Pin that the ComfyUI launch environment carries the amd-smi link directory on LD_LIBRARY_PATH, and that the link points at the SDK's library. Reword the ensure_therock_compat_shims doc so the amd-smi link is not described as an mpi_cxx write. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): pick the amd-smi soname by number, not by string When an in-place SDK upgrade leaves an older libamd_smi.so.<n> beside the current one, min() over the paths compared them as strings, so `.so.1` beat `.so.26` and the link pointed at a library torch had not loaded. Take the highest by number. Also look for an existing bare name in every library directory before choosing a soname: the loader finds it ahead of the shim, which sits last on the path. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * test(runtime): cover the examine probe's amd-smi link and pin its order Add a test that the torch probe in `rocm examine` picks up an existing amd-smi link directory and never writes one. Assert the link directory comes after the SDK's own in the ComfyUI and vLLM launch paths, gate the probe test on Linux like its siblings, reword the ensure_amd_smi_compat doc to match the any-directory check, and stop exporting runtime_compat_dir_for_python, which nothing outside its file uses. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(comfyui): write the amd-smi link on Linux only; tighten the probe test The ComfyUI launch wrote the amd-smi link on every platform, though only the Linux loader path uses it; gate it like LD_LIBRARY_PATH. Assert the examine probe test on the specific missing-link error, not on "not the OK version", which any failure satisfies. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(vllm): drop a leftover libmpi_cxx stub the amd-smi link would expose The compat directory now reaches the launch's loader path for the amd-smi link alone. A libmpi_cxx.so.40 stub written by an earlier run, before the host gained a real libmpi_cxx, would then shadow the real library there. Remove the stub when the launch does not need it, as is done for the stale numa link. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): say when the amd-smi link is missing; pin link repair A launch that cannot make the amd-smi link is silent and leaves torch counting zero devices. Have the examine torch probe note when the SDK ships libamd_smi.so.<n> but the environment has no link. Pin that a dangling link or one pointing at another SDK in the compat dir is repaired by the vLLM launch, and reflow the ensure_compat_symlink doc comment. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): shorten the first doc paragraph of amd_smi_link_missing clippy::too_long_first_doc_paragraph rejected it on Linux and on the Windows clippy job. Split the summary from the detail. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(vllm): remove a leftover mpi stub only when the host has a real libmpi_cxx ensure_mpi_cxx_compat answers None for a real libmpi_cxx, for no embedded stub and for a stub that could not be written. Only the first makes an existing stub harmful; in the others it may be the one torch loads. Gate the cleanup on a real libmpi_cxx being present and test both cases. Also name the tests that prove the examine note's claim that a launch or install writes the amd-smi link, and document why the ComfyUI launch ignores a failed link write. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): clear a stale mpi stub in the shared amd-smi helper The amd-smi link shares a directory with the vLLM launch's libmpi_cxx stub, and the ComfyUI launch, the install device probe and rocm examine now put that whole directory on the loader path. A stub left by an earlier vLLM run, on a host that has since gained a real libmpi_cxx, would load ahead of the real library there, but only the vLLM launch removed it. ensure_amd_smi_compat now removes such a stub before it returns the directory, so every caller that adds the directory gets the cleanup. The vLLM launch no longer keeps its own copy. The read-only variants behind rocm examine cannot write, so they hold the directory back while a stale stub is there, and they accept the link only when it points at the library a launch would pick, so examine no longer reports as healthy a link a launch would repoint. Also: the examine note says "no usable" link and drops the host-ROCm qualifier, the device-probe test uses the shared compat-dir constant, and the stub's file name is a public constant so the ComfyUI and vLLM tests stop hardcoding it. Those two tests derive their expectation from whether the host has a real libmpi_cxx, so their comments point at the rocm-core test that injects the host answer and covers both halves on any host. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): import Path on every platform for the stale mpi stub helpers mpi_cxx_stub_shadows_real_library and remove_stale_mpi_cxx_stub are not Linux-gated (their callers in managed_runtime are not), but Path was only imported on Linux, so the Windows build failed with E0425. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(runtime): clear a stale libnuma link from the shared compat dir The amd-smi link now puts the shared compat directory on the loader path of the ComfyUI launch, the install device probe and rocm examine, as well as the vLLM launch. A libnuma.so.1 link an older release wrote there hides the host's libnuma and breaks the torch import, but only the vLLM launch removed it. ensure_amd_smi_compat now removes that link before it returns the directory, the read-only variant holds the directory back while one is there, and the vLLM launch uses the shared cleanup instead of its own copy. Only a symlink counts: a regular file was not written by rocm-cli. The rocm examine note says a launch or install clears it. Also pins the directory-order rule of the soname choice, including skipping a directory with no soname, and the diagnostic's answer for an interpreter outside a bin or Scripts folder. The doc comments now state what the code does for those cases and for the shared directory, and architecture.md names the new cleanup. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit d189674) Signed-off-by: juhovainio <juho.vainio@amd.com>
) * ci: retire the self-hosted nightly lanes that never get a runner The R9700, MI350P and W7900 nightly jobs have sat queued for hours and been cancelled since at least 11-09. Remove them from nightly.yml and from the consolidated report that waited on them. These lanes now run on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu, Windows, WSL2, all on the Strix pool since #1018) are unchanged. Update the workflow contract tests and the docs to match. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs,test: take main's wording where #1066 and this change touch the same lines Preparation for merging main into this branch without conflicts; the next commits restore this change's own wording on top of the merge. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * feat(xtask): add e2e-queue-import to turn a queue run into nightly report inputs Reads a finished run of the shared GPU queue (its per-lane results artifact and the run's job log) and writes each lane's report files where `cargo xtask e2e-report` reads them, named for the lane and the queue's gate, plus a status section for the top of the support matrix. Wired in by the following commits. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * feat(e2e-report): label queue lanes by lane and gate, and keep advisory ones out of the verdict The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X lane is not the self-hosted one) and marks one the queue does not block on as advisory. release-readiness leaves advisory rows out of its failure count and still counts a blocking lane that came back with no result. xtask gains the `e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly platform check goes, since #1066 removed the per-PR W7900 lane it exempted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: keep the queue's GPUs in the support matrix and release readiness Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X, MI350X and MI355X, which only the queue runs, would never appear. e2e-report-nightly now fetches the newest queue-run-results artifact of a main run that finished in the last 24 hours (and its job log, for the tested branch and each lane's gate), imports it, and opens the matrix with a Queue lanes table. A missing, late or malformed queue run never fails the report: the page says so instead. The workflow contract pins the new steps and the native Linux preflight lane counts for this change plus #1066 (four). The docs describe the queue import, link the queue page instead of "(below; advisory)", and fix the "Of those" wording in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: say when the queue lanes are missing from the readiness page, and skip other-branch runs Review of the queue import found two gaps against "no GPU row goes missing": - When no queue run was imported, the Release-Readiness table still read "0 failure(s) across all platforms" and GO, and the absence was noted only on the E2E-Support-Matrix page. release-readiness now reads the matrix's absent note and marks the E2E row and adds a warning under the table. The verdict itself is unchanged: absence is still not a failure. - The fetch step took the single newest queue-run-results artifact and then dropped the whole import when its log said another branch, so a later dispatch of another branch hid a valid run of main. It now walks the five newest runs of the last 24 hours and takes the first that tested main. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: address review on the queue import and readiness doc - release_readiness: move queue_absent_reason above the total_e2e_failures doc block so each function keeps its own rustdoc. - nightly: import only a completed queue run (any conclusion), and scan every job of the run for the lane matrix instead of assuming the first. - docs: neutral pool wording, drop the leftover blank line. - workflow_contract: pin the new fetch-step guards. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: keep unnamed queue rows and directories apart; tidy the importer - An unnamed matrix row is now unnamed-row-N and an unnamed artifact directory unnamed-dir-N, so plan() can no longer merge two unrelated lanes into one. - Compile the three static regexes once (LazyLock), stop shadowing plan(), and share one make_dir helper in write_reports. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: a missing queue run makes the release-readiness verdict NO-GO A queue that never ran is not a pass. When the nightly report carries the 'no queue lanes' note, the verdict gets the failing gate 'queue lanes absent: <reason>' and reads NO-GO, instead of GO plus a warning. Tests and the docs line follow. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 9101286) Signed-off-by: juhovainio <juho.vainio@amd.com>
…ls (ROCMAI-703) (#1050) * fix(serve): say so in the summary when the post-launch smoke test fails (ROCMAI-703) After `rocm serve`, the deployment summary read `status: ready` with n/a metrics even when the first test request had failed: run_smoke_test discarded the error and returned empty metrics, so a server that could not answer looked the same as one that was not measured. run_smoke_test now returns the failure reason. When the test fails, the summary is headed "(smoke test failed)", the status reads "started, smoke test failed", and a note gives a one-line reason and points at `rocm logs`. The launch stays best-effort: the server keeps running and the exit code is unchanged. Fixes #523 Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * fix(serve): assert the smoke-failure summary end to end and without echoing the api key (ROCMAI-703) The first commit's tests echoed the rendered summary, which carries the endpoint api key, into assert messages, and CodeQL rust/cleartext-logging flagged them. The asserts now carry constant messages. Also in this change: - the failure reason is cleared of terminal control and format characters, and text after a cut-off escape sequence is kept; - a clean stream with no content token (a reasoning model) is no longer treated as a failure; - serve-55, a second `ready` mode of the existing e2e fault-injection hook (compiled out of release builds), asserts the heading, status row, engine reason, one chat request and the logs command over a pty with exit 0; - the README and test docs state what is and is not detected. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs(serve): say what the smoke-failure note does and does not detect (ROCMAI-703) The testing guide claimed a close-delimited stream cut after a token shows `n/a` metrics; it shows the metrics measured up to the cut. The README and the manual test steps now point at the guide's list of failure classes that are not detected, so they no longer imply the note catches everything. The failure reason no longer lets an engine add a backtick-quoted command next to the note's own logs hint. A test pins the note's "server may not be able to serve requests" warning. Comments no longer list scenario ids that drift. The guide records that two ready services on one model can make the verdict describe the other service. The earlier commit's message refers to the e2e scenario as serve-55; it is serve-56 after merging main. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * docs(serve): correct what the serve-57 steps and the smoke-failure notes claim (ROCMAI-703) A review found claims the code cannot back. The daemon-left-behind step cannot discriminate the autostart guard because the isolated config enables no watcher. The testing guide overstated `n/a` metrics for undetected classes and described a two-ready-services case that cannot happen, while omitting the real false pass: the smoke request is routed by model name, not to the launched service. A length-cap boundary had no test. The text now says what the code does, the cap is tested at 200/201 characters, a load-bearing line in the ready hook is commented, and prose cites @id tags instead of scenario numbers that had already drifted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit cab5db1) Signed-off-by: juhovainio <juho.vainio@amd.com>
…Ks (#1102) * ci: check that vLLM can install on the default release and nightly SDKs The nightly queue lanes lost their whole nightly channel on 2026-11-12 and every day since: #1033 made an unpinned nightly install resolve to nightly.repo.amd.com, whose SDK is 10.3.0aYYYYMMDD, and the vLLM engine has no discovery row or static pin for 10.3, so the pre-warm's `rocm engines install vllm` refused it. No pull-request check installs vLLM on a 10.x nightly SDK (the vLLM-default Instinct lanes only run in the scheduled queue nightly), so #1033 merged green. Add a CPU-only workflow that runs `rocm install sdk --dry-run` for the release and nightly channel on gfx942 and gfx950 and fails when the selected SDK has no route in the engine's two build tables, matched the way the engine matches them. It runs only on pull requests that touch the resolver (source.rs), the two tables or the check itself. It is advisory and not a required check: it reads the live indexes, so it can go red with no change in the pull request. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: key the vLLM route check's rust cache by the runner image The workflow-contract test hosted_linux_compile_caches_are_keyed_by_runner_image requires every hosted Linux job's setup-rust-toolchain to carry a cache-key naming the runner image. The new job had none, which failed the required Coverage check. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: widen the vLLM route check's path filter and say what it is stricter about Review findings on the check: - The version is chosen in apps/rocm/src/therock/resolve*.rs and version.rs, not source.rs alone, so a change only there would not have triggered the check. Add them to the path filter. - The script says a version with no row has "no route" and that the engine refuses it. The engine refuses only a line whose major has a discovery row; any other version falls back to the default static pin, a wheel built for another ROCm. Report the two outcomes separately and say in the docstring that the script is deliberately stricter than the engine there. - Use the setup-rust-toolchain pin the rest of the repo uses (v2.0.1). - Move the docs section above the canary's, so it no longer sits over the stray release-candidate paragraph, and list its self-test in testing.md. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: report a missing vLLM route as a warning, widen the path filter The route check was red on every PR touching its paths while the default nightly has no vLLM route, which trains reviewers to ignore it. A missing route now shows as a warning and a job summary and the job stays green; the build, the self-test and a check that cannot run (exit 2) still fail it. Also run on any change under apps/rocm/src/therock.rs and therock/, which all feed which SDK the default resolves to. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: fail the vLLM route check when the check itself breaks probe() returned a string for a dry-run timeout, a failing dry run and a plan with no usable version, so main() exited 1 for them just like a real missing route, and the workflow downgraded a regression in `rocm install sdk --dry-run` to a warning. Those cases, an unrunnable rocm and an unreadable build table now raise CheckError (exit 2), which fails the job; exit 1 stays reserved for a missing route. Findings print as warnings, the self-test covers each branch, and the docs and the arch list comment say what the check does. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 088faa4) Signed-off-by: juhovainio <juho.vainio@amd.com>
* docs: add a manifesto to the README State the goals the project builds toward, so users and contributors can see what the CLI is aiming for: bare metal to a served model in a few commands, the whole AMD GPU line with an honest answer when a GPU has no ROCm build, correctness tested on real hardware, no silent CPU fallback, asking before removing what a user has, actionable diagnosis, output for people, scripts and agents, and open development. The intro says the Tech Preview does not meet every goal yet. Sentences about current behavior are limited to what the CLI does today. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs: add an extensibility goal and point agents at the manifesto Add a ninth goal: ROCm CLI should adopt new engines, ROCm-based libraries and agentic-AI tooling as validated extensions, and take a user from a local machine to remote GPU compute without switching tools, with `rocm remote` as the first step. AGENTS.md now has agents read the manifesto before planning a change and raise it with the user when a request's approach would work against one of the goals, so the goals reach work done by agents and not only readers who open the README. When nobody can be asked, the agent implements nothing that conflicts and names the conflict in its report. The goals only govern choices AGENTS.md leaves open: its other rules, such as the ban on silent CPU fallback, stay binding whatever the user answers. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 2583a97) Signed-off-by: juhovainio <juho.vainio@amd.com>
… it does not (#1091) * fix(wsl): make a managed runtime load its own librocdxg, and say when it does not A ROCm wheel ships librocdxg.so.1 beside libhsa-runtime64, but the HSA runtime opens the unversioned librocdxg.so, which the wheel does not ship. On WSL2 the lookup therefore skips the runtime's own copy and loads the host's (/opt/rocm/lib from rocdxg-roct). The 10.2 and 10.3 nightly runtimes need hsaKmtSetPersistingCacheSize, which rocdxg-roct 1.2.2 does not export, and crash on first GPU use while `rocm examine` still reports wsl_rocdxg_ready. The lookup mechanism was observed with LD_DEBUG=libs on the 10.2 nightly (verified). Which component requests the symbol is not identified. - Wheel runtime installs and updates (install_wheel_runtime: `rocm install sdk`, `rocm update --apply`) on WSL link librocdxg.so to the runtime's own librocdxg.so.1 (a relative link) in the runtime's library directory, which libhsa-runtime64 searches through $ORIGIN (a DT_RPATH per readelf), so a bare <runtime>/bin/python is fixed too. The step reads the WSL gate itself. The report line says what was done; a failed link leaves the install at exit 0 with the manual command in the report. Tarballs ship both names and adopted or imported runtimes are never linked. The link is not tracked by the installer and is orphaned if the runtime is removed by package. - `rocm examine` lists, on WSL only, each managed runtime copy its runtime does not load (wsl_runtime_librocdxg, and summary.unlinked_runtime_librocdxg in --json) with the ln -s command that fixes a runtime installed earlier (for an adopted runtime that is a write into a directory the CLI did not create). It reports what the files show, not whether the host is affected. A dangling librocdxg.so is treated as present. driver_status is unchanged: it describes the host. Tests: the install entry point is driven on a simulated WSL host and a simulated native one (needs the e2e-test-hooks feature); the install's report helper is driven with the gate true and false on every Unix host and shown to link only the probe's library directories; the advised command and the install are shown to make the same relative link. Deleting the call in install_wheel_runtime is caught by the dead-code lint under `cargo clippy --workspace --all-targets -- -D warnings` (run: it reports the wrapper and its two helpers as never used), not by a test; `cargo test -p rocm --bin rocm therock::` stays green with a warning. A wrong gate or directory list passed through the wrapper is covered by the tests of the shared helpers; a different list passed to the core entry from inside the wrapper is not caught by any test. The install side has no scenario: `rocm install sdk` on WSL needs a real wheel download and a WSL host, and no default lane has one; the post-probe step is covered by unit tests and the call site by the dead-code lint. A failed link is reported with its error and the command to run once its cause is fixed, since the same link call failed. Tradeoffs: the link is made for every managed wheel runtime, including the 10.1 release (its bundled copy then replaces the host copy; checked to work); WSL1 counts as WSL; the examine line is one long "; "-joined line to keep its key: value shape. Notes: the loader claims (the runtime searches its own directory first; tarballs ship both names) were observed by hand, with readelf -d (libhsa-runtime64.so.1 carries RPATH $ORIGIN and no RUNPATH in the 10.1, 10.2 and 10.3 wheels; librocdxg.so.1 has no unversioned sibling) and LD_DEBUG=libs on the 10.2 nightly, not by an automated test. Runtimes installed before this change stay unlinked and examine only advises. Engine venvs (vLLM, ComfyUI) that pull rocm-sdk-core as a torch dependency are neither linked nor examined (not verified). The link is made before write_runtime_manifest, so a manifest failure leaves a link in an unregistered root (harmless on retry). The ExamineSummary gate (wsl.is_some()) is covered by scenario examine-52, not by a unit test. The update-check disabling is in push_planted_wsl_machine_env, for determinism. Fixes #1082 Signed-off-by: juhovainio <juho.vainio@amd.com> * docs(readme): link the WSL guide by URL so the Sphinx include resolves README.md is included into the Sphinx site, where a relative link to a file outside the docs source fails the -W build (myst.xref_missing). The README links other repository docs by absolute URL for the same reason. Signed-off-by: juhovainio <juho.vainio@amd.com> * docs(testing): link runtime_rocdxg.rs and drop a repeated sentence Cite the new module as a relative link, as docs/testing.md does for other source files, so the link check can catch a moved target, and remove the closing sentence of the install-side paragraph that restated the reason and the lint protection given a few lines earlier. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(wsl): treat a dangling librocdxg.so as missing, and replace it The previous change counted a dangling `librocdxg.so` as present and left it alone, so `ln -s` would not hit "File exists". That was wrong about the loader. LD_DEBUG=libs on the 10.2 nightly with a dangling librocdxg.so (pointing at a missing file) first on LD_LIBRARY_PATH showed the loader trying the link twice and continuing down the search path: open() on a broken symlink gives ENOENT, which it treats as a missing file, so the host's librocdxg was loaded and the process still exited 139. A runtime whose link goes dangling (a wheel changing the file, or the file being removed) therefore crashed again with no warning from `rocm examine` and no repair from the install. The name is now judged three ways: absent, present and resolving (a regular file or a working link: never replaced), and dangling. - `rocm examine` lists a dangling librocdxg.so like an absent one, and its advice is `ln -sfn`, which clears a dangling link (a plain `ln -s` fails with "File exists"). The line says "no librocdxg.so (or a dangling one)". - The install replaces a dangling link atomically: a temporary link in the same directory is renamed over it, so the name is never absent in between, and the temporary is removed if the rename fails. The report line says "replaced a dangling <path>". A file or a working link is left alone. - Limits, stated in the module doc and docs/wsl.md: the versioned name is the fixed `librocdxg.so.1` (a wheel whose soname changes is not detected or linked; no globbing), and a dangling librocdxg.so beside a runtime that ships no librocdxg.so.1 is not detected, since there is nothing to link it to. Also: claims about the gate now say WSL (it also fires on WSL1); managed_runtime_library_dirs documents that on Linux it includes the host's /usr/lib/wsl/lib; the startup update-check disabling is scoped to the Given steps that plant a managed runtime instead of every step using the planted-WSL helper (the harness consumes the scenario env on each run, so the advice step re-plants it); docs/testing.md credits the "only the probe's library paths" assertion to its real test in therock.rs. Tests: unit tests for detection, atomic replacement, no temporary left (also on a failed rename), a resolving link and a file untouched, and the advised command clearing a dangling link through sh; scenario examine-53 plants a dangling link, lists it, runs the advised command, and checks librocdxg.so reads as the runtime's own bytes. Signed-off-by: juhovainio <juho.vainio@amd.com> * docs(wsl): say ln -sfn everywhere, and test the temporary-link counter - The advised command is `ln -sfn librocdxg.so.1 '<dir>/librocdxg.so'`; docs/testing.md, docs/manual-testing.md and the docs/architecture.md row still said `ln -s` (which fails with "File exists" on a dangling link). The architecture row now also describes the dangling case. The command was run against GNU ln 9.7 as well as uutils ln: plain `ln -s` over a dangling link fails with "File exists", `ln -sfn` replaces it and the name resolves. - managed_runtime_library_dirs documents what it returns: every library entry the environment builder produces for each managed runtime (including each root's bin, as the builder collects them), plus the host's /usr/lib/wsl/lib on Linux. - A two-thread test replaces dangling links in one directory at once, so the counter in the temporary link's name is exercised: without it the threads collide on one temporary name and the test fails (checked by replacing the counter step with 0). Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(wsl): pin the versioned-copy rule, narrow the replace window - The versioned `librocdxg.so.1` decides whether a directory is listed or linked: a regular file or a resolving link is, and an absent, dangling or looping one is not (nothing valid to link to), in `rocm examine` and in the installer alike. No test planted a broken versioned copy; the whole matrix is now tested, through the installer (no link and no temporary left) and through examine's registry scan. The exclusion is deliberate and a broken versioned copy is not reported; the module doc and docs/wsl.md say so. - replace_with_link re-checks that the name still dangles just before the rename and leaves it alone otherwise. The window cannot be closed (rename has no "only if still dangling" form); the docs now say a name that becomes a working link between the re-check and the rename is replaced, instead of claiming "never replaced" unconditionally. - A failed removal of the temporary link is part of the returned error. A leftover temporary from a crashed process can only collide after pid reuse, and then symlink fails with "File exists", which is reported. - A test pins that examine's directory set contains every directory the install links (the two are computed separately and are not unified here). - A summary written before `unlinked_runtime_librocdxg` existed still deserialises (test). - Wording: claims about the gate and the link say WSL; "WSL2" stays only where the text describes what was observed. NameState::Dangling's doc covers link loops. A test comment names what actually produces the synthesized failure. The install test that asserts end state is renamed to what it checks. The e2e doc blocks are back on their own functions. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(wsl): link a name that vanished mid-replace, and test cleanup for real - In the dangling-link replacement, a name that is gone at the re-check (it vanished after the first check) is now linked, instead of being reported as "File exists" for a name that does not exist. Resolving at the re-check is still refused; the docs describe each case. - The cleanup test now reaches the rename-failure branch: the rename is injected to fail while the name still dangles, removal is the real one, and the test checks no temporary link is left on disk. The old test passed a path whose re-check read as absent, so it never reached the rename. A new test covers the vanished-name case. - The window description no longer implies a measurement. - README: re-wrapped the WSL paragraph. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(wsl): report a vanished name as linked, not replaced When the name vanished between the first check and the final check of a dangling-link replacement, the temporary link is renamed into place but nothing was replaced; the install line still said "replaced a dangling". replace_with_link now reports whether the name really dangled at the final check, so the three cases each print their own line: absent from the start and vanished at the final check say "linked", a dangling link that was replaced says "replaced a dangling ...". docs/wsl.md, docs/architecture.md, docs/testing.md and the module doc no longer say a name that resolves is "never" replaced: it is left alone when checked, the install refuses at its final check if it resolves by then, and a name that becomes a working link between that check and the rename is replaced. Signed-off-by: juhovainio <juho.vainio@amd.com> --------- Signed-off-by: juhovainio <juho.vainio@amd.com> (cherry picked from commit c21942f) Signed-off-by: juhovainio <juho.vainio@amd.com>
…89) (#1090) * feat(runtimes): show runtime provenance in list and examine (ROCMAI-189) The install report names a runtime's channel, package source, source layout generation and exact GPU target, but afterwards only its manifest knows them, so nothing told a user which stream or GPU an installed runtime was for. `rocm runtimes list` now prints a `provenance:` line under each runtime and `rocm examine` prints matching `active_runtime_*` lines for the active one, both from RuntimeProvenance::fields(). The layout and target choose the same recorded field `rocm update` uses; the source is the package index, else the artifact URL, never the adoption marker an adopted runtime keeps there. Values are trimmed, and a fact recorded nowhere or recorded blank prints `unrecorded`; nothing is guessed from other facts. Covered by unit tests and the runtime-lifecycle-19 and examine-49 scenarios on the mock lane; README, --help and the testing docs describe the new output. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(runtimes): cover adopted runtimes' unrecorded source in a scenario (ROCMAI-189) An adopted runtime installed nothing, so its manifest holds an adoption marker where an artifact URL would be, and the listing must report its source as unrecorded rather than show the marker. That was pinned only by unit tests that refer to the marker through its constant, so a change to the persisted value would have gone unnoticed. Scenario runtime-lifecycle-19 now plants an adopted runtime whose manifest carries the on-disk `adopted-read-only` marker and asserts it lists `source=unrecorded`. A unit test pins the persisted marker value against the constant. The examine provenance test gains an adopted active-runtime case beside the nightly one. The two adopt listing tests are merged into one table-driven test over the default and `--channel nightly` requests, sharing `provenance_line` with the runtimes_cmd tests. docs/testing.md describes the new coverage. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(runtimes): correct provenance claims, pin --runtime-id (ROCMAI-189) Several comments and docs described the provenance listing in ways the previous commits made untrue or left incomplete. The e2e prewarm comments said `rocm runtimes list` carries no `channel=` field. It now prints one on each runtime's `provenance:` line; the `{channel}-` key prefix guard is still needed because `RuntimeEntry::parse` reads only the row line, and the comments now say so. The provenance test's doc comment said each fact is read the way `rocm update` reads it. Only the layout and target are shared with `rocm update`, which does not read the recorded source. The README and docs/testing.md said an adopted runtime's channel is `nightly` after `--channel nightly` and `release` otherwise, missing that `--runtime-id therock-nightly:<family>` also names the channel. They now cover `--runtime-id`, and the adopt listing test gains that path beside the default and `--channel nightly` requests, so all three ways of naming the channel are pinned. The docs/architecture.md entries for the examine and runtime-lifecycle step files now describe the provenance steps they gained. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(runtimes): pin install_root before provenance in the listing (ROCMAI-189) The `cargo xtask e2e` prewarm reads the line directly after each runtime row of `rocm runtimes list` as that runtime's `install_root:`. The listing now also prints a `provenance:` line per runtime, and nothing pinned that it comes after `install_root:` rather than before: the existing provenance test searches ahead for the line and would not notice a reorder. A runtimes list test now pins the order row, `install_root:`, `provenance:` by index. The MIXED prewarm fixture gains `provenance:` lines in that position, so the prewarm's assessment is read against the current row order, and a new test checks that `RuntimeEntry::parse` ignores a provenance line, whose `name=value` fields resemble a row's. The fixture comments now state which fields their captures predate, and the `UNRECORDED_PROVENANCE` doc names `RuntimeProvenance::fields` as what prints it. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 9e50eac) Signed-off-by: juhovainio <juho.vainio@amd.com>
A queued e2e-gpu job for a superseded PR head, or for a closed or merged PR, still takes an MI300X runner and holds the concurrency group for about 45 minutes, and `gh run cancel` from outside does not dequeue it. On pull_request events e2e-gpu's first step, once it has a runner, asks the API for the PR's state and head sha. If the PR is closed or its head differs from the event's, the step records `stale` and every later step is skipped without failing the job. A job-level `if` cannot do this because it is evaluated before the job is queued. e2e-gpu keeps read scopes only (`contents: read`, `pull-requests: read`). If the PR cannot be read, or gh is missing, the job runs as before. The cancel is a separate GitHub-hosted job, cancel-stale-run, with no checkout and `actions: write`. e2e-gpu builds and runs PR code on a persistent self-hosted runner, so a write scope there would sit next to untrusted code, which this repo's workflows otherwise avoid. Only a same-repo PR is sent a cancel; a fork PR, whose self-hosted lanes only run after approval, just has e2e-gpu skip its steps. A failed cancel is tolerated. e2e-report is gated on always(), so it still runs after the cancel, but its missing-report check is gated on !cancelled() and is skipped in a cancelled run: the stale exemption is for runs that are not cancelled (fork PRs, or a cancel that fails or loses a race). e2e-gpu publishes a `stale` output so the consolidated report's missing-report check (here and in the identical copy in nightly.yml) does not count a stale-skipped lane as a missing report; it says so in the summary instead. The ci.yml and docs claims about fork runs are brought in line. Tests: the guard script runs under a stub `gh` for each outcome (including a missing gh and unreadable PR reads), every later step's `if:` is compared exactly, and the cancel job's permissions, runner, condition and single step are pinned. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 67c691b) Signed-off-by: juhovainio <juho.vainio@amd.com>
…backend (#855) * fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend Lemonade 11.5.1 pins TheRock 7.13.0 and llama.cpp b9752, and downloads TheRock from a location that does not serve ROCm 10.x, so on a ROCm 10.1 SDK its llamacpp:rocm backend could not be paired with the SDK in use. Lemonade 2026.44.1 pins TheRock 10.1.0 and llama.cpp b11580, whose release ships ROCm 10.1 (and 10.0 and 7.14) archives for Linux and Windows, and installs TheRock from AMD's whl-next wheel index or core tarball location. The embeddable layout, binary names and the lemond/lemonade flags rocm-cli uses are unchanged; the new archive digests are recorded. Not 2026.45.3, the newest stable release: its Linux lemond needs glibc 2.43 and OpenSSL 3.2.0 and exits with a loader error on Ubuntu 24.04, which has glibc 2.39 and OpenSSL 3.0.13. 2026.44.1's needs glibc 2.38 and OpenSSL 3.0.0, the same as every release back to 11.5.1. Two adaptations the new release line needs: - `lemonade backends` lists recipes rocm-cli did not know (llamacpp-hrx, ds4, thenoise) and adds Tier and Format columns. The parser matched recipe rows against a fixed list, so an unknown recipe's rows, and every indented row after them, were read as llamacpp backends. It now reads the recipe from the row layout, and reads Lemonade's update_required and update_available states as installable and installed instead of dropping the row. - The second alignment attempt took lemonade-sdk/llama.cpp's newest release, but a release can ship without some or all of its ROCm archives: b12013 through b12086 carried only qualification reports, and others skip one ROCm version or one platform. It now lists the 30 newest releases and picks the most recently published one that ships the exact archive for the active ROCm version on this platform. The runtime-16 and runtime-17 scenarios assumed the active SDK never equals the packaged pin, which is no longer true on a ROCm 10.1 lane: runtime-16 now checks that the installed pin equals the active SDK by whichever route the host takes, and runtime-17 compares the pin with the one inside the verified archive. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): compare the Lemonade archive extension ignoring case The runtime-17 step that finds the cached Lemonade archive compared the `.zip` extension with `ends_with`, which clippy's pedantic `case_sensitive_file_extension_comparisons` rejects. The e2e target is excluded from `--all-targets`, so only the Windows job's `cargo clippy -p e2e-cucumber --test e2e` compiles it and caught it. Derive the archive's stem once, comparing either extension ignoring ASCII case, and reuse it for the member path. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): refuse a host too old for the pinned lemond up front A Linux lemond built against a newer glibc or OpenSSL than the host has exits at once with a dynamic-loader error ("version `GLIBC_2.43' not found"). The user saw only "Lemonade server did not become ready", after the archive had been downloaded and extracted; this is how 2026.45.3 failed on the Ubuntu 24.04 GPU lanes. Record what the pinned lemond needs (glibc 2.38 and OpenSSL 3.0.0 for 2026.44.1, read with objdump -T) beside the archive digests, keyed by the Linux digest so a bump that does not re-read them fails a unit test. Read the host's versions as the newest GLIBC_/OPENSSL_ version node its libc.so.6 and libssl.so.3 define, the same thing the loader compares, locating them through the ldconfig cache. On Linux, `rocm engines install lemonade` now refuses such a host before downloading anything, and starting lemond is refused too. Both name the version needed and the version the host has for each short library. A host version that cannot be read is treated as unknown and never refuses. lemonade-installs-07 drives the install refusal through an e2e-test-hooks override of the host versions, and checks the message and that nothing was downloaded. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): refuse a host whose loader cache has no libssl.so.3 The OpenSSL half of the host check could not fire. The probe read only libssl.so.3, and every libssl.so.3 defines OPENSSL_3.0.0, which is all the pinned lemond needs. A host with only OpenSSL 1.1 has no libssl.so.3 at all, so the probe reported OpenSSL as unknown, never refused, and the user still got "Lemonade server did not become ready". Each library is now a version, missing, or unknown. It is missing when the ldconfig cache was read and lists libc.so.6 but not that library, and the refusal then says the host's loader cache has no libssl.so.3. A cache that could not be read, one that lists not even libc.so.6, and a library that does not parse still read as unknown and never refuse. The cache reading is now a pure function, unit-tested over planted `ldconfig -p` output, and the probe runs once per process. lemonade-installs-07 now reports a host the probe can actually produce, Ubuntu 20.04-like: glibc 2.31 and no libssl.so.3. The README, the docs and docs/manual-testing.md section 3 describe the same two outcomes, and the README no longer says every build needs glibc 2.38, since 2026.45.3 needs more. The start-time refusal has no scenario, because no lane can reach it: `rocm serve` runs the install check first, a Linux serve with a packaged llama-server never starts lemond, and the mock lane has no GPU for serve to resolve. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): stop calling Lemonade 11.5.1 the pin in two Windows rows The runtime-16 (ROCMAI-899) and serve-06 (ROCMAI-900) Windows rows were written while Lemonade 11.5.1 was pinned, and their reasons still said so. Say they were observed with 11.5.1 and have not been re-checked with 2026.44.1, which fetches TheRock elsewhere and whose llama.cpp build ships a ROCm 7.14 archive. Both scenarios are @nightly, so pull-request lanes have not run them yet; a stale XPASS on the first Windows nightly after this pin lands is the signal to delete the row. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): check the llama.cpp ROCm tag where the SDK is the packaged pin On a ROCm 10.1 lane the active SDK equals the pin Lemonade 2026.44.1 ships, so comparing the ROCm version cannot show a rewritten pin. Also compare the paired llamacpp.rocm-stable tag, and name the e2e-test-hooks feature in the manual-testing note. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * style(lemonade): cargo fmt the spawn helpers Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): stop calling Lemonade 11.5.1 the pin in a row-scope test comment Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): add an opt-out for the host-library check, and say what a refusal left undone The check reads only the ldconfig cache, so a host that provides OpenSSL 3 or glibc some other way (LD_LIBRARY_PATH, a Nix or Guix profile) was refused with no way past it. ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK turns both refusals into a warning, and the refusal names it (lemonade-installs-08). A refused re-run where the pinned Lemonade is already installed said "Nothing was downloaded." about a download it would never have made; it now says the installed copy stays as it is (lemonade-installs-09). Also: the backends parser no longer ends a recipe at a first-column line with no state; runtime-16 requires the install to report an alignment whenever the pin moved; and the ELF reader test skips on a host without ldconfig. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): compile the host-refusal outcome helpers on Linux only outside tests Their one caller, the host-library refusal, is Linux only, so a Windows build rejected them as dead code. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs: stop saying every published Lemonade build needs glibc 2.38 2026.45.3 needs glibc 2.43, so the floor belongs to the pinned build. Reword it in docs/wsl.md, the fix-wsl-5 rationale `rocm fix` prints, the matching note `rocm diagnose` prints, and WSL_MIN_UBUNTU's doc comment. Also say that the host-library check does not read GLIBCXX, and that `object` is one new crate for the standalone Lemonade adapter binary. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): read the backend state at its column, and name the version node a host library provides A first-column warning with a state word in its text ("... is installed but outdated") started a bogus recipe and dropped the llama.cpp rows after it. The state is now read at the Status column the header names; with no header, only a recipe-shaped first word starts a recipe. The host-library check reports the newest version node a library defines, not its release: Ubuntu 24.04's OpenSSL 3.0.13 defines up to OPENSSL_3.0.3. The message now says "this host's libc.so.6 provides up to GLIBC_2.31", and test rows use values the probe can produce. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs: list the new parser tests, drop unverifiable release-cadence figures, say who deletes the xfail rows Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 6803163) Signed-off-by: juhovainio <juho.vainio@amd.com>
…#1062) * fix(rocm-core): report a timed-out rocm fix step as such (ROCMAI-345) The fix runners read the probe adapter's collapsed exit code, so a helper stopped at the one-minute bound was reported as "usermod exited 124; group membership NOT changed." That names no timeout, offers no way to recover, and asserts a state nobody checked: the executor stops only the direct child, so usermod under sudo can still finish. Run each step through the bounded executor and report from its typed outcome. A command that exits non-zero or cannot start still says the change was NOT made; one that times out, is signalled or cannot be waited for now says the change may or may not have been made, names a command to check it, and the rocm fix command to re-run. The line and exit code 4 come from one value, and the re-run ids are the catalog's own constants. On Windows, a PowerShell read of the current User/Machine value that did not complete used to be taken as "unset": fix-2 then reported the override as gone and exited 0, and fix-9 skipped its already-set guard and ran setx. Both now stop with exit 4 before planning or prompting, as fix-6 already did. Covered by unit tests over an injected command runner for every outcome of each runner, and by e2e scenarios diagnose-15 (extended), 39 (tightened) and the new 62 (Linux helper never finishes), 63 and 64 (Windows reads fail). Docs, --help and the rocm-doctor skill describe the new exit-4 cases. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(rocm-core): keep the e2e suite polled and state fix read failures precisely (ROCMAI-345) diagnose-62 waits out fix-4's real one-minute timeout. Its step ran the CLI synchronously inside the single future the cucumber suite is polled on, so every concurrent scenario stalled for that minute and the TUI scenarios beside it overran their wall-clock caps, failing the hosted E2E lanes. The step now waits for the CLI on spawn_blocking. On Windows, the timeout line of a setx or PowerShell write said "Check with" before a PowerShell expression that cmd cannot run. Step now carries where its check runs, and those lines say "Check in PowerShell with". The Windows runners for fix-2, fix-6 and fix-9 write their own lines through injected stdout sinks, so the unit tests assert what is printed together with the exit code and what was run: a read that exits non-zero stops each fix with exit 4 and nothing written, a failed read prints no plan, and a dry run with a readable scope previews without running setx. diagnose-64 now previews fix-9 with --dry-run instead of applying it. The empty PATH hides PowerShell but not setx.exe, so a regressed read guard could otherwise set the runner's real User-scope HIP_VISIBLE_DEVICES. README, --help, the exit-code docs and the rocm-doctor skill now say exactly which fixes this covers: on Windows, fix-2, fix-6 and fix-9 with --device-index read the current value first, and a failed read exits 4 with no plan, under --dry-run too; the catalog entry printed before it is not a preview. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(architecture): say the Windows fix runners read before planning (ROCMAI-345) The runners.rs entry promised only print, dry-run and consent. On Windows fix-2, fix-6 and fix-9 now read the current env-scope value first and stop with exit 4, before the runner's own preview or any prompt, when that read fails, dry-run included; the entry should say so. "Consent first" becomes "consent before any write", and the report-only arms (fix-2 on Linux, fix-9 without a device index) are named as the exception, as the module header already does. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(rocm-doctor): read an older CLI's "exited 124" as a timeout (ROCMAI-345) The skill is federated from main ahead of a release, and its Phase 0 installs the release-channel CLI. A released CLI reports a timed-out fix step as "<program> exited 124; <what> NOT changed.", so an agent following the skill would tell the user nothing changed when the step may have landed. Treat that line as a timeout: say the change may or may not have landed, give the user the check for that fix (the group list on Linux, the User-scope variable on Windows), and offer the same re-run. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(rocm-core): name the proving tests and cover older CLIs in the skill (ROCMAI-345) The wording tests in fix/consent.rs pin each failure line against a stand-in step; their comments now name the runner tests that prove what each line claims, and say that no runner test drives a wait failure or a signal. The rocm-doctor skill special-cased only `exited 124`. A CLI released before this change also reports a signalled step as `exited -1` with `NOT changed`, and on Windows takes a failed read for an unset variable: fix-2 can then report the override gone and exit 0, and fix-9 can run setx over a value the user chose. One passage now covers both, with the check to run, and the exit-4 row in reference.md points at it. Also refresh three stale comments: what fix.rs says consent.rs holds, fix-2's class in the report_persistent_override doc, and a PR-relative note on the current_os WSL test. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * chore(catalog): publish the merged `failed` exit-code description (ROCMAI-345) The merge with main moved exit-code wording into `exit::MEANINGS`, and this branch's `failed` entry now says a timed-out or stopped command may have made its change. Regenerate the published catalog so tools read the same text the CLI prints. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(rocm-core): cover fix-2's Machine-scope read in the skill and catalog (ROCMAI-345) An older CLI swallows a failed read of either scope, so with an override set only in the Machine scope fix-2 can still report it gone. The skill's check for that case now reads both scopes, and since the output does not say which CLI produced it, the skill asks for the check on every Windows fix-2 and fix-9 run. The published `failed` description names the Machine read too, and the skill-eval replays are re-recorded against the new catalog text. Also stop the fix.rs module doc implying the consent.rs helpers are re-exported; only the options.rs names are. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(rocm-core): pin fix-2's Machine read under --dry-run, say what to do with a set value (ROCMAI-345) The published `failed` description says fix-2 reads its Machine value even under `--dry-run`, but only the User read had a dry-run test. A new test fails the Machine read under `--dry-run` with a User value set and asserts exit 4 with no plan; it goes red when the Machine read failure is swallowed. The skill's older-CLI check told the agent to read both scopes without saying what a set value means. It now says: a User value means re-running fix-2, a Machine value needs an elevated PowerShell, and for fix-9 an existing value is left for the user to change by hand. The lead-in says the Windows case applies to every run, since the output cannot tell an older CLI apart. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit ca170c9) Signed-off-by: juhovainio <juho.vainio@amd.com>
…0) (#847) * fix(installer): keep download progress inside the terminal (ROCMAI-770) curl takes its progress-bar width from COLUMNS in preference to asking the terminal, so a value inherited from a wider window -- what a shell leaves behind when its terminal is resized after it starts -- made every refresh run past the right margin and wrap. The carriage return ending each frame then landed a row below the bar, so a single download painted a stack of partial bars with a percentage torn across two rows instead of one bar updating in place. Measure the terminal and pass that width down, reserving the last column so the line stays clear of the margin. Resolve the download URL up front as well: release assets redirect to a signed host and the redirect responses carry no Content-Length, so curl drew its unknown-size "bouncing" bar for every hop before the real one began. wget's branch had a smaller version of the same problem: --show-progress without -q interleaves the bar with request logging. Covered by lifecycle-28, which drives the installer under a pseudo- terminal narrower than COLUMNS and fails on the unfixed script with all seven progress frames reaching or exceeding the terminal width. Signed-off-by: John Lybeck <john.lybeck@amd.com> * fix(installer): address review on the progress-bar fix (ROCMAI-770) The percentage assertion could not fail on the bug it was named for. It read the raw pty byte stream, but wrapping is something a terminal does while drawing, never something the writer puts in the stream, so an over-wide bar reaches it as one intact frame with a whole 100.0% in it. Render the capture through vt100 and assert on physical rows instead: screen().contents() stitches a wrapped row back onto its parent and would hide the tearing, so read rows() directly. The scenario now fails on the unfixed script with 11 progress rows where three downloads should leave three. Drop the redirect pre-resolution. It only ever shortened the bouncing bar from about three frames to one, since most of that wait is curl connecting rather than following hops, and it cost a HEAD per asset including both sidecars. It also never fell back as its comment claimed: curl prints url_effective even when the probe fails, so the original URL was never restored. Removing it leaves the width fix, which is what the reported stacking actually came from, fully covered. Three fixes to the width measurement itself: - A terminal whose size was never set reports 0x0. That was treated as a measured width and suppressed the bar entirely, where curl had previously drawn one at its own default. - tput sizes from its own stdin, which is the pipe under `curl ... | sh`, so the fallback read ncurses' 80-column default rather than the terminal. Point it at fd 2 like its sibling. - A zero-padded COLUMNS is an octal literal to POSIX arithmetic and dash rejects it outright. Strip leading zeros by hand; bash's 10# is not POSIX and dash rejects that too. wget keeps its bar but moves from -q to -nv, so a failed download still says why instead of leaving only this script's generic message. Covers the branches that had none: a terminal too narrow for a bar now says so rather than going silent, and the wget fallback is exercised by shadowing curl out of PATH. Each new scenario was confirmed to fail with its behaviour reverted. Signed-off-by: John Lybeck <john.lybeck@amd.com> * fix(e2e): keep the shadow-PATH helper compiling on Windows (ROCMAI-770) std::os::unix::fs::symlink has no Windows counterpart, and the Windows lifecycle lanes build this target even though every scenario reaching the helper is tagged @requires-os:linux. Gate the one call that has no portable spelling and leave the rest of the function shared. Verified by compiling the construct for x86_64-pc-windows-msvc as well as the host, rather than assuming: a full cross-check of the crate is not possible here, since its C dependencies need a Windows toolchain. Signed-off-by: John Lybeck <john.lybeck@amd.com> * chore: re-trigger CI (ROCMAI-770) GitHub Actions created no check suite for the previous head, so the PR sat with no CI on it. Seven third-party suites registered against that commit and the Actions one simply never appeared; the workflow has no path filters and the same trigger fired for both earlier commits, so this is an Actions-side miss rather than configuration. Empty on purpose: it carries no change, only a synchronize event. Signed-off-by: John Lybeck <john.lybeck@amd.com> * test(installer): cover the width sources the fix actually depends on (ROCMAI-770) Three of the behaviours this PR claims had no test that could fail if they regressed, and review was right that the gap mattered: each one sits behind the measurement chain, so a pty with a real size never reaches it. - Piping the installer into a shell, the shape it actually ships in, leaves stdin a pipe and only stderr a terminal. That is the whole reason the width is read from fd 2, and deleting the redirection kept every scenario green. The new scenario runs at 100 columns rather than 80 because ncurses answers 80 when it cannot read a terminal, which at this suite's usual width is indistinguishable from a correct measurement. - A terminal reporting no size is now stood in by a stty shim, since no real pty here will report 0x0 on demand, with tput left available so falling through yields the terminal's own width instead of the 80-column default a later guard would supply. - A zero-padded COLUMNS is only consulted when neither stty nor tput can answer, so both are shadowed out of PATH. Each was confirmed to fail with its own fix reverted, which an earlier attempt at two of them did not: they passed either way until the shape above was in place. The PATH helper grew shims and multiple names for this, assertions read the width the run actually used rather than a constant, and docs/testing.md names the new coverage as AGENTS.md requires. Signed-off-by: John Lybeck <john.lybeck@amd.com> * test(installer): stop the download notice counting as a progress frame (ROCMAI-770) Frames were picked out by carriage return alone, on the reasoning that only a downloader redraws in place. It does, but the line it redraws on also carries the preceding "download: <url>" notice, so that notice was being counted as a frame. Against a real release URL it is 106 columns wide and would have failed the width assertion on any terminal narrower than that; it passes in CI only because the loopback fixture's URL is short. Require a carriage return and a bar character together. Neither works alone: the notice redraws but carries no bar, and the get-started block prints "rocm install sdk # ..." which carries one but never redraws. Found by smoke-testing against the real nightly rather than the fixture. All six progress scenarios still fail with their own fix reverted. Signed-off-by: John Lybeck <john.lybeck@amd.com> --------- Signed-off-by: John Lybeck <john.lybeck@amd.com> (cherry picked from commit cdc3aee) Signed-off-by: juhovainio <juho.vainio@amd.com>
In `rocm dash` and interactive chat, the approval card for a mutating
assistant tool call (launch_server, install_sdk, install_engine,
stop_server, watcher_enable, watcher_disable) was rendered from the
call's argument object, but the approved replay went through
run_internal_mcp_call, which swapped in a nested `arguments` object
when one was present. A model-supplied call carrying such a field
showed one command on the card and ran a different one once the user
approved it.
The replay now validates and runs the object it is given, unchanged.
validate_chat_tool_call refuses any `arguments` field outright, so the
dash, chat and `rocm mcp-call` paths all refuse such a call before it
runs, and the dash and chat refuse it before a card is shown: no ROCm
tool declares a parameter of that name, so a call carrying one holds
two payloads and is refused rather than resolved to either.
The hidden `rocm mcp-call` command still accepts a pasted MCP
tools/call params envelope and unwraps it at its own boundary, but only
a pure envelope: `name`, `arguments` and the protocol's optional
`_meta`, which is dropped. It refuses an envelope naming a different
tool, and passes an object mixing tool fields with `arguments` through
whole, so validation refuses it. Arguments that are not a JSON object
are now refused rather than run as `{}`.
Tested by unit tests that the validator refuses a nested `arguments`
field for every ROCm tool name, that the dash card and the approved
replay agree, that the replay refuses a nested payload, that the
mcp-call boundary unwraps only a pure envelope, and that no tool in
either catalog declares an `arguments` parameter; and end to end by
chat-17, where a mock model answers with a two-payload launch_server
call and the model is told the call was refused and no approval prompt
is shown, chat-18, where a whole tools/call request handed to
`rocm mcp-call` runs the tool it names, and chat-19, where a request
with tool fields beside a nested `arguments` object is refused and
prints no tool result. The e2e mock server gains a scripted tool-call
reply keyed on the user's words, with its own harness test.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit c9d50ed)
Signed-off-by: juhovainio <juho.vainio@amd.com>
* feat(runtimes): add --json to rocm runtimes list (ROCMAI-148) Scripts that want the registered runtimes, which one is active and which is the rollback target currently have to scrape the text table and its `*`/`-` markers. `rocm runtimes list --json` prints the same listing as one JSON document on stdout, through the shared JsonOrText helper. The listing is now computed once into a value and both forms render from it, so the JSON `active`/`rollback_target` flags and the text markers cannot disagree. Unset values are null in the JSON, and the empty-registry next-step hint stays text-only. The text output of `rocm runtimes list` is unchanged except where `default_runtime_id` matches a runtime id only when letter case is ignored. The listing now applies one default-id rule, ASCII case ignored, to both the active marker and the problem report: the false "missing manifest for active_runtime_id" line printed next to an active runtime is gone, and runtime ids that differ only in case now report an ambiguous-runtime line instead. docs/json-output.md audits `--json` support per command family, states the rules those commands follow (stdout carries only the document, errors stay on stderr, documented fields change only additively), and records where the listing still differs from the resolver `rocm examine` uses. Whether to adopt a shared envelope across commands is left open there. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(runtimes): list the runtime the commands act on (ROCMAI-148) `rocm runtimes list` decided which runtime was active and which was the rollback target by rules of its own, so it could mark a runtime other than the one `rocm examine`, `rocm runtimes activate`, `rollback`, `uninstall` and `rocm update` act on: an active key in other letter case, a dead key whose default id still resolves, case-twin keys, or a previous key that rollback trims, matches ignoring case or reads as a runtime id. The listing now asks the resolvers those commands use: `current_runtime_manifest` for `active` and `select_runtime_manifest` on `previous_runtime_key` for `rollback_target`. `active_problem` reports the configuration with the same matching, so a dead active key is still reported as `missing_manifest_for_key` while the runtime the commands fall back to is marked active. The resolvers are unchanged. In the JSON, `active_runtime_id` is renamed `default_runtime_id`, the config value it carries; the text label stays `active_runtime_id:`. `registry`, `marker` and `install_root` are written lossily, so a path that is not valid UTF-8 no longer makes `--json` fail with empty stdout. `toolchain` is an enum serialized with the words the text and `rocm examine` print. docs/json-output.md describes which runtimes are marked, lists the surfaces that still resolve differently (the dash runtime manager, `rocm setup status`, `rocm engines install`), and states the exit codes: 1 for a failed command unless documented, 2 for a command line the argument parser rejects. The dash comment keeps only the dash's own rules and points there. A state matrix of in-memory unit tests covers every configuration row; scenarios runtime-lifecycle-24 and -25 check an active key in other letter case and a dead key falling back to the default id, each against `rocm examine --json`. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(runtimes): assert runtimes list --json prints nothing on stdout when it fails (ROCMAI-148) The JSON rules say a failing --json command prints its error to stderr, nothing on stdout, and exits 1, but every runtimes list scenario was a success path. Scenario runtime-lifecycle-29 replaces a registry entry with a folder of the same name, which the listing cannot read on any platform or as root (an entry that reads but does not parse is skipped, so it cannot stand in), and checks exit 1, an empty stdout, and a plain-text error naming that entry. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * feat(runtimes): include provenance in runtimes list --json (ROCMAI-148) The text listing names each runtime's channel, package source, source layout generation and exact GPU target on its `provenance:` line, but the JSON form left them out, so a script had to parse the text to learn which stream or GPU a runtime was installed for. Each runtime in `rocm runtimes list --json` now carries a `provenance` object with the same four facts under the same names, as strings, or `null` where the text prints `unrecorded`, so a script cannot take the placeholder for a recorded value. The JSON takes its names and order from a new `RuntimeProvenance::recorded_fields`, which `fields` (what the text and `rocm examine` print) is now derived from, so the two forms cannot name the facts differently. Unit tests pin the object for recorded, unrecorded, blank and padded values, check that the text line and the JSON object agree for each runtime, and add the object to the exact-key-set contract test. Scenario runtime-lifecycle-31 reads the provenance from the JSON for planted nightly, older and adopted runtimes and compares it with the text listing. It turns the startup update check off, as the examine scenarios do, because with wheel runtimes registered that check can print a progress line on stdout ahead of the document. README, docs/json-output.md, docs/testing.md and docs/manual-testing.md describe the field. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(runtimes): state the startup update check caveat and fix a misplaced test comment (ROCMAI-148) docs/json-output.md promises that --json leaves only the document on stdout. The startup update check that runs before most commands can still print a Python progress line on stdout first, once per check interval, when the runtime it checks is a wheel install whose recorded Python is unusable and the Python it finds instead does not fit. State that exception and the variable that turns the check off, so a script author knows about it until the follow-up moves the line. Also move the doc comment of the active-and-rollback-target list test back above that test; the provenance fixture helper had been inserted between them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit bfff521) Signed-off-by: juhovainio <juho.vainio@amd.com>
…compatible engines (ROCMAI-332) (#1073) The engine request and response envelopes carried no version. A daemon and an engine from different releases could not tell whether they spoke the same protocol, and a mismatch surfaced, at best, as a parse error after the engine had acted on a request it misread. Both envelopes now carry an optional `protocol_version`, written on every request and answer as `ENGINE_PROTOCOL_VERSION` (0.1.0). A message without it is read as 0.1.0, the format from before the field, so engines and daemons from earlier releases keep working; a message with no version is written without the field, and unknown fields are ignored so an additive change is read. A new `protocol_version` module holds the rule: MAJOR.MINOR.PATCH of plain decimal numbers without leading zeros that fit in 64 bits, compatible by the caret rule (major and minor must match while the major is 0, the major alone from 1.0.0), anything else unreadable and incompatible, and the bump policy, including that a new method a client may send raises the minor. Each side judges the other's version from the raw JSON object before decoding anything else. - The lemonade, vllm and aim engines, and `rocm __engine-stdio`, answer the request text through a shared `answer_engine_request`. An incompatible or unreadable version, including one present and neither a string nor null, gets `incompatible_protocol` whatever the method. A servable request naming an unknown method gets `unknown_method`, and input that is not a request envelope (including JSON that is not an object) gets `invalid_request`. Each is an error envelope on stdout with a clean exit; input that is not valid UTF-8 is not read as a request. A decoded request is refused the same way before its payload is read. - The shared stdio client that rocm and rocmd run engines through checks an answer's version before anything else in it, `ok` included, and for a single-object stdout that does not decode as an envelope, before its shape. An answer in a protocol it cannot use is an error naming both versions, with the child's failing exit status and stderr when there are any. Before an install or a launch, the client first sends `capabilities`, with an empty object as its payload and on the request's own arguments and environment minus the progress request, and sends the request only when that answer is in a compatible protocol and is not an error; otherwise the error says the request was not sent and why. The check lives in the shared client, for engine binaries it runs. rocm answers its built-in engines in its own process without it, and no shipped command sends an install or launch to an engine binary today. `stop` is not checked first, so a stop is never held behind a check. An engine that checks the version refuses a stop made for a protocol it cannot serve. An engine from before the field acts on the stop, and its unversioned answer is accepted while this release's protocol is 0.1.x. When the engine's answer is an error or is refused, the stop paths in rocm and rocmd still signal the server's recorded processes; only for aim, whose stop the engine alone confirms, does that leave the stop unconfirmed. An unusable stdout is described by what is wrong with it: nothing printed, other output before or after an answer (quoted), JSON that is not an envelope, or not JSON. Engine-written text the error quotes, including a decoder's message about it, is cut to a bounded length with control characters escaped. The note for an engine that answers and then exits non-zero says the answer was read. `unknown_method`, `invalid_request` and `invalid_payload` are shared constants beside `incompatible_protocol`, and `EngineMethod::ALL` lists every method. A shared `engine_contract` check behind the test-support feature holds every first-party engine, and rocm's built-in engines, to the same rules, and is itself tested against engines that each break one rule. Tests cover the rule as a table and property tests, the capabilities check as a state matrix against the fake engine, each engine's built stdio binary, and the client's diagnosis. e2e scenarios engine-ipc-03 to -07 show the daemon's stop report for answers it refuses and reads, and engine-ipc-08 to -12 show `rocm __engine-stdio vllm` answering requests it cannot serve. The engine plugin, testing and architecture docs describe the contract. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 75fc9c9) Signed-off-by: juhovainio <juho.vainio@amd.com>
* ci: run a pull request's MI300X check on the GPU queue The self-hosted MI300X runners are unreliable for pull requests, and a pull request's check should not have to wait for one of them. A new gate job, e2e-gpu-queue, dispatches queue-pr-run.yml on main with the pull request's branch (the runner runs only allow-listed workflows as they are on main), waits for the run, and publishes the release-channel report of its MI300X lane as e2e-gpu-report, the MI300X cell of the consolidated report. The self-hosted MI300X lane is kept as the second check and uploads e2e-gpu-selfhosted-report, shown as "MI300X (self-hosted)". The gate runs on a GitHub-hosted runner with only actions: write, for a pull request from this repository only (a fork's token cannot dispatch). A run the queue does not take (too many going, the nightly's window, no healthy MI300X worker) fails the gate with the queue's reason, which is capacity and not a failing test, and the missing-report check covers the gate like any other lane. The workflow contract now allows queue-pr-run.yml on the queue runner with the same no-credential rules as queue-run.yml, lets e2e-selfhosted.yml alone name it, and pins the gate's conditions and permissions. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: cancel a superseded queue gate and its run, retry API calls Review of the queue gate found: - the gate's `always()` kept its condition true through a cancellation, so a superseded push would not stop it polling for hours; it is dropped, and a cancelled gate now cancels the queue run it started so the runner that run holds is freed - one failed API call (a 502, a rate limit) ended the gate and orphaned its run; the lookups are now asked again - the branch name was checked only in the gate, which the pull request author edits; queue-pr-run.yml on main now checks it too - the permission test passed with a second permission beside actions: write; it now matches the whole block - the docs said two workflows, one runner, and did not say the queue tests the branch as it stands when the run starts, not the pull request's merge with its base Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: end the queue gate green when the nightly window skips it A refusal for the nightly's window is by design, and a red X every night for it would teach everyone to ignore the lane. The gate now ends green with a summary note to re-run after 07:00 UTC and exposes a `skipped` output; the missing-report check reads that as skipped, not missing, in both workflows that carry it. Every other refusal stays red. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: make the gate's permission test exact and update the stale queue docs The gate test accepted a second permission beside actions: write; it now compares the whole permissions mapping. The script scan stops at the next step. docs/ci-hardware-testing.md named two queue-report workflows and one MI300X check; it now names three and two. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: skip the queue gate while queue-pr-run.yml is not on main A dispatch runs main's copy of the workflow, so the pull request that adds queue-pr-run.yml got an HTTP 404 and a red gate. Say so and skip instead; any other error from the check stays red. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: cancel an undiscovered queue run on failure, retry the conclusion fetch The gate's cleanup ran only when cancelled, so a dispatched run whose id was never found stayed on the queue after the job failed. It now runs on failure too and looks the run up by its title when no id was kept. The conclusion fetch after the poll retries five times with a growing pause, as the poll does, instead of twice back to back. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: test the queue gate's script, say which MI300X lane a claim means Run the gate's script against a mock gh: the missing-workflow skip, the nightly refusal skip, retries, run discovery, the report copy and the verdict. The queue contract (the nightly: prefix of refused.txt and the report path) is named once in the test and in docs/nightly-on-the-queue.md. The missing-workflow skip now needs the branch to carry the file, so a file removed from main and the branch stays red. The branch-name error names the rules. A pull request has two MI300X checks, e2e-gpu-queue and e2e-gpu, on the same silicon: the workflow header, docs/ci-hardware-testing.md and docs/testing.md say which they mean. queue-pr-run.yml's comment now says a pending run can be cancelled by a newer one. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci: switch the queue gate on with a variable, skip Dependabot, bound the poll The gate does nothing until the repository variable QUEUE_PR_GATE is true, so the queue side can be deployed before it. Dependabot's read-only token cannot dispatch: its gate is skipped. The status poll stops after ten failed reads in a row. The cancel step looks for a dispatched run as long as the main step does. Counts and clock times the queue side owns are now invariants. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 0e830f3) Signed-off-by: juhovainio <juho.vainio@amd.com>
install.ps1 shipped without the MIT/copyright header that every other source file in the repo carries, because the header check never looked at PowerShell files at all: `licenserc.toml` did not include them. The same was true of PSScriptAnalyzerSettings.psd1 and of the WSL helper script under .github/scripts, which also had no header. Adding the globs alone is not enough. hawkeye has no built-in comment style for the `ps1`, `psm1` or `psd1` extensions, so a file matched by `includes` but with no matching rule is classified `unsupported` and silently skipped while the command still exits 0 -- the check would look green without inspecting the file. The explicit `[[rules]]` block mapping those extensions to the `script` comment style is what makes the files actually get checked. Only `hawkeye check --fail-on-unknown` exposes the gap, and neither the CI job nor the hook passes it. PowerShell files get an ASCII `Copyright (c)` header, rendered from a conditional in the header template, instead of the copyright sign the other file types use. A file with any non-ASCII byte needs a byte order mark to satisfy PSScriptAnalyzer, and install.ps1 cannot have one: the documented install path is `irm ... | iex`, which hands the parser a string, so the mark survives into it and the param() block stops being the script's first statement, silently dropping every installer parameter. Keeping the headers ASCII leaves install.ps1 as ASCII-only as its existing comment says and needs no analyzer suppression. The other PowerShell files follow for consistency, and LICENSE.TXT already uses the `(c)` form. Headers for every other file type render byte-for-byte as before. The license-headers hook already runs on every commit (it has no file-type filter), so no hook change is needed: a PowerShell-only edit reaches it. PSScriptAnalyzerSettings.psd1's comments named scripts that do not exist; they now describe what the lint covers (every tracked .ps1) and note that .psm1 and .psd1 are licence-checked but not linted. Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com> (cherry picked from commit f4b9945) Signed-off-by: juhovainio <juho.vainio@amd.com>
…guard .rocmNNN (#1109) * feat(vllm): add a ROCm 10.2 discover row and teach the local-version guard .rocmNNN The staging index publishes vLLM 0.32 wheels for ROCm 10.2 (local version spelled .rocm102), but the discover table stopped at 10.1, so the install failed closed on a 10.2 SDK. Add the 10.2.0 row (cp314, staging index, torch 2.14, torchvision 0.29). whl-next has no +rocm10.2 torch, so the row borrows the 10.1 stack and the new torch_stack_matches_sdk flag skips the local-version check for the torch triplet on that row only. The guard now also compares the compact .rocmNNN spelling, so a wheel for another line is still rejected. Refs #1108 Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): read the 10.2 torch stack from the staging index The staging host publishes torch 2.14.0, torchvision 0.29.2 and torchaudio 2.11.1 built for +rocm10.2.0rcN, so the 10.2 row no longer borrows the 10.1 stack. Drop the torch_stack_matches_sdk flag and the check skip it needed. Refs #1108 Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): reuse the SDK's torch stack for the 10.2 row Running the row on a real 10.2 SDK showed the staged torch does not work: the newest torch 2.14.* on the staging and nightly indexes is a +rocm10.3 build, and the staging +rocm10.2.0rcN torch needs device payloads and a rocm==10.2.0rc2 that staging does not serve. Add TorchStack::FromInstalledSdk: the row keeps the torch, torchvision and torchaudio the SDK install resolved, checks them against the row's series and ROCm line, and realigns to those exact builds from the nightly index. The other rows still discover theirs. Refs #1108 Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): fetch the 10.2 row's torch stack from the index the SDK came from The 10.2 row reuses the SDK's own torch, torchvision and torchaudio but fetched those exact builds again from the row's fixed nightly index. A release SDK's torch is not on that index, so the realign step would fail once 10.2 ships through the release channel. Read the index the runtime manifest records for the interpreter and use it for the torch install and both realign steps. A manifest that recorded none keeps the row's index, and an unreadable registry does the same but leaves a note. Rows that discover their stack are unchanged. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): honour exact torch pins and bare +rocmNNN, and correct the discover-rows doc Review fixes for the 10.2 discover row: - installed_torch_stack_pins read an exact torchaudio pin as a series prefix of a version that still carried its +local part, so an Exact pin rejected the release it names and accepted longer ones. An exact pin now means that release, a series keeps its prefix rule. - ensure_rocm_local_version_matches sent a local version starting with rocm to the dotted parser only, so a bare +rocm102 was rejected. The dotted spelling is now read from its first rocm segment wherever it sits, and the compact one is the fallback. - Assert the 10.2 row's torch_index_url, which stays the fallback index. - docs/vllm.md claimed all three rows share the stable index and resolve their torch pins fresh; that holds for 10.0 and 10.1 only. Signed-off-by: juhovainio <juho.vainio@amd.com> * fix(vllm): reject a rocm segment no spelling reads, and say when the torch index falls back Follow-up review fixes: - A local version with a rocm segment that is neither the dotted nor the compact spelling (+rocm, +rocmabc) passed the guard since the last change; it is a mismatch again, as it was before. - Only the row that reuses the SDK's torch stack looks up the recorded index, so an unreadable registry no longer adds a note to 10.0 and 10.1 installs, whose index is fixed by design. - That row also says so when the manifest records no index, and every failure on the torch install and the realigns carries the fallback notes, not just the remaining-deps install. - The unreadable-registry test skips itself where mode 000 does not block the reader (running as root). - docs/release-trust.md: an overridden pip base is recorded in the manifest and reused by later vLLM installs for that runtime. Signed-off-by: juhovainio <juho.vainio@amd.com> * refactor(vllm): attach the fallback notes to every failed discover step, and test them Review nits on the 10.2 discover row: - The notes about a fallback (companion pins, the torch index) were attached to individual failures one call at a time, so errors before the torch install and between the realigns still lost them. The sequence now records them as it goes and the entry point attaches them once, to whatever fails. - Test each real install step failing on the fallback index, not just the first. - The unreadable-registry case plants a directory where a manifest file belongs, so it fails for root too and no longer needs a skip. - Say 'no runtime manifest for this interpreter records a wheel index', which also covers an interpreter with no manifest at all. Signed-off-by: juhovainio <juho.vainio@amd.com> --------- Signed-off-by: juhovainio <juho.vainio@amd.com> (cherry picked from commit 44d599f) Signed-off-by: juhovainio <juho.vainio@amd.com>
…1070) Follow-up to #1011 review. The contract test now also requires the doc to give the run limit in hours next to its seconds (3 h (10800 s), and ties the workflow comment to timeout-minutes exactly: the 30 min, a margin derived from timeout-minutes less the two limits and 30 min, and the total, in the lines right above the timeout-minutes line. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 3209459) Signed-off-by: juhovainio <juho.vainio@amd.com>
….ops (#1103) * ci(e2e): fail the pre-warm when the runtime cannot import torchvision.ops A persisted pre-warm runtime can pass every freshness check and still be unusable: when its torch and torchvision come from different installs, `import torchvision` dies with `operator torchvision::nms does not exist`. Nothing ran that import before the suite, so each vLLM serve scenario found out for itself after waiting out its readiness timeout, and a lane that takes about 33 minutes when healthy ran 79 to 90 minutes and failed. The pre-warm now probes the active runtime's own Python, which is where vLLM installs its packages, for `import torchvision.ops`, with the runtime's recorded library directories on the loader path and a two-minute deadline: - Before the freshness decision, a runtime whose torch and torchvision do not match is uninstalled, so the decision installs afresh. A pinned run, a runtime marked ahead of the index, and a hang or any other import error are left alone. - After the engine check, which also runs when the runtime is reused, a runtime that still cannot import fails the pre-warm, naming the runtime and what to do. Where the default engine does not import torch (Lemonade) the finding is a warning. A runtime without torchvision is skipped. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci(e2e): repair, bound and scope the pre-warm's torchvision check Review round on the torchvision import check: - The probe puts the runtime's recorded `rocm_sdk.library_paths` (and the amd-smi shim) on the loader path, as the CLI's own probes do. A library that still will not load is a broken runtime; the skip remains only for a manifest that records no library directories. - The probe is killed after two minutes and reported, instead of stalling the pre-warm until the job times out. - A runtime whose torch and torchvision do not match is uninstalled before the freshness decision, the same repair as for a poisoned runtime, so that decision sees a registry without it and installs afresh. Other import failures still fail the pre-warm without deleting anything. - The finding is a warning where the default engine does not import torch (Lemonade). - The check moves to its own file to keep e2e_prewarm.rs under the module-size limit. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci(e2e): bound the probe's pipe readers and tighten its tests - The wait on the probe's reader threads is bounded, so a process that inherited the pipes cannot hold the pre-warm past the probe's deadline. The readers are detached and end when the pipes close. - The mismatch signature is kept in the error tail wherever it appears in stderr, not only in its last three lines. - The repair test's stand-in `rocm` removes the runtime's registry entry, so its final assertion fails when the uninstall is not requested; the table of cases that must not be uninstalled has a real hang case in place of a duplicate. - The source-order test checks the order of the calls rather than exact lines. - Tests that write and run a script hold a lock, since a concurrent fork makes the exec fail with ETXTBSY. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci(e2e): gate a unix-only test constant, anchor the mismatch match - `MISMATCH` is used only by unix tests, so it is dead code under `-D warnings` on Windows; gate it with them. - The operator-mismatch test requires both `torchvision::` and `does not exist`, since it decides whether a runtime is deleted. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> * ci(e2e): leave a pinned or ahead-of-index runtime in place The torchvision repair runs before the freshness decision, so it would also have removed a runtime the pre-warm never rolls back. It now declines when the run is pinned to an exact version, when `rocm update` marks the runtime `ahead_of_index`, or when that report cannot be read; the check after the engine install then fails with the manual advice. The real-interpreter test skips, rather than fails, when `python3` is not on PATH. The documentation says that a mismatch the engine install itself creates on a reused runtime is repaired on the next run. Signed-off-by: Tomas Saaristola <tsaarist@amd.com> --------- Signed-off-by: Tomas Saaristola <tsaarist@amd.com> (cherry picked from commit 6e23942) Signed-off-by: juhovainio <juho.vainio@amd.com>
`rocm serve` protects a vLLM endpoint (public bind, --require-api-key, `rocm remote serve`) by setting VLLM_API_KEY. vLLM checks that key only under a fixed set of path prefixes (/v1 and a few others), while the server also answers routes outside them that run or expose the model, such as /invocations, /tokenize, /detokenize and /metrics. Those routes answered clients that did not present the key. A keyed vLLM launch now also passes `--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`: a pure-ASGI, standard-library middleware compiled into the engine adapter, written atomically under the engine's data directory at launch and prepended to the server's PYTHONPATH (composed over the inherited, removed or explicitly set value, so the user's own PYTHONPATH and the runtime's are kept). It requires the bearer key, compared in constant time, on every HTTP and WebSocket route except GET/HEAD /health and CORS preflights, and raises instead of constructing when the key is empty, so a server that loaded it never answers a request unchecked. Being path-agnostic, it also covers routes a later vLLM adds. The module's copyright comment is exempted from the xtask ASCII-output contract, as install.sh's is; the rest of the file is ASCII. The launch command is built by one function, vllm_server_command, so a keyed launch can be asserted to carry --middleware, VLLM_API_KEY, the middleware directory ahead of the runtime's own PYTHONPATH entry and the selected GPU, and an unkeyed launch none of the key protection. Because /metrics is now covered, `rocm dash` cannot read engine metrics for a key-protected vLLM server and shows a persistent scrape warning, and `rocm bench load`, which sends no key, cannot benchmark one. The README says so, and that the every-route guarantee is the vLLM engine's: AIM runs the same vLLM server without this check, so routes such as /invocations, /tokenize and /metrics still answer there without the key. The manual-testing, testing and architecture docs describe the check; manual-testing sections after the new one are renumbered, and the three places that cited one of them by number now cite it by title. Tested by driving the shipped middleware per route under python3 with a stdlib ASGI harness (including CORS-preflight look-alikes and a WebSocket /health), by asserting the launch argv and environment, and end to end by serve-61 (`@id:serve-vllm-key-guards-every-route`), a `@nightly` GPU scenario that runs in the nightly runs on Instinct GPUs, which serve with vLLM (the MI300X job in nightly.yml and the shared GPU queue's Instinct lanes). A real keyed vLLM server refuses keyless calls on those routes, accepts keyed ones, and still answers /health. The e2e step passes the key through the environment, with a new time-bounded run helper, so it stays out of the recorded command log. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 42c2968) Signed-off-by: juhovainio <juho.vainio@amd.com>
* test(e2e): fail lanes that test nothing (ROCMAI-388, ROCMAI-450) A self-hosted GPU lane could pass without testing its GPU. The capability probe ignored `rocm examine`'s exit status, so a probe that failed read as a host with no GPU and every @requires-gpu scenario resolved to skip. Skipped scenarios never reach report.json, and the only outcome-based exit iterated its results, so a run in which everything was filtered out exited 0. Lanes now declare E2E_EXPECT_GPU: 1 on the self-hosted GPU lanes and the queue lane script, 0 on the WSL lanes, whose distro is deliberately left without ROCm's WSL driver bridge. The harness fails before any scenario when the probe could not read the binary (spawn failure, non-zero exit, empty output) or a declared GPU is missing, and after the run when nothing executed or, on a GPU-declaring lane, no @requires-gpu scenario executed. A workflow contract test holds every self-hosted lane to the declaration. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): bound probes, flag failed lanes (ROCMAI-388, ROCMAI-450) The lane checks added in the previous commit could still be defeated in two ways, and this closes both. A capability probe that never returned hung the whole run instead of refusing it. Each probe command (`rocm examine`, `rocm engines list`) now runs under a deadline, E2E_PROBE_TIMEOUT_SECS (whole seconds, 120 by default), and is killed and reaped when it passes. ProbeError gains Timeout, Wait (the child could not be waited on) and OutputNotClosed (the command exited but a process it started still holds its output), each naming the command and keeping the stderr read so far. The post-run version probe shares the deadline but only blanks the version columns. A lane failed by the post-run check (no scenario executed, or no @requires-gpu scenario on an E2E_EXPECT_GPU=1 lane) exited non-zero but still read green in the consolidated report when every scenario it resolved was a skip. The reason is now written to platform.json as `lane_failure`, and the report renders that lane as FAIL with the reason, in both the step summary and the HTML report. The single "Needs attention" list is split into a platform heading (lane-check failures) and a scenario heading (the expectation grid), so neither renderer shows two identical headings. Release readiness counts such a row, whose Fail cell is 0 and so never reaches the Total row, and does not count a row twice when it also has failed scenarios. Annotation and notice lines now escape `%`, CR and LF as the runner expects, so a probe's stderr tail can no longer truncate or corrupt a workflow command. The decisions the harness takes around these checks (reading the declaration, counting GPU scenarios, building the manifest, the final exit and the exact annotation lines) move out of tests/e2e.rs into pure functions in lane_verdict.rs, each unit-tested; the harness keeps only the glue that runs the probe and the scenarios and writes the files. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): tighten lane-verdict tests and docs (ROCMAI-388, ROCMAI-450) The GPU-lane check already counts @requires-multi-gpu scenarios as GPU scenarios, but its messages, doc comments and docs named @requires-gpu alone. The fatal reason and the E2E_EXPECT_GPU=0 note now name both tags, and the docs say that GPU scenarios the run's own configuration leaves out (@nightly, @merge-queue, outside an E2E_ONLY_LIFECYCLE selection) count as resolved but never as executed. New tests: the E2E_EXPECT_GPU=0 note's "N of M" arithmetic when some GPU scenarios ran (4 of 7), and a probe that reads stdin getting EOF at once rather than blocking until its deadline. Like its rocm-core sibling, the stdin test catches a missing stdin setting but does not pin null against inherit. The constant-equals-itself test is dropped; the workflow contract test already ties the name to the workflows. The workflow contract test now also fails when the step that runs the suite sets E2E_EXPECT_GPU itself, overriding the job-level declaration, when a lane has no such step, and when that step runs in WSL without a WSLENV of its own naming the variable. Stale comments corrected: the workflows now say the Windows lane's E2E_EXPECT_GPU=1 check backstops its best-effort GPU preflight, and that a harness preflight refusal shows as an EMPTY row rather than a missing report; run-e2e-lane.sh says a set-but-empty E2E_EXPECT_GPU leaves the lane undeclared. The wsl fixture gets the driver status a passthrough host reports, so tests built from it no longer reset the field. The markdown report folds any whitespace run in a lane-failure reason to one space, CRLF included, not only LF. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * test(e2e): let the post-kill drain test fail (ROCMAI-388, ROCMAI-450) The late writer in the post-kill drain test inherited stdout as well as stderr, so the stdout wait already held the probe open until `late` was written, and the test stayed green with the stderr wait removed. Send the writer's stdout to /dev/null so only the stderr wait can collect it. Also correct two notes in the testing docs: passing an @id to -n selects nothing, and the run then fails with "0 scenarios executed" rather than passing silently. Refs: #383 Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(e2e): name report.json origin, drop counts (ROCMAI-388, ROCMAI-450) The probe-grace rationale in capability.rs's output_within doc, and the leaked-thread note under it, rested on "three" probes. They now state the invariant: each probe runs once per suite (the capability probes cached by host_capability, and collect_versions' version probe), not once per sample. The Cargo.toml note on the copied spawn-and-reap helpers likewise says "these copies" rather than counting them. The claim that a harness preflight refusal leaves an empty report.json that is still uploaded now names where that holds: run_suite in tests/e2e-cucumber/tests/e2e.rs creates report.json before it calls lane_verdict::preflight, and on every self-hosted lane that ran its suite the results upload (and the WSL copy-out) is gated on always(). This is said in the e2e-report test doc, the e2e README and both nightly.yml comments. Rewrapped the over-long lines in nightly.yml's missing-report comment, lane_verdict.rs's parse_gpu_declaration doc and e2e-report's EXPECT_GPU_ENV doc. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 698b9d5) Signed-off-by: juhovainio <juho.vainio@amd.com>
Turn on four more Ruff rule families for every Python file in the repository: flake8-unused-arguments (ARG), flake8-blind-except (BLE), flake8-self (SLF) and the Pylint set (PL). SLF and the Pylint error rules report nothing. ARG needed two fixes: - `setup(app)` in the Sphinx config is removed. Sphinx calls the config's `setup` and discards what it returns, and the parallel-safe flags it returned only count for registered extensions, so it did nothing; a config without `setup` is skipped. - A dead `temp_state_root` parameter is dropped from `run_live_acceptance` in the local-assistant GPU test along with its argument at the single call site. What still fires is waived in ruff.toml. The global ignores give their reason, and each per-file entry states the rule new code under it has to keep: - PLR0912, PLR0913 and PLR0915 are ignored for every file, since acting on them means restructuring existing scripts. PLR2004 is ignored for every file as well: it flags inline numeric literals in comparisons, and naming them is a separate cleanup. - BLE001 is allowed in comfyui_therock_gpu_test.py, therock_sdk_install_test.py and vllm_therock_gpu_test.py. - PLC0415 and PLW0603 are allowed in vllm_route_check.py, whose self-test imports inside the function and rebinds a module global it restores in a `finally`. - ARG001, PLW0603 and PLW1510 are allowed in the rocm-doctor eval hooks and their tests; ARG001 covers only the unused parameters of setup() and teardown(). Those files are waived rather than edited because a change to skills/rocm-doctor/ on main opens a bump pull request in the federated copy. A waived rule checks nothing anywhere in that file, so new code in those files is reviewed by hand. ruff.toml and .pre-commit-config.yaml also described the Python scope as covering "engine workers", which is Rust-only; both now say every Python file in the repository, so they stay true as Python files are added. Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com> (cherry picked from commit 6427b50) Signed-off-by: juhovainio <juho.vainio@amd.com>
…452) (#1088) * fix(lemonade): escalate terminate_pid to SIGKILL after a grace period (ROCMAI-452) Lemonade's terminate_pid(pid, force) ignored `force` and only sent SIGTERM, and every caller follows it with an unbounded child.wait(). A lemond or llama-server that ignores SIGTERM therefore hung the CLI indefinitely during teardown. Route terminate_pid through rocm_core::terminate_verified with the Tree scope and the engine's 10 s stop grace: SIGTERM to the process tree, then SIGKILL for anything still running when `force` is set, so the call is bounded and the following wait returns. Tree scope matters because lemond runs each loaded model in a llama-server child, and killing lemond alone would orphan that child with the GPU still held. vLLM's serve_http had the same pattern in its readiness-failure teardown (terminate_process_tree, then wait); it now goes through a stop_spawned_server helper with the same bounded escalation. Document on terminate_process and terminate_process_tree that they only request termination and neither wait nor escalate, pointing at terminate_verified, and refresh the architecture entries. Regression tests cover a child and a grandchild that ignore SIGTERM, the unforced case, a cooperative child, and the Windows forced path. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(engines): report unconfirmed teardown stops (ROCMAI-452) The bounded teardown from the previous commit still returned only a bool, and every caller discards it before waiting on the child. A stop that could not be confirmed, either because the process outlived the grace periods or because its identity could not be read and nothing was signalled, was followed by the same silent wait. Both engines now write the reason for an unconfirmed stop to stderr. The text comes from a new TerminationOutcome::unconfirmed_reason(), which is None exactly when stopped() is true, so the warning cannot disagree with the decision it reports. vLLM's readiness-failure teardown moves into fail_unready_server. It owns the unreaped Child, so after an unconfirmed stop it kills the server before waiting, which bounds the wait on every path except a server that survives SIGKILL. A server the readiness poll already reaped is not signalled, because its PID may belong to an unrelated process by then. A failed write of the `failed` state is reported on stderr, so the readiness error stays the one returned. Lemonade keeps terminate_pid(pid, force) -> bool and does not kill on an unconfirmed stop, because it is called with bare PIDs; its docs now say that an Unverified process is left unsignalled and a following wait can block. Tests re-run the test binary to read the warning on real stderr together with the resulting process state, cover the production stop grace through both entry points, and check that a confirmed stop neither warns nor kills and is reaped before the teardown returns. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * fix(lemonade): bound the router serve teardown (ROCMAI-452) The router serve path, which moved into its own module when registered GGUF models gained a router route, stops a server that failed to load or verify with a plain SIGTERM to its process tree and then waits on the child. A lemond, or a backend under it, that ignores SIGTERM therefore hung the serve on that failure path, the same hang the previous commits removed from the other teardown sites. Route it through terminate_pid like its siblings: SIGTERM to the tree, SIGKILL after the stop grace, and a stderr warning when the stop cannot be confirmed. A test stages a fake lemond and a backend child that both ignore SIGTERM, and checks that the stop returns within a bound, reaps the server and kills the backend. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * docs(engines): say what the stop-reason test and STOP_SCOPE cover (ROCMAI-452) The unconfirmed_reason test pins the reason strings, so a consistent wrong edit to the production text and the test would stay green. Name the engine tests that prove the warning is printed exactly when a stop is unconfirmed, next to the state it describes, as the place the behaviour is actually checked. The STOP_SCOPE comment said Lemonade state always records the llama-server itself. That holds on the direct-serve path only: the router path records lemond, whose model runs in a child process, so a Single stop that escalates to SIGKILL can leave that child running. Say so, without changing the scope. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> (cherry picked from commit 340d987) Signed-off-by: juhovainio <juho.vainio@amd.com>
Contributor
Security scan found no critical or high findings on lines changed by this pull request17 medium, low, or unrated finding(s) are not listed; the scan run has them. View the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports the 32 commits that landed on
mainafterv0.3.0-rc.2was cut (46b9e3b5..340d9873) ontorelease/v0.3.0, inmainorder, so av0.3.0-rc.3can be tagged from the release branch.Every pick was made with
git cherry-pick -x -s, so each commit carries its(cherry picked from commit …)line. Each change was reviewed and tested in its original PR; this PR adds no new behavior of its own. The e2e scenarios those PRs added or updated come along with them.Conflict resolutions
Four picks conflicted, all with release-only docs. The release wording is kept and
main's behavior is taken:README.md(ROCMAI-455: keep endpoint API key files owner-only on every platform #1043): release wording, plus the new sentence that the endpoint key file is readable only by your user account.README.mdanddocs/wsl.md(fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend #855):main's text, since it documents the new Lemonade pin and the host glibc/OpenSSL check.docs/rocm-docs/conf.py(chore(lint): enable ARG, BLE, SLF and PL ruff rules #370):main's ruff change removed Sphinx'ssetup(), but the release branch added asetup()hook that points the Core SDK link at the latest docs. I kept the hook and added# noqa: ARG001to its callback, whose signature Sphinx fixes. The ruff check and format hooks pass on it.After this change the tree differs from
mainonly in the release-only docs (docs/rocm-docs/,docs/vllm.md,docs/wsl.md,CONTRIBUTING.mdandREADME.mdformatting) andlicenserc.toml.Verification
Run locally on this branch:
cargo clippy --workspace --all-targets -- -D warnings: cleancargo xtask smoke: okcargo test --workspace --all-targets --exclude rocm --no-fail-fast: passescargo test -p rocm: passes at--test-threads=4. At default parallelism a ComfyUI test (a different one each run) and once aservice_launchport test failed; each passes alone and in repeated runs, so they depend on machine load rather than on this change.origin/release/v0.3.0: no internal referencescargo xtask verify-commits --base origin/release/v0.3.0: all 32 commits signed and signed-offThe branch was pushed with
--no-verifybecause the localverify-commitshook defaults toorigin/mainas its base and flagsb38c87ef, a commit already onrelease/v0.3.0that is unsigned there.Not run locally: GPU-gated lanes and the Windows jobs. CI is the authority for those.