Skip to content

Backport main since v0.3.0-rc.2 to release/v0.3.0 - #1122

Merged
juhovainio merged 32 commits into
release/v0.3.0from
cherry-pick-main-to-release-v0.3.0-rc3
Nov 17, 2026
Merged

juhovainio merged 32 commits into
release/v0.3.0from
cherry-pick-main-to-release-v0.3.0-rc3

Conversation

@juhovainio

Copy link
Copy Markdown
Collaborator

Backports the 32 commits that landed on main after v0.3.0-rc.2 was cut (46b9e3b5..340d9873) onto release/v0.3.0, in main order, so a v0.3.0-rc.3 can be tagged from the release branch.

Every pick was made with git cherry-pick -x -s, so each commit carries its (cherry picked from commit …) line. Each change was reviewed and tested in its original PR; this PR adds no new behavior of its own. The e2e scenarios those PRs added or updated come along with them.

Conflict resolutions

Four picks conflicted, all with release-only docs. The release wording is kept and main's behavior is taken:

After this change the tree differs from main only in the release-only docs (docs/rocm-docs/, docs/vllm.md, docs/wsl.md, CONTRIBUTING.md and README.md formatting) and licenserc.toml.

Verification

Run locally on this branch:

  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo xtask smoke: ok
  • cargo test --workspace --all-targets --exclude rocm --no-fail-fast: passes
  • cargo test -p rocm: passes at --test-threads=4. At default parallelism a ComfyUI test (a different one each run) and once a service_launch port test failed; each passes alone and in repeated runs, so they depend on machine load rather than on this change.
  • conflict-marker grep: none; leak scan against origin/release/v0.3.0: no internal references
  • cargo xtask verify-commits --base origin/release/v0.3.0: all 32 commits signed and signed-off

The branch was pushed with --no-verify because the local verify-commits hook defaults to origin/main as its base and flags b38c87ef, a commit already on release/v0.3.0 that is unsigned there.

Not run locally: GPU-gated lanes and the Windows jobs. CI is the authority for those.

juhovainio and others added 30 commits November 17, 2026 22:19
…rix Halo Ubuntu (#1058) (#1097)

The row marked serve-aim-lifecycle-parity as an expected failure on native
Linux Strix Halo. It passed on both channels of the 2026-11-15 and
2026-11-16 nightlies, which the harness reports as a stale XPASS and fails
the lane on (exit 1), so the lane has been red for that alone.

Signed-off-by: juhovainio <juho.vainio@amd.com>
(cherry picked from commit 2980faf)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…1043)

* fix(serve): keep endpoint API key files owner-only on every platform (ROCMAI-455)

A served endpoint's API key file was created 0600 on Unix only when it
did not exist yet: re-storing a key over an existing file truncated it in
place, so a key file whose mode had been loosened (for example to 0644)
stayed readable by others. On Windows the file was written with default
permissions and inherited the data directory's ACL, so any account that
could read the data directory could read the key.

Add rocm_core::owner_only_file::open_new_owner_only, which creates a new
file only the current user can open: mode 0600 on Unix, set again on the
handle so the umask cannot leave it unreadable to its owner; on Windows a
protected DACL granting only the process token's user, applied by
CreateFileW itself so nothing is inherited. It never opens an existing
file or symbolic link.

Add atomic_write::write_owner_only_file_atomically for secrets: it stages
the bytes in such a file and publishes by rename alone, so the old file's
mode or DACL is never carried over and a symbolic link at the destination
is replaced rather than written through.

Endpoint keys are now stored with it, so a re-store repairs a loosened
key file, and the remote session key write uses open_new_owner_only,
gaining the Windows DACL. Docs and comments say "owner-only" instead of
"0600", and scenario serve-59 checks the mode of a real serve's key file
on the Linux GPU lanes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(serve): remove staged key copies, harden Windows open (ROCMAI-455)

A store killed between staging the endpoint key and renaming it into
place left a plaintext copy beside it
(<id>.endpoint-key.tmp-<pid>-<millis>-<n>) that nothing ever removed.
Clearing a key on stop, in both rocm and rocmd, now removes such copies
too; `rocm services remove` and `prune` delete and count them with their
record, and prune sweeps a copy whose record is gone under the usual age
gate. A copy is recognised only by the exact suffix the atomic writer
produces, so a service id crafted to look like one cannot get another
service's key deleted. A store racing a clear for the same service now
fails instead of bringing the cleared key back.

On Windows the owner-only create passed neither
FILE_FLAG_OPEN_REPARSE_POINT nor a long-path name, unlike std's
create_new: CREATE_NEW could follow a dangling symbolic link and create
the file at its target, and paths past MAX_PATH failed. It now does what
std does on both counts.

Add Windows tests for the symlink and long-path cases, a negative
control for the DACL check, and DACL checks of the endpoint and remote
session keys the apps store, through a test-support helper enabled only
as a dev-dependency. Extend scenarios service-cleanup-01 and -03, and
bring README, the services remove help and the docs in line, including
that on Windows a key-protected AIM serve stores its key but cannot
start.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(serve): pin the owner-only create mode and correct key-cleanup claims (ROCMAI-455)

The owner-only file is meant to be unreadable to others from the moment
it exists, not only once its mode is set again on the handle. Nothing
checked that: a create that asked for a wider mode would still have
passed every test, because the follow-up reset hid it. Split the create
into its own function and test the mode it leaves under umask 0, in a
re-executed child so the umask change cannot reach other tests. No
behaviour changes.

The cleanup comments overstated what removing a service's key does. The
previous commit's message and the doc comments said a store racing a
clear fails instead of bringing the cleared key back, fail-closed. That
is not guaranteed: the cleanup removes the key and the staged copies
present when the folder is listed, best-effort, and a concurrent store
for the same service may still publish a key or may fail. The comments
now say only that, and that an unlistable folder or an unremovable file
is left in place without being reported.

docs/testing.md now names the unit tests that fail without the repair
of a loosened key file, and says serve-59 guards only the fresh-create
path, which held before the repair too. docs/manual-testing.md states
that what happens to a key-protected AIM serve's key file on native
Windows when its server fails to start has not been verified, and keeps
the remote session key as the check there.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(core): pin owner-only publish and staged-key paths (ROCMAI-455)

Several properties the fix relies on held only by inspection. The
owner-only writer must never publish through a step that can fall back
to ReplaceFileW, which would carry the replaced file's DACL over to the
secret; a test-only counter at every such step now shows it is never
entered, and that the general writer is counted. The umask-0 "as
created" check went through a helper that production no longer had to
call; it now drives the real create through a test-only switch that
skips only the handle mode reset. Staged-name recognition is pinned to
reject empty digit runs.

staged_siblings read the empty parent of a bare relative name and so
found nothing; it now lists the current directory for it and returns
names in the form it was given, tested in a scratch directory via a
re-executed child so the working directory change stays contained.

service-cleanup-03 now also plants a staged key copy for the record
prune removes and checks it goes with the record, so that path has
scenario coverage and not only a unit test. The serve-59 step uses the
bounded serve runner, so a serve that never returns fails the scenario
instead of stalling the lane, and its comments state the fixed readiness
wait rather than claiming readiness is not awaited. Comments on the
restart key handoff and the Windows long-path conversion are corrected
to match the code.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 778c970)
Signed-off-by: juhovainio <juho.vainio@amd.com>
… review fixes, skip tags and commit messages (#1076)

* docs(contributing): check code-behavior claims in comments (ROCMAI-41)

A comment or doc line that states how the code behaves is easy to accept,
and no test reads it, yet a decision such as narrowing a test's scope or
dropping a guard often rests on one such sentence. AGENTS.md section 3 now
requires checking each such claim against the code it describes. Where the
claim is the reason for a decision, it names the place as a file and a
function, constant or test rather than a line number, which goes stale; a
claim about timing or ordering names where the enforcing mechanism is
defined. It also prefers an invariant to a count, and asks that a sentence
rewritten for a review finding be checked the same way.

Section 11 tells reviewers to treat such lines in a diff as claims to check,
the pre-flight checklist gains a matching item, and CONTRIBUTING.md and the
pull request template point at section 3.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(contributing): name a blocking objection's mechanism (ROCMAI-131)

An objection resting on a timing, ordering, concurrency or "this cannot
happen" argument is hard to answer when it does not say which mechanism it
assumes. AGENTS.md section 11 now asks a reviewer to name that mechanism and
where it is defined before marking such an objection blocking, and to post
it as a question when the mechanism cannot be located. CONTRIBUTING.md's
list of what reviewers look for points at the rule.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(contributing): review a review-fix commit as a whole (ROCMAI-44)

A commit answering review feedback is written in a narrow window and gets
the least scrutiny, so it is where new defects arrive. AGENTS.md section 11
gains a Review-fix commits group. The author diffs such a commit whole
against the head the reviewer saw, and says whether it changes behavior
outside the finding. Once the base has moved into the branch, the author
uses git range-diff instead, and checks whether each commit it marks removed
is the base's old history or a commit dropped since the review. A reviewed
head a force-push removed is fetched by its full SHA. The commit does not
widen what a documented user-facing option means or authorizes; a needed
widening goes in its own commit with the option's help text and docs.

A reviewer diffs such a revision whole as well, section 12's self-review
reminder names these rules, the pre-flight checklist gains a matching item,
and CONTRIBUTING.md and the pull request template point at section 11.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(contributing): treat e2e skip tags as deletions (ROCMAI-161)

A tag that makes an existing scenario skip on the blocking mock e2e job
removes its pre-merge coverage, yet the diff shows only a tag and no
coverage floor catches it. AGENTS.md section 8 now treats moving a scenario
off that job, in any change including CI plumbing, as a deletion: the PR
text names, for each direction the behavior could be broken in, which
blocking check still fails. A scenario that is the only pre-merge guard for
a seam says so in a comment beside it, and a neighbor it slows or
destabilizes is fixed at its own layer rather than by moving the guard.

Section 3's gated-lane bullet points such moves at section 8, the pre-flight
checklist gains a matching item, and CONTRIBUTING.md and the pull request
template point at section 8.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(contributing): review commit messages for accuracy (ROCMAI-182)

Pull requests here are normally squash-merged with the branch's commit
messages as the default squash body (rebase-merge is also enabled; either
way each commit message lands in public history), and that history is not
rewritten. AGENTS.md section 5 now requires each commit message to
describe what its own commit does, in words still true at the branch head.
A message whose words a later commit made misleading is reworded before
merge, and a commit whose whole subject a later commit removed is folded
into that later commit or dropped; either needs an approved force-push.

Section 11 tells reviewers to read the commit messages with the diff, the
pre-flight checklist gains a matching item, CONTRIBUTING.md gains a short
section on why commit messages are reviewed for accuracy, and the pull
request template gains a checkbox for it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(contributing): scope the code-claim rule to the change (ROCMAI-41)

The AGENTS.md section 3 rule, its pre-flight item and the CONTRIBUTING.md
reviewer bullet said every comment or doc line stating how the code
behaves is checked against that code, with no scope. Read on its own,
that asks a change to re-verify every existing comment in a file it
touches. They now cover the lines a change adds or edits, matching the
section 11 reviewer bullet ("in the diff") and the pull request template
("this PR adds or changes").

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 3a08d6b)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…t (ROCMAI-454) (#1039)

* fix(rocmd): stop the engine child when the supervisor cannot record it (ROCMAI-454)

std::process::Child does not kill on drop, so a failing engine_pid or
ready write after the engine was spawned left the engine running with no
supervisor, holding the GPU. When the failed write was the one recording
the PID, the record did not name the engine either, so `rocm services
stop` had nothing to stop and reported success.

A guard now terminates the engine's process tree through rocm-core's
terminate_recorded_pid_within and reaps it whenever the supervisor leaves
before waiting for it, on an error return or a panic. A waited-for child
is not signalled.

terminate_recovery_supervisor shares the same terminate_and_reap helper.
On Windows that adds a taskkill /T over the supervisor's descendants,
which the previous root-only kill did not reach.

The existing recovery-cleanup test re-executed itself with a filter
naming a stale module path, so it matched zero tests and the child
exited at once, which the test read as a kill. The filter is now derived
from module_path!() and the spawner waits for a marker proving the child
is sleeping.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(rocmd): report unconfirmed terminations and cover the engine's process tree (ROCMAI-454)

The engine guard and the recovery-supervisor cleanup now report a
termination they could not confirm instead of discarding the outcome,
and the tests exercise the shapes the earlier change only asserted for
a single process.

The kill tests now terminate real process trees: the sleeper can start
a grandchild, and the stand-in engine runs a shell with a child under
it, so the tests fail if the termination stops reaching below the
child. The termination step and the report sink are injectable, which
lets the tests hand the reap a TimedOut or Unverified outcome a real
child cannot produce: there the child is not reaped with a blocking
wait and is otherwise left alone, and a confirmed stop is still waited
for and reaped.

A termination that could not be confirmed is now reported. The engine
guard writes "rocmd: terminating the engine (pid N) ended <outcome>:
... may still be running" to its stderr, which is the service log, and
the recovery path attaches the same line to the restart error the
daemon records. Both failure arms of the restart go through one helper,
abandon_recovery_supervisor, so neither can terminate without
reporting. The line is written so that a failed write cannot panic
inside Drop.

A failed wait leaves the guard armed; the decision moved into
settle_wait so a test can hand it a failed wait. The engine's log is
emptied and then reopened in append mode, so a surviving engine cannot
write over a line the supervisor appended to the same file. Doc
comments now say the reap is conditional on the termination outcome.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(rocmd): pin the guard's default report and the per-run log truncation (ROCMAI-454)

The engine guard's report goes to the supervisor's stderr when it is
given no writer, but only injected writers were tested. A new test runs
the guard in a child test process and reads that process's real stderr
back, so the production default is exercised rather than assumed.

Each supervised run empties the service log before the engine writes to
it, and nothing pinned that. A test now seeds the log with a line from
an earlier run and checks it is gone and this run's output is there.

The PID write and the ready write now carry distinct error contexts, and
each guard test asserts which write failed, so a test cannot pass by
failing at a different record write than the one it targets.

The stand-in engine waits in these tests are bounded, and the tests
register the engine-spawn seam, so a run that times out kills its engine
instead of leaving it sleeping.

Comments and the architecture row are narrowed to what supervise.rs
itself does. An external kill of the supervisor (a signal, the OOM
killer) bypasses the guard; that is noted as not covered and tracked as
ROCMAI-957.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(rocmd): pin the already-exited reap and publish the sleeper marker atomically (ROCMAI-454)

A new Linux test hands TimedOut and Unverified to a child that has
already exited and asserts the child is reaped, pinning the try_wait in
terminate_and_reap_with. The test first waits for the exit with
waitid(WNOWAIT), which observes it without reaping, so the child is
certainly an unreaped zombie when the call is made and there is no
timing race.

The test sleeper now publishes its marker by writing a sibling file and
renaming it into place, and the spawner waits until the marker holds a
parseable grandchild PID. A half-written marker can no longer panic the
test and leave a sleeper tree running.

supervise_on_stand_in_rocm now tells a panic in the supervisor thread
(its result channel disconnected) apart from a timeout, so the failure
message points at the panic instead of claiming the supervisor hung.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(rocmd): say why the engine guard's termination cannot panic (ROCMAI-454)

The guard terminates the engine from Drop, which can run during an
unwind, where a panic aborts the process. Say at the call that the
termination path under it does not panic, and how long it blocks on
each platform.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit f95135c)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…er the default path in e2e (ROCMAI-480) (#1038)

* docs(install): say the release signing key is pinned and enforced (ROCMAI-480)

The installer comments and release-trust.md still described the release
signing key as unpublished and signature verification as opt-in. The key
is pinned in both installers, and every install verifies the detached
signature against it, so a missing .sig sidecar fails the install.

The docs now list only the trust roots that are still unpublished: the
runtime metadata signing key and the model recipe index signing key. The
same stale claim is fixed in testing.md and aim-catalog.md.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(install): cover default signature verification in the lifecycle lane (ROCMAI-480)

Every existing lifecycle install sets ROCM_CLI_REQUIRE_SIGNATURE=1, so a
regression that only bites when nobody asks for a signature would pass the
whole lane. Two Linux scenarios now run as a user does: lifecycle-36
installs a specific version with the requirement turned off and still
rejects a bundle signed by an untrusted key, and lifecycle-37 runs a
release install with the variable unset and still fails on a missing .sig.

The harness gains a per-scenario signature-requirement setting, takes the
channel as a parameter, and no longer lets an inherited
ROCM_CLI_SIGNING_PUBLIC_KEY_PATH/_PEM stand in for the pinned key.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(install): describe signature verification and trust-root slots exactly (ROCMAI-480)

Review found several statements that were broader or vaguer than the
code. Release and nightly CI sign every bundle archive the installers
download, but the nightly release also carries unsigned convenience
copies; the installers verify every install only while a release key is
pinned, and the release-channel floor changes no outcome today;
verify-pinned-keys also requires unpublished slots to stay empty and
cross-checks a CI-configured signing key. Say each of those as it is,
and drop a line-number reference to install.sh that had already drifted.

The installer changes are comments only.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(install): address review on signature-verification wording (ROCMAI-480)

Reflow the lines a partial edit left ragged, and say plainly what the new
lifecycle scenarios do not cover: the missing-.sig case with nothing asked
for runs on the release channel only, the Windows default path has no
scenario, and lifecycle-36 cannot tell a dropped channel argument from a
working one because the release-channel floor would force the same
rejection. Name the follow-up ticket in the installer comments instead of
calling the fail-closed change "planned".

The installer changes are comments only.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 058fa6e)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…a removed rollback target from the active marker (#1106)

* fix(runtimes): clear a removed rollback target from marker (ROCMAI-184)

Uninstalling the runtime that is the rollback target cleared
config.previous_runtime_key but left data/runtimes/active.json naming it,
so storage retention kept holding a key for a runtime that no longer
exists. Uninstall now clears it from the marker too.

The marker writer moves next to ActiveRuntimeMarker so activation and
uninstall share the one atomic write; the marker keeps naming the runtime
in use, and a rollback target that names another runtime is left alone.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): cover ROCm 7 and ROCm 10 runtimes side by side (ROCMAI-184)

Runtimes installed before the ROCm 10 layout and runtimes installed from
it coexist on one machine, and the CLI has to keep reading the older
record as it was written while it lists, switches between and removes the
newer ones. Nothing exercised that.

Mock-lane scenarios runtime-lifecycle-19/20/21 plant a ROCm 7 runtime
installed before the ROCm 10 layout plus ROCm 10 release and nightly
runtimes for one GPU family. They list, activate each in turn, uninstall
one runtime and then the active one while the others' folders and records
stay untouched, and refuse a runtime id shared by two runtimes with both
keys named. The Windows release lane also runs them, so the Windows paths
are covered on a required lane.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(runtimes): describe every active-runtime marker writer (ROCMAI-184)

The comments called write_active_runtime_marker the marker's one writer,
but a failed activation's restore writes or removes the marker directly,
and uninstalling the runtime in use and `rocm config
set-default-runtime`/`clear-default-runtime` remove it. The registry module
doc also said uninstall only reads and clears the marker, when it now
rewrites the rollback target too. Name each of them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 54df940)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…3) (#1045)

* fix(lemonade): write service state atomically under a lock (ROCMAI-453)

A Lemonade service's state file has several writers: the serve wrapper,
a healthcheck latching that inference was verified, and rocmd publishing
the Anthropic endpoint. Each merge read the file, patched it and wrote
it back in place with no lock, and a file it could not parse was
silently replaced by the patch alone. Two merges racing could drop each
other's keys, and a reader catching a write half-done, or a merge into
a torn file, could lose server_pid, so a later stop targeted the
wrapper instead of the server.

rocm-core now owns the state-file writers. merge_json_state_file and the
new write_json_state_file hold one cross-process lock per state
directory (.json-state.lock) across the whole read-modify-write and
publish through the shared atomic writer, so a lock-free reader always
sees a whole document. A merge into a file that is empty, not valid
JSON or not a JSON object is refused with a typed JsonStateFileRefused
and leaves the file byte for byte as it was; an absent file is still
created. Every Lemonade state write goes through these writers.

The serve paths need no change: their existing guards stop the server
and try to mark the state failed when a merge fails. The router test
that makes the ready merge hit a directory now expects the refusal the
merge reports when it cannot read the file. Hardening what happens when
a running server cannot be recorded or marked failed is left to a
follow-up.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(rocmd): retry the endpoint publish on a torn state file (ROCMAI-453)

rocmd records the Anthropic endpoint by merging into the engine's state
file. The merge now refuses a file it cannot merge into rather than
replace it, and the vLLM and AIM engines still rewrite their state
files in place, so the publish can catch one empty or cut short for a
moment and fail although the file is whole again an instant later.

Retry the publish only for that refusal (empty, or not valid JSON), up
to 5 attempts 200 ms apart. A whole document that is not an object, or
a failure to lock, read or write the file, is returned at once. The
error names the state file and says how many attempts were made.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(lemonade): prove the torn-state refusal and read lock (ROCMAI-453)

The ready-merge failure test used a directory where the state file
belongs, so it exercised a read error rather than the refusal of a torn
file, and pinned the "left it untouched and recorded nothing" message
without checking the file. It now tears the state file and asserts the
typed NotJson refusal together with the bytes left on disk.

Whether a merge reads the existing file under the state directory's
lock was only caught by a timing-dependent concurrency test. The read
now goes through an injectable seam, and a new test observes the lock
held at the moment of reading, so moving the read before the lock fails
deterministically.

Document that AIM's write_state, like vLLM's, takes no lock and writes
in place.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(lemonade): say a torn state file is left unmarked (ROCMAI-453)

The router serve comment, the direct server's supervisor doc, the
serve_http_resolving doc and the router_serve.rs architecture entry
said a failure after the spawn always marks the state `failed`. Since
merges refuse an empty, torn or non-object state file, that mark is
refused too and the file is left as it was, which the router test
already asserts. Say so in each place. No code changes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 2c28f39)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…called usable (#1092)

* fix(vllm): make vLLM's pinned tensorizer import before an install is called usable

vLLM's wheel pins tensorizer==2.10.1, which raises
KeyError: 'crypto_stream_salsa20_NONCEBYTES' at import under Python 3.14. The
ROCm 10.1.0 vLLM build is published for Python 3.14 only, so the pin cannot be
avoided, and `rocm engines install vllm` reported its dependencies satisfied
and the first `rocm serve` died in the engine core.

After installing, and on every later install of a managed runtime, the engine
imports tensorizer.serialization with the runtime's Python under the
environment the server is launched with (including PYTHONPATH; not python -I,
which would drop it) and with colour switched off, via the shared bounded
subprocess executor (five-minute timeouts for the probe and the upgrade). A
failure counts as tensorizer's own only when the last traceback frame is inside
the tensorizer package and the exception is not a library-load or
missing-module error; torch and shared-library failures are left to the device
check. Colour escapes are stripped before the traceback is read. When the import
is broken, tensorizer==2.12.1 is installed with --no-deps and imported again.

When this run installed vLLM (a first install, --reinstall or a repair), a
failed upgrade, a probe that cannot run, or a tensorizer that still cannot be
imported fails the install with the exception. On a runtime this run did not
install vLLM into the same three cases are warnings, so an install that used to
be a no-op (offline, say) stays one. A tensorizer that imports keeps vLLM's pin.

The upgraded release no longer matches vLLM's exact pin, so `uv pip check`
reports it. The repair assessment and the CLI's dependency_check treat vLLM's
own unmet tensorizer pin, with an installed release at or above the fixed one,
as an intended divergence: no reinstall loop, and dependency_check is
expected_divergence with a note naming the release the runtime holds. The note
claims only what is checked; the rule does not look at the Python version. A
reinstall re-checks and upgrades tensorizer again, so it stays the advice beside
a genuine violation, unlike a torch divergence.

Known limits:
- the check runs inside the engine install, before the CLI realigns torch, so a
  torch that cannot load at that moment reads as not tensorizer's and the next
  install checks again; a reinstall's re-upgrade depends on that too;
- a no-op install on an existing managed runtime now imports torch and
  tensorizer under the launch environment, which can write compatibility shims;
- on the `rocm install sdk` auto-install path a refused tensorizer upgrade can
  still print `dependency_check: satisfied` beside the refusal error, because a
  refused upgrade keeps vLLM's own pin and the metadata check is clean; the
  refusal text carries the truth and that output is deliberately not changed
  here;
- the upgrade is fetched from the default package index.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* test(e2e): cover the vLLM tensorizer check through rocm engines install

Fourteen Linux scenarios (ungated, no GPU or network) run the built CLI against
a planted managed runtime whose Python and uv are shell stand-ins sharing state
files, so each message is asserted together with the state it describes:

- the broken pin is upgraded, the warning, expected_divergence and the note's
  release are printed, and the runtime holds 2.12.1 (a coloured traceback, and a
  PYTHONPATH only the server's launch environment provides, included);
- a second install neither upgrades again nor reinstalls vLLM;
- a missing library, or a non-library exception raised below tensorizer, is left
  alone, and the probe is shown to have looked;
- an upgrade that cannot be fetched or does not help warns on a runtime this run
  did not install into, and fails a first install or a reinstall, saying so;
- repairing a genuine violation beside the upgraded tensorizer clears it and
  upgrades tensorizer again, and the reinstall the CLI advises, taken from the
  printed advice line, does the same.

docs/testing.md names how to run them (cargo xtask e2e -- -n 'vllm-tensorizer')
and docs/manual-testing.md adds the hand step for the real Python 3.14 import,
which no scenario covers.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): print an upgrade failure once, flatten the divergence note, link docs

- A failed tensorizer upgrade on an install that just installed vLLM built its
  message with the cause already interpolated and then chained the same error as
  the source, so the alternate Display printed "network unreachable: network
  unreachable". The message now carries the cause once, and the test counts it.
- The divergence note prints the installed release parsed from a `uv pip check`
  line, so it goes through sanitize_log_value like the other values in that
  report, and a test shows a newline in the version is flattened.
- docs/testing.md cites install.rs, repair.rs and runtime_install_finalize.rs as
  relative links, so the link check can see a moved file.
- The stand-in uv comment says why the repair scenario does not require
  --reinstall: the automatic repair does not pass it, and the scenario whose
  violation only a reinstall clears is the one that does.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* test(vllm): pin the tensorizer probe's remaining branches, fail closed on an unreadable pin

- The library-load rule has four independent markers; a table-driven test now
  raises an exception with tensorizer as the last frame and exactly one marker
  each (OSError, ImportError, ModuleNotFoundError, 'shared object file' and
  'DLL load failed' wrapped in another type), so deleting any one turns it red.
- The post-upgrade re-probe test asserts the second probe happened.
- The tensorizer override rule requires the pin to read as a plain release
  strictly below the fix; an unreadable, empty or marker-qualified pin, or one at
  or above the fix, stays a violation instead of passing as the override.
- The repair assessment names a torch and a tensorizer divergence in separate
  notes, each under the 200-character limit the neighbouring tests enforce.
- The e2e stand-in Python loses PYTHONPATH under -E as well as -I (and combined
  short flags such as -sE), so scenario 07 goes red for any of them.
- docs: the unit-test command selects by module and is counted, the scenarios
  are described as gated only on @requires-os:linux, the manual step covers hosts
  where rocm install sdk installs vLLM itself, and vllm.md says plainly that a
  retry after a failed install only warns.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(rocm-core): read a tensorizer version only in its plain shapes

The release parser behind the tensorizer override rule accepted any text after
a post segment and ignored everything after it, so a pin such as
2.10.1.postgarbage or 2.10.1.post1.foo counted as the plain release 2.10.1 and
could pass as the deliberate override: fail-open, the opposite of the direction
the rule is meant to fail in.

A version now reads as a release only when it is dotted ASCII-digit components,
optionally ending in one post<digits> component, optionally followed by a +local
segment. A bare post, text after the post component, a pre-release, a marker,
a dash or space, a non-ASCII digit, a component too large for u64 and the empty
string have no release, so a pin of that shape stays a violation and an
installed version of that shape is not fixed. Leading zeros read as the number
and trailing zero components are kept (a longer release compares greater than its
prefix, which already gives the right answer against the fix).

Tests cover post<digits> with and without +local, the rejected post shapes on the
pin and installed sides, trailing and leading zeros, and the unreadable shapes.
This also corrects the count in the previous commit's message: the library-load
rule has five markers, not four, and its table has five cases.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(rocm-core): require a well-formed local segment; pin the repair rule and the stand-in's flags

- A tensorizer version's +local segment must now be non-empty ASCII letters,
  digits and dots with no empty component and no second +. Real locals such as
  +rocm10.1.0.g98dff2a81.d20261002 and the short +local, +x, +1 still read; +,
  +a+b, +a..b, +a-b and a non-ASCII digit do not, on the pin side (stays a
  violation) or the installed side (not fixed). Because of that, a version with
  control characters is a plain violation, and the CLI test that exercised the
  divergence note's flattening now checks the violation line instead.
- The release parser's separate digits-only guard was redundant with u64 parsing
  (an equivalent mutant), so it is removed and a comment says that parsing is the
  guard: it rejects empty components, signs, non-ASCII digits, other text and
  overflow.
- A repair-level test pins that a marker-qualified, unreadable or post-garbage
  pin forces a reinstall instead of passing as the deliberate override.
- The 200-character assertion now covers only the two headlines; the
  expected-divergence lines carry the raw uv text and are not bounded by it.
- The e2e stand-in Python no longer reads an option's value, such as
  -Wignore::ImportWarning, as -E or -I, so it drops PYTHONPATH only where a real
  interpreter would; -E, -I, -sE, -Es and -sI still turn scenario 07 red.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* test(vllm): share the uv check frame, correct stale comments, pin the discover PYTHONPATH entry

- The tensorizer override rule splits a uv check line on the same constant
  violation_subject uses (now crate-visible) instead of repeating the literal; a
  test builds a line from the constant, and changing the constant turns it red.
- Two comments that still described only the torch case are corrected: the
  ExpectedDivergence doc now says a reinstall would undo a torch divergence but
  not a tensorizer one (the install upgrades tensorizer again), and the
  nothing-to-repair comment in repair_from_violations covers both intended
  divergences.
- The e2e module doc lists every file the stand-ins share and the marker files
  that pick their behaviour; the refusal step says this run installed vLLM,
  which holds for a first install as well as a reinstall.
- Scenario 07 now needs the exact entry the server's launch environment
  prepends to PYTHONPATH, and the run exports a decoy PYTHONPATH that replaces
  any host value, so dropping the prepend, dropping the launch environment or
  running python -I each turn it red, independent of the host.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* test(vllm): describe the stand-ins' shared state files exactly

The module doc said the scenario planter only reads tensorizer-version, but it
also seeds it, as the already upgraded release, for the two scenarios that plant
a violation beside the upgraded tensorizer, and the stand-in uv writes bin/vllm
on every vLLM install, not only when the file was missing.

Signed-off-by: juhovainio <juho.vainio@amd.com>

---------

Signed-off-by: juhovainio <juho.vainio@amd.com>
(cherry picked from commit cedbced)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…p the queue's GPUs in the wiki (#1063)

* ci: retire the self-hosted nightly lanes that never get a runner

The R9700, MI350P and W7900 nightly jobs have sat queued for hours and
been cancelled since at least 11-09. Remove them from nightly.yml and
from the consolidated report that waited on them. These lanes now run
on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu,
Windows, WSL2, all on the Strix pool since #1018) are unchanged.

Update the workflow contract tests and the docs to match.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs,test: take main's wording where #1066 and this change touch the same lines

Preparation for merging main into this branch without conflicts; the next
commits restore this change's own wording on top of the merge.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* feat(xtask): add e2e-queue-import to turn a queue run into nightly report inputs

Reads a finished run of the shared GPU queue (its per-lane results artifact
and the run's job log) and writes each lane's report files where
`cargo xtask e2e-report` reads them, named for the lane and the queue's gate,
plus a status section for the top of the support matrix. Wired in by the
following commits.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* feat(e2e-report): label queue lanes by lane and gate, and keep advisory ones out of the verdict

The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X
lane is not the self-hosted one) and marks one the queue does not block on as
advisory. release-readiness leaves advisory rows out of its failure count and
still counts a blocking lane that came back with no result. xtask gains the
`e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly
platform check goes, since #1066 removed the per-PR W7900 lane it exempted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: keep the queue's GPUs in the support matrix and release readiness

Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them
from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X,
MI350X and MI355X, which only the queue runs, would never appear.
e2e-report-nightly now fetches the newest queue-run-results artifact of a main
run that finished in the last 24 hours (and its job log, for the tested branch
and each lane's gate), imports it, and opens the matrix with a Queue lanes
table. A missing, late or malformed queue run never fails the report: the page
says so instead.

The workflow contract pins the new steps and the native Linux preflight lane
counts for this change plus #1066 (four). The docs describe the queue import,
link the queue page instead of "(below; advisory)", and fix the "Of those"
wording in testing.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: say when the queue lanes are missing from the readiness page, and skip other-branch runs

Review of the queue import found two gaps against "no GPU row goes missing":

- When no queue run was imported, the Release-Readiness table still read
  "0 failure(s) across all platforms" and GO, and the absence was noted only
  on the E2E-Support-Matrix page. release-readiness now reads the matrix's
  absent note and marks the E2E row and adds a warning under the table. The
  verdict itself is unchanged: absence is still not a failure.
- The fetch step took the single newest queue-run-results artifact and then
  dropped the whole import when its log said another branch, so a later
  dispatch of another branch hid a valid run of main. It now walks the five
  newest runs of the last 24 hours and takes the first that tested main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: address review on the queue import and readiness doc

- release_readiness: move queue_absent_reason above the total_e2e_failures
  doc block so each function keeps its own rustdoc.
- nightly: import only a completed queue run (any conclusion), and scan every
  job of the run for the lane matrix instead of assuming the first.
- docs: neutral pool wording, drop the leftover blank line.
- workflow_contract: pin the new fetch-step guards.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: keep unnamed queue rows and directories apart; tidy the importer

- An unnamed matrix row is now unnamed-row-N and an unnamed artifact
  directory unnamed-dir-N, so plan() can no longer merge two unrelated
  lanes into one.
- Compile the three static regexes once (LazyLock), stop shadowing plan(),
  and share one make_dir helper in write_reports.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit eb90e41)
Signed-off-by: juhovainio <juho.vainio@amd.com>
#1064)

* fix(runtime): keep a second amd-smi library out of the runtime's torch

On an MI300X with ROCm SDK 7.14.1 installed beside a system ROCm,
torch.cuda.device_count() reported 0 while HIP itself saw the device, so
`rocm install sdk` with the torch-alignment opt-out failed its device
check ("imported but reports 0 devices") and vLLM would have stopped with
"Failed to infer device type".

On ROCm, torch counts devices through the amdsmi Python package. That
package opens `libamd_smi.so` by its bare name, after a lookup relative
to its own path that only fits the SDK's own tree and misses in a virtual
environment. The SDK ships the library as `libamd_smi.so.26` only, so the
bare name resolves to the host's amd-smi, or to the copy inside the
amdsmi package. Torch has already loaded the SDK's own copy through
librccl, and two amd-smi libraries in one process enumerate no devices.

Link `libamd_smi.so` to the SDK's own library in the runtime's compat
directory and put that directory on the loader path, so the package opens
the library that is already loaded. The probe behind the install device
check, `rocm examine` and the vLLM launch all take it from one helper in
rocm-core. Nothing is written when the SDK already ships the bare name.

Measured on the MI300X: the same runtime reports no_devices before and
usable (1 device) after.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* test(runtime): cover the ComfyUI amd-smi link; fix a doc comment

Pin that the ComfyUI launch environment carries the amd-smi link
directory on LD_LIBRARY_PATH, and that the link points at the SDK's
library. Reword the ensure_therock_compat_shims doc so the amd-smi link
is not described as an mpi_cxx write.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): pick the amd-smi soname by number, not by string

When an in-place SDK upgrade leaves an older libamd_smi.so.<n> beside the
current one, min() over the paths compared them as strings, so `.so.1`
beat `.so.26` and the link pointed at a library torch had not loaded.
Take the highest by number. Also look for an existing bare name in every
library directory before choosing a soname: the loader finds it ahead of
the shim, which sits last on the path.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* test(runtime): cover the examine probe's amd-smi link and pin its order

Add a test that the torch probe in `rocm examine` picks up an existing
amd-smi link directory and never writes one. Assert the link directory
comes after the SDK's own in the ComfyUI and vLLM launch paths, gate the
probe test on Linux like its siblings, reword the ensure_amd_smi_compat
doc to match the any-directory check, and stop exporting
runtime_compat_dir_for_python, which nothing outside its file uses.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(comfyui): write the amd-smi link on Linux only; tighten the probe test

The ComfyUI launch wrote the amd-smi link on every platform, though only
the Linux loader path uses it; gate it like LD_LIBRARY_PATH. Assert the
examine probe test on the specific missing-link error, not on "not the OK
version", which any failure satisfies.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(vllm): drop a leftover libmpi_cxx stub the amd-smi link would expose

The compat directory now reaches the launch's loader path for the amd-smi
link alone. A libmpi_cxx.so.40 stub written by an earlier run, before the
host gained a real libmpi_cxx, would then shadow the real library there.
Remove the stub when the launch does not need it, as is done for the
stale numa link.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): say when the amd-smi link is missing; pin link repair

A launch that cannot make the amd-smi link is silent and leaves torch
counting zero devices. Have the examine torch probe note when the SDK ships
libamd_smi.so.<n> but the environment has no link. Pin that a dangling link
or one pointing at another SDK in the compat dir is repaired by the vLLM
launch, and reflow the ensure_compat_symlink doc comment.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): shorten the first doc paragraph of amd_smi_link_missing

clippy::too_long_first_doc_paragraph rejected it on Linux and on the
Windows clippy job. Split the summary from the detail.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(vllm): remove a leftover mpi stub only when the host has a real libmpi_cxx

ensure_mpi_cxx_compat answers None for a real libmpi_cxx, for no embedded
stub and for a stub that could not be written. Only the first makes an
existing stub harmful; in the others it may be the one torch loads. Gate
the cleanup on a real libmpi_cxx being present and test both cases.

Also name the tests that prove the examine note's claim that a launch or
install writes the amd-smi link, and document why the ComfyUI launch ignores
a failed link write.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): clear a stale mpi stub in the shared amd-smi helper

The amd-smi link shares a directory with the vLLM launch's libmpi_cxx
stub, and the ComfyUI launch, the install device probe and rocm examine
now put that whole directory on the loader path. A stub left by an
earlier vLLM run, on a host that has since gained a real libmpi_cxx,
would load ahead of the real library there, but only the vLLM launch
removed it.

ensure_amd_smi_compat now removes such a stub before it returns the
directory, so every caller that adds the directory gets the cleanup. The
vLLM launch no longer keeps its own copy. The read-only variants behind
rocm examine cannot write, so they hold the directory back while a stale
stub is there, and they accept the link only when it points at the
library a launch would pick, so examine no longer reports as healthy a
link a launch would repoint.

Also: the examine note says "no usable" link and drops the host-ROCm
qualifier, the device-probe test uses the shared compat-dir constant, and
the stub's file name is a public constant so the ComfyUI and vLLM tests
stop hardcoding it. Those two tests derive their expectation from whether
the host has a real libmpi_cxx, so their comments point at the rocm-core
test that injects the host answer and covers both halves on any host.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): import Path on every platform for the stale mpi stub helpers

mpi_cxx_stub_shadows_real_library and remove_stale_mpi_cxx_stub are not
Linux-gated (their callers in managed_runtime are not), but Path was only
imported on Linux, so the Windows build failed with E0425.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(runtime): clear a stale libnuma link from the shared compat dir

The amd-smi link now puts the shared compat directory on the loader path
of the ComfyUI launch, the install device probe and rocm examine, as well
as the vLLM launch. A libnuma.so.1 link an older release wrote there
hides the host's libnuma and breaks the torch import, but only the vLLM
launch removed it.

ensure_amd_smi_compat now removes that link before it returns the
directory, the read-only variant holds the directory back while one is
there, and the vLLM launch uses the shared cleanup instead of its own
copy. Only a symlink counts: a regular file was not written by rocm-cli.
The rocm examine note says a launch or install clears it.

Also pins the directory-order rule of the soname choice, including
skipping a directory with no soname, and the diagnostic's answer for an
interpreter outside a bin or Scripts folder. The doc comments now state
what the code does for those cases and for the shared directory, and
architecture.md names the new cleanup.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit d189674)
Signed-off-by: juhovainio <juho.vainio@amd.com>
)

* ci: retire the self-hosted nightly lanes that never get a runner

The R9700, MI350P and W7900 nightly jobs have sat queued for hours and
been cancelled since at least 11-09. Remove them from nightly.yml and
from the consolidated report that waited on them. These lanes now run
on the queue. The MI300X nightly and the Strix Halo nightlies (Ubuntu,
Windows, WSL2, all on the Strix pool since #1018) are unchanged.

Update the workflow contract tests and the docs to match.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs,test: take main's wording where #1066 and this change touch the same lines

Preparation for merging main into this branch without conflicts; the next
commits restore this change's own wording on top of the merge.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* feat(xtask): add e2e-queue-import to turn a queue run into nightly report inputs

Reads a finished run of the shared GPU queue (its per-lane results artifact
and the run's job log) and writes each lane's report files where
`cargo xtask e2e-report` reads them, named for the lane and the queue's gate,
plus a status section for the top of the support matrix. Wired in by the
following commits.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* feat(e2e-report): label queue lanes by lane and gate, and keep advisory ones out of the verdict

The e2e report names a lane of the shared GPU queue by the lane (a queue MI300X
lane is not the self-hosted one) and marks one the queue does not block on as
advisory. release-readiness leaves advisory rows out of its failure count and
still counts a blocking lane that came back with no result. xtask gains the
`e2e-queue-import` command, and the W7900 exemption in the per-PR/nightly
platform check goes, since #1066 removed the per-PR W7900 lane it exempted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: keep the queue's GPUs in the support matrix and release readiness

Retiring the self-hosted R9700, MI350P and W7900 nightly lanes would drop them
from the E2E-Support-Matrix and Release-Readiness wiki pages, and MI325X,
MI350X and MI355X, which only the queue runs, would never appear.
e2e-report-nightly now fetches the newest queue-run-results artifact of a main
run that finished in the last 24 hours (and its job log, for the tested branch
and each lane's gate), imports it, and opens the matrix with a Queue lanes
table. A missing, late or malformed queue run never fails the report: the page
says so instead.

The workflow contract pins the new steps and the native Linux preflight lane
counts for this change plus #1066 (four). The docs describe the queue import,
link the queue page instead of "(below; advisory)", and fix the "Of those"
wording in testing.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: say when the queue lanes are missing from the readiness page, and skip other-branch runs

Review of the queue import found two gaps against "no GPU row goes missing":

- When no queue run was imported, the Release-Readiness table still read
  "0 failure(s) across all platforms" and GO, and the absence was noted only
  on the E2E-Support-Matrix page. release-readiness now reads the matrix's
  absent note and marks the E2E row and adds a warning under the table. The
  verdict itself is unchanged: absence is still not a failure.
- The fetch step took the single newest queue-run-results artifact and then
  dropped the whole import when its log said another branch, so a later
  dispatch of another branch hid a valid run of main. It now walks the five
  newest runs of the last 24 hours and takes the first that tested main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: address review on the queue import and readiness doc

- release_readiness: move queue_absent_reason above the total_e2e_failures
  doc block so each function keeps its own rustdoc.
- nightly: import only a completed queue run (any conclusion), and scan every
  job of the run for the lane matrix instead of assuming the first.
- docs: neutral pool wording, drop the leftover blank line.
- workflow_contract: pin the new fetch-step guards.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: keep unnamed queue rows and directories apart; tidy the importer

- An unnamed matrix row is now unnamed-row-N and an unnamed artifact
  directory unnamed-dir-N, so plan() can no longer merge two unrelated
  lanes into one.
- Compile the three static regexes once (LazyLock), stop shadowing plan(),
  and share one make_dir helper in write_reports.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: a missing queue run makes the release-readiness verdict NO-GO

A queue that never ran is not a pass. When the nightly report carries the
'no queue lanes' note, the verdict gets the failing gate
'queue lanes absent: <reason>' and reads NO-GO, instead of GO plus a warning.
Tests and the docs line follow.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 9101286)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…ls (ROCMAI-703) (#1050)

* fix(serve): say so in the summary when the post-launch smoke test fails (ROCMAI-703)

After `rocm serve`, the deployment summary read `status: ready` with n/a
metrics even when the first test request had failed: run_smoke_test
discarded the error and returned empty metrics, so a server that could not
answer looked the same as one that was not measured.

run_smoke_test now returns the failure reason. When the test fails, the
summary is headed "(smoke test failed)", the status reads "started, smoke
test failed", and a note gives a one-line reason and points at `rocm logs`.
The launch stays best-effort: the server keeps running and the exit code is
unchanged.

Fixes #523

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* fix(serve): assert the smoke-failure summary end to end and without echoing the api key (ROCMAI-703)

The first commit's tests echoed the rendered summary, which carries the
endpoint api key, into assert messages, and CodeQL rust/cleartext-logging
flagged them. The asserts now carry constant messages.

Also in this change:
- the failure reason is cleared of terminal control and format characters,
  and text after a cut-off escape sequence is kept;
- a clean stream with no content token (a reasoning model) is no longer
  treated as a failure;
- serve-55, a second `ready` mode of the existing e2e fault-injection hook
  (compiled out of release builds), asserts the heading, status row, engine
  reason, one chat request and the logs command over a pty with exit 0;
- the README and test docs state what is and is not detected.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs(serve): say what the smoke-failure note does and does not detect (ROCMAI-703)

The testing guide claimed a close-delimited stream cut after a token shows
`n/a` metrics; it shows the metrics measured up to the cut. The README and
the manual test steps now point at the guide's list of failure classes that
are not detected, so they no longer imply the note catches everything.

The failure reason no longer lets an engine add a backtick-quoted command
next to the note's own logs hint. A test pins the note's "server may not be
able to serve requests" warning. Comments no longer list scenario ids that
drift. The guide records that two ready services on one model can make the
verdict describe the other service.

The earlier commit's message refers to the e2e scenario as serve-55; it is
serve-56 after merging main.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* docs(serve): correct what the serve-57 steps and the smoke-failure notes claim (ROCMAI-703)

A review found claims the code cannot back. The daemon-left-behind step
cannot discriminate the autostart guard because the isolated config
enables no watcher. The testing guide overstated `n/a` metrics for
undetected classes and described a two-ready-services case that cannot
happen, while omitting the real false pass: the smoke request is routed
by model name, not to the launched service. A length-cap boundary had no
test.

The text now says what the code does, the cap is tested at 200/201
characters, a load-bearing line in the ready hook is commented, and prose
cites @id tags instead of scenario numbers that had already drifted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit cab5db1)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…Ks (#1102)

* ci: check that vLLM can install on the default release and nightly SDKs

The nightly queue lanes lost their whole nightly channel on 2026-11-12 and
every day since: #1033 made an unpinned nightly install resolve to
nightly.repo.amd.com, whose SDK is 10.3.0aYYYYMMDD, and the vLLM engine has
no discovery row or static pin for 10.3, so the pre-warm's
`rocm engines install vllm` refused it. No pull-request check installs vLLM
on a 10.x nightly SDK (the vLLM-default Instinct lanes only run in the
scheduled queue nightly), so #1033 merged green.

Add a CPU-only workflow that runs `rocm install sdk --dry-run` for the
release and nightly channel on gfx942 and gfx950 and fails when the selected
SDK has no route in the engine's two build tables, matched the way the
engine matches them. It runs only on pull requests that touch the resolver
(source.rs), the two tables or the check itself.

It is advisory and not a required check: it reads the live indexes, so it
can go red with no change in the pull request.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: key the vLLM route check's rust cache by the runner image

The workflow-contract test hosted_linux_compile_caches_are_keyed_by_runner_image
requires every hosted Linux job's setup-rust-toolchain to carry a cache-key
naming the runner image. The new job had none, which failed the required
Coverage check.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: widen the vLLM route check's path filter and say what it is stricter about

Review findings on the check:

- The version is chosen in apps/rocm/src/therock/resolve*.rs and version.rs,
  not source.rs alone, so a change only there would not have triggered the
  check. Add them to the path filter.
- The script says a version with no row has "no route" and that the engine
  refuses it. The engine refuses only a line whose major has a discovery row;
  any other version falls back to the default static pin, a wheel built for
  another ROCm. Report the two outcomes separately and say in the docstring
  that the script is deliberately stricter than the engine there.
- Use the setup-rust-toolchain pin the rest of the repo uses (v2.0.1).
- Move the docs section above the canary's, so it no longer sits over the
  stray release-candidate paragraph, and list its self-test in testing.md.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: report a missing vLLM route as a warning, widen the path filter

The route check was red on every PR touching its paths while the default
nightly has no vLLM route, which trains reviewers to ignore it. A missing
route now shows as a warning and a job summary and the job stays green; the
build, the self-test and a check that cannot run (exit 2) still fail it.

Also run on any change under apps/rocm/src/therock.rs and therock/, which
all feed which SDK the default resolves to.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: fail the vLLM route check when the check itself breaks

probe() returned a string for a dry-run timeout, a failing dry run and a
plan with no usable version, so main() exited 1 for them just like a real
missing route, and the workflow downgraded a regression in `rocm install
sdk --dry-run` to a warning. Those cases, an unrunnable rocm and an
unreadable build table now raise CheckError (exit 2), which fails the job;
exit 1 stays reserved for a missing route. Findings print as warnings,
the self-test covers each branch, and the docs and the arch list comment
say what the check does.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 088faa4)
Signed-off-by: juhovainio <juho.vainio@amd.com>
* docs: add a manifesto to the README

State the goals the project builds toward, so users and contributors
can see what the CLI is aiming for: bare metal to a served model in a
few commands, the whole AMD GPU line with an honest answer when a GPU
has no ROCm build, correctness tested on real hardware, no silent CPU
fallback, asking before removing what a user has, actionable diagnosis,
output for people, scripts and agents, and open development.

The intro says the Tech Preview does not meet every goal yet. Sentences
about current behavior are limited to what the CLI does today.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs: add an extensibility goal and point agents at the manifesto

Add a ninth goal: ROCm CLI should adopt new engines, ROCm-based
libraries and agentic-AI tooling as validated extensions, and take a
user from a local machine to remote GPU compute without switching
tools, with `rocm remote` as the first step.

AGENTS.md now has agents read the manifesto before planning a change
and raise it with the user when a request's approach would work against
one of the goals, so the goals reach work done by agents and not only
readers who open the README. When nobody can be asked, the agent
implements nothing that conflicts and names the conflict in its report.
The goals only govern choices AGENTS.md leaves open: its other rules,
such as the ban on silent CPU fallback, stay binding whatever the user
answers.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 2583a97)
Signed-off-by: juhovainio <juho.vainio@amd.com>
… it does not (#1091)

* fix(wsl): make a managed runtime load its own librocdxg, and say when it does not

A ROCm wheel ships librocdxg.so.1 beside libhsa-runtime64, but the HSA
runtime opens the unversioned librocdxg.so, which the wheel does not
ship. On WSL2 the lookup therefore skips the runtime's own copy and
loads the host's (/opt/rocm/lib from rocdxg-roct). The 10.2 and 10.3
nightly runtimes need hsaKmtSetPersistingCacheSize, which rocdxg-roct
1.2.2 does not export, and crash on first GPU use while `rocm examine`
still reports wsl_rocdxg_ready.

The lookup mechanism was observed with LD_DEBUG=libs on the 10.2
nightly (verified). Which component requests the symbol is not
identified.

- Wheel runtime installs and updates (install_wheel_runtime: `rocm
  install sdk`, `rocm update --apply`) on WSL link librocdxg.so to the
  runtime's own librocdxg.so.1 (a relative link) in the runtime's
  library directory, which libhsa-runtime64 searches through $ORIGIN
  (a DT_RPATH per readelf), so a bare <runtime>/bin/python is fixed
  too. The step reads the WSL gate itself. The report line says what
  was done; a failed link leaves the install at exit 0 with the manual
  command in the report. Tarballs ship both names and adopted or
  imported runtimes are never linked. The link is not tracked by the
  installer and is orphaned if the runtime is removed by package.
- `rocm examine` lists, on WSL only, each managed runtime copy its
  runtime does not load (wsl_runtime_librocdxg, and
  summary.unlinked_runtime_librocdxg in --json) with the ln -s command
  that fixes a runtime installed earlier (for an adopted runtime that
  is a write into a directory the CLI did not create). It reports what
  the files show, not whether the host is affected. A dangling
  librocdxg.so is treated as present. driver_status is unchanged: it
  describes the host.

Tests: the install entry point is driven on a simulated WSL host and a
simulated native one (needs the e2e-test-hooks feature); the install's
report helper is driven with the gate true and false on every Unix host and
shown to link only the probe's library directories; the advised command
and the install are shown to make the same relative link. Deleting the
call in install_wheel_runtime is caught by the dead-code lint under
`cargo clippy --workspace --all-targets -- -D warnings` (run: it
reports the wrapper and its two helpers as never used), not by a test;
`cargo test -p rocm --bin rocm therock::` stays green with a warning.
A wrong gate or directory list passed through the wrapper is covered by
the tests of the shared helpers; a different list passed to the core
entry from inside the wrapper is not caught by any test. The install
side has no scenario: `rocm install sdk` on WSL needs a real wheel
download and a WSL host, and no default lane has one; the post-probe
step is covered by unit tests and the call site by the dead-code lint.
A failed link is reported with its error and the command to run once
its cause is fixed, since the same link call failed.

Tradeoffs: the link is made for every managed wheel runtime, including
the 10.1 release (its bundled copy then replaces the host copy; checked
to work); WSL1 counts as WSL; the examine line is one long
"; "-joined line to keep its key: value shape.

Notes: the loader claims (the runtime searches its own directory first;
tarballs ship both names) were observed by hand, with readelf -d
(libhsa-runtime64.so.1 carries RPATH $ORIGIN and no RUNPATH in the
10.1, 10.2 and 10.3 wheels; librocdxg.so.1 has no unversioned sibling)
and LD_DEBUG=libs on the 10.2 nightly, not by an automated test.
Runtimes installed before this change stay unlinked and examine only
advises. Engine venvs (vLLM, ComfyUI) that pull rocm-sdk-core as a
torch dependency are neither linked nor examined (not verified). The
link is made before write_runtime_manifest, so a manifest failure
leaves a link in an unregistered root (harmless on retry). The
ExamineSummary gate (wsl.is_some()) is covered by scenario examine-52,
not by a unit test. The update-check disabling is in
push_planted_wsl_machine_env, for determinism.

Fixes #1082

Signed-off-by: juhovainio <juho.vainio@amd.com>

* docs(readme): link the WSL guide by URL so the Sphinx include resolves

README.md is included into the Sphinx site, where a relative link to a file
outside the docs source fails the -W build (myst.xref_missing). The README
links other repository docs by absolute URL for the same reason.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* docs(testing): link runtime_rocdxg.rs and drop a repeated sentence

Cite the new module as a relative link, as docs/testing.md does for other
source files, so the link check can catch a moved target, and remove the
closing sentence of the install-side paragraph that restated the reason
and the lint protection given a few lines earlier.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(wsl): treat a dangling librocdxg.so as missing, and replace it

The previous change counted a dangling `librocdxg.so` as present and
left it alone, so `ln -s` would not hit "File exists". That was wrong
about the loader. LD_DEBUG=libs on the 10.2 nightly with a dangling
librocdxg.so (pointing at a missing file) first on LD_LIBRARY_PATH
showed the loader trying the link twice and continuing down the search
path: open() on a broken symlink gives ENOENT, which it treats as a
missing file, so the host's librocdxg was loaded and the process still
exited 139. A runtime whose link goes dangling (a wheel changing the
file, or the file being removed) therefore crashed again with no
warning from `rocm examine` and no repair from the install.

The name is now judged three ways: absent, present and resolving (a
regular file or a working link: never replaced), and dangling.

- `rocm examine` lists a dangling librocdxg.so like an absent one, and
  its advice is `ln -sfn`, which clears a dangling link (a plain
  `ln -s` fails with "File exists"). The line says "no librocdxg.so (or
  a dangling one)".
- The install replaces a dangling link atomically: a temporary link in
  the same directory is renamed over it, so the name is never absent in
  between, and the temporary is removed if the rename fails. The report
  line says "replaced a dangling <path>". A file or a working link is
  left alone.
- Limits, stated in the module doc and docs/wsl.md: the versioned name
  is the fixed `librocdxg.so.1` (a wheel whose soname changes is not
  detected or linked; no globbing), and a dangling librocdxg.so beside
  a runtime that ships no librocdxg.so.1 is not detected, since there
  is nothing to link it to.

Also: claims about the gate now say WSL (it also fires on WSL1);
managed_runtime_library_dirs documents that on Linux it includes the
host's /usr/lib/wsl/lib; the startup update-check disabling is scoped to
the Given steps that plant a managed runtime instead of every step using
the planted-WSL helper (the harness consumes the scenario env on each
run, so the advice step re-plants it); docs/testing.md credits the
"only the probe's library paths" assertion to its real test in
therock.rs.

Tests: unit tests for detection, atomic replacement, no temporary left
(also on a failed rename), a resolving link and a file untouched, and
the advised command clearing a dangling link through sh; scenario
examine-53 plants a dangling link, lists it, runs the advised command,
and checks librocdxg.so reads as the runtime's own bytes.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* docs(wsl): say ln -sfn everywhere, and test the temporary-link counter

- The advised command is `ln -sfn librocdxg.so.1 '<dir>/librocdxg.so'`;
  docs/testing.md, docs/manual-testing.md and the docs/architecture.md
  row still said `ln -s` (which fails with "File exists" on a dangling
  link). The architecture row now also describes the dangling case. The
  command was run against GNU ln 9.7 as well as uutils ln: plain
  `ln -s` over a dangling link fails with "File exists", `ln -sfn`
  replaces it and the name resolves.
- managed_runtime_library_dirs documents what it returns: every library
  entry the environment builder produces for each managed runtime
  (including each root's bin, as the builder collects them), plus the
  host's /usr/lib/wsl/lib on Linux.
- A two-thread test replaces dangling links in one directory at once, so
  the counter in the temporary link's name is exercised: without it the
  threads collide on one temporary name and the test fails (checked by
  replacing the counter step with 0).

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(wsl): pin the versioned-copy rule, narrow the replace window

- The versioned `librocdxg.so.1` decides whether a directory is listed
  or linked: a regular file or a resolving link is, and an absent,
  dangling or looping one is not (nothing valid to link to), in
  `rocm examine` and in the installer alike. No test planted a broken
  versioned copy; the whole matrix is now tested, through the installer
  (no link and no temporary left) and through examine's registry scan.
  The exclusion is deliberate and a broken versioned copy is not
  reported; the module doc and docs/wsl.md say so.
- replace_with_link re-checks that the name still dangles just before
  the rename and leaves it alone otherwise. The window cannot be closed
  (rename has no "only if still dangling" form); the docs now say a name
  that becomes a working link between the re-check and the rename is
  replaced, instead of claiming "never replaced" unconditionally.
- A failed removal of the temporary link is part of the returned error.
  A leftover temporary from a crashed process can only collide after pid
  reuse, and then symlink fails with "File exists", which is reported.
- A test pins that examine's directory set contains every directory the
  install links (the two are computed separately and are not unified
  here).
- A summary written before `unlinked_runtime_librocdxg` existed still
  deserialises (test).
- Wording: claims about the gate and the link say WSL; "WSL2" stays only
  where the text describes what was observed. NameState::Dangling's doc
  covers link loops. A test comment names what actually produces the
  synthesized failure. The install test that asserts end state is
  renamed to what it checks. The e2e doc blocks are back on their own
  functions.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(wsl): link a name that vanished mid-replace, and test cleanup for real

- In the dangling-link replacement, a name that is gone at the re-check
  (it vanished after the first check) is now linked, instead of being
  reported as "File exists" for a name that does not exist. Resolving at
  the re-check is still refused; the docs describe each case.
- The cleanup test now reaches the rename-failure branch: the rename is
  injected to fail while the name still dangles, removal is the real
  one, and the test checks no temporary link is left on disk. The old
  test passed a path whose re-check read as absent, so it never reached
  the rename. A new test covers the vanished-name case.
- The window description no longer implies a measurement.
- README: re-wrapped the WSL paragraph.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(wsl): report a vanished name as linked, not replaced

When the name vanished between the first check and the final check of a
dangling-link replacement, the temporary link is renamed into place but
nothing was replaced; the install line still said "replaced a dangling".
replace_with_link now reports whether the name really dangled at the final
check, so the three cases each print their own line: absent from the
start and vanished at the final check say "linked", a dangling link that
was replaced says "replaced a dangling ...".

docs/wsl.md, docs/architecture.md, docs/testing.md and the module doc no
longer say a name that resolves is "never" replaced: it is left alone when
checked, the install refuses at its final check if it resolves by then,
and a name that becomes a working link between that check and the rename
is replaced.

Signed-off-by: juhovainio <juho.vainio@amd.com>

---------

Signed-off-by: juhovainio <juho.vainio@amd.com>
(cherry picked from commit c21942f)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…89) (#1090)

* feat(runtimes): show runtime provenance in list and examine (ROCMAI-189)

The install report names a runtime's channel, package source, source
layout generation and exact GPU target, but afterwards only its manifest
knows them, so nothing told a user which stream or GPU an installed
runtime was for.

`rocm runtimes list` now prints a `provenance:` line under each runtime
and `rocm examine` prints matching `active_runtime_*` lines for the
active one, both from RuntimeProvenance::fields(). The layout and target
choose the same recorded field `rocm update` uses; the source is the
package index, else the artifact URL, never the adoption marker an
adopted runtime keeps there. Values are trimmed, and a fact recorded
nowhere or recorded blank prints `unrecorded`; nothing is guessed from
other facts.

Covered by unit tests and the runtime-lifecycle-19 and examine-49
scenarios on the mock lane; README, --help and the testing docs
describe the new output.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(runtimes): cover adopted runtimes' unrecorded source in a scenario (ROCMAI-189)

An adopted runtime installed nothing, so its manifest holds an adoption
marker where an artifact URL would be, and the listing must report its
source as unrecorded rather than show the marker. That was pinned only
by unit tests that refer to the marker through its constant, so a change
to the persisted value would have gone unnoticed.

Scenario runtime-lifecycle-19 now plants an adopted runtime whose
manifest carries the on-disk `adopted-read-only` marker and asserts it
lists `source=unrecorded`. A unit test pins the persisted marker value
against the constant.

The examine provenance test gains an adopted active-runtime case beside
the nightly one. The two adopt listing tests are merged into one
table-driven test over the default and `--channel nightly` requests,
sharing `provenance_line` with the runtimes_cmd tests.

docs/testing.md describes the new coverage.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(runtimes): correct provenance claims, pin --runtime-id (ROCMAI-189)

Several comments and docs described the provenance listing in ways the
previous commits made untrue or left incomplete.

The e2e prewarm comments said `rocm runtimes list` carries no
`channel=` field. It now prints one on each runtime's `provenance:`
line; the `{channel}-` key prefix guard is still needed because
`RuntimeEntry::parse` reads only the row line, and the comments now
say so.

The provenance test's doc comment said each fact is read the way
`rocm update` reads it. Only the layout and target are shared with
`rocm update`, which does not read the recorded source.

The README and docs/testing.md said an adopted runtime's channel is
`nightly` after `--channel nightly` and `release` otherwise, missing
that `--runtime-id therock-nightly:<family>` also names the channel.
They now cover `--runtime-id`, and the adopt listing test gains that
path beside the default and `--channel nightly` requests, so all three
ways of naming the channel are pinned.

The docs/architecture.md entries for the examine and runtime-lifecycle
step files now describe the provenance steps they gained.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(runtimes): pin install_root before provenance in the listing (ROCMAI-189)

The `cargo xtask e2e` prewarm reads the line directly after each
runtime row of `rocm runtimes list` as that runtime's `install_root:`.
The listing now also prints a `provenance:` line per runtime, and
nothing pinned that it comes after `install_root:` rather than before:
the existing provenance test searches ahead for the line and would not
notice a reorder.

A runtimes list test now pins the order row, `install_root:`,
`provenance:` by index. The MIXED prewarm fixture gains `provenance:`
lines in that position, so the prewarm's assessment is read against the
current row order, and a new test checks that `RuntimeEntry::parse`
ignores a provenance line, whose `name=value` fields resemble a row's.

The fixture comments now state which fields their captures predate,
and the `UNRECORDED_PROVENANCE` doc names `RuntimeProvenance::fields`
as what prints it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 9e50eac)
Signed-off-by: juhovainio <juho.vainio@amd.com>
A queued e2e-gpu job for a superseded PR head, or for a closed or merged
PR, still takes an MI300X runner and holds the concurrency group for
about 45 minutes, and `gh run cancel` from outside does not dequeue it.

On pull_request events e2e-gpu's first step, once it has a runner, asks
the API for the PR's state and head sha. If the PR is closed or its head
differs from the event's, the step records `stale` and every later step
is skipped without failing the job. A job-level `if` cannot do this
because it is evaluated before the job is queued. e2e-gpu keeps read
scopes only (`contents: read`, `pull-requests: read`). If the PR cannot
be read, or gh is missing, the job runs as before.

The cancel is a separate GitHub-hosted job, cancel-stale-run, with no
checkout and `actions: write`. e2e-gpu builds and runs PR code on a
persistent self-hosted runner, so a write scope there would sit next to
untrusted code, which this repo's workflows otherwise avoid. Only a same-repo PR is sent a cancel;
a fork PR, whose self-hosted lanes only run after approval, just has
e2e-gpu skip its steps. A failed cancel is tolerated.

e2e-report is gated on always(), so it still runs after the cancel, but its
missing-report check is gated on !cancelled() and is skipped in a cancelled
run: the stale exemption is for runs that are not cancelled (fork PRs, or a
cancel that fails or loses a race).

e2e-gpu publishes a `stale` output so the consolidated report's
missing-report check (here and in the identical copy in nightly.yml) does
not count a stale-skipped lane as a missing report; it says so in the
summary instead. The ci.yml and docs claims about fork runs are brought
in line.

Tests: the guard script runs under a stub `gh` for each outcome (including
a missing gh and unreadable PR reads), every later step's `if:` is compared
exactly, and the cancel job's permissions, runner, condition and single
step are pinned.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 67c691b)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…backend (#855)

* fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm backend

Lemonade 11.5.1 pins TheRock 7.13.0 and llama.cpp b9752, and downloads
TheRock from a location that does not serve ROCm 10.x, so on a ROCm 10.1
SDK its llamacpp:rocm backend could not be paired with the SDK in use.
Lemonade 2026.44.1 pins TheRock 10.1.0 and llama.cpp b11580, whose release
ships ROCm 10.1 (and 10.0 and 7.14) archives for Linux and Windows, and
installs TheRock from AMD's whl-next wheel index or core tarball location.
The embeddable layout, binary names and the lemond/lemonade flags rocm-cli
uses are unchanged; the new archive digests are recorded.

Not 2026.45.3, the newest stable release: its Linux lemond needs glibc
2.43 and OpenSSL 3.2.0 and exits with a loader error on Ubuntu 24.04,
which has glibc 2.39 and OpenSSL 3.0.13. 2026.44.1's needs glibc 2.38
and OpenSSL 3.0.0, the same as every release back to 11.5.1.

Two adaptations the new release line needs:

- `lemonade backends` lists recipes rocm-cli did not know (llamacpp-hrx,
  ds4, thenoise) and adds Tier and Format columns. The parser matched
  recipe rows against a fixed list, so an unknown recipe's rows, and every
  indented row after them, were read as llamacpp backends. It now reads
  the recipe from the row layout, and reads Lemonade's update_required and
  update_available states as installable and installed instead of
  dropping the row.
- The second alignment attempt took lemonade-sdk/llama.cpp's newest
  release, but a release can ship without some or all of its ROCm
  archives: b12013 through b12086 carried only qualification reports, and
  others skip one ROCm version or one platform. It now lists the 30 newest
  releases and picks the most recently published one that ships the exact
  archive for the active ROCm version on this platform.

The runtime-16 and runtime-17 scenarios assumed the active SDK never equals
the packaged pin, which is no longer true on a ROCm 10.1 lane: runtime-16
now checks that the installed pin equals the active SDK by whichever route
the host takes, and runtime-17 compares the pin with the one inside the
verified archive.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): compare the Lemonade archive extension ignoring case

The runtime-17 step that finds the cached Lemonade archive compared the
`.zip` extension with `ends_with`, which clippy's pedantic
`case_sensitive_file_extension_comparisons` rejects. The e2e target is
excluded from `--all-targets`, so only the Windows job's
`cargo clippy -p e2e-cucumber --test e2e` compiles it and caught it.
Derive the archive's stem once, comparing either extension ignoring ASCII
case, and reuse it for the member path.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): refuse a host too old for the pinned lemond up front

A Linux lemond built against a newer glibc or OpenSSL than the host has
exits at once with a dynamic-loader error ("version `GLIBC_2.43' not
found"). The user saw only "Lemonade server did not become ready", after
the archive had been downloaded and extracted; this is how 2026.45.3
failed on the Ubuntu 24.04 GPU lanes.

Record what the pinned lemond needs (glibc 2.38 and OpenSSL 3.0.0 for
2026.44.1, read with objdump -T) beside the archive digests, keyed by the
Linux digest so a bump that does not re-read them fails a unit test.
Read the host's versions as the newest GLIBC_/OPENSSL_ version node its
libc.so.6 and libssl.so.3 define, the same thing the loader compares,
locating them through the ldconfig cache. On Linux,
`rocm engines install lemonade` now refuses such a host before
downloading anything, and starting lemond is refused too. Both name the
version needed and the version the host has for each short library. A
host version that cannot be read is treated as unknown and never refuses.

lemonade-installs-07 drives the install refusal through an e2e-test-hooks
override of the host versions, and checks the message and that nothing
was downloaded.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): refuse a host whose loader cache has no libssl.so.3

The OpenSSL half of the host check could not fire. The probe read only
libssl.so.3, and every libssl.so.3 defines OPENSSL_3.0.0, which is all
the pinned lemond needs. A host with only OpenSSL 1.1 has no libssl.so.3
at all, so the probe reported OpenSSL as unknown, never refused, and the
user still got "Lemonade server did not become ready".

Each library is now a version, missing, or unknown. It is missing when
the ldconfig cache was read and lists libc.so.6 but not that library,
and the refusal then says the host's loader cache has no libssl.so.3.
A cache that could not be read, one that lists not even libc.so.6, and
a library that does not parse still read as unknown and never refuse.
The cache reading is now a pure function, unit-tested over planted
`ldconfig -p` output, and the probe runs once per process.

lemonade-installs-07 now reports a host the probe can actually produce,
Ubuntu 20.04-like: glibc 2.31 and no libssl.so.3. The README, the docs
and docs/manual-testing.md section 3 describe the same two outcomes, and
the README no longer says every build needs glibc 2.38, since 2026.45.3
needs more. The start-time refusal has no scenario, because no lane can
reach it: `rocm serve` runs the install check first, a Linux serve with
a packaged llama-server never starts lemond, and the mock lane has no
GPU for serve to resolve.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): stop calling Lemonade 11.5.1 the pin in two Windows rows

The runtime-16 (ROCMAI-899) and serve-06 (ROCMAI-900) Windows rows were
written while Lemonade 11.5.1 was pinned, and their reasons still said
so. Say they were observed with 11.5.1 and have not been re-checked with
2026.44.1, which fetches TheRock elsewhere and whose llama.cpp build
ships a ROCm 7.14 archive. Both scenarios are @nightly, so pull-request
lanes have not run them yet; a stale XPASS on the first Windows nightly
after this pin lands is the signal to delete the row.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): check the llama.cpp ROCm tag where the SDK is the packaged pin

On a ROCm 10.1 lane the active SDK equals the pin Lemonade 2026.44.1 ships,
so comparing the ROCm version cannot show a rewritten pin. Also compare the
paired llamacpp.rocm-stable tag, and name the e2e-test-hooks feature in the
manual-testing note.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* style(lemonade): cargo fmt the spawn helpers

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): stop calling Lemonade 11.5.1 the pin in a row-scope test comment

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): add an opt-out for the host-library check, and say what a refusal left undone

The check reads only the ldconfig cache, so a host that provides OpenSSL 3
or glibc some other way (LD_LIBRARY_PATH, a Nix or Guix profile) was refused
with no way past it. ROCM_CLI_DISABLE_LEMONADE_HOST_LIBRARY_CHECK turns both
refusals into a warning, and the refusal names it (lemonade-installs-08).

A refused re-run where the pinned Lemonade is already installed said
"Nothing was downloaded." about a download it would never have made; it
now says the installed copy stays as it is (lemonade-installs-09).

Also: the backends parser no longer ends a recipe at a first-column line
with no state; runtime-16 requires the install to report an alignment
whenever the pin moved; and the ELF reader test skips on a host without
ldconfig.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): compile the host-refusal outcome helpers on Linux only outside tests

Their one caller, the host-library refusal, is Linux only, so a Windows
build rejected them as dead code.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs: stop saying every published Lemonade build needs glibc 2.38

2026.45.3 needs glibc 2.43, so the floor belongs to the pinned build.
Reword it in docs/wsl.md, the fix-wsl-5 rationale `rocm fix` prints,
the matching note `rocm diagnose` prints, and WSL_MIN_UBUNTU's doc
comment. Also say that the host-library check does not read GLIBCXX,
and that `object` is one new crate for the standalone Lemonade adapter
binary.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): read the backend state at its column, and name the version node a host library provides

A first-column warning with a state word in its text ("... is installed
but outdated") started a bogus recipe and dropped the llama.cpp rows after
it. The state is now read at the Status column the header names; with no
header, only a recipe-shaped first word starts a recipe.

The host-library check reports the newest version node a library defines,
not its release: Ubuntu 24.04's OpenSSL 3.0.13 defines up to OPENSSL_3.0.3.
The message now says "this host's libc.so.6 provides up to GLIBC_2.31",
and test rows use values the probe can produce.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs: list the new parser tests, drop unverifiable release-cadence figures, say who deletes the xfail rows

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 6803163)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…#1062)

* fix(rocm-core): report a timed-out rocm fix step as such (ROCMAI-345)

The fix runners read the probe adapter's collapsed exit code, so a helper
stopped at the one-minute bound was reported as "usermod exited 124;
group membership NOT changed." That names no timeout, offers no way to
recover, and asserts a state nobody checked: the executor stops only the
direct child, so usermod under sudo can still finish.

Run each step through the bounded executor and report from its typed
outcome. A command that exits non-zero or cannot start still says the
change was NOT made; one that times out, is signalled or cannot be waited
for now says the change may or may not have been made, names a command to
check it, and the rocm fix command to re-run. The line and exit code 4
come from one value, and the re-run ids are the catalog's own constants.

On Windows, a PowerShell read of the current User/Machine value that did
not complete used to be taken as "unset": fix-2 then reported the override
as gone and exited 0, and fix-9 skipped its already-set guard and ran setx.
Both now stop with exit 4 before planning or prompting, as fix-6 already
did.

Covered by unit tests over an injected command runner for every outcome of
each runner, and by e2e scenarios diagnose-15 (extended), 39 (tightened)
and the new 62 (Linux helper never finishes), 63 and 64 (Windows reads
fail). Docs, --help and the rocm-doctor skill describe the new exit-4
cases.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(rocm-core): keep the e2e suite polled and state fix read failures precisely (ROCMAI-345)

diagnose-62 waits out fix-4's real one-minute timeout. Its step ran the
CLI synchronously inside the single future the cucumber suite is polled
on, so every concurrent scenario stalled for that minute and the TUI
scenarios beside it overran their wall-clock caps, failing the hosted
E2E lanes. The step now waits for the CLI on spawn_blocking.

On Windows, the timeout line of a setx or PowerShell write said "Check
with" before a PowerShell expression that cmd cannot run. Step now
carries where its check runs, and those lines say "Check in PowerShell
with".

The Windows runners for fix-2, fix-6 and fix-9 write their own lines
through injected stdout sinks, so the unit tests assert what is printed
together with the exit code and what was run: a read that exits non-zero
stops each fix with exit 4 and nothing written, a failed read prints no
plan, and a dry run with a readable scope previews without running setx.

diagnose-64 now previews fix-9 with --dry-run instead of applying it. The
empty PATH hides PowerShell but not setx.exe, so a regressed read guard
could otherwise set the runner's real User-scope HIP_VISIBLE_DEVICES.

README, --help, the exit-code docs and the rocm-doctor skill now say
exactly which fixes this covers: on Windows, fix-2, fix-6 and fix-9 with
--device-index read the current value first, and a failed read exits 4
with no plan, under --dry-run too; the catalog entry printed before it is
not a preview.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(architecture): say the Windows fix runners read before planning (ROCMAI-345)

The runners.rs entry promised only print, dry-run and consent. On Windows
fix-2, fix-6 and fix-9 now read the current env-scope value first and stop
with exit 4, before the runner's own preview or any prompt, when that read
fails, dry-run included; the entry should say so. "Consent first" becomes
"consent before any write", and the report-only arms (fix-2 on Linux,
fix-9 without a device index) are named as the exception, as the module
header already does.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(rocm-doctor): read an older CLI's "exited 124" as a timeout (ROCMAI-345)

The skill is federated from main ahead of a release, and its Phase 0
installs the release-channel CLI. A released CLI reports a timed-out fix
step as "<program> exited 124; <what> NOT changed.", so an agent
following the skill would tell the user nothing changed when the step
may have landed. Treat that line as a timeout: say the change may or may
not have landed, give the user the check for that fix (the group list on
Linux, the User-scope variable on Windows), and offer the same re-run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(rocm-core): name the proving tests and cover older CLIs in the skill (ROCMAI-345)

The wording tests in fix/consent.rs pin each failure line against a
stand-in step; their comments now name the runner tests that prove what
each line claims, and say that no runner test drives a wait failure or
a signal.

The rocm-doctor skill special-cased only `exited 124`. A CLI released
before this change also reports a signalled step as `exited -1` with
`NOT changed`, and on Windows takes a failed read for an unset variable:
fix-2 can then report the override gone and exit 0, and fix-9 can run
setx over a value the user chose. One passage now covers both, with the
check to run, and the exit-4 row in reference.md points at it.

Also refresh three stale comments: what fix.rs says consent.rs holds,
fix-2's class in the report_persistent_override doc, and a PR-relative
note on the current_os WSL test.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* chore(catalog): publish the merged `failed` exit-code description (ROCMAI-345)

The merge with main moved exit-code wording into `exit::MEANINGS`, and
this branch's `failed` entry now says a timed-out or stopped command may
have made its change. Regenerate the published catalog so tools read the
same text the CLI prints.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(rocm-core): cover fix-2's Machine-scope read in the skill and catalog (ROCMAI-345)

An older CLI swallows a failed read of either scope, so with an
override set only in the Machine scope fix-2 can still report it gone.
The skill's check for that case now reads both scopes, and since the
output does not say which CLI produced it, the skill asks for the check
on every Windows fix-2 and fix-9 run. The published `failed` description
names the Machine read too, and the skill-eval replays are re-recorded
against the new catalog text.

Also stop the fix.rs module doc implying the consent.rs helpers are
re-exported; only the options.rs names are.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(rocm-core): pin fix-2's Machine read under --dry-run, say what to do with a set value (ROCMAI-345)

The published `failed` description says fix-2 reads its Machine value
even under `--dry-run`, but only the User read had a dry-run test. A
new test fails the Machine read under `--dry-run` with a User value set
and asserts exit 4 with no plan; it goes red when the Machine read
failure is swallowed.

The skill's older-CLI check told the agent to read both scopes without
saying what a set value means. It now says: a User value means
re-running fix-2, a Machine value needs an elevated PowerShell, and for
fix-9 an existing value is left for the user to change by hand. The
lead-in says the Windows case applies to every run, since the output
cannot tell an older CLI apart.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit ca170c9)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…0) (#847)

* fix(installer): keep download progress inside the terminal (ROCMAI-770)

curl takes its progress-bar width from COLUMNS in preference to asking
the terminal, so a value inherited from a wider window -- what a shell
leaves behind when its terminal is resized after it starts -- made every
refresh run past the right margin and wrap. The carriage return ending
each frame then landed a row below the bar, so a single download painted
a stack of partial bars with a percentage torn across two rows instead
of one bar updating in place.

Measure the terminal and pass that width down, reserving the last column
so the line stays clear of the margin. Resolve the download URL up front
as well: release assets redirect to a signed host and the redirect
responses carry no Content-Length, so curl drew its unknown-size
"bouncing" bar for every hop before the real one began.

wget's branch had a smaller version of the same problem: --show-progress
without -q interleaves the bar with request logging.

Covered by lifecycle-28, which drives the installer under a pseudo-
terminal narrower than COLUMNS and fails on the unfixed script with all
seven progress frames reaching or exceeding the terminal width.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

* fix(installer): address review on the progress-bar fix (ROCMAI-770)

The percentage assertion could not fail on the bug it was named for.
It read the raw pty byte stream, but wrapping is something a terminal
does while drawing, never something the writer puts in the stream, so
an over-wide bar reaches it as one intact frame with a whole 100.0% in
it. Render the capture through vt100 and assert on physical rows
instead: screen().contents() stitches a wrapped row back onto its
parent and would hide the tearing, so read rows() directly. The
scenario now fails on the unfixed script with 11 progress rows where
three downloads should leave three.

Drop the redirect pre-resolution. It only ever shortened the bouncing
bar from about three frames to one, since most of that wait is curl
connecting rather than following hops, and it cost a HEAD per asset
including both sidecars. It also never fell back as its comment
claimed: curl prints url_effective even when the probe fails, so the
original URL was never restored. Removing it leaves the width fix,
which is what the reported stacking actually came from, fully covered.

Three fixes to the width measurement itself:

  - A terminal whose size was never set reports 0x0. That was treated
    as a measured width and suppressed the bar entirely, where curl
    had previously drawn one at its own default.
  - tput sizes from its own stdin, which is the pipe under
    `curl ... | sh`, so the fallback read ncurses' 80-column default
    rather than the terminal. Point it at fd 2 like its sibling.
  - A zero-padded COLUMNS is an octal literal to POSIX arithmetic and
    dash rejects it outright. Strip leading zeros by hand; bash's 10#
    is not POSIX and dash rejects that too.

wget keeps its bar but moves from -q to -nv, so a failed download still
says why instead of leaving only this script's generic message.

Covers the branches that had none: a terminal too narrow for a bar now
says so rather than going silent, and the wget fallback is exercised by
shadowing curl out of PATH. Each new scenario was confirmed to fail
with its behaviour reverted.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

* fix(e2e): keep the shadow-PATH helper compiling on Windows (ROCMAI-770)

std::os::unix::fs::symlink has no Windows counterpart, and the Windows
lifecycle lanes build this target even though every scenario reaching
the helper is tagged @requires-os:linux. Gate the one call that has no
portable spelling and leave the rest of the function shared.

Verified by compiling the construct for x86_64-pc-windows-msvc as well
as the host, rather than assuming: a full cross-check of the crate is
not possible here, since its C dependencies need a Windows toolchain.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

* chore: re-trigger CI (ROCMAI-770)

GitHub Actions created no check suite for the previous head, so the PR
sat with no CI on it. Seven third-party suites registered against that
commit and the Actions one simply never appeared; the workflow has no
path filters and the same trigger fired for both earlier commits, so
this is an Actions-side miss rather than configuration.

Empty on purpose: it carries no change, only a synchronize event.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

* test(installer): cover the width sources the fix actually depends on (ROCMAI-770)

Three of the behaviours this PR claims had no test that could fail if
they regressed, and review was right that the gap mattered: each one
sits behind the measurement chain, so a pty with a real size never
reaches it.

  - Piping the installer into a shell, the shape it actually ships in,
    leaves stdin a pipe and only stderr a terminal. That is the whole
    reason the width is read from fd 2, and deleting the redirection
    kept every scenario green. The new scenario runs at 100 columns
    rather than 80 because ncurses answers 80 when it cannot read a
    terminal, which at this suite's usual width is indistinguishable
    from a correct measurement.
  - A terminal reporting no size is now stood in by a stty shim, since
    no real pty here will report 0x0 on demand, with tput left
    available so falling through yields the terminal's own width
    instead of the 80-column default a later guard would supply.
  - A zero-padded COLUMNS is only consulted when neither stty nor tput
    can answer, so both are shadowed out of PATH.

Each was confirmed to fail with its own fix reverted, which an earlier
attempt at two of them did not: they passed either way until the shape
above was in place.

The PATH helper grew shims and multiple names for this, assertions read
the width the run actually used rather than a constant, and
docs/testing.md names the new coverage as AGENTS.md requires.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

* test(installer): stop the download notice counting as a progress frame (ROCMAI-770)

Frames were picked out by carriage return alone, on the reasoning that
only a downloader redraws in place. It does, but the line it redraws on
also carries the preceding "download: <url>" notice, so that notice was
being counted as a frame. Against a real release URL it is 106 columns
wide and would have failed the width assertion on any terminal narrower
than that; it passes in CI only because the loopback fixture's URL is
short.

Require a carriage return and a bar character together. Neither works
alone: the notice redraws but carries no bar, and the get-started block
prints "rocm install sdk  # ..." which carries one but never redraws.

Found by smoke-testing against the real nightly rather than the fixture.
All six progress scenarios still fail with their own fix reverted.

Signed-off-by: John Lybeck <john.lybeck@amd.com>

---------

Signed-off-by: John Lybeck <john.lybeck@amd.com>
(cherry picked from commit cdc3aee)
Signed-off-by: juhovainio <juho.vainio@amd.com>
In `rocm dash` and interactive chat, the approval card for a mutating
assistant tool call (launch_server, install_sdk, install_engine,
stop_server, watcher_enable, watcher_disable) was rendered from the
call's argument object, but the approved replay went through
run_internal_mcp_call, which swapped in a nested `arguments` object
when one was present. A model-supplied call carrying such a field
showed one command on the card and ran a different one once the user
approved it.

The replay now validates and runs the object it is given, unchanged.
validate_chat_tool_call refuses any `arguments` field outright, so the
dash, chat and `rocm mcp-call` paths all refuse such a call before it
runs, and the dash and chat refuse it before a card is shown: no ROCm
tool declares a parameter of that name, so a call carrying one holds
two payloads and is refused rather than resolved to either.

The hidden `rocm mcp-call` command still accepts a pasted MCP
tools/call params envelope and unwraps it at its own boundary, but only
a pure envelope: `name`, `arguments` and the protocol's optional
`_meta`, which is dropped. It refuses an envelope naming a different
tool, and passes an object mixing tool fields with `arguments` through
whole, so validation refuses it. Arguments that are not a JSON object
are now refused rather than run as `{}`.

Tested by unit tests that the validator refuses a nested `arguments`
field for every ROCm tool name, that the dash card and the approved
replay agree, that the replay refuses a nested payload, that the
mcp-call boundary unwraps only a pure envelope, and that no tool in
either catalog declares an `arguments` parameter; and end to end by
chat-17, where a mock model answers with a two-payload launch_server
call and the model is told the call was refused and no approval prompt
is shown, chat-18, where a whole tools/call request handed to
`rocm mcp-call` runs the tool it names, and chat-19, where a request
with tool fields beside a nested `arguments` object is refused and
prints no tool result. The e2e mock server gains a scripted tool-call
reply keyed on the user's words, with its own harness test.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit c9d50ed)
Signed-off-by: juhovainio <juho.vainio@amd.com>
* feat(runtimes): add --json to rocm runtimes list (ROCMAI-148)

Scripts that want the registered runtimes, which one is active and
which is the rollback target currently have to scrape the text table
and its `*`/`-` markers. `rocm runtimes list --json` prints the same
listing as one JSON document on stdout, through the shared JsonOrText
helper.

The listing is now computed once into a value and both forms render
from it, so the JSON `active`/`rollback_target` flags and the text
markers cannot disagree. Unset values are null in the JSON, and the
empty-registry next-step hint stays text-only.

The text output of `rocm runtimes list` is unchanged except where
`default_runtime_id` matches a runtime id only when letter case is
ignored. The listing now applies one default-id rule, ASCII case
ignored, to both the active marker and the problem report: the false
"missing manifest for active_runtime_id" line printed next to an
active runtime is gone, and runtime ids that differ only in case now
report an ambiguous-runtime line instead.

docs/json-output.md audits `--json` support per command family, states
the rules those commands follow (stdout carries only the document,
errors stay on stderr, documented fields change only additively), and
records where the listing still differs from the resolver `rocm
examine` uses. Whether to adopt a shared envelope across commands is
left open there.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(runtimes): list the runtime the commands act on (ROCMAI-148)

`rocm runtimes list` decided which runtime was active and which was the
rollback target by rules of its own, so it could mark a runtime other
than the one `rocm examine`, `rocm runtimes activate`, `rollback`,
`uninstall` and `rocm update` act on: an active key in other letter
case, a dead key whose default id still resolves, case-twin keys, or a
previous key that rollback trims, matches ignoring case or reads as a
runtime id.

The listing now asks the resolvers those commands use:
`current_runtime_manifest` for `active` and `select_runtime_manifest`
on `previous_runtime_key` for `rollback_target`. `active_problem`
reports the configuration with the same matching, so a dead active key
is still reported as `missing_manifest_for_key` while the runtime the
commands fall back to is marked active. The resolvers are unchanged.

In the JSON, `active_runtime_id` is renamed `default_runtime_id`, the
config value it carries; the text label stays `active_runtime_id:`.
`registry`, `marker` and `install_root` are written lossily, so a path
that is not valid UTF-8 no longer makes `--json` fail with empty
stdout. `toolchain` is an enum serialized with the words the text and
`rocm examine` print.

docs/json-output.md describes which runtimes are marked, lists the
surfaces that still resolve differently (the dash runtime manager,
`rocm setup status`, `rocm engines install`), and states the exit
codes: 1 for a failed command unless documented, 2 for a command line
the argument parser rejects. The dash comment keeps only the dash's
own rules and points there.

A state matrix of in-memory unit tests covers every configuration
row; scenarios runtime-lifecycle-24 and -25 check an active key in
other letter case and a dead key falling back to the default id, each
against `rocm examine --json`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(runtimes): assert runtimes list --json prints nothing on stdout when it fails (ROCMAI-148)

The JSON rules say a failing --json command prints its error to stderr,
nothing on stdout, and exits 1, but every runtimes list scenario was a
success path. Scenario runtime-lifecycle-29 replaces a registry entry
with a folder of the same name, which the listing cannot read on any
platform or as root (an entry that reads but does not parse is skipped,
so it cannot stand in), and checks exit 1, an empty stdout, and a
plain-text error naming that entry.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* feat(runtimes): include provenance in runtimes list --json (ROCMAI-148)

The text listing names each runtime's channel, package source, source
layout generation and exact GPU target on its `provenance:` line, but
the JSON form left them out, so a script had to parse the text to learn
which stream or GPU a runtime was installed for.

Each runtime in `rocm runtimes list --json` now carries a `provenance`
object with the same four facts under the same names, as strings, or
`null` where the text prints `unrecorded`, so a script cannot take the
placeholder for a recorded value. The JSON takes its names and order
from a new `RuntimeProvenance::recorded_fields`, which `fields` (what the
text and `rocm examine` print) is now derived from, so the two forms
cannot name the facts differently.

Unit tests pin the object for recorded, unrecorded, blank and padded
values, check that the text line and the JSON object agree for each
runtime, and add the object to the exact-key-set contract test. Scenario
runtime-lifecycle-31 reads the provenance from the JSON for planted
nightly, older and adopted runtimes and compares it with the text
listing. It turns the startup update check off, as the examine
scenarios do, because with wheel runtimes registered that check can
print a progress line on stdout ahead of the document. README,
docs/json-output.md, docs/testing.md and docs/manual-testing.md describe
the field.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(runtimes): state the startup update check caveat and fix a misplaced test comment (ROCMAI-148)

docs/json-output.md promises that --json leaves only the document on
stdout. The startup update check that runs before most commands can
still print a Python progress line on stdout first, once per check
interval, when the runtime it checks is a wheel install whose recorded
Python is unusable and the Python it finds instead does not fit.
State that exception and the variable that turns the check off, so a
script author knows about it until the follow-up moves the line.

Also move the doc comment of the active-and-rollback-target list test
back above that test; the provenance fixture helper had been inserted
between them.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit bfff521)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…compatible engines (ROCMAI-332) (#1073)

The engine request and response envelopes carried no version. A daemon and
an engine from different releases could not tell whether they spoke the
same protocol, and a mismatch surfaced, at best, as a parse error after the
engine had acted on a request it misread.

Both envelopes now carry an optional `protocol_version`, written on every
request and answer as `ENGINE_PROTOCOL_VERSION` (0.1.0). A message without
it is read as 0.1.0, the format from before the field, so engines and
daemons from earlier releases keep working; a message with no version is
written without the field, and unknown fields are ignored so an additive
change is read. A new `protocol_version` module holds the rule:
MAJOR.MINOR.PATCH of plain decimal numbers without leading zeros that fit
in 64 bits, compatible by the caret rule (major and minor must match while
the major is 0, the major alone from 1.0.0), anything else unreadable and
incompatible, and the bump policy, including that a new method a client may
send raises the minor.

Each side judges the other's version from the raw JSON object before
decoding anything else.
- The lemonade, vllm and aim engines, and `rocm __engine-stdio`, answer the
  request text through a shared `answer_engine_request`. An incompatible or
  unreadable version, including one present and neither a string nor null,
  gets `incompatible_protocol` whatever the method. A servable request
  naming an unknown method gets `unknown_method`, and input that is not a
  request envelope (including JSON that is not an object) gets
  `invalid_request`. Each is an error envelope on stdout with a clean exit;
  input that is not valid UTF-8 is not read as a request. A decoded request
  is refused the same way before its payload is read.
- The shared stdio client that rocm and rocmd run engines through checks an
  answer's version before anything else in it, `ok` included, and for a
  single-object stdout that does not decode as an envelope, before its
  shape. An answer in a protocol it cannot use is an error naming both
  versions, with the child's failing exit status and stderr when there are
  any.

Before an install or a launch, the client first sends `capabilities`, with
an empty object as its payload and on the request's own arguments and
environment minus the progress request, and sends the request only when
that answer is in a compatible protocol and is not an error; otherwise the
error says the request was not sent and why. The check lives in the shared
client, for engine binaries it runs. rocm answers its built-in engines in
its own process without it, and no shipped command sends an install or
launch to an engine binary today.

`stop` is not checked first, so a stop is never held behind a check. An
engine that checks the version refuses a stop made for a protocol it cannot
serve. An engine from before the field acts on the stop, and its unversioned
answer is accepted while this release's protocol is 0.1.x. When the
engine's answer is an error or is refused, the stop paths in rocm and rocmd
still signal the server's recorded processes; only for aim, whose stop the
engine alone confirms, does that leave the stop unconfirmed.

An unusable stdout is described by what is wrong with it: nothing printed,
other output before or after an answer (quoted), JSON that is not an
envelope, or not JSON. Engine-written text the error quotes, including a
decoder's message about it, is cut to a bounded length with control
characters escaped. The note for an engine that answers and then exits
non-zero says the answer was read.

`unknown_method`, `invalid_request` and `invalid_payload` are shared
constants beside `incompatible_protocol`, and `EngineMethod::ALL` lists
every method. A shared `engine_contract` check behind the test-support
feature holds every first-party engine, and rocm's built-in engines, to the
same rules, and is itself tested against engines that each break one rule.
Tests cover the rule as a table and property tests, the capabilities check
as a state matrix against the fake engine, each engine's built stdio
binary, and the client's diagnosis. e2e scenarios engine-ipc-03 to -07 show
the daemon's stop report for answers it refuses and reads, and engine-ipc-08
to -12 show `rocm __engine-stdio vllm` answering requests it cannot serve.
The engine plugin, testing and architecture docs describe the contract.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 75fc9c9)
Signed-off-by: juhovainio <juho.vainio@amd.com>
* ci: run a pull request's MI300X check on the GPU queue

The self-hosted MI300X runners are unreliable for pull requests, and a
pull request's check should not have to wait for one of them. A new gate
job, e2e-gpu-queue, dispatches queue-pr-run.yml on main with the pull
request's branch (the runner runs only allow-listed workflows as they
are on main), waits for the run, and publishes the release-channel
report of its MI300X lane as e2e-gpu-report, the MI300X cell of the
consolidated report. The self-hosted MI300X lane is kept as the second
check and uploads e2e-gpu-selfhosted-report, shown as "MI300X
(self-hosted)".

The gate runs on a GitHub-hosted runner with only actions: write, for a
pull request from this repository only (a fork's token cannot
dispatch). A run the queue does not take (too many going, the nightly's
window, no healthy MI300X worker) fails the gate with the queue's
reason, which is capacity and not a failing test, and the missing-report
check covers the gate like any other lane.

The workflow contract now allows queue-pr-run.yml on the queue runner
with the same no-credential rules as queue-run.yml, lets e2e-selfhosted.yml
alone name it, and pins the gate's conditions and permissions.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: cancel a superseded queue gate and its run, retry API calls

Review of the queue gate found:

- the gate's `always()` kept its condition true through a cancellation, so
  a superseded push would not stop it polling for hours; it is dropped,
  and a cancelled gate now cancels the queue run it started so the runner
  that run holds is freed
- one failed API call (a 502, a rate limit) ended the gate and orphaned
  its run; the lookups are now asked again
- the branch name was checked only in the gate, which the pull request
  author edits; queue-pr-run.yml on main now checks it too
- the permission test passed with a second permission beside
  actions: write; it now matches the whole block
- the docs said two workflows, one runner, and did not say the queue
  tests the branch as it stands when the run starts, not the pull
  request's merge with its base

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: end the queue gate green when the nightly window skips it

A refusal for the nightly's window is by design, and a red X every night
for it would teach everyone to ignore the lane. The gate now ends green
with a summary note to re-run after 07:00 UTC and exposes a `skipped`
output; the missing-report check reads that as skipped, not missing, in
both workflows that carry it. Every other refusal stays red.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: make the gate's permission test exact and update the stale queue docs

The gate test accepted a second permission beside actions: write; it now
compares the whole permissions mapping. The script scan stops at the next
step. docs/ci-hardware-testing.md named two queue-report workflows and one
MI300X check; it now names three and two.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: skip the queue gate while queue-pr-run.yml is not on main

A dispatch runs main's copy of the workflow, so the pull request that adds
queue-pr-run.yml got an HTTP 404 and a red gate. Say so and skip instead;
any other error from the check stays red.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: cancel an undiscovered queue run on failure, retry the conclusion fetch

The gate's cleanup ran only when cancelled, so a dispatched run whose id was
never found stayed on the queue after the job failed. It now runs on failure
too and looks the run up by its title when no id was kept. The conclusion
fetch after the poll retries five times with a growing pause, as the poll
does, instead of twice back to back.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: test the queue gate's script, say which MI300X lane a claim means

Run the gate's script against a mock gh: the missing-workflow skip, the
nightly refusal skip, retries, run discovery, the report copy and the verdict.
The queue contract (the nightly: prefix of refused.txt and the report path) is
named once in the test and in docs/nightly-on-the-queue.md.

The missing-workflow skip now needs the branch to carry the file, so a file
removed from main and the branch stays red. The branch-name error names the
rules. A pull request has two MI300X checks, e2e-gpu-queue and e2e-gpu, on the
same silicon: the workflow header, docs/ci-hardware-testing.md and
docs/testing.md say which they mean. queue-pr-run.yml's comment now says a
pending run can be cancelled by a newer one.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci: switch the queue gate on with a variable, skip Dependabot, bound the poll

The gate does nothing until the repository variable QUEUE_PR_GATE is true, so
the queue side can be deployed before it. Dependabot's read-only token cannot
dispatch: its gate is skipped. The status poll stops after ten failed reads in
a row. The cancel step looks for a dispatched run as long as the main step
does. Counts and clock times the queue side owns are now invariants.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 0e830f3)
Signed-off-by: juhovainio <juho.vainio@amd.com>
install.ps1 shipped without the MIT/copyright header that every other
source file in the repo carries, because the header check never looked at
PowerShell files at all: `licenserc.toml` did not include them. The same
was true of PSScriptAnalyzerSettings.psd1 and of the WSL helper script
under .github/scripts, which also had no header.

Adding the globs alone is not enough. hawkeye has no built-in comment style
for the `ps1`, `psm1` or `psd1` extensions, so a file matched by `includes`
but with no matching rule is classified `unsupported` and silently skipped
while the command still exits 0 -- the check would look green without
inspecting the file. The explicit `[[rules]]` block mapping those extensions
to the `script` comment style is what makes the files actually get checked.
Only `hawkeye check --fail-on-unknown` exposes the gap, and neither the CI job
nor the hook passes it.

PowerShell files get an ASCII `Copyright (c)` header, rendered from a
conditional in the header template, instead of the copyright sign the other
file types use. A file with any non-ASCII byte needs a byte order mark to
satisfy PSScriptAnalyzer, and install.ps1 cannot have one: the documented
install path is `irm ... | iex`, which hands the parser a string, so the mark
survives into it and the param() block stops being the script's first
statement, silently dropping every installer parameter. Keeping the headers
ASCII leaves install.ps1 as ASCII-only as its existing comment says and needs
no analyzer suppression. The other PowerShell files follow for consistency,
and LICENSE.TXT already uses the `(c)` form. Headers for every other file type
render byte-for-byte as before.

The license-headers hook already runs on every commit (it has no file-type
filter), so no hook change is needed: a PowerShell-only edit reaches it.

PSScriptAnalyzerSettings.psd1's comments named scripts that do not exist; they
now describe what the lint covers (every tracked .ps1) and note that .psm1 and
.psd1 are licence-checked but not linted.

Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com>
(cherry picked from commit f4b9945)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…guard .rocmNNN (#1109)

* feat(vllm): add a ROCm 10.2 discover row and teach the local-version guard .rocmNNN

The staging index publishes vLLM 0.32 wheels for ROCm 10.2 (local version
spelled .rocm102), but the discover table stopped at 10.1, so the install
failed closed on a 10.2 SDK.

Add the 10.2.0 row (cp314, staging index, torch 2.14, torchvision 0.29).
whl-next has no +rocm10.2 torch, so the row borrows the 10.1 stack and the
new torch_stack_matches_sdk flag skips the local-version check for the torch
triplet on that row only. The guard now also compares the compact .rocmNNN
spelling, so a wheel for another line is still rejected.

Refs #1108

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): read the 10.2 torch stack from the staging index

The staging host publishes torch 2.14.0, torchvision 0.29.2 and torchaudio
2.11.1 built for +rocm10.2.0rcN, so the 10.2 row no longer borrows the 10.1
stack. Drop the torch_stack_matches_sdk flag and the check skip it needed.

Refs #1108

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): reuse the SDK's torch stack for the 10.2 row

Running the row on a real 10.2 SDK showed the staged torch does not work:
the newest torch 2.14.* on the staging and nightly indexes is a +rocm10.3
build, and the staging +rocm10.2.0rcN torch needs device payloads and a
rocm==10.2.0rc2 that staging does not serve.

Add TorchStack::FromInstalledSdk: the row keeps the torch, torchvision and
torchaudio the SDK install resolved, checks them against the row's series and
ROCm line, and realigns to those exact builds from the nightly index. The
other rows still discover theirs.

Refs #1108

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): fetch the 10.2 row's torch stack from the index the SDK came from

The 10.2 row reuses the SDK's own torch, torchvision and torchaudio but
fetched those exact builds again from the row's fixed nightly index. A
release SDK's torch is not on that index, so the realign step would fail
once 10.2 ships through the release channel.

Read the index the runtime manifest records for the interpreter and use it
for the torch install and both realign steps. A manifest that recorded none
keeps the row's index, and an unreadable registry does the same but leaves a
note. Rows that discover their stack are unchanged.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): honour exact torch pins and bare +rocmNNN, and correct the discover-rows doc

Review fixes for the 10.2 discover row:
- installed_torch_stack_pins read an exact torchaudio pin as a series prefix
  of a version that still carried its +local part, so an Exact pin rejected
  the release it names and accepted longer ones. An exact pin now means that
  release, a series keeps its prefix rule.
- ensure_rocm_local_version_matches sent a local version starting with rocm
  to the dotted parser only, so a bare +rocm102 was rejected. The dotted
  spelling is now read from its first rocm segment wherever it sits, and the
  compact one is the fallback.
- Assert the 10.2 row's torch_index_url, which stays the fallback index.
- docs/vllm.md claimed all three rows share the stable index and resolve
  their torch pins fresh; that holds for 10.0 and 10.1 only.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* fix(vllm): reject a rocm segment no spelling reads, and say when the torch index falls back

Follow-up review fixes:
- A local version with a rocm segment that is neither the dotted nor the
  compact spelling (+rocm, +rocmabc) passed the guard since the last change;
  it is a mismatch again, as it was before.
- Only the row that reuses the SDK's torch stack looks up the recorded index,
  so an unreadable registry no longer adds a note to 10.0 and 10.1 installs,
  whose index is fixed by design.
- That row also says so when the manifest records no index, and every failure
  on the torch install and the realigns carries the fallback notes, not just
  the remaining-deps install.
- The unreadable-registry test skips itself where mode 000 does not block
  the reader (running as root).
- docs/release-trust.md: an overridden pip base is recorded in the manifest
  and reused by later vLLM installs for that runtime.

Signed-off-by: juhovainio <juho.vainio@amd.com>

* refactor(vllm): attach the fallback notes to every failed discover step, and test them

Review nits on the 10.2 discover row:
- The notes about a fallback (companion pins, the torch index) were attached
  to individual failures one call at a time, so errors before the torch
  install and between the realigns still lost them. The sequence now records
  them as it goes and the entry point attaches them once, to whatever fails.
- Test each real install step failing on the fallback index, not just the
  first.
- The unreadable-registry case plants a directory where a manifest file
  belongs, so it fails for root too and no longer needs a skip.
- Say 'no runtime manifest for this interpreter records a wheel index', which
  also covers an interpreter with no manifest at all.

Signed-off-by: juhovainio <juho.vainio@amd.com>

---------

Signed-off-by: juhovainio <juho.vainio@amd.com>
(cherry picked from commit 44d599f)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…1070)

Follow-up to #1011 review. The contract test now also requires the doc to
give the run limit in hours next to its seconds (3 h (10800 s), and ties the
workflow comment to timeout-minutes exactly: the 30 min, a margin derived
from timeout-minutes less the two limits and 30 min, and the total, in the
lines right above the timeout-minutes line.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 3209459)
Signed-off-by: juhovainio <juho.vainio@amd.com>
….ops (#1103)

* ci(e2e): fail the pre-warm when the runtime cannot import torchvision.ops

A persisted pre-warm runtime can pass every freshness check and still be
unusable: when its torch and torchvision come from different installs,
`import torchvision` dies with `operator torchvision::nms does not exist`.
Nothing ran that import before the suite, so each vLLM serve scenario found
out for itself after waiting out its readiness timeout, and a lane that takes
about 33 minutes when healthy ran 79 to 90 minutes and failed.

The pre-warm now probes the active runtime's own Python, which is where vLLM
installs its packages, for `import torchvision.ops`, with the runtime's
recorded library directories on the loader path and a two-minute deadline:

- Before the freshness decision, a runtime whose torch and torchvision do not
  match is uninstalled, so the decision installs afresh. A pinned run, a
  runtime marked ahead of the index, and a hang or any other import error are
  left alone.
- After the engine check, which also runs when the runtime is reused, a
  runtime that still cannot import fails the pre-warm, naming the runtime and
  what to do. Where the default engine does not import torch (Lemonade) the
  finding is a warning.

A runtime without torchvision is skipped.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci(e2e): repair, bound and scope the pre-warm's torchvision check

Review round on the torchvision import check:

- The probe puts the runtime's recorded `rocm_sdk.library_paths` (and the
  amd-smi shim) on the loader path, as the CLI's own probes do. A library that
  still will not load is a broken runtime; the skip remains only for a manifest
  that records no library directories.
- The probe is killed after two minutes and reported, instead of stalling the
  pre-warm until the job times out.
- A runtime whose torch and torchvision do not match is uninstalled before the
  freshness decision, the same repair as for a poisoned runtime, so that decision
  sees a registry without it and installs afresh. Other import failures still
  fail the pre-warm without deleting anything.
- The finding is a warning where the default engine does not import torch
  (Lemonade).
- The check moves to its own file to keep e2e_prewarm.rs under the module-size
  limit.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci(e2e): bound the probe's pipe readers and tighten its tests

- The wait on the probe's reader threads is bounded, so a process that
  inherited the pipes cannot hold the pre-warm past the probe's deadline. The
  readers are detached and end when the pipes close.
- The mismatch signature is kept in the error tail wherever it appears in
  stderr, not only in its last three lines.
- The repair test's stand-in `rocm` removes the runtime's registry entry, so its
  final assertion fails when the uninstall is not requested; the table of cases
  that must not be uninstalled has a real hang case in place of a duplicate.
- The source-order test checks the order of the calls rather than exact lines.
- Tests that write and run a script hold a lock, since a concurrent fork makes
  the exec fail with ETXTBSY.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci(e2e): gate a unix-only test constant, anchor the mismatch match

- `MISMATCH` is used only by unix tests, so it is dead code under
  `-D warnings` on Windows; gate it with them.
- The operator-mismatch test requires both `torchvision::` and `does not
  exist`, since it decides whether a runtime is deleted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

* ci(e2e): leave a pinned or ahead-of-index runtime in place

The torchvision repair runs before the freshness decision, so it would also
have removed a runtime the pre-warm never rolls back. It now declines when the
run is pinned to an exact version, when `rocm update` marks the runtime
`ahead_of_index`, or when that report cannot be read; the check after the engine
install then fails with the manual advice. The real-interpreter test skips,
rather than fails, when `python3` is not on PATH. The documentation says that a
mismatch the engine install itself creates on a reused runtime is repaired on
the next run.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>

---------

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
(cherry picked from commit 6e23942)
Signed-off-by: juhovainio <juho.vainio@amd.com>
`rocm serve` protects a vLLM endpoint (public bind, --require-api-key,
`rocm remote serve`) by setting VLLM_API_KEY. vLLM checks that key only
under a fixed set of path prefixes (/v1 and a few others), while the
server also answers routes outside them that run or expose the model,
such as /invocations, /tokenize, /detokenize and /metrics. Those routes
answered clients that did not present the key.

A keyed vLLM launch now also passes
`--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`: a pure-ASGI,
standard-library middleware compiled into the engine adapter, written
atomically under the engine's data directory at launch and prepended to
the server's PYTHONPATH (composed over the inherited, removed or
explicitly set value, so the user's own PYTHONPATH and the runtime's are
kept). It requires the bearer key, compared in constant time, on every
HTTP and WebSocket route except GET/HEAD /health and CORS preflights,
and raises instead of constructing when the key is empty, so a server
that loaded it never answers a request unchecked. Being path-agnostic,
it also covers routes a later vLLM adds. The module's copyright comment
is exempted from the xtask ASCII-output contract, as install.sh's is;
the rest of the file is ASCII.

The launch command is built by one function, vllm_server_command, so a
keyed launch can be asserted to carry --middleware, VLLM_API_KEY, the
middleware directory ahead of the runtime's own PYTHONPATH entry and the
selected GPU, and an unkeyed launch none of the key protection.

Because /metrics is now covered, `rocm dash` cannot read engine metrics
for a key-protected vLLM server and shows a persistent scrape warning,
and `rocm bench load`, which sends no key, cannot benchmark one. The
README says so, and that the every-route guarantee is the vLLM
engine's: AIM runs the same vLLM server without this check, so routes
such as /invocations, /tokenize and /metrics still answer there without
the key. The manual-testing, testing and architecture docs
describe the check; manual-testing sections after the new one are
renumbered, and the three places that cited one of them by number now
cite it by title.

Tested by driving the shipped middleware per route under python3 with a
stdlib ASGI harness (including CORS-preflight look-alikes and a
WebSocket /health), by asserting the launch argv and environment, and
end to end by serve-61 (`@id:serve-vllm-key-guards-every-route`), a
`@nightly` GPU scenario that runs in the nightly runs on Instinct GPUs,
which serve with vLLM (the MI300X job in nightly.yml and the shared GPU
queue's Instinct lanes). A real keyed vLLM server refuses
keyless calls on those routes, accepts keyed ones, and still answers
/health. The e2e step passes the key through the environment, with a
new time-bounded run helper, so it stays out of the recorded command
log.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 42c2968)
Signed-off-by: juhovainio <juho.vainio@amd.com>
* test(e2e): fail lanes that test nothing (ROCMAI-388, ROCMAI-450)

A self-hosted GPU lane could pass without testing its GPU. The capability
probe ignored `rocm examine`'s exit status, so a probe that failed read as
a host with no GPU and every @requires-gpu scenario resolved to skip.
Skipped scenarios never reach report.json, and the only outcome-based exit
iterated its results, so a run in which everything was filtered out exited
0.

Lanes now declare E2E_EXPECT_GPU: 1 on the self-hosted GPU lanes and the
queue lane script, 0 on the WSL lanes, whose distro is deliberately left
without ROCm's WSL driver bridge. The harness fails before any scenario
when the probe could not read the binary (spawn failure, non-zero exit,
empty output) or a declared GPU is missing, and after the run when nothing
executed or, on a GPU-declaring lane, no @requires-gpu scenario executed.
A workflow contract test holds every self-hosted lane to the declaration.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): bound probes, flag failed lanes (ROCMAI-388, ROCMAI-450)

The lane checks added in the previous commit could still be defeated
in two ways, and this closes both.

A capability probe that never returned hung the whole run instead of
refusing it. Each probe command (`rocm examine`, `rocm engines list`)
now runs under a deadline, E2E_PROBE_TIMEOUT_SECS (whole seconds, 120
by default), and is killed and reaped when it passes. ProbeError gains
Timeout, Wait (the child could not be waited on) and OutputNotClosed
(the command exited but a process it started still holds its output),
each naming the command and keeping the stderr read so far. The
post-run version probe shares the deadline but only blanks the version
columns.

A lane failed by the post-run check (no scenario executed, or no
@requires-gpu scenario on an E2E_EXPECT_GPU=1 lane) exited non-zero but
still read green in the consolidated report when every scenario it
resolved was a skip. The reason is now written to platform.json as
`lane_failure`, and the report renders that lane as FAIL with the
reason, in both the step summary and the HTML report. The single
"Needs attention" list is split into a platform heading (lane-check
failures) and a scenario heading (the expectation grid), so neither
renderer shows two identical headings. Release readiness counts such a
row, whose Fail cell is 0 and so never reaches the Total row, and does
not count a row twice when it also has failed scenarios.

Annotation and notice lines now escape `%`, CR and LF as the runner
expects, so a probe's stderr tail can no longer truncate or corrupt a
workflow command.

The decisions the harness takes around these checks (reading the
declaration, counting GPU scenarios, building the manifest, the final
exit and the exact annotation lines) move out of tests/e2e.rs into pure
functions in lane_verdict.rs, each unit-tested; the harness keeps only
the glue that runs the probe and the scenarios and writes the files.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): tighten lane-verdict tests and docs (ROCMAI-388, ROCMAI-450)

The GPU-lane check already counts @requires-multi-gpu scenarios as GPU
scenarios, but its messages, doc comments and docs named @requires-gpu
alone. The fatal reason and the E2E_EXPECT_GPU=0 note now name both
tags, and the docs say that GPU scenarios the run's own configuration
leaves out (@nightly, @merge-queue, outside an E2E_ONLY_LIFECYCLE
selection) count as resolved but never as executed.

New tests: the E2E_EXPECT_GPU=0 note's "N of M" arithmetic when some GPU
scenarios ran (4 of 7), and a probe that reads stdin getting EOF at once
rather than blocking until its deadline. Like its rocm-core sibling, the
stdin test catches a missing stdin setting but does not pin null against
inherit. The constant-equals-itself test is dropped; the workflow
contract test already ties the name to the workflows.

The workflow contract test now also fails when the step that runs the
suite sets E2E_EXPECT_GPU itself, overriding the job-level declaration,
when a lane has no such step, and when that step runs in WSL without a
WSLENV of its own naming the variable.

Stale comments corrected: the workflows now say the Windows lane's
E2E_EXPECT_GPU=1 check backstops its best-effort GPU preflight, and that
a harness preflight refusal shows as an EMPTY row rather than a missing
report; run-e2e-lane.sh says a set-but-empty E2E_EXPECT_GPU leaves the
lane undeclared. The wsl fixture gets the driver status a passthrough
host reports, so tests built from it no longer reset the field.

The markdown report folds any whitespace run in a lane-failure reason
to one space, CRLF included, not only LF.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* test(e2e): let the post-kill drain test fail (ROCMAI-388, ROCMAI-450)

The late writer in the post-kill drain test inherited stdout as well as
stderr, so the stdout wait already held the probe open until `late` was
written, and the test stayed green with the stderr wait removed. Send the
writer's stdout to /dev/null so only the stderr wait can collect it.

Also correct two notes in the testing docs: passing an @id to -n selects
nothing, and the run then fails with "0 scenarios executed" rather than
passing silently.

Refs: #383
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(e2e): name report.json origin, drop counts (ROCMAI-388, ROCMAI-450)

The probe-grace rationale in capability.rs's output_within doc, and the
leaked-thread note under it, rested on "three" probes. They now state the
invariant: each probe runs once per suite (the capability probes cached
by host_capability, and collect_versions' version probe), not once per
sample. The Cargo.toml note on the copied spawn-and-reap helpers likewise
says "these copies" rather than counting them.

The claim that a harness preflight refusal leaves an empty report.json
that is still uploaded now names where that holds: run_suite in
tests/e2e-cucumber/tests/e2e.rs creates report.json before it calls
lane_verdict::preflight, and on every self-hosted lane that ran its
suite the results upload (and the WSL copy-out) is gated on always().
This is said in the e2e-report test doc, the e2e README and both
nightly.yml comments.

Rewrapped the over-long lines in nightly.yml's missing-report comment,
lane_verdict.rs's parse_gpu_declaration doc and e2e-report's
EXPECT_GPU_ENV doc.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 698b9d5)
Signed-off-by: juhovainio <juho.vainio@amd.com>
siloteemu and others added 2 commits November 17, 2026 22:22
Turn on four more Ruff rule families for every Python file in the repository:
flake8-unused-arguments (ARG), flake8-blind-except (BLE), flake8-self (SLF)
and the Pylint set (PL).

SLF and the Pylint error rules report nothing. ARG needed two fixes:

- `setup(app)` in the Sphinx config is removed. Sphinx calls the config's
  `setup` and discards what it returns, and the parallel-safe flags it
  returned only count for registered extensions, so it did nothing; a config
  without `setup` is skipped.
- A dead `temp_state_root` parameter is dropped from `run_live_acceptance` in
  the local-assistant GPU test along with its argument at the single call
  site.

What still fires is waived in ruff.toml. The global ignores give their
reason, and each per-file entry states the rule new code under it has to
keep:

- PLR0912, PLR0913 and PLR0915 are ignored for every file, since acting on
  them means restructuring existing scripts. PLR2004 is ignored for every
  file as well: it flags inline numeric literals in comparisons, and naming
  them is a separate cleanup.
- BLE001 is allowed in comfyui_therock_gpu_test.py,
  therock_sdk_install_test.py and vllm_therock_gpu_test.py.
- PLC0415 and PLW0603 are allowed in vllm_route_check.py, whose self-test
  imports inside the function and rebinds a module global it restores in a
  `finally`.
- ARG001, PLW0603 and PLW1510 are allowed in the rocm-doctor eval hooks and
  their tests; ARG001 covers only the unused parameters of setup() and
  teardown(). Those files are waived rather than edited because a change to
  skills/rocm-doctor/ on main opens a bump pull request in the federated
  copy.

A waived rule checks nothing anywhere in that file, so new code in those
files is reviewed by hand.

ruff.toml and .pre-commit-config.yaml also described the Python scope as
covering "engine workers", which is Rust-only; both now say every Python
file in the repository, so they stay true as Python files are added.

Signed-off-by: Teemu Kuusisto <teemu.kuusisto@amd.com>
(cherry picked from commit 6427b50)
Signed-off-by: juhovainio <juho.vainio@amd.com>
…452) (#1088)

* fix(lemonade): escalate terminate_pid to SIGKILL after a grace period (ROCMAI-452)

Lemonade's terminate_pid(pid, force) ignored `force` and only sent
SIGTERM, and every caller follows it with an unbounded child.wait(). A
lemond or llama-server that ignores SIGTERM therefore hung the CLI
indefinitely during teardown.

Route terminate_pid through rocm_core::terminate_verified with the Tree
scope and the engine's 10 s stop grace: SIGTERM to the process tree,
then SIGKILL for anything still running when `force` is set, so the
call is bounded and the following wait returns. Tree scope matters
because lemond runs each loaded model in a llama-server child, and
killing lemond alone would orphan that child with the GPU still held.

vLLM's serve_http had the same pattern in its readiness-failure
teardown (terminate_process_tree, then wait); it now goes through a
stop_spawned_server helper with the same bounded escalation.

Document on terminate_process and terminate_process_tree that they only
request termination and neither wait nor escalate, pointing at
terminate_verified, and refresh the architecture entries. Regression
tests cover a child and a grandchild that ignore SIGTERM, the unforced
case, a cooperative child, and the Windows forced path.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(engines): report unconfirmed teardown stops (ROCMAI-452)

The bounded teardown from the previous commit still returned only a
bool, and every caller discards it before waiting on the child. A stop
that could not be confirmed, either because the process outlived the
grace periods or because its identity could not be read and nothing was
signalled, was followed by the same silent wait.

Both engines now write the reason for an unconfirmed stop to stderr.
The text comes from a new TerminationOutcome::unconfirmed_reason(),
which is None exactly when stopped() is true, so the warning cannot
disagree with the decision it reports.

vLLM's readiness-failure teardown moves into fail_unready_server. It
owns the unreaped Child, so after an unconfirmed stop it kills the
server before waiting, which bounds the wait on every path except a
server that survives SIGKILL. A server the readiness poll already
reaped is not signalled, because its PID may belong to an unrelated
process by then. A failed write of the `failed` state is reported on
stderr, so the readiness error stays the one returned.

Lemonade keeps terminate_pid(pid, force) -> bool and does not kill on
an unconfirmed stop, because it is called with bare PIDs; its docs now
say that an Unverified process is left unsignalled and a following
wait can block.

Tests re-run the test binary to read the warning on real stderr
together with the resulting process state, cover the production stop
grace through both entry points, and check that a confirmed stop
neither warns nor kills and is reaped before the teardown returns.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* fix(lemonade): bound the router serve teardown (ROCMAI-452)

The router serve path, which moved into its own module when registered
GGUF models gained a router route, stops a server that failed to load
or verify with a plain SIGTERM to its process tree and then waits on the
child. A lemond, or a backend under it, that ignores SIGTERM therefore
hung the serve on that failure path, the same hang the previous commits
removed from the other teardown sites.

Route it through terminate_pid like its siblings: SIGTERM to the tree,
SIGKILL after the stop grace, and a stderr warning when the stop cannot
be confirmed. A test stages a fake lemond and a backend child that both
ignore SIGTERM, and checks that the stop returns within a bound, reaps
the server and kills the backend.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

* docs(engines): say what the stop-reason test and STOP_SCOPE cover (ROCMAI-452)

The unconfirmed_reason test pins the reason strings, so a consistent
wrong edit to the production text and the test would stay green. Name
the engine tests that prove the warning is printed exactly when a stop
is unconfirmed, next to the state it describes, as the place the
behaviour is actually checked.

The STOP_SCOPE comment said Lemonade state always records the
llama-server itself. That holds on the direct-serve path only: the
router path records lemond, whose model runs in a child process, so a
Single stop that escalates to SIGKILL can leave that child running.
Say so, without changing the scope.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>

---------

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
(cherry picked from commit 340d987)
Signed-off-by: juhovainio <juho.vainio@amd.com>
@juhovainio
juhovainio requested a review from a team as a code owner November 17, 2026 20:39
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

17 medium, low, or unrated finding(s) are not listed; the scan run has them.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@juhovainio
juhovainio merged commit 07f72bb into release/v0.3.0 Nov 17, 2026
47 of 48 checks passed
@juhovainio
juhovainio deleted the cherry-pick-main-to-release-v0.3.0-rc3 branch November 17, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants