Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
79c5530
test(e2e): drop the serve-27 xfail row, the AIM stop now passes on St…
juhovainio Nov 16, 2026
a629db9
ROCMAI-455: keep endpoint API key files owner-only on every platform …
rominf Nov 16, 2026
57d391c
docs(contributing): add guidance on code claims, blocking objections,…
rominf Nov 16, 2026
51dfafe
fix(rocmd): stop the engine child when the supervisor cannot record i…
rominf Nov 16, 2026
49da089
docs(install): say the pinned release key is always enforced, and cov…
rominf Nov 16, 2026
9117d7e
ROCMAI-184: test ROCm 7 and ROCm 10 runtimes side by side, and clear …
rominf Nov 16, 2026
f029dc2
fix(lemonade): write service state atomically under a lock (ROCMAI-45…
rominf Nov 16, 2026
c4ae4ac
fix(vllm): make vLLM's pinned tensorizer import before an install is …
juhovainio Nov 16, 2026
15526c7
ci: retire the self-hosted nightly lanes that never get a runner; kee…
tomastola Nov 17, 2026
b392d54
fix(runtime): keep a second amd-smi library out of the runtime's torc…
tomastola Nov 17, 2026
0285f03
ci: a missing queue run makes the release-readiness verdict NO-GO (#1…
tomastola Nov 17, 2026
e85a864
fix(serve): say so in the summary when the post-launch smoke test fai…
tomastola Nov 17, 2026
e215192
ci: check that vLLM can install on the default release and nightly SD…
tomastola Nov 17, 2026
cb2c295
docs: add a manifesto to the README (#1099)
rominf Nov 17, 2026
2db608f
fix(wsl): make a managed runtime load its own librocdxg, and say when…
juhovainio Nov 17, 2026
17a7cda
feat(runtimes): show runtime provenance in list and examine (ROCMAI-1…
rominf Nov 17, 2026
12f5310
ci: make a stale MI300X E2E job end itself and cancel its run (#1110)
tomastola Nov 17, 2026
afcb9df
fix(lemonade): pin Lemonade 2026.44.1 so a ROCm 10.1 SDK gets a ROCm …
rominf Nov 17, 2026
49c95f8
fix(rocm-core): report a timed-out rocm fix step as such (ROCMAI-345)…
rominf Nov 17, 2026
aa014a5
fix(installer): keep download progress inside the terminal (ROCMAI-77…
johnl-amd Nov 17, 2026
2412032
fix(dash): run exactly the tool call the approval card shows (#980)
rominf Nov 17, 2026
072a32d
feat(runtimes): add --json to rocm runtimes list (ROCMAI-148) (#1098)
rominf Nov 17, 2026
e3519c9
fix(engine-protocol): version the engine stdio protocol and refuse in…
rominf Nov 17, 2026
73945e1
ci: run a pull request MI300X check on the GPU queue (#1104)
tomastola Nov 17, 2026
69fd39d
chore(license): enforce licence headers on PowerShell files (#369)
siloteemu Nov 17, 2026
55c838b
feat(vllm): add a ROCm 10.2 discover row and teach the local-version …
juhovainio Nov 17, 2026
889ed5d
test: pin the queue run timeout's comment and run limit more tightly …
tomastola Nov 17, 2026
16a359b
ci(e2e): fail the pre-warm when the runtime cannot import torchvision…
tomastola Nov 17, 2026
315d61c
fix(vllm): require the endpoint key on every server route (#974)
rominf Nov 17, 2026
91f16cb
test(e2e): fail lanes that test nothing (ROCMAI-388, ROCMAI-450) (#1068)
rominf Nov 17, 2026
aa3b1d8
chore(lint): enable ARG, BLE, SLF and PL ruff rules (#370)
siloteemu Nov 17, 2026
07f72bb
fix(engines): escalate Lemonade and vLLM teardown to SIGKILL (ROCMAI-…
rominf Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,7 @@ SPDX-License-Identifier: MIT
- [ ] If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per `docs/architecture.md` (the clap declaration in `cli.rs` and the dispatch wiring in `main.rs`/`lib.rs` are expected, not a violation).
- [ ] If this PR adds, renames or removes a production source file (or a `.rs` file under `xtask/src` or `tests/e2e-cucumber`), `docs/architecture.md` has the matching entry (`cargo xtask check-architecture-doc` fails otherwise), and the prose of any entry whose file changed role still describes it, which the check cannot verify.
- [ ] Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will *not* happen.
- [ ] Comments and doc lines this PR adds or changes that state how the code behaves were checked against that code, and each that is the reason for a decision names the file and the function, constant or test that makes it true (AGENTS.md §3).
- [ ] Any existing scenario this PR moves off the blocking mock `e2e` job (for example by a tag) is justified as a deletion (AGENTS.md §8).
- [ ] Commits answering review feedback were diffed whole against the reviewed head and widen no documented user-facing option's meaning or authority (AGENTS.md §11).
- [ ] Every commit message on the branch describes what its own commit does in words still true at the branch head (AGENTS.md §5).
4 changes: 4 additions & 0 deletions .github/scripts/Invoke-WslBash.ps1
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Copyright (c) Advanced Micro Devices, Inc., or its affiliates.
#
# SPDX-License-Identifier: MIT

# Runs a bash script inside the pinned WSL guest.
#
# Piping script text to `wsl.exe` from Windows PowerShell 5.1 is unreliable:
Expand Down
13 changes: 10 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1020,10 +1020,15 @@ jobs:
# serve-33/serve-34/serve-35 (the native-Windows vLLM refusal for `serve` and
# `engines install`) are folded in the same way: `@requires-os:windows`,
# hermetic, and otherwise run only on the optional self-hosted lane.
#
# runtime-lifecycle-19/20/21 (runtimes installed before and after the ROCm
# 10 layout, side by side) are folded in for the same reason: planted
# runtimes, no GPU or network, and their Windows paths are what this
# lane proves.
env:
E2E_INCLUDE_LIFECYCLE: "1"
E2E_ONLY_LIFECYCLE: "1"
E2E_ALSO_RUN_IDS: "comfyui-uv-install-failure-names-log,comfyui-uv-install-progress-streamed,comfyui-ambiguous-runtime-actionable,serve-vllm-refused-on-native-windows,serve-vllm-install-refused-on-native-windows,serve-vllm-install-with-runtime-refused-on-native-windows"
E2E_ALSO_RUN_IDS: "comfyui-uv-install-failure-names-log,comfyui-uv-install-progress-streamed,comfyui-ambiguous-runtime-actionable,serve-vllm-refused-on-native-windows,serve-vllm-install-refused-on-native-windows,serve-vllm-install-with-runtime-refused-on-native-windows,runtime-lifecycle-side-by-side-across-packaging-change,runtime-lifecycle-side-by-side-uninstall-keeps-the-others,runtime-lifecycle-side-by-side-shared-name-refused"
run: |
# Derive from CARGO_TARGET_DIR when it is set, falling back to the
# default target dir, so a future job-level CARGO_TARGET_DIR cannot
Expand Down Expand Up @@ -1869,8 +1874,10 @@ jobs:
# e2e-gpu-strix-windows). They live there — not here — so a job queued on an
# offline self-hosted runner (which GitHub cannot cancel) can never hold this
# workflow's concurrency group and stall its merge-required checks (EAI-7548).
# They are non-blocking (continue-on-error) and fork-safe (self-hosted runners
# do not execute untrusted fork PRs). See docs/ci-hardware-testing.md.
# They are non-blocking (none is a required check) and fork-restricted: GitHub does
# not start self-hosted jobs from an external fork's pull_request run without
# approval (whether that is every fork run or only a first-time contributor's
# is a repository setting). See docs/ci-hardware-testing.md.

# ── E2E tests (cucumber-rs) ───────────────────────────────────────────────
#
Expand Down
403 changes: 381 additions & 22 deletions .github/workflows/e2e-selfhosted.yml

Large diffs are not rendered by default.

439 changes: 169 additions & 270 deletions .github/workflows/nightly.yml

Large diffs are not rendered by default.

75 changes: 75 additions & 0 deletions .github/workflows/queue-pr-run.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# GPU E2E on the shared queue for one pull request's branch — start a run, wait
# for it, publish it.
#
# Dispatched by a pull request's gate job in e2e-selfhosted.yml, on main, with the
# branch to test. It is queue-run.yml with the pull request preset: the queue side
# runs one GPU model on the release channel, with a shorter wait and run time than
# the nightly, a limited number at once (the queue side sets the limit), and it
# does not take one while the nightly goes (it refuses, and this job then fails with that reason in refused.txt).
# Like queue-run.yml it needs no credential and no permission, and the runner
# runs only the allow-listed workflows as they are on main; that is enforced on
# the runner, outside this repository.
# What that protects is in docs/nightly-on-the-queue.md.
name: E2E on the queue (pull request)

# The run's title names the job that asked for it, which is how that job finds
# this run among the dispatched ones; the caller is only ever a title here.
run-name: Pull request run (caller ${{ inputs.caller }}; branch ${{ inputs.branch }})

on:
workflow_dispatch:
inputs:
branch:
description: "The branch of this repository to test"
type: string
required: true
caller:
description: "The run that asked for this one, for its title"
type: string
required: true

# Nothing here reads the repository. Uploading files uses the per-job runtime
# token GitHub issues to every job regardless of these permissions.
permissions: {}

# One at a time per branch: a run that has started is never cancelled for a
# newer one, but GitHub still cancels an older run that is only pending when a
# newer one joins the group, and the gate reports that run as having left no
# report. The queue side limits how many go at once.
concurrency:
group: queue-pr-run-${{ inputs.branch }}
cancel-in-progress: false

jobs:
run:
name: Run and publish
runs-on: [queue-report]
# A lane waits at most 45 min to be started, then runs for at most 75 min
# counted from its start. Finishing and publishing the run can take 30 min
# more, and 20 min are margin for the runner's own delays: 170 min.
timeout-minutes: 170
steps:
# Anyone who can dispatch names the branch, not only the gate job, so it is
# checked here, on main, and not only in the gate: a plain branch name that
# cannot start with a dash.
- name: Check the branch name
env:
BRANCH: ${{ inputs.branch }}
# One line: the workflow contract forbids backslashes in this file.
run: if [[ "${BRANCH}" =~ ^[A-Za-z0-9_][A-Za-z0-9._/-]{0,199}$ ]]; then exit 0; fi; echo "::error::not a branch name"; exit 1

- name: Run on the queue and publish
env:
BRANCH: ${{ inputs.branch }}
# One line: the workflow contract forbids backslashes in this file.
run: /opt/queue/bin/queue-publish --out "${RUNNER_TEMP}/results" --start "${BRANCH}" --kind pr

- name: Upload lane results
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: queue-pr-run-results
path: ${{ runner.temp }}/results
# The publisher's files only; a lane's hidden files are not published.
include-hidden-files: false
if-no-files-found: warn
4 changes: 2 additions & 2 deletions .github/workflows/queue-run.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
# prints the run's report and lane matrix, uploads the lanes' files, and passes
# or fails with the run's verdict. It needs no credential and no permission:
# the runner asks from where it stands. The runner is set up to run only this
# workflow and nightly-queue.yml as they are on main; that is enforced on the
# runner, outside this repository.
# workflow, nightly-queue.yml and queue-pr-run.yml as they are on main; that is
# enforced on the runner, outside this repository.
# What that protects is in docs/nightly-on-the-queue.md.
name: E2E on the queue

Expand Down
88 changes: 88 additions & 0 deletions .github/workflows/vllm-route-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: vLLM route check

# Advisory, and deliberately NOT a required check. Resolves the SDK an unpinned
# `rocm install sdk` picks on the release and the nightly channel, and warns
# when the vLLM engine has no install route for it (see
# scripts/vllm_route_check.py). It reads the live indexes, so it can warn on a
# PR that changed nothing: an index moved on to an SDK vLLM has no wheels for.
# Required, that would block unrelated merges, so a missing route only reports.
#
# Runs on a PR only when it touches what decides the answer: the resolver that
# picks the default SDK (therock.rs and therock/), the engine's two build
# tables, or this check. CPU only: a dry run, no install.
#
# A finding is reported as a warning plus a job summary and the job stays green
# (the step swallows exit code 1), so a known gap does not leave a permanent red
# X on every PR touching these paths. The build, the self-test and a check that
# cannot run (exit 2) still fail it.

on:
pull_request:
paths:
- apps/rocm/src/therock.rs
- apps/rocm/src/therock/**
- engines/vllm/src/wheel_builds.rs
- engines/vllm/src/discover_builds.rs
- scripts/vllm_route_check.py
- .github/workflows/vllm-route-check.yml
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CARGO_TERM_COLOR: always
RUSTFLAGS: -D warnings
AWS_LC_SYS_USE_SYSTEM: "0"

jobs:
vllm-route:
name: vLLM route for the default SDKs (advisory)
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions-rust-lang/setup-rust-toolchain@a188e68d0fd2bd36ad51dccb2b07ba6e75105f30 # v2.0.1
with:
cache-key: ubuntu-24.04

- name: Build rocm
run: cargo build --release -p rocm

- name: Self-test
run: python3 scripts/vllm_route_check.py --self-test

- name: Check the vLLM route of the default SDKs
id: route
# Exit 1 = a default SDK has no route: reported below, job stays green.
# Any other non-zero exit (2 = the check itself could not run, for
# example the build tables no longer parse, or `rocm install sdk
# --dry-run` timed out, failed or printed no plan) fails the job.
run: |
rc=0
python3 scripts/vllm_route_check.py --rocm target/release/rocm \
> "$RUNNER_TEMP/route.txt" 2>&1 || rc=$?
cat "$RUNNER_TEMP/route.txt"
echo "rc=$rc" >> "$GITHUB_OUTPUT"
if [ "$rc" -gt 1 ]; then exit "$rc"; fi

- name: Report a missing route
if: steps.route.outputs.rc == '1'
run: |
echo "::warning title=vLLM route::A default SDK has no vLLM install route. See the job summary."
{
echo "### vLLM route check: no route for a default SDK"
echo
echo "Advisory: add the discovery row once the wheels are published."
echo
echo '```'
cat "$RUNNER_TEMP/route.txt"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ repos:
args: [--fix=lf] # repo standardises on LF endings (see .gitattributes)
exclude: '^THIRD_PARTY_NOTICES\.txt$'

# Python: lint + format the helper scripts and engine workers.
# Python: lint + format every Python file in the repository.
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.17.3
hooks:
Expand Down
Loading
Loading