Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
174 changes: 158 additions & 16 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ env:
# with no timeout — holds a runner for six hours and, in the merge queue, stalls
# every queued PR behind it with no signal that anything is wrong. The values here
# are deliberately loose (roughly 4x the observed p100 for most jobs, which is
# under 3 min; the two Windows jobs are the exception, since they run for many
# minutes on a cold pull-request cache and keep a 45 min limit) so they never fire
# under 3 min; the two Windows build jobs are the exception, since they run for
# many minutes on a cold pull-request cache and keep a 45 min limit, and so is the
# hosted Windows E2E job, which also runs the suite and keeps 60) so they never fire
# on a merely slow run; they exist to convert a silent multi-hour stall into a
# prompt, legible failure. Raise a value if a job legitimately grows into it.
jobs:
Expand Down Expand Up @@ -1000,8 +1001,8 @@ jobs:
# every run, on a lane that determines when CI goes green.
#
# Deliberately NOT built with `rocm/e2e-test-hooks`, unlike the prebuilt
# lanes in e2e-selfhosted.yml / nightly.yml. Those run the full suite,
# whose scripted failure seams are compiled out without the feature.
# lanes in e2e-selfhosted.yml / nightly.yml. Those run scenarios whose
# scripted failure seams are compiled out without the feature.
# `E2E_ONLY_LIFECYCLE` restricts this lane to @lifecycle scenarios, none
# of which touch a seam — and this lane packages and installs the binary
# through the real installer, so it should ship exactly what a release
Expand All @@ -1016,8 +1017,8 @@ jobs:
# (e2e-selfhosted.yml) is optional and not a required check. They are not
# tagged `@lifecycle` themselves — they are not release packaging/
# install/uninstall scenarios, and tagging them so would drop them out of
# nightly.yml's and e2e-selfhosted.yml's full-suite runs, neither of
# which sets `E2E_INCLUDE_LIFECYCLE`.
# nightly.yml's and e2e-selfhosted.yml's runs, neither of which sets
# `E2E_INCLUDE_LIFECYCLE`.
#
# serve-33/serve-34/serve-35 (the native-Windows vLLM refusal for `serve` and
# `engines install`) are folded in the same way: `@requires-os:windows`,
Expand Down Expand Up @@ -1990,10 +1991,9 @@ jobs:
# e2e-selfhosted.yml: the artifact-name contract in `xtask::e2e_report`
# reads `name: e2e-*` lines literally. Artifact names are unique per run.
# The 24.04 leg keeps `e2e-report`, which the consolidated report below
# picks up through its `*-report` glob. The 26.04 leg's artifact is named
# outside both that glob and the `e2e-` namespace: the report has one
# mock platform, and two inputs labelled as it would collide. It is
# uploaded for inspection only.
# downloads by that exact name. The 26.04 leg's artifact is named outside
# the `e2e-` namespace: the report has one mock Linux platform, and two
# inputs labelled as it would collide. It is uploaded for inspection only.
- name: Upload E2E report
if: always() && steps.gate.outputs.run == 'true' && matrix.os == 'ubuntu-24.04'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand All @@ -2008,15 +2008,157 @@ jobs:
name: ubuntu-26.04-e2e-results
path: tests/e2e-cucumber/results/

# The same suite on a GitHub-hosted Windows runner. The self-hosted GPU lanes
# (e2e-selfhosted.yml) run only the scenarios that need their GPU
# (E2E_GPU_ONLY), so this is where everything else meets native Windows —
# path handling, the Windows probes, the PowerShell surfaces. Built with
# `rocm/e2e-test-hooks` by `cargo xtask e2e`, like the Linux lane above and
# unlike windows-build-and-test's lifecycle run, which must ship exactly what
# a release ships. Scenarios on a simulated machine are `@requires-os:linux`
# and skip here: the Windows probes ask the registry, not a filesystem.
e2e-windows:
name: E2E tests (Windows)
timeout-minutes: 60
runs-on: windows-latest
# Only `changes`: unlike the Linux `e2e` job this does not wait for the
# Linux `build-and-test`, which tells it nothing about the Windows build and
# would put that job's duration in front of this one's.
needs: [changes]
# Same compile cache as windows-build-and-test (see build-and-test for the
# design: Azure backend, written only by push and merge_group), under its
# own prefix — this build carries `rocm/e2e-test-hooks`, so its entries
# would never match that job's anyway.
env:
CARGO_INCREMENTAL: "0"
RUSTC_WRAPPER: sccache
SCCACHE_AZURE_CONNECTION_STRING: ${{ secrets.CACHE_AZURE_CONNECTION_STRING }}
SCCACHE_AZURE_BLOB_CONTAINER: cache
SCCACHE_AZURE_KEY_PREFIX: rocm-cli/e2e-windows
SCCACHE_AZURE_RW_MODE: ${{ (github.event_name == 'push' || github.event_name == 'merge_group') && 'READ_WRITE' || 'READ_ONLY' }}
if: >-
always()
&& needs.changes.result == 'success'
&& (github.event_name != 'workflow_dispatch'
|| inputs.platform == 'all' || inputs.platform == 'mock')
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

# Whether the real work runs this time: on a dispatch, or when the change
# is heavy (`merge_group` forces heavy). The job itself always runs, so the
# check is always reported — required checks that never report stall the
# merge queue.
- name: Decide whether to run
id: gate
shell: bash
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ] \
|| [ "${{ needs.changes.outputs.heavy }}" = "true" ]; then
echo "run=true" >> "$GITHUB_OUTPUT"
else
echo "run=false" >> "$GITHUB_OUTPUT"
fi

# Installs sccache. Runs BEFORE setup-rust-toolchain, whose rust-cache step
# runs `cargo metadata` and honours RUSTC_WRAPPER. `version` is pinned so
# the compiler wrapper does not float with the latest sccache release.
# continue-on-error for the same reason as build-and-test: a failed install
# is handled by the probe, not by failing the job.
- name: Set up sccache
if: steps.gate.outputs.run == 'true'
continue-on-error: true
uses: mozilla-actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11
with:
version: v0.17.0

# Keep the remote backend only if it answers — see the equivalent step in
# build-and-test for why this is a probe, and why failure clears the Azure
# variables rather than the wrapper.
- name: Probe sccache cache backend
id: sccache_probe
if: steps.gate.outputs.run == 'true'
shell: pwsh
run: |
# This script decides whether to use the cache; a non-zero status from
# the commands it runs is an answer, not a job failure. Pin the native
# error preference off so a future runner default cannot turn a failed
# probe into a failed step.
$PSNativeCommandUseErrorActionPreference = $false
# Every Out-File below passes -Encoding utf8 explicitly. PowerShell 7
# (`shell: pwsh`) already defaults to UTF-8, but Windows PowerShell 5.1
# defaults to UTF-16LE, which the runner cannot parse — so an edit that
# switched this step to `shell: powershell` would silently corrupt the
# environment file rather than fail visibly.
function Disable-Backend {
"SCCACHE_AZURE_CONNECTION_STRING=" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
"SCCACHE_AZURE_BLOB_CONTAINER=" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
}
if (-not (Get-Command sccache -ErrorAction SilentlyContinue)) {
Write-Output "::warning::sccache is not installed; compiling without it."
"RUSTC_WRAPPER=" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
# Nothing will read the credential now; don't leave it in the job env.
Disable-Backend
exit 0
}
# sccache is the compiler wrapper from here on, remote backend or not,
# so there are statistics worth reporting either way.
"active=true" | Out-File -FilePath $env:GITHUB_OUTPUT -Append -Encoding utf8
if (-not $env:SCCACHE_AZURE_CONNECTION_STRING) {
Write-Output "::notice::No cache credential available (expected on forks); compiling without a shared cache."
Disable-Backend
exit 0
}
sccache --start-server
if ($LASTEXITCODE -ne 0) {
Write-Output "::warning::sccache could not reach the cache backend; compiling without a shared cache."
# Leave no half-started server behind for the cargo steps to find.
sccache --stop-server *> $null
Disable-Backend
}
exit 0

- uses: actions-rust-lang/setup-rust-toolchain@a188e68d0fd2bd36ad51dccb2b07ba6e75105f30 # v2.0.1
if: steps.gate.outputs.run == 'true'
with:
cache-on-failure: true

- name: Run E2E tests
if: steps.gate.outputs.run == 'true'
shell: pwsh
run: cargo xtask e2e

- name: sccache stats
if: always() && steps.sccache_probe.outputs.active == 'true'
shell: pwsh
run: sccache --show-stats

# Its own grid in the run Summary; see e2e-report for why the consolidated
# (required) report does not wait for this lane.
- name: Report summary
if: always() && steps.gate.outputs.run == 'true'
shell: pwsh
run: cargo xtask e2e-report --artifacts-dir tests/e2e-cucumber/results --html-out tests/e2e-cucumber/results/consolidated.html >> $env:GITHUB_STEP_SUMMARY

- name: Upload E2E report
if: always() && steps.gate.outputs.run == 'true'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: e2e-windows-report
path: tests/e2e-cucumber/results/

# Consolidate the mock (GitHub-hosted) platform's report into the cross-platform
# report: a platform × tier matrix in the run Summary plus a single merged HTML
# artifact. `if: always()` so a failing platform still appears; runs on
# GitHub-hosted ubuntu (no GPU needed — it only parses report.json).
#
# The self-hosted GPU platforms are consolidated by e2e-selfhosted.yml's own
# report job (they moved there so an offline runner can't stall this workflow's
# required checks). This job only depends on the mock `e2e` lane; the `*-report`
# glob still auto-includes any new GitHub-hosted platform added here.
# required checks). This job only depends on the mock `e2e` lane, and downloads
# only its artifact. `e2e-windows` summarizes itself instead: waiting on it here
# would put a cold Windows release build in front of this REQUIRED check, and
# matching it by glob would include it or not depending on which job finished
# first.
e2e-report:
name: E2E consolidated report
runs-on: ubuntu-24.04 # one LTS: only merges report.json files
Expand All @@ -2043,9 +2185,9 @@ jobs:
# built against one Ubuntu release's glibc is never restored on another.
cache-key: ubuntu-24.04

# Pull every e2e artifact matching the glob. `xtask e2e-report` turns each
# into one labeled platform and the `*-report` glob makes new platforms
# appear automatically. Layout note: download-artifact@v8 extracts a match
# Pull the mock lane's artifact (named exactly; see the job comment for why
# not a glob). `xtask e2e-report` turns it into one labeled platform.
# Layout note: download-artifact@v8 extracts a match
# into e2e-artifacts/<artifact-name>/ when several match, but flattens
# straight into e2e-artifacts/ when EXACTLY ONE matches (its source picks
# the root path when `artifacts.length === 1`). After the ci.yml ⇄
Expand All @@ -2055,7 +2197,7 @@ jobs:
- name: Download all E2E reports
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: '*-report'
pattern: 'e2e-report'
path: e2e-artifacts

- name: Build consolidated report + step summary
Expand Down
30 changes: 26 additions & 4 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,9 +264,22 @@ jobs:
# workflow sets this unconditionally; here it lets a scoped dispatch confirm
# a single nightly scenario (e.g. the 27B serve) without the full nightly run.
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# On a pull request and in the merge queue (which this lane skips; the
# lanes that run there share this expression), of the scenarios that need
# a real GPU only the `@gpu-smoke` canaries (and, in the queue, the
# @merge-queue serves) run. Every GPU scenario that only needs the CLI to
# see a GPU runs on a simulated machine on the hosted `E2E tests` lane. Off
# those events the narrowing does not apply (the @nightly and @merge-queue
# gates still do). Keeps the GPU lanes, which set merge latency, short.
E2E_GPU_SMOKE_ONLY: "${{ (github.event_name == 'merge_group' || github.event_name == 'pull_request') && '1' || '' }}"
# A real GPU lane: run the `@requires-real-gpu` scenarios that a simulated
# host cannot stand in for. See HardwareMode in src/expectation.rs.
E2E_HARDWARE: real
# ...and ONLY those, on every event: everything that needs no GPU runs on
# the GitHub-hosted Linux and Windows lanes in ci.yml (`E2E tests`,
# `E2E tests (Windows)`), so a GPU runner's time is not spent on it. See
# restrict_to_lane in src/expectation.rs.
E2E_GPU_ONLY: "1"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down Expand Up @@ -529,6 +542,7 @@ jobs:
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# See the e2e-gpu lane.
E2E_HARDWARE: real
E2E_GPU_ONLY: "1"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down Expand Up @@ -688,8 +702,11 @@ jobs:
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Heavy @merge-queue serves run only in the merge queue.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
# GPU smoke test only on pull requests and in the merge queue; see the e2e-gpu lane.
E2E_GPU_SMOKE_ONLY: "${{ (github.event_name == 'merge_group' || github.event_name == 'pull_request') && '1' || '' }}"
# See the e2e-gpu lane.
E2E_HARDWARE: real
E2E_GPU_ONLY: "1"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down Expand Up @@ -993,9 +1010,12 @@ jobs:
# WSL2 works on these hosts; pin `--id Microsoft.WSL` in the install step so
# a failed install can't report green; expect ~5min/job for distro install
# since distros are unregistered between jobs by design; static TA runners
# are not needed for WSL coverage. Same suite as every other platform —
# scenarios the host cannot satisfy resolve to skip from the capability
# probe, so nothing is filtered out here.
# are not needed for WSL coverage. Unlike the GPU lanes, it sets no
# E2E_GPU_ONLY: scenarios the host cannot satisfy resolve to skip from the
# capability probe, the usual tag gates (@nightly, @merge-queue, @lifecycle,
# @requires-docker) apply as on every lane, and on a pull request and in the
# merge queue its real-GPU scenarios narrow to the smoke set
# (E2E_GPU_SMOKE_ONLY). Nothing is filtered out by hand.
e2e-wsl:
name: E2E tests (Strix Halo, WSL2, ${{ matrix.sdk_version }})
# 120min: like the native pool lanes, every job here is cold (fresh distro,
Expand Down Expand Up @@ -1043,6 +1063,8 @@ jobs:
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Heavy @merge-queue serves run only in the merge queue.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
# GPU smoke test only on pull requests and in the merge queue; see the e2e-gpu lane.
E2E_GPU_SMOKE_ONLY: "${{ (github.event_name == 'merge_group' || github.event_name == 'pull_request') && '1' || '' }}"
# See the e2e-gpu lane.
E2E_HARDWARE: real
# apps/rocm/build.rs falls back to GitHub's own GITHUB_HEAD_REF/
Expand Down Expand Up @@ -1490,7 +1512,7 @@ jobs:
# Forward the job-level E2E_* env vars into the guest: `wsl.exe`
# does not share the calling process's environment unless a
# variable is named in WSLENV.
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:E2E_HARDWARE:ROCM_CLI_RELEASE_REF:MATRIX_SDK_VERSION:GITHUB_EVENT_INPUTS_NAME_FILTER'
$env:WSLENV = 'E2E_SERVE_TIMEOUT_SECS:E2E_TUI_TIMEOUT_SECS:E2E_INCLUDE_NIGHTLY:E2E_MERGE_QUEUE:E2E_GPU_SMOKE_ONLY:E2E_HARDWARE:ROCM_CLI_RELEASE_REF:MATRIX_SDK_VERSION:GITHUB_EVENT_INPUTS_NAME_FILTER'
& "$env:GITHUB_WORKSPACE\.github\scripts\Invoke-WslBash.ps1" -PipeFail -Script @'
export PATH="$HOME/.cargo/bin:$PATH"
cd /root/work/rocm-cli
Expand Down
9 changes: 9 additions & 0 deletions crates/e2e-report/src/consolidated/descriptor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,13 @@ const KNOWN_PLATFORMS: &[KnownPlatform] = &[
os: "Linux",
slugs: &["mock"],
},
// The GitHub-hosted Windows lane: no GPU, native Windows.
KnownPlatform {
artifact: "e2e-windows-report",
platform: "Mock",
os: "Windows",
slugs: &["mock-windows"],
},
KnownPlatform {
artifact: "e2e-gpu-report",
platform: "MI300X",
Expand Down Expand Up @@ -275,6 +282,7 @@ mod tests {
fn parse_descriptor_maps_known_artifacts() {
for (name, platform, os) in [
("e2e-report", "Mock", "Linux"),
("e2e-windows-report", "Mock", "Windows"),
("e2e-gpu-report", "MI300X", "Linux"),
("e2e-gpu-rad3-report", "R9700", "Linux"),
("e2e-gpu-mi350p-report", "MI350P", "Linux"),
Expand Down Expand Up @@ -404,6 +412,7 @@ mod tests {
// values the harness writes and the artifact names the workflows upload.
for (slug, artifact) in [
("mock", "e2e-report"),
("mock-windows", "e2e-windows-report"),
("mi300x", "e2e-gpu-report"),
("gfx1201", "e2e-gpu-rad3-report"),
("mi350p", "e2e-gpu-mi350p-report"),
Expand Down
Loading
Loading