Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
c1d5dd8
ci: run pull requests on one Linux and one Windows GPU node
tomastola Nov 10, 2026
bfd8ce8
docs: say which per-PR lanes skip pull_request
tomastola Nov 11, 2026
6409402
docs: release-candidate gate runs the full self-hosted matrix
tomastola Nov 11, 2026
6235d9f
ci: pin the whole per-PR gate; guard the nightly Strix job; fix stale…
tomastola Nov 11, 2026
c88d773
ci: tighten the nightly Strix path guard; fix remaining per-PR Strix/…
tomastola Nov 11, 2026
9eb2799
ci: drop W7900 from merge_group; fix the last per-PR Strix claims
tomastola Nov 11, 2026
d68476e
ci: drop e2e-gpu-strix-ubuntu from merge_group
tomastola Nov 11, 2026
e7a3ce0
docs, ci: make every merge_group claim match the workflow
tomastola Nov 11, 2026
66d2bc3
docs, tests: fix stale merge-queue and per-PR claims, re-wrap two lines
tomastola Nov 11, 2026
0b2d5dc
ci: pin the PR-skipping lanes in #1032's closed-PR if table
tomastola Nov 12, 2026
bfde201
docs, ci: drop the stale Docker 'unconfirmed' and per-PR budget claims
tomastola Nov 12, 2026
3158466
tests: say Docker is reachable on the nightly Strix pool, and serve-2…
tomastola Nov 12, 2026
b7d8f46
tests: declare serve-aim-lifecycle-parity an expected failure on Stri…
tomastola Nov 12, 2026
b2f23df
docs, ci: reference rocm-cli#1058 instead of pinning a run ID and fai…
tomastola Nov 12, 2026
c91e228
tests, docs: keep WSL2 out of the serve-27 xfail row; fix two stale c…
tomastola Nov 12, 2026
ee855d7
ci, docs: pin that e2e-wsl and Windows Strix set E2E_MERGE_QUEUE; say…
tomastola Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ name: E2E self-hosted
# THIS workflow's supersession, never the required checks in ci.yml. Only
# pull_request runs are superseded (see `concurrency:` below).
#
# Per-PR scope: a pull request runs exactly one Linux GPU node (`e2e-gpu`,
# MI300X: the only lane where vLLM is the effective engine) and one Windows GPU
# pool (the Strix Halo pool: `e2e-gpu-strix-windows` and `e2e-wsl`). The Strix
# Ubuntu and W7900 lanes keep their push / dispatch triggers and run nightly
# (nightly.yml); neither starts on pull_request or merge_group.
#
# These lanes are non-blocking because none of their check names are in
# branch protection's required-status-check list (confirmed 2026-09-11) — not
# because of `continue-on-error`, which is `false` on every lane but
Expand Down Expand Up @@ -110,7 +116,8 @@ jobs:
outputs:
heavy: ${{ steps.filter.outputs.heavy || steps.all.outputs.forced }}
# Narrow GPU-serve gate; forced true off-PR so the merge queue runs the
# full matrix (every lane but MI300X, which skips merge_group) and its
# full matrix (every lane but MI300X, Strix Ubuntu and W7900, which skip
# merge_group) and its
# required checks are never starved.
serve: ${{ steps.filter.outputs.serve || steps.all.outputs.forced }}
# ROCMAI-430: the three most recent rows of the official release list
Expand Down Expand Up @@ -465,16 +472,24 @@ jobs:
# ratio the 35 was chosen for.
# An include_nightly dispatch also runs the AIM serve scenarios (serve-26/27,
# @nightly, up to 1200s readiness each on a cold pull) and the large-model
# serve, which cannot fit the per-PR 45min budget.
# serve, which cannot fit the 45min budget of an ordinary push run.
timeout-minutes: ${{ inputs.include_nightly && 105 || 45 }}
runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]
needs: [changes]
# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-ubuntu.
# Not on pull_request: a PR gets exactly one Linux GPU node (MI300X, the only
# lane where vLLM runs) and one Windows pool (Strix Halo). Not on
# merge_group either: over 7 days 35% of its merge-queue jobs (98 of 281) were
# cancelled at the 45 min cap, and the queue does not wait for this
# non-required check. Still runs on push (main, release/**), dispatch and
# nightly (nightly.yml, e2e-gpu-nightly-strix, same pool labels).
if: >-
always()
&& needs.changes.result == 'success'
&& (
(github.event_name != 'workflow_dispatch'
&& github.event_name != 'pull_request'
Comment thread
tomastola marked this conversation as resolved.
Resolved
&& github.event_name != 'merge_group'
&& needs.changes.outputs.serve == 'true')
|| (github.event_name == 'workflow_dispatch'
&& (inputs.platform == 'all' || inputs.platform == 'strix-ubuntu'))
Expand All @@ -486,7 +501,7 @@ jobs:
channel: [release, nightly]
# See `e2e-gpu` (ROCMAI-430): layered alongside the pre-existing
# `channel` dimension above, not replacing it — this lane is the only
# per-PR job with two independent matrix axes.
# job with two independent matrix axes.
sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }}
# The nightly channel only carries the latest (pre-)release builds, so
# it cannot resolve the n-1/n-2 released versions the matrix pins
Expand All @@ -508,8 +523,6 @@ jobs:
# Match the MI300X dispatch path: opt into the platform-adaptive large-model
# scenario only when the manual include_nightly input is enabled.
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Heavy @merge-queue serves run only in the merge queue.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down Expand Up @@ -1599,11 +1612,18 @@ jobs:
fail-fast: false
matrix:
sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }}
# Not on pull_request or merge_group: one runner on borrowed quota, 58-70% of

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"58-70% of its PR jobs never started" does not match the PR table (W7900: 202 of 507 jobs never started, about 40%); the PR body also says 70%. Please correct the figure here or state the denominator.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the 58-70% figure does not match the PR table. I will correct it in the workflow comment and the docs in a small follow-up PR from main once this merges, rather than re-spin the queue entry.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I will not correct the W7900 figure. The W7900 lane is being removed entirely by a separate PR (#1063's work), so this comment and the matching docs text go away with it; a number fix here would be churn. The tracker-ID point is a separate small PR: #1065.

# its PR jobs never started before being superseded, and a merge_group run
# would queue behind it the same way (the queue does not wait for this
# check). Still runs on push (main, release/**), dispatch and nightly
# (nightly.yml, e2e-gpu-nightly-w7900).
if: >-
always()
&& needs.changes.result == 'success'
&& (
(github.event_name != 'workflow_dispatch'
&& github.event_name != 'pull_request'
&& github.event_name != 'merge_group'
&& needs.changes.outputs.serve == 'true')
|| (github.event_name == 'workflow_dispatch'
&& (inputs.platform == 'all' || inputs.platform == 'w7900'))
Expand All @@ -1616,8 +1636,6 @@ jobs:
env:
E2E_SERVE_TIMEOUT_SECS: "300"
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Full serve matrix at the merge-queue gate.
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
Expand Down
47 changes: 20 additions & 27 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -671,10 +671,11 @@ jobs:
name: e2e-gpu-mi350p-${{ matrix.channel }}-report
path: tests/e2e-cucumber/results/

# W7900 counterpart to e2e-gpu-nightly. Mirrors the per-PR e2e-gpu-w7900 lane
# in e2e-selfhosted.yml (w7900 label, shared uv cache, GPU preflight), with
# @nightly scenarios enabled. Keep the two in step: the report crate asserts
# the nightly lanes publish the same platforms as the per-PR lanes.
# W7900 counterpart to e2e-gpu-nightly. Mirrors the e2e-gpu-w7900 lane (which
# no longer runs on pull_request) in e2e-selfhosted.yml (w7900 label, shared
# uv cache, GPU preflight), with @nightly scenarios enabled. Keep the two in
# step: the report crate asserts the nightly lanes publish the same platforms
# as the lanes in e2e-selfhosted.yml.
e2e-gpu-nightly-w7900:
name: E2E tests (W7900, incl. nightly-only)
timeout-minutes: 90
Expand Down Expand Up @@ -762,29 +763,24 @@ jobs:
# Strix Ubuntu job.
e2e-gpu-nightly-strix:
name: E2E tests (Strix Halo, incl. nightly-only)
# 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits when
# this host has Docker; on a host without it they skip and cost nothing.
# Whether it has Docker is unconfirmed: nothing in this job installs or checks
# it, so the "Report whether Docker is reachable" step logs the answer. If it
# does not, no nightly lane runs AIM serve and a pool-based one is needed.
# 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits; on a
Comment thread
tomastola marked this conversation as resolved.
Resolved
# host without Docker they skip and cost nothing. The "Report whether Docker
# is reachable" step logs the answer each run. serve-27's known failure on
# this pool is an xfail row in expectations.toml (rocm-cli#1058).
timeout-minutes: 120
# `native` disambiguates this runner from any host that might carry a bare
# `wsl` label alongside `strix-halo` (the static WSL runner this repo used
# before the WSL lanes moved to the ephemeral pool) -- the paths below
# exist only on the native host, whether or not that other runner remains
# registered.
runs-on: [self-hosted, linux, strix-halo, native]
# The DevLab Dispatch pool, identical to the e2e-gpu-strix-ubuntu lane in
# e2e-selfhosted.yml, which runs on push and dispatch but not on pull_request
# or merge_group (pinned by workflow_contract.rs). This lane used to target
# the static `native` host, which is offline: every nightly sat queued until
# the 12h run cap and the lane never ran. Pool hosts are cold,
# so there are no host-specific HOME or cache-dir overrides.
runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]
Comment thread
tomastola marked this conversation as resolved.
Resolved

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nightly Strix lane now runs on the same pool whose per-PR twin hangs to its cap in about 19% of jobs, and Docker on pool hosts is still unconfirmed (the step only warns). Please do a workflow_dispatch run of this lane before relying on it, so a hang does not show up as another silent nightly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not dispatching nightly.yml from this branch: it builds and publishes nightly release assets, so a dispatch from a PR branch would touch the shared release tag. What I can show is run 43025895270 of e2e-selfhosted.yml on the same pool (strix-ubuntu, include_nightly=true): Docker was reachable, serve-26 passed on both channel legs, serve-27 fails at stopping the AIM service (no xfail row yet), and nothing hung. The hang risk you name (about 19% of the per-PR twin's jobs) is bounded by the job's timeout-minutes: 120, and the job is continue-on-error, so a hang cannot block the nightly. The first scheduled nightly after merge is the end-to-end check, and I will workflow_dispatch nightly on main right after merge if you want it sooner. Leaving this thread open until that run exists.

continue-on-error: true
strategy:
fail-fast: false
matrix:
channel: [release, nightly]
env:
Comment thread
tomastola marked this conversation as resolved.
Resolved
HOME: /home/ubuntu/actions-runner/e2e-home
CARGO_HOME: /home/ubuntu/actions-runner/.cargo
RUSTUP_HOME: /home/ubuntu/actions-runner/.rustup
TMPDIR: /home/ubuntu/actions-runner/tmp
PIP_CACHE_DIR: /home/ubuntu/actions-runner/pip-cache
E2E_INCLUDE_NIGHTLY: "1"
E2E_SERVE_TIMEOUT_SECS: "300"
E2E_TUI_TIMEOUT_SECS: "90"
Expand All @@ -793,11 +789,8 @@ jobs:
with:
persist-credentials: false

- name: Prepare writable dirs on the nvme + reclaim GPU from stray E2E procs
run: |
mkdir -p "$HOME" "$TMPDIR" "$PIP_CACHE_DIR"
echo "prepared"
bash scripts/reclaim-gpu.sh
- name: Reclaim GPU from stray E2E processes
run: bash scripts/reclaim-gpu.sh

# Bounded wait for a free GPU, then an assertion that no leaked E2E engine
# process survives. gfx1151 shares system memory, so the floor is smaller
Expand All @@ -815,11 +808,11 @@ jobs:

- name: Ensure Rust toolchain
run: |
if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$CARGO_HOME/bin/cargo" ]; then
if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$HOME/.cargo/bin/cargo" ]; then
curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \
| sh -s -- -y --no-modify-path --default-toolchain none
fi
echo "$CARGO_HOME/bin" >> "$GITHUB_PATH"
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"

# serve-26/27 (AIM) need Docker reachable on this host; without it they skip
# silently. Say which it is, so a nightly that never ran them is visible.
Expand Down
Loading
Loading