Repository navigation
ci: run pull requests on one Linux GPU node and one Windows GPU pool; merge queue drops its native Linux GPU lanes #1018
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c1d5dd8
bfd8ce8
6409402
6235d9f
c88d773
9eb2799
d68476e
e7a3ce0
66d2bc3
0b2d5dc
bfde201
3158466
b7d8f46
b2f23df
c91e228
ee855d7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,6 +9,12 @@ name: E2E self-hosted | |
| # THIS workflow's supersession, never the required checks in ci.yml. Only | ||
| # pull_request runs are superseded (see `concurrency:` below). | ||
| # | ||
| # Per-PR scope: a pull request runs exactly one Linux GPU node (`e2e-gpu`, | ||
| # MI300X: the only lane where vLLM is the effective engine) and one Windows GPU | ||
| # pool (the Strix Halo pool: `e2e-gpu-strix-windows` and `e2e-wsl`). The Strix | ||
| # Ubuntu and W7900 lanes keep their push / dispatch triggers and run nightly | ||
| # (nightly.yml); neither starts on pull_request or merge_group. | ||
| # | ||
| # These lanes are non-blocking because none of their check names are in | ||
| # branch protection's required-status-check list (confirmed 2026-09-11) — not | ||
| # because of `continue-on-error`, which is `false` on every lane but | ||
|
|
@@ -110,7 +116,8 @@ jobs: | |
| outputs: | ||
| heavy: ${{ steps.filter.outputs.heavy || steps.all.outputs.forced }} | ||
| # Narrow GPU-serve gate; forced true off-PR so the merge queue runs the | ||
| # full matrix (every lane but MI300X, which skips merge_group) and its | ||
| # full matrix (every lane but MI300X, Strix Ubuntu and W7900, which skip | ||
| # merge_group) and its | ||
| # required checks are never starved. | ||
| serve: ${{ steps.filter.outputs.serve || steps.all.outputs.forced }} | ||
| # ROCMAI-430: the three most recent rows of the official release list | ||
|
|
@@ -465,16 +472,24 @@ jobs: | |
| # ratio the 35 was chosen for. | ||
| # An include_nightly dispatch also runs the AIM serve scenarios (serve-26/27, | ||
| # @nightly, up to 1200s readiness each on a cold pull) and the large-model | ||
| # serve, which cannot fit the per-PR 45min budget. | ||
| # serve, which cannot fit the 45min budget of an ordinary push run. | ||
| timeout-minutes: ${{ inputs.include_nightly && 105 || 45 }} | ||
| runs-on: [self-hosted, linux, devlab-dispatch, strix-halo] | ||
| needs: [changes] | ||
| # See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-ubuntu. | ||
| # Not on pull_request: a PR gets exactly one Linux GPU node (MI300X, the only | ||
| # lane where vLLM runs) and one Windows pool (Strix Halo). Not on | ||
| # merge_group either: over 7 days 35% of its merge-queue jobs (98 of 281) were | ||
| # cancelled at the 45 min cap, and the queue does not wait for this | ||
| # non-required check. Still runs on push (main, release/**), dispatch and | ||
| # nightly (nightly.yml, e2e-gpu-nightly-strix, same pool labels). | ||
| if: >- | ||
| always() | ||
| && needs.changes.result == 'success' | ||
| && ( | ||
| (github.event_name != 'workflow_dispatch' | ||
| && github.event_name != 'pull_request' | ||
| && github.event_name != 'merge_group' | ||
| && needs.changes.outputs.serve == 'true') | ||
| || (github.event_name == 'workflow_dispatch' | ||
| && (inputs.platform == 'all' || inputs.platform == 'strix-ubuntu')) | ||
|
|
@@ -486,7 +501,7 @@ jobs: | |
| channel: [release, nightly] | ||
| # See `e2e-gpu` (ROCMAI-430): layered alongside the pre-existing | ||
| # `channel` dimension above, not replacing it — this lane is the only | ||
| # per-PR job with two independent matrix axes. | ||
| # job with two independent matrix axes. | ||
| sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }} | ||
| # The nightly channel only carries the latest (pre-)release builds, so | ||
| # it cannot resolve the n-1/n-2 released versions the matrix pins | ||
|
|
@@ -508,8 +523,6 @@ jobs: | |
| # Match the MI300X dispatch path: opt into the platform-adaptive large-model | ||
| # scenario only when the manual include_nightly input is enabled. | ||
| E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}" | ||
| # Heavy @merge-queue serves run only in the merge queue. | ||
| E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}" | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
|
|
@@ -1599,11 +1612,18 @@ jobs: | |
| fail-fast: false | ||
| matrix: | ||
| sdk_version: ${{ fromJSON(needs.changes.result == 'skipped' && '["current"]' || needs.changes.outputs.sdk_version_matrix) }} | ||
| # Not on pull_request or merge_group: one runner on borrowed quota, 58-70% of | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "58-70% of its PR jobs never started" does not match the PR table (W7900: 202 of 507 jobs never started, about 40%); the PR body also says 70%. Please correct the figure here or state the denominator.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed, the 58-70% figure does not match the PR table. I will correct it in the workflow comment and the docs in a small follow-up PR from main once this merges, rather than re-spin the queue entry.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| # its PR jobs never started before being superseded, and a merge_group run | ||
| # would queue behind it the same way (the queue does not wait for this | ||
| # check). Still runs on push (main, release/**), dispatch and nightly | ||
| # (nightly.yml, e2e-gpu-nightly-w7900). | ||
| if: >- | ||
| always() | ||
| && needs.changes.result == 'success' | ||
| && ( | ||
| (github.event_name != 'workflow_dispatch' | ||
| && github.event_name != 'pull_request' | ||
| && github.event_name != 'merge_group' | ||
| && needs.changes.outputs.serve == 'true') | ||
| || (github.event_name == 'workflow_dispatch' | ||
| && (inputs.platform == 'all' || inputs.platform == 'w7900')) | ||
|
|
@@ -1616,8 +1636,6 @@ jobs: | |
| env: | ||
| E2E_SERVE_TIMEOUT_SECS: "300" | ||
| E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}" | ||
| # Full serve matrix at the merge-queue gate. | ||
| E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}" | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -671,10 +671,11 @@ jobs: | |
| name: e2e-gpu-mi350p-${{ matrix.channel }}-report | ||
| path: tests/e2e-cucumber/results/ | ||
|
|
||
| # W7900 counterpart to e2e-gpu-nightly. Mirrors the per-PR e2e-gpu-w7900 lane | ||
| # in e2e-selfhosted.yml (w7900 label, shared uv cache, GPU preflight), with | ||
| # @nightly scenarios enabled. Keep the two in step: the report crate asserts | ||
| # the nightly lanes publish the same platforms as the per-PR lanes. | ||
| # W7900 counterpart to e2e-gpu-nightly. Mirrors the e2e-gpu-w7900 lane (which | ||
| # no longer runs on pull_request) in e2e-selfhosted.yml (w7900 label, shared | ||
| # uv cache, GPU preflight), with @nightly scenarios enabled. Keep the two in | ||
| # step: the report crate asserts the nightly lanes publish the same platforms | ||
| # as the lanes in e2e-selfhosted.yml. | ||
| e2e-gpu-nightly-w7900: | ||
| name: E2E tests (W7900, incl. nightly-only) | ||
| timeout-minutes: 90 | ||
|
|
@@ -762,29 +763,24 @@ jobs: | |
| # Strix Ubuntu job. | ||
| e2e-gpu-nightly-strix: | ||
| name: E2E tests (Strix Halo, incl. nightly-only) | ||
| # 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits when | ||
| # this host has Docker; on a host without it they skip and cost nothing. | ||
| # Whether it has Docker is unconfirmed: nothing in this job installs or checks | ||
| # it, so the "Report whether Docker is reachable" step logs the answer. If it | ||
| # does not, no nightly lane runs AIM serve and a pool-based one is needed. | ||
| # 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits; on a | ||
|
tomastola marked this conversation as resolved.
Resolved
|
||
| # host without Docker they skip and cost nothing. The "Report whether Docker | ||
| # is reachable" step logs the answer each run. serve-27's known failure on | ||
| # this pool is an xfail row in expectations.toml (rocm-cli#1058). | ||
| timeout-minutes: 120 | ||
| # `native` disambiguates this runner from any host that might carry a bare | ||
| # `wsl` label alongside `strix-halo` (the static WSL runner this repo used | ||
| # before the WSL lanes moved to the ephemeral pool) -- the paths below | ||
| # exist only on the native host, whether or not that other runner remains | ||
| # registered. | ||
| runs-on: [self-hosted, linux, strix-halo, native] | ||
| # The DevLab Dispatch pool, identical to the e2e-gpu-strix-ubuntu lane in | ||
| # e2e-selfhosted.yml, which runs on push and dispatch but not on pull_request | ||
| # or merge_group (pinned by workflow_contract.rs). This lane used to target | ||
| # the static `native` host, which is offline: every nightly sat queued until | ||
| # the 12h run cap and the lane never ran. Pool hosts are cold, | ||
| # so there are no host-specific HOME or cache-dir overrides. | ||
| runs-on: [self-hosted, linux, devlab-dispatch, strix-halo] | ||
|
tomastola marked this conversation as resolved.
Resolved
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The nightly Strix lane now runs on the same pool whose per-PR twin hangs to its cap in about 19% of jobs, and Docker on pool hosts is still unconfirmed (the step only warns). Please do a workflow_dispatch run of this lane before relying on it, so a hang does not show up as another silent nightly.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not dispatching |
||
| continue-on-error: true | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| channel: [release, nightly] | ||
| env: | ||
|
tomastola marked this conversation as resolved.
Resolved
|
||
| HOME: /home/ubuntu/actions-runner/e2e-home | ||
| CARGO_HOME: /home/ubuntu/actions-runner/.cargo | ||
| RUSTUP_HOME: /home/ubuntu/actions-runner/.rustup | ||
| TMPDIR: /home/ubuntu/actions-runner/tmp | ||
| PIP_CACHE_DIR: /home/ubuntu/actions-runner/pip-cache | ||
| E2E_INCLUDE_NIGHTLY: "1" | ||
| E2E_SERVE_TIMEOUT_SECS: "300" | ||
| E2E_TUI_TIMEOUT_SECS: "90" | ||
|
|
@@ -793,11 +789,8 @@ jobs: | |
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Prepare writable dirs on the nvme + reclaim GPU from stray E2E procs | ||
| run: | | ||
| mkdir -p "$HOME" "$TMPDIR" "$PIP_CACHE_DIR" | ||
| echo "prepared" | ||
| bash scripts/reclaim-gpu.sh | ||
| - name: Reclaim GPU from stray E2E processes | ||
| run: bash scripts/reclaim-gpu.sh | ||
|
|
||
| # Bounded wait for a free GPU, then an assertion that no leaked E2E engine | ||
| # process survives. gfx1151 shares system memory, so the floor is smaller | ||
|
|
@@ -815,11 +808,11 @@ jobs: | |
|
|
||
| - name: Ensure Rust toolchain | ||
| run: | | ||
| if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$CARGO_HOME/bin/cargo" ]; then | ||
| if ! command -v cargo >/dev/null 2>&1 && [ ! -x "$HOME/.cargo/bin/cargo" ]; then | ||
| curl --proto '=https' --tlsv1.2 -fsSL https://sh.rustup.rs \ | ||
| | sh -s -- -y --no-modify-path --default-toolchain none | ||
| fi | ||
| echo "$CARGO_HOME/bin" >> "$GITHUB_PATH" | ||
| echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" | ||
|
|
||
| # serve-26/27 (AIM) need Docker reachable on this host; without it they skip | ||
| # silently. Say which it is, so a nightly that never ran them is visible. | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.