Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
58a93aa
ci(e2e): reclaim engine processes leaked into the pre-warm tree (EAI-…
tomastola Sep 15, 2026
cf6902a
ci(e2e): fail the reclaim self-test on the branches it guards (EAI-8751)
tomastola Sep 17, 2026
12f34d9
ci(e2e): enforce every reclaim root and marker individually (EAI-8751)
tomastola Sep 18, 2026
3390dc8
ci(e2e): close review round on the reclaim self-test (EAI-8751)
tomastola Sep 21, 2026
4bfdb1c
ci(e2e): re-verify pid identity before SIGTERM too (EAI-8751)
tomastola Sep 21, 2026
df8bc9a
ci(e2e): give the reclaim self-test a real containment case (EAI-8751)
tomastola Sep 21, 2026
ca136ca
ci(e2e): correct the bystander count in a self-test comment (EAI-8751)
tomastola Sep 21, 2026
f3ca541
ci(e2e): close five review findings on the reclaim self-test (EAI-8751)
tomastola Sep 22, 2026
ac83031
ci(e2e): tighten reclaim self-test evidence and assertions (EAI-8751)
tomastola Sep 23, 2026
f416bf2
ci(e2e): reclaim leaks whose argv carries the record delimiters (EAI-…
tomastola Sep 23, 2026
35cbc77
ci(e2e): anchor the scenario root and stop overclaiming the rest (EAI…
tomastola Sep 25, 2026
8e871c2
ci(e2e): stop the self-test leaking decoys, and correct two claims (E…
tomastola Sep 25, 2026
b56fb8b
ci(e2e): correct three comment claims the code does not make (EAI-8751)
tomastola Sep 25, 2026
51cba17
ci(e2e): cover report_holders and the rm root, pin WSL diagnostics (E…
tomastola Sep 29, 2026
96f6591
ci(e2e): stop promising a /workload serve is never touched (EAI-8751)
tomastola Sep 29, 2026
2a665ae
ci(e2e): carry the signalled/terminated fix to the Windows mirrors (E…
tomastola Sep 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,22 @@ jobs:
if: needs.changes.outputs.heavy == 'true'
run: bash scripts/setup-wsl-portable-build-deps.sh --self-test

# Runs here, on the GitHub-hosted lane, rather than on the GPU runners the
# reclaim protects: the matching rules are what regress, and they are
# provable against decoy processes with no GPU present.
- name: GPU reclaim self-test
if: needs.changes.outputs.heavy == 'true'
# Twice, because one of the rules is conditional on the environment.
# The scenario root follows TMPDIR, and the self-test can only assert
# the redirected form when the shell running it actually has one — a
# hosted runner does not, so a single plain run reports that arm as
# "not exercised" and the nightly Strix lane's redirect would be the
# first place a break showed up. The second run supplies the redirect.
run: |
bash scripts/reclaim-gpu.sh --self-test
mkdir -p "${RUNNER_TEMP}/redirected-tmp"
TMPDIR="${RUNNER_TEMP}/redirected-tmp" bash scripts/reclaim-gpu.sh --self-test

- name: Release readiness self-test
if: needs.changes.outputs.heavy == 'true'
run: python scripts/release_readiness.py --self-test
Expand Down
130 changes: 79 additions & 51 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,25 +177,14 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# Reclaim the GPU before running: a serve leaked by a killed/timed-out prior
# run (its Drop teardown never executed) can keep an engine process spinning
# on the GPU, starving this job's serves until it hits the timeout. Kill only
# e2e leftovers — scoped to /tmp/rocm-e2e-* and the e2e-target/e2e-shared
# trees — never the runner or any /workload manual-testing processes.
# Reclaim the GPU before running: a serve leaked by a killed or
# timed-out prior run (its Drop teardown never executed) keeps an engine
# process on the card and starves this job's serves. Scoped to E2E-owned
# trees: a manual /workload serve is matched only if its command line
# also names an E2E root — see the exposure documented in
# scripts/reclaim-gpu.sh.
- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
# NOTE: no unanchored `pkill -f 'vulkan/llama-server'` — the lemonade
# Vulkan assistant an e2e scenario spawns lives under /tmp/rocm-e2e-*
# and is already caught by the first line; an unanchored pattern would
# also kill a legitimate /workload manual-testing serve on this shared
# self-hosted runner.
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"
run: bash scripts/reclaim-gpu.sh

# GPU preflight: fail fast (~90s) instead of hanging to the job cap when the
# GPU is missing, the driver is wedged, or VRAM is still saturated by a
Expand Down Expand Up @@ -235,6 +224,10 @@ jobs:
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
bash scripts/reclaim-gpu.sh --report-holders || true
exit 1

# cache: false — on this self-hosted runner we persist the build cache
Expand Down Expand Up @@ -409,14 +402,14 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# Reclaim the GPU before running: a serve leaked by a killed or
# timed-out prior run (its Drop teardown never executed) keeps an engine
# process on the card and starves this job's serves. Scoped to E2E-owned
# trees: a manual /workload serve is matched only if its command line
# also names an E2E root — see the exposure documented in
# scripts/reclaim-gpu.sh.
- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"
run: bash scripts/reclaim-gpu.sh

# GPU preflight: bounded wait so a missing/wedged/held GPU fails fast (~90s)
# instead of hanging to the job cap. Transient contention (VRAM still
Expand Down Expand Up @@ -557,6 +550,10 @@ jobs:
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
bash scripts/reclaim-gpu.sh --report-holders || true
exit 1

# Pool hosts don't preinstall cargo/rustup; bootstrap ourselves with
Expand Down Expand Up @@ -673,10 +670,33 @@ jobs:
- name: Reclaim GPU from stray E2E processes
shell: powershell
run: |
Get-CimInstance Win32_Process -ErrorAction SilentlyContinue |
Where-Object { $_.CommandLine -match 'rocm-e2e|__engine-serve-http|e2e-target|e2e-shared' -and $_.CommandLine -match 'llama-server|vllm|rocm ' } |
ForEach-Object { Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue }
Write-Host "reclaimed"
# Mirrors the SHAPE of scripts/reclaim-gpu.sh's rule — an E2E-owned root
# AND an engine process — but is not equivalent to it. Native Windows has
# no bash, so the rule is restated here rather than shared. Keep the root
# alternation in step with the script's E2E_ROOTS (`e2e-prewarm` is the
# shared runtime tree the engine binaries actually run from); xtask's
# reclaim_roots_are_mirrored_in_the_powershell_reclaims test pins that half.
#
# The engine half diverges in two pre-existing ways, tracked in EAI-8815.
# They fail in OPPOSITE directions, and only one can cost a bystander:
# - a MISS: `rocm ` (literal trailing space) cannot match Windows'
# `rocm.exe daemon`, so such a leak is not reclaimed here;
# - an OVER-MATCH: `__engine-serve-http` sits in the root alternation
# rather than the engine one, so on Windows a non-E2E managed serve
# satisfies the root half from that marker alone and can be killed.
# Do not read this as "same markers as the script" until that is fixed.
$leaked = Get-CimInstance Win32_Process -ErrorAction SilentlyContinue |
Where-Object { $_.CommandLine -match 'rocm-e2e|e2e-prewarm|e2e-shared|e2e-target|__engine-serve-http' -and $_.CommandLine -match 'llama-server|vllm|rocm ' }
foreach ($p in $leaked) {
Write-Host "reclaim: terminating pid=$($p.ProcessId) cmd=$($p.CommandLine)"
Stop-Process -Id $p.ProcessId -Force -ErrorAction SilentlyContinue
}
# SIGNALLED, not terminated, for the same reason the bash summary says
# so: -ErrorAction SilentlyContinue swallows a Stop-Process that could
# not kill its target, and this count is the SELECTED set, so a
# process that survived is still counted. Weaker than the bash line,
# which at least counts only what it re-verified before signalling.
Write-Host "reclaim: $(@($leaked).Count) process(es) signalled"

# GPU preflight (best-effort on Windows): bounded wait for the GPU to be
# available so a wedged/held GPU fails fast instead of hanging to the cap.
Expand Down Expand Up @@ -1211,6 +1231,14 @@ jobs:
exit 0
fi
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${pool} never dropped below the floor — a serve is likely still holding the GPU"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
# From the guest clone, like every sibling step: `wsl -- bash <file>`
# gets no --cd, so a bare relative path would resolve against whatever
# directory WSL maps the Windows step's cwd to, and `|| true` would
# hide a miss.
(cd /root/work/rocm-cli && bash scripts/reclaim-gpu.sh --report-holders) || true
exit 1
# This body runs under a bare `bash <file>`: Invoke-WslBash.ps1 adds
# `-eo pipefail` only when passed `-PipeFail`, which this step does
Expand Down Expand Up @@ -1378,18 +1406,14 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The pod is usually fresh (scale-to-zero recreates it per job), but the
# work PVC is not, and two jobs can share one pod's life. Same reclaim as
# e2e-gpu, scoped to e2e leftovers only.
# Reclaim the GPU before running: a serve leaked by a killed or
# timed-out prior run (its Drop teardown never executed) keeps an engine
# process on the card and starves this job's serves. Scoped to E2E-owned
# trees: a manual /workload serve is matched only if its command line
# also names an E2E root — see the exposure documented in
# scripts/reclaim-gpu.sh.
- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"
run: bash scripts/reclaim-gpu.sh

# Bounded wait, as in e2e-gpu. The pod requests `amd.com/gpu: 1`, so
# rocm-smi sees exactly the one card Kueue admitted it for. The floor is
Expand Down Expand Up @@ -1420,6 +1444,10 @@ jobs:
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
bash scripts/reclaim-gpu.sh --report-holders || true
exit 1

# cache: false for the same reason as e2e-gpu — the build cache is kept in
Expand Down Expand Up @@ -1506,18 +1534,14 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The pod is usually fresh (the runner scales to zero between jobs), but the
# work volume is not, and two jobs can share one pod's life. Same reclaim as
# e2e-gpu, scoped to e2e leftovers only.
# Reclaim the GPU before running: a serve leaked by a killed or
# timed-out prior run (its Drop teardown never executed) keeps an engine
# process on the card and starves this job's serves. Scoped to E2E-owned
# trees: a manual /workload serve is matched only if its command line
# also names an E2E root — see the exposure documented in
# scripts/reclaim-gpu.sh.
- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"
run: bash scripts/reclaim-gpu.sh

# Bounded wait, as in e2e-gpu. The pod is admitted for exactly one card, so
# rocm-smi sees that card alone. The floor matches the other Instinct lane
Expand Down Expand Up @@ -1549,6 +1573,10 @@ jobs:
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
# Name what is holding the card instead of leaving the next reader to
# reconstruct it: --report-holders prints per-process VRAM and the
# engine processes visible here. Never fails the step itself.
bash scripts/reclaim-gpu.sh --report-holders || true
exit 1

# cache: false for the same reason as e2e-gpu — the build cache is kept in
Expand Down
Loading
Loading