Repository navigation
fix(ci): install CPU-only torch wheels in GitHub Actions - #3
Conversation
025cbcc to
8b70263
Compare
Gate 3 — upstream-readiness: PASSEDVerified by actually building the mirror branch against pure upstream, per Clean cherry-pick onto Upstream-cleanliness checks:
This commit is contributable to Note on the stack-hygiene checkThis PR is the case that exposed a flaw in the old instructions. It is not. Only the Gate status
Not merging until Gate 2 lands. |
The Linux CI jobs fail while installing the package:
[Errno 122] Disk quota exceeded
Errno 122 is EDQUOT, hit while unpacking the CUDA torch stack that
`pip install -e .` resolves by default. torch alone is 554.6 MB and the
accompanying nvidia-* wheels add another ~2.4 GB, which no longer fits
the runner's quota. The failure occurs during "Installing build
dependencies", i.e. the isolated PEP 517 build environment is already
enough to exceed it on its own.
The trigger is a runner image change, not a code change:
* upstream 11_bernhard_dev, 2026-10-20, Image: ubuntu-24.04 -> success
* same workflows, 2026-10-23, Image: ubuntu-26.04 -> failure
Both resolve an identical wheel set (torch 2.14.1, 554.6 MB). The
`ubuntu-latest` label began migrating to Ubuntu 26 on 2026-10-19
(actions/runner-images#14748), between those two runs. Both workflows
already carry "Free up disk space" steps, so these jobs were running
close to the limit before the migration.
Fix: install CPU-only torch from download.pytorch.org, then build with
--no-build-isolation so the build reuses it instead of resolving torch
a second time. --no-build-isolation needs the build dependencies present
up front, hence the explicit setuptools/wheel/numpy install; the
`setuptools<81` bound matches the pin already in install_requires.
This loses no coverage: BUILD_CUDA_EXT keys off CUDA_HOME/ROCM_HOME,
which is unset on these runners, so the CUDA sources were never compiled
here. tests/smoke_tests/Dockerfile already installs torch this way.
macOS keeps the plain `pip install -e .` unchanged: its PyPI wheels are
already CPU-only, and leaving it alone keeps the default PEP 517
build-isolation path under test somewhere in the matrix.
8b70263 to
652e080
Compare
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
Gate 2 — adversarial review recordReviewed in a separate task with no authoring context, per copilot-instructions.md §4.7. The code Findings fixed
Findings accepted, not fixed
Probes that found nothing
Gate status
Reviewer's verdict was merge into the stack: yes; send upstream as-is: no — the "no" referred |
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
Gate 3 — re-verified against the amended commitThe commit was amended after review (macOS reverted to the plain install, narrative rewritten), so Cherry-pick applied with a clean auto-merge, no conflict. Gate 3 passes — the commit is Byproduct: this run found a bug in our own docsExecuting the recipe rather than reading it exposed a defect in the §6 text: it starts with Fixed in #4 — both §4.8 and §6 now name the target branch explicitly instead of relying on Gate summary
Ready for a human to press Rebase and merge. Per §5 an agent does not merge into Note the ordering dependency: #4 is currently red on every Linux job with this exact |
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
Mirror PR opened upstream: Emerge-Lab#602
Gate 3 re-verified against current This PR (#3) can now be stacked. Tracking no longer depends on it staying open — per §6.2 the mirror PR is the long-lived artifact, and it is independent of this one by construction. When this merges, GitHub closes it and deletes Lifecycle from here (§6.2 → §6.3):
Upstream checks are Framing note: upstream's CI is currently green (last run 2026-10-20 on |
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3): a
draft PR titled `[do-not-mirror]`, carrying the `do-not-mirror` label, with
the template's "Never upstream" box ticked. Four redundant markers, on
purpose. It is folded into the base commit by fixup rather than merged, so
the stack keeps exactly one fork commit.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
`integration/3.0` is both what BMW pins and the base every later feature
rebases onto, so a defect merged there is multiplied across every consumer
and every subsequent PR. Opening a PR was previously enough to get a change
onto the stack; now it is not.
A feature may be stacked only after three gates:
1. CI green on the final commit.
2. An adversarial review by someone other than the author. Section 4.7
defines what that means concretely -- revert the fix and prove the new
test fails, hunt edge cases, determinism, trust boundaries, observation
and config layout, hot-path cost, scope creep, and likely upstream
objections -- and requires the findings to be written down, including
the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit is
cherry-picked onto a branch cut from pure `upstream/3.0` *before*
merging, and must apply without conflict and produce an upstream-clean
diff. We stack only commits we could contribute today.
`chore(fork):` commits are exempt from gate 3 -- fork-only infrastructure
can never be upstream-ready -- and gain their own flow (section 3.3). Two
orthogonal axes must both be marked, because they fail in different
directions: `do-not-mirror` (must never reach upstream; marked four times
over -- title prefix, label, `chore(fork):` commit type, and the template's
"Never upstream" box) and `do-not-merge` (the merge button must never be
pressed on it; marked by title prefix and label). Merging such a PR would
put a *second* `chore(fork):` commit on the stack, and section 7 asserts
there is exactly one, so every future sync would abort until someone
unpicked it by hand. It is folded into the base commit by fixup instead, so
the stack keeps exactly one fork commit and the PR ends closed, not merged.
Also fixes a real ambiguity in the old text. Section 3 listed fork-only
*files* and fork-only *hunks inside shared files* as one flat list, and
section 4.5 then checked them with a file-name diff. That check cannot see
hunks, and it falsely flags any PR that legitimately edits a shared file --
which is exactly what the CI fix in #3 does to `ci.yml` and `install.yml`.
Split into 3.1 (whole files) and 3.2 (hunks in shared files), each with a
check that actually detects it.
Smaller changes:
- the default branch *is* the process. A clone checks out `integration/3.0`,
so a rule that lives only in an open PR has no effect on any agent or on
the BMW submodule consumers. Section 2 states this, 4.1 now opens by
listing in-flight process PRs (the two labels AND-ed, which excludes the
standing draft PR), and 3.3 requires the fold-in window to be kept short
and the output of agents that ran during it to be re-checked.
- document `chore/<slug>` branch naming, previously undefined
- agents may not review their own work, nor merge to `integration/3.0` on
their own initiative
- section 6 no longer rebuilds the mirror branch from scratch, since gate 3
already built and verified it, and now requires re-running the suites on
the mirror branch because it has a different base than CI tested
What
Install CPU-only torch wheels in the Linux GitHub Actions jobs, and build with
--no-build-isolationso the build reuses them.Why
Every Linux job fails while installing the package:
Errno 122isEDQUOT, raised while unpacking the CUDA torch stack thatpip install -e .resolves by default: torch is 554.6 MB and thenvidia-*wheels add roughly 2.4 GB more. The failure lands during "Installing build
dependencies" - the isolated PEP 517 build environment is enough to exceed
the quota on its own, before the runtime install is even attempted.
This is a runner image change, not a code change
11_bernhard_devImage:ubuntu-24.04ubuntu-26.04Same workflows, same resolved wheel set, three days apart. The
ubuntu-latestlabel began migrating to Ubuntu 26 on 2026-10-19,
which falls between those two runs. (That issue is cited for the rollout date
only - it documents the migration, not a quota change.)
Both workflows already carry
Free up disk spacesteps, so these jobs wererunning close to the limit well before the migration. Nothing in the repo
changed; the headroom did.
How
--no-build-isolationstops pip from resolving torch a second time for thebuild environment (which would pull CUDA wheels from PyPI regardless of what
the runtime environment already has). It requires build deps up front, hence
the explicit
setuptools/wheel/numpyline. Thesetuptools<81boundmirrors the pin already present in
install_requires.tests/smoke_tests/Dockerfilealready installs torch exactly this way, sothis aligns CI with the container the smoke tests run in.
Coverage
No GPU coverage is lost.
setup.pygates the CUDA extension onBUILD_CUDA_EXT, which keys offCUDA_HOME/ROCM_HOME- i.e. presence of aCUDA toolkit, not a GPU and not the torch wheel variant. Neither is set on
these runners, so the CUDA sources were never compiled in CI. Verified by
diffing a CUDA-torch build log against this one: no
nvcc, no-ltorch_cudain either.
macOS is deliberately left on the plain
pip install -e .. Its PyPI wheelsare already CPU-only, so converting it would be a no-op on disk usage while
removing the last job that exercises the default PEP 517 build-isolation path.
The macOS install behaviour in this PR is byte-for-byte the pre-existing one.
The one real trade-off: the Linux half of
install.ymlno longer exercisesbuild isolation. That is the cost of the fix, and the macOS job is what keeps
the path covered.
Known limitation
--index-urlreplaces PyPI rather than supplementing it, so a handful ofnon-torch dependencies are now served from
download.pytorch.org(it mirrorsthem). If a future dependency ever constrains
torch, pip could resolve aCUDA build from PyPI and the quota failure would return. It would fail loudly
and identically, but the cause would be non-obvious - worth a pin or an
--extra-index-urlrework if that ever bites.Testing
All 12 checks green on this branch (Unit / C Drive / Eval / Notebooks / Smoke
Performance on Linux, install matrix on Linux + macOS, Docker smoke 14m39s).
Follow-ups (not in this PR)
[build-system] requiresstill pinsnumpy<2.0while the runtime resolvesnumpy 2.x and the extension is rebuilt against 2.x headers. The pin is stale
and misleading; worth a separate change.
torchcannot simply be dropped frombuild-system.requires-setup.pyimportstorch.utils.cpp_extensionat module scope.