Repository navigation
Conversation
ASCS-eV/PufferDrive is consumed as a git submodule internally while feeding
features back to Emerge-Lab/PufferDrive. This commit carries everything that
exists only in the fork, and is the single place fork-only content lives.
Branch model:
3.0 verbatim, fast-forward-only mirror of upstream/3.0
integration/3.0 default branch; this commit plus a linear stack of
single-commit features; what internal consumers pin
feature/<slug> one feature, exactly one commit
mirror/<slug> cherry-picked onto pure upstream/3.0 for upstream PRs
integration/3.0 is the default branch because GitHub reads the PR template,
the contributing guidelines, the agent instructions and scheduled workflows
only from the default branch, and none of those files may exist on 3.0
without destroying the mirror. A clone checks out the default branch, so the
copy of the instructions here is the process as far as every agent and every
submodule consumer is concerned; a rule that lives only in an open PR has no
effect.
Nothing reaches integration/3.0 until three gates pass:
1. CI green on the final commit.
2. Adversarial review by someone other than the author, with the findings
written down -- including the probes that found nothing.
3. Upstream-readiness proven rather than asserted: the commit cherry-picks
onto a branch cut from pure upstream/3.0 and produces an upstream-clean
diff. We stack only commits we could contribute today.
chore(fork): commits are exempt from gate 3, since fork-only infrastructure
can never be upstream-ready. They carry two orthogonal markers: do-not-mirror
(must never reach upstream) and do-not-merge (the merge button is never
pressed -- merging would create a second chore(fork): commit and break the
single-commit invariant that every sync asserts). Such a PR is folded into
this commit by fixup and closed unmerged.
Contents:
- .github/copilot-instructions.md: normative process for coding agents
- CONTRIBUTING.md: human orientation, same rules
- .github/pull_request_template.md: testing and stack-hygiene gates
- .github/workflows/sync-mirror.yml: fast-forward-only mirror sync that
refuses any non-fast-forward instead of rewriting the upstream base
- AGENTS.md / CLAUDE.md: fenced pointer so agents find the fork process
without colliding with upstream-owned content
- ci.yml / install.yml: run CI for PRs targeting integration/3.0, which
upstream's branch filters would otherwise skip
A standing draft PR from integration/3.0 into 3.0 gives a continuous CI
verdict on the combined stack; it must never be merged.
Everything here is fork-only and must never be cherry-picked upstream.
The Linux CI jobs fail while installing the package:
[Errno 122] Disk quota exceeded
Errno 122 is EDQUOT, hit while unpacking the CUDA torch stack that
`pip install -e .` resolves by default. torch alone is 554.6 MB and the
accompanying nvidia-* wheels add another ~2.4 GB, which no longer fits
the runner's quota. The failure occurs during "Installing build
dependencies", i.e. the isolated PEP 517 build environment is already
enough to exceed it on its own.
The trigger is a runner image change, not a code change:
* upstream 11_bernhard_dev, 2026-10-20, Image: ubuntu-24.04 -> success
* same workflows, 2026-10-23, Image: ubuntu-26.04 -> failure
Both resolve an identical wheel set (torch 2.14.1, 554.6 MB). The
`ubuntu-latest` label began migrating to Ubuntu 26 on 2026-10-19
(actions/runner-images#14748), between those two runs. Both workflows
already carry "Free up disk space" steps, so these jobs were running
close to the limit before the migration.
Fix: install CPU-only torch from download.pytorch.org, then build with
--no-build-isolation so the build reuses it instead of resolving torch
a second time. --no-build-isolation needs the build dependencies present
up front, hence the explicit setuptools/wheel/numpy install; the
`setuptools<81` bound matches the pin already in install_requires.
This loses no coverage: BUILD_CUDA_EXT keys off CUDA_HOME/ROCM_HOME,
which is unset on these runners, so the CUDA sources were never compiled
here. tests/smoke_tests/Dockerfile already installs torch this way.
macOS keeps the plain `pip install -e .` unchanged: its PyPI wheels are
already CPU-only, and leaving it alone keeps the default PEP 517
build-isolation path under test somewhere in the matrix.
The Makefiles were the only part of the dev workflow that required a POSIX shell. The targets themselves were portable; their recipe bodies were not -- mkdir -p, rm -rf, command -v, for...done and $(MAKE) -C meant no test suite could be driven from a Windows checkout. Move the task definitions into a justfile and the logic they invoked into scripts/devtasks.py (stdlib only, so CI needs no extra dependency). Every recipe body is now a single command with no shell operators, so it behaves identically under sh, PowerShell and cmd. just is a command runner, not a build system, so the prerequisite graph Make provided is gone. The staleness checks are reimplemented in devtasks.py with the same mtime semantics: the C extension and each C test binary are rebuilt only when one of their sources or headers is newer. A warm `just test-c` still compiles nothing. tests/drive/Makefile is folded into the same helper; no caller used its unit and smoke targets separately. The root Makefile is kept as a thin shim that forwards to just, so `make test` and `make PYTHON=/path/to/python test` keep working. The C suite now runs natively on Windows. test_error and test_drive_map_cache fork() a child to assert on abort paths, which Windows has no equivalent for; they are skipped there behind an explicit notice so a native run cannot be mistaken for a full pass. The Python suites still require WSL, Linux or macOS because setup.py emits GCC/Clang-only flags and drive.h needs unistd.h. README states that limit instead of leaving it to be discovered by failure.
13 of 16 tasks
jdsika
force-pushed
the
integration/3.0
branch
from
October 26, 2026 14:45
b3fcee3 to
eddee28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces
Makefileandtests/drive/Makefilewith ajusttask runner plus a stdlib-onlyscripts/devtasks.pyhelper. A thinMakefileshim is kept somake testandmake PYTHON=/path/to/python testkeep working.Why
The Makefiles were the only part of the dev workflow that required a POSIX shell. The targets were portable; the recipe bodies were not —
mkdir -p,rm -rf,command -v,for…doneand$(MAKE) -Cmeant no suite could be driven from a Windows checkout at all.Every recipe body is now a single command with no shell operators, so it behaves identically under
sh, PowerShell andcmd. The measurable result: the C sim suite now runs natively on Windows, which it could not before.justis a command runner, not a build system, so Make's prerequisite graph is gone. The staleness checks are reimplemented indevtasks.pywith the same mtime semantics, so incrementality is preserved — a warmjust test-ccompiles nothing (verified: 0.7 s, 0 recompiles).Honest scoping note for the reviewer: the portability win comes from moving logic into Python, not from
justitself.justbuys the cross-platform front end (windows-shell,os_family()) and a self-documenting--list;devtasks.pydoes the real work.How tested
uvx pre-commit run --all-filesjust test-cjust test-cfork()tests, no skip notice, 0 recompilesjust perftest_simulator_raw_perf, 42117 SPSjust test-unitjust test-evaljust test-notebooksjust python=/nonexistent/python check-venvmake JUST=… PYTHON=/nonexistent/python check-venvmake JUST=… PYTHON=/usr/bin/python3 check-venvpython -m pip install rust-just→just --versionpre-commit run --all-filesmake rebuild— run asjust build(a dependency oftest-unit/test-eval) in WSL. No.c/.hfile is touched by this PR, so no new warnings are possible.make test-unit/make test-eval— run asjust test-unit/just test-eval, both passmake test-c— run asjust test-con both Windows and WSLmake test-notebooks— in progress, not yet green. No notebook is touched by this PR.make test-docker-smoke— not run. No sim dynamics, reward, observation or training-loop code is touched; this PR changes only task-runner plumbing and docs.make -C tests/drive perf— run asjust perf; no hot path is touched, so this proves the recipe, not a perf deltamake testbefore marking ready — blocked on notebooks above. This is why the PR is a draft.New behaviour is covered by a new test / the fix has a test that fails without it:
Not applicable in the usual sense, and I would rather say so than invent a test. This PR adds no simulator behaviour — it replaces the mechanism that runs the tests. The only meaningful verification is executing the existing suites through the new runner on both a POSIX and a Windows host, which is the table above. The one genuinely new behaviour, the Windows skip of the two
fork()tests, is asserted by the run itself: Windows reports 12 executed with the skip notice present, WSL reports 14 with it absent.Stack hygiene
git rev-list --count origin/integration/3.0..HEAD→1)integration/3.0git diff --name-only origin/integration/3.0...HEAD)AGENTS.md,CLAUDE.md, and thepull_requestbranch filters of
ci.yml/install.ymlare untouched in those specific placesci.ymlis in the diff, which is expected and allowed: only its twomakerun-lines change. The'integration/3.0'entry in itspull_requestbranch filter is deliberately untouched (§3.2).Merge gate
explicitly accepted in the thread — not done. I authored this change, so under §5 I must not review it; it needs a fresh-context reviewer or a human.
mirror/<slug>cherry-picked onto pureupstream/3.0, applied without conflict, diff is upstream-cleanGate 3 result:
Cherry-picked onto a detached worktree at pure
upstream/3.0, applied without conflict. Both abort guards passed (no fork-only file, no fork-only hunk):Upstream disposition
Emerge-Lab/PufferDriverequested now — needs a human to verify the finaltitle, body and diff first, and the body must carry the authorship disclosure from
copilot-instructions.md §6.1
chore(fork):change, Gate 3 exemptRisk / notes
Why this is a draft:
just test-notebookswas still running when the PR was opened, sojust testhas not completed end-to-end. Do not mark ready until it has.Reviewer: the two highest-risk areas.
justhas no timestamp graph, sois_stale()/newest_mtime()indevtasks.pyre-implement it by hand. If that logic is wrong, builds go stale silently and tests pass against old binaries. Verified warm-run compiles nothing on both platforms, but please attack this specifically.--interpreter "{{ python }}"leaked literal quotes intoargvundercmd.exe /c, which made a present venv look missing. It is now passed via an exportedPUFFER_PYTHONenv var, which bypasses shell quoting entirely. Anything else crossing the just→shell→Python boundary deserves the same suspicion.Behaviour changes.
tests/drive's separateunitandsmoketargets are merged into onec-testscommand. No caller used them separately.test-c,perfanddocker-smokedeliberately use the system interpreter, not.venv. The C suite is the only one that works natively on Windows, so requiring a venv would have broken the one thing this PR enables.ensure-test-depsnow passes--pythontouv pip install. The old Makefile did not, somake PYTHON=… test-notebooksinstalled into the wrong environment. Fixed here because this PR promotes that override to documented behaviour.Pre-existing README inaccuracies corrected (please sanity-check these rather than take my word):
make test-smokewas documented but never existed, andmake testwas described as running a "replay-HTML smoke test" while omittingtest-eval, which it actually runs.CI. Two
Install juststeps are added, usingpip install rust-justrather than a third-party action — this repo currently uses onlyactions/checkoutandactions/setup-python, and I did not want to widen that. It also means CI genuinely exercises the justfile instead of bypassing it.Known follow-ups, deliberately not in this PR.
runs-on: windows-latestjob runningjust test-c; it will need a human to add it to branch protection's required checks.CONTRIBUTING.md,.github/pull_request_template.md,.github/copilot-instructions.md,.github/workflows/sync-mirror.yml) still saymake …and are now stale. They are intentionally excluded from this commit — they are fork-only files under §3.1 and must land as a separatechore(fork):change, or this PR would fail Gate 3.setup.pybranches only onLinux/Darwinwith GCC-only flags, so the extension cannot build with MSVC. WSL/Docker remains the answer; README now says so explicitly.