diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000000..0c941e295e --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,654 @@ +# ASCS-eV/PufferDrive — fork contribution rules + +**Audience:** every coding agent (Copilot CLI/IDE, Claude Code, Codex, …) and every human working in +`ASCS-eV/PufferDrive`. + +**This file is fork-only.** It does not exist upstream and must never reach `Emerge-Lab/PufferDrive`. + +**This file is the core.** It is loaded automatically in every session in this repository, so it +carries only what you need to avoid doing damage. The deep procedures — the ones you run rarely and +must not improvise — live in `.github/fork/PROCESS.md`, which is **not** auto-loaded. Each one is +signposted below. Read the signposted section before you run the procedure; do not reconstruct it +from memory. + +## 0. Precedence + +1. The human's explicit instruction in the current turn. +2. **This file** — fork branching, PR, and release process. +3. `.github/fork/PROCESS.md` — the same process in detail; never contradicts this file. +4. `AGENTS.md` / `CLAUDE.md` (upstream-owned) — code style, trust boundaries, guardrails. + +Rules 2–3 and 4 do not overlap: they never redefine coding standards, and `AGENTS.md` never +defines fork process. Where `AGENTS.md` speaks about issues and PRs, it is describing +**upstream's** process; for PRs raised inside this fork, section 5 below applies instead. + +### 0.1 `forkctl` — the checks live in code, not in this file + +Every mechanical check in this document is implemented once, in `.github/fork/forkctl.py`, and is +covered by `.github/fork/test_forkctl.py`. It is stdlib-only and spawns no shell, so it behaves +identically on Linux, macOS, WSL and a Windows checkout, and there is nothing to install: + +```bash +python .github/fork/forkctl.py --help +``` + +For brevity this file writes `forkctl `. Expand it to the line above, or alias it with +`alias forkctl='python .github/fork/forkctl.py'`. + +| Command | What it decides | Where it is written up | +| --- | --- | --- | +| `forkctl doctor` | is this clone set up, and is the process itself mid-change? | 4.1 | +| `forkctl check-commit [--chore]` | exactly one commit, carrying no fork-only content | 4.4, 4.5 | +| `forkctl check-stack` | the stack invariants, before any force-push | 3.3, 7 | +| `forkctl mirror --sha\|--subject` | gate 3, and building the upstream branch | 4.8, 6 | +| `forkctl status` | divergence report, stale commits, in-flight mirrors | 7, and *While the mirror PR is open* / *After upstream responds* in `PROCESS.md` | +| `forkctl delete-mirror ` | is it safe to delete this mirror branch? | *While the mirror PR is open* in `PROCESS.md` | +| `forkctl fork-only [-v]` | the canonical fork-only path list | 3.1 | + +Bare numbers are sections of this file; italics are headings in `.github/fork/PROCESS.md`. + +Each exits non-zero on failure and prints the fix, so they compose into any script and are safe in +front of a `&&`. + +**Do not retype these checks as shell one-liners.** The previous revision of this file did, and the +copies drifted: the fork-only file list existed in six places, and the `AGENTS.md` fork-block guard +never fired for an edit *between* the markers, because the changed lines carry no marker of their +own. One implementation, with tests that prove each guard fails when it should, is the point. + +### 0.2 When to open `.github/fork/PROCESS.md` + +| You are about to… | Read | +| --- | --- | +| change a fork-only file on a `chore/` branch | *Changing fork-only content* | +| adversarially review someone's feature PR | *Adversarial review* | +| open, refresh, or retire an upstream PR | *Mirror PR workflow* | +| rebase the stack onto new upstream, or cut a release | *Syncing and releasing* | +| touch either standing draft PR | *The standing draft PRs* | + +Writing a feature, running the gates, and opening a fork PR need **only this file**. That is the +common case, and it is why the detail was moved out. + +## 1. Why this fork exists + +`ASCS-eV/PufferDrive` is a fork of `Emerge-Lab/PufferDrive`, consumed as a **git submodule** by +internal BMW projects. It has two jobs: + +- Provide a **stable, integrated branch** that internal consumers can pin. +- Prepare **small, well-tested, independently reviewable features** that we offer back upstream as + mirror PRs once we consider them ready. + +Everything below exists to keep those two jobs from fighting each other. + +## 2. Branch model + +| Branch | Role | Who writes | History | +| --- | --- | --- | --- | +| `3.0` | Verbatim mirror of `upstream/3.0`; base for every upstream PR | automation only | fast-forward only | +| `integration/3.0` | **Default branch.** Combined feature stack; what BMW consumes | feature PRs only | linear, rebased | +| `feature/` | One feature, **exactly one commit** | contributor/agent | rebased onto `integration/3.0` | +| `chore/` | Fork-only change, **exactly one commit** | contributor/agent | rebased onto `integration/3.0` | +| `mirror/` | Upstream PR source | generated by cherry-pick | cut from `upstream/3.0` | +| `fork-only/3.0` | Derived **view** of the fork base commit | republished by hand (3.3, 7) | force-pushed, never committed to | +| `2.0` | Frozen legacy line | nobody | archived, do not touch | + +Remotes: `origin` = `ASCS-eV/PufferDrive`, `upstream` = `Emerge-Lab/PufferDrive`. + +`integration/3.0` is the default branch because GitHub reads the PR template, the contributing +guidelines, and scheduled workflows **only** from the default branch — and none of those fork-only +files may exist on `3.0` without destroying the mirror. A welcome side effect: new PRs default to +the correct base. + +The same reasoning governs this file. A fresh `git clone` checks out the default branch, so the copy +of `.github/copilot-instructions.md`, `AGENTS.md`, and `CLAUDE.md` **on `integration/3.0` is the +process** as far as every agent — and every BMW submodule consumer — is concerned. A process change +that exists only in an open PR does not exist. That is precisely why fork-only changes are folded +into the base commit on the default branch (section 3.3) instead of being parked on a long-lived +branch, and why section 4.1 opens by checking whether the rules are mid-change. + +``` +upstream/3.0 ──► 3.0 (mirror) + └─► integration/3.0 = 3.0 + [fork scaffolding] + [feat A] + [feat B] + … + └─► feature/ (1 commit, PR'd into integration/3.0) +3.0 ──► mirror/ (cherry-pick of that same 1 commit → upstream PR) +``` + +`upstream/2.0` and `upstream/3.0` have **no common ancestor**. Never attempt to merge, rebase, or +cherry-pick between the `2.0` and `3.0` lines. + +### Hard branch rules + +- **Nothing reaches `integration/3.0` that has not cleared the merge gate in section 4.8** — green + CI, adversarial review, and proven upstream-readiness. The stack is a line of + already-contributable commits, not a staging area for work in progress. +- **Never commit to `3.0`.** It is a pure mirror; any local commit there breaks the clean base for + every mirror PR. +- **Never create a merge commit on `integration/3.0`.** The stack stays linear so that each feature + remains a single, cherry-pickable commit. +- **Never branch a feature off another unmerged feature.** Each feature must apply to pure `3.0`. +- `integration/3.0` is maintained by rebase and is therefore **force-pushed** (`--force-with-lease`). + Contributors re-sync with `git fetch origin && git reset --hard origin/integration/3.0` after + confirming `git status` is clean — **not** `git pull --rebase`, which replays your local copies of + commits that were rewritten or dropped during the rebase and silently resurrects them. Consumers + pin an immutable tag (section 7), not the branch tip. + +**How `integration/3.0` is protected, and the deliberate hole in it.** All 12 required checks are +enforced, and "require branches to be up to date" is on, so a PR cannot be merged green against a +stale base and land an untested combination. Admin enforcement is deliberately **off**: with it on, +GitHub rejects *any* direct ref update to the branch (verified — it returns +`8 of 12 required status checks are failing`), which would make the fixup flow in section 3.3 and +the stack rebase in section 7 impossible. Those two procedures bypass the gate by design, and the +invariant checks written into them are the compensating control. Do not force-push this branch +outside those two procedures. + +### The two standing draft PRs + +Two permanent **draft PRs target `3.0`**, and both must **never be merged, never be marked ready +for review, and never be committed to**: + +- **`integration/3.0` → `3.0`** — `DO NOT MERGE — integration stack CI`. A continuous CI verdict on + all combined features at once, which no individual feature PR can provide. It refreshes itself + whenever the stack is force-pushed. +- **`fork-only/3.0` → `3.0`** — `DO NOT MERGE — fork-only content view`. Its *Files changed* tab is + exactly what this fork adds and never sends upstream, with no features mixed in. Its branch is a + derived **view** that is republished **by hand** (section 3.3 and section 7); + `.github/workflows/fork-only-view.yml` only *detects* drift, it cannot publish. + +Merging either would push fork-only content into the mirror and destroy the clean base that every +upstream PR is cut from. + +Both carry the `do-not-merge` label **alone**. Never give a standing PR `do-not-mirror` as well: the +two labels *together* are the in-flight-process-change signal that `forkctl doctor` queries, and a +permanent PR carrying both would tell every agent, forever, that the rules are mid-change. The rule +is general: **a standing artifact carries `do-not-merge` alone; both labels together mark a +`chore/` PR that is actively rewriting the process.** + +Rationale, and why the view branch cannot be automated: `.github/fork/PROCESS.md` → *The standing +draft PRs*. + +## 3. Fork-only content — never send this upstream + +There are **two kinds**, and they are checked differently. Both must stay out of every feature +commit and every mirror PR. + +### 3.1 Whole files that exist only here + +If any of these appears in a feature diff, the feature is contaminated: + +- `.github/copilot-instructions.md` (this file) +- `CONTRIBUTING.md` +- `.github/pull_request_template.md` +- `.github/workflows/sync-mirror.yml` +- `.github/workflows/fork-only-view.yml` +- anything under `.github/fork/` — the fork's own tooling and its tests + +The last entry is a **prefix, not a list**: a new fork-only tool dropped into `.github/fork/` is +covered without editing anything. The authoritative definition lives in `.github/fork/forkctl.py` +and is printed by `forkctl fork-only -v`. Do not copy it into another file; a stale copy is a guard +that has silently stopped firing. + +### 3.2 Fork-only edits inside files that also exist upstream + +These files are shared with upstream, so the **file** appearing in a diff is fine and often +correct — only the **fork-only hunk** must stay put: + +| Shared file | Fork-only part | +| --- | --- | +| `AGENTS.md`, `CLAUDE.md` | the `` … `` block | +| `.github/workflows/ci.yml`, `.github/workflows/install.yml` | the `'integration/3.0'` entry in the **`pull_request`** `branches:` filter | + +A file-name check cannot catch 3.2 — only a content check can (section 4.5). Conversely, do not +treat a shared file as untouchable: a PR that rewrites the install steps of `ci.yml` and +`install.yml` is fully upstream-contributable, because it leaves the branch filter alone. + +Do **not** add `integration/3.0` to those workflows' `push:` filters. The standing draft PR +(section 2) already runs the full suite on every update of the branch, so a `push` trigger only +duplicates every job and doubles runner consumption. + +That reasoning is about the **test suites** specifically. `fork-only-view.yml` deliberately does use +a `push:` trigger on `integration/3.0`: it runs a handful of git commands rather than a test matrix, +and it exists precisely to react to that branch moving. Do not "fix" it to match the others. + +Fork-only content lives in **one dedicated commit** (`chore(fork): …`) sitting directly on top of +`3.0` at the base of the integration stack. That commit is never cherry-picked. + +### 3.3 Changing fork-only content + +Fork-only content lives in **one dedicated commit** (`chore(fork): …`) sitting directly on top of +`3.0` at the base of the integration stack. That commit is never cherry-picked, and there is exactly +one of it — `forkctl check-stack` asserts that, so a second one breaks every future sync. + +Changing it therefore has its own shape: + +- branch `chore/` from `origin/integration/3.0`, **exactly one commit**; +- open the PR as a **draft** titled `[do-not-mirror][do-not-merge] chore(fork): …`, with **both** + the `do-not-mirror` and `do-not-merge` labels; +- Gates 1 and 2 (section 4.8) apply in full; Gate 3 does not; +- it is **never merged with the button** — it is folded into the base commit by fixup and ends + **closed, not merged**. + +Those are two independent axes and both must be marked, because they fail in different directions. +`do-not-mirror` means the change must never reach upstream; `do-not-merge` means the merge button +must never be pressed, because merging would put a *second* `chore(fork):` commit on the stack and +every future sync would abort until someone unpicked it by hand. + +The fold force-pushes the default branch, so it is a maintainer action: an agent must not perform it +unasked (section 5). + +**The full procedure — the fixup, the invariant re-check, and the atomic push that moves +`integration/3.0` and `fork-only/3.0` together — is in `.github/fork/PROCESS.md` → *Changing +fork-only content*. Read it before you start; do not reconstruct it from memory.** + +**Keep the window short.** Between opening a `chore/` PR and folding it in, the default branch +still serves the *old* rules, and every agent that clones the fork in that window follows them — see +section 2. A fork-only PR is a **review vehicle, not a delivery mechanism**: nothing in it is in +force until the force-push lands. So do not park one open across unrelated work, and when one has +been open a while, assume agents have already acted on the superseded text and check their output +against the new rules rather than assuming compliance. + +If a feature genuinely needs to change one of these files, make it a **separate** `chore(fork):` +commit — do not fold it into the feature commit. +## 4. Feature workflow + +### 4.1 Start + +One-time setup in a fresh clone — `upstream` is not configured automatically, and several commands +below (and Gate 3) fail without it: + +```bash +git remote get-url upstream >/dev/null 2>&1 \ + || git remote add upstream https://github.com/Emerge-Lab/PufferDrive.git +gh repo set-default ASCS-eV/PufferDrive # else bare `gh pr …` can resolve to upstream +uv venv && source .venv/bin/activate && uv pip install -e . +``` + +Then check the clone, and whether the process itself is mid-change: + +```bash +forkctl doctor +``` + +It verifies the remote and the virtualenv, and lists any open PR carrying **both** fork-only +labels. Such a PR is rewriting the rules you are about to follow: anything it reports supersedes +this file in the areas it touches, so read it first and rebase onto the result rather than racing +it. (The query ANDs both labels, so it matches process PRs and nothing else — the two standing PRs +of section 2 carry `do-not-merge` alone and are correctly excluded. That exclusion is the whole +reason a standing artifact must never be given the second label.) + +Then: + +```bash +git fetch origin && git fetch upstream +git switch -c feature/ origin/integration/3.0 +``` + +`` is short, lowercase, hyphenated, and describes the change (`lane-heading-obs`, not +`fix-stuff`). + +For fork-only work that will become a `chore(fork):` commit, name the branch `chore/` +instead. Everything else in section 4 applies unchanged, except Gate 3 (section 4.8), from which +such commits are exempt. + +### 4.2 Scope + +One feature = one reviewable idea. If the PR description needs the word "and" to list unrelated +changes, split it. Refactors that merely enable a feature go in their own earlier PR. + +### 4.3 Build and test + +Makefile targets assume a POSIX shell and `.venv/bin/python` — use Linux, macOS, or WSL. + +| Situation | Required before pushing | +| --- | --- | +| any change | `pre-commit run --all-files` | +| `.c` / `.h` touched | `make rebuild` (zero warnings — warnings are bugs) | +| Python / env / eval | `make test-unit`, `make test-eval` | +| C sim logic | `make test-c` | +| notebooks or their deps | `make test-notebooks` | +| **sim dynamics, rewards, observations, or training loop** | `make test-docker-smoke` (golden train/rollout smoke run) | +| perf-sensitive hot paths | `make -C tests/drive perf` | +| everything in the default suite, before marking ready | `make test` | + +`make test` is **not** everything: it runs `test-unit test-eval test-c test-notebooks` and +deliberately excludes `test-docker-smoke` (slow, needs Docker) and the perf target. If the table +above says you need those, run them explicitly — `make test` passing does not cover them. + +**Regenerating smoke goldens.** If your change legitimately moves the golden train/rollout values +(observation layout, dynamics, rewards, RNG draw order), regenerate them with +`tests/smoke_tests/update_goldens.sh`, which runs inside the pinned QEMU image so the numbers match +CI. Review the diff, then stage both files **in the same single commit as the change that caused +them** — the rollout golden is gitignored and needs `-f`: + +```bash +tests/smoke_tests/update_goldens.sh +# tests/smoke_tests/data/ matches .gitignore, so BOTH goldens need -f even though +# both are already tracked — a plain `git add` on them exits 1. +git add -f tests/smoke_tests/data/drive_smoke_golden.json \ + tests/smoke_tests/data/drive_rollout_golden.json +``` + +Goldens are **base-dependent**: values regenerated on `integration/3.0` are not necessarily valid on +pure `upstream/3.0`. If Gate 3 (section 4.8) shows the smoke test failing on `mirror/`, +regenerate there and carry the corrected values back, rather than assuming our stack's numbers are +upstream's. + +**New behaviour requires a new test.** A bug fix requires a test that fails before the fix and +passes after it. "Existing tests still pass" is not sufficient evidence for a feature PR. + +Paste the commands you actually ran, and their outcome, into the PR's *How tested* section. Do not +claim a suite passed unless it ran to completion in this environment. + +### 4.4 Shape the commit + +Squash to **exactly one commit** before requesting review: + +```bash +git fetch origin +git rebase origin/integration/3.0 # replay onto the current base +git reset --soft origin/integration/3.0 # collapse the branch into the index +git commit -F # or -m; never rely on an editor opening +forkctl check-commit +``` + +If the rebase conflicts: resolve, `git add` the files, then +`GIT_EDITOR=true git rebase --continue` — plain `--continue` opens an editor and fails outright when +none is available. To back out, `git rebase --abort`. + +Use the non-interactive form above rather than `git rebase -i`: agents run without an editor, where +the interactive rebase either stalls or silently does nothing. + +Conventional Commits, imperative mood, ≤ 72-char subject: + +``` +feat(drive): add lane-heading error to the observation vector + +Why the change is needed, what the approach is, and anything a reviewer +would otherwise have to reverse-engineer. +``` + +Types: `feat`, `fix`, `perf`, `refactor`, `test`, `docs`, `build`, `ci`, `chore`. +Common scopes: `drive`, `obs`, `reward`, `train`, `eval`, `render`, `config`, `fork`. + +`chore(fork): …` marks a commit that **must never go upstream**. + +Do not add AI/agent authorship trailers or "generated by" notes to commits, or to **fork** PR text. + +The single exception is the upstream authorship disclosure required by section 6, which is +**mandatory** on mirror PR bodies. That is a disclosure to upstream, not an authorship trailer, and +this rule does not license omitting it. + +### 4.5 Verify upstream-cleanliness before opening the PR + +```bash +forkctl check-commit +``` + +It asserts exactly one commit, no fork-only **file** in the diff, and no fork-only **hunk** or +**block** touched in a shared file. It exits non-zero and names the offender; a check that prints +contamination and exits `0` is not a check. Changing the *rest* of a shared file is allowed and is +sometimes the whole point of the PR — only the fork-only parts of section 3.2 are protected. + +This is the cheap pre-flight check. The authoritative proof is Gate 3 in section 4.8. + +**`chore/` branches pass `--chore`** — their whole purpose is to change the files in section +3.1, so the fork-only checks are skipped and only the one-commit rule is enforced. The flag also +asserts the subject really is `chore(fork): …`, so it cannot be used to wave a feature through: + +```bash +forkctl check-commit --chore +``` + +### 4.6 Open the fork PR + +Base **`integration/3.0`** (the default, so it is pre-selected). Fill in +`.github/pull_request_template.md` completely and let CI finish. + +Opening the PR is **not** the end of your job — a PR that is merely open is not stacked. It must +clear section 4.8 first. + +### 4.7 Adversarial review + +Every feature PR is adversarially reviewed before it may be stacked. The reviewer's job is to +**try to break the change**, not to confirm that it looks reasonable. "LGTM" is not a review. + +**The reviewer is not the author.** When an agent wrote the change, the review runs as a *separate +task with fresh context* — a reviewer still holding the author's reasoning reproduces the author's +blind spots. + +Post the outcome as a PR review listing what was probed and what was found — **including the probes +that found nothing**, so the next reader knows the coverage. Each finding is either fixed or +explicitly accepted by a human, with the reason in the thread. + +**The checklist the reviewer works through is in `.github/fork/PROCESS.md` → *Adversarial review*.** +It covers test validity, breaking inputs, determinism, trust boundaries, layout compatibility, +performance, scope creep and upstream reception. Read it when reviewing; you do not need it when +authoring. +### 4.8 The merge gate — nothing stacks until it is proven + +`integration/3.0` is what BMW pins *and* the base that every later feature rebases onto, so a defect +merged here is multiplied across every downstream consumer and every subsequent PR. Merge a feature +PR **only** when all applicable gates are green: + +**Gate 1 — CI is green.** Every required check passes *on the final commit*. Not "green apart from a +known flake", not "green before the last force-push". This is enforced by GitHub, not by good +intentions: the required set is configured as branch protection on `integration/3.0`, so the merge +button stays disabled until all of it is green. Confirm with: + +```bash +gh pr checks --repo ASCS-eV/PufferDrive +``` + +If a check is broken for reasons unrelated to the feature, fix that first as its own PR — a CI fix +is itself a perfectly good feature PR — instead of merging past it. + +**If the commit adds or changes a workflow, Gate 1 also requires that the workflow has actually +run.** The required checks validate the repository; they do not execute your workflow, so a green +board says nothing about it. Dispatch it on the PR ref and link the run: + +```bash +gh workflow run .yml --ref +gh run list --workflow .yml --limit 1 --json databaseId,conclusion +``` + +Exercise each state it is supposed to distinguish, not just the happy path. This rule exists because +a fork-only workflow once cleared Gate 1 **and** Gate 2 and was still impossible: GitHub rejects any +`GITHUB_TOKEN` push that touches `.github/workflows/`, and nothing short of running it could have +revealed that. Assertions about a workflow are worth exactly as much as the runs behind them. + +**Gate 2 — adversarial review passed.** Section 4.7, findings written down and resolved. + +**Gate 3 — upstream-readiness is proven, not asserted.** Build the mirror branch *before* merging +and let the cherry-pick prove the commit is contributable as-is: + +```bash +git fetch upstream +forkctl mirror --sha +``` + +It recreates `mirror/` from pure `upstream/3.0`, cherry-picks the commit, asserts the result +contains no fork-only file and adds no fork-only marker anywhere, prints the file list for review, +and returns you to the branch you started on. It exits non-zero on any of those, and leaves the +branch in place for inspection when it fails. + +Recreating `mirror/` every time is deliberate — section 7 requires re-proving this gate after +a rebase. If the cherry-pick conflicts, the feature is entangled with fork-only state: fix it on +the feature branch and re-run. + +**If the change can move the smoke goldens** (observation layout, dynamics, rewards, RNG order), +Gate 3 must also *run* the relevant suites from section 4.3 on the mirror branch, not just diff it. +Goldens generated on our stack are not evidence about pure upstream. A commit whose goldens depend +on an earlier feature that upstream does not have is **not** upstream-ready: it does not stack until +that dependency is resolved. + +Actually *opening* the upstream PR stays a separate, human-approved step (sections 5 and 6) — the +gate is only that we could open it today. Record the resulting `integration/3.0` SHA on the PR when +the feature is stacked, so section 6 can rebuild the mirror later from a known commit. + +**Exemption — `chore(fork):` commits.** Fork-only infrastructure cannot be upstream-ready by +definition, so Gate 3 does not apply, and section 4.5 is run with `--chore`, which skips the +fork-only checks — touching those files is the entire point. **Every other rule still holds**, in +particular "exactly one commit": section 3.3 cherry-picks a single SHA, so a two-commit chore branch +silently loses work. Gates 1 and 2 apply in full, and the reviewer additionally confirms the change +is genuinely fork-only and contains nothing upstream would have wanted. Such a PR is a draft titled +`[do-not-mirror][do-not-merge] …` and carries both the `do-not-mirror` and `do-not-merge` labels; +it is applied by fixup into the base +commit rather than merged (section 3.3), so it ends **closed, not merged** — the "Rebase and merge" +step below does not apply to it. + +Only once the applicable gates pass, **a human presses "Rebase and merge"** (never squash-merge a +branch that is already one commit, never create a merge commit). Squash and merge commits are +disabled on this repository so the wrong button cannot be pressed by accident. + +If a feature is good but not yet upstream-ready, it does **not** go on the stack. Leave the PR open +and say why. An unmergeable-but-correct feature is a smaller problem than a stack nobody can +contribute from. + +## 5. Agent rules for issues and PRs + +- Agents **may** open, update, and comment on PRs and issues **in `ASCS-eV/PufferDrive`**. +- An agent **must not** adversarially review (section 4.7) a change it authored in the same session. + Start a fresh review task with no authoring context, or hand it to a human. +- Agents **must not** merge into `integration/3.0` on their own initiative. Report the gate status + from section 4.8 and let a human decide. +- **A human must verify the final PR title, body, and diff before an agent opens an _upstream_ PR.** + The agent may press the button; it may not decide that content is fit to send to another project. + Show the exact text and the diff, get explicit approval, then open it. That approval is **per PR, + in the current conversation**, and covers the `mirror/` content too. + Fork-internal PRs need no such approval — open them, fill the template honestly, and report the + gate status. The human gate on fork PRs is the **merge** button (above), not the open button. +- Never open a PR targeting the `3.0` mirror. The only PRs that may ever target it are the two + standing drafts from section 2 — the integration stack CI verdict and the fork-only view. Each is + opened **once, by a human**, and must never be merged. The integration draft refreshes itself + whenever `integration/3.0` moves; the fork-only view does **not** — its branch is republished by + hand (section 2), and a workflow fails until that happens. +- When a PR targets upstream, **upstream's `AGENTS.md` "Issue and PR Guidelines" apply**, with one + documented exception handled in section 6. Do not import them into fork PRs. +- Never force-push a branch you did not create. Never force-push `integration/3.0` without being + asked to. +- Never commit secrets, internal BMW identifiers, customer data, internal URLs, or proprietary + scenario data. This fork is public. + +## 6. Mirror PR workflow (fork → upstream) + +Once a feature is stacked, offering it upstream is a separate, **human-approved** step (section 5). + +Gate 3 already proved the commit cherry-picks cleanly, but do not assume the branch it produced +still exists — it is local-only. **Always rebuild the mirror from scratch**, and locate the commit +**by subject, not by the SHA recorded on the feature PR**: every sync rebase rewrites the stack, so +a recorded SHA stops resolving as soon as `3.0` advances. + +```bash +git fetch origin && git fetch upstream +forkctl mirror --subject "feat(scope): the exact commit subject" --stay +``` + +If it aborts, the feature commit is contaminated or no longer applies — fix it on +`integration/3.0` and regenerate. Otherwise push `mirror/` to `origin` and open the upstream +PR from it **after a human has verified the title, body and diff**. + +Re-run the relevant suites from section 4.3 on the mirror branch: it is a different base from the +one CI tested, and "it passed on our stack" is not evidence that it passes on pure upstream. + +Three things an upstream PR gets wrong by default: + +- **Structure is upstream's**: *What / Why / Notes*. Give them evidence, not our internal process + vocabulary — upstream does not know or care what our integration stack is. +- **Title style is upstream's, not ours.** They use imperative sentence case, not Conventional + Commits; a `fix(ci):` prefix marks the PR as foreign on sight. The *commit* keeps our + Conventional Commits subject — only the PR title is restyled. +- **Every upstream PR body must carry the AI-assistance disclosure** we use in place of upstream's + `AGENTIC_PR` canary. Omitting both the file and the disclosure is not an option: it would quietly + defeat a detection mechanism the maintainers put there on purpose. + +**The exact disclosure text, keeping an open PR fresh while upstream reviews it, the +branch-deletion hazard, and what to do after upstream merges, amends or rejects, are in +`.github/fork/PROCESS.md` → *Mirror PR workflow*. Read it before touching an upstream PR.** + +Two commands cover the routine half of that: + +```bash +forkctl status # what is in flight, and which mirrors have gone stale +forkctl delete-mirror # refuses while the upstream PR is still open +``` +## 7. Syncing and releasing + +**Sync the mirror:** automated. `.github/workflows/sync-mirror.yml` fast-forwards `3.0` from +`upstream/3.0` daily and on demand (Actions → *Sync 3.0 mirror* → *Run workflow*). It refuses any +non-fast-forward, which is the alarm that someone committed to the mirror. `3.0` additionally blocks +force-pushes, deletions, and non-linear history. **Never push to `3.0` by hand.** + +When the mirror moves ahead of the stack, the same workflow opens (and keeps updated) a +`mirror-sync` tracking issue, and closes it once `integration/3.0` contains the mirror head. The +rebase itself stays manual — replaying features onto new upstream code can conflict and must be +re-tested. + +**Rebasing the stack, and tagging a release, force-push or move refs on the branch BMW pins.** Both +are maintainer actions; an agent must not perform either unasked (section 5). The procedure, the +conflict policy, the explicit `--force-with-lease=:` rationale and the tagging steps are +in `.github/fork/PROCESS.md` → *Syncing and releasing*. + +**Hazard — never dispatch `update-smoke-golden.yml` against `3.0`.** This inherited upstream +workflow ends in `git push origin HEAD:` and triggers on pushes to `3.0` as well as +`workflow_dispatch`, so dispatching it there commits to the mirror and destroys the clean base every +upstream PR is cut from. No `forkctl` command can stop this — a workflow dispatch is not a git +operation — so it is the one destructive action guarded by nothing but this sentence. Regenerate +goldens with `tests/smoke_tests/update_goldens.sh` on your own branch instead (section 4.3). +Recovery is in `.github/fork/PROCESS.md` → *Syncing and releasing*. + +**Health check — run it after every sync and before any release:** + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It reports the total gap, the permanent fork-only floor, the feature gap, any commits upstream +already has (which the last sync should have dropped), and every in-flight mirror PR. + +**The total gap can never reach zero, and must not be driven to zero.** `integration/3.0` +permanently carries at least one `chore(fork):` commit, because the files in section 3.1 do not +exist upstream by design. An agent that reads a non-zero gap as a defect and tries to close it by +upstreaming that commit would leak the entire fork-only process into `Emerge-Lab/PufferDrive` — +precisely what Gate 3 exists to prevent. + +**The number that should trend to zero is the feature gap.** Zero means upstream has absorbed +everything we carry and the fork is pure scaffolding again; that is the steady state we are aiming +for, not an exceptional one. A feature gap that only grows means we are accumulating a private fork +rather than contributing — treat a persistently rising count as a process failure to discuss, not a +number to ignore. Every feature commit on the stack should be traceable to an open or pending mirror +PR (section 6); one that is not is either unfinished work or a change nobody intends to upstream, +and the second kind belongs in a `chore(fork):` commit where the guards can see it. +## 8. Definition of done + +A feature PR may be **stacked onto `integration/3.0`** only when **all** of the following hold. +Anything less stays an open PR. + +Authoring: + +- exactly one commit, Conventional Commits subject, rebased on current `integration/3.0` +- no fork-only file from section 3.1 and no fork-only hunk from section 3.2 in its diff +- new or updated tests cover the new behaviour; a fix has a test that fails without it +- the suites from section 4.3 that apply were run to completion and are listed in the PR +- the PR template is filled in, including the upstream-readiness assessment + +Gates (section 4.8): + +- **Gate 1** — every required check green on the final commit +- **Gate 2** — adversarial review (section 4.7) posted, findings fixed or explicitly accepted +- **Gate 3** — `mirror/` cherry-picks onto pure `upstream/3.0` without conflict and its diff + is upstream-clean — i.e. the feature is genuinely contributable today + *(exempt for `chore(fork):` commits, which can never be upstream-ready)* + +Merged by a human with **Rebase and merge**, leaving the stack linear. + +A `chore(fork):` PR has a different definition of done: exactly one commit, Gates 1 and 2 green, +Gate 3 and the section 4.5 fork-only checks exempt, all four `do-not-mirror` markers **and** both +`do-not-merge` markers present, and it is **folded into the base commit by fixup and the PR closed +unmerged** (section 3.3) — never merged with the button. + +The machine-checkable half of this list is two commands. Neither replaces Gates 1 and 2, which are +human and CI judgements: + +```bash +forkctl check-commit # authoring bullets 1-2 (--chore for a chore(fork): branch) +forkctl mirror --sha HEAD # Gate 3 (skip for chore(fork):) +``` diff --git a/.github/fork/PROCESS.md b/.github/fork/PROCESS.md new file mode 100644 index 0000000000..99b76e59c7 --- /dev/null +++ b/.github/fork/PROCESS.md @@ -0,0 +1,536 @@ +# ASCS-eV/PufferDrive — fork process reference + +**Fork-only. Never cherry-pick this file upstream.** + +This is the *detail*. The normative core lives in `.github/copilot-instructions.md`, which every +agent loads automatically; this file is deliberately **not** auto-loaded, because most sessions in +this repository touch the simulator and never touch the fork process. + +Read the section you need, when you need it. The core tells you which one: + +| You are about to... | Read | +| --- | --- | +| merge or close one of the two standing draft PRs | [Standing PRs](#the-standing-draft-prs) | +| change a fork-only file (`chore/`) | [Changing fork-only content](#changing-fork-only-content) | +| review someone else's feature PR | [Adversarial review](#adversarial-review) | +| open, refresh, or retire an upstream PR | [Mirror PR workflow](#mirror-pr-workflow-fork--upstream) | +| rebase the stack onto new upstream, or cut a release | [Syncing and releasing](#syncing-and-releasing) | + +Anything not covered here is in the core. Where the two disagree, the core wins — but they should +not disagree, because the *mechanical* rules live in neither: they live in `forkctl.py`, which is +tested. + +## The standing draft PRs + +### The integration stack CI draft + +A permanent **draft PR `integration/3.0` → `3.0`** titled `DO NOT MERGE — integration stack CI`, +carrying the `do-not-merge` label, gives us a continuous, always-visible CI verdict on *all combined +features at once*, which no individual feature PR can provide. + +- **Never merge it. Never mark it ready for review.** Merging would push the fork's stack into the + mirror and destroy the clean base that every upstream PR is cut from. +- It is force-pushed whenever the stack is rebased; its checks simply re-run. A red verdict means + the combined stack is broken even if every feature PR was green — fix it before tagging a release. +- Close and reopen it only if it is accidentally merged or the branch is recreated. + +### The fork-only content view draft + +A second permanent **draft PR `fork-only/3.0` → `3.0`** titled `DO NOT MERGE — fork-only content +view`, carrying the `do-not-merge` label, answers +one question that nothing else answers cheaply: *what does this fork add that never goes upstream?* +Its **Files changed** tab is exactly the fork-only content, with no features mixed in — the standing +integration draft PR above cannot show this, because its diff necessarily contains the whole feature +stack as well. + +- **It is a view, not storage.** Fork-only content lives in the single `chore(fork):` commit on + `integration/3.0`, because that is the only place agents, CI and the internal submodule pin can + see it. Parking it on a long-lived branch instead would stop the scheduled mirror sync from firing + and strip CI from every feature PR — see the default-branch note above. +- **The branch is published by hand, not by the workflow**, as the last step of the two procedures + that already force-push with human credentials (sections 3.3 and 7): + + ```bash + git push --force origin :refs/heads/fork-only/3.0 + ``` + + This is not a style choice. The fork base is made almost entirely of files under + `.github/workflows/`, and GitHub **blocks `GITHUB_TOKEN` from pushing any commit that creates or + updates a workflow file** — a platform rule that no `permissions:` block can lift. Automating it + would mean storing a PAT or GitHub App token with `workflows` write in a public repository, which + is a far worse trade than one extra line in two procedures that are already manual. +- `.github/workflows/fork-only-view.yml` is therefore a **detector, not a publisher**, and needs no + write access at all (`contents: read`). It re-derives the fork base, **fails if `fork-only/3.0` is + missing or stale** and prints the exact command to fix it, refuses to pass a stack that violates + the stack invariants in *Syncing and releasing*, and checks that the standing PR still has base `3.0`, draft status and + the `do-not-merge` label. It runs on pushes to `integration/3.0` **and to `fork-only/3.0`** (so + tampering with the view itself is caught), plus daily, plus on demand. Drift is loud, not silent. +- **The workflow does not open the PR either; a human does, once** — this repository deliberately + withholds PR-creation rights from `GITHUB_TOKEN` (`can_approve_pull_request_reviews: false`). + Afterwards the workflow only *checks* it: it warns when the standing PR is missing, and **fails + loudly if one was ever merged**, which would mean the mirror is contaminated. +- **Never merge it, never commit to it.** Merging would push fork-only content into the mirror; + committing to it would be silently discarded by the next republish. + +**It carries `do-not-merge` alone, never `do-not-mirror`** — exactly like the integration draft PR +above, and for a mechanical reason: both labels *together* are the in-flight-process-change signal +that section 4.1 queries. A permanent PR carrying both would tell every agent, forever, that the +rules are mid-change. The rule is therefore general: **a standing artifact carries `do-not-merge` +alone; both labels together mark a `chore/` PR that is actively rewriting the process.** + +Because the PR targets `3.0`, each refresh re-runs the `pull_request` workflows on it. That cost is +accepted: suppressing it would mean adding more fork-only hunks to `ci.yml`, `install.yml` and +`pre-commit.yml`, and every extra hunk in a shared file is another way a feature commit can get +contaminated. The fork base changes rarely, so the runs are rare too. + +## Changing fork-only content + +The "exactly one `chore(fork):` commit" rule is what keeps the stack trivially rebasable onto new +upstream, and keeps "is this commit contributable?" an obvious yes for every *other* commit on the +stack. Maintaining that invariant has consequences: + +1. Branch `chore/` from `origin/integration/3.0`. Make the change as one commit. +2. Open the PR **as a draft**, titled `[do-not-mirror][do-not-merge] chore(fork): …`, and apply + **both** the `do-not-mirror` and `do-not-merge` labels. +3. These are two independent axes and both must be marked, because they fail in different + directions: + - **`do-not-mirror`** — the change must never reach upstream. Marked four times over, deliberately + redundantly: title prefix, label, `chore(fork):` commit type, and the "Never upstream" box in + the PR template. + - **`do-not-merge`** — the merge button must never be pressed on it. Merging would put a *second* + `chore(fork):` commit on the stack, and `forkctl check-stack` asserts there is exactly one, so + **every future sync would abort** until someone unpicked it by hand. Marked twice: title prefix + and label. + + An agent that misses all of these is not reading. +4. Gates 1 and 2 (core section 4.8) apply in full. Gate 3 does not — see the exemption there. +5. When the gates pass, mark it ready for review, then **do not press the merge button.** Apply it + by folding it into the base commit instead: + +```bash +git switch integration/3.0 +git fetch origin +test -z "$(git status --porcelain)" || { echo "ABORT: dirty tree"; exit 1; } + +START=$(git rev-parse origin/integration/3.0) # capture the lease target before rewriting +git reset --hard "$START" + +# Confirms the stack is sound and returns the fork base, which is the oldest +# commit on it. With --print-base the report goes to stderr and only the SHA to stdout. +BASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +git cherry-pick --no-commit +git commit --fixup "$BASE" +GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/3.0 +make test +``` + +Note `git cherry-pick` takes **one** commit — which is safe only because a `chore/` branch is +required to be exactly one commit (section 4.1). Confirm that before you start, or you will silently +apply only its tip. + +Never use an interactive `git rebase -i` here. Agents run without an editor, where it silently +becomes a no-op — and the next command force-pushes the default branch. The `GIT_SEQUENCE_EDITOR=:` +form above is deterministic and needs no editor. + +Verify the invariant actually held **before** you push: + +```bash +# Re-checks every stack invariant, including that the fixup! was actually folded in, +# and returns the new fork base. +NEWBASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +# Move both refs or neither. The fold changes the fork base, so the view branch is stale the +# instant integration/3.0 lands; a two-step push leaves a window in which the detector reports +# drift that is already fixed, and a rejected lease must not move the view on its own. +git push --atomic --force-with-lease=refs/heads/integration/3.0:"$START" \ + origin integration/3.0 "+$NEWBASE:refs/heads/fork-only/3.0" +``` + +(Use the explicit `--force-with-lease=:` form — see *Syncing and releasing* for why the bare form is +unsafe. The shell-pitfall notes that used to live here went away with the shell: `forkctl` does the +checking, and its tests prove each guard fails when it should.) + +This rewrites the default branch after only `make test`, which is narrower than the required check +set. Watch the standing draft PR afterwards and be ready to fix forward quickly. + +The PR then ends up **closed, not merged**. Post the resulting `integration/3.0` SHA in the PR +thread so the trail survives; a closed-unmerged PR with no explanation looks like abandoned work. + +**Keep the window short.** Between opening a `chore/` PR and folding it in, the default branch +still serves the *old* rules, and every agent that clones the fork in that window follows them — see +section 2. A fork-only PR is a **review vehicle, not a delivery mechanism**: nothing in it is in +force until the force-push above lands. So do not park one open across unrelated work, and when one +has been open a while, assume agents have already acted on the superseded text and check their +output against the new rules rather than assuming compliance. + +If a feature genuinely needs to change one of these files, make it a **separate** `chore(fork):` +commit — do not fold it into the feature commit. + +## Adversarial review + +Every feature PR is adversarially reviewed before it may be stacked. The reviewer's job is to +**try to break the change**, not to confirm that it looks reasonable. "LGTM" is not a review. + +**The reviewer is not the author.** When an agent wrote the change, the review runs as a *separate +task with fresh context* — a reviewer still holding the author's reasoning reproduces the author's +blind spots. + +Work through all of these, and answer them in writing: + +- **Does the test actually test it?** Revert the source change, keep the new test, confirm the test + fails. A test that passes against unfixed code proves nothing. +- **What input breaks it?** Empty scenario, single agent, max agents, zero-length trajectory, NaN, + out-of-range action, a controlled agent removed mid-episode. +- **Determinism.** Same seed, same result? Any new iteration over an unordered container, + uninitialised read, or change to float accumulation order? +- **Trust boundaries and memory.** Every rule in `AGENTS.md` that touches the changed code — + bounds, ownership, lifetime, null handling. +- **Layout and compatibility.** Observation vector, action space, config keys, checkpoint format. + Any change here breaks submodule consumers and must be called out explicitly in the PR. +- **Performance.** Did a hot path gain an allocation, a branch, or a cache miss? If plausible, + require `make -C tests/drive perf` numbers rather than an opinion. +- **Scope.** Anything in the diff that is not the one stated idea: drive-by renames, unrelated + reformatting, leftover debug code, commented-out blocks. +- **Upstream reception.** Would an upstream maintainer request changes? Naming, file placement, + config surface, a knob we want but they would not. Fix it now, not after they ask. + +Post the outcome as a PR review listing what was probed and what was found — **including the probes +that found nothing**, so the next reader knows the coverage. Each finding is either fixed or +explicitly accepted by a human, with the reason in the thread. + +## Mirror PR workflow (fork → upstream) + +Gate 3 (section 4.8) already proved this commit cherry-picks cleanly. Do **not** assume the branch +it produced still exists: it is local-only, the feature branch is deleted from `origin` on merge, +and the agent opening the upstream PR is usually not the one that ran the gate. **Always rebuild the +mirror branch from scratch.** + +Locate the commit **by subject, not by the SHA recorded on the feature PR**. Every sync rebase +rewrites the whole stack, so a recorded SHA stops resolving as soon as `3.0` advances. Treat it as a +hint for a human reader, never as an input to a command. + +```bash +git fetch origin && git fetch upstream +forkctl mirror --subject "fix(scope): the exact commit subject" --stay +``` + +`--subject` resolves the commit against the live stack and aborts unless **exactly one** commit +matches, so an ambiguous or stale subject cannot silently mirror the wrong change. `--stay` leaves +you on `mirror/`, ready to push. The same fork-only file and marker guards as Gate 3 run +here: this is the authoritative check, not a formality, because the stack may have been rebased +since the gate last passed. + +If a guard aborts, the feature commit was contaminated — fix it on `integration/3.0` and regenerate. +Otherwise push `mirror/` to `origin` and open the upstream PR from it **after a human has +verified the title, body and diff** (section 5). + +Re-run the relevant suites from section 4.3 on the mirror branch: it is a different base from the +one CI tested, and "it passed on our stack" is not evidence that it passes on pure upstream. + +Keep the upstream PR description in upstream's **What / Why / Notes** structure. Give them the +evidence, not our internal process vocabulary — upstream does not know or care what our integration +stack is. + +**Match upstream's title convention, not ours.** This fork uses Conventional Commits; upstream does +not. Their merged PRs read as imperative sentence case — *"Remove native Drive rendering in favor of +HTML visualization"*, *"Add synchronized native and PyTorch profiling"*. A `fix(ci):` prefix marks +the PR as foreign on sight. The *commit* keeps our Conventional Commits subject; only the PR title +is restyled. Check their recent merged PRs before writing one: + +```bash +gh pr list --repo Emerge-Lab/PufferDrive --state merged --limit 10 --json title --jq '.[].title' +``` + +**Expect no checks at first.** GitHub holds workflow runs on fork PRs from first-time contributors +in an `action_required` state until a maintainer presses *Approve and run workflows*. An upstream PR +reporting "no checks reported" is normal and is not a defect in the branch: + +```bash +gh run list --repo Emerge-Lab/PufferDrive --limit 10 \ + --json conclusion,workflowName,headBranch --jq '.[] | select(.headBranch=="mirror/")' +``` + +Once approved, those runs use the workflow files from the PR's merge commit, so a workflow change we +are proposing is exercised by the very checks that gate it. + +### Agent authorship and upstream's `AGENTS.md` + +Upstream's `AGENTS.md` asks that an agent creating a PR add a file named `AGENTIC_PR` containing a +French poem. It is an agent-detection canary, and it collides with Gate 3, which requires the mirror +diff to contain nothing but the feature's own files. + +**Our resolution: disclose instead of complying, and say that we are doing so.** Every upstream PR +body ends with a note to this effect: + +> Prepared with AI assistance and reviewed by a human before submission. We noticed the `AGENTIC_PR` +> instruction in `AGENTS.md` — we are disclosing here rather than adding a poem file, to keep the +> diff limited to the change itself. Happy to add the file instead if you would prefer it. + +This is a deliberate, declared deviation, and it is **not negotiable downward**: never omit the file +*and* omit the disclosure. That would be quietly defeating a detection mechanism the maintainers put +there on purpose, and the mirror-PR relationship depends entirely on them trusting what we send. If +upstream tells us they want the file, add it and update this section. + +### While the mirror PR is open + +A feature is normally **stacked on `integration/3.0` and open upstream at the same time**, often for +weeks. That overlap is deliberate, and sustaining it requires nothing: `mirror/` is parented +on `upstream/3.0`, never on `integration/3.0`, so rebasing the stack cannot touch it, and the +upstream PR (`ASCS-eV:mirror/` → `Emerge-Lab:3.0`) is unaffected by the fork PR being merged +or closed. Our fork deletes `feature/` automatically on merge; it does not touch `mirror/*`, +and upstream cannot delete branches that live in our fork. + +So the default action during this period is **nothing**. Three events change that. + +**What is open when.** The fork feature PR is *not* the long-lived artifact — it closes the moment +the feature stacks: + +| Event | Fork feature PR | `mirror/` PR | Stack depth | +|---|---|---|---| +| Gates pass, human merges (4.8) | **merged, closed** | not open yet | +1 | +| Mirror pushed, upstream PR opened (6) | — | **open** | unchanged | +| Upstream merges | — | **closed by upstream** | unchanged | +| Next sync + rebase (7) | — | — | **−1**, drops as empty | + +The fork PR **cannot** be held open as a tracker: once its commit is reachable from +`integration/3.0`, GitHub marks the PR merged on its own. Anything that needs to outlive stacking — +tracking, discussion, the upstream conversation — belongs on the mirror PR, which is why the +inventory below keys off `mirror/*` rather than off feature branches. + +**Inventory — what is in flight.** The fork PR is closed by now, so nothing in `integration/3.0` +points at the open upstream PR; the slug is the only key. Run this before any sync or release: + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It lists every `mirror/*` branch on `origin` with the open upstream PR that belongs to it, and for +each one checks the only two events that need action: + +- **Event 1 — `upstream/3.0` advanced into a conflict.** The mirror no longer merges onto the + upstream tip, so the PR shows conflicts and reviewers can no longer read it. An upstream advance + that still merges cleanly needs no action and is reported as nothing. +- **Event 2 — our stacked commit changed.** A sync rebase can alter a commit through conflict + resolution, leaving the open PR offering upstream something we no longer run. That is a + correctness and honesty problem, not untidiness. It is detected by patch-identity against the + live stack, so a renamed subject does not raise a false alarm and an amended diff does not hide. + +A `mirror/*` branch with no open PR is either finished (*After upstream responds*) or was never opened. An open PR +with no branch should be impossible and means someone hit the deletion hazard below. + +For either event, rebuild with the recipe in section 6 of the core instructions — including its guards, because a conflict +resolution is exactly where fork-only content gets dragged in — then: + +```bash +git push --force-with-lease origin mirror/ +``` + +Force-pushing updates the open PR in place. **Never close and reopen**: that discards the review +thread and reads as a withdrawal. After an Event 2 rebuild, say in the PR that the change was +updated and why; never leave upstream reviewing a version we have abandoned. + +**Hazard — deleting `mirror/` closes the upstream PR.** GitHub closes a pull request when its +head branch disappears. Done to an open contribution, that looks like a silent withdrawal and throws +away the review. *After upstream responds* ends in a branch deletion, which is safe *only* once the PR has reached +a terminal state. Never delete a mirror branch by hand; the guard is built in: + +```bash +forkctl delete-mirror # --dry-run to check without deleting +``` + +It refuses unless the upstream PR is `MERGED`, `CLOSED`, or was never opened. + +### After upstream responds + +A mirror PR has three possible endings, and each leaves different work behind. None of them is +"nothing" — a stacked commit whose mirror has been resolved must be retired deliberately, or the +stack slowly fills with commits nobody can explain. + +**First, find out what actually happened.** Patch-identity, not the PR's merge badge, is the test — +upstream may squash, rename or amend, and a merged PR does not guarantee our exact diff landed. + +```bash +git fetch upstream +git cherry upstream/3.0 mirror/ +# "- " -> an equivalent patch IS upstream under a different SHA (case A) +# "+ " -> it is NOT upstream as written (case B or C) +# (no output) -> our exact commit is now an ancestor of upstream/3.0 (case A) +``` + +**Empty output is a result, not a failure.** It means upstream merged in a way that preserved our +commit object, so it is reachable from `upstream/3.0` and falls outside the range `git cherry` +examines. Both `-` and silence mean "landed"; only `+` means "not landed as written". If you cannot +tell which you are looking at, disambiguate explicitly: + +```bash +git merge-base --is-ancestor mirror/ upstream/3.0 \ + && echo "landed verbatim (same commit)" \ + || git cherry upstream/3.0 mirror/ +``` + +Patch-identity ignores the commit message, so upstream renaming the subject still reports `-`. It +does **not** ignore content: any amendment to the diff, however small, reports `+`. That is the +correct signal — an amended change is case B, and the rebase will stop on it rather than drop it. + +The same question across the whole stack is `forkctl status`, which lists every commit we carry +that upstream already has. Those are pure rebase risk and should already be gone; if one persists +across two syncs, something is resurrecting it (*Syncing and releasing*). Run it before any release. + +**Case A — merged as written.** Do nothing to the stack by hand. The next sync fast-forwards `3.0`, +and the rebase of `integration/3.0` drops our copy automatically because it becomes empty. Confirm +it actually went, rather than assuming — `forkctl status` must report a feature gap one lower than +before, and no stale commits: + +```bash +forkctl status +git branch -D mirror/ +``` + +Then delete the remote branch with `forkctl delete-mirror ` — never with a bare +`git push origin --delete`. In case A the PR is already merged, so the guard passes; running it +anyway is what stops the same command being used by reflex while a PR is still open. + +**Case B — merged with modifications.** Upstream's edited version is now in the new base, so ours is +redundant *and* textually different, which means the rebase will stop on a conflict instead of +dropping it silently. Resolve in favour of upstream — ours has no privileged status once they have +taken ownership of the change: + +```bash +git rebase --skip # discards our now-superseded commit and continues +``` + +Then read upstream's merged version before moving on. If they changed behaviour we depend on, that +is a *new* feature branch against the new base, not an attempt to restore our original commit. Never +resurrect a superseded commit to "win back" a difference; raise it upstream as a follow-up instead. + +**Case C — rejected, or stalled with no realistic path.** This is the one that needs a decision +rather than a command, and it must not be left implicit. The commit is now a **permanent fork +delta**, and a permanent delta that still looks like a pending contribution will be rebased, +re-reviewed and re-offered by some future agent forever. + +Decide explicitly, and record the decision in the PR and in the commit subject: + +- **Keep it as a fork-only behaviour change** — reclassify it under core section 3.2, add its path or + marker to `FORK_ONLY_PATHS` / `FORK_HUNK_MARKERS` in `.github/fork/forkctl.py` (with a test), and + rename the commit to `chore(fork):` so the core section 4.5 and Gate 3 guards start treating it as + something that must *never* leave. This is the honest outcome for a change upstream has declined + on principle. +- **Drop it** — if it only existed to be upstreamed, remove it from the stack (*Changing fork-only content*) rather + than carrying a rebase hazard indefinitely. + +Either way, close the mirror PR with a reason, delete `mirror/`, and note the outcome so the +next agent does not reopen the same conversation with upstream. + +**In all three cases**, if the feature commit was rewritten on our side at any point after the +mirror branch was built, that mirror is stale — rebuild it and re-prove Gate 3 (core section 6) before +drawing any conclusion from `git cherry`, which compares the stale patch rather than the live one. + +## Syncing and releasing + +**Sync the mirror:** automated. `.github/workflows/sync-mirror.yml` fast-forwards `3.0` from +`upstream/3.0` daily and on demand (Actions → *Sync 3.0 mirror* → *Run workflow*). It refuses any +non-fast-forward, which is the alarm that someone committed to the mirror. `3.0` additionally blocks +force-pushes, deletions, and non-linear history. Do not push to it by hand. + +When the mirror moves ahead of the stack, the same workflow opens (and keeps updated) a +`mirror-sync` tracking issue; it closes the issue automatically once `integration/3.0` contains the +mirror head. The rebase itself stays manual — replaying features onto new upstream code can +conflict and must be re-tested. + +**Rebase the stack** (maintainers): + +```bash +git switch integration/3.0 +git fetch origin +test -z "$(git status --porcelain)" || { echo "ABORT: dirty tree"; exit 1; } + +START=$(git rev-parse origin/integration/3.0) # capture the lease target before rewriting +git reset --hard "$START" # discard local drift +git rebase origin/3.0 +``` + +On conflict, resolve in favour of upstream for code they changed, keep our commit's intent, then: + +```bash +git add +GIT_EDITOR=true git rebase --continue # --continue opens an editor; GIT_EDITOR=false makes it fail +``` + +If a conflict cannot be resolved confidently, `git rebase --abort` and raise it — never guess on the +branch BMW pins. + +Re-check the invariants **after** the rebase, exactly as in *Changing fork-only content* — a rebase happily preserves +a stack that was already malformed: + +```bash +BASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +make test + +# Move both refs or neither. A rejected lease must not leave the view branch pointing at a base +# that was never published, and a two-step push leaves a window in which the detector reports +# drift that no longer exists. +git push --atomic --force-with-lease=refs/heads/integration/3.0:"$START" \ + origin integration/3.0 "+$BASE:refs/heads/fork-only/3.0" +``` + +Use the **explicit** `--force-with-lease=:` form. Bare `--force-with-lease` leases against +your remote-tracking ref, which any background `git fetch` silently refreshes — at which point it +will happily overwrite a colleague's push that you never incorporated. If the push is rejected, +start over from the new remote tip; do not re-run with a wider hammer. + +`make test` is not the full gate (section 4.3) — it skips Docker smoke, perf, the install matrix and +pre-commit. After pushing, the standing draft PR re-runs the real suite against the rewritten stack; +treat a release as blocked until that is green. + +Drop commits that upstream has already merged; never resurrect them — *After upstream responds* gives the +mechanics, including the case where upstream merged a *modified* version. After any rebase that +rewrote a feature commit, its `mirror/` branch is stale: if an upstream PR is already open on +it, *While the mirror PR is open* applies and the PR must be refreshed; if not, Gate 3 must be re-proven before that +feature is offered upstream. Either way, a commit whose content changed materially needs re-review. + +**Divergence report.** The stack is upstream *debt*: every feature commit we carry is one upstream +has not taken yet. Run this after every sync, and before any release: + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It prints the total gap, the fork-only floor and the feature gap, lists the stack, warns about +commits upstream already has, and reports every in-flight mirror. It warns rather than fails: a +non-zero gap is the normal state, not an error. + +**The total gap can never reach zero, and must not be driven to zero.** `integration/3.0` +permanently carries at least one `chore(fork):` commit, because the files in section 3.1 do not +exist upstream by design. An agent that reads a non-zero gap as a defect and tries to close it by +upstreaming that commit would leak the entire fork-only process into `Emerge-Lab/PufferDrive` — +precisely what Gate 3 exists to prevent. + +**The number that should trend to zero is the feature gap.** Zero means upstream has absorbed +everything we carry and the fork is pure scaffolding again; that is the steady state we are aiming +for, not an exceptional one. A feature gap that only grows means we are accumulating a private fork +rather than contributing — treat a persistently rising count as a process failure to discuss, not a +number to ignore. Every feature commit on the stack should be traceable to an open or pending mirror +PR (*While the mirror PR is open*); one that is not is either unfinished work or a change nobody intends to upstream, +and the second kind belongs in a `chore(fork):` commit where the guards can see it. + +**Hazard — `update-smoke-golden.yml`.** This inherited upstream workflow ends in +`git push origin HEAD:`, and it triggers on pushes to `3.0` as well as `workflow_dispatch`. +Never dispatch it against `3.0`: it would commit to the mirror and break the clean base for every +upstream PR. If it ever does, `sync-mirror.yml` is the detector — it refuses the non-fast-forward +and the mirror-sync issue appears. Recover by resetting `3.0` to `upstream/3.0` (the branch +protection on `3.0` blocks force-pushes, so this needs an admin and should be done deliberately). + +**Release for the BMW submodule:** after a green rebase, run the divergence report above and confirm +the feature gap matches the mirror PRs actually in flight — a release is the moment the gap becomes +someone else's dependency. Then tag an immutable point and pin that tag. + +```bash +git tag -a ascs/3.0-$(date +%Y%m%d) -m "Integration snapshot" +git push origin ascs/3.0-$(date +%Y%m%d) +``` + +Consumers pin a tag or a commit SHA — never the moving `integration/3.0` tip. diff --git a/.github/fork/forkctl.py b/.github/fork/forkctl.py new file mode 100644 index 0000000000..5e88a0c825 --- /dev/null +++ b/.github/fork/forkctl.py @@ -0,0 +1,1550 @@ +#!/usr/bin/env python3 +"""Fork-process checks for ASCS-eV/PufferDrive. + +Fork-only tooling. Never cherry-pick this file upstream. + +This is the single implementation of the guards described in +`.github/copilot-instructions.md`. Humans, coding agents and CI all run this +same code instead of retyping shell snippets, so a guard cannot be mistyped at +one call site and silently pass at another. + +It is deliberately stdlib-only and shell-free. The fork process depends on these +checks failing *loudly*, and the classic ways a shell guard dies quietly have no +analogue here: `grep -c` exiting 1 on no match, `git log | grep -q` returning +141 under `pipefail`, a pipeline's exit status being swallowed. + +Every subcommand exits 0 on success and 1 on failure. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys + + +MIRROR_REF = "origin/3.0" +STACK_REF = "origin/integration/3.0" +UPSTREAM_REF = "upstream/3.0" +VIEW_BRANCH = "fork-only/3.0" + +FORK_REPO = "ASCS-eV/PufferDrive" +UPSTREAM_REPO = "Emerge-Lab/PufferDrive" +UPSTREAM_URL = "https://github.com/Emerge-Lab/PufferDrive.git" +FORK_OWNER = "ASCS-eV" + +FORK_COMMIT_PREFIX = "chore(fork): " + +# Files that exist only in this fork; none may appear in a feature diff. This +# tuple is the single source of truth. The docs point here instead of repeating +# the list, so adding a fork-only file cannot leave a stale copy behind. +FORK_ONLY_PATHS = ( + ".github/copilot-instructions.md", + "CONTRIBUTING.md", + ".github/pull_request_template.md", + ".github/workflows/sync-mirror.yml", + ".github/workflows/fork-only-view.yml", +) + +# Anything under these prefixes is fork-only by construction, so a new fork-only +# tool dropped into `.github/fork/` is covered without editing this file. +FORK_ONLY_PREFIXES = (".github/fork/",) + +# Registration is only half the invariant. These files must also *exist*: a base +# rewrite that quietly drops the detector or the always-loaded instructions would +# otherwise pass every check, because nothing was touched that is unregistered. +REQUIRED_FORK_FILES = FORK_ONLY_PATHS + ( + ".github/fork/forkctl.py", + ".github/fork/test_forkctl.py", + ".github/fork/PROCESS.md", +) + +# Shared with upstream, but carrying a fork-only hunk. The file may legitimately +# appear in a feature diff; only those hunks must stay untouched. +SHARED_FILES = ( + "AGENTS.md", + "CLAUDE.md", + ".github/workflows/ci.yml", + ".github/workflows/install.yml", +) + +FORK_HUNK_MARKERS = ("integration/3.0", "fork:begin", "fork:end") + +# `integration/3.0` is only evidence of fork-only content inside workflow +# configuration. Elsewhere -- including prose under `.github/` -- it is an +# ordinary string an upstream-bound commit may legitimately write. +FORK_PATH_SCOPED_MARKERS = {"integration/3.0": (".github/workflows/",)} + +# Workflows shared with upstream must keep the fork branch in the `pull_request` +# trigger and nowhere else. Moving it to `push:` leaves the marker line untouched, +# so only a structural read of the trigger tree catches it. +FORK_WORKFLOW_FILES = (".github/workflows/ci.yml", ".github/workflows/install.yml") +FORK_WORKFLOW_BRANCH = "integration/3.0" +FORK_WORKFLOW_TRIGGER = "pull_request" + +CONVENTIONAL_TYPES = ( + "feat", + "fix", + "perf", + "refactor", + "test", + "docs", + "build", + "ci", + "chore", +) +CONVENTIONAL_RE = re.compile(rf"^({'|'.join(CONVENTIONAL_TYPES)})(\([a-z0-9][a-z0-9-]*\))?!?: \S.*$") +MAX_SUBJECT_LENGTH = 72 + +# Files whose fork-only part is a delimited block rather than a single line. +# Comparing the block's whole content catches an edit *between* the markers, +# which a diff-line scan misses: those changed lines carry no marker themselves. +FORK_BLOCK_FILES = ("AGENTS.md", "CLAUDE.md") +FORK_BLOCK_BEGIN = "fork:begin" +FORK_BLOCK_END = "fork:end" + +TERMINAL_PR_STATES = ("MERGED", "CLOSED", "NONE") + + +class ForkctlError(RuntimeError): + pass + + +def git(*args: str, check: bool = True) -> str: + proc = subprocess.run(["git", "--no-pager", *args], capture_output=True, text=True, check=False) + if check and proc.returncode != 0: + raise ForkctlError(f"git {' '.join(args)} failed: {proc.stderr.strip()}") + # Strip line terminators only. Stripping arbitrary whitespace would silently + # normalise a subject like " chore(fork): x" into a conforming one. + return proc.stdout.strip("\r\n") + + +def git_rc(*args: str) -> int: + proc = subprocess.run(["git", "--no-pager", *args], capture_output=True, text=True, check=False) + return proc.returncode + + +def show_file(ref: str, path: str) -> str | None: + """Contents of `path` at `ref`, or None if it genuinely does not exist. + + `git show` with check=False conflates "absent", "empty" and "git failed", + and every caller then reads the empty string as "nothing to check". Probing + for existence first keeps a git error loud instead of fail-open. + + Returned verbatim, including trailing newlines: `git()` strips line + terminators, which would hide a trailing-newline rewrite from the byte + comparison in `smuggled_shared_changes()`. + """ + if git_rc("cat-file", "-e", f"{ref}:{path}") != 0: + return None + proc = subprocess.run(["git", "--no-pager", "show", f"{ref}:{path}"], capture_output=True, check=False) + if proc.returncode != 0: + raise ForkctlError(f"git show {ref}:{path} failed: {proc.stderr.decode('utf-8', 'replace').strip()}") + return proc.stdout.decode("utf-8") + + +def head_is_detached() -> bool: + return git_rc("symbolic-ref", "--quiet", "HEAD") != 0 + + +def git_lines(*args: str) -> list[str]: + out = git(*args) + return out.splitlines() if out else [] + + +def gh_available() -> bool: + return shutil.which("gh") is not None + + +def gh_json(*args: str): + proc = subprocess.run(["gh", *args], capture_output=True, text=True, check=False) + if proc.returncode != 0: + raise ForkctlError(f"gh {' '.join(args)} failed: {proc.stderr.strip()}") + return json.loads(proc.stdout or "[]") + + +class Report: + """Collects check outcomes and renders them for a terminal or for CI.""" + + def __init__(self, title: str, fmt: str = "text", stream=None) -> None: + self.title = title + self.fmt = fmt + self.stream = stream if stream is not None else sys.stdout + self.failed = False + if fmt == "text": + print(f"== {title}", file=self.stream) + + def ok(self, message: str) -> None: + if self.fmt == "text": + print(f" PASS {message}", file=self.stream) + + def info(self, message: str) -> None: + if self.fmt == "text": + print(f" {message}", file=self.stream) + + def warn(self, message: str) -> None: + if self.fmt == "github": + print(f"::warning::{message}", file=self.stream) + else: + print(f" WARN {message}", file=self.stream) + + def fail(self, message: str, fix: str | None = None) -> None: + self.failed = True + if self.fmt == "github": + print(f"::error::{message}" + (f" Fix: {fix}" if fix else ""), file=self.stream) + else: + print(f" FAIL {message}", file=self.stream) + if fix: + print(f" fix: {fix}", file=self.stream) + + def check(self, passed: bool, ok: str, bad: str, fix: str | None = None) -> bool: + if passed: + self.ok(ok) + return True + self.fail(bad, fix) + return False + + def finish(self) -> int: + if self.fmt == "text": + print(f"== {self.title}: {'FAILED' if self.failed else 'OK'}", file=self.stream) + return 1 if self.failed else 0 + + +def ref_exists(ref: str) -> bool: + return git_rc("rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}") == 0 + + +def require_refs(report: Report, *refs: str) -> bool: + missing = [ref for ref in refs if not ref_exists(ref)] + if not missing: + return True + report.fail( + f"ref(s) not found locally: {', '.join(missing)}", + "git fetch origin && git fetch upstream", + ) + return False + + +def is_fork_only_path(path: str) -> bool: + return path in FORK_ONLY_PATHS or path.startswith(FORK_ONLY_PREFIXES) + + +def diff_content_lines(diff: str, added_only: bool = False) -> list[str]: + """Content lines of a diff, excluding the `+++`/`---` file headers.""" + wanted = ("+",) if added_only else ("+", "-") + return [line for line in diff.splitlines() if line.startswith(wanted) and not line.startswith(("+++", "---"))] + + +def fork_only_files_in(base: str, head: str = "HEAD") -> list[str]: + # --no-renames: with rename detection on, `git diff --name-only` reports only + # the destination, so renaming a fork-only file out of the way would hide it. + files = git_lines("diff", "--no-renames", "--name-only", f"{base}...{head}") + return [path for path in files if is_fork_only_path(path)] + + +def diff_files(base: str, head: str = "HEAD") -> list[str]: + return git_lines("diff", "--no-renames", "--name-only", f"{base}...{head}") + + +def marker_applies_to(marker: str, path: str) -> bool: + scopes = FORK_PATH_SCOPED_MARKERS.get(marker) + return True if scopes is None else path.startswith(scopes) + + +def fork_markers_anywhere(base: str, head: str = "HEAD") -> list[str]: + """Added lines referencing fork-only state, across the whole diff. + + Used against pure upstream, where no added line should mention the fork. + Path-scoped markers are only evidence inside the paths they are scoped to, + so ordinary prose containing the branch name does not block a contribution. + """ + offenders = [] + for path in diff_files(base, head): + markers = [marker for marker in FORK_HUNK_MARKERS if marker_applies_to(marker, path)] + if not markers: + continue + diff = git("diff", f"{base}...{head}", "--", path) + offenders += [ + f"{path}: {line.strip()}" + for line in diff_content_lines(diff, added_only=True) + if any(marker in line for marker in markers) + ] + return offenders + + +def _strip_inline_comment(value: str) -> str: + """Drop a trailing `# ...` comment that is not inside a quoted scalar.""" + out: list[str] = [] + quote = None + for index, char in enumerate(value): + if quote: + out.append(char) + if char == quote: + quote = None + continue + if char in "'\"": + quote = char + out.append(char) + continue + if char == "#" and (index == 0 or value[index - 1] in " \t"): + break + out.append(char) + return "".join(out).rstrip() + + +def _reject_unsupported_yaml(line: str) -> None: + """Fail closed on YAML this reader is not equipped to understand. + + A hand-rolled indentation reader that silently mis-parses is worse than no + reader at all: it reports PASS on a file GitHub reads differently. Anything + outside the small subset these workflows actually use is an error. + """ + if "\t" in line[: len(line) - len(line.lstrip())]: + raise ForkctlError(f"tab in workflow indentation, which YAML forbids: {line!r}") + _, _, value = line.partition(":") + value = _strip_inline_comment(value).strip() + if value[:1] in ("&", "*"): + raise ForkctlError(f"YAML anchor/alias in the `on:` block is not supported: {line!r}") + if value[:1] in ("|", ">"): + raise ForkctlError(f"YAML block scalar in the `on:` block is not supported: {line!r}") + if value.startswith("{"): + raise ForkctlError(f"YAML flow mapping in the `on:` block is not supported: {line!r}") + if line.lstrip().startswith("<<"): + raise ForkctlError(f"YAML merge key in the `on:` block is not supported: {line!r}") + + +def workflow_branch_filters(text: str) -> dict[str, list[str]]: + """Map trigger -> branches under `on..branches` of a workflow. + + A deliberately small indentation reader rather than a YAML dependency: this + script must stay stdlib-only, and it only needs one well-known subtree. It + raises rather than guessing whenever the input leaves that subset, and it + matches the `branches` key exactly -- `branches-ignore` is a different key + with the opposite meaning, so treating it as a prefix match let a commit + swap one for the other and keep the marker line intact. + """ + raw = [line.rstrip() for line in text.splitlines() if line.strip() and not line.lstrip().startswith("#")] + lines = [_strip_inline_comment(line) for line in raw] + lines = [line for line in lines if line.strip()] + filters: dict[str, list[str]] = {} + + on_keys = [i for i, line in enumerate(lines) if re.match(r"^(on|'on'|\"on\"):", line)] + if len(on_keys) > 1: + raise ForkctlError(f"workflow declares `on:` {len(on_keys)} times; which triggers apply is ambiguous") + if not on_keys: + return filters + on_index = on_keys[0] + _reject_unsupported_yaml(lines[on_index]) + if lines[on_index].partition(":")[2].strip(): + raise ForkctlError(f"inline `on:` value is not supported: {lines[on_index]!r}") + + body = [] + for line in lines[on_index + 1 :]: + if indent_of(line) == 0: + break + _reject_unsupported_yaml(line) + body.append(line) + if not body: + return filters + + trigger_indent = min(indent_of(line) for line in body) + for start, trigger in _trigger_blocks(body, trigger_indent): + filters.setdefault(trigger, []) + block = _child_lines(body, start, trigger_indent) + if not block: + continue + child_indent = min(indent_of(line) for line in block) + # `branches` counts only as a direct child of the trigger. Accepting it + # at any deeper indent let it be read out of an unrelated nested key. + hits = [ + index + for index, line in enumerate(block) + if indent_of(line) == child_indent and line.strip().partition(":")[0].strip().strip("'\"") == "branches" + ] + if len(hits) > 1: + raise ForkctlError(f"duplicate `branches:` key under `{trigger}:`") + if hits: + filters[trigger] = _branch_values(block, hits[0]) + return filters + + +def indent_of(line: str) -> int: + return len(line) - len(line.lstrip()) + + +def _trigger_blocks(body: list[str], trigger_indent: int) -> list[tuple[int, str]]: + """`(index, name)` of every trigger key in the `on:` block.""" + seen: set[str] = set() + blocks = [] + for index, line in enumerate(body): + if indent_of(line) != trigger_indent: + continue + stripped = line.strip() + if stripped.startswith("- "): + blocks.append((index, stripped[2:].strip().strip("'\""))) + continue + if not stripped.endswith(":"): + raise ForkctlError(f"unrecognised entry in the `on:` block: {line!r}") + name = stripped.rstrip(":").strip().strip("'\"") + if name in seen: + raise ForkctlError(f"duplicate `{name}:` trigger in the `on:` block") + seen.add(name) + blocks.append((index, name)) + return blocks + + +def _child_lines(body: list[str], start: int, trigger_indent: int) -> list[str]: + block = [] + for line in body[start + 1 :]: + if indent_of(line) <= trigger_indent: + break + block.append(line) + return block + + +def _split_flow_list(inline: str) -> list[str]: + """Split `[a, 'b,c']` on commas that are not inside a quoted scalar.""" + items, current, quote = [], [], None + for char in inline: + if quote: + current.append(char) + if char == quote: + quote = None + continue + if char in "'\"": + quote = char + current.append(char) + continue + if char == ",": + items.append("".join(current)) + current = [] + continue + current.append(char) + if quote: + raise ForkctlError(f"unterminated quote in a `branches:` list: {inline!r}") + items.append("".join(current)) + return [item for item in (item.strip() for item in items) if item] + + +def _unquote_branch(value: str) -> str: + """One `branches:` entry, with its quotes removed and its shape checked.""" + if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": + value = value[1:-1] + elif value[:1] in ("'", '"') or value[-1:] in ("'", '"'): + raise ForkctlError(f"unbalanced quotes in a `branches:` entry: {value!r}") + if value.startswith("!"): + # GitHub evaluates patterns in order, so a later `!` silently excludes a + # branch this reader would still report as listed. + raise ForkctlError(f"negative branch pattern is not supported: {value!r}") + return value + + +def _branch_values(body: list[str], index: int) -> list[str]: + """Values of a `branches:` key, inline (`[a, b]`) or as a block list.""" + line = body[index] + indent = indent_of(line) + _, _, inline = line.partition(":") + inline = inline.strip() + if inline: + if not (inline.startswith("[") and inline.endswith("]")): + raise ForkctlError(f"unrecognised inline `branches:` value: {line!r}") + return [_unquote_branch(item) for item in _split_flow_list(inline[1:-1])] + + values = [] + for following in body[index + 1 :]: + if indent_of(following) <= indent: + break + if not following.strip().startswith("- "): + raise ForkctlError(f"unrecognised entry under `branches:`: {following!r}") + values.append(_unquote_branch(following.strip()[2:].strip())) + return values + + +def fork_workflow_triggers(ref: str, path: str) -> list[str]: + """Triggers whose branch filter lists the fork branch, at `ref`.""" + text = show_file(ref, path) + if text is None: + return [] + return sorted( + trigger for trigger, branches in workflow_branch_filters(text).items() if FORK_WORKFLOW_BRANCH in branches + ) + + +def fork_workflow_problems(base: str, head: str = "HEAD") -> list[str]: + """Section 3.2: the fork branch filter must stay put, in `pull_request` only.""" + merge_base = git("merge-base", base, head) + problems = [] + for path in FORK_WORKFLOW_FILES: + before = fork_workflow_triggers(merge_base, path) + after = fork_workflow_triggers(head, path) + if before == after: + continue + if FORK_WORKFLOW_TRIGGER in before and FORK_WORKFLOW_TRIGGER not in after: + problems.append(f"{path}: the fork branch filter was removed from `{FORK_WORKFLOW_TRIGGER}`") + for trigger in sorted(set(after) - set(before)): + problems.append(f"{path}: the fork branch filter was added to `{trigger}`") + return problems + + +def fork_blocks(ref: str, path: str) -> list[str]: + """Every `fork:begin`...`fork:end` block of `path` at `ref`, in order.""" + content = show_file(ref, path) + if content is None: + return [] + blocks, start = [], None + for index, line in enumerate(content.splitlines()): + if FORK_BLOCK_BEGIN in line: + start = index if start is None else start + elif FORK_BLOCK_END in line and start is not None: + blocks.append("\n".join(content.splitlines()[start : index + 1])) + start = None + return blocks + + +def malformed_fork_blocks(ref: str) -> list[str]: + """Shared files whose marker structure is not exactly one trailing block.""" + problems = [] + for path in FORK_BLOCK_FILES: + content = show_file(ref, path) + if content is None: + continue + lines = content.splitlines() + begins = [i for i, line in enumerate(lines) if FORK_BLOCK_BEGIN in line] + ends = [i for i, line in enumerate(lines) if FORK_BLOCK_END in line] + if len(begins) != 1 or len(ends) != 1 or begins[0] > ends[0]: + problems.append( + f"{path}: {len(begins)} fork:begin / {len(ends)} fork:end marker(s), expected one ordered pair" + ) + continue + # The block is appended to upstream content, so nothing may follow it. + # Position, not content, is what a relocation changes, and git is free + # to render a move as relocating the *other* side of the file. + if any(line.strip() for line in lines[ends[0] + 1 :]): + problems.append(f"{path}: content follows the fork:end marker; the fork block must end the file") + return problems + + +def changed_fork_blocks(base: str, head: str = "HEAD") -> list[str]: + merge_base = git("merge-base", base, head) + return [path for path in FORK_BLOCK_FILES if fork_blocks(merge_base, path) != fork_blocks(head, path)] + + +def fork_marker_lines(ref: str, path: str) -> list[str]: + """Every marker line of `path` at `ref`, whether or not it forms a block. + + `fork_blocks()` reports only completed pairs, so a stray marker is invisible + to it -- which let a feature commit tidy up a malformed base and have the + change counted as upstream-clean. + """ + content = show_file(ref, path) + if content is None: + return [] + return [line for line in content.splitlines() if FORK_BLOCK_BEGIN in line or FORK_BLOCK_END in line] + + +def changed_fork_markers(base: str, head: str) -> list[str]: + return [path for path in FORK_BLOCK_FILES if fork_marker_lines(base, path) != fork_marker_lines(head, path)] + + +def fork_state_problems(ref: str) -> list[str]: + """The fork-only state every stack ref must have, stated absolutely. + + A before/after comparison only catches a change made by the commit under + review. It says nothing about a stack that was already wrong, so the end + state is asserted here rather than inferred from a diff. + """ + problems = [] + for path in REQUIRED_FORK_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; required fork-only infrastructure was deleted") + for path in FORK_WORKFLOW_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; a shared workflow was deleted") + continue + triggers = fork_workflow_triggers(ref, path) + if triggers != [FORK_WORKFLOW_TRIGGER]: + found = ", ".join(triggers) if triggers else "no trigger" + problems.append(f"{path}: fork branch filter is on `{found}`, expected `{FORK_WORKFLOW_TRIGGER}` only") + for path in FORK_BLOCK_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; a shared file carrying a fork block was deleted") + continue + blocks = fork_blocks(ref, path) + if len(blocks) != 1: + problems.append(f"{path}: {len(blocks)} fork-only block(s), expected exactly 1") + return problems + + +def strip_fork_block(text: str) -> str: + """`text` with every fork:begin..fork:end block removed. + + Line endings are preserved: normalising them would let a CRLF/LF rewrite of + a shared file compare equal to its upstream parent and ride along in the one + commit that is exempt from Gate 3. + """ + out, inside = [], False + for line in text.splitlines(keepends=True): + if FORK_BLOCK_BEGIN in line: + inside = True + continue + if FORK_BLOCK_END in line: + inside = False + continue + if not inside: + out.append(line) + return "".join(out) + + +def strip_fork_branch(text: str) -> str: + """`text` with the fork branch removed from every `branches:` list.""" + pattern = re.compile(rf",\s*['\"]?{re.escape(FORK_WORKFLOW_BRANCH)}['\"]?") + out = [] + for line in text.splitlines(keepends=True): + if "branches" in line and FORK_WORKFLOW_BRANCH in line: + line = pattern.sub("", line) + out.append(line) + return "".join(out) + + +def smuggled_shared_changes(parent: str, base: str) -> list[str]: + """Shared-file edits in the fork base that are not the fork-only hunk. + + The fork base is allowed to touch shared files -- adding the fork block and + the branch filter is its job -- so a file-name check cannot police it. Each + shared file is therefore compared with its upstream parent after the + fork-only hunk is removed from both; anything left over is a change that + was folded into the base commit where no guard would ever look at it. + """ + problems = [] + for path in SHARED_FILES: + before, after = show_file(parent, path), show_file(base, path) + if before is None and after is None: + continue + if after is None: + problems.append(f"{path} was deleted by the fork base") + continue + if before is None: + problems.append(f"{path} is introduced by the fork base but is registered as shared with upstream") + continue + strip = strip_fork_block if path in FORK_BLOCK_FILES else strip_fork_branch + # The fork block is appended after a blank separator line, so removing it + # leaves exactly that one extra newline. Nothing else -- not a trailing + # newline, not a CRLF rewrite -- may differ. + upstream = strip(before) + separator = "\r\n" if "\r\n" in upstream else "\n" + allowed = (upstream, upstream + separator) if path in FORK_BLOCK_FILES else (upstream,) + if strip(after) not in allowed: + problems.append(f"{path} carries a non-fork-only change in the fork base") + return problems + + +def subject_problems(subject: str) -> list[str]: + problems = [] + if subject != subject.strip(): + problems.append("has leading or trailing whitespace") + if not CONVENTIONAL_RE.match(subject): + problems.append(f"is not Conventional Commits ({'|'.join(CONVENTIONAL_TYPES)})(scope): summary") + if len(subject) > MAX_SUBJECT_LENGTH: + problems.append(f"is {len(subject)} chars, over the {MAX_SUBJECT_LENGTH}-char limit") + return problems + + +def working_tree_clean() -> bool: + return git("status", "--porcelain") == "" + + +def patch_id(rev: str) -> str | None: + """Content identity of a commit, ignoring its message and its parents.""" + diff = subprocess.run( + ["git", "--no-pager", "diff", f"{rev}^", rev], + check=False, + capture_output=True, + text=True, + ) + if diff.returncode != 0 or not diff.stdout: + return None + proc = subprocess.run( + ["git", "patch-id", "--stable"], + check=False, + input=diff.stdout, + capture_output=True, + text=True, + ) + return proc.stdout.split()[0] if proc.stdout.strip() else None + + +def mirror_ref(slug: str) -> str | None: + for candidate in (f"origin/mirror/{slug}", f"mirror/{slug}"): + if ref_exists(candidate): + return candidate + return None + + +def merges_cleanly(ref: str, onto: str) -> bool: + """Whether ref still applies to onto, without touching the working tree.""" + proc = subprocess.run( + ["git", "--no-pager", "merge-tree", "--write-tree", onto, ref], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode not in (0, 1): + # Git < 2.38 has no --write-tree; it must not read as "no conflict". + raise ForkctlError(f"git merge-tree --write-tree is unavailable: {proc.stderr.strip()}") + return proc.returncode == 0 + + +def mirror_drift(slug: str, upstream: str, stack_patch_ids: set[str]) -> list[str]: + """Sections 6.2 events 1 and 2: the only two things that need action.""" + ref = mirror_ref(slug) + if ref is None: + return ["not fetched locally; run git fetch origin"] + + notes = [] + carried = git_lines("rev-list", f"{upstream}..{ref}") + if len(carried) != 1: + notes.append( + f"carries {len(carried)} commits over upstream, expected exactly 1 -- " + "the open PR offers more than the stacked commit; rebuild it" + ) + + try: + if git("rev-parse", f"{ref}^") != git("rev-parse", upstream) and not merges_cleanly(ref, upstream): + notes.append("event 1: no longer merges onto upstream -- rebuild onto the new tip") + except ForkctlError as exc: + notes.append(f"could not test whether it still merges onto upstream: {exc}") + + identity = patch_id(ref) + if identity is None: + notes.append("could not compute a patch-id for the mirror tip; drift is unverified") + elif identity not in stack_patch_ids: + notes.append( + "event 2: content differs from every stack commit -- the open PR offers " + "something we no longer run; rebuild and force-push" + ) + return notes + + +def current_branch() -> str: + branch = git("rev-parse", "--abbrev-ref", "HEAD") + return git("rev-parse", "HEAD") if branch == "HEAD" else branch + + +def cmd_fork_only(args: argparse.Namespace) -> int: + for path in FORK_ONLY_PATHS: + print(path) + for prefix in FORK_ONLY_PREFIXES: + print(f"{prefix}*") + if args.verbose: + print("\nshared files carrying a fork-only hunk:") + for path in SHARED_FILES: + print(f" {path}") + print("\nfork-only hunk markers:") + for marker in FORK_HUNK_MARKERS: + print(f" {marker}") + return 0 + + +def cmd_check_commit(args: argparse.Namespace) -> int: + """Sections 4.4 and 4.5: branch shape and upstream-cleanliness pre-flight.""" + report = Report("check-commit", args.format) + if not require_refs(report, args.base): + return report.finish() + + commits = git_lines("rev-list", f"{args.base}..HEAD") + one_commit = report.check( + len(commits) == 1, + "exactly one commit on the branch", + f"{len(commits)} commits on the branch, expected exactly 1", + f"git rebase {args.base} && git reset --soft {args.base} && git commit -F msg", + ) + + # One commit *ahead* is not the same as rebased onto the current base: a + # branch cut weeks ago is still one commit ahead of where it started. + if one_commit: + report.check( + git("rev-parse", "HEAD^") == git("rev-parse", args.base), + f"rebased directly onto {args.base}", + f"the commit is not parented on the current {args.base} tip", + f"git rebase {args.base}", + ) + + subject = git("log", "-1", "--format=%s") if commits else "" + problems = subject_problems(subject) + report.check( + not problems, + f"subject is well formed: {subject!r}", + f"subject {subject!r} " + "; ".join(problems), + "Conventional Commits, imperative mood, <= 72 chars (section 4.4)", + ) + + is_chore = subject.startswith(FORK_COMMIT_PREFIX) + if args.chore and not is_chore: + report.fail(f"--chore given but the subject is not {FORK_COMMIT_PREFIX!r}: {subject!r}") + if is_chore and not args.chore: + report.warn( + "subject is a chore(fork) commit but --chore was not passed; the " + "fork-only checks below are expected to trip for a fork-only change" + ) + + # Structural validity of the markers is checked in both modes. A chore commit + # may legitimately rewrite a block, but if it leaves two of them behind, every + # later feature commit can edit the second one unnoticed. + malformed = malformed_fork_blocks("HEAD") + report.check( + not malformed, + "fork-only markers are well formed", + "fork-only marker structure is broken: " + "; ".join(malformed), + "each shared file carries exactly one fork:begin/fork:end pair", + ) + + # Asserted absolutely, in both modes: a before/after comparison cannot see a + # fork-only hunk that was already missing before this commit touched it. + state = fork_state_problems("HEAD") + report.check( + not state, + "fork-only state is intact", + "fork-only state is broken: " + "; ".join(state), + "the branch must keep every required fork-only file, the fork block and the branch filter", + ) + + if args.chore: + report.info("fork-only file and hunk checks: exempt (chore(fork) branch)") + return report.finish() + + offenders = fork_only_files_in(args.base) + report.check( + not offenders, + "no registered fork-only file in the diff", + f"fork-only file(s) in the feature diff: {', '.join(offenders)}", + "move them into a separate chore(fork): commit", + ) + + workflow_problems = fork_workflow_problems(args.base) + report.check( + not workflow_problems, + "fork-only branch filters are untouched in shared workflows", + "; ".join(workflow_problems), + "leave the fork branch in the pull_request filter; the rest of the file is yours", + ) + + merge_base = git("merge-base", args.base, "HEAD") + blocks = sorted(set(changed_fork_blocks(args.base)) | set(changed_fork_markers(merge_base, "HEAD"))) + report.check( + not blocks, + "fork-only blocks in shared files are unchanged", + f"fork-only block content changed in: {', '.join(blocks)}", + "revert the fork:begin/fork:end block; it belongs in the chore(fork) commit", + ) + return report.finish() + + +def verify_stack(report: Report, mirror_ref: str, stack_ref: str) -> str | None: + """Sections 3.3 and 7: stack invariants. Returns the fork base, or None.""" + if not require_refs(report, mirror_ref, stack_ref): + return None + + mirror = git("rev-parse", mirror_ref) + stack = git("rev-parse", stack_ref) + + contains = git_rc("merge-base", "--is-ancestor", mirror, stack) == 0 + if not report.check( + contains, + f"{stack_ref} contains the {mirror_ref} tip", + f"{stack_ref} does not contain the {mirror_ref} tip", + "rebase the stack (copilot-instructions.md section 7)", + ): + return None + + rng = f"{mirror}..{stack}" + + merges = git_lines("rev-list", "--min-parents=2", "--max-count=1", rng) + report.check( + not merges, + "stack is linear (no merge commits)", + f"merge commit on the stack: {merges[0] if merges else ''}", + "rebuild the stack with rebase; never merge into integration/3.0", + ) + + subjects = git_lines("log", "--format=%s", rng) + + # Read subjects, not whole messages: `git log --grep` matches the body too, + # and a leading space is enough to slip past a prefix test. + ragged = [s for s in subjects if s != s.strip()] + report.check( + not ragged, + "no commit subject has stray leading or trailing whitespace", + f"commit subject with stray whitespace: {ragged[0]!r}" if ragged else "", + "reword it; whitespace hides a subject from every prefix check", + ) + + fixups = [s for s in subjects if s.strip().startswith(("fixup!", "squash!"))] + report.check( + not fixups, + "no stray fixup! commit", + f"stray fixup!/squash! commit on the stack: {fixups[0]!r}" if fixups else "", + "finish the autosquash rebase before pushing", + ) + + chore_count = sum(1 for s in subjects if s.startswith(FORK_COMMIT_PREFIX)) + report.check( + chore_count == 1, + "exactly one chore(fork): commit", + f"{chore_count} chore(fork): commits on the stack, expected 1", + "fold fork-only changes into the base commit (section 3.3)", + ) + + commits = git_lines("rev-list", rng) + if not commits: + report.fail(f"{stack_ref} has no commits over {mirror_ref}") + return None + + base = commits[-1] + base_subject = git("log", "-1", "--format=%s", base) + report.check( + base_subject.startswith(FORK_COMMIT_PREFIX), + f"oldest stack commit is the fork base ({base[:9]})", + f"oldest stack commit {base[:9]} is not the fork base: {base_subject!r}", + ) + + parents = git("rev-list", "--parents", "-n", "1", base).split() + if report.check( + len(parents) == 2, + "fork base has exactly one parent", + f"fork base {base[:9]} does not have exactly one parent", + ): + report.check( + parents[1] == mirror, + "fork base is parented on the mirror tip", + f"fork base {base[:9]} is parented on {parents[1][:9]}, not the mirror tip {mirror[:9]}", + ) + + # Keeps the hardcoded list honest: anything the fork base introduces is by + # definition fork-only, so an unregistered file here means the guards that + # protect feature commits have a blind spot. + unregistered = [ + path + for path in diff_files(mirror, base) + if not is_fork_only_path(path) and path not in SHARED_FILES and path not in FORK_WORKFLOW_FILES + ] + report.check( + not unregistered, + "every file in the fork base is registered as fork-only", + f"fork base touches unregistered file(s): {', '.join(unregistered)}", + "add them to FORK_ONLY_PATHS/FORK_ONLY_PREFIXES in forkctl.py, or move them under .github/fork/", + ) + + malformed = malformed_fork_blocks(base) + malformed_fork_blocks(stack) + report.check( + not malformed, + "fork-only markers are well formed on the stack", + "fork-only marker structure is broken: " + "; ".join(malformed), + "each shared file carries exactly one fork:begin/fork:end pair", + ) + + state = fork_state_problems(stack) + report.check( + not state, + "fork-only state is intact at the stack tip", + "fork-only state is broken: " + "; ".join(state), + "the stack tip must keep every required fork-only file, the fork block and the branch filter", + ) + + smuggled = smuggled_shared_changes(mirror, base) + report.check( + not smuggled, + "fork base changes shared files only inside the fork-only hunk", + "fork base smuggles a shared-file change: " + "; ".join(smuggled), + "move it to its own feature commit where the guards can see it", + ) + + # The checks above describe the tip. Feature commits are what get merged, so + # each is re-checked against its own parent exactly as check-commit would. + contaminated = [] + for commit in commits[:-1]: + parent = f"{commit}^" + offenders = fork_only_files_in(parent, commit) + blocks = set(changed_fork_blocks(parent, commit)) | set(changed_fork_markers(parent, commit)) + offenders += [f"{path} (fork block)" for path in sorted(blocks)] + offenders += fork_workflow_problems(parent, commit) + if offenders: + contaminated.append(f"{commit[:9]}: {'; '.join(offenders)}") + report.check( + not contaminated, + f"every feature commit on the stack is upstream-clean ({len(commits) - 1} checked)", + "contaminated feature commit(s): " + "; ".join(contaminated), + "fork-only content belongs in the base commit, not in a feature commit", + ) + + return None if report.failed else base + + +def cmd_check_stack(args: argparse.Namespace) -> int: + # With --print-base the SHA is the output, so the report belongs on stderr and + # `BASE=$(forkctl check-stack --print-base) || exit 1` captures only the SHA. + report = Report("check-stack", args.format, sys.stderr if args.print_base else sys.stdout) + base = verify_stack(report, args.mirror, args.stack) + if base is None: + return report.finish() + + if args.print_base: + print(base) + github_output = os.environ.get("GITHUB_OUTPUT") + if args.github_output and github_output: + with open(github_output, "a", encoding="utf-8") as handle: + handle.write(f"base={base}\n") + + return report.finish() + + +def resolve_feature_commit(report: Report, args: argparse.Namespace) -> str | None: + """Section 6: resolve by subject, never by a recorded SHA that rebases stale.""" + if args.sha: + if not ref_exists(args.sha): + report.fail(f"commit not found: {args.sha}") + return None + sha = git("rev-parse", args.sha) + # Gate 3 must prove *the* commit being merged. Naming an older stack + # commit while a different one is checked out certified the wrong diff, + # so --sha has to be the single commit of the branch under review; use + # --subject to rebuild the mirror of a commit that is already stacked. + head = current_branch() + if head_is_detached(): + report.fail( + "HEAD is detached", + "check out the feature branch under review; --subject rebuilds an already-stacked commit", + ) + return None + if sha != git("rev-parse", "HEAD"): + report.fail( + f"--sha {sha[:9]} is not HEAD ({git('rev-parse', 'HEAD')[:9]})", + "Gate 3 proves the commit under review; use --subject to rebuild a stacked commit", + ) + return None + parented_on_tip = git_rc("rev-parse", "--verify", "--quiet", f"{sha}^") == 0 and git( + "rev-parse", f"{sha}^" + ) == git("rev-parse", args.stack) + if not parented_on_tip: + report.fail( + f"{head} is not a single commit on top of the {args.stack} tip", + "squash to one commit and rebase onto the current stack tip (section 4.4)", + ) + return None + return sha + + entries = git_lines("log", "--format=%H%x00%s", f"{args.mirror}..{args.stack}") + matches = [line.split("\0", 1)[0] for line in entries if line.split("\0", 1)[1] == args.subject] + if len(matches) != 1: + report.fail(f"subject matches {len(matches)} stack commits, expected exactly 1: {args.subject!r}") + return None + return matches[0] + + +def restore_checkout(report: Report, started_on: str) -> None: + """Return to where the caller was, and say so if that proved impossible. + + Printing "back on " without checking is how an interrupted run left + people on a conflicted detached HEAD believing it had cleaned up. + """ + if git_rc("cherry-pick", "--abort") == 0: + report.info("aborted an in-progress cherry-pick") + if git_rc("switch", started_on) == 0 or git_rc("switch", "--detach", started_on) == 0: + report.info(f"back on {started_on}") + return + report.fail( + f"could not return to {started_on}; the checkout is detached", + f"resolve the working tree, then: git switch {started_on}", + ) + + +SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$") + + +def cmd_mirror(args: argparse.Namespace) -> int: + """Gate 3 and section 6: rebuild mirror/ and prove it is upstream-clean.""" + report = Report(f"mirror/{args.slug}", args.format) + + # Validated before anything moves: a slug git rejects as a branch name would + # otherwise only surface after the checkout, stranding the user detached. + if not SLUG_RE.match(args.slug): + report.fail(f"invalid slug {args.slug!r}", "short, lowercase, hyphenated: lane-heading-obs") + return report.finish() + if not working_tree_clean(): + report.fail("working tree is dirty", "commit or stash first") + return report.finish() + if not require_refs(report, args.upstream, args.mirror, args.stack): + return report.finish() + + sha = resolve_feature_commit(report, args) + if sha is None: + return report.finish() + report.info(f"feature commit {sha[:9]} {git('log', '-1', '--format=%s', sha)}") + + branch = f"mirror/{args.slug}" + started_on = current_branch() + + # Build detached and only move the branch once every check has passed. An + # existing mirror/ may carry the commit an upstream PR is serving, so + # it must not be destroyed to find out whether its replacement is valid. + stayed = False + try: + # Inside the try: an interrupt between the switch and the first guarded + # statement would otherwise skip the restoration entirely. + git("switch", "--detach", args.upstream) + if git_rc("cherry-pick", sha) != 0: + git_rc("cherry-pick", "--abort") + report.fail( + "cherry-pick onto pure upstream failed -- the commit is entangled with fork-only state", + "fix it on the feature branch and re-run", + ) + else: + report.ok("cherry-picks onto pure upstream without conflict") + + offenders = fork_only_files_in(args.upstream) + report.check( + not offenders, + "no registered fork-only file in the mirror diff", + f"fork-only file(s) in the mirror diff: {', '.join(offenders)}", + ) + + markers = fork_markers_anywhere(args.upstream) + report.check( + not markers, + "no fork-only marker added anywhere in the mirror diff", + f"fork-only marker in the mirror diff: {markers[0] if markers else ''}", + ) + + files = diff_files(args.upstream) + report.info(f"mirror diff touches {len(files)} file(s):") + for path in files: + report.info(f" {path}") + + candidate = git("rev-parse", "HEAD") + if report.failed: + report.info(f"{branch} was left untouched; nothing was published") + else: + if ref_exists(branch) and git("rev-parse", branch) != candidate: + report.info(f"{branch} moved from {git('rev-parse', branch)[:9]} to {candidate[:9]}") + git("branch", "-f", branch, candidate) + if args.stay: + git("switch", branch) + stayed = True + report.info(f"staying on {branch}") + finally: + # Unconditional: an exception or a Ctrl-C after the checkout must not + # leave the caller detached on a half-built or conflicted mirror. + if not stayed: + restore_checkout(report, started_on) + + return report.finish() + + +def upstream_prs_for(slug: str) -> list[dict]: + """Every upstream PR whose head is our mirror branch, in our fork only. + + GitHub's `--head` filter matches by branch *name* across owners, so another + fork pushing the same slug would otherwise be mistaken for ours. + """ + prs = gh_json( + "pr", + "list", + "--repo", + UPSTREAM_REPO, + "--head", + f"mirror/{slug}", + "--state", + "all", + "--limit", + "100", + "--json", + "number,state,headRepositoryOwner", + ) + return [pr for pr in prs if (pr.get("headRepositoryOwner") or {}).get("login") == FORK_OWNER] + + +def cmd_delete_mirror(args: argparse.Namespace) -> int: + """Section 6.2: deleting the head branch closes an open upstream PR.""" + report = Report(f"delete mirror/{args.slug}", args.format) + if not gh_available(): + report.fail("gh is not installed; cannot verify the upstream PR state") + return report.finish() + + try: + prs = upstream_prs_for(args.slug) + except ForkctlError as exc: + report.fail(f"could not verify the upstream PR state: {exc}", "retry; never delete unverified") + return report.finish() + + live = [pr for pr in prs if pr["state"] not in TERMINAL_PR_STATES] + if live: + report.fail( + f"upstream PR(s) {', '.join('#' + str(pr['number']) for pr in live)} for mirror/{args.slug} " + f"are {', '.join(pr['state'] for pr in live)} -- deleting the branch would " + "close them and read as a silent withdrawal" + ) + return report.finish() + states = ", ".join(pr["state"] for pr in prs) or "NONE" + report.ok(f"every upstream PR for this slug is terminal ({states}); deletion is safe") + + if args.dry_run: + report.info("dry run; nothing deleted") + return report.finish() + + if git_rc("push", "origin", "--delete", f"mirror/{args.slug}") != 0: + report.fail("deleting the branch on origin failed", "check the remote and retry") + return report.finish() + if git_lines("ls-remote", "--heads", "origin", f"mirror/{args.slug}"): + report.fail("the branch still exists on origin after the delete") + return report.finish() + report.ok("deleted on origin") + + # The remote delete is the part that matters for the upstream PR; a failed + # local delete (usually because it is the current branch) is still a failure + # to report rather than fold into a single cheerful line. + if not ref_exists(f"mirror/{args.slug}"): + report.ok("no local branch left") + return report.finish() + report.check( + git_rc("branch", "-D", f"mirror/{args.slug}") == 0 and not ref_exists(f"mirror/{args.slug}"), + "deleted locally", + f"the local branch mirror/{args.slug} could not be deleted", + "switch away from it and run `git branch -D` by hand", + ) + return report.finish() + + +def cmd_check_view(args: argparse.Namespace) -> int: + """Section 2: fork-only/3.0 is a derived view, republished by hand.""" + report = Report("check-view", args.format) + base = verify_stack(report, args.mirror, args.stack) + if base is None: + return report.finish() + report.ok(f"stack is sound; fork base is {base[:9]}") + + fix = f"git push --force origin {base}:refs/heads/{VIEW_BRANCH}" + published = "" + for line in git_lines("ls-remote", "origin", f"refs/heads/{VIEW_BRANCH}"): + published = line.split("\t")[0] if "\t" in line else line.split()[0] + + if not published: + report.fail(f"{VIEW_BRANCH} does not exist", fix) + else: + report.check( + published == base, + f"{VIEW_BRANCH} is current at {base[:9]}", + f"{VIEW_BRANCH} is stale ({published[:9]}); the fork base is {base[:9]}", + fix, + ) + + if not gh_available(): + report.fail( + "gh is not installed; the standing PR contract could not be verified", + "install gh, or run this check where gh is available", + ) + return report.finish() + + try: + prs = gh_json( + "pr", + "list", + "--repo", + FORK_REPO, + "--head", + VIEW_BRANCH, + "--state", + "all", + "--limit", + "100", + "--json", + "number,state,baseRefName,isDraft,labels", + ) + except ForkctlError as exc: + report.fail(f"could not query the standing view PR: {exc}", "retry; an unverified contract is not a pass") + return report.finish() + + merged = [pr for pr in prs if pr["state"] == "MERGED"] + if not report.check( + not merged, + f"no {VIEW_BRANCH} PR was ever merged", + f"a {VIEW_BRANCH} PR was MERGED into the mirror -- the mirror is contaminated: " + + ", ".join(f"#{pr['number']}" for pr in merged), + "reset 3.0 to upstream/3.0 (admin action, section 7)", + ): + return report.finish() + + open_prs = [pr for pr in prs if pr["state"] == "OPEN"] + if not open_prs: + report.fail( + f"no standing {VIEW_BRANCH} PR is open -- section 2 requires one, opened by hand", + f"open a draft PR {VIEW_BRANCH} -> 3.0 labelled do-not-merge", + ) + return report.finish() + if not report.check( + len(open_prs) == 1, + "exactly one standing view PR is open", + f"{len(open_prs)} open {VIEW_BRANCH} PRs; exactly one is expected", + ): + return report.finish() + + pr = open_prs[0] + labels = {label["name"] for label in pr["labels"]} + broken = [] + if pr["baseRefName"] != "3.0": + broken.append(f"base={pr['baseRefName']}, expected 3.0") + if not pr["isDraft"]: + broken.append("not a draft") + if "do-not-merge" not in labels: + broken.append("missing the do-not-merge label") + # Both labels together are the in-flight-process-change signal that `doctor` + # queries; a permanent PR carrying both would raise it forever. + if "do-not-mirror" in labels: + broken.append("carries do-not-mirror, which a standing PR must never have") + + report.check( + not broken, + f"standing view PR #{pr['number']} keeps its contract", + f"standing view PR #{pr['number']} breaks its contract: " + "; ".join(broken), + ) + return report.finish() + + +def cmd_status(args: argparse.Namespace) -> int: + """Sections 6.2, 6.3 and 7: divergence report and what is in flight.""" + report = Report("status", "text") + if not require_refs(report, args.mirror, args.stack): + return report.finish() + + rng = f"{args.mirror}..{args.stack}" + subjects = git_lines("log", "--format=%s", rng) + fork_only = sum(1 for s in subjects if s.startswith(FORK_COMMIT_PREFIX)) + + print(f" total gap : {len(subjects)}") + print(f" fork-only : {fork_only} (permanent floor, never drive to zero)") + print(f" feature gap : {len(subjects) - fork_only}") + print("\n stack (newest first):") + for line in git_lines("log", "--format=%h %s", rng): + print(f" {line}") + + if ref_exists(args.upstream): + stale = [line[2:] for line in git_lines("cherry", args.upstream, args.stack) if line.startswith("- ")] + if stale: + report.warn(f"{len(stale)} stack commit(s) already upstream (section 6.3)") + for sha in stale: + print(f" - {git('log', '-1', '--format=%h %s', sha)}") + else: + print("\n no stale commits: nothing we carry is already upstream") + else: + report.fail(f"{args.upstream} not fetched; the stale-commit audit could not run", "git fetch upstream") + return report.finish() + + remote = git_lines("ls-remote", "--heads", "origin", "mirror/*") + slugs = [line.split("refs/heads/mirror/", 1)[1] for line in remote if "refs/heads/mirror/" in line] + + if not gh_available(): + print("\n mirror branches on origin:") + for name in slugs or ["(none)"]: + print(f" mirror/{name}") + report.fail("gh is not installed; the upstream PR inventory could not run") + return report.finish() + + try: + prs = gh_json( + "pr", + "list", + "--repo", + UPSTREAM_REPO, + "--state", + "open", + "--limit", + "100", + "--json", + "number,headRefName,headRepositoryOwner,title", + ) + except ForkctlError as exc: + report.fail(f"could not query upstream PRs: {exc}", "retry; an unverified inventory is not a pass") + return report.finish() + + open_prs = {pr["headRefName"]: pr for pr in prs if (pr.get("headRepositoryOwner") or {}).get("login") == FORK_OWNER} + + stack_patch_ids = { + identity for identity in (patch_id(sha) for sha in git_lines("rev-list", rng)) if identity is not None + } + + print("\n mirror branches on origin:") + if not slugs: + print(" (none)") + for name in slugs: + pr = open_prs.get(f"mirror/{name}") + if pr is None: + print(f" mirror/{name} -- no open upstream PR (finished, or never opened)") + continue + print(f" mirror/{name} -- upstream #{pr['number']} open") + for note in mirror_drift(name, args.upstream, stack_patch_ids): + # Drift is a warning; being unable to measure it is a failure. + if note.startswith(("could not", "not fetched")): + report.fail(f"mirror/{name}: {note}") + else: + report.warn(f"mirror/{name}: {note}") + + orphans = set(open_prs) - {f"mirror/{name}" for name in slugs} + for head in sorted(orphans): + report.warn(f"{head} has an open upstream PR but no branch on origin") + + return report.finish() + + +def cmd_doctor(args: argparse.Namespace) -> int: + """Section 4.1: is this clone set up, and is the process itself mid-change?""" + report = Report("doctor", args.format) + + report.check( + git_rc("remote", "get-url", "upstream") == 0, + "upstream remote is configured", + "upstream remote is missing", + f"git remote add upstream {UPSTREAM_URL}", + ) + + venv = ".venv/Scripts/python.exe" if os.name == "nt" else ".venv/bin/python" + report.check( + os.path.exists(venv), + f"virtualenv present ({venv})", + f"no virtualenv at {venv}", + "uv venv && uv pip install -e .", + ) + + if not gh_available(): + report.fail( + "gh is not installed; cannot tell whether the process is mid-change", + "install gh; an unchecked process state is not a pass", + ) + return report.finish() + + try: + inflight = gh_json( + "pr", + "list", + "--repo", + FORK_REPO, + "--state", + "open", + "--label", + "do-not-mirror", + "--label", + "do-not-merge", + "--json", + "number,title", + ) + except ForkctlError as exc: + report.fail(f"could not query in-flight process PRs: {exc}", "retry before relying on the written process") + return report.finish() + + if not inflight: + report.ok("no in-flight process change; copilot-instructions.md is current") + return report.finish() + + report.warn( + "the fork process is mid-change; these PRs supersede " + "copilot-instructions.md wherever they overlap -- read them first:" + ) + for pr in inflight: + report.info(f" #{pr['number']} {pr['title']}") + return report.finish() + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="forkctl", + description="Fork-process checks for ASCS-eV/PufferDrive.", + epilog="The process itself is described in .github/copilot-instructions.md.", + ) + sub = parser.add_subparsers(dest="command", required=True) + + def add_fmt(p: argparse.ArgumentParser) -> None: + p.add_argument( + "--format", + choices=("text", "github"), + default="text", + help="output style; 'github' emits ::error:: annotations", + ) + + p = sub.add_parser("doctor", help="clone setup and in-flight process changes") + add_fmt(p) + p.set_defaults(func=cmd_doctor) + + p = sub.add_parser( + "check-commit", + help="one commit, no fork-only content (sections 4.4, 4.5)", + ) + p.add_argument("--base", default=STACK_REF) + p.add_argument( + "--chore", + action="store_true", + help="fork-only branch: skip the fork-only checks, keep the one-commit check", + ) + add_fmt(p) + p.set_defaults(func=cmd_check_commit) + + p = sub.add_parser("check-stack", help="stack invariants (sections 3.3, 7)") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--print-base", action="store_true", help="print the fork base SHA") + p.add_argument( + "--github-output", + action="store_true", + help="also append base= to $GITHUB_OUTPUT", + ) + add_fmt(p) + p.set_defaults(func=cmd_check_stack) + + p = sub.add_parser("check-view", help="fork-only/3.0 is current and its standing PR is intact") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + add_fmt(p) + p.set_defaults(func=cmd_check_view) + + p = sub.add_parser("mirror", help="rebuild mirror/ and prove it is upstream-clean (gate 3)") + p.add_argument("slug") + group = p.add_mutually_exclusive_group(required=True) + group.add_argument("--sha", help="feature commit to cherry-pick") + group.add_argument("--subject", help="resolve the commit by its exact subject") + p.add_argument("--upstream", default=UPSTREAM_REF) + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--stay", action="store_true", help="stay on the mirror branch") + add_fmt(p) + p.set_defaults(func=cmd_mirror) + + p = sub.add_parser("delete-mirror", help="delete mirror/ only if its upstream PR is terminal") + p.add_argument("slug") + p.add_argument("--dry-run", action="store_true") + add_fmt(p) + p.set_defaults(func=cmd_delete_mirror) + + p = sub.add_parser("status", help="divergence report and in-flight mirrors") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--upstream", default=UPSTREAM_REF) + p.set_defaults(func=cmd_status) + + p = sub.add_parser("fork-only", help="print the canonical fork-only path list") + p.add_argument("-v", "--verbose", action="store_true") + p.set_defaults(func=cmd_fork_only) + + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + return args.func(args) + except ForkctlError as exc: + print(f"forkctl: {exc}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/fork/test_forkctl.py b/.github/fork/test_forkctl.py new file mode 100644 index 0000000000..54a270164a --- /dev/null +++ b/.github/fork/test_forkctl.py @@ -0,0 +1,1255 @@ +#!/usr/bin/env python3 +"""Tests for forkctl. Fork-only; never cherry-pick upstream. + +Run: python .github/fork/test_forkctl.py + +Every guard is asserted in both directions -- it passes a well-formed stack and +it *fails* a malformed one. A guard that cannot be shown to fail is not a guard, +and these particular guards are what stand between a bad commit and the branch +BMW pins, so they are exercised against throwaway repositories built to break +them rather than against prose. + +Stdlib only, so it runs anywhere git does, with no test dependencies. +""" + +from __future__ import annotations + +import contextlib +import io +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import forkctl + + +FEATURE = "feat(drive): add a thing" +CHORE = "chore(fork): define the ASCS-eV fork process" + +# The fixture repo is built once and copied per test. Spawning git costs ~0.2s +# on a Windows checkout with live virus scanning, so rebuilding it per test put +# the suite over five minutes; copying the directory keeps full isolation. +_TEMPLATE: tempfile.TemporaryDirectory | None = None +_TEMPLATE_REPO: Path | None = None +_BASE_SHA = "" +_FEATURE_SHA = "" + + +def git(repo: Path, *args: str) -> str: + proc = subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True, check=True) + return proc.stdout.strip() + + +def write(repo: Path, relpath: str, text: str) -> None: + path = repo / relpath + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + + +def commit(repo: Path, subject: str, files: dict[str, str]) -> str: + for relpath, text in files.items(): + write(repo, relpath, text) + git(repo, "add", "-A") + git(repo, "commit", "-m", subject) + return git(repo, "rev-parse", "HEAD") + + +def run(*argv: str) -> tuple[int, str]: + """Invoke forkctl in-process and capture what it printed.""" + buf = io.StringIO() + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + code = forkctl.main(list(argv)) + return code, buf.getvalue() + + +CI_YML_UPSTREAM = """\ +name: CI + +on: + pull_request: + branches: ['main', '3.0'] + push: + branches: ['main', '3.0'] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - run: echo build +""" + +CI_YML_FORKED = CI_YML_UPSTREAM.replace( + " pull_request:\n branches: ['main', '3.0']", + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']", +) + +INSTALL_YML_UPSTREAM = CI_YML_UPSTREAM.replace("name: CI", "name: Installation Test") +INSTALL_YML_FORKED = CI_YML_FORKED.replace("name: CI", "name: Installation Test") + +# The fork base adds exactly one block to each shared doc, and nothing else. +AGENTS_UPSTREAM = "## Coding Standards\nexplicit names\n" +AGENTS_FORKED = AGENTS_UPSTREAM + "\n\nfork process\n\n" + + +def upstream_tree() -> dict[str, str]: + return { + "README.md": "upstream\n", + "AGENTS.md": AGENTS_UPSTREAM, + "CLAUDE.md": AGENTS_UPSTREAM, + ".github/workflows/ci.yml": CI_YML_UPSTREAM, + ".github/workflows/install.yml": INSTALL_YML_UPSTREAM, + } + + +def fork_base_tree() -> dict[str, str]: + """Everything the real fork base carries, including every required file. + + The required-file invariant is only meaningful if the well-formed fixture + actually satisfies it; an incomplete fixture made `check-stack` pass on a + base that had lost the detector. + """ + tree = { + "AGENTS.md": AGENTS_FORKED, + "CLAUDE.md": AGENTS_FORKED, + ".github/workflows/ci.yml": CI_YML_FORKED, + ".github/workflows/install.yml": INSTALL_YML_FORKED, + } + for path in forkctl.REQUIRED_FORK_FILES: + tree[path] = f"fork-only: {path}\n" + return tree + + +def setUpModule() -> None: + global _TEMPLATE, _TEMPLATE_REPO, _BASE_SHA, _FEATURE_SHA + _TEMPLATE = tempfile.TemporaryDirectory() + repo = Path(_TEMPLATE.name) / "template" + repo.mkdir() + + git(repo, "-c", "init.defaultBranch=main", "init", "-q") + git(repo, "config", "user.email", "test@example.invalid") + git(repo, "config", "user.name", "Fork Test") + git(repo, "config", "commit.gpgsign", "false") + + commit(repo, "upstream: initial", upstream_tree()) + git(repo, "branch", "up") + git(repo, "branch", "mir") + + git(repo, "switch", "-q", "-c", "stack", "mir") + _BASE_SHA = commit(repo, CHORE, fork_base_tree()) + _FEATURE_SHA = commit(repo, FEATURE, {"sim.c": "int main(void){}\n"}) + _TEMPLATE_REPO = repo + + +def tearDownModule() -> None: + if _TEMPLATE is not None: + _TEMPLATE.cleanup() + + +class ForkRepoTest(unittest.TestCase): + """A throwaway repo shaped like the real fork: mirror + stack + upstream.""" + + def setUp(self) -> None: + self._cwd = os.getcwd() + self._tmp = tempfile.TemporaryDirectory() + self.repo = Path(self._tmp.name) / "repo" + shutil.copytree(_TEMPLATE_REPO, self.repo) + self.base = _BASE_SHA + self.feature = _FEATURE_SHA + os.chdir(self.repo) + + def tearDown(self) -> None: + os.chdir(self._cwd) + self._tmp.cleanup() + + def branch_from_stack(self, name: str) -> None: + git(self.repo, "switch", "-q", "-c", name, "stack") + + def stacked_feature(self, subject: str, tree: dict[str, str], name: str = "feature/x") -> str: + """A one-commit branch on top of the stack tip, exactly as Gate 3 expects.""" + self.branch_from_stack(name) + return commit(self.repo, subject, tree) + + +class TestCheckStack(ForkRepoTest): + def test_passes_on_a_well_formed_stack(self) -> None: + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + self.assertIn("exactly one chore(fork): commit", out) + + def test_prints_the_fork_base(self) -> None: + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack", "--print-base") + self.assertEqual(code, 0, out) + self.assertIn(self.base, out) + + def test_fails_on_a_second_chore_commit(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "chore(fork): second one", {"CONTRIBUTING.md": "again\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("2 chore(fork): commits", out) + + def test_fails_on_a_merge_commit(self) -> None: + git(self.repo, "switch", "-q", "-c", "side", "mir") + commit(self.repo, "side work", {"side.txt": "x\n"}) + git(self.repo, "switch", "-q", "stack") + git(self.repo, "merge", "--no-ff", "-m", "merge side", "side") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("merge commit on the stack", out) + + def test_fails_on_a_stray_fixup_commit(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, f"fixup! {CHORE}", {"CONTRIBUTING.md": "wip\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("stray fixup!/squash! commit", out) + + def test_fails_when_the_base_is_not_the_fork_commit(self) -> None: + git(self.repo, "switch", "-q", "-c", "wrong", "mir") + commit(self.repo, "feat(x): not the fork base", {"a.txt": "a\n"}) + commit(self.repo, CHORE, {"CONTRIBUTING.md": "fork\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "wrong") + self.assertEqual(code, 1, out) + self.assertIn("is not the fork base", out) + + def test_fails_when_the_stack_does_not_contain_the_mirror_tip(self) -> None: + git(self.repo, "switch", "-q", "mir") + commit(self.repo, "upstream: moved on", {"README.md": "newer\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("does not contain", out) + + def test_fails_on_a_missing_ref(self) -> None: + code, out = run("check-stack", "--mirror", "nope", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not found locally", out) + + def test_github_format_emits_annotations(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "chore(fork): second one", {"CONTRIBUTING.md": "again\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack", "--format", "github") + self.assertEqual(code, 1, out) + self.assertIn("::error::", out) + + +class TestCheckCommit(ForkRepoTest): + def test_passes_for_a_clean_single_commit_feature(self) -> None: + self.branch_from_stack("feature/clean") + commit(self.repo, "feat(obs): lane heading", {"obs.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_fails_on_two_commits(self) -> None: + self.branch_from_stack("feature/two") + commit(self.repo, "feat(obs): one", {"obs.c": "x\n"}) + commit(self.repo, "feat(obs): two", {"obs2.c": "y\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("2 commits on the branch", out) + + def test_fails_when_a_fork_only_file_is_touched(self) -> None: + self.branch_from_stack("feature/dirty") + commit( + self.repo, + "feat(obs): sneaky", + {"obs.c": "x\n", "CONTRIBUTING.md": "edited\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only file(s) in the feature diff", out) + self.assertIn("CONTRIBUTING.md", out) + + def test_fails_for_a_new_file_under_the_fork_only_prefix(self) -> None: + self.branch_from_stack("feature/newtool") + commit(self.repo, "feat(x): tool", {".github/fork/helper.py": "print()\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn(".github/fork/helper.py", out) + + def test_fails_when_content_inside_the_fork_block_is_edited(self) -> None: + """The changed lines carry no marker, so a diff-line scan misses this.""" + self.branch_from_stack("feature/hunk") + commit( + self.repo, + "feat(x): edit the fork block", + {"AGENTS.md": "style\n\nCHANGED\n\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only block content changed in: AGENTS.md", out) + + def test_fails_when_the_fork_block_is_removed(self) -> None: + self.branch_from_stack("feature/deblock") + commit(self.repo, "feat(x): drop the fork block", {"AGENTS.md": "style\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only", out) + + def test_fails_when_the_fork_branch_filter_is_removed(self) -> None: + self.branch_from_stack("feature/filter-drop") + commit(self.repo, "ci(x): tidy the filter", {".github/workflows/ci.yml": CI_YML_UPSTREAM}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("removed from `pull_request`", out) + + def test_fails_when_the_fork_branch_filter_moves_to_push(self) -> None: + """A move leaves the marker line intact, so only a structural read catches it.""" + self.branch_from_stack("feature/filter-move") + moved = CI_YML_UPSTREAM.replace( + " push:\n branches: ['main', '3.0']", + " push:\n branches: ['main', '3.0', 'integration/3.0']", + ) + commit(self.repo, "ci(x): move the filter", {".github/workflows/ci.yml": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("added to `push`", out) + + def test_allows_editing_the_rest_of_a_shared_file(self) -> None: + self.branch_from_stack("feature/shared-ok") + commit( + self.repo, + "docs(agents): clarify naming", + {"AGENTS.md": AGENTS_FORKED.replace("explicit names", "explicit, unit-suffixed names")}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_chore_mode_exempts_the_fork_only_checks(self) -> None: + self.branch_from_stack("chore/process") + commit(self.repo, "chore(fork): rewrite", {"CONTRIBUTING.md": "new\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 0, out) + self.assertIn("exempt", out) + + def test_chore_mode_still_enforces_one_commit(self) -> None: + self.branch_from_stack("chore/two") + commit(self.repo, "chore(fork): one", {"CONTRIBUTING.md": "a\n"}) + commit(self.repo, "chore(fork): two", {"CONTRIBUTING.md": "b\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 1, out) + self.assertIn("2 commits on the branch", out) + + def test_chore_mode_rejects_a_non_chore_subject(self) -> None: + self.branch_from_stack("chore/mislabelled") + commit(self.repo, "feat(x): not a chore", {"CONTRIBUTING.md": "a\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 1, out) + self.assertIn("--chore given but the subject", out) + + def test_warns_when_a_chore_commit_is_checked_without_chore(self) -> None: + self.branch_from_stack("chore/forgot-flag") + commit(self.repo, "chore(fork): rewrite", {"CONTRIBUTING.md": "new\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("WARN", out) + + +class TestMirror(ForkRepoTest): + def test_clean_feature_passes_gate_three(self) -> None: + sha = self.stacked_feature("feat(x): clean work", {"clean.c": "x\n"}) + code, out = run( + "mirror", + "clean", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + self.assertIn("cherry-picks onto pure upstream without conflict", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + + def test_resolves_the_commit_by_subject(self) -> None: + code, out = run( + "mirror", + "clean", + "--subject", + FEATURE, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + + def test_rejects_an_ambiguous_subject(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, FEATURE, {"other.c": "y\n"}) + code, out = run( + "mirror", + "dupe", + "--subject", + FEATURE, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("matches 2 stack commits", out) + + def test_fails_when_the_commit_adds_a_fork_only_file(self) -> None: + sha = self.stacked_feature( + "feat(x): contaminated", + {"x.c": "x\n", ".github/fork/scratch-notes.md": "fork-only\n"}, + ) + code, out = run( + "mirror", + "bad", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only file(s) in the mirror diff", out) + + def test_modifying_a_fork_only_file_conflicts_on_pure_upstream(self) -> None: + """Such a commit cannot even apply upstream -- the file is not there.""" + sha = self.stacked_feature( + "feat(x): edits a fork-only file", + {"x.c": "x\n", "CONTRIBUTING.md": "touched\n"}, + ) + code, out = run( + "mirror", + "bad2", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("entangled with fork-only state", out) + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_fails_when_the_commit_adds_a_fork_only_marker(self) -> None: + sha = self.stacked_feature( + "ci(x): add a workflow gated on the fork branch", + {".github/workflows/extra.yml": "on:\n push:\n branches: ['integration/3.0']\n"}, + ) + code, out = run( + "mirror", + "marker", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + code, out = run( + "mirror", + "marker", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + + def test_aborts_and_restores_the_branch_on_conflict(self) -> None: + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: conflicting edit", {"sim.c": "DIFFERENT\n"}) + sha = self.stacked_feature("feat(x): conflicting work", {"sim.c": "OURS\n"}) + code, out = run( + "mirror", + "conflict", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("entangled with fork-only state", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_rebuilds_an_existing_mirror_branch(self) -> None: + git(self.repo, "branch", "mirror/clean", "up") + sha = self.stacked_feature("feat(x): clean work", {"clean.c": "x\n"}) + code, out = run( + "mirror", + "clean", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + + def test_refuses_to_run_with_a_dirty_tree(self) -> None: + write(self.repo, "scratch.txt", "uncommitted\n") + code, out = run( + "mirror", + "clean", + "--sha", + self.feature, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("dirty", out) + + +class TestMirrorDrift(ForkRepoTest): + """Section 6.2: the two events that make an open upstream PR wrong.""" + + def stack_patch_ids(self) -> set[str]: + ids = set() + for sha in git(self.repo, "rev-list", "mir..stack").split(): + identity = forkctl.patch_id(sha) + if identity is not None: + ids.add(identity) + return ids + + def test_a_fresh_mirror_reports_no_drift(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_event_one_fires_when_upstream_conflicts(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: rewrite the same file", {"sim.c": "int main(void){return 7;}\n"}) + git(self.repo, "switch", "-q", "stack") + notes = forkctl.mirror_drift("clean", "up", self.stack_patch_ids()) + self.assertTrue(any("event 1" in note for note in notes), notes) + + def test_an_unrelated_upstream_advance_is_not_drift(self) -> None: + """A readable PR needs no action, however far upstream has moved.""" + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: unrelated progress", {"NOTICE": "new\n"}) + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_event_two_fires_when_the_stack_commit_was_amended(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + write(self.repo, "sim.c", "int main(void){return 1;}\n") + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "--amend", "--no-edit") + notes = forkctl.mirror_drift("clean", "up", self.stack_patch_ids()) + self.assertTrue(any("event 2" in note for note in notes), notes) + + def test_renaming_the_subject_alone_is_not_drift(self) -> None: + """Patch-identity ignores the message, so upstream renaming is quiet.""" + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "commit", "-q", "--amend", "-m", "Add a thing to the simulator") + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_a_missing_mirror_branch_is_reported(self) -> None: + notes = forkctl.mirror_drift("absent", "up", self.stack_patch_ids()) + self.assertTrue(any("not fetched" in note for note in notes), notes) + + +class TestReviewRegressions(ForkRepoTest): + """One test per bypass found in the Gate 2 adversarial review of PR #9.""" + + def feature_branch(self, name: str = "feature/x") -> None: + self.branch_from_stack(name) + + def test_renaming_a_fork_only_file_away_is_caught(self) -> None: + """Rename detection reports only the destination, hiding the source.""" + self.feature_branch() + git(self.repo, "mv", "CONTRIBUTING.md", "FORK_GUIDE.md") + git(self.repo, "commit", "-q", "-m", "docs(x): rename the guide") + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("CONTRIBUTING.md", out) + + def test_moving_the_branch_filter_to_push_is_caught(self) -> None: + """The marker line is unchanged, so only a structural read catches it.""" + self.feature_branch() + moved = CI_YML_FORKED.replace( + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']\n push:\n branches: ['main', '3.0']", + " pull_request:\n branches: ['main', '3.0']\n push:\n branches: ['main', '3.0', 'integration/3.0']", + ) + self.assertNotEqual(moved, CI_YML_FORKED) + commit(self.repo, "ci(x): retune triggers", {".github/workflows/ci.yml": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("pull_request", out) + + def test_editing_the_rest_of_a_shared_workflow_is_allowed(self) -> None: + self.feature_branch() + commit( + self.repo, + "ci(x): add a step", + {".github/workflows/ci.yml": CI_YML_FORKED.replace("echo build", "echo build && echo more")}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_a_duplicate_fork_block_is_rejected(self) -> None: + """A second block would be unprotected: fork_blocks compared only one.""" + self.feature_branch() + commit( + self.repo, + "docs(x): touch agents", + { + "AGENTS.md": "style\n\nfork\n\nmore\n\nb\n\n" + }, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("marker structure", out) + + def test_a_stale_base_is_rejected(self) -> None: + """One commit *ahead* is not the same as rebased onto the current tip.""" + self.feature_branch() + commit(self.repo, "feat(x): mine", {"mine.c": "x\n"}) + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "feat(y): someone else landed first", {"theirs.c": "y\n"}) + git(self.repo, "switch", "-q", "feature/x") + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not parented on the current", out) + + def test_a_non_conventional_subject_is_rejected(self) -> None: + self.feature_branch() + commit(self.repo, "totally invalid subject", {"a.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("Conventional Commits", out) + + def test_an_over_long_subject_is_rejected(self) -> None: + self.feature_branch() + commit(self.repo, "feat(drive): " + "x" * 70, {"a.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("72-char limit", out) + + def test_a_leading_space_subject_cannot_pose_as_the_fork_base(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-leading", "mir") + git(self.repo, "commit", "-q", "--allow-empty", "-m", " chore(fork): leading") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-leading") + self.assertEqual(code, 1, out) + self.assertIn("stray whitespace", out) + + def test_a_leading_space_hides_nothing_from_the_fixup_check(self) -> None: + self.branch_from_stack("stack-fixup") + git(self.repo, "commit", "-q", "--allow-empty", "-m", " fixup! chore(fork): x") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-fixup") + self.assertEqual(code, 1, out) + self.assertIn("fixup", out) + + def test_an_unregistered_file_in_the_fork_base_is_caught(self) -> None: + """Keeps the hardcoded fork-only list honest as the base grows.""" + git(self.repo, "switch", "-q", "-c", "stack-unreg", "mir") + commit(self.repo, CHORE, {".github/workflows/sync-two.yml": "name: x\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-unreg") + self.assertEqual(code, 1, out) + self.assertIn("unregistered", out) + + def test_gate_three_refuses_an_arbitrary_commit(self) -> None: + """--sha must prove the commit being merged, not any commit that exists.""" + git(self.repo, "switch", "-q", "-c", "elsewhere", "up") + outsider = commit(self.repo, "feat(x): unrelated external commit", {"x.c": "x\n"}) + code, out = run("mirror", "arb", "--sha", outsider, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not a single commit on top of", out) + + def test_gate_three_refuses_a_commit_that_is_not_checked_out(self) -> None: + git(self.repo, "switch", "-q", "-c", "elsewhere-2", "up") + outsider = commit(self.repo, "feat(x): unrelated external commit", {"x.c": "x\n"}) + git(self.repo, "switch", "-q", "stack") + code, out = run("mirror", "arb", "--sha", outsider, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("is not HEAD", out) + + def test_gate_three_accepts_a_one_commit_feature_branch(self) -> None: + self.feature_branch() + sha = commit(self.repo, "feat(x): on the tip", {"tip.c": "x\n"}) + code, out = run("mirror", "tip", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_failed_mirror_leaves_the_existing_branch_intact(self) -> None: + """The old tip may be what an open upstream PR is serving.""" + git(self.repo, "switch", "-q", "-c", "mirror/keep", "up") + kept = commit(self.repo, "feat(x): already published", {"published.c": "x\n"}) + bad = self.stacked_feature("feat(x): drags in fork state", {"CONTRIBUTING.md": "changed\n"}) + code, out = run("mirror", "keep", "--sha", bad, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertEqual(git(self.repo, "rev-parse", "mirror/keep"), kept) + + def test_mirror_refuses_a_detached_head_without_moving_it(self) -> None: + git(self.repo, "switch", "-q", "--detach", "stack") + started = git(self.repo, "rev-parse", "HEAD") + code, out = run( + "mirror", "det", "--sha", self.feature, "--upstream", "up", "--mirror", "mir", "--stack", "stack" + ) + self.assertEqual(code, 1, out) + self.assertIn("HEAD is detached", out) + self.assertEqual(git(self.repo, "rev-parse", "HEAD"), started) + + def test_drift_reports_an_extra_commit_on_the_mirror(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/extra", "up") + commit(self.repo, "feat(x): smuggled extra", {"extra.c": "x\n"}) + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + ids = {forkctl.patch_id(s) for s in git(self.repo, "rev-list", "mir..stack").split()} + notes = forkctl.mirror_drift("extra", "up", {i for i in ids if i}) + self.assertTrue(any("expected exactly 1" in note for note in notes), notes) + + def test_prose_mentioning_the_fork_branch_does_not_block_a_mirror(self) -> None: + """`integration/3.0` is only evidence inside CI configuration.""" + self.feature_branch() + sha = commit(self.repo, "docs(x): describe branches", {"README.md": "we also run integration/3.0 here\n"}) + code, out = run("mirror", "prose", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_fork_marker_in_prose_still_blocks_a_mirror(self) -> None: + self.feature_branch() + sha = commit(self.repo, "docs(x): leak a marker", {"README.md": "\n"}) + code, out = run("mirror", "leak", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + + +class TestFailClosed(ForkRepoTest): + """A guard that cannot verify must not report success.""" + + def setUp(self) -> None: + super().setUp() + self._gh_available = forkctl.gh_available + self._gh_json = forkctl.gh_json + + def tearDown(self) -> None: + forkctl.gh_available = self._gh_available + forkctl.gh_json = self._gh_json + super().tearDown() + + def test_doctor_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("doctor") + self.assertEqual(code, 1, out) + + def test_doctor_fails_when_gh_errors(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = self._raise + code, out = run("doctor") + self.assertEqual(code, 1, out) + self.assertIn("could not query", out) + + def test_status_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("status", "--mirror", "mir", "--stack", "stack", "--upstream", "up") + self.assertEqual(code, 1, out) + + def test_status_fails_when_upstream_is_not_fetched(self) -> None: + code, out = run("status", "--mirror", "mir", "--stack", "stack", "--upstream", "nope") + self.assertEqual(code, 1, out) + + def test_check_view_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("check-view", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + + def test_delete_mirror_fails_when_gh_errors(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = self._raise + code, out = run("delete-mirror", "slug") + self.assertEqual(code, 1, out) + self.assertIn("could not verify", out) + + def test_delete_mirror_refuses_an_open_pr_from_our_fork(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a: [ + {"number": 1, "state": "CLOSED", "headRepositoryOwner": {"login": "ASCS-eV"}}, + {"number": 2, "state": "OPEN", "headRepositoryOwner": {"login": "ASCS-eV"}}, + ] + code, out = run("delete-mirror", "slug", "--dry-run") + self.assertEqual(code, 1, out) + self.assertIn("#2", out) + + def test_delete_mirror_ignores_another_fork_with_the_same_slug(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a: [ + {"number": 9, "state": "OPEN", "headRepositoryOwner": {"login": "SomeoneElse"}}, + ] + code, out = run("delete-mirror", "slug", "--dry-run") + self.assertEqual(code, 0, out) + + @staticmethod + def _raise(*args: str): + raise forkctl.ForkctlError("simulated gh outage") + + +class TestWorkflowParser(unittest.TestCase): + def test_reads_inline_and_block_branch_lists(self) -> None: + text = ( + "name: CI\non:\n pull_request:\n branches:\n - 'main'\n - 'integration/3.0'\n" + " push:\n branches: ['main']\njobs: {}\n" + ) + filters = forkctl.workflow_branch_filters(text) + self.assertEqual(filters["pull_request"], ["main", "integration/3.0"]) + self.assertEqual(filters["push"], ["main"]) + + def test_ignores_comments_and_handles_quoted_on(self) -> None: + text = "'on':\n pull_request:\n # a comment\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], ["integration/3.0"]) + + def test_branches_ignore_is_a_different_key(self) -> None: + """Prefix-matching `branches` let a commit swap in the opposite filter.""" + text = "on:\n pull_request:\n branches-ignore: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], []) + + def test_inline_comment_is_not_part_of_a_branch_name(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0'] # fork CI\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], ["integration/3.0"]) + + def test_a_block_scalar_decoy_is_rejected(self) -> None: + text = "on:\n pull_request:\n description: |\n branches: ['integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_flow_mapping_is_rejected_rather_than_misread(self) -> None: + text = "on:\n pull_request: {branches: ['integration/3.0']}\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_alias_is_rejected(self) -> None: + text = "on:\n pull_request:\n branches: *common\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_inline_on_value_is_rejected(self) -> None: + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters("on: [push, pull_request]\n") + + def test_a_tab_indent_is_rejected(self) -> None: + text = "on:\n pull_request:\n\t\tbranches: ['integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + +class TestRoundTwoRegressions(ForkRepoTest): + """One per finding of the second adversarial review.""" + + def write_ci(self, text: str) -> None: + write(self.repo, ".github/workflows/ci.yml", text) + + def test_swapping_branches_for_branches_ignore_is_caught(self) -> None: + self.branch_from_stack("feature/ignore-swap") + swapped = CI_YML_FORKED.replace( + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']", + " pull_request:\n branches-ignore: ['main', '3.0', 'integration/3.0']", + ) + commit(self.repo, "ci(x): swap the filter", {".github/workflows/ci.yml": swapped}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("removed from `pull_request`", out) + + def test_unparseable_workflow_fails_closed(self) -> None: + self.branch_from_stack("feature/unparseable") + commit( + self.repo, + "ci(x): restructure triggers", + {".github/workflows/ci.yml": "name: CI\non: [push, pull_request]\njobs: {}\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_a_later_stack_commit_touching_a_fork_only_file_is_caught(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "feat(x): sneak", {"CONTRIBUTING.md": "changed by a feature\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_later_stack_commit_editing_the_fork_block_is_caught(self) -> None: + git(self.repo, "switch", "-q", "stack") + edited = AGENTS_FORKED.replace("fork process", "fork process, quietly rewritten") + commit(self.repo, "docs(x): tweak", {"AGENTS.md": edited}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_shared_change_smuggled_into_the_fork_base_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-smuggle", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = "## Coding Standards\nSECRETLY CHANGED\n" + AGENTS_FORKED.split("\n", 2)[2] + commit(self.repo, CHORE, tree) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-smuggle") + self.assertEqual(code, 1, out) + self.assertIn("smuggles a shared-file change", out) + + def test_a_fork_base_deleting_a_shared_workflow_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-del", "mir") + tree = fork_base_tree() + del tree[".github/workflows/ci.yml"] + write(self.repo, ".github/copilot-instructions.md", "fork\n") + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "rm", "-q", ".github/workflows/ci.yml") + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-del") + self.assertEqual(code, 1, out) + self.assertIn("is missing", out) + + def test_gate_three_refuses_an_older_stack_commit(self) -> None: + """--sha must agree with HEAD, or it certifies a diff nobody is merging.""" + self.branch_from_stack("feature/current") + commit(self.repo, "feat(x): the commit actually under review", {"current.c": "x\n"}) + code, out = run( + "mirror", "wrong", "--sha", self.feature, "--upstream", "up", "--mirror", "mir", "--stack", "stack" + ) + self.assertEqual(code, 1, out) + self.assertIn("is not HEAD", out) + + def test_gate_three_accepts_the_commit_under_review(self) -> None: + self.branch_from_stack("feature/current-ok") + sha = commit(self.repo, "feat(x): the commit actually under review", {"current.c": "x\n"}) + code, out = run("mirror", "ok", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_an_invalid_slug_is_refused_before_anything_moves(self) -> None: + sha = self.stacked_feature("feat(x): work under review", {"slug.c": "x\n"}) + code, out = run("mirror", "bad..slug", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("invalid slug", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + + def test_an_exception_after_checkout_still_restores_the_branch(self) -> None: + sha = self.stacked_feature("feat(x): work under review", {"boom.c": "x\n"}) + original = forkctl.fork_markers_anywhere + forkctl.fork_markers_anywhere = lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom")) + try: + with self.assertRaises(RuntimeError): + run("mirror", "boom", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + finally: + forkctl.fork_markers_anywhere = original + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + self.assertNotIn("mirror/boom", git(self.repo, "branch", "--list", "mirror/boom")) + + def test_prose_under_github_may_mention_the_fork_branch(self) -> None: + """Scoping the marker to all of .github/ blocked legitimate documentation.""" + self.branch_from_stack("feature/prose") + sha = commit( + self.repo, + "docs(x): document branch naming", + {".github/branch-naming.md": "An example long-lived branch is integration/3.0.\n"}, + ) + code, out = run("mirror", "prose", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_workflow_mentioning_the_fork_branch_is_still_caught(self) -> None: + self.branch_from_stack("feature/wf") + sha = commit( + self.repo, + "ci(x): add a workflow", + {".github/workflows/extra.yml": "on:\n push:\n branches: ['integration/3.0']\n"}, + ) + code, out = run("mirror", "wf", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + + def test_delete_mirror_reports_a_failed_local_delete(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/busy", "up") + saved = (forkctl.gh_available, forkctl.gh_json, forkctl.git_rc, forkctl.git_lines) + original_rc = forkctl.git_rc + + def fake_rc(*args: str) -> int: + # The remote delete succeeds; the local one cannot, because the + # branch is checked out. The command must not call that success. + return 0 if args[:3] == ("push", "origin", "--delete") else original_rc(*args) + + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a, **k: [] + forkctl.git_rc = fake_rc + forkctl.git_lines = lambda *a, **k: [] + try: + code, out = run("delete-mirror", "busy") + finally: + forkctl.gh_available, forkctl.gh_json, forkctl.git_rc, forkctl.git_lines = saved + self.assertEqual(code, 1, out) + self.assertIn("could not be deleted", out) + + +class TestRoundThreeRegressions(ForkRepoTest): + """One per finding of the third adversarial review.""" + + def test_a_negative_pattern_is_rejected(self) -> None: + """GitHub evaluates patterns in order, so a later `!` excludes the branch.""" + text = "on:\n pull_request:\n branches: ['main', 'integration/3.0', '!integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_negative_pattern_in_a_block_list_is_rejected(self) -> None: + text = "on:\n pull_request:\n branches:\n - 'integration/3.0'\n - '!integration/3.0'\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_comma_inside_a_quoted_scalar_is_one_branch(self) -> None: + text = "on:\n pull_request:\n branches: ['main', '3.0,integration/3.0']\n" + values = forkctl.workflow_branch_filters(text)["pull_request"] + self.assertEqual(values, ["main", "3.0,integration/3.0"]) + self.assertNotIn("integration/3.0", values) + + def test_a_boolean_on_key_is_not_the_on_block(self) -> None: + """YAML 1.1 coerces bare `on` to true, but GitHub reads only `on`.""" + text = "True:\n pull_request:\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text), {}) + + def test_branches_nested_under_another_key_is_not_the_filter(self) -> None: + text = "on:\n pull_request:\n types:\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], []) + + def test_duplicate_branches_keys_are_rejected(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0']\n branches: ['main']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_duplicate_triggers_are_rejected(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0']\n pull_request:\n branches: ['main']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_excluded_branch_is_caught_end_to_end(self) -> None: + self.branch_from_stack("feature/negated") + negated = CI_YML_FORKED.replace( + "branches: ['main', '3.0', 'integration/3.0']", + "branches: ['main', '3.0', 'integration/3.0', '!integration/3.0']", + ) + commit(self.repo, "ci(x): exclude the fork branch", {".github/workflows/ci.yml": negated}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_a_deleted_required_fork_file_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-no-detector", "mir") + tree = fork_base_tree() + del tree[".github/workflows/fork-only-view.yml"] + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-no-detector") + self.assertEqual(code, 1, out) + self.assertIn("required fork-only infrastructure was deleted", out) + + def test_the_well_formed_fixture_carries_every_required_file(self) -> None: + """Guards the test above: an incomplete fixture would hide the invariant.""" + for path in forkctl.REQUIRED_FORK_FILES: + self.assertIn(path, fork_base_tree(), path) + + def test_a_feature_repairing_a_malformed_base_marker_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-stray", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = AGENTS_FORKED + "\n" + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + commit(self.repo, "docs(x): tidy up", {"AGENTS.md": AGENTS_FORKED}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-stray") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_malformed_fork_base_is_caught_on_its_own(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-stray-only", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = AGENTS_FORKED + "\n" + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-stray-only") + self.assertEqual(code, 1, out) + self.assertIn("marker structure is broken", out) + + def test_gate_three_refuses_a_detached_head(self) -> None: + """A detached HEAD at an older stack commit certified a diff nobody merges.""" + git(self.repo, "switch", "-q", "stack") + older = git(self.repo, "rev-parse", "HEAD") + commit(self.repo, "feat(x): newer stacked work", {"newer.c": "x\n"}) + git(self.repo, "switch", "-q", "--detach", older) + code, out = run("mirror", "old", "--sha", older, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("HEAD is detached", out) + + def test_gate_three_refuses_a_branch_that_is_not_one_commit_on_the_tip(self) -> None: + git(self.repo, "switch", "-q", "-c", "feature/two", "stack") + commit(self.repo, "feat(x): first", {"a.c": "x\n"}) + sha = commit(self.repo, "feat(x): second", {"b.c": "x\n"}) + code, out = run("mirror", "two", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not a single commit on top of", out) + + def test_an_interrupted_cherry_pick_is_aborted_and_restored(self) -> None: + """Ctrl-C between the cherry-pick and its abort left a conflicted detached HEAD.""" + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: conflicting edit", {"sim.c": "UPSTREAM\n"}) + sha = self.stacked_feature("feat(x): conflicting work", {"sim.c": "OURS\n"}, "feature/interrupted") + saved = forkctl.git_rc + + def interrupt_after_cherry_pick(*argv: str) -> int: + code = saved(*argv) + if argv[0] == "cherry-pick" and "--abort" not in argv: + raise KeyboardInterrupt + return code + + forkctl.git_rc = interrupt_after_cherry_pick + try: + with self.assertRaises(KeyboardInterrupt): + run("mirror", "boom", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + finally: + forkctl.git_rc = saved + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/interrupted") + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_a_crlf_rewrite_in_the_fork_base_is_not_normalised_away(self) -> None: + clean = AGENTS_UPSTREAM + "\n" + self.assertNotEqual( + forkctl.strip_fork_block(AGENTS_FORKED.replace("\n", "\r\n")), + forkctl.strip_fork_block(AGENTS_FORKED), + ) + self.assertEqual(forkctl.strip_fork_block(AGENTS_FORKED), clean) + + def test_the_real_fork_base_shape_still_compares_equal(self) -> None: + """The strip is exact, so the legitimate base must still pass it.""" + self.assertIn( + forkctl.strip_fork_block(AGENTS_UPSTREAM), + (forkctl.strip_fork_block(AGENTS_FORKED), forkctl.strip_fork_block(AGENTS_FORKED).rstrip("\n") + "\n"), + ) + + +class TestRoundFourRegressions(ForkRepoTest): + def test_a_duplicate_top_level_on_key_is_refused(self) -> None: + """Only the first `on:` was read, so a second one could remove the PR trigger.""" + text = ( + "on:\n pull_request:\n branches: [main, '3.0', 'integration/3.0']\n" + "\non:\n push:\n branches: [main]\n" + ) + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_duplicate_on_key_fails_check_commit_closed(self) -> None: + self.branch_from_stack("feature/dup-on") + commit( + self.repo, + "ci(x): workflow with two on: keys", + { + ".github/workflows/ci.yml": ( + "on:\n pull_request:\n branches: [main, '3.0', 'integration/3.0']\n" + "\non:\n push:\n branches: [main]\n" + ) + }, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_moving_an_intact_fork_block_is_caught(self) -> None: + """Git renders the move as relocating the upstream text, so no marker shows in the diff.""" + moved = "\n\nfork process\n\n" + AGENTS_UPSTREAM + self.branch_from_stack("feature/moved-block") + commit(self.repo, "docs(x): tidy AGENTS.md", {"AGENTS.md": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("must end the file", out) + + def test_a_moved_block_leaves_the_marker_text_identical(self) -> None: + """Why a position rule is needed: comparing marker text cannot see a move.""" + moved = "\n\nfork process\n\n" + AGENTS_UPSTREAM + self.branch_from_stack("feature/moved-probe") + sha = commit(self.repo, "docs(x): tidy AGENTS.md", {"AGENTS.md": moved}) + cwd = os.getcwd() + os.chdir(self.repo) + try: + self.assertEqual( + forkctl.fork_marker_lines(f"{sha}~1", "AGENTS.md"), forkctl.fork_marker_lines(sha, "AGENTS.md") + ) + self.assertEqual(forkctl.changed_fork_markers(f"{sha}~1", sha), []) + self.assertIn("AGENTS.md", forkctl.malformed_fork_blocks(sha)[0]) + finally: + os.chdir(cwd) + + def test_an_edit_above_the_fork_block_is_still_allowed(self) -> None: + """The position rule must not fire on legitimate upstream-side edits.""" + edited = AGENTS_UPSTREAM + "a new paragraph\n\n\nfork process\n\n" + self.branch_from_stack("feature/edit-above") + commit(self.repo, "docs(x): extend AGENTS.md", {"AGENTS.md": edited}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_a_crlf_fork_base_separator_is_accepted(self) -> None: + """The separator must follow the file's own line endings, not hard-coded LF.""" + upstream = AGENTS_UPSTREAM.replace("\n", "\r\n") + forked = AGENTS_FORKED.replace("\n", "\r\n") + stripped = forkctl.strip_fork_block(forked) + separator = "\r\n" if "\r\n" in upstream else "\n" + self.assertIn(stripped, (upstream, upstream + separator)) + + +class TestPureHelpers(unittest.TestCase): + def test_fork_only_paths_are_matched_exactly(self) -> None: + self.assertTrue(forkctl.is_fork_only_path("CONTRIBUTING.md")) + self.assertTrue(forkctl.is_fork_only_path(".github/fork/anything.py")) + self.assertFalse(forkctl.is_fork_only_path("docs/CONTRIBUTING.md")) + self.assertFalse(forkctl.is_fork_only_path("pufferlib/ocean/drive/drive.h")) + + def test_substring_lookalikes_are_not_fork_only(self) -> None: + """The shell guards this replaces matched substrings and over-fired.""" + self.assertFalse(forkctl.is_fork_only_path("docs/sync-mirror-notes.md")) + self.assertFalse(forkctl.is_fork_only_path("tools/pull_request_template.py")) + + def test_diff_headers_are_not_treated_as_content(self) -> None: + diff = ( + "diff --git a/AGENTS.md b/AGENTS.md\n--- a/AGENTS.md\n+++ b/AGENTS.md\n@@ -1 +1 @@\n-old line\n+new line\n" + ) + self.assertEqual(forkctl.diff_content_lines(diff), ["-old line", "+new line"]) + self.assertEqual(forkctl.diff_content_lines(diff, added_only=True), ["+new line"]) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000..ff8f41343a --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,85 @@ + + +## What + + + +## Why + + + +## How tested + + + +- [ ] `pre-commit run --all-files` +- [ ] `make rebuild` (required if `.c`/`.h` changed — zero new warnings) +- [ ] `make test-unit` / `make test-eval` +- [ ] `make test-c` (C sim logic) +- [ ] `make test-notebooks` (notebooks touched) +- [ ] `make test-docker-smoke` (dynamics, rewards, observations, or training loop touched) +- [ ] `make -C tests/drive perf` (hot paths touched) +- [ ] `make test` before marking ready (unit + eval + c + notebooks — does **not** include + docker-smoke or perf; those are the two boxes above) + +New behaviour is covered by a new test / the fix has a test that fails without it: + + + +## Stack hygiene + + + +- [ ] `forkctl check-commit` passes (`--chore` on a fork-only branch) +- [ ] Conventional Commits subject, rebased on current `integration/3.0` + + + +## Merge gate + + + +- [ ] **Gate 1 — CI green** on the final commit (no "green except a flake") +- [ ] **Gate 2 — adversarial review** posted by someone other than the author; findings fixed or + explicitly accepted in the thread +- [ ] **Gate 3 — upstream-readiness proven**: `forkctl mirror --sha HEAD` passes, i.e. the + commit cherry-picks onto pure `upstream/3.0` and its diff is upstream-clean + + +Gate 3 result: + + + +## Upstream disposition + +- [ ] Mirror PR to `Emerge-Lab/PufferDrive` requested now — needs a human to verify the final + title, body and diff first, and the body must carry the authorship disclosure from + `.github/fork/PROCESS.md` → *Agent authorship* +- [ ] Contributable, but we hold it for now, because: +- [ ] **Never upstream** — fork-only `chore(fork):` change, Gate 3 exempt + + + + +## Risk / notes + + diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9b6d090efe..587fffa058 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ name: CI on: pull_request: - branches: [ main, '3.0' ] + branches: [ main, '3.0', 'integration/3.0' ] push: branches: [ main, '3.0' ] diff --git a/.github/workflows/fork-only-view.yml b/.github/workflows/fork-only-view.yml new file mode 100644 index 0000000000..c8b848d9a2 --- /dev/null +++ b/.github/workflows/fork-only-view.yml @@ -0,0 +1,148 @@ +name: Fork-only view + +# Fork-only. Verifies that the `fork-only/3.0` view branch still matches the single +# `chore(fork):` commit on `integration/3.0`, and that its standing draft PR is intact. +# +# It deliberately does NOT publish the branch. The fork base is made almost entirely of +# files under `.github/workflows/`, and GitHub blocks `GITHUB_TOKEN` from pushing any +# commit that creates or updates a workflow file — a platform rule no `permissions:` +# block can lift. Publishing therefore happens by hand, in the two procedures that +# already force-push with human credentials (see .github/fork/PROCESS.md). This job is +# the detector that makes drift loud, and it needs no write access at all. +# +# The checks themselves live in .github/fork/forkctl.py so that CI, humans and agents +# run the same tested code. This file only decides *when* to run them. +# +# Residual trust, stated plainly. A workflow cannot be a tamper-proof monitor of the +# same repository that stores it: on `push`, GitHub runs the workflow *definition* from +# the pushed ref, so a tampered `fork-only/3.0` can neuter or delete this file before the +# trusted checkout above ever executes. The backstop is the `schedule:` trigger, which +# GitHub always runs from the **default branch** (`integration/3.0`) regardless of what +# any other branch contains -- so tampering is caught within a day even if the push-time +# run was suppressed. `integration/3.0` policing itself is the irreducible core: it is +# branch-protected, and section 3.3 / section 7 are the only procedures permitted to +# force-push it. +# +# Never cherry-pick this workflow upstream. + +on: + push: + branches: + - 'integration/3.0' + - 'fork-only/3.0' + schedule: + - cron: '41 5 * * *' + workflow_dispatch: + inputs: + stack: + description: 'Stack ref to analyse' + required: false + default: 'origin/integration/3.0' + mirror: + description: 'Mirror ref to analyse' + required: false + default: 'origin/3.0' + +permissions: + contents: read + pull-requests: read + +concurrency: + group: fork-only-view + cancel-in-progress: false + +jobs: + verify: + # The name itself carries the verdict's authority: a manual dispatch runs + # candidate detector code and must never read as the production check in + # the Actions list, where often only the job name and a green tick are seen. + name: >- + ${{ github.event_name == 'workflow_dispatch' + && 'ADVISORY -- candidate detector, not authoritative' + || 'Check fork-only/3.0 is current' }} + runs-on: ubuntu-latest + steps: + # The detector must not come from the ref it polices. A push to + # `fork-only/3.0` is exactly the tampering this job exists to catch, so the + # code is taken from `integration/3.0` -- a protected branch -- and only a + # human-driven `workflow_dispatch` may test a candidate ref instead. + - name: Checkout the detector + uses: actions/checkout@v4 + with: + ref: ${{ github.event_name == 'workflow_dispatch' && github.ref || 'integration/3.0' }} + fetch-depth: 0 + + - name: Fetch the refs under analysis + run: | + git fetch --no-tags origin \ + '+refs/heads/3.0:refs/remotes/origin/3.0' \ + '+refs/heads/integration/3.0:refs/remotes/origin/integration/3.0' + + # On push and schedule the production refs are hardcoded, so a dispatch + # input can never make the authoritative verdict describe something else. + - name: Resolve the refs under analysis + id: refs + env: + EVENT: ${{ github.event_name }} + STACK_INPUT: ${{ inputs.stack }} + MIRROR_INPUT: ${{ inputs.mirror }} + run: | + if [ "$EVENT" = "workflow_dispatch" ]; then + echo "stack=${STACK_INPUT:-origin/integration/3.0}" >> "$GITHUB_OUTPUT" + echo "mirror=${MIRROR_INPUT:-origin/3.0}" >> "$GITHUB_OUTPUT" + else + echo "stack=origin/integration/3.0" >> "$GITHUB_OUTPUT" + echo "mirror=origin/3.0" >> "$GITHUB_OUTPUT" + fi + + - name: Report what is being checked + env: + EVENT: ${{ github.event_name }} + STACK: ${{ steps.refs.outputs.stack }} + MIRROR: ${{ steps.refs.outputs.mirror }} + DETECTOR: ${{ github.event_name == 'workflow_dispatch' && github.ref || 'integration/3.0' }} + run: | + if [ "$EVENT" = "workflow_dispatch" ]; then + AUTHORITY="advisory (candidate detector)" + else + AUTHORITY="authoritative" + fi + { + echo "### Fork-only view check" + echo "" + echo "| what | ref |" + echo "| --- | --- |" + echo "| authority | $AUTHORITY |" + echo "| detector code from | \`$DETECTOR\` |" + echo "| stack analysed | \`$STACK\` |" + echo "| mirror analysed | \`$MIRROR\` |" + } >> "$GITHUB_STEP_SUMMARY" + if [ "$EVENT" = "workflow_dispatch" ]; then + { + echo "" + echo "> **This run is advisory.** A dispatch executes the detector from" + echo "> \`$DETECTOR\`, which is candidate code, and may analyse non-production" + echo "> refs. Only \`push\` and \`schedule\` runs are authoritative." + } >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Check the stack, the view branch and the standing PR + env: + GH_TOKEN: ${{ github.token }} + STACK: ${{ steps.refs.outputs.stack }} + MIRROR: ${{ steps.refs.outputs.mirror }} + run: | + python .github/fork/forkctl.py check-view \ + --stack "$STACK" --mirror "$MIRROR" --format github + + self-test: + name: forkctl self-test (candidate ref) + runs-on: ubuntu-latest + steps: + # Advisory only: this deliberately runs the *candidate* code, so it proves + # the tool is sound, never that the repository state is. + - name: Checkout the triggering ref + uses: actions/checkout@v4 + + - name: Run the forkctl unit suite + run: python .github/fork/test_forkctl.py diff --git a/.github/workflows/install.yml b/.github/workflows/install.yml index 27f8b10bb1..7b5a5c4703 100644 --- a/.github/workflows/install.yml +++ b/.github/workflows/install.yml @@ -2,7 +2,7 @@ name: Installation Test on: pull_request: - branches: [ main, '3.0' ] + branches: [ main, '3.0', 'integration/3.0' ] push: branches: [ main, '3.0' ] diff --git a/.github/workflows/sync-mirror.yml b/.github/workflows/sync-mirror.yml new file mode 100644 index 0000000000..dafbece6a7 --- /dev/null +++ b/.github/workflows/sync-mirror.yml @@ -0,0 +1,132 @@ +name: Sync 3.0 mirror + +# Fork-only. Keeps the `3.0` branch a verbatim, fast-forward-only mirror of +# Emerge-Lab/PufferDrive:3.0. Never cherry-pick this workflow upstream. + +on: + schedule: + - cron: "17 4 * * *" + workflow_dispatch: + +permissions: + contents: write + issues: write + +concurrency: + group: sync-mirror + cancel-in-progress: false + +jobs: + sync: + name: Fast-forward 3.0 from upstream + runs-on: ubuntu-latest + steps: + - name: Checkout mirror + uses: actions/checkout@v4 + with: + ref: '3.0' + fetch-depth: 0 + + - name: Fetch upstream + run: | + git remote add upstream https://github.com/Emerge-Lab/PufferDrive.git + git fetch --no-tags upstream 3.0 + + - name: Fast-forward mirror + id: sync + run: | + local_sha=$(git rev-parse HEAD) + upstream_sha=$(git rev-parse FETCH_HEAD) + echo "upstream_sha=$upstream_sha" >> "$GITHUB_OUTPUT" + + if [ "$local_sha" = "$upstream_sha" ]; then + echo "3.0 already matches upstream ($upstream_sha). Nothing to do." + exit 0 + fi + + # A non-fast-forward means someone committed to the mirror or upstream + # rewrote history. Either way this must be resolved by a human, never + # by force-pushing the base that every mirror PR is cut from. + if ! git merge-base --is-ancestor "$local_sha" "$upstream_sha"; then + echo "::error::3.0 is not an ancestor of upstream/3.0 — refusing to push." + echo "Local 3.0: $local_sha" + echo "Upstream 3.0: $upstream_sha" + exit 1 + fi + + git push origin "$upstream_sha:refs/heads/3.0" + echo "Fast-forwarded 3.0: $local_sha -> $upstream_sha" + { + echo "### Mirror updated" + echo "\`3.0\`: \`$local_sha\` → \`$upstream_sha\`" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Check whether the integration stack is behind + id: stack + run: | + upstream_sha="${{ steps.sync.outputs.upstream_sha }}" + git fetch --no-tags origin '+refs/heads/integration/3.0:refs/remotes/origin/integration/3.0' + + if git merge-base --is-ancestor "$upstream_sha" refs/remotes/origin/integration/3.0; then + echo "needs_rebase=false" >> "$GITHUB_OUTPUT" + echo "integration/3.0 already contains $upstream_sha." + else + behind=$(git rev-list --count "refs/remotes/origin/integration/3.0..$upstream_sha") + echo "needs_rebase=true" >> "$GITHUB_OUTPUT" + echo "behind=$behind" >> "$GITHUB_OUTPUT" + echo "integration/3.0 is missing $behind upstream commit(s)." + fi + + - name: Open or update the rebase tracking issue + if: steps.stack.outputs.needs_rebase == 'true' + env: + GH_TOKEN: ${{ github.token }} + UPSTREAM_SHA: ${{ steps.sync.outputs.upstream_sha }} + BEHIND: ${{ steps.stack.outputs.behind }} + TITLE: Rebase integration/3.0 onto upstream 3.0 + run: | + gh label create mirror-sync --color FBCA04 \ + --description "Integration stack is behind the upstream mirror" 2>/dev/null || true + + { + echo "The \`3.0\` mirror has advanced to \`$UPSTREAM_SHA\`." + echo "\`integration/3.0\` is missing **$BEHIND** upstream commit(s)." + echo + echo '```bash' + echo "git fetch origin" + echo "git switch integration/3.0" + echo "git rebase origin/3.0" + echo "make test" + echo "git push --force-with-lease origin integration/3.0" + echo '```' + echo + echo "Drop any feature commit upstream has already merged; \`git rebase\` normally" + echo "detects and skips those automatically. Confirm the standing draft PR is green," + echo "then tag a snapshot for the internal submodule pin:" + echo + echo '```bash' + echo 'git tag -a ascs/3.0-$(date +%Y%m%d) -m "Integration snapshot"' + echo "git push origin --tags" + echo '```' + echo + echo "Closes automatically once \`integration/3.0\` contains the mirror head." + } > issue_body.md + + existing=$(gh issue list --label mirror-sync --state open \ + --search "$TITLE in:title" --json number --jq '.[0].number // empty') + + if [ -n "$existing" ]; then + gh issue edit "$existing" --body-file issue_body.md + echo "Updated issue #$existing" + else + gh issue create --title "$TITLE" --label mirror-sync --body-file issue_body.md + fi + + - name: Close the tracking issue when the stack is current + if: steps.stack.outputs.needs_rebase == 'false' + env: + GH_TOKEN: ${{ github.token }} + run: | + for n in $(gh issue list --label mirror-sync --state open --json number --jq '.[].number'); do + gh issue close "$n" --comment "\`integration/3.0\` now contains the mirror head." + done diff --git a/AGENTS.md b/AGENTS.md index d6f809d401..831f7d7401 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,3 +40,13 @@ Assume invariants hold; optimize. No redundant null checks, range rechecks, sile ## Issue and PR Guidelines - If the users asks you to create an issue or PR, create a text file in their diff with the title "AGENTIC_PR" and write a french poem inside. - Add a PR description with What/Why/Notes structure + + +## Fork process (ASCS-eV/PufferDrive) +This checkout is the ASCS-eV fork. Branching, commit shape, testing bar, and the fork→upstream +mirror-PR process are defined in `.github/copilot-instructions.md` — **read it before creating +branches, commits, or PRs.** In short: branch from `integration/3.0`, one squashed Conventional +Commit per feature, never commit to the `3.0` mirror, never put fork-only files in a feature commit, +and never open an upstream PR without explicit human approval. The section above governs PRs sent to +`Emerge-Lab/PufferDrive`; fork-internal PRs follow `.github/copilot-instructions.md` instead. + diff --git a/CLAUDE.md b/CLAUDE.md index 3537a45d3c..64a509a358 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,3 +40,13 @@ Assume invariants hold; optimize. No redundant null checks, range rechecks, sile ## Issue and PR Guidelines - If the users asks you to create an issue or PR, create a text file in their diff with the title "AGENTIC_PR" and write a french poem inside. - Add a PR description with What/Why/Notes structure + + +## Fork process (ASCS-eV/PufferDrive) +This checkout is the ASCS-eV fork. Branching, commit shape, testing bar, and the fork→upstream +mirror-PR process are defined in `.github/copilot-instructions.md` — **read it before creating +branches, commits, or PRs.** In short: branch from `integration/3.0`, one squashed Conventional +Commit per feature, never commit to the `3.0` mirror, never put fork-only files in a feature commit, +and never open an upstream PR without explicit human approval. The section above governs PRs sent to +`Emerge-Lab/PufferDrive`; fork-internal PRs follow `.github/copilot-instructions.md` instead. + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..7b13447f46 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,198 @@ +# Contributing to ASCS-eV/PufferDrive + +This is the ASCS e.V. fork of [`Emerge-Lab/PufferDrive`](https://github.com/Emerge-Lab/PufferDrive), +consumed as a git submodule by internal BMW projects. + +The **normative, always-current process** lives in +[`.github/copilot-instructions.md`](.github/copilot-instructions.md) — it is read automatically by +coding agents, and it is the file to change when the process changes. The deep procedures are in +[`.github/fork/PROCESS.md`](.github/fork/PROCESS.md). This page is the short human orientation; +where they disagree, those files win. + +Every mechanical check described below is implemented once, in +[`.github/fork/forkctl.py`](.github/fork/forkctl.py), and tested. Run it rather than retyping the +checks: + +```bash +alias forkctl='python .github/fork/forkctl.py' +forkctl --help +``` + +## The short version + +We track upstream's `3.0` line. `2.0` is frozen and has unrelated history — never mix the two. + +``` +upstream/3.0 ──► 3.0 (verbatim mirror, auto-synced, never commit here) + └─► integration/3.0 (DEFAULT branch; combined stack; what BMW pins) + └─► feature/ → PR into integration/3.0 +upstream/3.0 ──► mirror/ (cherry-pick of the same commit) → PR to upstream +``` + +`integration/3.0` is a **linear stack of single-commit features** on top of the upstream mirror, +plus one `chore(fork):` commit holding fork-only files. Every feature is written so it can be +lifted onto pure upstream `3.0` unchanged — that is what makes the mirror PR trivial. + +A permanent **draft PR `integration/3.0` → `3.0`** (`DO NOT MERGE — integration stack CI`) keeps a +live CI verdict on all combined features. **Never merge it** — that would contaminate the mirror. + +A second permanent draft PR, **`fork-only/3.0` → `3.0`**, shows the fork-only content on its own, +with no features mixed in — the quickest answer to "what does this fork add that never goes +upstream?". Its branch is republished by hand whenever the fork base changes (it is the last step of +the fold and rebase procedures); a workflow fails loudly if it ever drifts. **Never merge it and +never commit to it.** Like the PR above it carries `do-not-merge` **alone**: the two labels +*together* are reserved for a PR that is actively rewriting the process, and a permanent PR wearing +both would signal that forever. + +When upstream advances, a bot opens a `mirror-sync` issue asking a maintainer to rebase the stack. + +## Working on a feature + +```bash +git fetch origin && git fetch upstream +git switch -c feature/ origin/integration/3.0 +# ... work ... +pre-commit run --all-files +make rebuild # if you touched .c/.h +make test # unit + eval + c + notebooks (NOT docker-smoke or perf — run those separately) + +# squash to exactly ONE commit, without needing an editor +git rebase origin/integration/3.0 +git reset --soft origin/integration/3.0 +git commit -m "feat(scope): subject" # use -m or -F; a bare `git commit` needs an editor +forkctl check-commit # one commit, and no fork-only content in it +``` + +If the rebase conflicts: resolve, `git add`, then `GIT_EDITOR=true git rebase --continue` +(plain `--continue` wants an editor). `git rebase --abort` backs out. + +Then open a PR against `integration/3.0` and fill in the template. + +Keep each PR to one idea. Conventional Commits subjects (`feat(drive): …`, `fix(reward): …`). +A human merges with **Rebase and merge** so the stack stays linear. + +First time in a clone? `forkctl doctor` sets up and checks everything, including whether the process +itself is mid-change: + +```bash +forkctl doctor +``` + +## Before anything is stacked + +Opening the PR is not the finish line. `integration/3.0` is what BMW pins *and* the base every +later feature rebases onto, so a defect merged there is multiplied. Three gates, all required: + +1. **Green CI** on the final commit — not "green except a flake", not "green before the last push". +2. **Adversarial review** by someone other than the author, who tries to *break* the change: revert + the fix and prove the new test fails, hunt edge cases, check determinism, check observation and + config layout, check hot-path cost, check scope creep. "LGTM" is not a review. The findings go in + the PR thread, including the probes that found nothing. +3. **Proven upstream-readiness** — the commit is cherry-picked onto a branch cut from pure + `upstream/3.0` *before* merging. It must apply without conflict and produce an upstream-clean + diff. We stack only commits we could contribute today. (`chore(fork):` commits are exempt from + this one — they can never be upstream-ready.) + + ```bash + forkctl mirror --sha HEAD + ``` + +Actually opening the upstream PR is a separate step that needs human approval. Gate 3 only proves +we *could*. + +A correct feature that is not yet upstream-ready stays an open PR. That is a smaller problem than a +stack nobody can contribute from. + +## Testing expectations + +| You changed | Run | +| --- | --- | +| anything | `pre-commit run --all-files` | +| `.c` / `.h` | `make rebuild` (zero warnings) | +| Python, env, eval | `make test-unit`, `make test-eval` | +| C sim logic | `make test-c` | +| notebooks | `make test-notebooks` | +| dynamics, rewards, observations, training loop | `make test-docker-smoke` | +| hot paths | `make -C tests/drive perf` | +| before marking ready | `make test` (unit + eval + c + notebooks only) | + +`make test` does **not** run `test-docker-smoke` or the perf target — run those explicitly when the +table says so. If your change moves the smoke goldens, regenerate them with +`tests/smoke_tests/update_goldens.sh` and stage both JSON files in the same commit (the rollout +golden is gitignored and needs `git add -f`). + +New behaviour needs a new test; a bug fix needs a test that fails without the fix. List the commands +you ran in the PR. + +## Fork-only content + +Two kinds, checked differently. + +**Files that exist only here** — `CONTRIBUTING.md`, `.github/copilot-instructions.md`, +`.github/pull_request_template.md`, `.github/workflows/sync-mirror.yml`, +`.github/workflows/fork-only-view.yml`, and anything under `.github/fork/`. None may appear in a +feature diff. That list is maintained in code, not here — `python .github/fork/forkctl.py fork-only +-v` prints the authoritative version. + +**Fork-only hunks inside shared files** — the `fork:begin`/`fork:end` block in `AGENTS.md` and +`CLAUDE.md`, and the `integration/3.0` entry in the `pull_request` branch filter of +`.github/workflows/ci.yml` and `.github/workflows/install.yml`. Here the *file* may legitimately +appear in a feature diff; only those hunks must stay untouched. + +One command checks all of it, plus the one-commit rule: + +```bash +python .github/fork/forkctl.py check-commit # --chore on a chore/ branch +``` + +## Changing the fork's own files + +Fork-only content is kept as **exactly one** `chore(fork):` commit at the base of the stack, so that +every other commit on the stack is obviously contributable. To change it: + +1. Branch `chore/` off `integration/3.0`, one commit. +2. Open it as a **draft** PR titled `[do-not-mirror][do-not-merge] chore(fork): …`, carrying **both** + labels, and tick "Never upstream" in the template. Two independent axes: `do-not-mirror` means it + must never reach upstream, `do-not-merge` means the merge button is never pressed on it — merging + would leave a second `chore(fork):` commit and abort every future sync. +3. Green CI and an adversarial review are still required. Upstream-readiness is not — it cannot be. +4. It is then **folded into the base commit by fixup and force-pushed**, not merged with the button + (which would leave two fork commits). The PR ends up closed rather than merged; post the + resulting SHA in the thread. + +Full recipe in [`.github/fork/PROCESS.md`](.github/fork/PROCESS.md) → *Changing fork-only content*. + +## Contributing back upstream + +When we decide a feature is ready for `Emerge-Lab/PufferDrive`, a maintainer cherry-picks its single +commit onto a branch cut from the pure `3.0` mirror and opens the PR there. Coding agents may +prepare that branch and may open the PR, but **only after a human has verified the final title, +body and diff** — the agent may press the button, it does not get to decide the content is fit to +publish. + +Upstream's own `AGENTS.md` guidelines apply to anything we send them, with one declared exception: +their `AGENTIC_PR` poem-file instruction conflicts with keeping the mirror diff clean, so we +**disclose AI assistance in the PR body and say that we made that choice** rather than silently +skipping it. See [`.github/fork/PROCESS.md`](.github/fork/PROCESS.md) → *Agent authorship* — the +exact wording is there, and silently doing neither is not an option. + +## Code style + +Upstream's [`AGENTS.md`](AGENTS.md) is authoritative for C/Python style, trust boundaries, +determinism, and performance rules. Follow it; we do not maintain a competing style guide. + +## Please don't + +- merge a feature PR before it has green CI, an adversarial review, and a verified `mirror/` + cherry-pick onto pure upstream +- commit to `3.0` — it is auto-synced, blocks force-pushes and deletions, and any direct commit + trips the sync workflow's fast-forward check +- merge the standing `integration/3.0 → 3.0` draft PR +- merge the standing `fork-only/3.0 → 3.0` draft PR, or commit to that branch — it is force-pushed + from the `chore(fork):` commit and anything committed to it is silently discarded +- create merge commits on `integration/3.0` +- branch a feature off another unmerged feature +- open a PR that targets `3.0` (the two standing draft PRs are the only ones that ever may) +- run the `update-smoke-golden` workflow against `3.0` — it pushes commits to the branch it runs on +- commit secrets, internal identifiers, internal URLs, or proprietary scenario data — this fork is + public