From 2198c04f791d45e160811005f364943f4d81817e Mon Sep 17 00:00:00 2001 From: Carlo van Driesten Date: Fri, 23 Oct 2026 10:43:00 +0200 Subject: [PATCH] chore(fork): define the ASCS-eV fork process ASCS-eV/PufferDrive is consumed as a git submodule internally while feeding features back to Emerge-Lab/PufferDrive. This commit carries everything that exists only in the fork, and is the single place fork-only content lives. Branch model: 3.0 verbatim, fast-forward-only mirror of upstream/3.0 integration/3.0 default branch; this commit plus a linear stack of single-commit features; what internal consumers pin feature/ one feature, exactly one commit mirror/ cherry-picked onto pure upstream/3.0 for upstream PRs integration/3.0 is the default branch because GitHub reads the PR template, the contributing guidelines, the agent instructions and scheduled workflows only from the default branch, and none of those files may exist on 3.0 without destroying the mirror. A clone checks out the default branch, so the copy of the instructions here is the process as far as every agent and every submodule consumer is concerned; a rule that lives only in an open PR has no effect. Nothing reaches integration/3.0 until three gates pass: 1. CI green on the final commit. 2. Adversarial review by someone other than the author, with the findings written down -- including the probes that found nothing. 3. Upstream-readiness proven rather than asserted: the commit cherry-picks onto a branch cut from pure upstream/3.0 and produces an upstream-clean diff. We stack only commits we could contribute today. chore(fork): commits are exempt from gate 3, since fork-only infrastructure can never be upstream-ready. They carry two orthogonal markers: do-not-mirror (must never reach upstream) and do-not-merge (the merge button is never pressed -- merging would create a second chore(fork): commit and break the single-commit invariant that every sync asserts). Such a PR is folded into this commit by fixup and closed unmerged. Contents: - .github/copilot-instructions.md: normative process for coding agents - CONTRIBUTING.md: human orientation, same rules - .github/pull_request_template.md: testing and stack-hygiene gates - .github/workflows/sync-mirror.yml: fast-forward-only mirror sync that refuses any non-fast-forward instead of rewriting the upstream base - AGENTS.md / CLAUDE.md: fenced pointer so agents find the fork process without colliding with upstream-owned content - ci.yml / install.yml: run CI for PRs targeting integration/3.0, which upstream's branch filters would otherwise skip A standing draft PR from integration/3.0 into 3.0 gives a continuous CI verdict on the combined stack; it must never be merged. Everything here is fork-only and must never be cherry-picked upstream. --- .github/copilot-instructions.md | 654 +++++++++++ .github/fork/PROCESS.md | 536 +++++++++ .github/fork/forkctl.py | 1550 ++++++++++++++++++++++++++ .github/fork/test_forkctl.py | 1255 +++++++++++++++++++++ .github/pull_request_template.md | 85 ++ .github/workflows/ci.yml | 2 +- .github/workflows/fork-only-view.yml | 148 +++ .github/workflows/install.yml | 2 +- .github/workflows/sync-mirror.yml | 132 +++ AGENTS.md | 10 + CLAUDE.md | 10 + CONTRIBUTING.md | 198 ++++ 12 files changed, 4580 insertions(+), 2 deletions(-) create mode 100644 .github/copilot-instructions.md create mode 100644 .github/fork/PROCESS.md create mode 100644 .github/fork/forkctl.py create mode 100644 .github/fork/test_forkctl.py create mode 100644 .github/pull_request_template.md create mode 100644 .github/workflows/fork-only-view.yml create mode 100644 .github/workflows/sync-mirror.yml create mode 100644 CONTRIBUTING.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000000..0c941e295e --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,654 @@ +# ASCS-eV/PufferDrive — fork contribution rules + +**Audience:** every coding agent (Copilot CLI/IDE, Claude Code, Codex, …) and every human working in +`ASCS-eV/PufferDrive`. + +**This file is fork-only.** It does not exist upstream and must never reach `Emerge-Lab/PufferDrive`. + +**This file is the core.** It is loaded automatically in every session in this repository, so it +carries only what you need to avoid doing damage. The deep procedures — the ones you run rarely and +must not improvise — live in `.github/fork/PROCESS.md`, which is **not** auto-loaded. Each one is +signposted below. Read the signposted section before you run the procedure; do not reconstruct it +from memory. + +## 0. Precedence + +1. The human's explicit instruction in the current turn. +2. **This file** — fork branching, PR, and release process. +3. `.github/fork/PROCESS.md` — the same process in detail; never contradicts this file. +4. `AGENTS.md` / `CLAUDE.md` (upstream-owned) — code style, trust boundaries, guardrails. + +Rules 2–3 and 4 do not overlap: they never redefine coding standards, and `AGENTS.md` never +defines fork process. Where `AGENTS.md` speaks about issues and PRs, it is describing +**upstream's** process; for PRs raised inside this fork, section 5 below applies instead. + +### 0.1 `forkctl` — the checks live in code, not in this file + +Every mechanical check in this document is implemented once, in `.github/fork/forkctl.py`, and is +covered by `.github/fork/test_forkctl.py`. It is stdlib-only and spawns no shell, so it behaves +identically on Linux, macOS, WSL and a Windows checkout, and there is nothing to install: + +```bash +python .github/fork/forkctl.py --help +``` + +For brevity this file writes `forkctl `. Expand it to the line above, or alias it with +`alias forkctl='python .github/fork/forkctl.py'`. + +| Command | What it decides | Where it is written up | +| --- | --- | --- | +| `forkctl doctor` | is this clone set up, and is the process itself mid-change? | 4.1 | +| `forkctl check-commit [--chore]` | exactly one commit, carrying no fork-only content | 4.4, 4.5 | +| `forkctl check-stack` | the stack invariants, before any force-push | 3.3, 7 | +| `forkctl mirror --sha\|--subject` | gate 3, and building the upstream branch | 4.8, 6 | +| `forkctl status` | divergence report, stale commits, in-flight mirrors | 7, and *While the mirror PR is open* / *After upstream responds* in `PROCESS.md` | +| `forkctl delete-mirror ` | is it safe to delete this mirror branch? | *While the mirror PR is open* in `PROCESS.md` | +| `forkctl fork-only [-v]` | the canonical fork-only path list | 3.1 | + +Bare numbers are sections of this file; italics are headings in `.github/fork/PROCESS.md`. + +Each exits non-zero on failure and prints the fix, so they compose into any script and are safe in +front of a `&&`. + +**Do not retype these checks as shell one-liners.** The previous revision of this file did, and the +copies drifted: the fork-only file list existed in six places, and the `AGENTS.md` fork-block guard +never fired for an edit *between* the markers, because the changed lines carry no marker of their +own. One implementation, with tests that prove each guard fails when it should, is the point. + +### 0.2 When to open `.github/fork/PROCESS.md` + +| You are about to… | Read | +| --- | --- | +| change a fork-only file on a `chore/` branch | *Changing fork-only content* | +| adversarially review someone's feature PR | *Adversarial review* | +| open, refresh, or retire an upstream PR | *Mirror PR workflow* | +| rebase the stack onto new upstream, or cut a release | *Syncing and releasing* | +| touch either standing draft PR | *The standing draft PRs* | + +Writing a feature, running the gates, and opening a fork PR need **only this file**. That is the +common case, and it is why the detail was moved out. + +## 1. Why this fork exists + +`ASCS-eV/PufferDrive` is a fork of `Emerge-Lab/PufferDrive`, consumed as a **git submodule** by +internal BMW projects. It has two jobs: + +- Provide a **stable, integrated branch** that internal consumers can pin. +- Prepare **small, well-tested, independently reviewable features** that we offer back upstream as + mirror PRs once we consider them ready. + +Everything below exists to keep those two jobs from fighting each other. + +## 2. Branch model + +| Branch | Role | Who writes | History | +| --- | --- | --- | --- | +| `3.0` | Verbatim mirror of `upstream/3.0`; base for every upstream PR | automation only | fast-forward only | +| `integration/3.0` | **Default branch.** Combined feature stack; what BMW consumes | feature PRs only | linear, rebased | +| `feature/` | One feature, **exactly one commit** | contributor/agent | rebased onto `integration/3.0` | +| `chore/` | Fork-only change, **exactly one commit** | contributor/agent | rebased onto `integration/3.0` | +| `mirror/` | Upstream PR source | generated by cherry-pick | cut from `upstream/3.0` | +| `fork-only/3.0` | Derived **view** of the fork base commit | republished by hand (3.3, 7) | force-pushed, never committed to | +| `2.0` | Frozen legacy line | nobody | archived, do not touch | + +Remotes: `origin` = `ASCS-eV/PufferDrive`, `upstream` = `Emerge-Lab/PufferDrive`. + +`integration/3.0` is the default branch because GitHub reads the PR template, the contributing +guidelines, and scheduled workflows **only** from the default branch — and none of those fork-only +files may exist on `3.0` without destroying the mirror. A welcome side effect: new PRs default to +the correct base. + +The same reasoning governs this file. A fresh `git clone` checks out the default branch, so the copy +of `.github/copilot-instructions.md`, `AGENTS.md`, and `CLAUDE.md` **on `integration/3.0` is the +process** as far as every agent — and every BMW submodule consumer — is concerned. A process change +that exists only in an open PR does not exist. That is precisely why fork-only changes are folded +into the base commit on the default branch (section 3.3) instead of being parked on a long-lived +branch, and why section 4.1 opens by checking whether the rules are mid-change. + +``` +upstream/3.0 ──► 3.0 (mirror) + └─► integration/3.0 = 3.0 + [fork scaffolding] + [feat A] + [feat B] + … + └─► feature/ (1 commit, PR'd into integration/3.0) +3.0 ──► mirror/ (cherry-pick of that same 1 commit → upstream PR) +``` + +`upstream/2.0` and `upstream/3.0` have **no common ancestor**. Never attempt to merge, rebase, or +cherry-pick between the `2.0` and `3.0` lines. + +### Hard branch rules + +- **Nothing reaches `integration/3.0` that has not cleared the merge gate in section 4.8** — green + CI, adversarial review, and proven upstream-readiness. The stack is a line of + already-contributable commits, not a staging area for work in progress. +- **Never commit to `3.0`.** It is a pure mirror; any local commit there breaks the clean base for + every mirror PR. +- **Never create a merge commit on `integration/3.0`.** The stack stays linear so that each feature + remains a single, cherry-pickable commit. +- **Never branch a feature off another unmerged feature.** Each feature must apply to pure `3.0`. +- `integration/3.0` is maintained by rebase and is therefore **force-pushed** (`--force-with-lease`). + Contributors re-sync with `git fetch origin && git reset --hard origin/integration/3.0` after + confirming `git status` is clean — **not** `git pull --rebase`, which replays your local copies of + commits that were rewritten or dropped during the rebase and silently resurrects them. Consumers + pin an immutable tag (section 7), not the branch tip. + +**How `integration/3.0` is protected, and the deliberate hole in it.** All 12 required checks are +enforced, and "require branches to be up to date" is on, so a PR cannot be merged green against a +stale base and land an untested combination. Admin enforcement is deliberately **off**: with it on, +GitHub rejects *any* direct ref update to the branch (verified — it returns +`8 of 12 required status checks are failing`), which would make the fixup flow in section 3.3 and +the stack rebase in section 7 impossible. Those two procedures bypass the gate by design, and the +invariant checks written into them are the compensating control. Do not force-push this branch +outside those two procedures. + +### The two standing draft PRs + +Two permanent **draft PRs target `3.0`**, and both must **never be merged, never be marked ready +for review, and never be committed to**: + +- **`integration/3.0` → `3.0`** — `DO NOT MERGE — integration stack CI`. A continuous CI verdict on + all combined features at once, which no individual feature PR can provide. It refreshes itself + whenever the stack is force-pushed. +- **`fork-only/3.0` → `3.0`** — `DO NOT MERGE — fork-only content view`. Its *Files changed* tab is + exactly what this fork adds and never sends upstream, with no features mixed in. Its branch is a + derived **view** that is republished **by hand** (section 3.3 and section 7); + `.github/workflows/fork-only-view.yml` only *detects* drift, it cannot publish. + +Merging either would push fork-only content into the mirror and destroy the clean base that every +upstream PR is cut from. + +Both carry the `do-not-merge` label **alone**. Never give a standing PR `do-not-mirror` as well: the +two labels *together* are the in-flight-process-change signal that `forkctl doctor` queries, and a +permanent PR carrying both would tell every agent, forever, that the rules are mid-change. The rule +is general: **a standing artifact carries `do-not-merge` alone; both labels together mark a +`chore/` PR that is actively rewriting the process.** + +Rationale, and why the view branch cannot be automated: `.github/fork/PROCESS.md` → *The standing +draft PRs*. + +## 3. Fork-only content — never send this upstream + +There are **two kinds**, and they are checked differently. Both must stay out of every feature +commit and every mirror PR. + +### 3.1 Whole files that exist only here + +If any of these appears in a feature diff, the feature is contaminated: + +- `.github/copilot-instructions.md` (this file) +- `CONTRIBUTING.md` +- `.github/pull_request_template.md` +- `.github/workflows/sync-mirror.yml` +- `.github/workflows/fork-only-view.yml` +- anything under `.github/fork/` — the fork's own tooling and its tests + +The last entry is a **prefix, not a list**: a new fork-only tool dropped into `.github/fork/` is +covered without editing anything. The authoritative definition lives in `.github/fork/forkctl.py` +and is printed by `forkctl fork-only -v`. Do not copy it into another file; a stale copy is a guard +that has silently stopped firing. + +### 3.2 Fork-only edits inside files that also exist upstream + +These files are shared with upstream, so the **file** appearing in a diff is fine and often +correct — only the **fork-only hunk** must stay put: + +| Shared file | Fork-only part | +| --- | --- | +| `AGENTS.md`, `CLAUDE.md` | the `` … `` block | +| `.github/workflows/ci.yml`, `.github/workflows/install.yml` | the `'integration/3.0'` entry in the **`pull_request`** `branches:` filter | + +A file-name check cannot catch 3.2 — only a content check can (section 4.5). Conversely, do not +treat a shared file as untouchable: a PR that rewrites the install steps of `ci.yml` and +`install.yml` is fully upstream-contributable, because it leaves the branch filter alone. + +Do **not** add `integration/3.0` to those workflows' `push:` filters. The standing draft PR +(section 2) already runs the full suite on every update of the branch, so a `push` trigger only +duplicates every job and doubles runner consumption. + +That reasoning is about the **test suites** specifically. `fork-only-view.yml` deliberately does use +a `push:` trigger on `integration/3.0`: it runs a handful of git commands rather than a test matrix, +and it exists precisely to react to that branch moving. Do not "fix" it to match the others. + +Fork-only content lives in **one dedicated commit** (`chore(fork): …`) sitting directly on top of +`3.0` at the base of the integration stack. That commit is never cherry-picked. + +### 3.3 Changing fork-only content + +Fork-only content lives in **one dedicated commit** (`chore(fork): …`) sitting directly on top of +`3.0` at the base of the integration stack. That commit is never cherry-picked, and there is exactly +one of it — `forkctl check-stack` asserts that, so a second one breaks every future sync. + +Changing it therefore has its own shape: + +- branch `chore/` from `origin/integration/3.0`, **exactly one commit**; +- open the PR as a **draft** titled `[do-not-mirror][do-not-merge] chore(fork): …`, with **both** + the `do-not-mirror` and `do-not-merge` labels; +- Gates 1 and 2 (section 4.8) apply in full; Gate 3 does not; +- it is **never merged with the button** — it is folded into the base commit by fixup and ends + **closed, not merged**. + +Those are two independent axes and both must be marked, because they fail in different directions. +`do-not-mirror` means the change must never reach upstream; `do-not-merge` means the merge button +must never be pressed, because merging would put a *second* `chore(fork):` commit on the stack and +every future sync would abort until someone unpicked it by hand. + +The fold force-pushes the default branch, so it is a maintainer action: an agent must not perform it +unasked (section 5). + +**The full procedure — the fixup, the invariant re-check, and the atomic push that moves +`integration/3.0` and `fork-only/3.0` together — is in `.github/fork/PROCESS.md` → *Changing +fork-only content*. Read it before you start; do not reconstruct it from memory.** + +**Keep the window short.** Between opening a `chore/` PR and folding it in, the default branch +still serves the *old* rules, and every agent that clones the fork in that window follows them — see +section 2. A fork-only PR is a **review vehicle, not a delivery mechanism**: nothing in it is in +force until the force-push lands. So do not park one open across unrelated work, and when one has +been open a while, assume agents have already acted on the superseded text and check their output +against the new rules rather than assuming compliance. + +If a feature genuinely needs to change one of these files, make it a **separate** `chore(fork):` +commit — do not fold it into the feature commit. +## 4. Feature workflow + +### 4.1 Start + +One-time setup in a fresh clone — `upstream` is not configured automatically, and several commands +below (and Gate 3) fail without it: + +```bash +git remote get-url upstream >/dev/null 2>&1 \ + || git remote add upstream https://github.com/Emerge-Lab/PufferDrive.git +gh repo set-default ASCS-eV/PufferDrive # else bare `gh pr …` can resolve to upstream +uv venv && source .venv/bin/activate && uv pip install -e . +``` + +Then check the clone, and whether the process itself is mid-change: + +```bash +forkctl doctor +``` + +It verifies the remote and the virtualenv, and lists any open PR carrying **both** fork-only +labels. Such a PR is rewriting the rules you are about to follow: anything it reports supersedes +this file in the areas it touches, so read it first and rebase onto the result rather than racing +it. (The query ANDs both labels, so it matches process PRs and nothing else — the two standing PRs +of section 2 carry `do-not-merge` alone and are correctly excluded. That exclusion is the whole +reason a standing artifact must never be given the second label.) + +Then: + +```bash +git fetch origin && git fetch upstream +git switch -c feature/ origin/integration/3.0 +``` + +`` is short, lowercase, hyphenated, and describes the change (`lane-heading-obs`, not +`fix-stuff`). + +For fork-only work that will become a `chore(fork):` commit, name the branch `chore/` +instead. Everything else in section 4 applies unchanged, except Gate 3 (section 4.8), from which +such commits are exempt. + +### 4.2 Scope + +One feature = one reviewable idea. If the PR description needs the word "and" to list unrelated +changes, split it. Refactors that merely enable a feature go in their own earlier PR. + +### 4.3 Build and test + +Makefile targets assume a POSIX shell and `.venv/bin/python` — use Linux, macOS, or WSL. + +| Situation | Required before pushing | +| --- | --- | +| any change | `pre-commit run --all-files` | +| `.c` / `.h` touched | `make rebuild` (zero warnings — warnings are bugs) | +| Python / env / eval | `make test-unit`, `make test-eval` | +| C sim logic | `make test-c` | +| notebooks or their deps | `make test-notebooks` | +| **sim dynamics, rewards, observations, or training loop** | `make test-docker-smoke` (golden train/rollout smoke run) | +| perf-sensitive hot paths | `make -C tests/drive perf` | +| everything in the default suite, before marking ready | `make test` | + +`make test` is **not** everything: it runs `test-unit test-eval test-c test-notebooks` and +deliberately excludes `test-docker-smoke` (slow, needs Docker) and the perf target. If the table +above says you need those, run them explicitly — `make test` passing does not cover them. + +**Regenerating smoke goldens.** If your change legitimately moves the golden train/rollout values +(observation layout, dynamics, rewards, RNG draw order), regenerate them with +`tests/smoke_tests/update_goldens.sh`, which runs inside the pinned QEMU image so the numbers match +CI. Review the diff, then stage both files **in the same single commit as the change that caused +them** — the rollout golden is gitignored and needs `-f`: + +```bash +tests/smoke_tests/update_goldens.sh +# tests/smoke_tests/data/ matches .gitignore, so BOTH goldens need -f even though +# both are already tracked — a plain `git add` on them exits 1. +git add -f tests/smoke_tests/data/drive_smoke_golden.json \ + tests/smoke_tests/data/drive_rollout_golden.json +``` + +Goldens are **base-dependent**: values regenerated on `integration/3.0` are not necessarily valid on +pure `upstream/3.0`. If Gate 3 (section 4.8) shows the smoke test failing on `mirror/`, +regenerate there and carry the corrected values back, rather than assuming our stack's numbers are +upstream's. + +**New behaviour requires a new test.** A bug fix requires a test that fails before the fix and +passes after it. "Existing tests still pass" is not sufficient evidence for a feature PR. + +Paste the commands you actually ran, and their outcome, into the PR's *How tested* section. Do not +claim a suite passed unless it ran to completion in this environment. + +### 4.4 Shape the commit + +Squash to **exactly one commit** before requesting review: + +```bash +git fetch origin +git rebase origin/integration/3.0 # replay onto the current base +git reset --soft origin/integration/3.0 # collapse the branch into the index +git commit -F # or -m; never rely on an editor opening +forkctl check-commit +``` + +If the rebase conflicts: resolve, `git add` the files, then +`GIT_EDITOR=true git rebase --continue` — plain `--continue` opens an editor and fails outright when +none is available. To back out, `git rebase --abort`. + +Use the non-interactive form above rather than `git rebase -i`: agents run without an editor, where +the interactive rebase either stalls or silently does nothing. + +Conventional Commits, imperative mood, ≤ 72-char subject: + +``` +feat(drive): add lane-heading error to the observation vector + +Why the change is needed, what the approach is, and anything a reviewer +would otherwise have to reverse-engineer. +``` + +Types: `feat`, `fix`, `perf`, `refactor`, `test`, `docs`, `build`, `ci`, `chore`. +Common scopes: `drive`, `obs`, `reward`, `train`, `eval`, `render`, `config`, `fork`. + +`chore(fork): …` marks a commit that **must never go upstream**. + +Do not add AI/agent authorship trailers or "generated by" notes to commits, or to **fork** PR text. + +The single exception is the upstream authorship disclosure required by section 6, which is +**mandatory** on mirror PR bodies. That is a disclosure to upstream, not an authorship trailer, and +this rule does not license omitting it. + +### 4.5 Verify upstream-cleanliness before opening the PR + +```bash +forkctl check-commit +``` + +It asserts exactly one commit, no fork-only **file** in the diff, and no fork-only **hunk** or +**block** touched in a shared file. It exits non-zero and names the offender; a check that prints +contamination and exits `0` is not a check. Changing the *rest* of a shared file is allowed and is +sometimes the whole point of the PR — only the fork-only parts of section 3.2 are protected. + +This is the cheap pre-flight check. The authoritative proof is Gate 3 in section 4.8. + +**`chore/` branches pass `--chore`** — their whole purpose is to change the files in section +3.1, so the fork-only checks are skipped and only the one-commit rule is enforced. The flag also +asserts the subject really is `chore(fork): …`, so it cannot be used to wave a feature through: + +```bash +forkctl check-commit --chore +``` + +### 4.6 Open the fork PR + +Base **`integration/3.0`** (the default, so it is pre-selected). Fill in +`.github/pull_request_template.md` completely and let CI finish. + +Opening the PR is **not** the end of your job — a PR that is merely open is not stacked. It must +clear section 4.8 first. + +### 4.7 Adversarial review + +Every feature PR is adversarially reviewed before it may be stacked. The reviewer's job is to +**try to break the change**, not to confirm that it looks reasonable. "LGTM" is not a review. + +**The reviewer is not the author.** When an agent wrote the change, the review runs as a *separate +task with fresh context* — a reviewer still holding the author's reasoning reproduces the author's +blind spots. + +Post the outcome as a PR review listing what was probed and what was found — **including the probes +that found nothing**, so the next reader knows the coverage. Each finding is either fixed or +explicitly accepted by a human, with the reason in the thread. + +**The checklist the reviewer works through is in `.github/fork/PROCESS.md` → *Adversarial review*.** +It covers test validity, breaking inputs, determinism, trust boundaries, layout compatibility, +performance, scope creep and upstream reception. Read it when reviewing; you do not need it when +authoring. +### 4.8 The merge gate — nothing stacks until it is proven + +`integration/3.0` is what BMW pins *and* the base that every later feature rebases onto, so a defect +merged here is multiplied across every downstream consumer and every subsequent PR. Merge a feature +PR **only** when all applicable gates are green: + +**Gate 1 — CI is green.** Every required check passes *on the final commit*. Not "green apart from a +known flake", not "green before the last force-push". This is enforced by GitHub, not by good +intentions: the required set is configured as branch protection on `integration/3.0`, so the merge +button stays disabled until all of it is green. Confirm with: + +```bash +gh pr checks --repo ASCS-eV/PufferDrive +``` + +If a check is broken for reasons unrelated to the feature, fix that first as its own PR — a CI fix +is itself a perfectly good feature PR — instead of merging past it. + +**If the commit adds or changes a workflow, Gate 1 also requires that the workflow has actually +run.** The required checks validate the repository; they do not execute your workflow, so a green +board says nothing about it. Dispatch it on the PR ref and link the run: + +```bash +gh workflow run .yml --ref +gh run list --workflow .yml --limit 1 --json databaseId,conclusion +``` + +Exercise each state it is supposed to distinguish, not just the happy path. This rule exists because +a fork-only workflow once cleared Gate 1 **and** Gate 2 and was still impossible: GitHub rejects any +`GITHUB_TOKEN` push that touches `.github/workflows/`, and nothing short of running it could have +revealed that. Assertions about a workflow are worth exactly as much as the runs behind them. + +**Gate 2 — adversarial review passed.** Section 4.7, findings written down and resolved. + +**Gate 3 — upstream-readiness is proven, not asserted.** Build the mirror branch *before* merging +and let the cherry-pick prove the commit is contributable as-is: + +```bash +git fetch upstream +forkctl mirror --sha +``` + +It recreates `mirror/` from pure `upstream/3.0`, cherry-picks the commit, asserts the result +contains no fork-only file and adds no fork-only marker anywhere, prints the file list for review, +and returns you to the branch you started on. It exits non-zero on any of those, and leaves the +branch in place for inspection when it fails. + +Recreating `mirror/` every time is deliberate — section 7 requires re-proving this gate after +a rebase. If the cherry-pick conflicts, the feature is entangled with fork-only state: fix it on +the feature branch and re-run. + +**If the change can move the smoke goldens** (observation layout, dynamics, rewards, RNG order), +Gate 3 must also *run* the relevant suites from section 4.3 on the mirror branch, not just diff it. +Goldens generated on our stack are not evidence about pure upstream. A commit whose goldens depend +on an earlier feature that upstream does not have is **not** upstream-ready: it does not stack until +that dependency is resolved. + +Actually *opening* the upstream PR stays a separate, human-approved step (sections 5 and 6) — the +gate is only that we could open it today. Record the resulting `integration/3.0` SHA on the PR when +the feature is stacked, so section 6 can rebuild the mirror later from a known commit. + +**Exemption — `chore(fork):` commits.** Fork-only infrastructure cannot be upstream-ready by +definition, so Gate 3 does not apply, and section 4.5 is run with `--chore`, which skips the +fork-only checks — touching those files is the entire point. **Every other rule still holds**, in +particular "exactly one commit": section 3.3 cherry-picks a single SHA, so a two-commit chore branch +silently loses work. Gates 1 and 2 apply in full, and the reviewer additionally confirms the change +is genuinely fork-only and contains nothing upstream would have wanted. Such a PR is a draft titled +`[do-not-mirror][do-not-merge] …` and carries both the `do-not-mirror` and `do-not-merge` labels; +it is applied by fixup into the base +commit rather than merged (section 3.3), so it ends **closed, not merged** — the "Rebase and merge" +step below does not apply to it. + +Only once the applicable gates pass, **a human presses "Rebase and merge"** (never squash-merge a +branch that is already one commit, never create a merge commit). Squash and merge commits are +disabled on this repository so the wrong button cannot be pressed by accident. + +If a feature is good but not yet upstream-ready, it does **not** go on the stack. Leave the PR open +and say why. An unmergeable-but-correct feature is a smaller problem than a stack nobody can +contribute from. + +## 5. Agent rules for issues and PRs + +- Agents **may** open, update, and comment on PRs and issues **in `ASCS-eV/PufferDrive`**. +- An agent **must not** adversarially review (section 4.7) a change it authored in the same session. + Start a fresh review task with no authoring context, or hand it to a human. +- Agents **must not** merge into `integration/3.0` on their own initiative. Report the gate status + from section 4.8 and let a human decide. +- **A human must verify the final PR title, body, and diff before an agent opens an _upstream_ PR.** + The agent may press the button; it may not decide that content is fit to send to another project. + Show the exact text and the diff, get explicit approval, then open it. That approval is **per PR, + in the current conversation**, and covers the `mirror/` content too. + Fork-internal PRs need no such approval — open them, fill the template honestly, and report the + gate status. The human gate on fork PRs is the **merge** button (above), not the open button. +- Never open a PR targeting the `3.0` mirror. The only PRs that may ever target it are the two + standing drafts from section 2 — the integration stack CI verdict and the fork-only view. Each is + opened **once, by a human**, and must never be merged. The integration draft refreshes itself + whenever `integration/3.0` moves; the fork-only view does **not** — its branch is republished by + hand (section 2), and a workflow fails until that happens. +- When a PR targets upstream, **upstream's `AGENTS.md` "Issue and PR Guidelines" apply**, with one + documented exception handled in section 6. Do not import them into fork PRs. +- Never force-push a branch you did not create. Never force-push `integration/3.0` without being + asked to. +- Never commit secrets, internal BMW identifiers, customer data, internal URLs, or proprietary + scenario data. This fork is public. + +## 6. Mirror PR workflow (fork → upstream) + +Once a feature is stacked, offering it upstream is a separate, **human-approved** step (section 5). + +Gate 3 already proved the commit cherry-picks cleanly, but do not assume the branch it produced +still exists — it is local-only. **Always rebuild the mirror from scratch**, and locate the commit +**by subject, not by the SHA recorded on the feature PR**: every sync rebase rewrites the stack, so +a recorded SHA stops resolving as soon as `3.0` advances. + +```bash +git fetch origin && git fetch upstream +forkctl mirror --subject "feat(scope): the exact commit subject" --stay +``` + +If it aborts, the feature commit is contaminated or no longer applies — fix it on +`integration/3.0` and regenerate. Otherwise push `mirror/` to `origin` and open the upstream +PR from it **after a human has verified the title, body and diff**. + +Re-run the relevant suites from section 4.3 on the mirror branch: it is a different base from the +one CI tested, and "it passed on our stack" is not evidence that it passes on pure upstream. + +Three things an upstream PR gets wrong by default: + +- **Structure is upstream's**: *What / Why / Notes*. Give them evidence, not our internal process + vocabulary — upstream does not know or care what our integration stack is. +- **Title style is upstream's, not ours.** They use imperative sentence case, not Conventional + Commits; a `fix(ci):` prefix marks the PR as foreign on sight. The *commit* keeps our + Conventional Commits subject — only the PR title is restyled. +- **Every upstream PR body must carry the AI-assistance disclosure** we use in place of upstream's + `AGENTIC_PR` canary. Omitting both the file and the disclosure is not an option: it would quietly + defeat a detection mechanism the maintainers put there on purpose. + +**The exact disclosure text, keeping an open PR fresh while upstream reviews it, the +branch-deletion hazard, and what to do after upstream merges, amends or rejects, are in +`.github/fork/PROCESS.md` → *Mirror PR workflow*. Read it before touching an upstream PR.** + +Two commands cover the routine half of that: + +```bash +forkctl status # what is in flight, and which mirrors have gone stale +forkctl delete-mirror # refuses while the upstream PR is still open +``` +## 7. Syncing and releasing + +**Sync the mirror:** automated. `.github/workflows/sync-mirror.yml` fast-forwards `3.0` from +`upstream/3.0` daily and on demand (Actions → *Sync 3.0 mirror* → *Run workflow*). It refuses any +non-fast-forward, which is the alarm that someone committed to the mirror. `3.0` additionally blocks +force-pushes, deletions, and non-linear history. **Never push to `3.0` by hand.** + +When the mirror moves ahead of the stack, the same workflow opens (and keeps updated) a +`mirror-sync` tracking issue, and closes it once `integration/3.0` contains the mirror head. The +rebase itself stays manual — replaying features onto new upstream code can conflict and must be +re-tested. + +**Rebasing the stack, and tagging a release, force-push or move refs on the branch BMW pins.** Both +are maintainer actions; an agent must not perform either unasked (section 5). The procedure, the +conflict policy, the explicit `--force-with-lease=:` rationale and the tagging steps are +in `.github/fork/PROCESS.md` → *Syncing and releasing*. + +**Hazard — never dispatch `update-smoke-golden.yml` against `3.0`.** This inherited upstream +workflow ends in `git push origin HEAD:` and triggers on pushes to `3.0` as well as +`workflow_dispatch`, so dispatching it there commits to the mirror and destroys the clean base every +upstream PR is cut from. No `forkctl` command can stop this — a workflow dispatch is not a git +operation — so it is the one destructive action guarded by nothing but this sentence. Regenerate +goldens with `tests/smoke_tests/update_goldens.sh` on your own branch instead (section 4.3). +Recovery is in `.github/fork/PROCESS.md` → *Syncing and releasing*. + +**Health check — run it after every sync and before any release:** + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It reports the total gap, the permanent fork-only floor, the feature gap, any commits upstream +already has (which the last sync should have dropped), and every in-flight mirror PR. + +**The total gap can never reach zero, and must not be driven to zero.** `integration/3.0` +permanently carries at least one `chore(fork):` commit, because the files in section 3.1 do not +exist upstream by design. An agent that reads a non-zero gap as a defect and tries to close it by +upstreaming that commit would leak the entire fork-only process into `Emerge-Lab/PufferDrive` — +precisely what Gate 3 exists to prevent. + +**The number that should trend to zero is the feature gap.** Zero means upstream has absorbed +everything we carry and the fork is pure scaffolding again; that is the steady state we are aiming +for, not an exceptional one. A feature gap that only grows means we are accumulating a private fork +rather than contributing — treat a persistently rising count as a process failure to discuss, not a +number to ignore. Every feature commit on the stack should be traceable to an open or pending mirror +PR (section 6); one that is not is either unfinished work or a change nobody intends to upstream, +and the second kind belongs in a `chore(fork):` commit where the guards can see it. +## 8. Definition of done + +A feature PR may be **stacked onto `integration/3.0`** only when **all** of the following hold. +Anything less stays an open PR. + +Authoring: + +- exactly one commit, Conventional Commits subject, rebased on current `integration/3.0` +- no fork-only file from section 3.1 and no fork-only hunk from section 3.2 in its diff +- new or updated tests cover the new behaviour; a fix has a test that fails without it +- the suites from section 4.3 that apply were run to completion and are listed in the PR +- the PR template is filled in, including the upstream-readiness assessment + +Gates (section 4.8): + +- **Gate 1** — every required check green on the final commit +- **Gate 2** — adversarial review (section 4.7) posted, findings fixed or explicitly accepted +- **Gate 3** — `mirror/` cherry-picks onto pure `upstream/3.0` without conflict and its diff + is upstream-clean — i.e. the feature is genuinely contributable today + *(exempt for `chore(fork):` commits, which can never be upstream-ready)* + +Merged by a human with **Rebase and merge**, leaving the stack linear. + +A `chore(fork):` PR has a different definition of done: exactly one commit, Gates 1 and 2 green, +Gate 3 and the section 4.5 fork-only checks exempt, all four `do-not-mirror` markers **and** both +`do-not-merge` markers present, and it is **folded into the base commit by fixup and the PR closed +unmerged** (section 3.3) — never merged with the button. + +The machine-checkable half of this list is two commands. Neither replaces Gates 1 and 2, which are +human and CI judgements: + +```bash +forkctl check-commit # authoring bullets 1-2 (--chore for a chore(fork): branch) +forkctl mirror --sha HEAD # Gate 3 (skip for chore(fork):) +``` diff --git a/.github/fork/PROCESS.md b/.github/fork/PROCESS.md new file mode 100644 index 0000000000..99b76e59c7 --- /dev/null +++ b/.github/fork/PROCESS.md @@ -0,0 +1,536 @@ +# ASCS-eV/PufferDrive — fork process reference + +**Fork-only. Never cherry-pick this file upstream.** + +This is the *detail*. The normative core lives in `.github/copilot-instructions.md`, which every +agent loads automatically; this file is deliberately **not** auto-loaded, because most sessions in +this repository touch the simulator and never touch the fork process. + +Read the section you need, when you need it. The core tells you which one: + +| You are about to... | Read | +| --- | --- | +| merge or close one of the two standing draft PRs | [Standing PRs](#the-standing-draft-prs) | +| change a fork-only file (`chore/`) | [Changing fork-only content](#changing-fork-only-content) | +| review someone else's feature PR | [Adversarial review](#adversarial-review) | +| open, refresh, or retire an upstream PR | [Mirror PR workflow](#mirror-pr-workflow-fork--upstream) | +| rebase the stack onto new upstream, or cut a release | [Syncing and releasing](#syncing-and-releasing) | + +Anything not covered here is in the core. Where the two disagree, the core wins — but they should +not disagree, because the *mechanical* rules live in neither: they live in `forkctl.py`, which is +tested. + +## The standing draft PRs + +### The integration stack CI draft + +A permanent **draft PR `integration/3.0` → `3.0`** titled `DO NOT MERGE — integration stack CI`, +carrying the `do-not-merge` label, gives us a continuous, always-visible CI verdict on *all combined +features at once*, which no individual feature PR can provide. + +- **Never merge it. Never mark it ready for review.** Merging would push the fork's stack into the + mirror and destroy the clean base that every upstream PR is cut from. +- It is force-pushed whenever the stack is rebased; its checks simply re-run. A red verdict means + the combined stack is broken even if every feature PR was green — fix it before tagging a release. +- Close and reopen it only if it is accidentally merged or the branch is recreated. + +### The fork-only content view draft + +A second permanent **draft PR `fork-only/3.0` → `3.0`** titled `DO NOT MERGE — fork-only content +view`, carrying the `do-not-merge` label, answers +one question that nothing else answers cheaply: *what does this fork add that never goes upstream?* +Its **Files changed** tab is exactly the fork-only content, with no features mixed in — the standing +integration draft PR above cannot show this, because its diff necessarily contains the whole feature +stack as well. + +- **It is a view, not storage.** Fork-only content lives in the single `chore(fork):` commit on + `integration/3.0`, because that is the only place agents, CI and the internal submodule pin can + see it. Parking it on a long-lived branch instead would stop the scheduled mirror sync from firing + and strip CI from every feature PR — see the default-branch note above. +- **The branch is published by hand, not by the workflow**, as the last step of the two procedures + that already force-push with human credentials (sections 3.3 and 7): + + ```bash + git push --force origin :refs/heads/fork-only/3.0 + ``` + + This is not a style choice. The fork base is made almost entirely of files under + `.github/workflows/`, and GitHub **blocks `GITHUB_TOKEN` from pushing any commit that creates or + updates a workflow file** — a platform rule that no `permissions:` block can lift. Automating it + would mean storing a PAT or GitHub App token with `workflows` write in a public repository, which + is a far worse trade than one extra line in two procedures that are already manual. +- `.github/workflows/fork-only-view.yml` is therefore a **detector, not a publisher**, and needs no + write access at all (`contents: read`). It re-derives the fork base, **fails if `fork-only/3.0` is + missing or stale** and prints the exact command to fix it, refuses to pass a stack that violates + the stack invariants in *Syncing and releasing*, and checks that the standing PR still has base `3.0`, draft status and + the `do-not-merge` label. It runs on pushes to `integration/3.0` **and to `fork-only/3.0`** (so + tampering with the view itself is caught), plus daily, plus on demand. Drift is loud, not silent. +- **The workflow does not open the PR either; a human does, once** — this repository deliberately + withholds PR-creation rights from `GITHUB_TOKEN` (`can_approve_pull_request_reviews: false`). + Afterwards the workflow only *checks* it: it warns when the standing PR is missing, and **fails + loudly if one was ever merged**, which would mean the mirror is contaminated. +- **Never merge it, never commit to it.** Merging would push fork-only content into the mirror; + committing to it would be silently discarded by the next republish. + +**It carries `do-not-merge` alone, never `do-not-mirror`** — exactly like the integration draft PR +above, and for a mechanical reason: both labels *together* are the in-flight-process-change signal +that section 4.1 queries. A permanent PR carrying both would tell every agent, forever, that the +rules are mid-change. The rule is therefore general: **a standing artifact carries `do-not-merge` +alone; both labels together mark a `chore/` PR that is actively rewriting the process.** + +Because the PR targets `3.0`, each refresh re-runs the `pull_request` workflows on it. That cost is +accepted: suppressing it would mean adding more fork-only hunks to `ci.yml`, `install.yml` and +`pre-commit.yml`, and every extra hunk in a shared file is another way a feature commit can get +contaminated. The fork base changes rarely, so the runs are rare too. + +## Changing fork-only content + +The "exactly one `chore(fork):` commit" rule is what keeps the stack trivially rebasable onto new +upstream, and keeps "is this commit contributable?" an obvious yes for every *other* commit on the +stack. Maintaining that invariant has consequences: + +1. Branch `chore/` from `origin/integration/3.0`. Make the change as one commit. +2. Open the PR **as a draft**, titled `[do-not-mirror][do-not-merge] chore(fork): …`, and apply + **both** the `do-not-mirror` and `do-not-merge` labels. +3. These are two independent axes and both must be marked, because they fail in different + directions: + - **`do-not-mirror`** — the change must never reach upstream. Marked four times over, deliberately + redundantly: title prefix, label, `chore(fork):` commit type, and the "Never upstream" box in + the PR template. + - **`do-not-merge`** — the merge button must never be pressed on it. Merging would put a *second* + `chore(fork):` commit on the stack, and `forkctl check-stack` asserts there is exactly one, so + **every future sync would abort** until someone unpicked it by hand. Marked twice: title prefix + and label. + + An agent that misses all of these is not reading. +4. Gates 1 and 2 (core section 4.8) apply in full. Gate 3 does not — see the exemption there. +5. When the gates pass, mark it ready for review, then **do not press the merge button.** Apply it + by folding it into the base commit instead: + +```bash +git switch integration/3.0 +git fetch origin +test -z "$(git status --porcelain)" || { echo "ABORT: dirty tree"; exit 1; } + +START=$(git rev-parse origin/integration/3.0) # capture the lease target before rewriting +git reset --hard "$START" + +# Confirms the stack is sound and returns the fork base, which is the oldest +# commit on it. With --print-base the report goes to stderr and only the SHA to stdout. +BASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +git cherry-pick --no-commit +git commit --fixup "$BASE" +GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/3.0 +make test +``` + +Note `git cherry-pick` takes **one** commit — which is safe only because a `chore/` branch is +required to be exactly one commit (section 4.1). Confirm that before you start, or you will silently +apply only its tip. + +Never use an interactive `git rebase -i` here. Agents run without an editor, where it silently +becomes a no-op — and the next command force-pushes the default branch. The `GIT_SEQUENCE_EDITOR=:` +form above is deterministic and needs no editor. + +Verify the invariant actually held **before** you push: + +```bash +# Re-checks every stack invariant, including that the fixup! was actually folded in, +# and returns the new fork base. +NEWBASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +# Move both refs or neither. The fold changes the fork base, so the view branch is stale the +# instant integration/3.0 lands; a two-step push leaves a window in which the detector reports +# drift that is already fixed, and a rejected lease must not move the view on its own. +git push --atomic --force-with-lease=refs/heads/integration/3.0:"$START" \ + origin integration/3.0 "+$NEWBASE:refs/heads/fork-only/3.0" +``` + +(Use the explicit `--force-with-lease=:` form — see *Syncing and releasing* for why the bare form is +unsafe. The shell-pitfall notes that used to live here went away with the shell: `forkctl` does the +checking, and its tests prove each guard fails when it should.) + +This rewrites the default branch after only `make test`, which is narrower than the required check +set. Watch the standing draft PR afterwards and be ready to fix forward quickly. + +The PR then ends up **closed, not merged**. Post the resulting `integration/3.0` SHA in the PR +thread so the trail survives; a closed-unmerged PR with no explanation looks like abandoned work. + +**Keep the window short.** Between opening a `chore/` PR and folding it in, the default branch +still serves the *old* rules, and every agent that clones the fork in that window follows them — see +section 2. A fork-only PR is a **review vehicle, not a delivery mechanism**: nothing in it is in +force until the force-push above lands. So do not park one open across unrelated work, and when one +has been open a while, assume agents have already acted on the superseded text and check their +output against the new rules rather than assuming compliance. + +If a feature genuinely needs to change one of these files, make it a **separate** `chore(fork):` +commit — do not fold it into the feature commit. + +## Adversarial review + +Every feature PR is adversarially reviewed before it may be stacked. The reviewer's job is to +**try to break the change**, not to confirm that it looks reasonable. "LGTM" is not a review. + +**The reviewer is not the author.** When an agent wrote the change, the review runs as a *separate +task with fresh context* — a reviewer still holding the author's reasoning reproduces the author's +blind spots. + +Work through all of these, and answer them in writing: + +- **Does the test actually test it?** Revert the source change, keep the new test, confirm the test + fails. A test that passes against unfixed code proves nothing. +- **What input breaks it?** Empty scenario, single agent, max agents, zero-length trajectory, NaN, + out-of-range action, a controlled agent removed mid-episode. +- **Determinism.** Same seed, same result? Any new iteration over an unordered container, + uninitialised read, or change to float accumulation order? +- **Trust boundaries and memory.** Every rule in `AGENTS.md` that touches the changed code — + bounds, ownership, lifetime, null handling. +- **Layout and compatibility.** Observation vector, action space, config keys, checkpoint format. + Any change here breaks submodule consumers and must be called out explicitly in the PR. +- **Performance.** Did a hot path gain an allocation, a branch, or a cache miss? If plausible, + require `make -C tests/drive perf` numbers rather than an opinion. +- **Scope.** Anything in the diff that is not the one stated idea: drive-by renames, unrelated + reformatting, leftover debug code, commented-out blocks. +- **Upstream reception.** Would an upstream maintainer request changes? Naming, file placement, + config surface, a knob we want but they would not. Fix it now, not after they ask. + +Post the outcome as a PR review listing what was probed and what was found — **including the probes +that found nothing**, so the next reader knows the coverage. Each finding is either fixed or +explicitly accepted by a human, with the reason in the thread. + +## Mirror PR workflow (fork → upstream) + +Gate 3 (section 4.8) already proved this commit cherry-picks cleanly. Do **not** assume the branch +it produced still exists: it is local-only, the feature branch is deleted from `origin` on merge, +and the agent opening the upstream PR is usually not the one that ran the gate. **Always rebuild the +mirror branch from scratch.** + +Locate the commit **by subject, not by the SHA recorded on the feature PR**. Every sync rebase +rewrites the whole stack, so a recorded SHA stops resolving as soon as `3.0` advances. Treat it as a +hint for a human reader, never as an input to a command. + +```bash +git fetch origin && git fetch upstream +forkctl mirror --subject "fix(scope): the exact commit subject" --stay +``` + +`--subject` resolves the commit against the live stack and aborts unless **exactly one** commit +matches, so an ambiguous or stale subject cannot silently mirror the wrong change. `--stay` leaves +you on `mirror/`, ready to push. The same fork-only file and marker guards as Gate 3 run +here: this is the authoritative check, not a formality, because the stack may have been rebased +since the gate last passed. + +If a guard aborts, the feature commit was contaminated — fix it on `integration/3.0` and regenerate. +Otherwise push `mirror/` to `origin` and open the upstream PR from it **after a human has +verified the title, body and diff** (section 5). + +Re-run the relevant suites from section 4.3 on the mirror branch: it is a different base from the +one CI tested, and "it passed on our stack" is not evidence that it passes on pure upstream. + +Keep the upstream PR description in upstream's **What / Why / Notes** structure. Give them the +evidence, not our internal process vocabulary — upstream does not know or care what our integration +stack is. + +**Match upstream's title convention, not ours.** This fork uses Conventional Commits; upstream does +not. Their merged PRs read as imperative sentence case — *"Remove native Drive rendering in favor of +HTML visualization"*, *"Add synchronized native and PyTorch profiling"*. A `fix(ci):` prefix marks +the PR as foreign on sight. The *commit* keeps our Conventional Commits subject; only the PR title +is restyled. Check their recent merged PRs before writing one: + +```bash +gh pr list --repo Emerge-Lab/PufferDrive --state merged --limit 10 --json title --jq '.[].title' +``` + +**Expect no checks at first.** GitHub holds workflow runs on fork PRs from first-time contributors +in an `action_required` state until a maintainer presses *Approve and run workflows*. An upstream PR +reporting "no checks reported" is normal and is not a defect in the branch: + +```bash +gh run list --repo Emerge-Lab/PufferDrive --limit 10 \ + --json conclusion,workflowName,headBranch --jq '.[] | select(.headBranch=="mirror/")' +``` + +Once approved, those runs use the workflow files from the PR's merge commit, so a workflow change we +are proposing is exercised by the very checks that gate it. + +### Agent authorship and upstream's `AGENTS.md` + +Upstream's `AGENTS.md` asks that an agent creating a PR add a file named `AGENTIC_PR` containing a +French poem. It is an agent-detection canary, and it collides with Gate 3, which requires the mirror +diff to contain nothing but the feature's own files. + +**Our resolution: disclose instead of complying, and say that we are doing so.** Every upstream PR +body ends with a note to this effect: + +> Prepared with AI assistance and reviewed by a human before submission. We noticed the `AGENTIC_PR` +> instruction in `AGENTS.md` — we are disclosing here rather than adding a poem file, to keep the +> diff limited to the change itself. Happy to add the file instead if you would prefer it. + +This is a deliberate, declared deviation, and it is **not negotiable downward**: never omit the file +*and* omit the disclosure. That would be quietly defeating a detection mechanism the maintainers put +there on purpose, and the mirror-PR relationship depends entirely on them trusting what we send. If +upstream tells us they want the file, add it and update this section. + +### While the mirror PR is open + +A feature is normally **stacked on `integration/3.0` and open upstream at the same time**, often for +weeks. That overlap is deliberate, and sustaining it requires nothing: `mirror/` is parented +on `upstream/3.0`, never on `integration/3.0`, so rebasing the stack cannot touch it, and the +upstream PR (`ASCS-eV:mirror/` → `Emerge-Lab:3.0`) is unaffected by the fork PR being merged +or closed. Our fork deletes `feature/` automatically on merge; it does not touch `mirror/*`, +and upstream cannot delete branches that live in our fork. + +So the default action during this period is **nothing**. Three events change that. + +**What is open when.** The fork feature PR is *not* the long-lived artifact — it closes the moment +the feature stacks: + +| Event | Fork feature PR | `mirror/` PR | Stack depth | +|---|---|---|---| +| Gates pass, human merges (4.8) | **merged, closed** | not open yet | +1 | +| Mirror pushed, upstream PR opened (6) | — | **open** | unchanged | +| Upstream merges | — | **closed by upstream** | unchanged | +| Next sync + rebase (7) | — | — | **−1**, drops as empty | + +The fork PR **cannot** be held open as a tracker: once its commit is reachable from +`integration/3.0`, GitHub marks the PR merged on its own. Anything that needs to outlive stacking — +tracking, discussion, the upstream conversation — belongs on the mirror PR, which is why the +inventory below keys off `mirror/*` rather than off feature branches. + +**Inventory — what is in flight.** The fork PR is closed by now, so nothing in `integration/3.0` +points at the open upstream PR; the slug is the only key. Run this before any sync or release: + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It lists every `mirror/*` branch on `origin` with the open upstream PR that belongs to it, and for +each one checks the only two events that need action: + +- **Event 1 — `upstream/3.0` advanced into a conflict.** The mirror no longer merges onto the + upstream tip, so the PR shows conflicts and reviewers can no longer read it. An upstream advance + that still merges cleanly needs no action and is reported as nothing. +- **Event 2 — our stacked commit changed.** A sync rebase can alter a commit through conflict + resolution, leaving the open PR offering upstream something we no longer run. That is a + correctness and honesty problem, not untidiness. It is detected by patch-identity against the + live stack, so a renamed subject does not raise a false alarm and an amended diff does not hide. + +A `mirror/*` branch with no open PR is either finished (*After upstream responds*) or was never opened. An open PR +with no branch should be impossible and means someone hit the deletion hazard below. + +For either event, rebuild with the recipe in section 6 of the core instructions — including its guards, because a conflict +resolution is exactly where fork-only content gets dragged in — then: + +```bash +git push --force-with-lease origin mirror/ +``` + +Force-pushing updates the open PR in place. **Never close and reopen**: that discards the review +thread and reads as a withdrawal. After an Event 2 rebuild, say in the PR that the change was +updated and why; never leave upstream reviewing a version we have abandoned. + +**Hazard — deleting `mirror/` closes the upstream PR.** GitHub closes a pull request when its +head branch disappears. Done to an open contribution, that looks like a silent withdrawal and throws +away the review. *After upstream responds* ends in a branch deletion, which is safe *only* once the PR has reached +a terminal state. Never delete a mirror branch by hand; the guard is built in: + +```bash +forkctl delete-mirror # --dry-run to check without deleting +``` + +It refuses unless the upstream PR is `MERGED`, `CLOSED`, or was never opened. + +### After upstream responds + +A mirror PR has three possible endings, and each leaves different work behind. None of them is +"nothing" — a stacked commit whose mirror has been resolved must be retired deliberately, or the +stack slowly fills with commits nobody can explain. + +**First, find out what actually happened.** Patch-identity, not the PR's merge badge, is the test — +upstream may squash, rename or amend, and a merged PR does not guarantee our exact diff landed. + +```bash +git fetch upstream +git cherry upstream/3.0 mirror/ +# "- " -> an equivalent patch IS upstream under a different SHA (case A) +# "+ " -> it is NOT upstream as written (case B or C) +# (no output) -> our exact commit is now an ancestor of upstream/3.0 (case A) +``` + +**Empty output is a result, not a failure.** It means upstream merged in a way that preserved our +commit object, so it is reachable from `upstream/3.0` and falls outside the range `git cherry` +examines. Both `-` and silence mean "landed"; only `+` means "not landed as written". If you cannot +tell which you are looking at, disambiguate explicitly: + +```bash +git merge-base --is-ancestor mirror/ upstream/3.0 \ + && echo "landed verbatim (same commit)" \ + || git cherry upstream/3.0 mirror/ +``` + +Patch-identity ignores the commit message, so upstream renaming the subject still reports `-`. It +does **not** ignore content: any amendment to the diff, however small, reports `+`. That is the +correct signal — an amended change is case B, and the rebase will stop on it rather than drop it. + +The same question across the whole stack is `forkctl status`, which lists every commit we carry +that upstream already has. Those are pure rebase risk and should already be gone; if one persists +across two syncs, something is resurrecting it (*Syncing and releasing*). Run it before any release. + +**Case A — merged as written.** Do nothing to the stack by hand. The next sync fast-forwards `3.0`, +and the rebase of `integration/3.0` drops our copy automatically because it becomes empty. Confirm +it actually went, rather than assuming — `forkctl status` must report a feature gap one lower than +before, and no stale commits: + +```bash +forkctl status +git branch -D mirror/ +``` + +Then delete the remote branch with `forkctl delete-mirror ` — never with a bare +`git push origin --delete`. In case A the PR is already merged, so the guard passes; running it +anyway is what stops the same command being used by reflex while a PR is still open. + +**Case B — merged with modifications.** Upstream's edited version is now in the new base, so ours is +redundant *and* textually different, which means the rebase will stop on a conflict instead of +dropping it silently. Resolve in favour of upstream — ours has no privileged status once they have +taken ownership of the change: + +```bash +git rebase --skip # discards our now-superseded commit and continues +``` + +Then read upstream's merged version before moving on. If they changed behaviour we depend on, that +is a *new* feature branch against the new base, not an attempt to restore our original commit. Never +resurrect a superseded commit to "win back" a difference; raise it upstream as a follow-up instead. + +**Case C — rejected, or stalled with no realistic path.** This is the one that needs a decision +rather than a command, and it must not be left implicit. The commit is now a **permanent fork +delta**, and a permanent delta that still looks like a pending contribution will be rebased, +re-reviewed and re-offered by some future agent forever. + +Decide explicitly, and record the decision in the PR and in the commit subject: + +- **Keep it as a fork-only behaviour change** — reclassify it under core section 3.2, add its path or + marker to `FORK_ONLY_PATHS` / `FORK_HUNK_MARKERS` in `.github/fork/forkctl.py` (with a test), and + rename the commit to `chore(fork):` so the core section 4.5 and Gate 3 guards start treating it as + something that must *never* leave. This is the honest outcome for a change upstream has declined + on principle. +- **Drop it** — if it only existed to be upstreamed, remove it from the stack (*Changing fork-only content*) rather + than carrying a rebase hazard indefinitely. + +Either way, close the mirror PR with a reason, delete `mirror/`, and note the outcome so the +next agent does not reopen the same conversation with upstream. + +**In all three cases**, if the feature commit was rewritten on our side at any point after the +mirror branch was built, that mirror is stale — rebuild it and re-prove Gate 3 (core section 6) before +drawing any conclusion from `git cherry`, which compares the stale patch rather than the live one. + +## Syncing and releasing + +**Sync the mirror:** automated. `.github/workflows/sync-mirror.yml` fast-forwards `3.0` from +`upstream/3.0` daily and on demand (Actions → *Sync 3.0 mirror* → *Run workflow*). It refuses any +non-fast-forward, which is the alarm that someone committed to the mirror. `3.0` additionally blocks +force-pushes, deletions, and non-linear history. Do not push to it by hand. + +When the mirror moves ahead of the stack, the same workflow opens (and keeps updated) a +`mirror-sync` tracking issue; it closes the issue automatically once `integration/3.0` contains the +mirror head. The rebase itself stays manual — replaying features onto new upstream code can +conflict and must be re-tested. + +**Rebase the stack** (maintainers): + +```bash +git switch integration/3.0 +git fetch origin +test -z "$(git status --porcelain)" || { echo "ABORT: dirty tree"; exit 1; } + +START=$(git rev-parse origin/integration/3.0) # capture the lease target before rewriting +git reset --hard "$START" # discard local drift +git rebase origin/3.0 +``` + +On conflict, resolve in favour of upstream for code they changed, keep our commit's intent, then: + +```bash +git add +GIT_EDITOR=true git rebase --continue # --continue opens an editor; GIT_EDITOR=false makes it fail +``` + +If a conflict cannot be resolved confidently, `git rebase --abort` and raise it — never guess on the +branch BMW pins. + +Re-check the invariants **after** the rebase, exactly as in *Changing fork-only content* — a rebase happily preserves +a stack that was already malformed: + +```bash +BASE=$(forkctl check-stack --stack HEAD --print-base) || exit 1 + +make test + +# Move both refs or neither. A rejected lease must not leave the view branch pointing at a base +# that was never published, and a two-step push leaves a window in which the detector reports +# drift that no longer exists. +git push --atomic --force-with-lease=refs/heads/integration/3.0:"$START" \ + origin integration/3.0 "+$BASE:refs/heads/fork-only/3.0" +``` + +Use the **explicit** `--force-with-lease=:` form. Bare `--force-with-lease` leases against +your remote-tracking ref, which any background `git fetch` silently refreshes — at which point it +will happily overwrite a colleague's push that you never incorporated. If the push is rejected, +start over from the new remote tip; do not re-run with a wider hammer. + +`make test` is not the full gate (section 4.3) — it skips Docker smoke, perf, the install matrix and +pre-commit. After pushing, the standing draft PR re-runs the real suite against the rewritten stack; +treat a release as blocked until that is green. + +Drop commits that upstream has already merged; never resurrect them — *After upstream responds* gives the +mechanics, including the case where upstream merged a *modified* version. After any rebase that +rewrote a feature commit, its `mirror/` branch is stale: if an upstream PR is already open on +it, *While the mirror PR is open* applies and the PR must be refreshed; if not, Gate 3 must be re-proven before that +feature is offered upstream. Either way, a commit whose content changed materially needs re-review. + +**Divergence report.** The stack is upstream *debt*: every feature commit we carry is one upstream +has not taken yet. Run this after every sync, and before any release: + +```bash +git fetch origin && git fetch upstream +forkctl status +``` + +It prints the total gap, the fork-only floor and the feature gap, lists the stack, warns about +commits upstream already has, and reports every in-flight mirror. It warns rather than fails: a +non-zero gap is the normal state, not an error. + +**The total gap can never reach zero, and must not be driven to zero.** `integration/3.0` +permanently carries at least one `chore(fork):` commit, because the files in section 3.1 do not +exist upstream by design. An agent that reads a non-zero gap as a defect and tries to close it by +upstreaming that commit would leak the entire fork-only process into `Emerge-Lab/PufferDrive` — +precisely what Gate 3 exists to prevent. + +**The number that should trend to zero is the feature gap.** Zero means upstream has absorbed +everything we carry and the fork is pure scaffolding again; that is the steady state we are aiming +for, not an exceptional one. A feature gap that only grows means we are accumulating a private fork +rather than contributing — treat a persistently rising count as a process failure to discuss, not a +number to ignore. Every feature commit on the stack should be traceable to an open or pending mirror +PR (*While the mirror PR is open*); one that is not is either unfinished work or a change nobody intends to upstream, +and the second kind belongs in a `chore(fork):` commit where the guards can see it. + +**Hazard — `update-smoke-golden.yml`.** This inherited upstream workflow ends in +`git push origin HEAD:`, and it triggers on pushes to `3.0` as well as `workflow_dispatch`. +Never dispatch it against `3.0`: it would commit to the mirror and break the clean base for every +upstream PR. If it ever does, `sync-mirror.yml` is the detector — it refuses the non-fast-forward +and the mirror-sync issue appears. Recover by resetting `3.0` to `upstream/3.0` (the branch +protection on `3.0` blocks force-pushes, so this needs an admin and should be done deliberately). + +**Release for the BMW submodule:** after a green rebase, run the divergence report above and confirm +the feature gap matches the mirror PRs actually in flight — a release is the moment the gap becomes +someone else's dependency. Then tag an immutable point and pin that tag. + +```bash +git tag -a ascs/3.0-$(date +%Y%m%d) -m "Integration snapshot" +git push origin ascs/3.0-$(date +%Y%m%d) +``` + +Consumers pin a tag or a commit SHA — never the moving `integration/3.0` tip. diff --git a/.github/fork/forkctl.py b/.github/fork/forkctl.py new file mode 100644 index 0000000000..5e88a0c825 --- /dev/null +++ b/.github/fork/forkctl.py @@ -0,0 +1,1550 @@ +#!/usr/bin/env python3 +"""Fork-process checks for ASCS-eV/PufferDrive. + +Fork-only tooling. Never cherry-pick this file upstream. + +This is the single implementation of the guards described in +`.github/copilot-instructions.md`. Humans, coding agents and CI all run this +same code instead of retyping shell snippets, so a guard cannot be mistyped at +one call site and silently pass at another. + +It is deliberately stdlib-only and shell-free. The fork process depends on these +checks failing *loudly*, and the classic ways a shell guard dies quietly have no +analogue here: `grep -c` exiting 1 on no match, `git log | grep -q` returning +141 under `pipefail`, a pipeline's exit status being swallowed. + +Every subcommand exits 0 on success and 1 on failure. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys + + +MIRROR_REF = "origin/3.0" +STACK_REF = "origin/integration/3.0" +UPSTREAM_REF = "upstream/3.0" +VIEW_BRANCH = "fork-only/3.0" + +FORK_REPO = "ASCS-eV/PufferDrive" +UPSTREAM_REPO = "Emerge-Lab/PufferDrive" +UPSTREAM_URL = "https://github.com/Emerge-Lab/PufferDrive.git" +FORK_OWNER = "ASCS-eV" + +FORK_COMMIT_PREFIX = "chore(fork): " + +# Files that exist only in this fork; none may appear in a feature diff. This +# tuple is the single source of truth. The docs point here instead of repeating +# the list, so adding a fork-only file cannot leave a stale copy behind. +FORK_ONLY_PATHS = ( + ".github/copilot-instructions.md", + "CONTRIBUTING.md", + ".github/pull_request_template.md", + ".github/workflows/sync-mirror.yml", + ".github/workflows/fork-only-view.yml", +) + +# Anything under these prefixes is fork-only by construction, so a new fork-only +# tool dropped into `.github/fork/` is covered without editing this file. +FORK_ONLY_PREFIXES = (".github/fork/",) + +# Registration is only half the invariant. These files must also *exist*: a base +# rewrite that quietly drops the detector or the always-loaded instructions would +# otherwise pass every check, because nothing was touched that is unregistered. +REQUIRED_FORK_FILES = FORK_ONLY_PATHS + ( + ".github/fork/forkctl.py", + ".github/fork/test_forkctl.py", + ".github/fork/PROCESS.md", +) + +# Shared with upstream, but carrying a fork-only hunk. The file may legitimately +# appear in a feature diff; only those hunks must stay untouched. +SHARED_FILES = ( + "AGENTS.md", + "CLAUDE.md", + ".github/workflows/ci.yml", + ".github/workflows/install.yml", +) + +FORK_HUNK_MARKERS = ("integration/3.0", "fork:begin", "fork:end") + +# `integration/3.0` is only evidence of fork-only content inside workflow +# configuration. Elsewhere -- including prose under `.github/` -- it is an +# ordinary string an upstream-bound commit may legitimately write. +FORK_PATH_SCOPED_MARKERS = {"integration/3.0": (".github/workflows/",)} + +# Workflows shared with upstream must keep the fork branch in the `pull_request` +# trigger and nowhere else. Moving it to `push:` leaves the marker line untouched, +# so only a structural read of the trigger tree catches it. +FORK_WORKFLOW_FILES = (".github/workflows/ci.yml", ".github/workflows/install.yml") +FORK_WORKFLOW_BRANCH = "integration/3.0" +FORK_WORKFLOW_TRIGGER = "pull_request" + +CONVENTIONAL_TYPES = ( + "feat", + "fix", + "perf", + "refactor", + "test", + "docs", + "build", + "ci", + "chore", +) +CONVENTIONAL_RE = re.compile(rf"^({'|'.join(CONVENTIONAL_TYPES)})(\([a-z0-9][a-z0-9-]*\))?!?: \S.*$") +MAX_SUBJECT_LENGTH = 72 + +# Files whose fork-only part is a delimited block rather than a single line. +# Comparing the block's whole content catches an edit *between* the markers, +# which a diff-line scan misses: those changed lines carry no marker themselves. +FORK_BLOCK_FILES = ("AGENTS.md", "CLAUDE.md") +FORK_BLOCK_BEGIN = "fork:begin" +FORK_BLOCK_END = "fork:end" + +TERMINAL_PR_STATES = ("MERGED", "CLOSED", "NONE") + + +class ForkctlError(RuntimeError): + pass + + +def git(*args: str, check: bool = True) -> str: + proc = subprocess.run(["git", "--no-pager", *args], capture_output=True, text=True, check=False) + if check and proc.returncode != 0: + raise ForkctlError(f"git {' '.join(args)} failed: {proc.stderr.strip()}") + # Strip line terminators only. Stripping arbitrary whitespace would silently + # normalise a subject like " chore(fork): x" into a conforming one. + return proc.stdout.strip("\r\n") + + +def git_rc(*args: str) -> int: + proc = subprocess.run(["git", "--no-pager", *args], capture_output=True, text=True, check=False) + return proc.returncode + + +def show_file(ref: str, path: str) -> str | None: + """Contents of `path` at `ref`, or None if it genuinely does not exist. + + `git show` with check=False conflates "absent", "empty" and "git failed", + and every caller then reads the empty string as "nothing to check". Probing + for existence first keeps a git error loud instead of fail-open. + + Returned verbatim, including trailing newlines: `git()` strips line + terminators, which would hide a trailing-newline rewrite from the byte + comparison in `smuggled_shared_changes()`. + """ + if git_rc("cat-file", "-e", f"{ref}:{path}") != 0: + return None + proc = subprocess.run(["git", "--no-pager", "show", f"{ref}:{path}"], capture_output=True, check=False) + if proc.returncode != 0: + raise ForkctlError(f"git show {ref}:{path} failed: {proc.stderr.decode('utf-8', 'replace').strip()}") + return proc.stdout.decode("utf-8") + + +def head_is_detached() -> bool: + return git_rc("symbolic-ref", "--quiet", "HEAD") != 0 + + +def git_lines(*args: str) -> list[str]: + out = git(*args) + return out.splitlines() if out else [] + + +def gh_available() -> bool: + return shutil.which("gh") is not None + + +def gh_json(*args: str): + proc = subprocess.run(["gh", *args], capture_output=True, text=True, check=False) + if proc.returncode != 0: + raise ForkctlError(f"gh {' '.join(args)} failed: {proc.stderr.strip()}") + return json.loads(proc.stdout or "[]") + + +class Report: + """Collects check outcomes and renders them for a terminal or for CI.""" + + def __init__(self, title: str, fmt: str = "text", stream=None) -> None: + self.title = title + self.fmt = fmt + self.stream = stream if stream is not None else sys.stdout + self.failed = False + if fmt == "text": + print(f"== {title}", file=self.stream) + + def ok(self, message: str) -> None: + if self.fmt == "text": + print(f" PASS {message}", file=self.stream) + + def info(self, message: str) -> None: + if self.fmt == "text": + print(f" {message}", file=self.stream) + + def warn(self, message: str) -> None: + if self.fmt == "github": + print(f"::warning::{message}", file=self.stream) + else: + print(f" WARN {message}", file=self.stream) + + def fail(self, message: str, fix: str | None = None) -> None: + self.failed = True + if self.fmt == "github": + print(f"::error::{message}" + (f" Fix: {fix}" if fix else ""), file=self.stream) + else: + print(f" FAIL {message}", file=self.stream) + if fix: + print(f" fix: {fix}", file=self.stream) + + def check(self, passed: bool, ok: str, bad: str, fix: str | None = None) -> bool: + if passed: + self.ok(ok) + return True + self.fail(bad, fix) + return False + + def finish(self) -> int: + if self.fmt == "text": + print(f"== {self.title}: {'FAILED' if self.failed else 'OK'}", file=self.stream) + return 1 if self.failed else 0 + + +def ref_exists(ref: str) -> bool: + return git_rc("rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}") == 0 + + +def require_refs(report: Report, *refs: str) -> bool: + missing = [ref for ref in refs if not ref_exists(ref)] + if not missing: + return True + report.fail( + f"ref(s) not found locally: {', '.join(missing)}", + "git fetch origin && git fetch upstream", + ) + return False + + +def is_fork_only_path(path: str) -> bool: + return path in FORK_ONLY_PATHS or path.startswith(FORK_ONLY_PREFIXES) + + +def diff_content_lines(diff: str, added_only: bool = False) -> list[str]: + """Content lines of a diff, excluding the `+++`/`---` file headers.""" + wanted = ("+",) if added_only else ("+", "-") + return [line for line in diff.splitlines() if line.startswith(wanted) and not line.startswith(("+++", "---"))] + + +def fork_only_files_in(base: str, head: str = "HEAD") -> list[str]: + # --no-renames: with rename detection on, `git diff --name-only` reports only + # the destination, so renaming a fork-only file out of the way would hide it. + files = git_lines("diff", "--no-renames", "--name-only", f"{base}...{head}") + return [path for path in files if is_fork_only_path(path)] + + +def diff_files(base: str, head: str = "HEAD") -> list[str]: + return git_lines("diff", "--no-renames", "--name-only", f"{base}...{head}") + + +def marker_applies_to(marker: str, path: str) -> bool: + scopes = FORK_PATH_SCOPED_MARKERS.get(marker) + return True if scopes is None else path.startswith(scopes) + + +def fork_markers_anywhere(base: str, head: str = "HEAD") -> list[str]: + """Added lines referencing fork-only state, across the whole diff. + + Used against pure upstream, where no added line should mention the fork. + Path-scoped markers are only evidence inside the paths they are scoped to, + so ordinary prose containing the branch name does not block a contribution. + """ + offenders = [] + for path in diff_files(base, head): + markers = [marker for marker in FORK_HUNK_MARKERS if marker_applies_to(marker, path)] + if not markers: + continue + diff = git("diff", f"{base}...{head}", "--", path) + offenders += [ + f"{path}: {line.strip()}" + for line in diff_content_lines(diff, added_only=True) + if any(marker in line for marker in markers) + ] + return offenders + + +def _strip_inline_comment(value: str) -> str: + """Drop a trailing `# ...` comment that is not inside a quoted scalar.""" + out: list[str] = [] + quote = None + for index, char in enumerate(value): + if quote: + out.append(char) + if char == quote: + quote = None + continue + if char in "'\"": + quote = char + out.append(char) + continue + if char == "#" and (index == 0 or value[index - 1] in " \t"): + break + out.append(char) + return "".join(out).rstrip() + + +def _reject_unsupported_yaml(line: str) -> None: + """Fail closed on YAML this reader is not equipped to understand. + + A hand-rolled indentation reader that silently mis-parses is worse than no + reader at all: it reports PASS on a file GitHub reads differently. Anything + outside the small subset these workflows actually use is an error. + """ + if "\t" in line[: len(line) - len(line.lstrip())]: + raise ForkctlError(f"tab in workflow indentation, which YAML forbids: {line!r}") + _, _, value = line.partition(":") + value = _strip_inline_comment(value).strip() + if value[:1] in ("&", "*"): + raise ForkctlError(f"YAML anchor/alias in the `on:` block is not supported: {line!r}") + if value[:1] in ("|", ">"): + raise ForkctlError(f"YAML block scalar in the `on:` block is not supported: {line!r}") + if value.startswith("{"): + raise ForkctlError(f"YAML flow mapping in the `on:` block is not supported: {line!r}") + if line.lstrip().startswith("<<"): + raise ForkctlError(f"YAML merge key in the `on:` block is not supported: {line!r}") + + +def workflow_branch_filters(text: str) -> dict[str, list[str]]: + """Map trigger -> branches under `on..branches` of a workflow. + + A deliberately small indentation reader rather than a YAML dependency: this + script must stay stdlib-only, and it only needs one well-known subtree. It + raises rather than guessing whenever the input leaves that subset, and it + matches the `branches` key exactly -- `branches-ignore` is a different key + with the opposite meaning, so treating it as a prefix match let a commit + swap one for the other and keep the marker line intact. + """ + raw = [line.rstrip() for line in text.splitlines() if line.strip() and not line.lstrip().startswith("#")] + lines = [_strip_inline_comment(line) for line in raw] + lines = [line for line in lines if line.strip()] + filters: dict[str, list[str]] = {} + + on_keys = [i for i, line in enumerate(lines) if re.match(r"^(on|'on'|\"on\"):", line)] + if len(on_keys) > 1: + raise ForkctlError(f"workflow declares `on:` {len(on_keys)} times; which triggers apply is ambiguous") + if not on_keys: + return filters + on_index = on_keys[0] + _reject_unsupported_yaml(lines[on_index]) + if lines[on_index].partition(":")[2].strip(): + raise ForkctlError(f"inline `on:` value is not supported: {lines[on_index]!r}") + + body = [] + for line in lines[on_index + 1 :]: + if indent_of(line) == 0: + break + _reject_unsupported_yaml(line) + body.append(line) + if not body: + return filters + + trigger_indent = min(indent_of(line) for line in body) + for start, trigger in _trigger_blocks(body, trigger_indent): + filters.setdefault(trigger, []) + block = _child_lines(body, start, trigger_indent) + if not block: + continue + child_indent = min(indent_of(line) for line in block) + # `branches` counts only as a direct child of the trigger. Accepting it + # at any deeper indent let it be read out of an unrelated nested key. + hits = [ + index + for index, line in enumerate(block) + if indent_of(line) == child_indent and line.strip().partition(":")[0].strip().strip("'\"") == "branches" + ] + if len(hits) > 1: + raise ForkctlError(f"duplicate `branches:` key under `{trigger}:`") + if hits: + filters[trigger] = _branch_values(block, hits[0]) + return filters + + +def indent_of(line: str) -> int: + return len(line) - len(line.lstrip()) + + +def _trigger_blocks(body: list[str], trigger_indent: int) -> list[tuple[int, str]]: + """`(index, name)` of every trigger key in the `on:` block.""" + seen: set[str] = set() + blocks = [] + for index, line in enumerate(body): + if indent_of(line) != trigger_indent: + continue + stripped = line.strip() + if stripped.startswith("- "): + blocks.append((index, stripped[2:].strip().strip("'\""))) + continue + if not stripped.endswith(":"): + raise ForkctlError(f"unrecognised entry in the `on:` block: {line!r}") + name = stripped.rstrip(":").strip().strip("'\"") + if name in seen: + raise ForkctlError(f"duplicate `{name}:` trigger in the `on:` block") + seen.add(name) + blocks.append((index, name)) + return blocks + + +def _child_lines(body: list[str], start: int, trigger_indent: int) -> list[str]: + block = [] + for line in body[start + 1 :]: + if indent_of(line) <= trigger_indent: + break + block.append(line) + return block + + +def _split_flow_list(inline: str) -> list[str]: + """Split `[a, 'b,c']` on commas that are not inside a quoted scalar.""" + items, current, quote = [], [], None + for char in inline: + if quote: + current.append(char) + if char == quote: + quote = None + continue + if char in "'\"": + quote = char + current.append(char) + continue + if char == ",": + items.append("".join(current)) + current = [] + continue + current.append(char) + if quote: + raise ForkctlError(f"unterminated quote in a `branches:` list: {inline!r}") + items.append("".join(current)) + return [item for item in (item.strip() for item in items) if item] + + +def _unquote_branch(value: str) -> str: + """One `branches:` entry, with its quotes removed and its shape checked.""" + if len(value) >= 2 and value[0] == value[-1] and value[0] in "'\"": + value = value[1:-1] + elif value[:1] in ("'", '"') or value[-1:] in ("'", '"'): + raise ForkctlError(f"unbalanced quotes in a `branches:` entry: {value!r}") + if value.startswith("!"): + # GitHub evaluates patterns in order, so a later `!` silently excludes a + # branch this reader would still report as listed. + raise ForkctlError(f"negative branch pattern is not supported: {value!r}") + return value + + +def _branch_values(body: list[str], index: int) -> list[str]: + """Values of a `branches:` key, inline (`[a, b]`) or as a block list.""" + line = body[index] + indent = indent_of(line) + _, _, inline = line.partition(":") + inline = inline.strip() + if inline: + if not (inline.startswith("[") and inline.endswith("]")): + raise ForkctlError(f"unrecognised inline `branches:` value: {line!r}") + return [_unquote_branch(item) for item in _split_flow_list(inline[1:-1])] + + values = [] + for following in body[index + 1 :]: + if indent_of(following) <= indent: + break + if not following.strip().startswith("- "): + raise ForkctlError(f"unrecognised entry under `branches:`: {following!r}") + values.append(_unquote_branch(following.strip()[2:].strip())) + return values + + +def fork_workflow_triggers(ref: str, path: str) -> list[str]: + """Triggers whose branch filter lists the fork branch, at `ref`.""" + text = show_file(ref, path) + if text is None: + return [] + return sorted( + trigger for trigger, branches in workflow_branch_filters(text).items() if FORK_WORKFLOW_BRANCH in branches + ) + + +def fork_workflow_problems(base: str, head: str = "HEAD") -> list[str]: + """Section 3.2: the fork branch filter must stay put, in `pull_request` only.""" + merge_base = git("merge-base", base, head) + problems = [] + for path in FORK_WORKFLOW_FILES: + before = fork_workflow_triggers(merge_base, path) + after = fork_workflow_triggers(head, path) + if before == after: + continue + if FORK_WORKFLOW_TRIGGER in before and FORK_WORKFLOW_TRIGGER not in after: + problems.append(f"{path}: the fork branch filter was removed from `{FORK_WORKFLOW_TRIGGER}`") + for trigger in sorted(set(after) - set(before)): + problems.append(f"{path}: the fork branch filter was added to `{trigger}`") + return problems + + +def fork_blocks(ref: str, path: str) -> list[str]: + """Every `fork:begin`...`fork:end` block of `path` at `ref`, in order.""" + content = show_file(ref, path) + if content is None: + return [] + blocks, start = [], None + for index, line in enumerate(content.splitlines()): + if FORK_BLOCK_BEGIN in line: + start = index if start is None else start + elif FORK_BLOCK_END in line and start is not None: + blocks.append("\n".join(content.splitlines()[start : index + 1])) + start = None + return blocks + + +def malformed_fork_blocks(ref: str) -> list[str]: + """Shared files whose marker structure is not exactly one trailing block.""" + problems = [] + for path in FORK_BLOCK_FILES: + content = show_file(ref, path) + if content is None: + continue + lines = content.splitlines() + begins = [i for i, line in enumerate(lines) if FORK_BLOCK_BEGIN in line] + ends = [i for i, line in enumerate(lines) if FORK_BLOCK_END in line] + if len(begins) != 1 or len(ends) != 1 or begins[0] > ends[0]: + problems.append( + f"{path}: {len(begins)} fork:begin / {len(ends)} fork:end marker(s), expected one ordered pair" + ) + continue + # The block is appended to upstream content, so nothing may follow it. + # Position, not content, is what a relocation changes, and git is free + # to render a move as relocating the *other* side of the file. + if any(line.strip() for line in lines[ends[0] + 1 :]): + problems.append(f"{path}: content follows the fork:end marker; the fork block must end the file") + return problems + + +def changed_fork_blocks(base: str, head: str = "HEAD") -> list[str]: + merge_base = git("merge-base", base, head) + return [path for path in FORK_BLOCK_FILES if fork_blocks(merge_base, path) != fork_blocks(head, path)] + + +def fork_marker_lines(ref: str, path: str) -> list[str]: + """Every marker line of `path` at `ref`, whether or not it forms a block. + + `fork_blocks()` reports only completed pairs, so a stray marker is invisible + to it -- which let a feature commit tidy up a malformed base and have the + change counted as upstream-clean. + """ + content = show_file(ref, path) + if content is None: + return [] + return [line for line in content.splitlines() if FORK_BLOCK_BEGIN in line or FORK_BLOCK_END in line] + + +def changed_fork_markers(base: str, head: str) -> list[str]: + return [path for path in FORK_BLOCK_FILES if fork_marker_lines(base, path) != fork_marker_lines(head, path)] + + +def fork_state_problems(ref: str) -> list[str]: + """The fork-only state every stack ref must have, stated absolutely. + + A before/after comparison only catches a change made by the commit under + review. It says nothing about a stack that was already wrong, so the end + state is asserted here rather than inferred from a diff. + """ + problems = [] + for path in REQUIRED_FORK_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; required fork-only infrastructure was deleted") + for path in FORK_WORKFLOW_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; a shared workflow was deleted") + continue + triggers = fork_workflow_triggers(ref, path) + if triggers != [FORK_WORKFLOW_TRIGGER]: + found = ", ".join(triggers) if triggers else "no trigger" + problems.append(f"{path}: fork branch filter is on `{found}`, expected `{FORK_WORKFLOW_TRIGGER}` only") + for path in FORK_BLOCK_FILES: + if show_file(ref, path) is None: + problems.append(f"{path} is missing; a shared file carrying a fork block was deleted") + continue + blocks = fork_blocks(ref, path) + if len(blocks) != 1: + problems.append(f"{path}: {len(blocks)} fork-only block(s), expected exactly 1") + return problems + + +def strip_fork_block(text: str) -> str: + """`text` with every fork:begin..fork:end block removed. + + Line endings are preserved: normalising them would let a CRLF/LF rewrite of + a shared file compare equal to its upstream parent and ride along in the one + commit that is exempt from Gate 3. + """ + out, inside = [], False + for line in text.splitlines(keepends=True): + if FORK_BLOCK_BEGIN in line: + inside = True + continue + if FORK_BLOCK_END in line: + inside = False + continue + if not inside: + out.append(line) + return "".join(out) + + +def strip_fork_branch(text: str) -> str: + """`text` with the fork branch removed from every `branches:` list.""" + pattern = re.compile(rf",\s*['\"]?{re.escape(FORK_WORKFLOW_BRANCH)}['\"]?") + out = [] + for line in text.splitlines(keepends=True): + if "branches" in line and FORK_WORKFLOW_BRANCH in line: + line = pattern.sub("", line) + out.append(line) + return "".join(out) + + +def smuggled_shared_changes(parent: str, base: str) -> list[str]: + """Shared-file edits in the fork base that are not the fork-only hunk. + + The fork base is allowed to touch shared files -- adding the fork block and + the branch filter is its job -- so a file-name check cannot police it. Each + shared file is therefore compared with its upstream parent after the + fork-only hunk is removed from both; anything left over is a change that + was folded into the base commit where no guard would ever look at it. + """ + problems = [] + for path in SHARED_FILES: + before, after = show_file(parent, path), show_file(base, path) + if before is None and after is None: + continue + if after is None: + problems.append(f"{path} was deleted by the fork base") + continue + if before is None: + problems.append(f"{path} is introduced by the fork base but is registered as shared with upstream") + continue + strip = strip_fork_block if path in FORK_BLOCK_FILES else strip_fork_branch + # The fork block is appended after a blank separator line, so removing it + # leaves exactly that one extra newline. Nothing else -- not a trailing + # newline, not a CRLF rewrite -- may differ. + upstream = strip(before) + separator = "\r\n" if "\r\n" in upstream else "\n" + allowed = (upstream, upstream + separator) if path in FORK_BLOCK_FILES else (upstream,) + if strip(after) not in allowed: + problems.append(f"{path} carries a non-fork-only change in the fork base") + return problems + + +def subject_problems(subject: str) -> list[str]: + problems = [] + if subject != subject.strip(): + problems.append("has leading or trailing whitespace") + if not CONVENTIONAL_RE.match(subject): + problems.append(f"is not Conventional Commits ({'|'.join(CONVENTIONAL_TYPES)})(scope): summary") + if len(subject) > MAX_SUBJECT_LENGTH: + problems.append(f"is {len(subject)} chars, over the {MAX_SUBJECT_LENGTH}-char limit") + return problems + + +def working_tree_clean() -> bool: + return git("status", "--porcelain") == "" + + +def patch_id(rev: str) -> str | None: + """Content identity of a commit, ignoring its message and its parents.""" + diff = subprocess.run( + ["git", "--no-pager", "diff", f"{rev}^", rev], + check=False, + capture_output=True, + text=True, + ) + if diff.returncode != 0 or not diff.stdout: + return None + proc = subprocess.run( + ["git", "patch-id", "--stable"], + check=False, + input=diff.stdout, + capture_output=True, + text=True, + ) + return proc.stdout.split()[0] if proc.stdout.strip() else None + + +def mirror_ref(slug: str) -> str | None: + for candidate in (f"origin/mirror/{slug}", f"mirror/{slug}"): + if ref_exists(candidate): + return candidate + return None + + +def merges_cleanly(ref: str, onto: str) -> bool: + """Whether ref still applies to onto, without touching the working tree.""" + proc = subprocess.run( + ["git", "--no-pager", "merge-tree", "--write-tree", onto, ref], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode not in (0, 1): + # Git < 2.38 has no --write-tree; it must not read as "no conflict". + raise ForkctlError(f"git merge-tree --write-tree is unavailable: {proc.stderr.strip()}") + return proc.returncode == 0 + + +def mirror_drift(slug: str, upstream: str, stack_patch_ids: set[str]) -> list[str]: + """Sections 6.2 events 1 and 2: the only two things that need action.""" + ref = mirror_ref(slug) + if ref is None: + return ["not fetched locally; run git fetch origin"] + + notes = [] + carried = git_lines("rev-list", f"{upstream}..{ref}") + if len(carried) != 1: + notes.append( + f"carries {len(carried)} commits over upstream, expected exactly 1 -- " + "the open PR offers more than the stacked commit; rebuild it" + ) + + try: + if git("rev-parse", f"{ref}^") != git("rev-parse", upstream) and not merges_cleanly(ref, upstream): + notes.append("event 1: no longer merges onto upstream -- rebuild onto the new tip") + except ForkctlError as exc: + notes.append(f"could not test whether it still merges onto upstream: {exc}") + + identity = patch_id(ref) + if identity is None: + notes.append("could not compute a patch-id for the mirror tip; drift is unverified") + elif identity not in stack_patch_ids: + notes.append( + "event 2: content differs from every stack commit -- the open PR offers " + "something we no longer run; rebuild and force-push" + ) + return notes + + +def current_branch() -> str: + branch = git("rev-parse", "--abbrev-ref", "HEAD") + return git("rev-parse", "HEAD") if branch == "HEAD" else branch + + +def cmd_fork_only(args: argparse.Namespace) -> int: + for path in FORK_ONLY_PATHS: + print(path) + for prefix in FORK_ONLY_PREFIXES: + print(f"{prefix}*") + if args.verbose: + print("\nshared files carrying a fork-only hunk:") + for path in SHARED_FILES: + print(f" {path}") + print("\nfork-only hunk markers:") + for marker in FORK_HUNK_MARKERS: + print(f" {marker}") + return 0 + + +def cmd_check_commit(args: argparse.Namespace) -> int: + """Sections 4.4 and 4.5: branch shape and upstream-cleanliness pre-flight.""" + report = Report("check-commit", args.format) + if not require_refs(report, args.base): + return report.finish() + + commits = git_lines("rev-list", f"{args.base}..HEAD") + one_commit = report.check( + len(commits) == 1, + "exactly one commit on the branch", + f"{len(commits)} commits on the branch, expected exactly 1", + f"git rebase {args.base} && git reset --soft {args.base} && git commit -F msg", + ) + + # One commit *ahead* is not the same as rebased onto the current base: a + # branch cut weeks ago is still one commit ahead of where it started. + if one_commit: + report.check( + git("rev-parse", "HEAD^") == git("rev-parse", args.base), + f"rebased directly onto {args.base}", + f"the commit is not parented on the current {args.base} tip", + f"git rebase {args.base}", + ) + + subject = git("log", "-1", "--format=%s") if commits else "" + problems = subject_problems(subject) + report.check( + not problems, + f"subject is well formed: {subject!r}", + f"subject {subject!r} " + "; ".join(problems), + "Conventional Commits, imperative mood, <= 72 chars (section 4.4)", + ) + + is_chore = subject.startswith(FORK_COMMIT_PREFIX) + if args.chore and not is_chore: + report.fail(f"--chore given but the subject is not {FORK_COMMIT_PREFIX!r}: {subject!r}") + if is_chore and not args.chore: + report.warn( + "subject is a chore(fork) commit but --chore was not passed; the " + "fork-only checks below are expected to trip for a fork-only change" + ) + + # Structural validity of the markers is checked in both modes. A chore commit + # may legitimately rewrite a block, but if it leaves two of them behind, every + # later feature commit can edit the second one unnoticed. + malformed = malformed_fork_blocks("HEAD") + report.check( + not malformed, + "fork-only markers are well formed", + "fork-only marker structure is broken: " + "; ".join(malformed), + "each shared file carries exactly one fork:begin/fork:end pair", + ) + + # Asserted absolutely, in both modes: a before/after comparison cannot see a + # fork-only hunk that was already missing before this commit touched it. + state = fork_state_problems("HEAD") + report.check( + not state, + "fork-only state is intact", + "fork-only state is broken: " + "; ".join(state), + "the branch must keep every required fork-only file, the fork block and the branch filter", + ) + + if args.chore: + report.info("fork-only file and hunk checks: exempt (chore(fork) branch)") + return report.finish() + + offenders = fork_only_files_in(args.base) + report.check( + not offenders, + "no registered fork-only file in the diff", + f"fork-only file(s) in the feature diff: {', '.join(offenders)}", + "move them into a separate chore(fork): commit", + ) + + workflow_problems = fork_workflow_problems(args.base) + report.check( + not workflow_problems, + "fork-only branch filters are untouched in shared workflows", + "; ".join(workflow_problems), + "leave the fork branch in the pull_request filter; the rest of the file is yours", + ) + + merge_base = git("merge-base", args.base, "HEAD") + blocks = sorted(set(changed_fork_blocks(args.base)) | set(changed_fork_markers(merge_base, "HEAD"))) + report.check( + not blocks, + "fork-only blocks in shared files are unchanged", + f"fork-only block content changed in: {', '.join(blocks)}", + "revert the fork:begin/fork:end block; it belongs in the chore(fork) commit", + ) + return report.finish() + + +def verify_stack(report: Report, mirror_ref: str, stack_ref: str) -> str | None: + """Sections 3.3 and 7: stack invariants. Returns the fork base, or None.""" + if not require_refs(report, mirror_ref, stack_ref): + return None + + mirror = git("rev-parse", mirror_ref) + stack = git("rev-parse", stack_ref) + + contains = git_rc("merge-base", "--is-ancestor", mirror, stack) == 0 + if not report.check( + contains, + f"{stack_ref} contains the {mirror_ref} tip", + f"{stack_ref} does not contain the {mirror_ref} tip", + "rebase the stack (copilot-instructions.md section 7)", + ): + return None + + rng = f"{mirror}..{stack}" + + merges = git_lines("rev-list", "--min-parents=2", "--max-count=1", rng) + report.check( + not merges, + "stack is linear (no merge commits)", + f"merge commit on the stack: {merges[0] if merges else ''}", + "rebuild the stack with rebase; never merge into integration/3.0", + ) + + subjects = git_lines("log", "--format=%s", rng) + + # Read subjects, not whole messages: `git log --grep` matches the body too, + # and a leading space is enough to slip past a prefix test. + ragged = [s for s in subjects if s != s.strip()] + report.check( + not ragged, + "no commit subject has stray leading or trailing whitespace", + f"commit subject with stray whitespace: {ragged[0]!r}" if ragged else "", + "reword it; whitespace hides a subject from every prefix check", + ) + + fixups = [s for s in subjects if s.strip().startswith(("fixup!", "squash!"))] + report.check( + not fixups, + "no stray fixup! commit", + f"stray fixup!/squash! commit on the stack: {fixups[0]!r}" if fixups else "", + "finish the autosquash rebase before pushing", + ) + + chore_count = sum(1 for s in subjects if s.startswith(FORK_COMMIT_PREFIX)) + report.check( + chore_count == 1, + "exactly one chore(fork): commit", + f"{chore_count} chore(fork): commits on the stack, expected 1", + "fold fork-only changes into the base commit (section 3.3)", + ) + + commits = git_lines("rev-list", rng) + if not commits: + report.fail(f"{stack_ref} has no commits over {mirror_ref}") + return None + + base = commits[-1] + base_subject = git("log", "-1", "--format=%s", base) + report.check( + base_subject.startswith(FORK_COMMIT_PREFIX), + f"oldest stack commit is the fork base ({base[:9]})", + f"oldest stack commit {base[:9]} is not the fork base: {base_subject!r}", + ) + + parents = git("rev-list", "--parents", "-n", "1", base).split() + if report.check( + len(parents) == 2, + "fork base has exactly one parent", + f"fork base {base[:9]} does not have exactly one parent", + ): + report.check( + parents[1] == mirror, + "fork base is parented on the mirror tip", + f"fork base {base[:9]} is parented on {parents[1][:9]}, not the mirror tip {mirror[:9]}", + ) + + # Keeps the hardcoded list honest: anything the fork base introduces is by + # definition fork-only, so an unregistered file here means the guards that + # protect feature commits have a blind spot. + unregistered = [ + path + for path in diff_files(mirror, base) + if not is_fork_only_path(path) and path not in SHARED_FILES and path not in FORK_WORKFLOW_FILES + ] + report.check( + not unregistered, + "every file in the fork base is registered as fork-only", + f"fork base touches unregistered file(s): {', '.join(unregistered)}", + "add them to FORK_ONLY_PATHS/FORK_ONLY_PREFIXES in forkctl.py, or move them under .github/fork/", + ) + + malformed = malformed_fork_blocks(base) + malformed_fork_blocks(stack) + report.check( + not malformed, + "fork-only markers are well formed on the stack", + "fork-only marker structure is broken: " + "; ".join(malformed), + "each shared file carries exactly one fork:begin/fork:end pair", + ) + + state = fork_state_problems(stack) + report.check( + not state, + "fork-only state is intact at the stack tip", + "fork-only state is broken: " + "; ".join(state), + "the stack tip must keep every required fork-only file, the fork block and the branch filter", + ) + + smuggled = smuggled_shared_changes(mirror, base) + report.check( + not smuggled, + "fork base changes shared files only inside the fork-only hunk", + "fork base smuggles a shared-file change: " + "; ".join(smuggled), + "move it to its own feature commit where the guards can see it", + ) + + # The checks above describe the tip. Feature commits are what get merged, so + # each is re-checked against its own parent exactly as check-commit would. + contaminated = [] + for commit in commits[:-1]: + parent = f"{commit}^" + offenders = fork_only_files_in(parent, commit) + blocks = set(changed_fork_blocks(parent, commit)) | set(changed_fork_markers(parent, commit)) + offenders += [f"{path} (fork block)" for path in sorted(blocks)] + offenders += fork_workflow_problems(parent, commit) + if offenders: + contaminated.append(f"{commit[:9]}: {'; '.join(offenders)}") + report.check( + not contaminated, + f"every feature commit on the stack is upstream-clean ({len(commits) - 1} checked)", + "contaminated feature commit(s): " + "; ".join(contaminated), + "fork-only content belongs in the base commit, not in a feature commit", + ) + + return None if report.failed else base + + +def cmd_check_stack(args: argparse.Namespace) -> int: + # With --print-base the SHA is the output, so the report belongs on stderr and + # `BASE=$(forkctl check-stack --print-base) || exit 1` captures only the SHA. + report = Report("check-stack", args.format, sys.stderr if args.print_base else sys.stdout) + base = verify_stack(report, args.mirror, args.stack) + if base is None: + return report.finish() + + if args.print_base: + print(base) + github_output = os.environ.get("GITHUB_OUTPUT") + if args.github_output and github_output: + with open(github_output, "a", encoding="utf-8") as handle: + handle.write(f"base={base}\n") + + return report.finish() + + +def resolve_feature_commit(report: Report, args: argparse.Namespace) -> str | None: + """Section 6: resolve by subject, never by a recorded SHA that rebases stale.""" + if args.sha: + if not ref_exists(args.sha): + report.fail(f"commit not found: {args.sha}") + return None + sha = git("rev-parse", args.sha) + # Gate 3 must prove *the* commit being merged. Naming an older stack + # commit while a different one is checked out certified the wrong diff, + # so --sha has to be the single commit of the branch under review; use + # --subject to rebuild the mirror of a commit that is already stacked. + head = current_branch() + if head_is_detached(): + report.fail( + "HEAD is detached", + "check out the feature branch under review; --subject rebuilds an already-stacked commit", + ) + return None + if sha != git("rev-parse", "HEAD"): + report.fail( + f"--sha {sha[:9]} is not HEAD ({git('rev-parse', 'HEAD')[:9]})", + "Gate 3 proves the commit under review; use --subject to rebuild a stacked commit", + ) + return None + parented_on_tip = git_rc("rev-parse", "--verify", "--quiet", f"{sha}^") == 0 and git( + "rev-parse", f"{sha}^" + ) == git("rev-parse", args.stack) + if not parented_on_tip: + report.fail( + f"{head} is not a single commit on top of the {args.stack} tip", + "squash to one commit and rebase onto the current stack tip (section 4.4)", + ) + return None + return sha + + entries = git_lines("log", "--format=%H%x00%s", f"{args.mirror}..{args.stack}") + matches = [line.split("\0", 1)[0] for line in entries if line.split("\0", 1)[1] == args.subject] + if len(matches) != 1: + report.fail(f"subject matches {len(matches)} stack commits, expected exactly 1: {args.subject!r}") + return None + return matches[0] + + +def restore_checkout(report: Report, started_on: str) -> None: + """Return to where the caller was, and say so if that proved impossible. + + Printing "back on " without checking is how an interrupted run left + people on a conflicted detached HEAD believing it had cleaned up. + """ + if git_rc("cherry-pick", "--abort") == 0: + report.info("aborted an in-progress cherry-pick") + if git_rc("switch", started_on) == 0 or git_rc("switch", "--detach", started_on) == 0: + report.info(f"back on {started_on}") + return + report.fail( + f"could not return to {started_on}; the checkout is detached", + f"resolve the working tree, then: git switch {started_on}", + ) + + +SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$") + + +def cmd_mirror(args: argparse.Namespace) -> int: + """Gate 3 and section 6: rebuild mirror/ and prove it is upstream-clean.""" + report = Report(f"mirror/{args.slug}", args.format) + + # Validated before anything moves: a slug git rejects as a branch name would + # otherwise only surface after the checkout, stranding the user detached. + if not SLUG_RE.match(args.slug): + report.fail(f"invalid slug {args.slug!r}", "short, lowercase, hyphenated: lane-heading-obs") + return report.finish() + if not working_tree_clean(): + report.fail("working tree is dirty", "commit or stash first") + return report.finish() + if not require_refs(report, args.upstream, args.mirror, args.stack): + return report.finish() + + sha = resolve_feature_commit(report, args) + if sha is None: + return report.finish() + report.info(f"feature commit {sha[:9]} {git('log', '-1', '--format=%s', sha)}") + + branch = f"mirror/{args.slug}" + started_on = current_branch() + + # Build detached and only move the branch once every check has passed. An + # existing mirror/ may carry the commit an upstream PR is serving, so + # it must not be destroyed to find out whether its replacement is valid. + stayed = False + try: + # Inside the try: an interrupt between the switch and the first guarded + # statement would otherwise skip the restoration entirely. + git("switch", "--detach", args.upstream) + if git_rc("cherry-pick", sha) != 0: + git_rc("cherry-pick", "--abort") + report.fail( + "cherry-pick onto pure upstream failed -- the commit is entangled with fork-only state", + "fix it on the feature branch and re-run", + ) + else: + report.ok("cherry-picks onto pure upstream without conflict") + + offenders = fork_only_files_in(args.upstream) + report.check( + not offenders, + "no registered fork-only file in the mirror diff", + f"fork-only file(s) in the mirror diff: {', '.join(offenders)}", + ) + + markers = fork_markers_anywhere(args.upstream) + report.check( + not markers, + "no fork-only marker added anywhere in the mirror diff", + f"fork-only marker in the mirror diff: {markers[0] if markers else ''}", + ) + + files = diff_files(args.upstream) + report.info(f"mirror diff touches {len(files)} file(s):") + for path in files: + report.info(f" {path}") + + candidate = git("rev-parse", "HEAD") + if report.failed: + report.info(f"{branch} was left untouched; nothing was published") + else: + if ref_exists(branch) and git("rev-parse", branch) != candidate: + report.info(f"{branch} moved from {git('rev-parse', branch)[:9]} to {candidate[:9]}") + git("branch", "-f", branch, candidate) + if args.stay: + git("switch", branch) + stayed = True + report.info(f"staying on {branch}") + finally: + # Unconditional: an exception or a Ctrl-C after the checkout must not + # leave the caller detached on a half-built or conflicted mirror. + if not stayed: + restore_checkout(report, started_on) + + return report.finish() + + +def upstream_prs_for(slug: str) -> list[dict]: + """Every upstream PR whose head is our mirror branch, in our fork only. + + GitHub's `--head` filter matches by branch *name* across owners, so another + fork pushing the same slug would otherwise be mistaken for ours. + """ + prs = gh_json( + "pr", + "list", + "--repo", + UPSTREAM_REPO, + "--head", + f"mirror/{slug}", + "--state", + "all", + "--limit", + "100", + "--json", + "number,state,headRepositoryOwner", + ) + return [pr for pr in prs if (pr.get("headRepositoryOwner") or {}).get("login") == FORK_OWNER] + + +def cmd_delete_mirror(args: argparse.Namespace) -> int: + """Section 6.2: deleting the head branch closes an open upstream PR.""" + report = Report(f"delete mirror/{args.slug}", args.format) + if not gh_available(): + report.fail("gh is not installed; cannot verify the upstream PR state") + return report.finish() + + try: + prs = upstream_prs_for(args.slug) + except ForkctlError as exc: + report.fail(f"could not verify the upstream PR state: {exc}", "retry; never delete unverified") + return report.finish() + + live = [pr for pr in prs if pr["state"] not in TERMINAL_PR_STATES] + if live: + report.fail( + f"upstream PR(s) {', '.join('#' + str(pr['number']) for pr in live)} for mirror/{args.slug} " + f"are {', '.join(pr['state'] for pr in live)} -- deleting the branch would " + "close them and read as a silent withdrawal" + ) + return report.finish() + states = ", ".join(pr["state"] for pr in prs) or "NONE" + report.ok(f"every upstream PR for this slug is terminal ({states}); deletion is safe") + + if args.dry_run: + report.info("dry run; nothing deleted") + return report.finish() + + if git_rc("push", "origin", "--delete", f"mirror/{args.slug}") != 0: + report.fail("deleting the branch on origin failed", "check the remote and retry") + return report.finish() + if git_lines("ls-remote", "--heads", "origin", f"mirror/{args.slug}"): + report.fail("the branch still exists on origin after the delete") + return report.finish() + report.ok("deleted on origin") + + # The remote delete is the part that matters for the upstream PR; a failed + # local delete (usually because it is the current branch) is still a failure + # to report rather than fold into a single cheerful line. + if not ref_exists(f"mirror/{args.slug}"): + report.ok("no local branch left") + return report.finish() + report.check( + git_rc("branch", "-D", f"mirror/{args.slug}") == 0 and not ref_exists(f"mirror/{args.slug}"), + "deleted locally", + f"the local branch mirror/{args.slug} could not be deleted", + "switch away from it and run `git branch -D` by hand", + ) + return report.finish() + + +def cmd_check_view(args: argparse.Namespace) -> int: + """Section 2: fork-only/3.0 is a derived view, republished by hand.""" + report = Report("check-view", args.format) + base = verify_stack(report, args.mirror, args.stack) + if base is None: + return report.finish() + report.ok(f"stack is sound; fork base is {base[:9]}") + + fix = f"git push --force origin {base}:refs/heads/{VIEW_BRANCH}" + published = "" + for line in git_lines("ls-remote", "origin", f"refs/heads/{VIEW_BRANCH}"): + published = line.split("\t")[0] if "\t" in line else line.split()[0] + + if not published: + report.fail(f"{VIEW_BRANCH} does not exist", fix) + else: + report.check( + published == base, + f"{VIEW_BRANCH} is current at {base[:9]}", + f"{VIEW_BRANCH} is stale ({published[:9]}); the fork base is {base[:9]}", + fix, + ) + + if not gh_available(): + report.fail( + "gh is not installed; the standing PR contract could not be verified", + "install gh, or run this check where gh is available", + ) + return report.finish() + + try: + prs = gh_json( + "pr", + "list", + "--repo", + FORK_REPO, + "--head", + VIEW_BRANCH, + "--state", + "all", + "--limit", + "100", + "--json", + "number,state,baseRefName,isDraft,labels", + ) + except ForkctlError as exc: + report.fail(f"could not query the standing view PR: {exc}", "retry; an unverified contract is not a pass") + return report.finish() + + merged = [pr for pr in prs if pr["state"] == "MERGED"] + if not report.check( + not merged, + f"no {VIEW_BRANCH} PR was ever merged", + f"a {VIEW_BRANCH} PR was MERGED into the mirror -- the mirror is contaminated: " + + ", ".join(f"#{pr['number']}" for pr in merged), + "reset 3.0 to upstream/3.0 (admin action, section 7)", + ): + return report.finish() + + open_prs = [pr for pr in prs if pr["state"] == "OPEN"] + if not open_prs: + report.fail( + f"no standing {VIEW_BRANCH} PR is open -- section 2 requires one, opened by hand", + f"open a draft PR {VIEW_BRANCH} -> 3.0 labelled do-not-merge", + ) + return report.finish() + if not report.check( + len(open_prs) == 1, + "exactly one standing view PR is open", + f"{len(open_prs)} open {VIEW_BRANCH} PRs; exactly one is expected", + ): + return report.finish() + + pr = open_prs[0] + labels = {label["name"] for label in pr["labels"]} + broken = [] + if pr["baseRefName"] != "3.0": + broken.append(f"base={pr['baseRefName']}, expected 3.0") + if not pr["isDraft"]: + broken.append("not a draft") + if "do-not-merge" not in labels: + broken.append("missing the do-not-merge label") + # Both labels together are the in-flight-process-change signal that `doctor` + # queries; a permanent PR carrying both would raise it forever. + if "do-not-mirror" in labels: + broken.append("carries do-not-mirror, which a standing PR must never have") + + report.check( + not broken, + f"standing view PR #{pr['number']} keeps its contract", + f"standing view PR #{pr['number']} breaks its contract: " + "; ".join(broken), + ) + return report.finish() + + +def cmd_status(args: argparse.Namespace) -> int: + """Sections 6.2, 6.3 and 7: divergence report and what is in flight.""" + report = Report("status", "text") + if not require_refs(report, args.mirror, args.stack): + return report.finish() + + rng = f"{args.mirror}..{args.stack}" + subjects = git_lines("log", "--format=%s", rng) + fork_only = sum(1 for s in subjects if s.startswith(FORK_COMMIT_PREFIX)) + + print(f" total gap : {len(subjects)}") + print(f" fork-only : {fork_only} (permanent floor, never drive to zero)") + print(f" feature gap : {len(subjects) - fork_only}") + print("\n stack (newest first):") + for line in git_lines("log", "--format=%h %s", rng): + print(f" {line}") + + if ref_exists(args.upstream): + stale = [line[2:] for line in git_lines("cherry", args.upstream, args.stack) if line.startswith("- ")] + if stale: + report.warn(f"{len(stale)} stack commit(s) already upstream (section 6.3)") + for sha in stale: + print(f" - {git('log', '-1', '--format=%h %s', sha)}") + else: + print("\n no stale commits: nothing we carry is already upstream") + else: + report.fail(f"{args.upstream} not fetched; the stale-commit audit could not run", "git fetch upstream") + return report.finish() + + remote = git_lines("ls-remote", "--heads", "origin", "mirror/*") + slugs = [line.split("refs/heads/mirror/", 1)[1] for line in remote if "refs/heads/mirror/" in line] + + if not gh_available(): + print("\n mirror branches on origin:") + for name in slugs or ["(none)"]: + print(f" mirror/{name}") + report.fail("gh is not installed; the upstream PR inventory could not run") + return report.finish() + + try: + prs = gh_json( + "pr", + "list", + "--repo", + UPSTREAM_REPO, + "--state", + "open", + "--limit", + "100", + "--json", + "number,headRefName,headRepositoryOwner,title", + ) + except ForkctlError as exc: + report.fail(f"could not query upstream PRs: {exc}", "retry; an unverified inventory is not a pass") + return report.finish() + + open_prs = {pr["headRefName"]: pr for pr in prs if (pr.get("headRepositoryOwner") or {}).get("login") == FORK_OWNER} + + stack_patch_ids = { + identity for identity in (patch_id(sha) for sha in git_lines("rev-list", rng)) if identity is not None + } + + print("\n mirror branches on origin:") + if not slugs: + print(" (none)") + for name in slugs: + pr = open_prs.get(f"mirror/{name}") + if pr is None: + print(f" mirror/{name} -- no open upstream PR (finished, or never opened)") + continue + print(f" mirror/{name} -- upstream #{pr['number']} open") + for note in mirror_drift(name, args.upstream, stack_patch_ids): + # Drift is a warning; being unable to measure it is a failure. + if note.startswith(("could not", "not fetched")): + report.fail(f"mirror/{name}: {note}") + else: + report.warn(f"mirror/{name}: {note}") + + orphans = set(open_prs) - {f"mirror/{name}" for name in slugs} + for head in sorted(orphans): + report.warn(f"{head} has an open upstream PR but no branch on origin") + + return report.finish() + + +def cmd_doctor(args: argparse.Namespace) -> int: + """Section 4.1: is this clone set up, and is the process itself mid-change?""" + report = Report("doctor", args.format) + + report.check( + git_rc("remote", "get-url", "upstream") == 0, + "upstream remote is configured", + "upstream remote is missing", + f"git remote add upstream {UPSTREAM_URL}", + ) + + venv = ".venv/Scripts/python.exe" if os.name == "nt" else ".venv/bin/python" + report.check( + os.path.exists(venv), + f"virtualenv present ({venv})", + f"no virtualenv at {venv}", + "uv venv && uv pip install -e .", + ) + + if not gh_available(): + report.fail( + "gh is not installed; cannot tell whether the process is mid-change", + "install gh; an unchecked process state is not a pass", + ) + return report.finish() + + try: + inflight = gh_json( + "pr", + "list", + "--repo", + FORK_REPO, + "--state", + "open", + "--label", + "do-not-mirror", + "--label", + "do-not-merge", + "--json", + "number,title", + ) + except ForkctlError as exc: + report.fail(f"could not query in-flight process PRs: {exc}", "retry before relying on the written process") + return report.finish() + + if not inflight: + report.ok("no in-flight process change; copilot-instructions.md is current") + return report.finish() + + report.warn( + "the fork process is mid-change; these PRs supersede " + "copilot-instructions.md wherever they overlap -- read them first:" + ) + for pr in inflight: + report.info(f" #{pr['number']} {pr['title']}") + return report.finish() + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="forkctl", + description="Fork-process checks for ASCS-eV/PufferDrive.", + epilog="The process itself is described in .github/copilot-instructions.md.", + ) + sub = parser.add_subparsers(dest="command", required=True) + + def add_fmt(p: argparse.ArgumentParser) -> None: + p.add_argument( + "--format", + choices=("text", "github"), + default="text", + help="output style; 'github' emits ::error:: annotations", + ) + + p = sub.add_parser("doctor", help="clone setup and in-flight process changes") + add_fmt(p) + p.set_defaults(func=cmd_doctor) + + p = sub.add_parser( + "check-commit", + help="one commit, no fork-only content (sections 4.4, 4.5)", + ) + p.add_argument("--base", default=STACK_REF) + p.add_argument( + "--chore", + action="store_true", + help="fork-only branch: skip the fork-only checks, keep the one-commit check", + ) + add_fmt(p) + p.set_defaults(func=cmd_check_commit) + + p = sub.add_parser("check-stack", help="stack invariants (sections 3.3, 7)") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--print-base", action="store_true", help="print the fork base SHA") + p.add_argument( + "--github-output", + action="store_true", + help="also append base= to $GITHUB_OUTPUT", + ) + add_fmt(p) + p.set_defaults(func=cmd_check_stack) + + p = sub.add_parser("check-view", help="fork-only/3.0 is current and its standing PR is intact") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + add_fmt(p) + p.set_defaults(func=cmd_check_view) + + p = sub.add_parser("mirror", help="rebuild mirror/ and prove it is upstream-clean (gate 3)") + p.add_argument("slug") + group = p.add_mutually_exclusive_group(required=True) + group.add_argument("--sha", help="feature commit to cherry-pick") + group.add_argument("--subject", help="resolve the commit by its exact subject") + p.add_argument("--upstream", default=UPSTREAM_REF) + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--stay", action="store_true", help="stay on the mirror branch") + add_fmt(p) + p.set_defaults(func=cmd_mirror) + + p = sub.add_parser("delete-mirror", help="delete mirror/ only if its upstream PR is terminal") + p.add_argument("slug") + p.add_argument("--dry-run", action="store_true") + add_fmt(p) + p.set_defaults(func=cmd_delete_mirror) + + p = sub.add_parser("status", help="divergence report and in-flight mirrors") + p.add_argument("--mirror", default=MIRROR_REF) + p.add_argument("--stack", default=STACK_REF) + p.add_argument("--upstream", default=UPSTREAM_REF) + p.set_defaults(func=cmd_status) + + p = sub.add_parser("fork-only", help="print the canonical fork-only path list") + p.add_argument("-v", "--verbose", action="store_true") + p.set_defaults(func=cmd_fork_only) + + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + return args.func(args) + except ForkctlError as exc: + print(f"forkctl: {exc}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/fork/test_forkctl.py b/.github/fork/test_forkctl.py new file mode 100644 index 0000000000..54a270164a --- /dev/null +++ b/.github/fork/test_forkctl.py @@ -0,0 +1,1255 @@ +#!/usr/bin/env python3 +"""Tests for forkctl. Fork-only; never cherry-pick upstream. + +Run: python .github/fork/test_forkctl.py + +Every guard is asserted in both directions -- it passes a well-formed stack and +it *fails* a malformed one. A guard that cannot be shown to fail is not a guard, +and these particular guards are what stand between a bad commit and the branch +BMW pins, so they are exercised against throwaway repositories built to break +them rather than against prose. + +Stdlib only, so it runs anywhere git does, with no test dependencies. +""" + +from __future__ import annotations + +import contextlib +import io +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import forkctl + + +FEATURE = "feat(drive): add a thing" +CHORE = "chore(fork): define the ASCS-eV fork process" + +# The fixture repo is built once and copied per test. Spawning git costs ~0.2s +# on a Windows checkout with live virus scanning, so rebuilding it per test put +# the suite over five minutes; copying the directory keeps full isolation. +_TEMPLATE: tempfile.TemporaryDirectory | None = None +_TEMPLATE_REPO: Path | None = None +_BASE_SHA = "" +_FEATURE_SHA = "" + + +def git(repo: Path, *args: str) -> str: + proc = subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True, check=True) + return proc.stdout.strip() + + +def write(repo: Path, relpath: str, text: str) -> None: + path = repo / relpath + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + + +def commit(repo: Path, subject: str, files: dict[str, str]) -> str: + for relpath, text in files.items(): + write(repo, relpath, text) + git(repo, "add", "-A") + git(repo, "commit", "-m", subject) + return git(repo, "rev-parse", "HEAD") + + +def run(*argv: str) -> tuple[int, str]: + """Invoke forkctl in-process and capture what it printed.""" + buf = io.StringIO() + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf): + code = forkctl.main(list(argv)) + return code, buf.getvalue() + + +CI_YML_UPSTREAM = """\ +name: CI + +on: + pull_request: + branches: ['main', '3.0'] + push: + branches: ['main', '3.0'] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - run: echo build +""" + +CI_YML_FORKED = CI_YML_UPSTREAM.replace( + " pull_request:\n branches: ['main', '3.0']", + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']", +) + +INSTALL_YML_UPSTREAM = CI_YML_UPSTREAM.replace("name: CI", "name: Installation Test") +INSTALL_YML_FORKED = CI_YML_FORKED.replace("name: CI", "name: Installation Test") + +# The fork base adds exactly one block to each shared doc, and nothing else. +AGENTS_UPSTREAM = "## Coding Standards\nexplicit names\n" +AGENTS_FORKED = AGENTS_UPSTREAM + "\n\nfork process\n\n" + + +def upstream_tree() -> dict[str, str]: + return { + "README.md": "upstream\n", + "AGENTS.md": AGENTS_UPSTREAM, + "CLAUDE.md": AGENTS_UPSTREAM, + ".github/workflows/ci.yml": CI_YML_UPSTREAM, + ".github/workflows/install.yml": INSTALL_YML_UPSTREAM, + } + + +def fork_base_tree() -> dict[str, str]: + """Everything the real fork base carries, including every required file. + + The required-file invariant is only meaningful if the well-formed fixture + actually satisfies it; an incomplete fixture made `check-stack` pass on a + base that had lost the detector. + """ + tree = { + "AGENTS.md": AGENTS_FORKED, + "CLAUDE.md": AGENTS_FORKED, + ".github/workflows/ci.yml": CI_YML_FORKED, + ".github/workflows/install.yml": INSTALL_YML_FORKED, + } + for path in forkctl.REQUIRED_FORK_FILES: + tree[path] = f"fork-only: {path}\n" + return tree + + +def setUpModule() -> None: + global _TEMPLATE, _TEMPLATE_REPO, _BASE_SHA, _FEATURE_SHA + _TEMPLATE = tempfile.TemporaryDirectory() + repo = Path(_TEMPLATE.name) / "template" + repo.mkdir() + + git(repo, "-c", "init.defaultBranch=main", "init", "-q") + git(repo, "config", "user.email", "test@example.invalid") + git(repo, "config", "user.name", "Fork Test") + git(repo, "config", "commit.gpgsign", "false") + + commit(repo, "upstream: initial", upstream_tree()) + git(repo, "branch", "up") + git(repo, "branch", "mir") + + git(repo, "switch", "-q", "-c", "stack", "mir") + _BASE_SHA = commit(repo, CHORE, fork_base_tree()) + _FEATURE_SHA = commit(repo, FEATURE, {"sim.c": "int main(void){}\n"}) + _TEMPLATE_REPO = repo + + +def tearDownModule() -> None: + if _TEMPLATE is not None: + _TEMPLATE.cleanup() + + +class ForkRepoTest(unittest.TestCase): + """A throwaway repo shaped like the real fork: mirror + stack + upstream.""" + + def setUp(self) -> None: + self._cwd = os.getcwd() + self._tmp = tempfile.TemporaryDirectory() + self.repo = Path(self._tmp.name) / "repo" + shutil.copytree(_TEMPLATE_REPO, self.repo) + self.base = _BASE_SHA + self.feature = _FEATURE_SHA + os.chdir(self.repo) + + def tearDown(self) -> None: + os.chdir(self._cwd) + self._tmp.cleanup() + + def branch_from_stack(self, name: str) -> None: + git(self.repo, "switch", "-q", "-c", name, "stack") + + def stacked_feature(self, subject: str, tree: dict[str, str], name: str = "feature/x") -> str: + """A one-commit branch on top of the stack tip, exactly as Gate 3 expects.""" + self.branch_from_stack(name) + return commit(self.repo, subject, tree) + + +class TestCheckStack(ForkRepoTest): + def test_passes_on_a_well_formed_stack(self) -> None: + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + self.assertIn("exactly one chore(fork): commit", out) + + def test_prints_the_fork_base(self) -> None: + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack", "--print-base") + self.assertEqual(code, 0, out) + self.assertIn(self.base, out) + + def test_fails_on_a_second_chore_commit(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "chore(fork): second one", {"CONTRIBUTING.md": "again\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("2 chore(fork): commits", out) + + def test_fails_on_a_merge_commit(self) -> None: + git(self.repo, "switch", "-q", "-c", "side", "mir") + commit(self.repo, "side work", {"side.txt": "x\n"}) + git(self.repo, "switch", "-q", "stack") + git(self.repo, "merge", "--no-ff", "-m", "merge side", "side") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("merge commit on the stack", out) + + def test_fails_on_a_stray_fixup_commit(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, f"fixup! {CHORE}", {"CONTRIBUTING.md": "wip\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("stray fixup!/squash! commit", out) + + def test_fails_when_the_base_is_not_the_fork_commit(self) -> None: + git(self.repo, "switch", "-q", "-c", "wrong", "mir") + commit(self.repo, "feat(x): not the fork base", {"a.txt": "a\n"}) + commit(self.repo, CHORE, {"CONTRIBUTING.md": "fork\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "wrong") + self.assertEqual(code, 1, out) + self.assertIn("is not the fork base", out) + + def test_fails_when_the_stack_does_not_contain_the_mirror_tip(self) -> None: + git(self.repo, "switch", "-q", "mir") + commit(self.repo, "upstream: moved on", {"README.md": "newer\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("does not contain", out) + + def test_fails_on_a_missing_ref(self) -> None: + code, out = run("check-stack", "--mirror", "nope", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not found locally", out) + + def test_github_format_emits_annotations(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "chore(fork): second one", {"CONTRIBUTING.md": "again\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack", "--format", "github") + self.assertEqual(code, 1, out) + self.assertIn("::error::", out) + + +class TestCheckCommit(ForkRepoTest): + def test_passes_for_a_clean_single_commit_feature(self) -> None: + self.branch_from_stack("feature/clean") + commit(self.repo, "feat(obs): lane heading", {"obs.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_fails_on_two_commits(self) -> None: + self.branch_from_stack("feature/two") + commit(self.repo, "feat(obs): one", {"obs.c": "x\n"}) + commit(self.repo, "feat(obs): two", {"obs2.c": "y\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("2 commits on the branch", out) + + def test_fails_when_a_fork_only_file_is_touched(self) -> None: + self.branch_from_stack("feature/dirty") + commit( + self.repo, + "feat(obs): sneaky", + {"obs.c": "x\n", "CONTRIBUTING.md": "edited\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only file(s) in the feature diff", out) + self.assertIn("CONTRIBUTING.md", out) + + def test_fails_for_a_new_file_under_the_fork_only_prefix(self) -> None: + self.branch_from_stack("feature/newtool") + commit(self.repo, "feat(x): tool", {".github/fork/helper.py": "print()\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn(".github/fork/helper.py", out) + + def test_fails_when_content_inside_the_fork_block_is_edited(self) -> None: + """The changed lines carry no marker, so a diff-line scan misses this.""" + self.branch_from_stack("feature/hunk") + commit( + self.repo, + "feat(x): edit the fork block", + {"AGENTS.md": "style\n\nCHANGED\n\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only block content changed in: AGENTS.md", out) + + def test_fails_when_the_fork_block_is_removed(self) -> None: + self.branch_from_stack("feature/deblock") + commit(self.repo, "feat(x): drop the fork block", {"AGENTS.md": "style\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only", out) + + def test_fails_when_the_fork_branch_filter_is_removed(self) -> None: + self.branch_from_stack("feature/filter-drop") + commit(self.repo, "ci(x): tidy the filter", {".github/workflows/ci.yml": CI_YML_UPSTREAM}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("removed from `pull_request`", out) + + def test_fails_when_the_fork_branch_filter_moves_to_push(self) -> None: + """A move leaves the marker line intact, so only a structural read catches it.""" + self.branch_from_stack("feature/filter-move") + moved = CI_YML_UPSTREAM.replace( + " push:\n branches: ['main', '3.0']", + " push:\n branches: ['main', '3.0', 'integration/3.0']", + ) + commit(self.repo, "ci(x): move the filter", {".github/workflows/ci.yml": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("added to `push`", out) + + def test_allows_editing_the_rest_of_a_shared_file(self) -> None: + self.branch_from_stack("feature/shared-ok") + commit( + self.repo, + "docs(agents): clarify naming", + {"AGENTS.md": AGENTS_FORKED.replace("explicit names", "explicit, unit-suffixed names")}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_chore_mode_exempts_the_fork_only_checks(self) -> None: + self.branch_from_stack("chore/process") + commit(self.repo, "chore(fork): rewrite", {"CONTRIBUTING.md": "new\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 0, out) + self.assertIn("exempt", out) + + def test_chore_mode_still_enforces_one_commit(self) -> None: + self.branch_from_stack("chore/two") + commit(self.repo, "chore(fork): one", {"CONTRIBUTING.md": "a\n"}) + commit(self.repo, "chore(fork): two", {"CONTRIBUTING.md": "b\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 1, out) + self.assertIn("2 commits on the branch", out) + + def test_chore_mode_rejects_a_non_chore_subject(self) -> None: + self.branch_from_stack("chore/mislabelled") + commit(self.repo, "feat(x): not a chore", {"CONTRIBUTING.md": "a\n"}) + code, out = run("check-commit", "--base", "stack", "--chore") + self.assertEqual(code, 1, out) + self.assertIn("--chore given but the subject", out) + + def test_warns_when_a_chore_commit_is_checked_without_chore(self) -> None: + self.branch_from_stack("chore/forgot-flag") + commit(self.repo, "chore(fork): rewrite", {"CONTRIBUTING.md": "new\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("WARN", out) + + +class TestMirror(ForkRepoTest): + def test_clean_feature_passes_gate_three(self) -> None: + sha = self.stacked_feature("feat(x): clean work", {"clean.c": "x\n"}) + code, out = run( + "mirror", + "clean", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + self.assertIn("cherry-picks onto pure upstream without conflict", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + + def test_resolves_the_commit_by_subject(self) -> None: + code, out = run( + "mirror", + "clean", + "--subject", + FEATURE, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + + def test_rejects_an_ambiguous_subject(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, FEATURE, {"other.c": "y\n"}) + code, out = run( + "mirror", + "dupe", + "--subject", + FEATURE, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("matches 2 stack commits", out) + + def test_fails_when_the_commit_adds_a_fork_only_file(self) -> None: + sha = self.stacked_feature( + "feat(x): contaminated", + {"x.c": "x\n", ".github/fork/scratch-notes.md": "fork-only\n"}, + ) + code, out = run( + "mirror", + "bad", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only file(s) in the mirror diff", out) + + def test_modifying_a_fork_only_file_conflicts_on_pure_upstream(self) -> None: + """Such a commit cannot even apply upstream -- the file is not there.""" + sha = self.stacked_feature( + "feat(x): edits a fork-only file", + {"x.c": "x\n", "CONTRIBUTING.md": "touched\n"}, + ) + code, out = run( + "mirror", + "bad2", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("entangled with fork-only state", out) + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_fails_when_the_commit_adds_a_fork_only_marker(self) -> None: + sha = self.stacked_feature( + "ci(x): add a workflow gated on the fork branch", + {".github/workflows/extra.yml": "on:\n push:\n branches: ['integration/3.0']\n"}, + ) + code, out = run( + "mirror", + "marker", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + code, out = run( + "mirror", + "marker", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + + def test_aborts_and_restores_the_branch_on_conflict(self) -> None: + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: conflicting edit", {"sim.c": "DIFFERENT\n"}) + sha = self.stacked_feature("feat(x): conflicting work", {"sim.c": "OURS\n"}) + code, out = run( + "mirror", + "conflict", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("entangled with fork-only state", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_rebuilds_an_existing_mirror_branch(self) -> None: + git(self.repo, "branch", "mirror/clean", "up") + sha = self.stacked_feature("feat(x): clean work", {"clean.c": "x\n"}) + code, out = run( + "mirror", + "clean", + "--sha", + sha, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 0, out) + + def test_refuses_to_run_with_a_dirty_tree(self) -> None: + write(self.repo, "scratch.txt", "uncommitted\n") + code, out = run( + "mirror", + "clean", + "--sha", + self.feature, + "--upstream", + "up", + "--mirror", + "mir", + "--stack", + "stack", + ) + self.assertEqual(code, 1, out) + self.assertIn("dirty", out) + + +class TestMirrorDrift(ForkRepoTest): + """Section 6.2: the two events that make an open upstream PR wrong.""" + + def stack_patch_ids(self) -> set[str]: + ids = set() + for sha in git(self.repo, "rev-list", "mir..stack").split(): + identity = forkctl.patch_id(sha) + if identity is not None: + ids.add(identity) + return ids + + def test_a_fresh_mirror_reports_no_drift(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_event_one_fires_when_upstream_conflicts(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: rewrite the same file", {"sim.c": "int main(void){return 7;}\n"}) + git(self.repo, "switch", "-q", "stack") + notes = forkctl.mirror_drift("clean", "up", self.stack_patch_ids()) + self.assertTrue(any("event 1" in note for note in notes), notes) + + def test_an_unrelated_upstream_advance_is_not_drift(self) -> None: + """A readable PR needs no action, however far upstream has moved.""" + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: unrelated progress", {"NOTICE": "new\n"}) + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_event_two_fires_when_the_stack_commit_was_amended(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + write(self.repo, "sim.c", "int main(void){return 1;}\n") + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "--amend", "--no-edit") + notes = forkctl.mirror_drift("clean", "up", self.stack_patch_ids()) + self.assertTrue(any("event 2" in note for note in notes), notes) + + def test_renaming_the_subject_alone_is_not_drift(self) -> None: + """Patch-identity ignores the message, so upstream renaming is quiet.""" + git(self.repo, "switch", "-q", "-c", "mirror/clean", "up") + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "commit", "-q", "--amend", "-m", "Add a thing to the simulator") + git(self.repo, "switch", "-q", "stack") + self.assertEqual(forkctl.mirror_drift("clean", "up", self.stack_patch_ids()), []) + + def test_a_missing_mirror_branch_is_reported(self) -> None: + notes = forkctl.mirror_drift("absent", "up", self.stack_patch_ids()) + self.assertTrue(any("not fetched" in note for note in notes), notes) + + +class TestReviewRegressions(ForkRepoTest): + """One test per bypass found in the Gate 2 adversarial review of PR #9.""" + + def feature_branch(self, name: str = "feature/x") -> None: + self.branch_from_stack(name) + + def test_renaming_a_fork_only_file_away_is_caught(self) -> None: + """Rename detection reports only the destination, hiding the source.""" + self.feature_branch() + git(self.repo, "mv", "CONTRIBUTING.md", "FORK_GUIDE.md") + git(self.repo, "commit", "-q", "-m", "docs(x): rename the guide") + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("CONTRIBUTING.md", out) + + def test_moving_the_branch_filter_to_push_is_caught(self) -> None: + """The marker line is unchanged, so only a structural read catches it.""" + self.feature_branch() + moved = CI_YML_FORKED.replace( + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']\n push:\n branches: ['main', '3.0']", + " pull_request:\n branches: ['main', '3.0']\n push:\n branches: ['main', '3.0', 'integration/3.0']", + ) + self.assertNotEqual(moved, CI_YML_FORKED) + commit(self.repo, "ci(x): retune triggers", {".github/workflows/ci.yml": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("pull_request", out) + + def test_editing_the_rest_of_a_shared_workflow_is_allowed(self) -> None: + self.feature_branch() + commit( + self.repo, + "ci(x): add a step", + {".github/workflows/ci.yml": CI_YML_FORKED.replace("echo build", "echo build && echo more")}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_a_duplicate_fork_block_is_rejected(self) -> None: + """A second block would be unprotected: fork_blocks compared only one.""" + self.feature_branch() + commit( + self.repo, + "docs(x): touch agents", + { + "AGENTS.md": "style\n\nfork\n\nmore\n\nb\n\n" + }, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("marker structure", out) + + def test_a_stale_base_is_rejected(self) -> None: + """One commit *ahead* is not the same as rebased onto the current tip.""" + self.feature_branch() + commit(self.repo, "feat(x): mine", {"mine.c": "x\n"}) + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "feat(y): someone else landed first", {"theirs.c": "y\n"}) + git(self.repo, "switch", "-q", "feature/x") + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not parented on the current", out) + + def test_a_non_conventional_subject_is_rejected(self) -> None: + self.feature_branch() + commit(self.repo, "totally invalid subject", {"a.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("Conventional Commits", out) + + def test_an_over_long_subject_is_rejected(self) -> None: + self.feature_branch() + commit(self.repo, "feat(drive): " + "x" * 70, {"a.c": "x\n"}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("72-char limit", out) + + def test_a_leading_space_subject_cannot_pose_as_the_fork_base(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-leading", "mir") + git(self.repo, "commit", "-q", "--allow-empty", "-m", " chore(fork): leading") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-leading") + self.assertEqual(code, 1, out) + self.assertIn("stray whitespace", out) + + def test_a_leading_space_hides_nothing_from_the_fixup_check(self) -> None: + self.branch_from_stack("stack-fixup") + git(self.repo, "commit", "-q", "--allow-empty", "-m", " fixup! chore(fork): x") + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-fixup") + self.assertEqual(code, 1, out) + self.assertIn("fixup", out) + + def test_an_unregistered_file_in_the_fork_base_is_caught(self) -> None: + """Keeps the hardcoded fork-only list honest as the base grows.""" + git(self.repo, "switch", "-q", "-c", "stack-unreg", "mir") + commit(self.repo, CHORE, {".github/workflows/sync-two.yml": "name: x\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-unreg") + self.assertEqual(code, 1, out) + self.assertIn("unregistered", out) + + def test_gate_three_refuses_an_arbitrary_commit(self) -> None: + """--sha must prove the commit being merged, not any commit that exists.""" + git(self.repo, "switch", "-q", "-c", "elsewhere", "up") + outsider = commit(self.repo, "feat(x): unrelated external commit", {"x.c": "x\n"}) + code, out = run("mirror", "arb", "--sha", outsider, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not a single commit on top of", out) + + def test_gate_three_refuses_a_commit_that_is_not_checked_out(self) -> None: + git(self.repo, "switch", "-q", "-c", "elsewhere-2", "up") + outsider = commit(self.repo, "feat(x): unrelated external commit", {"x.c": "x\n"}) + git(self.repo, "switch", "-q", "stack") + code, out = run("mirror", "arb", "--sha", outsider, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("is not HEAD", out) + + def test_gate_three_accepts_a_one_commit_feature_branch(self) -> None: + self.feature_branch() + sha = commit(self.repo, "feat(x): on the tip", {"tip.c": "x\n"}) + code, out = run("mirror", "tip", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_failed_mirror_leaves_the_existing_branch_intact(self) -> None: + """The old tip may be what an open upstream PR is serving.""" + git(self.repo, "switch", "-q", "-c", "mirror/keep", "up") + kept = commit(self.repo, "feat(x): already published", {"published.c": "x\n"}) + bad = self.stacked_feature("feat(x): drags in fork state", {"CONTRIBUTING.md": "changed\n"}) + code, out = run("mirror", "keep", "--sha", bad, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertEqual(git(self.repo, "rev-parse", "mirror/keep"), kept) + + def test_mirror_refuses_a_detached_head_without_moving_it(self) -> None: + git(self.repo, "switch", "-q", "--detach", "stack") + started = git(self.repo, "rev-parse", "HEAD") + code, out = run( + "mirror", "det", "--sha", self.feature, "--upstream", "up", "--mirror", "mir", "--stack", "stack" + ) + self.assertEqual(code, 1, out) + self.assertIn("HEAD is detached", out) + self.assertEqual(git(self.repo, "rev-parse", "HEAD"), started) + + def test_drift_reports_an_extra_commit_on_the_mirror(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/extra", "up") + commit(self.repo, "feat(x): smuggled extra", {"extra.c": "x\n"}) + git(self.repo, "cherry-pick", self.feature) + git(self.repo, "switch", "-q", "stack") + ids = {forkctl.patch_id(s) for s in git(self.repo, "rev-list", "mir..stack").split()} + notes = forkctl.mirror_drift("extra", "up", {i for i in ids if i}) + self.assertTrue(any("expected exactly 1" in note for note in notes), notes) + + def test_prose_mentioning_the_fork_branch_does_not_block_a_mirror(self) -> None: + """`integration/3.0` is only evidence inside CI configuration.""" + self.feature_branch() + sha = commit(self.repo, "docs(x): describe branches", {"README.md": "we also run integration/3.0 here\n"}) + code, out = run("mirror", "prose", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_fork_marker_in_prose_still_blocks_a_mirror(self) -> None: + self.feature_branch() + sha = commit(self.repo, "docs(x): leak a marker", {"README.md": "\n"}) + code, out = run("mirror", "leak", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + + +class TestFailClosed(ForkRepoTest): + """A guard that cannot verify must not report success.""" + + def setUp(self) -> None: + super().setUp() + self._gh_available = forkctl.gh_available + self._gh_json = forkctl.gh_json + + def tearDown(self) -> None: + forkctl.gh_available = self._gh_available + forkctl.gh_json = self._gh_json + super().tearDown() + + def test_doctor_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("doctor") + self.assertEqual(code, 1, out) + + def test_doctor_fails_when_gh_errors(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = self._raise + code, out = run("doctor") + self.assertEqual(code, 1, out) + self.assertIn("could not query", out) + + def test_status_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("status", "--mirror", "mir", "--stack", "stack", "--upstream", "up") + self.assertEqual(code, 1, out) + + def test_status_fails_when_upstream_is_not_fetched(self) -> None: + code, out = run("status", "--mirror", "mir", "--stack", "stack", "--upstream", "nope") + self.assertEqual(code, 1, out) + + def test_check_view_fails_without_gh(self) -> None: + forkctl.gh_available = lambda: False + code, out = run("check-view", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + + def test_delete_mirror_fails_when_gh_errors(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = self._raise + code, out = run("delete-mirror", "slug") + self.assertEqual(code, 1, out) + self.assertIn("could not verify", out) + + def test_delete_mirror_refuses_an_open_pr_from_our_fork(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a: [ + {"number": 1, "state": "CLOSED", "headRepositoryOwner": {"login": "ASCS-eV"}}, + {"number": 2, "state": "OPEN", "headRepositoryOwner": {"login": "ASCS-eV"}}, + ] + code, out = run("delete-mirror", "slug", "--dry-run") + self.assertEqual(code, 1, out) + self.assertIn("#2", out) + + def test_delete_mirror_ignores_another_fork_with_the_same_slug(self) -> None: + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a: [ + {"number": 9, "state": "OPEN", "headRepositoryOwner": {"login": "SomeoneElse"}}, + ] + code, out = run("delete-mirror", "slug", "--dry-run") + self.assertEqual(code, 0, out) + + @staticmethod + def _raise(*args: str): + raise forkctl.ForkctlError("simulated gh outage") + + +class TestWorkflowParser(unittest.TestCase): + def test_reads_inline_and_block_branch_lists(self) -> None: + text = ( + "name: CI\non:\n pull_request:\n branches:\n - 'main'\n - 'integration/3.0'\n" + " push:\n branches: ['main']\njobs: {}\n" + ) + filters = forkctl.workflow_branch_filters(text) + self.assertEqual(filters["pull_request"], ["main", "integration/3.0"]) + self.assertEqual(filters["push"], ["main"]) + + def test_ignores_comments_and_handles_quoted_on(self) -> None: + text = "'on':\n pull_request:\n # a comment\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], ["integration/3.0"]) + + def test_branches_ignore_is_a_different_key(self) -> None: + """Prefix-matching `branches` let a commit swap in the opposite filter.""" + text = "on:\n pull_request:\n branches-ignore: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], []) + + def test_inline_comment_is_not_part_of_a_branch_name(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0'] # fork CI\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], ["integration/3.0"]) + + def test_a_block_scalar_decoy_is_rejected(self) -> None: + text = "on:\n pull_request:\n description: |\n branches: ['integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_flow_mapping_is_rejected_rather_than_misread(self) -> None: + text = "on:\n pull_request: {branches: ['integration/3.0']}\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_alias_is_rejected(self) -> None: + text = "on:\n pull_request:\n branches: *common\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_inline_on_value_is_rejected(self) -> None: + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters("on: [push, pull_request]\n") + + def test_a_tab_indent_is_rejected(self) -> None: + text = "on:\n pull_request:\n\t\tbranches: ['integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + +class TestRoundTwoRegressions(ForkRepoTest): + """One per finding of the second adversarial review.""" + + def write_ci(self, text: str) -> None: + write(self.repo, ".github/workflows/ci.yml", text) + + def test_swapping_branches_for_branches_ignore_is_caught(self) -> None: + self.branch_from_stack("feature/ignore-swap") + swapped = CI_YML_FORKED.replace( + " pull_request:\n branches: ['main', '3.0', 'integration/3.0']", + " pull_request:\n branches-ignore: ['main', '3.0', 'integration/3.0']", + ) + commit(self.repo, "ci(x): swap the filter", {".github/workflows/ci.yml": swapped}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("removed from `pull_request`", out) + + def test_unparseable_workflow_fails_closed(self) -> None: + self.branch_from_stack("feature/unparseable") + commit( + self.repo, + "ci(x): restructure triggers", + {".github/workflows/ci.yml": "name: CI\non: [push, pull_request]\njobs: {}\n"}, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_a_later_stack_commit_touching_a_fork_only_file_is_caught(self) -> None: + git(self.repo, "switch", "-q", "stack") + commit(self.repo, "feat(x): sneak", {"CONTRIBUTING.md": "changed by a feature\n"}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_later_stack_commit_editing_the_fork_block_is_caught(self) -> None: + git(self.repo, "switch", "-q", "stack") + edited = AGENTS_FORKED.replace("fork process", "fork process, quietly rewritten") + commit(self.repo, "docs(x): tweak", {"AGENTS.md": edited}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_shared_change_smuggled_into_the_fork_base_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-smuggle", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = "## Coding Standards\nSECRETLY CHANGED\n" + AGENTS_FORKED.split("\n", 2)[2] + commit(self.repo, CHORE, tree) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-smuggle") + self.assertEqual(code, 1, out) + self.assertIn("smuggles a shared-file change", out) + + def test_a_fork_base_deleting_a_shared_workflow_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-del", "mir") + tree = fork_base_tree() + del tree[".github/workflows/ci.yml"] + write(self.repo, ".github/copilot-instructions.md", "fork\n") + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "rm", "-q", ".github/workflows/ci.yml") + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-del") + self.assertEqual(code, 1, out) + self.assertIn("is missing", out) + + def test_gate_three_refuses_an_older_stack_commit(self) -> None: + """--sha must agree with HEAD, or it certifies a diff nobody is merging.""" + self.branch_from_stack("feature/current") + commit(self.repo, "feat(x): the commit actually under review", {"current.c": "x\n"}) + code, out = run( + "mirror", "wrong", "--sha", self.feature, "--upstream", "up", "--mirror", "mir", "--stack", "stack" + ) + self.assertEqual(code, 1, out) + self.assertIn("is not HEAD", out) + + def test_gate_three_accepts_the_commit_under_review(self) -> None: + self.branch_from_stack("feature/current-ok") + sha = commit(self.repo, "feat(x): the commit actually under review", {"current.c": "x\n"}) + code, out = run("mirror", "ok", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_an_invalid_slug_is_refused_before_anything_moves(self) -> None: + sha = self.stacked_feature("feat(x): work under review", {"slug.c": "x\n"}) + code, out = run("mirror", "bad..slug", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("invalid slug", out) + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + + def test_an_exception_after_checkout_still_restores_the_branch(self) -> None: + sha = self.stacked_feature("feat(x): work under review", {"boom.c": "x\n"}) + original = forkctl.fork_markers_anywhere + forkctl.fork_markers_anywhere = lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom")) + try: + with self.assertRaises(RuntimeError): + run("mirror", "boom", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + finally: + forkctl.fork_markers_anywhere = original + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/x") + self.assertNotIn("mirror/boom", git(self.repo, "branch", "--list", "mirror/boom")) + + def test_prose_under_github_may_mention_the_fork_branch(self) -> None: + """Scoping the marker to all of .github/ blocked legitimate documentation.""" + self.branch_from_stack("feature/prose") + sha = commit( + self.repo, + "docs(x): document branch naming", + {".github/branch-naming.md": "An example long-lived branch is integration/3.0.\n"}, + ) + code, out = run("mirror", "prose", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 0, out) + + def test_a_workflow_mentioning_the_fork_branch_is_still_caught(self) -> None: + self.branch_from_stack("feature/wf") + sha = commit( + self.repo, + "ci(x): add a workflow", + {".github/workflows/extra.yml": "on:\n push:\n branches: ['integration/3.0']\n"}, + ) + code, out = run("mirror", "wf", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("fork-only marker", out) + + def test_delete_mirror_reports_a_failed_local_delete(self) -> None: + git(self.repo, "switch", "-q", "-c", "mirror/busy", "up") + saved = (forkctl.gh_available, forkctl.gh_json, forkctl.git_rc, forkctl.git_lines) + original_rc = forkctl.git_rc + + def fake_rc(*args: str) -> int: + # The remote delete succeeds; the local one cannot, because the + # branch is checked out. The command must not call that success. + return 0 if args[:3] == ("push", "origin", "--delete") else original_rc(*args) + + forkctl.gh_available = lambda: True + forkctl.gh_json = lambda *a, **k: [] + forkctl.git_rc = fake_rc + forkctl.git_lines = lambda *a, **k: [] + try: + code, out = run("delete-mirror", "busy") + finally: + forkctl.gh_available, forkctl.gh_json, forkctl.git_rc, forkctl.git_lines = saved + self.assertEqual(code, 1, out) + self.assertIn("could not be deleted", out) + + +class TestRoundThreeRegressions(ForkRepoTest): + """One per finding of the third adversarial review.""" + + def test_a_negative_pattern_is_rejected(self) -> None: + """GitHub evaluates patterns in order, so a later `!` excludes the branch.""" + text = "on:\n pull_request:\n branches: ['main', 'integration/3.0', '!integration/3.0']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_negative_pattern_in_a_block_list_is_rejected(self) -> None: + text = "on:\n pull_request:\n branches:\n - 'integration/3.0'\n - '!integration/3.0'\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_comma_inside_a_quoted_scalar_is_one_branch(self) -> None: + text = "on:\n pull_request:\n branches: ['main', '3.0,integration/3.0']\n" + values = forkctl.workflow_branch_filters(text)["pull_request"] + self.assertEqual(values, ["main", "3.0,integration/3.0"]) + self.assertNotIn("integration/3.0", values) + + def test_a_boolean_on_key_is_not_the_on_block(self) -> None: + """YAML 1.1 coerces bare `on` to true, but GitHub reads only `on`.""" + text = "True:\n pull_request:\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text), {}) + + def test_branches_nested_under_another_key_is_not_the_filter(self) -> None: + text = "on:\n pull_request:\n types:\n branches: ['integration/3.0']\n" + self.assertEqual(forkctl.workflow_branch_filters(text)["pull_request"], []) + + def test_duplicate_branches_keys_are_rejected(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0']\n branches: ['main']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_duplicate_triggers_are_rejected(self) -> None: + text = "on:\n pull_request:\n branches: ['integration/3.0']\n pull_request:\n branches: ['main']\n" + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_an_excluded_branch_is_caught_end_to_end(self) -> None: + self.branch_from_stack("feature/negated") + negated = CI_YML_FORKED.replace( + "branches: ['main', '3.0', 'integration/3.0']", + "branches: ['main', '3.0', 'integration/3.0', '!integration/3.0']", + ) + commit(self.repo, "ci(x): exclude the fork branch", {".github/workflows/ci.yml": negated}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_a_deleted_required_fork_file_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-no-detector", "mir") + tree = fork_base_tree() + del tree[".github/workflows/fork-only-view.yml"] + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-no-detector") + self.assertEqual(code, 1, out) + self.assertIn("required fork-only infrastructure was deleted", out) + + def test_the_well_formed_fixture_carries_every_required_file(self) -> None: + """Guards the test above: an incomplete fixture would hide the invariant.""" + for path in forkctl.REQUIRED_FORK_FILES: + self.assertIn(path, fork_base_tree(), path) + + def test_a_feature_repairing_a_malformed_base_marker_is_caught(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-stray", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = AGENTS_FORKED + "\n" + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + commit(self.repo, "docs(x): tidy up", {"AGENTS.md": AGENTS_FORKED}) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-stray") + self.assertEqual(code, 1, out) + self.assertIn("contaminated feature commit", out) + + def test_a_malformed_fork_base_is_caught_on_its_own(self) -> None: + git(self.repo, "switch", "-q", "-c", "stack-stray-only", "mir") + tree = fork_base_tree() + tree["AGENTS.md"] = AGENTS_FORKED + "\n" + for path, text in tree.items(): + write(self.repo, path, text) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", CHORE) + code, out = run("check-stack", "--mirror", "mir", "--stack", "stack-stray-only") + self.assertEqual(code, 1, out) + self.assertIn("marker structure is broken", out) + + def test_gate_three_refuses_a_detached_head(self) -> None: + """A detached HEAD at an older stack commit certified a diff nobody merges.""" + git(self.repo, "switch", "-q", "stack") + older = git(self.repo, "rev-parse", "HEAD") + commit(self.repo, "feat(x): newer stacked work", {"newer.c": "x\n"}) + git(self.repo, "switch", "-q", "--detach", older) + code, out = run("mirror", "old", "--sha", older, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("HEAD is detached", out) + + def test_gate_three_refuses_a_branch_that_is_not_one_commit_on_the_tip(self) -> None: + git(self.repo, "switch", "-q", "-c", "feature/two", "stack") + commit(self.repo, "feat(x): first", {"a.c": "x\n"}) + sha = commit(self.repo, "feat(x): second", {"b.c": "x\n"}) + code, out = run("mirror", "two", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + self.assertEqual(code, 1, out) + self.assertIn("not a single commit on top of", out) + + def test_an_interrupted_cherry_pick_is_aborted_and_restored(self) -> None: + """Ctrl-C between the cherry-pick and its abort left a conflicted detached HEAD.""" + git(self.repo, "switch", "-q", "up") + commit(self.repo, "upstream: conflicting edit", {"sim.c": "UPSTREAM\n"}) + sha = self.stacked_feature("feat(x): conflicting work", {"sim.c": "OURS\n"}, "feature/interrupted") + saved = forkctl.git_rc + + def interrupt_after_cherry_pick(*argv: str) -> int: + code = saved(*argv) + if argv[0] == "cherry-pick" and "--abort" not in argv: + raise KeyboardInterrupt + return code + + forkctl.git_rc = interrupt_after_cherry_pick + try: + with self.assertRaises(KeyboardInterrupt): + run("mirror", "boom", "--sha", sha, "--upstream", "up", "--mirror", "mir", "--stack", "stack") + finally: + forkctl.git_rc = saved + self.assertEqual(git(self.repo, "rev-parse", "--abbrev-ref", "HEAD"), "feature/interrupted") + self.assertEqual(git(self.repo, "status", "--porcelain"), "") + + def test_a_crlf_rewrite_in_the_fork_base_is_not_normalised_away(self) -> None: + clean = AGENTS_UPSTREAM + "\n" + self.assertNotEqual( + forkctl.strip_fork_block(AGENTS_FORKED.replace("\n", "\r\n")), + forkctl.strip_fork_block(AGENTS_FORKED), + ) + self.assertEqual(forkctl.strip_fork_block(AGENTS_FORKED), clean) + + def test_the_real_fork_base_shape_still_compares_equal(self) -> None: + """The strip is exact, so the legitimate base must still pass it.""" + self.assertIn( + forkctl.strip_fork_block(AGENTS_UPSTREAM), + (forkctl.strip_fork_block(AGENTS_FORKED), forkctl.strip_fork_block(AGENTS_FORKED).rstrip("\n") + "\n"), + ) + + +class TestRoundFourRegressions(ForkRepoTest): + def test_a_duplicate_top_level_on_key_is_refused(self) -> None: + """Only the first `on:` was read, so a second one could remove the PR trigger.""" + text = ( + "on:\n pull_request:\n branches: [main, '3.0', 'integration/3.0']\n" + "\non:\n push:\n branches: [main]\n" + ) + with self.assertRaises(forkctl.ForkctlError): + forkctl.workflow_branch_filters(text) + + def test_a_duplicate_on_key_fails_check_commit_closed(self) -> None: + self.branch_from_stack("feature/dup-on") + commit( + self.repo, + "ci(x): workflow with two on: keys", + { + ".github/workflows/ci.yml": ( + "on:\n pull_request:\n branches: [main, '3.0', 'integration/3.0']\n" + "\non:\n push:\n branches: [main]\n" + ) + }, + ) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + + def test_moving_an_intact_fork_block_is_caught(self) -> None: + """Git renders the move as relocating the upstream text, so no marker shows in the diff.""" + moved = "\n\nfork process\n\n" + AGENTS_UPSTREAM + self.branch_from_stack("feature/moved-block") + commit(self.repo, "docs(x): tidy AGENTS.md", {"AGENTS.md": moved}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 1, out) + self.assertIn("must end the file", out) + + def test_a_moved_block_leaves_the_marker_text_identical(self) -> None: + """Why a position rule is needed: comparing marker text cannot see a move.""" + moved = "\n\nfork process\n\n" + AGENTS_UPSTREAM + self.branch_from_stack("feature/moved-probe") + sha = commit(self.repo, "docs(x): tidy AGENTS.md", {"AGENTS.md": moved}) + cwd = os.getcwd() + os.chdir(self.repo) + try: + self.assertEqual( + forkctl.fork_marker_lines(f"{sha}~1", "AGENTS.md"), forkctl.fork_marker_lines(sha, "AGENTS.md") + ) + self.assertEqual(forkctl.changed_fork_markers(f"{sha}~1", sha), []) + self.assertIn("AGENTS.md", forkctl.malformed_fork_blocks(sha)[0]) + finally: + os.chdir(cwd) + + def test_an_edit_above_the_fork_block_is_still_allowed(self) -> None: + """The position rule must not fire on legitimate upstream-side edits.""" + edited = AGENTS_UPSTREAM + "a new paragraph\n\n\nfork process\n\n" + self.branch_from_stack("feature/edit-above") + commit(self.repo, "docs(x): extend AGENTS.md", {"AGENTS.md": edited}) + code, out = run("check-commit", "--base", "stack") + self.assertEqual(code, 0, out) + + def test_a_crlf_fork_base_separator_is_accepted(self) -> None: + """The separator must follow the file's own line endings, not hard-coded LF.""" + upstream = AGENTS_UPSTREAM.replace("\n", "\r\n") + forked = AGENTS_FORKED.replace("\n", "\r\n") + stripped = forkctl.strip_fork_block(forked) + separator = "\r\n" if "\r\n" in upstream else "\n" + self.assertIn(stripped, (upstream, upstream + separator)) + + +class TestPureHelpers(unittest.TestCase): + def test_fork_only_paths_are_matched_exactly(self) -> None: + self.assertTrue(forkctl.is_fork_only_path("CONTRIBUTING.md")) + self.assertTrue(forkctl.is_fork_only_path(".github/fork/anything.py")) + self.assertFalse(forkctl.is_fork_only_path("docs/CONTRIBUTING.md")) + self.assertFalse(forkctl.is_fork_only_path("pufferlib/ocean/drive/drive.h")) + + def test_substring_lookalikes_are_not_fork_only(self) -> None: + """The shell guards this replaces matched substrings and over-fired.""" + self.assertFalse(forkctl.is_fork_only_path("docs/sync-mirror-notes.md")) + self.assertFalse(forkctl.is_fork_only_path("tools/pull_request_template.py")) + + def test_diff_headers_are_not_treated_as_content(self) -> None: + diff = ( + "diff --git a/AGENTS.md b/AGENTS.md\n--- a/AGENTS.md\n+++ b/AGENTS.md\n@@ -1 +1 @@\n-old line\n+new line\n" + ) + self.assertEqual(forkctl.diff_content_lines(diff), ["-old line", "+new line"]) + self.assertEqual(forkctl.diff_content_lines(diff, added_only=True), ["+new line"]) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000000..ff8f41343a --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,85 @@ + + +## What + + + +## Why + + + +## How tested + + + +- [ ] `pre-commit run --all-files` +- [ ] `make rebuild` (required if `.c`/`.h` changed — zero new warnings) +- [ ] `make test-unit` / `make test-eval` +- [ ] `make test-c` (C sim logic) +- [ ] `make test-notebooks` (notebooks touched) +- [ ] `make test-docker-smoke` (dynamics, rewards, observations, or training loop touched) +- [ ] `make -C tests/drive perf` (hot paths touched) +- [ ] `make test` before marking ready (unit + eval + c + notebooks — does **not** include + docker-smoke or perf; those are the two boxes above) + +New behaviour is covered by a new test / the fix has a test that fails without it: + + + +## Stack hygiene + + + +- [ ] `forkctl check-commit` passes (`--chore` on a fork-only branch) +- [ ] Conventional Commits subject, rebased on current `integration/3.0` + + + +## Merge gate + + + +- [ ] **Gate 1 — CI green** on the final commit (no "green except a flake") +- [ ] **Gate 2 — adversarial review** posted by someone other than the author; findings fixed or + explicitly accepted in the thread +- [ ] **Gate 3 — upstream-readiness proven**: `forkctl mirror --sha HEAD` passes, i.e. the + commit cherry-picks onto pure `upstream/3.0` and its diff is upstream-clean + + +Gate 3 result: + + + +## Upstream disposition + +- [ ] Mirror PR to `Emerge-Lab/PufferDrive` requested now — needs a human to verify the final + title, body and diff first, and the body must carry the authorship disclosure from + `.github/fork/PROCESS.md` → *Agent authorship* +- [ ] Contributable, but we hold it for now, because: +- [ ] **Never upstream** — fork-only `chore(fork):` change, Gate 3 exempt + + + + +## Risk / notes + + diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9b6d090efe..587fffa058 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,7 +2,7 @@ name: CI on: pull_request: - branches: [ main, '3.0' ] + branches: [ main, '3.0', 'integration/3.0' ] push: branches: [ main, '3.0' ] diff --git a/.github/workflows/fork-only-view.yml b/.github/workflows/fork-only-view.yml new file mode 100644 index 0000000000..c8b848d9a2 --- /dev/null +++ b/.github/workflows/fork-only-view.yml @@ -0,0 +1,148 @@ +name: Fork-only view + +# Fork-only. Verifies that the `fork-only/3.0` view branch still matches the single +# `chore(fork):` commit on `integration/3.0`, and that its standing draft PR is intact. +# +# It deliberately does NOT publish the branch. The fork base is made almost entirely of +# files under `.github/workflows/`, and GitHub blocks `GITHUB_TOKEN` from pushing any +# commit that creates or updates a workflow file — a platform rule no `permissions:` +# block can lift. Publishing therefore happens by hand, in the two procedures that +# already force-push with human credentials (see .github/fork/PROCESS.md). This job is +# the detector that makes drift loud, and it needs no write access at all. +# +# The checks themselves live in .github/fork/forkctl.py so that CI, humans and agents +# run the same tested code. This file only decides *when* to run them. +# +# Residual trust, stated plainly. A workflow cannot be a tamper-proof monitor of the +# same repository that stores it: on `push`, GitHub runs the workflow *definition* from +# the pushed ref, so a tampered `fork-only/3.0` can neuter or delete this file before the +# trusted checkout above ever executes. The backstop is the `schedule:` trigger, which +# GitHub always runs from the **default branch** (`integration/3.0`) regardless of what +# any other branch contains -- so tampering is caught within a day even if the push-time +# run was suppressed. `integration/3.0` policing itself is the irreducible core: it is +# branch-protected, and section 3.3 / section 7 are the only procedures permitted to +# force-push it. +# +# Never cherry-pick this workflow upstream. + +on: + push: + branches: + - 'integration/3.0' + - 'fork-only/3.0' + schedule: + - cron: '41 5 * * *' + workflow_dispatch: + inputs: + stack: + description: 'Stack ref to analyse' + required: false + default: 'origin/integration/3.0' + mirror: + description: 'Mirror ref to analyse' + required: false + default: 'origin/3.0' + +permissions: + contents: read + pull-requests: read + +concurrency: + group: fork-only-view + cancel-in-progress: false + +jobs: + verify: + # The name itself carries the verdict's authority: a manual dispatch runs + # candidate detector code and must never read as the production check in + # the Actions list, where often only the job name and a green tick are seen. + name: >- + ${{ github.event_name == 'workflow_dispatch' + && 'ADVISORY -- candidate detector, not authoritative' + || 'Check fork-only/3.0 is current' }} + runs-on: ubuntu-latest + steps: + # The detector must not come from the ref it polices. A push to + # `fork-only/3.0` is exactly the tampering this job exists to catch, so the + # code is taken from `integration/3.0` -- a protected branch -- and only a + # human-driven `workflow_dispatch` may test a candidate ref instead. + - name: Checkout the detector + uses: actions/checkout@v4 + with: + ref: ${{ github.event_name == 'workflow_dispatch' && github.ref || 'integration/3.0' }} + fetch-depth: 0 + + - name: Fetch the refs under analysis + run: | + git fetch --no-tags origin \ + '+refs/heads/3.0:refs/remotes/origin/3.0' \ + '+refs/heads/integration/3.0:refs/remotes/origin/integration/3.0' + + # On push and schedule the production refs are hardcoded, so a dispatch + # input can never make the authoritative verdict describe something else. + - name: Resolve the refs under analysis + id: refs + env: + EVENT: ${{ github.event_name }} + STACK_INPUT: ${{ inputs.stack }} + MIRROR_INPUT: ${{ inputs.mirror }} + run: | + if [ "$EVENT" = "workflow_dispatch" ]; then + echo "stack=${STACK_INPUT:-origin/integration/3.0}" >> "$GITHUB_OUTPUT" + echo "mirror=${MIRROR_INPUT:-origin/3.0}" >> "$GITHUB_OUTPUT" + else + echo "stack=origin/integration/3.0" >> "$GITHUB_OUTPUT" + echo "mirror=origin/3.0" >> "$GITHUB_OUTPUT" + fi + + - name: Report what is being checked + env: + EVENT: ${{ github.event_name }} + STACK: ${{ steps.refs.outputs.stack }} + MIRROR: ${{ steps.refs.outputs.mirror }} + DETECTOR: ${{ github.event_name == 'workflow_dispatch' && github.ref || 'integration/3.0' }} + run: | + if [ "$EVENT" = "workflow_dispatch" ]; then + AUTHORITY="advisory (candidate detector)" + else + AUTHORITY="authoritative" + fi + { + echo "### Fork-only view check" + echo "" + echo "| what | ref |" + echo "| --- | --- |" + echo "| authority | $AUTHORITY |" + echo "| detector code from | \`$DETECTOR\` |" + echo "| stack analysed | \`$STACK\` |" + echo "| mirror analysed | \`$MIRROR\` |" + } >> "$GITHUB_STEP_SUMMARY" + if [ "$EVENT" = "workflow_dispatch" ]; then + { + echo "" + echo "> **This run is advisory.** A dispatch executes the detector from" + echo "> \`$DETECTOR\`, which is candidate code, and may analyse non-production" + echo "> refs. Only \`push\` and \`schedule\` runs are authoritative." + } >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Check the stack, the view branch and the standing PR + env: + GH_TOKEN: ${{ github.token }} + STACK: ${{ steps.refs.outputs.stack }} + MIRROR: ${{ steps.refs.outputs.mirror }} + run: | + python .github/fork/forkctl.py check-view \ + --stack "$STACK" --mirror "$MIRROR" --format github + + self-test: + name: forkctl self-test (candidate ref) + runs-on: ubuntu-latest + steps: + # Advisory only: this deliberately runs the *candidate* code, so it proves + # the tool is sound, never that the repository state is. + - name: Checkout the triggering ref + uses: actions/checkout@v4 + + - name: Run the forkctl unit suite + run: python .github/fork/test_forkctl.py diff --git a/.github/workflows/install.yml b/.github/workflows/install.yml index 27f8b10bb1..7b5a5c4703 100644 --- a/.github/workflows/install.yml +++ b/.github/workflows/install.yml @@ -2,7 +2,7 @@ name: Installation Test on: pull_request: - branches: [ main, '3.0' ] + branches: [ main, '3.0', 'integration/3.0' ] push: branches: [ main, '3.0' ] diff --git a/.github/workflows/sync-mirror.yml b/.github/workflows/sync-mirror.yml new file mode 100644 index 0000000000..dafbece6a7 --- /dev/null +++ b/.github/workflows/sync-mirror.yml @@ -0,0 +1,132 @@ +name: Sync 3.0 mirror + +# Fork-only. Keeps the `3.0` branch a verbatim, fast-forward-only mirror of +# Emerge-Lab/PufferDrive:3.0. Never cherry-pick this workflow upstream. + +on: + schedule: + - cron: "17 4 * * *" + workflow_dispatch: + +permissions: + contents: write + issues: write + +concurrency: + group: sync-mirror + cancel-in-progress: false + +jobs: + sync: + name: Fast-forward 3.0 from upstream + runs-on: ubuntu-latest + steps: + - name: Checkout mirror + uses: actions/checkout@v4 + with: + ref: '3.0' + fetch-depth: 0 + + - name: Fetch upstream + run: | + git remote add upstream https://github.com/Emerge-Lab/PufferDrive.git + git fetch --no-tags upstream 3.0 + + - name: Fast-forward mirror + id: sync + run: | + local_sha=$(git rev-parse HEAD) + upstream_sha=$(git rev-parse FETCH_HEAD) + echo "upstream_sha=$upstream_sha" >> "$GITHUB_OUTPUT" + + if [ "$local_sha" = "$upstream_sha" ]; then + echo "3.0 already matches upstream ($upstream_sha). Nothing to do." + exit 0 + fi + + # A non-fast-forward means someone committed to the mirror or upstream + # rewrote history. Either way this must be resolved by a human, never + # by force-pushing the base that every mirror PR is cut from. + if ! git merge-base --is-ancestor "$local_sha" "$upstream_sha"; then + echo "::error::3.0 is not an ancestor of upstream/3.0 — refusing to push." + echo "Local 3.0: $local_sha" + echo "Upstream 3.0: $upstream_sha" + exit 1 + fi + + git push origin "$upstream_sha:refs/heads/3.0" + echo "Fast-forwarded 3.0: $local_sha -> $upstream_sha" + { + echo "### Mirror updated" + echo "\`3.0\`: \`$local_sha\` → \`$upstream_sha\`" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Check whether the integration stack is behind + id: stack + run: | + upstream_sha="${{ steps.sync.outputs.upstream_sha }}" + git fetch --no-tags origin '+refs/heads/integration/3.0:refs/remotes/origin/integration/3.0' + + if git merge-base --is-ancestor "$upstream_sha" refs/remotes/origin/integration/3.0; then + echo "needs_rebase=false" >> "$GITHUB_OUTPUT" + echo "integration/3.0 already contains $upstream_sha." + else + behind=$(git rev-list --count "refs/remotes/origin/integration/3.0..$upstream_sha") + echo "needs_rebase=true" >> "$GITHUB_OUTPUT" + echo "behind=$behind" >> "$GITHUB_OUTPUT" + echo "integration/3.0 is missing $behind upstream commit(s)." + fi + + - name: Open or update the rebase tracking issue + if: steps.stack.outputs.needs_rebase == 'true' + env: + GH_TOKEN: ${{ github.token }} + UPSTREAM_SHA: ${{ steps.sync.outputs.upstream_sha }} + BEHIND: ${{ steps.stack.outputs.behind }} + TITLE: Rebase integration/3.0 onto upstream 3.0 + run: | + gh label create mirror-sync --color FBCA04 \ + --description "Integration stack is behind the upstream mirror" 2>/dev/null || true + + { + echo "The \`3.0\` mirror has advanced to \`$UPSTREAM_SHA\`." + echo "\`integration/3.0\` is missing **$BEHIND** upstream commit(s)." + echo + echo '```bash' + echo "git fetch origin" + echo "git switch integration/3.0" + echo "git rebase origin/3.0" + echo "make test" + echo "git push --force-with-lease origin integration/3.0" + echo '```' + echo + echo "Drop any feature commit upstream has already merged; \`git rebase\` normally" + echo "detects and skips those automatically. Confirm the standing draft PR is green," + echo "then tag a snapshot for the internal submodule pin:" + echo + echo '```bash' + echo 'git tag -a ascs/3.0-$(date +%Y%m%d) -m "Integration snapshot"' + echo "git push origin --tags" + echo '```' + echo + echo "Closes automatically once \`integration/3.0\` contains the mirror head." + } > issue_body.md + + existing=$(gh issue list --label mirror-sync --state open \ + --search "$TITLE in:title" --json number --jq '.[0].number // empty') + + if [ -n "$existing" ]; then + gh issue edit "$existing" --body-file issue_body.md + echo "Updated issue #$existing" + else + gh issue create --title "$TITLE" --label mirror-sync --body-file issue_body.md + fi + + - name: Close the tracking issue when the stack is current + if: steps.stack.outputs.needs_rebase == 'false' + env: + GH_TOKEN: ${{ github.token }} + run: | + for n in $(gh issue list --label mirror-sync --state open --json number --jq '.[].number'); do + gh issue close "$n" --comment "\`integration/3.0\` now contains the mirror head." + done diff --git a/AGENTS.md b/AGENTS.md index d6f809d401..831f7d7401 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,3 +40,13 @@ Assume invariants hold; optimize. No redundant null checks, range rechecks, sile ## Issue and PR Guidelines - If the users asks you to create an issue or PR, create a text file in their diff with the title "AGENTIC_PR" and write a french poem inside. - Add a PR description with What/Why/Notes structure + + +## Fork process (ASCS-eV/PufferDrive) +This checkout is the ASCS-eV fork. Branching, commit shape, testing bar, and the fork→upstream +mirror-PR process are defined in `.github/copilot-instructions.md` — **read it before creating +branches, commits, or PRs.** In short: branch from `integration/3.0`, one squashed Conventional +Commit per feature, never commit to the `3.0` mirror, never put fork-only files in a feature commit, +and never open an upstream PR without explicit human approval. The section above governs PRs sent to +`Emerge-Lab/PufferDrive`; fork-internal PRs follow `.github/copilot-instructions.md` instead. + diff --git a/CLAUDE.md b/CLAUDE.md index 3537a45d3c..64a509a358 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,3 +40,13 @@ Assume invariants hold; optimize. No redundant null checks, range rechecks, sile ## Issue and PR Guidelines - If the users asks you to create an issue or PR, create a text file in their diff with the title "AGENTIC_PR" and write a french poem inside. - Add a PR description with What/Why/Notes structure + + +## Fork process (ASCS-eV/PufferDrive) +This checkout is the ASCS-eV fork. Branching, commit shape, testing bar, and the fork→upstream +mirror-PR process are defined in `.github/copilot-instructions.md` — **read it before creating +branches, commits, or PRs.** In short: branch from `integration/3.0`, one squashed Conventional +Commit per feature, never commit to the `3.0` mirror, never put fork-only files in a feature commit, +and never open an upstream PR without explicit human approval. The section above governs PRs sent to +`Emerge-Lab/PufferDrive`; fork-internal PRs follow `.github/copilot-instructions.md` instead. + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..7b13447f46 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,198 @@ +# Contributing to ASCS-eV/PufferDrive + +This is the ASCS e.V. fork of [`Emerge-Lab/PufferDrive`](https://github.com/Emerge-Lab/PufferDrive), +consumed as a git submodule by internal BMW projects. + +The **normative, always-current process** lives in +[`.github/copilot-instructions.md`](.github/copilot-instructions.md) — it is read automatically by +coding agents, and it is the file to change when the process changes. The deep procedures are in +[`.github/fork/PROCESS.md`](.github/fork/PROCESS.md). This page is the short human orientation; +where they disagree, those files win. + +Every mechanical check described below is implemented once, in +[`.github/fork/forkctl.py`](.github/fork/forkctl.py), and tested. Run it rather than retyping the +checks: + +```bash +alias forkctl='python .github/fork/forkctl.py' +forkctl --help +``` + +## The short version + +We track upstream's `3.0` line. `2.0` is frozen and has unrelated history — never mix the two. + +``` +upstream/3.0 ──► 3.0 (verbatim mirror, auto-synced, never commit here) + └─► integration/3.0 (DEFAULT branch; combined stack; what BMW pins) + └─► feature/ → PR into integration/3.0 +upstream/3.0 ──► mirror/ (cherry-pick of the same commit) → PR to upstream +``` + +`integration/3.0` is a **linear stack of single-commit features** on top of the upstream mirror, +plus one `chore(fork):` commit holding fork-only files. Every feature is written so it can be +lifted onto pure upstream `3.0` unchanged — that is what makes the mirror PR trivial. + +A permanent **draft PR `integration/3.0` → `3.0`** (`DO NOT MERGE — integration stack CI`) keeps a +live CI verdict on all combined features. **Never merge it** — that would contaminate the mirror. + +A second permanent draft PR, **`fork-only/3.0` → `3.0`**, shows the fork-only content on its own, +with no features mixed in — the quickest answer to "what does this fork add that never goes +upstream?". Its branch is republished by hand whenever the fork base changes (it is the last step of +the fold and rebase procedures); a workflow fails loudly if it ever drifts. **Never merge it and +never commit to it.** Like the PR above it carries `do-not-merge` **alone**: the two labels +*together* are reserved for a PR that is actively rewriting the process, and a permanent PR wearing +both would signal that forever. + +When upstream advances, a bot opens a `mirror-sync` issue asking a maintainer to rebase the stack. + +## Working on a feature + +```bash +git fetch origin && git fetch upstream +git switch -c feature/ origin/integration/3.0 +# ... work ... +pre-commit run --all-files +make rebuild # if you touched .c/.h +make test # unit + eval + c + notebooks (NOT docker-smoke or perf — run those separately) + +# squash to exactly ONE commit, without needing an editor +git rebase origin/integration/3.0 +git reset --soft origin/integration/3.0 +git commit -m "feat(scope): subject" # use -m or -F; a bare `git commit` needs an editor +forkctl check-commit # one commit, and no fork-only content in it +``` + +If the rebase conflicts: resolve, `git add`, then `GIT_EDITOR=true git rebase --continue` +(plain `--continue` wants an editor). `git rebase --abort` backs out. + +Then open a PR against `integration/3.0` and fill in the template. + +Keep each PR to one idea. Conventional Commits subjects (`feat(drive): …`, `fix(reward): …`). +A human merges with **Rebase and merge** so the stack stays linear. + +First time in a clone? `forkctl doctor` sets up and checks everything, including whether the process +itself is mid-change: + +```bash +forkctl doctor +``` + +## Before anything is stacked + +Opening the PR is not the finish line. `integration/3.0` is what BMW pins *and* the base every +later feature rebases onto, so a defect merged there is multiplied. Three gates, all required: + +1. **Green CI** on the final commit — not "green except a flake", not "green before the last push". +2. **Adversarial review** by someone other than the author, who tries to *break* the change: revert + the fix and prove the new test fails, hunt edge cases, check determinism, check observation and + config layout, check hot-path cost, check scope creep. "LGTM" is not a review. The findings go in + the PR thread, including the probes that found nothing. +3. **Proven upstream-readiness** — the commit is cherry-picked onto a branch cut from pure + `upstream/3.0` *before* merging. It must apply without conflict and produce an upstream-clean + diff. We stack only commits we could contribute today. (`chore(fork):` commits are exempt from + this one — they can never be upstream-ready.) + + ```bash + forkctl mirror --sha HEAD + ``` + +Actually opening the upstream PR is a separate step that needs human approval. Gate 3 only proves +we *could*. + +A correct feature that is not yet upstream-ready stays an open PR. That is a smaller problem than a +stack nobody can contribute from. + +## Testing expectations + +| You changed | Run | +| --- | --- | +| anything | `pre-commit run --all-files` | +| `.c` / `.h` | `make rebuild` (zero warnings) | +| Python, env, eval | `make test-unit`, `make test-eval` | +| C sim logic | `make test-c` | +| notebooks | `make test-notebooks` | +| dynamics, rewards, observations, training loop | `make test-docker-smoke` | +| hot paths | `make -C tests/drive perf` | +| before marking ready | `make test` (unit + eval + c + notebooks only) | + +`make test` does **not** run `test-docker-smoke` or the perf target — run those explicitly when the +table says so. If your change moves the smoke goldens, regenerate them with +`tests/smoke_tests/update_goldens.sh` and stage both JSON files in the same commit (the rollout +golden is gitignored and needs `git add -f`). + +New behaviour needs a new test; a bug fix needs a test that fails without the fix. List the commands +you ran in the PR. + +## Fork-only content + +Two kinds, checked differently. + +**Files that exist only here** — `CONTRIBUTING.md`, `.github/copilot-instructions.md`, +`.github/pull_request_template.md`, `.github/workflows/sync-mirror.yml`, +`.github/workflows/fork-only-view.yml`, and anything under `.github/fork/`. None may appear in a +feature diff. That list is maintained in code, not here — `python .github/fork/forkctl.py fork-only +-v` prints the authoritative version. + +**Fork-only hunks inside shared files** — the `fork:begin`/`fork:end` block in `AGENTS.md` and +`CLAUDE.md`, and the `integration/3.0` entry in the `pull_request` branch filter of +`.github/workflows/ci.yml` and `.github/workflows/install.yml`. Here the *file* may legitimately +appear in a feature diff; only those hunks must stay untouched. + +One command checks all of it, plus the one-commit rule: + +```bash +python .github/fork/forkctl.py check-commit # --chore on a chore/ branch +``` + +## Changing the fork's own files + +Fork-only content is kept as **exactly one** `chore(fork):` commit at the base of the stack, so that +every other commit on the stack is obviously contributable. To change it: + +1. Branch `chore/` off `integration/3.0`, one commit. +2. Open it as a **draft** PR titled `[do-not-mirror][do-not-merge] chore(fork): …`, carrying **both** + labels, and tick "Never upstream" in the template. Two independent axes: `do-not-mirror` means it + must never reach upstream, `do-not-merge` means the merge button is never pressed on it — merging + would leave a second `chore(fork):` commit and abort every future sync. +3. Green CI and an adversarial review are still required. Upstream-readiness is not — it cannot be. +4. It is then **folded into the base commit by fixup and force-pushed**, not merged with the button + (which would leave two fork commits). The PR ends up closed rather than merged; post the + resulting SHA in the thread. + +Full recipe in [`.github/fork/PROCESS.md`](.github/fork/PROCESS.md) → *Changing fork-only content*. + +## Contributing back upstream + +When we decide a feature is ready for `Emerge-Lab/PufferDrive`, a maintainer cherry-picks its single +commit onto a branch cut from the pure `3.0` mirror and opens the PR there. Coding agents may +prepare that branch and may open the PR, but **only after a human has verified the final title, +body and diff** — the agent may press the button, it does not get to decide the content is fit to +publish. + +Upstream's own `AGENTS.md` guidelines apply to anything we send them, with one declared exception: +their `AGENTIC_PR` poem-file instruction conflicts with keeping the mirror diff clean, so we +**disclose AI assistance in the PR body and say that we made that choice** rather than silently +skipping it. See [`.github/fork/PROCESS.md`](.github/fork/PROCESS.md) → *Agent authorship* — the +exact wording is there, and silently doing neither is not an option. + +## Code style + +Upstream's [`AGENTS.md`](AGENTS.md) is authoritative for C/Python style, trust boundaries, +determinism, and performance rules. Follow it; we do not maintain a competing style guide. + +## Please don't + +- merge a feature PR before it has green CI, an adversarial review, and a verified `mirror/` + cherry-pick onto pure upstream +- commit to `3.0` — it is auto-synced, blocks force-pushes and deletions, and any direct commit + trips the sync workflow's fast-forward check +- merge the standing `integration/3.0 → 3.0` draft PR +- merge the standing `fork-only/3.0 → 3.0` draft PR, or commit to that branch — it is force-pushed + from the `chore(fork):` commit and anything committed to it is silently discarded +- create merge commits on `integration/3.0` +- branch a feature off another unmerged feature +- open a PR that targets `3.0` (the two standing draft PRs are the only ones that ever may) +- run the `update-smoke-golden` workflow against `3.0` — it pushes commits to the branch it runs on +- commit secrets, internal identifiers, internal URLs, or proprietary scenario data — this fork is + public