fix: release lane fails, runs in series, and posts malformed issues #694

What happened?

The release lane tried v7.14.0 at v7-next in the test repository, and a scenario failed. Nothing was tagged and v7 did not move.

• accepted-spec: no label autopilot; no comment with (https://github.com/DiluxOne/diluxone-test/actions/runs/42419682452)
• spec-to-draft: no comment with ; the pull request does not say Closes Jobs waiting for the organisation's VM #230; the pull request's branch is not ^feat/[0-9]+-; the pull request's title is not ^feat((.+))?: (https://github.com/DiluxOne/diluxone-test/actions/runs/42421347374)
• prd-edited-after-acceptance: label accepted still there; waiting is False (https://github.com/DiluxOne/diluxone-test/actions/runs/42423484033)
• decision-sub-issues: no label accepted (https://github.com/DiluxOne/diluxone-test/actions/runs/42425205919)
• claim-pull-request: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• bug-reproduced: no label bug:confirmed; the pull request does not say Closes Auto-merge can merge a head pushed after its decision #237; the pull request's branch is not ^fix/[0-9]+-; the pull request's title is not ^fix((.+))?: (https://github.com/DiluxOne/diluxone-test/actions/runs/42426875979)
• question-answered: the pull request does not say Closes feat(review): the count first, and pre-existing findings never block #238 (https://github.com/DiluxOne/diluxone-test/actions/runs/42428751004)
• stop-hint-continue: the pull request does not say Closes feat(digest): what became of the review's findings, and its noise #239 (https://github.com/DiluxOne/diluxone-test/actions/runs/42430741616)
• draft-in-conflict: gh api -X PUT: gh: Branch scenario/draft-in-conflict not found (HTTP 404) (no run)
• ready-behind-main: gh api -X PUT: gh: Branch scenario/ready-behind-main not found (HTTP 404) (no run)
• merge-queue: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• low-risk-merges: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• release-issue: the pull request's title is not ^chore: release (https://github.com/DiluxOne/diluxone-test/actions/runs/42433587271)
• bug-not-reproduced: no label bug:unconfirmed (https://github.com/DiluxOne/diluxone-test/actions/runs/42419682452)
• review-changes-asked: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• review-replied: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• fork-without-secrets: gh repo fork DiluxOne/diluxone-test: failed to fork: HTTP 403: Resource not accessible by personal access token (https://api.github.com/repos/DiluxOne/diluxone-test/forks) (no run)
• high-risk-waits: gh api repos/DiluxOne/diluxone-test/git/refs -f: gh: Reference already exists (HTTP 422) (no run)
• play: a play job ended in failure: a step that broke, or its play step past its timeout of 240 minutes (timeout-minutes in release-lane.yml) (https://github.com/DiluxOne/.github/actions/runs/42419485251)

The lane's run: https://github.com/DiluxOne/.github/actions/runs/42419485251 (22:12 to 01:37 UTC, 3 h 25 min: the serial group alone played 17 scenarios one after another).

Problem

The release lane cannot release v7.14.0, and when it fails it takes hours to say so. Three things are wrong today. First, a scenario leaves its scenario/<id> branch behind, and the next run fails on it before testing anything (HTTP 422 "Reference already exists", 7 scenarios here; 2 more fail with 404 on a branch they expected). Second, the serial group plays its 17 scenarios one after another in one job, because they share the test repository's one core (scenarios/), with fixed waits of 5 and 10 minutes and a failing scenario waiting out its 20 minutes: the run took 3 h 25 min while the organisation has hundreds of runners idle. Third, the bug the lane opens (this one) is born incomplete: it does not follow the issue schema, so it waits on a person instead of being fixed. On top of that, the scenarios that did run show the candidate or the scenarios' fixtures out of step: issues the scenarios open stay incomplete (the bug bodies use ### What happened and ### How to reproduce, not the report's sections), no dx-prd-seal, no accepted, pull requests without Closes #, and fork-without-secrets fails with 403 because the test token cannot fork.

Scope

• Every scenario deletes its own leftover scenario/<id> branch (and closes its leftover issues and pull requests) before it starts, and cleans them after.
• Every scenario gets its own core in the test repository (scenarios/<id>/), so none waits on another's change; the serial group goes away and every scenario plays in its own job, at once.
• Fixed waits become polling (every 15 seconds) for the state they wait for, and a scenario stops as soon as an expectation can no longer be met.
• The bug the lane opens follows the issue schema (report, Problem, Scope, Out of scope, Acceptance criteria, Criticality), so it passes G1 and goes the reproduced-bug way.
• The scenarios' bodies use the sections the schema knows; each failing scenario above passes, or its cause in the engine is fixed.
• fork-without-secrets uses a token that may fork (or the scenario says which permission it lacks and is fixed with it).
• The play step's and job's timeouts come down to what the parallel suite needs.
• The same for every other workflow of this repository and its reusable workflows: a job that waits in series on another it does not need runs at once; a fixed sleep becomes polling; a job that cannot be needed is skipped before it starts a runner.
• Every issue, pull request and comment the engine writes (the lane's bugs, the release issue, sub-issues, the triage's rewrites, the scenarios' fixtures) is checked against its schema or convention before it is posted, by the script that checks a person's, so none is born incomplete or with a wrong title, branch or Closes #.
• A bug the engine opens itself (the release lane's dx-release-lane, the watch's, like the nightly's dx-nightly-finding today) carries the Type Bug and goes through the triage and the reproduction: issue-triage.yml skips every issue a bot opens (user.type != 'Bot'), so this one got no labels and no reproduction.
• The autopilot's stop and its answer hold together with the gates: a G2 "Acceptance withdrawn" comment (or any comment of the bot after the stop) never hides the stop from review-reply.yml, an accepted added again after a stop takes needs-human off and the autopilot goes on, and the autopilot reads the answers of those who decide in the issue's comments with the PRD, so it never asks again what was answered.
• Whole paths, not single events: a table in docs/ maps every path docs/model.md describes (a person's bug to its merge, a reproduced bug merged by policy, a feature accepted and built, a claim and its release, a stop answered, an edit after acceptance and the new acceptance, the release and its watch) and each setting that changes it (acceptance by policy, who-decides, a claim, merge on its own or by a person, its ceiling) to the scenario that plays it; every row has one, and release-scenarios.py's --test fails on a row without one.
• No model call is spent twice on the same question by an automatic re-run, and each run of the triage, the autopilot and the review says in its summary the runner minutes and model tokens it spent, so the cost is measured.

Out of scope

Changing the gates or what they check. Rewriting main to correct the message of #684's commit. Tagging v7.14.0 by hand.

Acceptance criteria

C1: Given a scenario/<id> branch left by an earlier run, when the lane plays that scenario, then it deletes the branch first and the scenario does not fail with 422.
C2: Given tests/release/scenarios.yml, when release-scenarios.py groups it, then every scenario is its own group and no two scenarios write to the same core path.
C3: Given tests/release/scenarios.yml, when release-scenarios.py plans it, then no scenario has a fixed wait (every wait polls its state every 15 seconds) and each scenario's planned limit is 15 minutes or less, which its --test holds.
C4: Given a scenario fails, when the lane opens its bug, then python3 scripts/issue-schema.py check passes on the bug's body.
C5: Given release-lane.yml, autopilot.yml, org-pull-request.yml and issue-repro.yml, when a job's needs is read, then each entry is either used (its outputs or its result) or carries a comment # order: <why> that names the gate or the order it keeps; a script's --test checks every workflow of this repository for it, and the needs that are neither are removed.
C6: Given an automatic re-run of the triage or the review (not a person's re-run) on the same input (the head SHA for a pull request, the body's hash for an issue) with the same prompt and policy version, when it runs, then it reuses the earlier answer and calls no model; a person's re-run calls the model. Every run of the triage, the autopilot and the review writes its runner minutes and model tokens in its step summary. Tests hold both.
C7: Given any issue or pull request the engine opens, when it is posted, then issue-schema.py check and the conventions check pass on it, and a test per writer holds it.
C8: Given the release lane opens a bug, when it is opened, then its Type is Bug and the triage and the reproduction run on it as on a person's bug.
C10: Given an issue whose autopilot stopped, when its PRD is edited, accepted again and a person who decides answers in a comment, then the autopilot goes on without autopilot:continue and does not ask again what the answer settled; a scenario plays it.
C11: Given the table of paths in docs/, when release-scenarios.py --test runs, then every path and setting in it names a scenario of tests/release/scenarios.yml, and every one of those scenarios exists.
C9: Given the merge of this issue's last pull request, when the release lane runs on v7.14.0's candidate, then every scenario passes, the run takes 20 minutes or less from Plan to "Release, bug or issue", and the lane tags v7.14.0 and moves v7; that run, linked here, is the evidence, and a scenario that fails in the engine is fixed under this issue.

Criticality

Critical

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

Spec: approach

The release lane (tests/release/scenarios.yml, scripts/release-scenarios.py, release-lane.yml):

• play first deletes its own scenario/<id> branch (a 404 is fine) and closes the issues and pull requests a scenario left behind (found by a dx-scenario: <id> HTML-comment mark in each body it opens), then cleans them again after. A scenario that expects a branch makes it itself instead of assuming it exists (the two 404s).
• Each scenario gets its own core: the engine-test pack's core becomes scenarios/*/** and fixed-change.file becomes scenarios/{scenario}/{n}.md. {scenario} comes from the issue's mark, and org.py core_of and the core wait compare the first directory under scenarios/. Only issues of the same scenario share a core, so next_step:wait:core still waits (its two issues are one scenario's). Then serial:, the serial group and its checks in partition go away: groups gives one group per scenario, and partition fails when two scenarios could write to the same core path.
• {wait: N} steps are replaced by {until: <expectation>}, polled every 15 s. holds grows a hopeless verdict (for example a label we expect gone is still there after the job that removes it has finished, or an absent label has appeared), so a scenario stops as soon as it can no longer pass. The plan holds a limit per scenario of 15 min or less, and --test checks it. The play step's timeout becomes 20 min and the job's 30.
• The bug the lane opens (decide, act in release-lane.py) is written in the schema's sections: report, Problem, Scope, Out of scope, Acceptance criteria, Criticality. It is created with gh api with type=Bug and its dx-release-lane mark. The scenarios' bodies use the report's sections, and the bodies the engine writes carry Closes #. Any scenario that still fails gets its engine cause fixed under this issue.
• fork-without-secrets forks with a new secret DX_TEST_FORK_TOKEN, a test-account token that may fork the test repository. The scenario names that permission when the secret is missing.

The engine's own writes: a new scripts/engine-post.py (check issue|pull) runs issue-schema.py check and the conventions check (title, branch, Closes #) on a body before any writer posts it: release-lane.py, release-issue.py, nightly-review.py, the autopilot's PR and sub-issues, the triage's rewrites and the scenarios' fixtures. Each writer's --test feeds its output through it. issue-triage.yml (line 534's condition) and issue-repro.yml (line 304) treat a bot issue that starts with the dx-release-lane mark (and the watch's mark) the way they treat dx-nightly-finding, so the triage and the reproduction run on it.

Workflows: needs-chain.py gets a second check. Each needs entry must be used (needs.X.outputs or needs.X.result in the job), or carry # order: <why>. Needs that are neither are removed across .github/workflows/. A job that cannot be needed gets a job-level if: before it starts a runner. sleep N loops waiting for a state become 15-s polling.

Cost: a new scripts/model-cache.py. Its key is the head SHA (pull request) or the body's hash (issue), plus the prompt's hash and the policy version. On an automatic re-run (github.run_attempt > 1 and github.triggering_actor is the bot, or the event's sender is the bot) with a stored answer, it reuses that answer and skips the model step. A person's re-run always calls the model. model-cache.py summary writes the runner minutes (from the job start time) and the model tokens (from the action's execution file) to $GITHUB_STEP_SUMMARY in issue-triage.yml, autopilot.yml and claude-review.yml.

Options for the maintainer (where the cached answer lives):

• (A, recommended) An artifact of the earlier run, found by the run id and the key, saved only on the earlier run. It touches nothing on GitHub that people read, and it expires with the run's retention.
• (B) The Actions cache, keyed by the key. Simpler, but any workflow on the default branch can read and write it, and an evicted entry turns into a paid call.

Widened: one new secret (DX_TEST_FORK_TOKEN, a token that may fork in the test repository). The triage and the reproduction now run on the lane's own bot bugs. Nothing else widens.

Spec: files

• scripts/release-scenarios.py, tests/release/scenarios.yml, kinds/engine-test/pack.yml, scripts/org.py
• scripts/release-lane.py, scripts/release-issue.py, scripts/nightly-review.py, scripts/autopilot.py, scripts/triage-act.sh
• scripts/engine-post.py (new), scripts/model-cache.py (new), scripts/needs-chain.py
• .github/workflows/release-lane.yml, issue-triage.yml, issue-repro.yml, autopilot.yml, claude-review.yml, org-pull-request.yml, and every other workflow under .github/workflows/ whose needs or sleeps change
• docs/architecture.md, docs/testing-and-quality.md, kinds/README.md, CONTRIBUTING.md (what the lane, the needs check and the cost summary do)

Spec: tests

• C1 → scripts/release-scenarios.py (--test: play deletes a leftover scenario branch first, and a 422 does not happen)
• C2 → scripts/release-scenarios.py (--test: groups is one per scenario, and no two scenarios share a core path)
• C3 → scripts/release-scenarios.py (--test: no fixed wait, polling every 15 s, each planned limit ≤ 15 min)
• C4 → scripts/release-lane.py (--test: the bug body passes issue-schema.py check)
• C5 → scripts/needs-chain.py (--test, and run on every workflow of the repository)
• C6 → scripts/model-cache.py (--test: an automatic re-run reuses the answer, a person's re-run calls the model, the summary has minutes and tokens)
• C7 → scripts/engine-post.py (--test, one case per writer), plus each writer's own --test
• C8 → scripts/release-lane.py (--test: Type Bug and the mark; check_workflow holds the triage and repro conditions)
• C9 → the release lane's run on v7.14.0's candidate after the merge, linked on fix: release lane fails, runs in series, and posts malformed issues #694; scripts/release-scenarios.py --test holds the limits it relies on