Skip to content

fix: release candidate v7.14.0 fails scenario accepted-spec - #714

Merged
soydiloreto merged 9 commits into
mainfrom
fix/713-release-candidate-v7-14-0
Nov 10, 2026
Merged

soydiloreto merged 9 commits into
mainfrom
fix/713-release-candidate-v7-14-0

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

C1, a cancelled check is not a failed one. Today waiting.py blocker reads only statusCheckRollup.state, and GitHub folds a CANCELLED check run into FAILURE. The query (FIELDS and ONE) will also read the head's contexts(first: 100) (CheckRun name conclusion status, StatusContext context state). A pure failed_checks(rollup) decides "a failed check" from the contexts. When the rollup says FAILURE/ERROR but no context failed, the state is taken as pending or green, whichever the other contexts say. I will search auto-merge.yml and the scripts it calls (and autopilot.py's ready/merge paths) for any read of the rollup's state, and each one I find gets the same rule. GitHub's native auto-merge waits only on required checks, so no ruleset or gate changes (out of scope). There are two ways to say which cancelled runs don't count, and the maintainer picks:

  • (a) Recommended. No CANCELLED check run is a failure, the way scripts/needed-a-person.py FAILED already reads checks (FAILURE, TIMED_OUT, STARTUP_FAILURE). This needs no extra read. A superseded run is replaced by its newer run, and a run a person cancelled is no failure of the code.
  • (b) Only a CANCELLED check run whose annotation says its concurrency group cancelled it ("Canceling since a higher priority waiting request … exists") is skipped, and any other cancellation still counts as failed. This is closer to the PRD's wording, but it reads annotations(first: 5) { message } per check run and depends on GitHub's message text.

C2, waiting for the autopilot's result. play already polls expect until the scenario's limit. The failing rows show that the autopilot had not finished within 10 minutes, or that it saw an issue opened already accepted as "unsealed" (ready-behind-main, stop-hint-continue). Each scenario whose expect or until needs the autopilot's result (a dx-prd-seal comment, the autopilot or autopilot:core-wait label, a pull request that closes its issue) will change in three ways:

  • It opens its issue without accepted.
  • It waits until: {labels: [awaiting-acceptance]}, the triage's verdict, then takes label: accepted. The acceptance then seals the triaged body, as a maintainer's would.
  • It waits until the seal (the dx-prd-seal marker comment), then until the autopilot's result, before its own steps and expect. Its limit is minutes: 15, the most plan allows.

This applies to accepted-spec, spec-to-draft, prd-edited-after-acceptance, stop-hint-continue, core-one-at-a-time (its before issue too), ready-behind-main, question-answered and draft-in-conflict. A new pure autopilot_problems(scenarios), run by coverage and --test, names every scenario of that kind that opens its issue accepted, or that has no until between its label: accepted and its expect.

C3, claim before acceptance. claim-and-release opens its issue plain, with the steps comment: /claim, until claimed, label: accepted, comment: /unclaim. It still covers next_step:wait:claim. autopilot_problems also refuses a scenario whose /claim step comes after its accepted.

C4, a writer's release. release-issue expects what the test identity (role write) gets: the comment "No release pull request:" (scripts/release-issue.py's refusal) and no pull request. A pure check in --test matches the refusal's text in release-issue.py, so the two cannot drift apart.

C5, complete bodies. bug and bug-not-reproduced gain the report form's ### Version (bug_report.yml requires What happened?, Steps to reproduce and Version). decision becomes a full PRD (Problem, Scope, Out of scope, Acceptance criteria, Criticality) plus a ### Repositories section naming the test repository. decision-sub-issues opens it plain, waits until awaiting-acceptance, and then accepts it. fixture_problems stops exempting decision. It also checks each Bug-typed body against the report form's required fields, read from bug_report.yml.

C6 is the release lane itself, run again on v7.14.0's candidate with the fix.

The comments in scenarios.yml and the header of release-scenarios.py will describe these waits. This widens no permission, secret, lane or limit; the scenario limit stays at 15 minutes or less.

💡 Why

Closes #713. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given a pull request whose only non-green check run is one cancelled by its concurrency group, when the waiting view and auto-merge read its checks, then they do not say "fix the failed check" and auto-merge is not held by it, tested in scripts/waiting.py --test.
    Test: scripts/waiting.py --test (fixtures/waiting.json: a pull request whose only non-green check run is the cancelled "What waits" view)
  • C2 Given a scenario whose expectations need the autopilot's result (a seal, the autopilot label, a pull request), when release-scenarios.py checks them, then it waits for the autopilot's run with until instead of the first run after the step, tested in release-scenarios.py --test from tests/release/scenarios.yml.
    Test: scripts/release-scenarios.py --test (autopilot_problems over tests/release/scenarios.yml, and a fake that only seals after the triage)
  • C3 Given claim-and-release, when it plays, then /claim is commented before the issue is accepted, tested in release-scenarios.py --test.
    Test: scripts/release-scenarios.py --test (claim-and-release's /claim comes before accepted)
  • C4 Given release-issue played by an identity with the write role, when it checks the issue, then it expects the refusal for a writer, tested in release-scenarios.py --test.
    Test: scripts/release-scenarios.py --test (release-issue expects release-issue.py's refusal for a writer)
  • C5 Given the bodies of decision-sub-issues and bug-not-reproduced, when issue-schema.py checks them, then they pass, and the bug body has every section of the report form, tested in release-scenarios.py --test.
    Test: scripts/release-scenarios.py --test (fixture_problems: decision and bug bodies pass issue-schema.py, and the bug bodies have every required field of bug_report.yml)
  • C6 Given v7.14.0's candidate with the fix, when the release lane plays every scenario in the test repository, then each passes and the lane releases it.
    Test: .github/workflows/release-lane.yml (the lane's run on v7.14.0's candidate)

🧭 Decisions

  • 2026-11-09: C1: option (a), the Spec's recommended one: no CANCELLED check run is a failed one (FAILURE, TIMED_OUT, STARTUP_FAILURE, as needed-a-person.py reads them), with no read of annotations; a rollup whose contexts were not all read keeps GitHub's state. auto-merge.yml reads no rollup state (GitHub's native auto-merge waits only on required checks), so only waiting.py changes.
  • 2026-11-09: C4: the refusal a writer gets is said by .github/workflows/issue-accepted.yml (its release job), not scripts/release-issue.py as the Spec put it, so the self-test matches the expected words against that line of issue-accepted.yml.
  • 2026-11-10: C5: decision-sub-issues expects what a decision gets in the test repository, accepted and its seal, waiting until the seal within 15 minutes, with no sub-issue comment (the maintainer's option 1); issue-accepted.yml is not widened.

🙌 Credits

🧪 How I tested it

  • python3 scripts/waiting.py --test: C1, from fixtures/waiting.json. A pull request whose only non-green check run is the cancelled "What waits" view gets no "fix the failed check", and a failed, timed-out or erroring check still counts as failed.
  • python3 scripts/release-scenarios.py --test and coverage over the real tests/release/scenarios.yml: C2 to C5. autopilot_problems finds nothing, and a fake repository seals only after the triage. claim-and-release comments /claim before accepted. release-issue expects the writer's refusal. decision-sub-issues expects accepted and its seal, as the maintainer answered. The decision and bug bodies pass issue-schema.py and the bug report form's required fields.
  • Every python3/bash scripts/… command of pull-request.yml's scripts job, run locally (the repository has no make pre-pr). All pass except plugin-i18n.sh, design-lint.py, review-brief.sh and runner-vm/dx-runner-slot.sh, which need gettext, stylelint, a pull request's environment or docker. They fail the same way on the base commit 33002d5.
  • Not run here: C6 itself, the release lane's run on v7.14.0's candidate in the test repository. Its next run, once this is on v7-next, is the proof.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added autopilot Opened by the autopilot from an accepted issue autopilot:run-1 The autopilot's runs on this pull request labels Nov 9, 2026
@dilux-bot

dilux-bot Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 9, 2026
dilux-bot Bot added 5 commits November 9, 2026 22:24
The waiting view read only statusCheckRollup.state, which GitHub folds to FAILURE when a check run is cancelled, so a pull request whose only non-green run was the view's own job, superseded by its concurrency group, was told to fix the failed check. It now reads the head's contexts and counts only a run concluded FAILURE, TIMED_OUT or STARTUP_FAILURE, or a status FAILURE or ERROR, as failed; with none, the state is pending or green from the other contexts.

Assisted-by: Claude Opus 5.5 (Anthropic)
The scenarios that read the autopilot's result (its seal, the autopilot label, its pull request) opened their issue accepted, or accepted it before the triage answered, so the autopilot saw it unsealed or the expectation was read before it ran. Each now opens its issue plain, waits until awaiting-acceptance, accepts it, and waits until the seal before the autopilot's result, within 15 minutes; core-one-at-a-time does the same for its before issue, with steps that take `of: before`. A new autopilot_problems, run by coverage, names any scenario that reads that result too early, and the self-test plays accepted-spec against a fake repository that seals only after the triage.

Assisted-by: Claude Opus 5.5 (Anthropic)
claim-and-release opened its issue accepted and then commented /claim, which the claim refuses once the autopilot has started. It now opens the issue plain, claims it, waits until claimed, and only then accepts it before /unclaim. autopilot_problems refuses any scenario whose /claim comes after its issue's acceptance, and the self-test plays it against a fake that refuses a late claim.

Assisted-by: Claude Opus 5.5 (Anthropic)
The test identity has the write role, and only a person with the admin or maintain role approves a release, so release-issue can never get a release pull request. It now expects issue-accepted.yml's "No release pull request:" refusal and no pull request (`pull: false`, which holds now reads); the self-test matches the expected words against the writer's refusal in issue-accepted.yml, so the two cannot drift apart.

Assisted-by: Claude Opus 5.5 (Anthropic)
The decision body had no PRD, so the triage left it incomplete, and bug-not-reproduced had no Version and no steps, so the triage asked for more. The decision is now a full PRD whose Repositories names only the test repository, opened plain and accepted after its triage; both bug bodies carry every field bug_report.yml requires. fixture_problems no longer exempts the decision, and checks each Bug body against the form's required fields, read from bug_report.yml.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot

dilux-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot has a question before it goes on. C1 to C5 are committed and their tests pass. C6 (every scenario passes in the test repository) cannot pass as written, because decision-sub-issues expects a comment with "sub-issue". Only scripts/decision.py posts that comment, and issue-accepted.yml runs it only in DiluxOne/.github (if: github.repository == 'DiluxOne/.github'). In diluxone-test nothing posts it, even with the decision body fixed by C5 (opened plain, accepted after its triage). Which do you want? 1. (Recommended) decision-sub-issues expects what a decision gets in the test repository: accepted and its seal (<!-- dx-prd-seal -->), with until the seal, and no sub-issue comment. decision.py --test already covers opening the sub-issues, and this changes only tests/release/scenarios.yml and widens nothing. 2. Let the decision job also run in the test repository (org.yml test-repo). This widens a workflow condition and gives a test the bot's issue-write token for the whole organisation, which is outside this PRD's scope. 3. Remove decision-sub-issues from the scenarios. One more thing: for C1 I took the Spec's recommended option (a), so no cancelled check run counts as failed (logged as a decision). Say so if you want (b) instead.

Answer here in your own words; the autopilot goes on with your answer (autopilot:waiting).

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: answer the autopilot's question #714

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member

Option 1: decision-sub-issues expects what a decision gets in the test repository (accepted and its seal, with until the seal, no sub-issue comment). Don't widen issue-accepted.yml. For C1, option (a) is right: no cancelled check run counts as failed.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the autopilot:free-2 The autopilot's runs on this pull request that do not count towards its cap label Nov 10, 2026
dilux-bot Bot added 2 commits November 10, 2026 00:09
The release lane's run on v7.14.0's candidate with this fix is C6's proof, and it cannot run before the candidate is on v7-next. This brings docs/versioning.md in step with how the lane now plays a scenario that needs the autopilot's result (opened without accepted, accepted after its triage, then waiting until the seal and the result, with a /claim before accepted), and docs/workflows.md with the waiting view's rule that a cancelled check run is not a failed check.

Assisted-by: Claude Opus 5.5 (Anthropic)
The maintainer's answer on #713 settles what a decision gets in the test repository, the only one its body names: accepted and its seal, waiting until the seal, with no sub-issue comment, and issue-accepted.yml stays as it is. The scenario now waits until the seal within 15 minutes and expects it, and release-scenarios.py --test holds both, with decision-sub-issues among the scenarios that read the autopilot's result.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 10, 2026 00:09
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 10, 2026 00:09
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 10, 2026
@dilux-bot

dilux-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity high · type fix

Nothing that must be fixed · 2 minor · 1 pre-existing, not this pull request's

First review. I read the diff only and could not run the --test suites here. In scripts/waiting.py, a cancelled check run is no longer read as a failed check. In tests/release/scenarios.yml and scripts/release-scenarios.py, scenarios that need the autopilot's result now open their issue plain and wait for the triage verdict, the acceptance and the seal. The new autopilot_problems, form_required/missing_fields, of: before steps and pull: false have self-tests. I found no blocker or major problem. Minors: (1) scripts/claim.py (lines 309, 852) still reads the raw rollup state, so a claimed pull request with only a cancelled run still counts as stalled; outside the criteria, listed for the maintainer. (2) checks_state treats a completed ACTION_REQUIRED or STALE run as neither failed nor running, so the board can say "review it and merge it" while a fork's run waits for approval; GitHub's required checks still hold the merge. (3) decision-sub-issues no longer checks any sub-issue, so its name overstates it (the maintainer's option 1). (4) The description does not mention the new of: before on comment/label steps or expect.pull: false, though docs/versioning.md does. No new capability, so no version:major suggestion. C6 cannot be proven by this pull request.

  • minor (pre-existing) scripts/claim.py:852: claim.py still reads the raw rollup state, so a cancelled check run counts as a failure there
  • minor scripts/waiting.py:153: A completed ACTION_REQUIRED or STALE run is read as green by checks_state
  • minor tests/release/scenarios.yml:292: decision-sub-issues no longer exercises any sub-issue

Policy floor: high (touches high-risk paths: scripts/fixtures/waiting.json, scripts/release-scenarios.py, scripts/waiting.py, tests/release/scenarios.yml). Reviewed 6ed6600 (whole pull request; review 1 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.46, 14 turns

@soydiloreto
soydiloreto merged commit 0fbbcaa into main Nov 10, 2026
33 checks passed
@soydiloreto
soydiloreto deleted the fix/713-release-candidate-v7-14-0 branch November 10, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot:free-2 The autopilot's runs on this pull request that do not count towards its cap autopilot:run-1 The autopilot's runs on this pull request autopilot Opened by the autopilot from an accepted issue complexity:high Set by the Claude review risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: release candidate v7.14.0 fails scenario accepted-spec

1 participant