Skip to content

Fix - Let org members waive the spec gate with qa-waived, report refusals as neutral, and build assets in CI - #19

Merged
iamprazol merged 2 commits into
mainfrom
feat/qa-waived-neutral-refusal-build
Oct 28, 2026
Merged

iamprazol merged 2 commits into
mainfrom
feat/qa-waived-neutral-refusal-build

Conversation

@iamprazol

Copy link
Copy Markdown
Collaborator

Summary

Three changes to the PR gates in suite.yml and pro-suite.yml.

1. A qa-waived label can waive the spec gate. Some changes cannot have the spec the gate asks for (a Stripe or PayPal webhook, a cron handler, a subscription row with no order), because the harness cannot create that state. Reviewers were already waiving these by comment, with nothing to say who agreed. Now a member of the organisation applies the qa-waived label.

  • A new script, gate-refusal.mjs, checks that the label is on the PR and who applied it. The person must be an organisation member, or have write access on the repo. A bot never counts.
  • A waived PR runs the suite on whatever its diff maps to. Only the "PR must carry a spec" requirement is waived, not the tests. The PR comment names who waived it.
  • It covers both gates: no regression spec, and a changed spec that cannot be selected.
  • The label takes effect on the next run. The caller templates now list labeled in pull_request.types, and the reusable workflows let only qa-waived through on that event, so any other label is a skipped job. Existing callers need that one line added. @claudegrill suite re-runs for any caller without it.

2. A refused PR no longer fails the job. Nothing was tested and nothing broke, so a red check read like a test failure.

  • suite.yml: the job passes with a warning, and a separate grey check run, "QA suite — refused", says why. This needs checks: write on the suite job. Without it the check run is skipped with a warning and the job still passes.
  • pro-suite.yml: the gate job passes and reports no modes, so the whole matrix is skipped (grey). No new permission.
  • The 👀 → 🚀 reaction now shows 😕 for a refusal.

3. CI builds the product's assets. A new "Build the product's assets" step runs a build command declared in the product's .themegrill-qa/suite.json, after install and before the site boots. A product with no build skips it and behaves as before. This is documented in SUITE.md.

Why: CI tested whatever compiled CSS/JS was committed, so a source fix failed until the rebuilt files were committed too (user-registration #1535, #1548). Anything the build emits that is gitignored never existed on a runner at all. User Registration's React bundle (chunks/) is one, and its setup-wizard specs failed or skipped in CI because of it.

The matching build lines for user-registration and user-registration-pro are already on their develop branches.

Verification

  • gate-refusal.mjs against a stubbed gh, 10 cases: member, write access, read-only access, a bot, a label re-applied by an outsider, no event found, no label, API failure, two gates tripped, nothing tripped. Plus the usage errors.
  • The new steps' bash, lifted from both workflow files and run against a stubbed gh, for both workflows, including a token without checks: write.
  • actionlint on both workflows: nothing new, apart from two info-level shellcheck notes (SC2016) on a node -e pattern main already uses.
  • setup-product.mjs check-workflow on the updated templates: clean. synchronize and an unknown type are still flagged.
  • The build, against real UR: build:no-makepot takes about 20 seconds and changes no tracked files. With the bundle built, the setup-wizard specs pass 6 of 6 on a fresh Playground site. The pro command (UR_PRO=true) also builds in about 15 seconds with no tracked changes.

Not verified

Nothing has run in GitHub Actions. Callers pin @main, so it can only be exercised after merge. Specifically unproven:

  • the job-level if allow-list for labeled + qa-waived
  • the organisation-membership check with GITHUB_TOKEN (it may not see private members, which is why write access is the fallback)
  • the neutral check run being created with checks: write
  • the modes=[] output skipping the pro matrix, and the answer job staying quiet on a refusal

To test after merge: open a PR with a source change and no spec, confirm the grey refusal, apply qa-waived, and confirm the suite runs.

Notes

  • labeled fires on every label. Other labels cost one skipped job and no runner minutes.
  • A labeled run of qa-waived on a PR that was never refused will run the suite again.
  • The "Refusal as neutral" and "build in CI" items came from the UR e2e analysis: refused PRs were merged despite a red check, and CI kept failing on build output.

🤖 Generated with Claude Code

@iamprazol
iamprazol merged commit 586bc92 into main Oct 28, 2026
2 checks passed
@iamprazol
iamprazol deleted the feat/qa-waived-neutral-refusal-build branch October 28, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant