ci: a per-package suite-duration ratchet — Test Core reds a suite over its measured ceiling

.github/workflows/ci.yml

@@ -1391,7 +1391,10 @@ jobs:
       # redden it. Ordering after the verdict is what makes "cannot delay"
       # true, and the two flags are what make "cannot redden" true — a
       # diagnostics fault here (an artifact-service 403 is the measured one,
-      # #14469) must not evict a green run.
+      # #14469) must not evict a green run. The suite-duration ceiling check
+      # at the end of this job reads the same captures and is the one step
+      # here that DOES red the job, on purpose: it is a gate, not a report, and
+      # a capture that never arrived reads NOT MEASURED there, never red.
       #
       # ⛔ Steps, deliberately, NOT a new `needs:` entry: this job's
       # `OS_AGGREGATOR_MEMBERS` / `OS_AGGREGATOR_NON_MEMBERS` block is asserted
@@ -1447,6 +1450,38 @@ jobs:
           if-no-files-found: ignore
           retention-days: 14
 
+      # ── The per-package suite-duration ratchet (#16468) ──────────────────
+      #
+      # A GATE, unlike the two reporting steps above: NO `continue-on-error`,
+      # so a package whose suite ran past its ceiling reds `Test Core`, naming
+      # the package and its slowest test files. The ceiling is the package's
+      # slowest executed run in the refresh window x 1.25, generated into
+      # scripts/test-shard-timings.json (`ceilings`) by the refresh lane and
+      # held by every later refresh; it rises only by a ruled entry in the
+      # script's RULED_CEILING_RAISES. A `provisional` package prints "no
+      # ceiling: provisional" and is not red; a package absent from the dataset
+      # is red (the new-package shape). See the script header for the ruling.
+      #
+      # HERE, in the aggregator, rather than in each shard, because the CLI's
+      # file-level slices land on three shards: the three windows are SUMMED
+      # and the sum graded against the whole package's ceiling -- the quantity
+      # the ceiling was measured in. It reads the shard captures the download
+      # step above already fetched (each package's turbo execution window via
+      # samplesFromSummary, and every vitest file line); a capture that did not
+      # arrive leaves its packages ungraded and says so, never red.
+      #
+      # NOT MEASURED, exit 0, while the dataset carries no `ceilings`: they are
+      # generated only where the run-summary artifacts are downloadable (the
+      # refresh lane), so the first refresh after this lands writes them, and
+      # until then every run prints a warning annotation saying so.
+      #
+      # LAST, after the reporting steps, so the timing table is published
+      # whatever this decides. `always()`: it reads only what the shards already
+      # captured, so a red shard's packages are graded too.
+      - name: Check the per-package suite-duration ceilings
+        if: always()
+        run: node scripts/check-test-suite-ceilings.mjs --captures "$RUNNER_TEMP/test-timings" --label 'Test Core'
+
 
   # ── Temporal conformance against live, non-UTC servers (ADR-0053 D-A3) ─────
   #

.github/workflows/lint.yml

@@ -4369,6 +4369,16 @@ jobs:
       - name: Test-timing parser self-test
         run: node scripts/report-test-timings.mjs --self-test
 
+      # The per-package suite-duration ratchet's own instrument (#16468). The
+      # gate runs in ci.yml's `Test Core` aggregator, where a clean tree with
+      # every suite under its ceiling reads OK whether or not the over-ceiling,
+      # absent-package and summed-slice rules still work -- so its self-test,
+      # which feeds those shapes in, runs here with teeth. Invoked as `node` for
+      # the GATE INVOCATION IDIOM reason at the top of this file. In-process
+      # fixtures, no IO, milliseconds.
+      - name: Suite-duration ceiling check self-test
+        run: node scripts/check-test-suite-ceilings.mjs --self-test
+
       # Hand-written `.d.mts` mirrors (#10549). `scripts/js-comment-mask.mjs`
       # and `scripts/check-regen-pending.mjs` are untyped `.mjs` that each ship
       # a hand-written declaration beside them, and both files say "keep this in

.github/workflows/shard-timings-refresh.yml

@@ -33,6 +33,16 @@
 #      quoted verbatim into the PR body and a human decides. See "THE PINS" below.
 #   ⛔ It never opens a PR when the regenerated file is byte-identical. A weekly
 #      no-op PR trains everyone to ignore this PR.
+#   ⛔ It never moves a suite-duration ceiling (#16468). The dataset also
+#      carries `ceilings`, which `Test Core` grades every executed package
+#      against (scripts/check-test-suite-ceilings.mjs). The generator writes
+#      them in the same pass and HOLDS every ceiling the committed dataset
+#      already has; a package gets a new one only when it had none (its slowest
+#      executed run in this window x 1.25). A ceiling rises only by a ruling,
+#      recorded in that script, so a refresh whose window ran slower REPORTS
+#      the gap in the PR body instead of absorbing it. The first refresh after
+#      the check landed writes the first table; until it merges the check reads
+#      NOT MEASURED.
 #
 # HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE
 # ----------------------------------------------------------------------------
@@ -634,6 +644,53 @@ jobs:
             }
           ')
 
+          # The suite-duration ceilings (#16468), read out of the dataset like the
+          # two lines above. The prior table is the COMMITTED dataset, read from
+          # git because the bins step has already copied the refreshed file over
+          # it. A refresh never moves a held ceiling, so the line counts what is
+          # new and what is uncapped, and the spread table below marks every
+          # held ceiling this window would have set differently -- and every one
+          # a run of this window already went over.
+          CEILING_LINE=$(node -e '
+            const fs = require("fs");
+            const { execFileSync } = require("child_process");
+            const d = JSON.parse(fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/refreshed.json", "utf8"));
+            if (!d.ceilings || typeof d.ceilings !== "object" || !d.uncapped || typeof d.uncapped !== "object") {
+              throw new Error("the refreshed dataset carries no `ceilings` / `uncapped`");
+            }
+            const prior = JSON.parse(execFileSync("git", ["show", "HEAD:scripts/test-shard-timings.json"], { encoding: "utf8" }));
+            const held = prior.ceilings && typeof prior.ceilings === "object" ? prior.ceilings : null;
+            const names = Object.keys(d.ceilings);
+            const fresh = held === null ? names : names.filter((n) => !Object.hasOwn(held, n));
+            const uncapped = Object.entries(d.uncapped).map(([n, why]) => n + " (" + why + ")");
+            const h = d.provenance.ceilingHeadroom;
+            const over = (fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/spread.md", "utf8").match(/OVER its held ceiling/g) || []).length;
+            const lines = [];
+            if (held === null) {
+              lines.push(
+                "⚠️ This refresh writes the FIRST suite-duration ceiling table: " + names.length + " package(s) get a " +
+                "ceiling, each its slowest executed run in this window x " + h + ". From the merge of this PR the `Test Core` " +
+                "check (scripts/check-test-suite-ceilings.mjs) grades every executed package against it -- until then it " +
+                "reads NOT MEASURED. Every later refresh HOLDS these numbers and a ceiling rises only by a ruling, so " +
+                "review the ceiling column below before merging."
+              );
+            } else {
+              lines.push(
+                "Suite-duration ceilings: " + (names.length - fresh.length) + " held unchanged (a refresh never moves one; " +
+                "a raise is a ruling), " + fresh.length + " new" + (fresh.length ? " (" + fresh.join(", ") + ")" : "") +
+                ", each new one its slowest executed run in this window x " + h + "."
+              );
+            }
+            lines.push(uncapped.length ? "No ceiling: " + uncapped.join(", ") + "." : "Every package carries a ceiling.");
+            if (over > 0) {
+              lines.push(
+                "⚠️ " + over + " package(s) ran OVER their held ceiling in this window (marked in the table): the brake " +
+                "fired on main. This refresh does not raise them; that takes a ruling."
+              );
+            }
+            console.log(lines.join("\n\n"));
+          ')
+
           SHARDS=$(node -e '
             const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
             const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID);
@@ -675,10 +732,12 @@ jobs:
             echo "because a weekly lane cannot know which cards a given run ought to retire. If this is"
             echo "the first refresh to land, retire those two by hand as part of merging it."
             echo
-            echo "## Sample depth and run-to-run spread"
+            echo "## Sample depth, run-to-run spread and suite-duration ceilings"
             echo
             echo "$DEPTH_LINE"
             echo
+            echo "$CEILING_LINE"
+            echo
             cat "$RUNNER_TEMP/refresh/spread.md"
             echo
             echo "## Measured per-shard suite time on the newest run in the set"