Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 36 additions & 23 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -478,31 +478,44 @@ jobs:
# margin for a cold Turbo cache; the old 45 left a hung job "running" for
# half an hour past any plausible healthy finish.
#
# ⚠ RAISED 30 -> 45 (#16173), and the raise is still load-bearing. It was
# made because shard 5/6 was being killed at the 30-minute wall (12+
# observations at 30:16-30:21 with every other job green; the green band
# topped out at 28:56, a ~1 minute margin), and so the tail could be
# measured uncensored before anyone re-derived
# `scripts/test-shard-timings.json`. Both are done: the dataset was
# refreshed (#20388, then #21826) and the split ran the whole CLI alone on
# shard 1/6 (#21487) until #22075 cut it into 3 slices. HELD at 45 by that
# change: the re-size is owed after it lands, from the measured post-change
# Test Core job walls, never from a prediction (#22075 carries it).
# History: RAISED 30 -> 45 (#16173) while one shard ran the whole CLI and
# was killed at the 30-minute wall; held at 45 until #22075 cut the CLI
# into 3 slices (5919483472).
#
# ⛔ REVERT CONDITION, restated against measured wall time, not a card:
# back to `30` only when the slowest shard's JOB wall time stays at or
# under 24 minutes (80% of 30, so the margin the old wall lacked is
# there) on every scheduled run for a week. It does not today. On the 16
# main runs after the #21826 refresh (37413386379 to 37467882762), shard
# 1/6 — the CLI alone — read 6m14s-35m43s of job wall time, over 30
# minutes on 9 of them (34m39s run 37415122516, 35m43s run 37453598388,
# 34m08s run 37460325624), so the condition as first written ("once
# #16173 lands") would now kill that shard; and its slowest reading is
# already 79% of this 45. Read the `Test Core (1/6)` job's started and
# completed times before touching this value. It carries no other expiry
# — a raise with no revert condition beside it becomes permanent by
# ⚠ RE-SIZED 45 -> 50 (#22075), derived so that BOTH stall-guard verdicts
# land before this wall on every observed run. Window: every completed,
# non-cancelled CI run whose tested tree carries 5919483472, runs
# 37917885267 to 37956350887 (2026-10-09 10:29Z-16:03Z): 5 schedule,
# 6 push, 30 merge_group and 25 pull_request runs, 396 `Test Core (N/6)`
# jobs; 24 cancelled runs excluded. Slowest job wall per event, in
# minutes: schedule 21.0, push 20.9, merge_group 24.1, pull_request 27.1;
# p90 of all 396 is 22.0. Measured with
# `node scripts/measure-stall-guard-headroom.mjs` over those jobs, the
# worst p + s (job start to the end of "Run this shard's tests") is
# 26m38s, on run 37923333449 `Test Core (3/6)`. Three readings set the
# value, and the largest answer wins:
# * the window verdict, a silent freeze killed 10m after the last
# output: p + s + 10m = 36m38s;
# * the cap-deferred verdict, when the liveness probe sees the suite
# still writing: p + s + 20m = 46m38s, inside 50 by 3m22s. It missed
# 45 on 8 of the 396 jobs and would miss 40 on 60;
# * the 80% wall margin: the slowest job wall, 27m05s on that same job
# (70 packages executed, the CLI as all 3 slices), is within 80% of
# 50 (40.0).
#
# ⛔ REVERT CONDITION, restated against measurement, not a card: a value
# below 50 is allowed only when, on a re-measured window of Test Core
# runs (`node scripts/measure-stall-guard-headroom.mjs` over the jobs
# payloads of every completed pull_request, merge_group, push and
# schedule run in it), BOTH guard verdicts -- the window path
# (p + s + 10m) and the cap-deferred path (p + s + 20m) -- land before the
# new value on every observation, AND the slowest `Test Core (N/6)` job
# wall is at or under 80% of it. Read the pull_request and merge_group
# runs, not the hourly schedule alone: scheduled runs mostly replay the
# cache their commit's push run saved. It carries no other expiry — a
# raise with no revert condition beside it becomes permanent by
# forgetting — and the stall guard stays the primary hang detector.
timeout-minutes: 45
timeout-minutes: 50
permissions:
contents: read
strategy:
Expand Down
Loading