Skip to content

ci(rerun-safety): a truthful verdict, passes bounded by the job's time, and a card on a red night - #23975

Merged
os-zhuang merged 7 commits into
mainfrom
claude/issue-23880-rerun-safety-nightly
Oct 17, 2026
Merged

os-zhuang merged 7 commits into
mainfrom
claude/issue-23880-rerun-safety-nightly

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Part of #23880
Clause-②: no

Rerun Safety (the nightly that runs the whole suite twice in one working tree) has concluded success on one of its last thirty scheduled nights (2026-09-26), and nothing filed a card. This PR changes only that job's own shape and its reporting. No package code or package test is edited.

  • scripts/rerun-safety-verdict.mjs now separates a task turbo failed from a task turbo interrupted after another task failed. It also says how much of each pass actually ran: MEASURED, PARTIALLY MEASURED, or INCOMPLETE/NOT MEASURED. It flags a failure whose every failed test is a vitest timeout, and says how to tell load from a leak.
  • .github/workflows/rerun-safety-nightly.yml has three changes:
    • each pass runs under timeout set to the time the job has left, so a slow night ends in a verdict and a failure conclusion instead of a silent cancelled;
    • --only stops each pass re-executing the 69 build tasks that Build packages just built;
    • the header records the measured wall-time growth.
  • .github/workflows/scheduled-full-run-card.yml adds 'Rerun Safety' to workflow_run.workflows, so a red night files or refreshes a card.
  • scripts/ci/scheduled-full-run.mjs (patch round, claim 6218695689) gives that card its workflow's cadence. A closed map, CARD_CADENCES, sets CI and Lint & Type Check to hourly and Rerun Safety to nightly; a workflow outside the map is refused, never defaulted. For the two hourly workflows every byte the card carries is unchanged, so the open CI card hourly full run: red on main (CI) #23885 is still found and refreshed.

Why Part of: Done-when item 3 ("a nightly run concludes success, or names only failures that already have cards") can only be read on a nightly after this lands. Two full passes also no longer fit one 4-vCPU runner in 170 minutes. That capacity choice belongs to the maintainer, after the first nightly reading; it is set out under Capacity below. #23880 remains open for it.

Readings: the failing tests, per night and pass

Source: each run's job log via MCP get_job_logs. That tool returns the last 5,000 lines. The REST log zip answers 403 from this container (results-receiver.actions.githubusercontent.com, connect_rejected). A full log is 290k to 436k lines, so pass 1's detail is readable only on nights whose pass 2 stopped early. The other nights are marked NOT MEASURED.

Column meanings:

  • FAILED: turbo's own report, from its Failed: footer and per-task ERROR command … exited (1) line.
  • INTERRUPTED: the task printed only ELIFECYCLE Test failed., 0.1 to 0.5 s after the real failure's, with no ERROR line and no footer entry.
night run pass FAILED the failing test and its error line INTERRUPTED turbo: started / successful
10-02 36962761984 1 dogfood activity-parent-read-gate.dogfood.test.ts:102 "list: every row returned is about a record the member can open…"; activity-text-predicate.dogfood.test.ts:465 "control: the unmasking reader filters and groups by every column…"; audit-log-parent-read-gate.dogfood.test.ts:144 "list: every row returned that names a record…". Each: Error: Test timed out in 5000ms. cli 138 / 136
10-02 2 core src/security/platform-admin.test.ts:278 "nor by @objectstack/core's security entry — the published surface lost them" (a dynamic import('./index.js')): Test timed out in 5000ms. driver-sql, spec, lint 42 / 20
10-13 38344867129 1 NOT MEASURED (verdict line: cli, dogfood)
10-13 2 runtime src/declared-capability-absence-warn-once.test.ts "the dispatcher door > reports a declared 501 ONCE across N requests…": Test timed out in 30000ms. verify, http-conformance 132 / 127
10-14 38509291145 1 dogfood activity-text-predicate.dogfood.test.ts:470 "control: the unmasking reader filters and groups by every column of a pinned parent, as before": Test timed out in 5000ms. cli 143 / 141
10-14 2 core src/security/auth-gate.test.ts:571 "BOUNDARY — newly exempt is exactly POST on the four-segment hook route…": Test timed out in 5000ms. spec, lint 49 / 24
10-15 38679312462 1 NOT MEASURED (verdict line: cli, dogfood)
10-15 2 runtime src/domains/share-links-refusal-order.test.ts:356 "email audience: the refusal is the one an unknown token and a password-less email link answer": Test timed out in 5000ms. verify (it logged Received SIGINT), cli, organizations 142 / 132
10-16 38848504327 1 NOT MEASURED: no verdict ran
10-16 2 none: cancelled by the job timeout 67 min in, still printing

Every real failure in the table is a vitest timeout. The pass-2 "newly failing" sets the old verdict printed are not rerun-safety findings either:

  • 10-14: core, lint, spec;
  • 10-15: cli, organizations, runtime, verify.

Each was one real timeout plus the suites turbo stopped beside it.

Pass 1's root cause

  • @objectstack/cli#test is not a pass-1 failure. On both nights whose pass 1 is readable (10-02, 10-14), turbo's footer names @objectstack/dogfood#test alone. cli printed ELIFECYCLE 0.16 to 0.23 s later, with no ERROR line: turbo, which runs without --continue, stopped it when dogfood failed. 10-13 and 10-15 print the same verdict shape; their detail is NOT MEASURED. The old verdict read every ELIFECYCLE as a failure. That was the defect, and this PR removes it.
  • @objectstack/dogfood#test is vitest 5000 ms timeouts under this job's load. I ran the failing files locally, unloaded, on an idle 4-core container at 98ba1bf1d2. All passed:
    • activity-text-predicate's control test: 3,377 ms, which is 68% of vitest's default timeout. The file's 22 tests sum to 12.9 s.
    • activity-parent-read-gate's list test: 1,063 ms.
    • audit-log-parent-read-gate's list test: 974 ms.
  • Why the load is higher here than in the hourly battery:
    • The hourly Dogfood job runs dogfood alone, in three slices, one per 4-vCPU runner.
    • Here the whole dogfood suite runs beside up to three other heavy suites: 34 min on 10-14, starting 64 min into the pass. Turbo runs at --concurrency=4, each task at vitest's default of three workers, so twelve workers share four vCPUs.
    • The tail of the pass is where dogfood, cli, runtime and verify overlap.
  • That is this job's environment. This PR makes the job say so, and does not change the tests. How the job gets its capacity back is the decision under Capacity.

The pass-2-only packages: each passes a second run in one tree

Measured locally at 98ba1bf1d2, unloaded, --maxWorkers=2, two consecutive runs in the same worktree:

package run 1 run 2
@objectstack/core 108 files / 2,586 tests passed, 72 s 108 / 2,586 passed, 70 s
@objectstack/runtime (--project local) 402 files / 6,122 tests passed, 679 s 402 / 6,122 passed, 654 s

After both runs, git status showed no new or ignored state beyond dist/ and node_modules/.

The specific tests, unloaded:

  • auth-gate BOUNDARY: 1,055 ms.
  • share-links "email audience": 1,057 / 1,086 / 1,135 / 1,136 ms over four runs.
  • warn-once: 79 ms.

The 10-13 warn-once test ran 79 ms here and hit its own 30 s bound in pass 2. That gap is beyond plain CPU contention. What held it is NOT MEASURED. It is one occurrence that passes twice locally, so it is noted here and not carded.

The timeout's cause: wall-time growth, not a hang, not one new suite

  • 10-16 step timestamps:
    • setup, install and build: 7 min (04:02:11 to 04:09:14);
    • pass 1: 100.4 min (04:09:14 to 05:49:39);
    • pass 2: started 05:49:39 and was cancelled at the job's 170-minute timeout (job end 06:57:11). Its log was still printing at 06:54:45 (verify and example-todo booting), and the 15-minute stall guard never fired.
  • 10-12, the same: pass 1 113 min; pass 2 cancelled 56 min in.
  • Pass 1 duration by night:
    • 09-26, the last green night: 38 min (pass 2 also 38);
    • 10-02: 66; 10-03: 65.5;
    • 10-14: 98; 10-15: 97; 10-16: 100.
  • Queue-tier test task time in scripts/test-shard-timings.json:
    • by refresh date: 123.8 min (measured 09-28), 163.0 (10-05), 177.2 (10-08), 187.1 (10-12), 193.6 (10-13);
    • @objectstack/cli alone: 733 s, 1,703, 1,739, 2,109, 2,206.

So two full passes need roughly 200 minutes of a 170-minute job. A night "finished" only because pass 2 stopped at its first failure: on 10-14 it ran 7 minutes and started 49 of 143 tasks. A night that did not fail early was cancelled, with no verdict. Raising timeout-minutes is ruled out by triage's scope and the test-budget rulings, and this PR does not raise it.

What this PR does about it

  1. Each pass is bounded by the time the job has left.
    • The bound is budget = (deadline − now)/60 − PASS_RESERVE_MINUTES, with JOB_TIMEOUT_MINUTES: 170 set beside timeout-minutes: 170.
    • It is spelled timeout --kill-after=2m "${budget}m" and sits inside the stall guard, as the command it wraps.
    • Measured locally (guard, then timeout 45s, then turbo over core, types and formula): exit 124 came back through the guard. The log ended at Shutting down Turborepo tasks... / Force killed Turborepo tasks: @objectstack/core#test, with no Tasks: summary. No process was left behind.
    • The verdict reads 124 or 137 as a pass that did not finish: INCOMPLETE, rerun property NOT MEASURED, exit 1. The job concludes failure, which the card filer reads; cancelled it does not read.
  2. --only on both passes. turbo run test --force --dry=json at 98ba1bf1d2 plans 83 test, 69 build and 1 @objectstack/spec#gen:migration-registry task.
    • All 69 builds are in pnpm build's own plan (Build packages).
    • The spec's test script runs pnpm gen:migration-registry itself.
    • With --only, a pass runs only the 83 test tasks, and pass 2 no longer rebuilds dist/ over what pass 1 left there.
    • NOT MEASURED: how many minutes this saves a nightly pass. That needs the first nightly after landing.
  3. The verdict tells the truth about every night above. Here is run 38509291145 (10-14), read from its job log by the old script and by this one.

Old (what the job printed):

  Newly failing on pass 2: @objectstack/core#test, @objectstack/lint#test, @objectstack/spec#test
  rerun property: MEASURED
  pass 1 exit 1, failing tasks: @objectstack/cli#test, @objectstack/dogfood#test

New:

  Newly failing on pass 2: @objectstack/core#test
  @objectstack/core#test: every failed test hit vitest's test timeout -- load-shaped. A leak fails an assertion, not a clock: run `pnpm --filter @objectstack/core run test` twice in one tree on its own; a leak fails the second run there, a load timeout does not.
  Failing in both passes: (none)
  Failed on pass 1 and never finished by pass 2 (NOT re-measured, not recovered): @objectstack/dogfood#test
  Interrupted by turbo in pass 1 after another task failed (not failures, unmeasured): @objectstack/cli#test
  Interrupted by turbo in pass 2 after another task failed (not failures, unmeasured): @objectstack/lint#test, @objectstack/spec#test
  Pass 1 stopped at its first failure (turbo runs without --continue); the tasks still running then were interrupted and are unmeasured in that pass.
  Pass 2 stopped at its first failure (turbo runs without --continue) after starting 49 of the 143 tasks the longer pass started; every task it did not finish is unmeasured in that pass.

  rerun property: PARTIALLY MEASURED
  pass 1 exit 1, failing tasks: @objectstack/dogfood#test (timeout-only)
         interrupted by turbo (unmeasured): @objectstack/cli#test
         turbo: 141 successful of 143 started
  pass 2 exit 1, failing tasks: @objectstack/core#test (timeout-only)
         interrupted by turbo (unmeasured): @objectstack/lint#test, @objectstack/spec#test
         turbo: 24 successful of 49 started

The 10-02 log (run 36962761984) reads the same way: newly failing core alone, not core, driver-sql, lint, spec. Pass 2 started 42 of 138 tasks.

The reporting change: a card for a red night, in the night's own cadence

scheduled-full-run-card.yml now lists 'Rerun Safety'. I measured the existing mechanism against a once-a-day workflow:

  • Identity and dedupe fit. The card is per workflow name, so it is distinct from the CI and Lint cards. Refresh-in-place suits a nightly streak: one card, and no comment per night.
  • The conclusion guard missed every timeout night. The guard reads failure || timed_out, but a job timeout here concluded cancelled: runs 38848504327 and 38202695171 are cancelled at both run and job level. That gap is closed from the job side by change 1: an overrun night now concludes failure.
  • The card body's cadence was hourly for every watched workflow. This patch round closes that in scripts/ci/scheduled-full-run.mjs:
    • CARD_CADENCES is a closed map from the byte-exact workflow name to its cadence: CI hourly, Lint & Type Check hourly, Rerun Safety nightly. cardIdentity refuses a name outside it and names the map in the refusal, so adding a workflow to the card's watched list means adding its cadence in the same change.
    • The cadence drives the title prefix, the marker prefix and every cadence sentence in renderBody: the marker line, the "runs on main…" paragraph, the remedy lines, the "What landed since the last green … run" heading, and the empty-range flake reading.
    • Rerun Safety therefore files nightly full run: red on main (Rerun Safety) with marker os-nightly-full-run-rerun-safety. Its body says "every night", "the last green nightly run" and "green the night before", and has no hourly wording.
    • The hourly entry is the old text byte for byte. The CI card keeps hourly full run: red on main (CI), marker os-hourly-full-run-ci, and the same marker line and body.
    • renderBody also refuses an identity built by hand without a cadence, rather than default one.
    • The range for Rerun Safety starts at its last green schedule run, 09-26 (run 36216601546). The compare API caps at 250 commits, and the body already says when it truncated.

Capacity: the maintainer's choice, after the first nightly reading

Under this PR, a night whose two passes do not fit gets a loud INCOMPLETE and no longer a silent cancelled. It is still not a measurement. Once the first nightly after landing prints its two pass durations (the budget lines print them), there are three ways to make two full passes fit again:

  • A. Legs.
    • What: a matrix of legs (dogfood | cli | the rest, split in two by scripts/partition-test-shards.mjs with cli and dogfood excluded), each running its own package set twice in its own tree.
    • Fit: each leg has an estimated 2 × 45 + 8 min of work at today's timings.
    • Cost: standard runners and about three more setup-and-build laps a night.
    • Loses: detection of pollution between legs.
    • Scales: add a leg.
  • B. A larger runner.
    • Keeps one tree and the full property.
    • Needs an organization larger-runner label and is billed per minute. Whether one is configured is NOT MEASURED; every runs-on in this repo is ubuntu-latest.
  • C. Keep one runner under this PR's bound. The nightly measures only what fits, and says so.

Recommendation, after that reading: if pass 1 still exceeds about 78 minutes, choose A. That threshold is 170 minus 7 minutes of setup and build minus the 5-minute reserve, halved.

Split cards (self-split rule)

One lane card was warranted. The seat filed it from the dev's draft in the report on #23880:

  • Dogfood lane: packages/qa/dogfood/test/activity-text-predicate.dogfood.test.ts. Its 22 tests sum to 12.9 s unloaded, over the 10 s per-file rule (ruling record 6181973363), and it declares no budget. Its "control: the unmasking reader filters and groups by every column…" test runs 3,377 ms of vitest's 5,000 ms default on an idle 4-core box. It timed out in this job's pass 1 on 10-02 and 10-14.

The other timed-out tests are not carded. core and runtime each pass a second run in one tree. The 10-02 dogfood list tests run about 1 s unloaded and are load-only.

Verification

The branch merged origin/main at 68ffa5dc1 (a merge, no rebase). The gate union ran after the final commit, at 240a2a565.

Round 1 (verdict and nightly):

  • pnpm check:rerun-safety-verdict: rerun-safety-verdict --self-test: OK (23 batteries, 23 registered), exit 0. The fixtures are the 10-14 and 10-15 log shapes, colour codes included.
  • Ablation, one behaviour, on committed code via scripts/ablation-replace.mjs:
    • Mutation: put back the pre-fix union, adding failing.add(id) on every ELIFECYCLE line. Landing proof: anchor 1 → 0, blob 050a7fc6ae10 → aec8629e130f.
    • Result: --self-test: 9 FAILED, including the end-to-end case printing Newly failing on pass 2: @objectstack/core#test, @objectstack/lint#test, @objectstack/spec#test.
    • Restored: blob == HEAD (050a7fc6ae10), git diff HEAD empty.
    • The first attempt was a no-op: its replacement re-contained the anchor, ablation-replace refused it as "a drop of 0", and that reading was discarded.
  • Real logs: both scripts were run on the 10-14 and 10-02 job logs; output as above. The old script reproduces the job's printed verdict exactly.

Patch round (cadence):

  • node scripts/ci/scheduled-full-run.mjs --self-test: scheduled-full-run: self-test OK, exit 0. The identity battery now has 39 cases (28 before), and its floor rises from 23 to 34 for the eleven new ones:
    • the CI title and marker as the literals hourly full run: red on main (CI) and os-hourly-full-run-ci;
    • the CI marker line as a literal, in all three body shapes;
    • the open CI card exactly as filed (title and marker line), still found by its title, and still found by its marker alone after a rename;
    • Lint & Type Check hourly;
    • Rerun Safety nightly (title and marker);
    • a nightly body with no hourly wording in any of the three shapes, and an hourly body with no nightly wording;
    • a workflow outside the map (Spec Coverage) refused by name;
    • a hand-built identity with no cadence refused by renderBody;
    • the nightly and hourly identities never matching each other's cards.
  • Byte-identity against the pre-change module (one-off comparison, not a kept file): for CI and Lint & Type Check, title, marker, workflow name, labels and three body shapes match the module at b48f908ad. Result: byte-identical: 14 of 14.
  • node scripts/ci/scheduled-full-run.mjs --check-concurrency: exit 0.
  • Ablations, each on committed 240a2a565 via scripts/ablation-replace.mjs, each restored to blob == HEAD c33af68bc2ee with git diff HEAD empty. Every mutation landed (anchor 1 → 0) and each self-test went red:
    1. The refusal made a default (if (false …) in place of the map check) → Error: cadence: a workflow outside the map was not refused by name (accepted).
    2. CI mapped to nightly → Error: identity: CI title is 'nightly full run: red on main (CI)'.
    3. The hourly every word changed to 'each hour' → Error: cadence: the CI marker line moved: …. The pre-existing cases do not pin this line; only the new marker-line pin catches it.

Gates and tier:

  • Gates at 240a2a565: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derives 53 commands for this diff. Each was run with its exit code captured before any pipe, and all 53 exited 0.
    • These include node scripts/ci/scheduled-full-run.mjs --self-test, and check:stall-guard-budget, which still sees both rerun-safety sites: window 15m, cap 30m, budget 170m, slack 140m.
    • They also include check:workflow-status-functions, check:agent-test-spelling, check:pm-dispatch-gates (fast tier, 1,884 cases, 320 s) and check:rerun-safety-verdict.
    • --ran reconciliation: 53 derived, 53 run, 0 NOT-MEASURED, 0 UNRUN, with a derived zero, because all 53 recorded an exit code.
  • node scripts/pm/check-governed-merges.mjs --branch claude/issue-23880-rerun-safety-nightly at 240a2a565: GOVERNED, landing tier H, .github/workflows/** ×2.
  • Changed lines: 1,089 (+876 / −213) over 4 files. The round-1 body's "899 (+705 / −194)" was read at 58669da5e. Round 1 ended at fdc14b452 with 908 (+713 / −195); the cadence leg adds the rest.

Acceptance notes

  • The verdict's old union reading was encoded in a self-test case named for the 2026-09-08 "footer under-reports" shape. Read against this card's logs, that shape is turbo's fail-fast interruption. The case is replaced, not respelled.
  • The 10-13 runtime warn-once timeout (79 ms unloaded, 30 s bound hit) is an observation. It is not carded: one occurrence, and it passes twice locally.
  • Rerun Safety runs test and not test:repo, while the root pnpm test runs both. That is out of this card's scope, noted only.
  • CARD_CADENCES and scheduled-full-run-card.yml's workflows: list are kept in step by the refusal, at the first red run of an unmapped workflow. No self-test reads the workflow file to compare them, deliberately: that would add a cross-file check to this gate family, and no new gate is in scope.
  • NOT MEASURED:
    • the nightly effect of --only and of the bound (this seat cannot dispatch the workflow);
    • pass 1 detail on 10-13, 10-15 and 10-16;
    • whether a larger-runner label exists for this organization.

维护者速读(草稿)

改了什么:只动「Rerun Safety」夜间任务自己的四处。

  1. 判定脚本:分清「真失败」和「被 turbo 顺手停掉的」,并如实写出每一轮到底测了多少。
  2. 夜间工作流:每一轮只用任务剩余的时间跑,超时就给出「未测完」的结论,不再被平台静默取消;两轮里也不再重复构建已构建好的包。
  3. 红卡工作流:把这个夜间任务加入监听名单,红了会自动开卡或刷新卡。
  4. 红卡文案:按工作流区分「每小时」和「每晚」。夜间任务的卡写「每晚」;CI 等每小时任务的卡一个字都不变,已开着的 CI 卡照常被找到、被刷新,不会重复开卡。

为什么改:这个任务 30 晚里只绿过 1 晚,却没人收到通知。读日志发现:

  • 每晚报的 cli 失败其实是被 turbo 中途停掉的,真正失败的是 dogfood,而且都是 5 秒超时,属于负载问题。
  • 第二轮「新增失败」的包每晚都不同,同样只是第一个超时的包加上旁边被停掉的包。
  • 整个测试集三周里变慢了约 2.6 倍,两轮已经塞不进 170 分钟。没测完的那几晚被平台直接取消,既没有结论,也没有发卡。
  • 新开的夜间红卡如果沿用原来的文案,会写成「每小时跑一次」,误导看卡的人。

风险与代价(含回滚):

  • 不改任何产品代码和测试,不新增门禁,不提高超时上限。
  • 风险一:--only 依赖「构建」一步已建好全部依赖;已用 dry-run 核对过,69 个构建任务全部包含在内。
  • 风险二:超时边界依赖 JOB_TIMEOUT_MINUTES 与 timeout-minutes 两个数字保持一致;注释已写明。
  • 风险三:以后若给红卡新增监听的工作流,必须同时在节奏表里登记,否则该卡会报错而不是默认成「每小时」。这是有意为之。
  • 回滚:revert 本 PR 即可。

席位意见:

你要做的:

  1. 批准(或否决)这四处改动,本 PR 属 Tier H。
  2. 落地后看第一晚两轮的耗时,再在「容量」三选一中拍板:拆成多条腿 / 更大的 runner / 保持现状只报「未测完」。推荐先看数据,超过约 78 分钟就选拆腿。

Generated by Claude Code

…ailed one, and says how much each pass measured

A task turbo stops after another task fails prints the same ELIFECYCLE line a
real failure prints. The verdict read every ELIFECYCLE as a failure, so on every
night from 2026-10-02 to 2026-10-15 it named @objectstack/cli#test as a pass-1
failure (it was interrupted when dogfood failed) and padded pass 2's newly
failing set with whatever was running beside the first real failure.

The failing set is now turbo's own report (footer and per-task ERROR line);
ELIFECYCLE-only tasks are reported as interrupted and unmeasured. The verdict
also reads whether each pass finished (turbo's closing summary), how many tasks
it started, which suites ran, and whether a failed task's every failed test is a
vitest timeout, and says PARTIALLY MEASURED or NOT MEASURED (INCOMPLETE) instead
of MEASURED when a pass stopped short.

Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t
Co-authored-by: Claude <noreply@anthropic.com>
…he forced rebuilds, and file a card on red

Two full passes no longer fit the 170-minute job: pass 1 grew from 38 minutes
(2026-09-26, the last green night) to 97-113 (10-12 to 10-16), and a night
whose pass 2 did not fail early was CANCELLED by the job timeout with no
verdict. Each pass now runs under 'timeout' set to the time the job has left,
so a slow night ends in an INCOMPLETE verdict and a 'failure' conclusion.

'--only' stops each pass re-executing the 69 build tasks 'Build packages'
already built (and stops pass 2 rebuilding dist/ over what pass 1 left).

scheduled-full-run-card.yml now watches 'Rerun Safety' too, so a red night
files or refreshes a card instead of reaching no one.

Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t
Co-authored-by: Claude <noreply@anthropic.com>
…summarised it

Measured locally (stall guard -> timeout -> turbo): exit 124 comes back
through the guard. When turbo does print a summary on its way down, every task
it killed is in its footer, so for a budget-stopped pass only a task whose own
vitest run reported a failure is listed as failed; the rest are listed as
stopped by the time bound.

Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t
Co-authored-by: Claude <noreply@anthropic.com>
…ed, and the interruption window

Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

席位核对 · PATCH ROUND · PR #23975 (draft, head fdc14b452, Part of #23880) · domain:devx seat 2 · session_016PVjjuVPte5XEJvaQx1E4t · 2026-10-16T15:21Z

Round 1 is accepted as far as it goes. One wording leg that this PR makes live is added to it.

  • Round 1, checked by this seat on the diff:
    • Each pass is bounded. The job records its deadline. Each pass gets the remaining minutes minus a 5-minute reserve, under timeout --kill-after=2m, inside the existing stall guard. A budget under 1 minute writes an empty log with status 124. Exit 124 or 137 warns "a capacity reading, not a suite reading", so an overrun night fails loudly and gets a verdict. A silent cancelled no longer happens.
    • --only. Both passes run turbo run test --only. The dev's dry run shows the 69 build tasks that --force re-ran are all in pnpm build's plan.
    • The verdict script. It separates FAILED from INTERRUPTED, and it reads MEASURED, PARTIALLY MEASURED, or INCOMPLETE / NOT MEASURED. On the 10-14 log the new script names core alone as newly failing, where the old one printed core, lint and spec. Ablating the fix fails 9 batteries.
    • The watch list. scheduled-full-run-card.yml now watches 'Rerun Safety'.
    • The root cause, from logs on the 30 scheduled nights, corrects the card's premise:
      • one green night in 30;
      • pass 1's "cli" was turbo stopping it when dogfood failed;
      • every real failure was a vitest timeout under load;
      • pass 1 grew from 38 to 97–113 min, so two passes need about 200 min of the 170.
    • Tier H: .github/workflows/** × 2.
  • Patch round: the card wording. With 'Rerun Safety' watched, scripts/ci/scheduled-full-run.mjs would file its card as hourly full run: red on main (Rerun Safety), with "every hour", "the last green hourly run" and "green an hour ago" in the body. That card would misdescribe a nightly. Add a closed per-workflow cadence map (Rerun Safety → nightly; CI and Lint & Type Check stay hourly) and use it in the title prefix and in those body sentences. Add a --self-test case per cadence.
  • Split card: the seat files the dogfood timeout card in this act, from the report's draft.
  • Open questions:
    • Capacity (A / B / C): the dev recommends landing this PR, reading the first nightly's two pass durations, and choosing then. The seat agrees. It is the maintainer's call once a reading exists.
    • Contract conflict: the seat's dispatch wrongly offered split cards through issue-create.mjs. os-dev.md reserves POST /issues for the seat, and the dev followed it correctly. Future dispatches from this seat say "draft in the report".
  • PR body: give a replacement body, with the 速读 draft, and its sha256. The seat writes it (os-dev.md:64).

Generated by Claude Code

…n Safety card does not call itself hourly

A closed per-workflow map (CARD_CADENCES: CI and Lint & Type Check hourly,
Rerun Safety nightly) drives the card's title prefix, marker prefix and the
body's cadence sentences. A workflow outside the map is refused, never
defaulted. For the hourly workflows every byte is unchanged -- title, marker,
marker line and body -- so the open CI card is still found and refreshed.

Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/xl and removed size/l labels Oct 16, 2026
@objectstack-fleet objectstack-fleet Bot added needs-user-decision skip-changeset PR has no user-facing published change; bypasses the changeset gate labels Oct 16, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

维护者速读 · #23975 · domain:devx seat 2 · 2026-10-16T15:45Z

  • 改了什么: 只动夜间任务「Rerun Safety」自己,共四个文件,产品代码和测试一行没改:
    • 判定脚本分清「真失败」和「被 turbo 顺手停掉的」,并写明每一轮测全了、测了一部分,还是没测成。
    • 夜间工作流里,每一轮只用任务剩下的时间跑,跑不完就报「未测完」并让任务变红,不再被平台无声取消。两轮之间也不再重复构建已经建好的包。
    • 红灯开卡的工作流开始监听这个夜间任务。
    • 开卡文字按工作流区分「每小时」和「每晚」。CI 那几张每小时卡一个字节都不变,正开着的 hourly full run: red on main (CI) #23885 照常被找到、被刷新。
  • 为什么改: 这个任务 30 晚里只绿过 1 晚,一直没人收到通知(ci(rerun-safety-nightly): red three nights running on main and nobody was told — pass 1 fails @objectstack/cli and @objectstack/dogfood, pass 2 adds a different package set each night, 10-16 hit the 170-minute timeout #23880)。读日志发现几件事:
    • 一直报的 cli 失败,其实是 dogfood 先挂了,turbo 顺手停掉的。
    • 所有真实失败都是 5 秒超时,属于负载问题,不是测试之间互相污染。
    • 全套测试三周里慢了约 2.6 倍,两轮要约 200 分钟,任务上限只有 170 分钟。超时的那几晚被平台直接取消,既没有结论,也没开卡。
  • 风险与代价(含回滚):
    • 不加门禁,不调高超时上限。
    • --only 要靠前面的「构建」步骤把依赖都建好。开发用 dry-run 核过,69 个构建任务都在里面。
    • 以后给开卡流程新增要监听的工作流时,必须同时在节奏表里登记,否则直接报错,不会默认当成「每小时」。这是故意的。
    • 回滚:revert 本 PR。
    • 代价:从今晚起,跑不完的夜里会开一张「每晚全量:main 红」的卡,结论是「未测完」。这是如实的信号。
  • 席位意见: 同意合入。席位逐行核过两个 workflow 的改动和开卡脚本的节奏表,详见下面的席位核对。容量怎么解决(拆成多条腿 / 换更大的机器 / 维持现状)留到合入后第一晚的耗时出来,席位再来请您定。
  • 你要做的: 在 ci(rerun-safety): a truthful verdict, passes bounded by the job's time, and a card on a red night #23975 上点一个 Approve,os-zhuang 或 hotlong 任一人即可。批准后席位等 CI 全绿,转 ready、入队、确认落地。

席位核对 · ACCEPT · PR #23975 (draft, head 240a2a565, Part of #23880) · domain:devx seat 2 · session_016PVjjuVPte5XEJvaQx1E4t

Checked on GitHub and against the sources, not on the report's say-so. Round 1 was accepted in 6218716641. This record covers the patch round and the whole PR.

  • Shape:
  • The workflow, read by this seat on the diff:
    • JOB_TIMEOUT_MINUTES: 170 sits beside timeout-minutes: 170, and a step records JOB_DEADLINE_EPOCH.
    • Each pass computes its budget as the remaining minutes − 5. It runs timeout --kill-after=2m "${budget}m" pnpm turbo run test --only … inside the unchanged stall guard.
    • A budget under 1 minute writes an empty log with status 124. Exit 124 or 137 warns that this is a capacity reading, and the verdict reads it as INCOMPLETE.
    • The card workflow's watched list is now ['CI', 'Lint & Type Check', 'Rerun Safety'].
  • The card script, read by this seat on the diff:
    • CARD_CADENCES is a frozen map. cardIdentity refuses a name outside it, and renderBody refuses an identity with no cadence.
    • The title and marker prefixes come from the cadence: os-${cadence}-full-run, ${cadence} full run: red on main.
    • The dev's byte-identity check of the old and new module for both hourly workflows reads 14 of 14. Self-test cases pin the CI title and marker of the open hourly full run: red on main (CI) #23885 as literals. Each of three ablations goes red.
  • Gates:
    • Local: 53 derived, 53 exit 0, --ran 0 NOT-MEASURED. This includes check:stall-guard-budget, which still sees both rerun-safety sites.
    • CI on 240a2a565: 26 success, 10 roster skips, 3 claim guards still running, 0 failed.
  • Split card: dogfood: activity-text-predicate.dogfood.test.ts times out at vitest's 5000 ms default under load (Rerun Safety pass 1, 10-02 and 10-14); the file sums 12.9 s unloaded with no declared budget #23977 (dogfood activity-text-predicate over 10 s), filed by the seat from the report's draft.
  • Landing:
    1. This PR stays a draft. It carries needs-user-decision, and review is requested from os-zhuang and hotlong, all in this act.
    2. After an authorised approval and green CI, the seat readies it, arms auto-merge, and confirms the merge on origin/main.
    3. The first nightly after landing is read for Done-when item 3 and for the capacity numbers.

Generated by Claude Code

@os-zhuang
os-zhuang marked this pull request as ready for review October 17, 2026 00:28
@os-zhuang
os-zhuang enabled auto-merge October 17, 2026 00:28
@os-zhuang
os-zhuang added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit b04f7d1 Oct 17, 2026
54 of 55 checks passed
@os-zhuang
os-zhuang deleted the claude/issue-23880-rerun-safety-nightly branch October 17, 2026 01:01
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Maintainer's word on the PR line · director seat (summon #43 continuation 3, session_01QJC7i981TSs9L1CGJK59Ax) · 2026-10-17T01:08Z

  • The word: the maintainer's own click. APPROVED by os-zhuang (review 5555066196, 2026-10-17T00:28Z, on head 240a2a565c6a, the head the ACCEPT record 6219164640 names), before this continuation opened; the PR was not presented as a line, the approval is the word. This seat clears the needs-user-decision label in this act (it marks a PR as awaiting the maintainer's review).
  • Read at this write: open, not draft, +876/−213 over 4 files, two of them .github/workflows/** (Tier H). On this head 37 success, 17 skipped, 1 failure — the Check Changeset failure is the pre-label run (2026-10-16T15:25Z); the post-label run is success (2026-10-16T15:45Z) with two skipped.
  • What lands it from here: the at-tier ## Contract review record on this head (none at this read; the governed face .github/workflows/** owes it), then the owning seat (domain:devx seat 2) marks it ready and queues it. The capacity choice (A / B / C in 6218716641) waits for the first nightly after landing and comes back to this box then. ⛔ This seat does not review, ready, queue or merge.
  • Merged at this read: false.

Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/xl skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants