Repository navigation
ci(rerun-safety): a truthful verdict, passes bounded by the job's time, and a card on a red night - #23975
Merged
Conversation
…ailed one, and says how much each pass measured A task turbo stops after another task fails prints the same ELIFECYCLE line a real failure prints. The verdict read every ELIFECYCLE as a failure, so on every night from 2026-10-02 to 2026-10-15 it named @objectstack/cli#test as a pass-1 failure (it was interrupted when dogfood failed) and padded pass 2's newly failing set with whatever was running beside the first real failure. The failing set is now turbo's own report (footer and per-task ERROR line); ELIFECYCLE-only tasks are reported as interrupted and unmeasured. The verdict also reads whether each pass finished (turbo's closing summary), how many tasks it started, which suites ran, and whether a failed task's every failed test is a vitest timeout, and says PARTIALLY MEASURED or NOT MEASURED (INCOMPLETE) instead of MEASURED when a pass stopped short. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
…he forced rebuilds, and file a card on red Two full passes no longer fit the 170-minute job: pass 1 grew from 38 minutes (2026-09-26, the last green night) to 97-113 (10-12 to 10-16), and a night whose pass 2 did not fail early was CANCELLED by the job timeout with no verdict. Each pass now runs under 'timeout' set to the time the job has left, so a slow night ends in an INCOMPLETE verdict and a 'failure' conclusion. '--only' stops each pass re-executing the 69 build tasks 'Build packages' already built (and stops pass 2 rebuilding dist/ over what pass 1 left). scheduled-full-run-card.yml now watches 'Rerun Safety' too, so a red night files or refreshes a card instead of reaching no one. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
…summarised it Measured locally (stall guard -> timeout -> turbo): exit 124 comes back through the guard. When turbo does print a summary on its way down, every task it killed is in its footer, so for a budget-stopped pass only a task whose own vitest run reported a failure is listed as failed; the rest are listed as stopped by the time bound. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
…is red Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
…ed, and the interruption window Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
Contributor
Author
席位核对 · PATCH ROUND · PR #23975 (draft, head
|
…run-safety-nightly
…n Safety card does not call itself hourly A closed per-workflow map (CARD_CADENCES: CI and Lint & Type Check hourly, Rerun Safety nightly) drives the card's title prefix, marker prefix and the body's cadence sentences. A workflow outside the map is refused, never defaulted. For the hourly workflows every byte is unchanged -- title, marker, marker line and body -- so the open CI card is still found and refreshed. Claude-Session: https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t Co-authored-by: Claude <noreply@anthropic.com>
Contributor
Author
维护者速读 · #23975 ·
|
This was referenced Oct 16, 2026
os-zhuang
approved these changes
Oct 17, 2026
os-zhuang
marked this pull request as ready for review
October 17, 2026 00:28
os-zhuang
enabled auto-merge
October 17, 2026 00:28
Contributor
Author
|
Maintainer's word on the PR line · director seat (summon #43 continuation 3,
Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #23880
Clause-②: no
Rerun Safety(the nightly that runs the whole suite twice in one working tree) has concludedsuccesson one of its last thirty scheduled nights (2026-09-26), and nothing filed a card. This PR changes only that job's own shape and its reporting. No package code or package test is edited.scripts/rerun-safety-verdict.mjsnow separates a task turbo failed from a task turbo interrupted after another task failed. It also says how much of each pass actually ran: MEASURED, PARTIALLY MEASURED, or INCOMPLETE/NOT MEASURED. It flags a failure whose every failed test is a vitest timeout, and says how to tell load from a leak..github/workflows/rerun-safety-nightly.ymlhas three changes:timeoutset to the time the job has left, so a slow night ends in a verdict and afailureconclusion instead of a silentcancelled;--onlystops each pass re-executing the 69 build tasks thatBuild packagesjust built;.github/workflows/scheduled-full-run-card.ymladds'Rerun Safety'toworkflow_run.workflows, so a red night files or refreshes a card.scripts/ci/scheduled-full-run.mjs(patch round, claim6218695689) gives that card its workflow's cadence. A closed map,CARD_CADENCES, setsCIandLint & Type Checkto hourly andRerun Safetyto nightly; a workflow outside the map is refused, never defaulted. For the two hourly workflows every byte the card carries is unchanged, so the open CI card hourly full run: red on main (CI) #23885 is still found and refreshed.Why
Part of: Done-when item 3 ("a nightly run concludessuccess, or names only failures that already have cards") can only be read on a nightly after this lands. Two full passes also no longer fit one 4-vCPU runner in 170 minutes. That capacity choice belongs to the maintainer, after the first nightly reading; it is set out under Capacity below. #23880 remains open for it.Readings: the failing tests, per night and pass
Source: each run's job log via MCP
get_job_logs. That tool returns the last 5,000 lines. The REST log zip answers 403 from this container (results-receiver.actions.githubusercontent.com,connect_rejected). A full log is 290k to 436k lines, so pass 1's detail is readable only on nights whose pass 2 stopped early. The other nights are marked NOT MEASURED.Column meanings:
Failed:footer and per-taskERROR command … exited (1)line.ELIFECYCLE Test failed., 0.1 to 0.5 s after the real failure's, with no ERROR line and no footer entry.activity-parent-read-gate.dogfood.test.ts:102"list: every row returned is about a record the member can open…";activity-text-predicate.dogfood.test.ts:465"control: the unmasking reader filters and groups by every column…";audit-log-parent-read-gate.dogfood.test.ts:144"list: every row returned that names a record…". Each:Error: Test timed out in 5000ms.src/security/platform-admin.test.ts:278"nor by@objectstack/core's security entry — the published surface lost them" (a dynamicimport('./index.js')):Test timed out in 5000ms.src/declared-capability-absence-warn-once.test.ts"the dispatcher door > reports a declared 501 ONCE across N requests…":Test timed out in 30000ms.activity-text-predicate.dogfood.test.ts:470"control: the unmasking reader filters and groups by every column of a pinned parent, as before":Test timed out in 5000ms.src/security/auth-gate.test.ts:571"BOUNDARY — newly exempt is exactly POST on the four-segment hook route…":Test timed out in 5000ms.src/domains/share-links-refusal-order.test.ts:356"email audience: the refusal is the one an unknown token and a password-less email link answer":Test timed out in 5000ms.Received SIGINT), cli, organizationsEvery real failure in the table is a vitest timeout. The pass-2 "newly failing" sets the old verdict printed are not rerun-safety findings either:
Each was one real timeout plus the suites turbo stopped beside it.
Pass 1's root cause
@objectstack/cli#testis not a pass-1 failure. On both nights whose pass 1 is readable (10-02, 10-14), turbo's footer names@objectstack/dogfood#testalone. cli printed ELIFECYCLE 0.16 to 0.23 s later, with no ERROR line: turbo, which runs without--continue, stopped it when dogfood failed. 10-13 and 10-15 print the same verdict shape; their detail is NOT MEASURED. The old verdict read every ELIFECYCLE as a failure. That was the defect, and this PR removes it.@objectstack/dogfood#testis vitest 5000 ms timeouts under this job's load. I ran the failing files locally, unloaded, on an idle 4-core container at98ba1bf1d2. All passed:Dogfoodjob runs dogfood alone, in three slices, one per 4-vCPU runner.--concurrency=4, each task at vitest's default of three workers, so twelve workers share four vCPUs.The pass-2-only packages: each passes a second run in one tree
Measured locally at
98ba1bf1d2, unloaded,--maxWorkers=2, two consecutive runs in the same worktree:@objectstack/core@objectstack/runtime(--project local)After both runs,
git statusshowed no new or ignored state beyonddist/andnode_modules/.The specific tests, unloaded:
The 10-13 warn-once test ran 79 ms here and hit its own 30 s bound in pass 2. That gap is beyond plain CPU contention. What held it is NOT MEASURED. It is one occurrence that passes twice locally, so it is noted here and not carded.
The timeout's cause: wall-time growth, not a hang, not one new suite
scripts/test-shard-timings.json:@objectstack/clialone: 733 s, 1,703, 1,739, 2,109, 2,206.So two full passes need roughly 200 minutes of a 170-minute job. A night "finished" only because pass 2 stopped at its first failure: on 10-14 it ran 7 minutes and started 49 of 143 tasks. A night that did not fail early was
cancelled, with no verdict. Raisingtimeout-minutesis ruled out by triage's scope and the test-budget rulings, and this PR does not raise it.What this PR does about it
budget = (deadline − now)/60 − PASS_RESERVE_MINUTES, withJOB_TIMEOUT_MINUTES: 170set besidetimeout-minutes: 170.timeout --kill-after=2m "${budget}m"and sits inside the stall guard, as the command it wraps.timeout 45s, then turbo over core, types and formula): exit124came back through the guard. The log ended atShutting down Turborepo tasks.../Force killed Turborepo tasks: @objectstack/core#test, with noTasks:summary. No process was left behind.INCOMPLETE, rerun propertyNOT MEASURED, exit 1. The job concludesfailure, which the card filer reads;cancelledit does not read.--onlyon both passes.turbo run test --force --dry=jsonat98ba1bf1d2plans 83test, 69buildand 1@objectstack/spec#gen:migration-registrytask.pnpm build's own plan (Build packages).testscript runspnpm gen:migration-registryitself.--only, a pass runs only the 83 test tasks, and pass 2 no longer rebuildsdist/over what pass 1 left there.Old (what the job printed):
New:
The 10-02 log (run 36962761984) reads the same way: newly failing
corealone, notcore, driver-sql, lint, spec. Pass 2 started 42 of 138 tasks.The reporting change: a card for a red night, in the night's own cadence
scheduled-full-run-card.ymlnow lists'Rerun Safety'. I measured the existing mechanism against a once-a-day workflow:failure || timed_out, but a job timeout here concludedcancelled: runs 38848504327 and 38202695171 arecancelledat both run and job level. That gap is closed from the job side by change 1: an overrun night now concludesfailure.scripts/ci/scheduled-full-run.mjs:CARD_CADENCESis a closed map from the byte-exact workflow name to its cadence:CIhourly,Lint & Type Checkhourly,Rerun Safetynightly.cardIdentityrefuses a name outside it and names the map in the refusal, so adding a workflow to the card's watched list means adding its cadence in the same change.renderBody: the marker line, the "runs onmain…" paragraph, the remedy lines, the "What landed since the last green … run" heading, and the empty-range flake reading.Rerun Safetytherefore filesnightly full run: red on main (Rerun Safety)with markeros-nightly-full-run-rerun-safety. Its body says "every night", "the last green nightly run" and "green the night before", and has no hourly wording.hourly full run: red on main (CI), markeros-hourly-full-run-ci, and the same marker line and body.renderBodyalso refuses an identity built by hand without a cadence, rather than default one.Rerun Safetystarts at its last greenschedulerun, 09-26 (run 36216601546). The compare API caps at 250 commits, and the body already says when it truncated.Capacity: the maintainer's choice, after the first nightly reading
Under this PR, a night whose two passes do not fit gets a loud INCOMPLETE and no longer a silent
cancelled. It is still not a measurement. Once the first nightly after landing prints its two pass durations (the budget lines print them), there are three ways to make two full passes fit again:scripts/partition-test-shards.mjswith cli and dogfood excluded), each running its own package set twice in its own tree.runs-onin this repo isubuntu-latest.Recommendation, after that reading: if pass 1 still exceeds about 78 minutes, choose A. That threshold is 170 minus 7 minutes of setup and build minus the 5-minute reserve, halved.
pnpm test只能跑绿一次:InMemoryDriver 默认把数据落盘,datasource-autoconnect 测试在第二次运行必然失败 #4065) wrote.objectstack/data/memory-driver.jsonunder the suite's own working directory, and A keeps that.Split cards (self-split rule)
One lane card was warranted. The seat filed it from the dev's draft in the report on #23880:
packages/qa/dogfood/test/activity-text-predicate.dogfood.test.ts. Its 22 tests sum to 12.9 s unloaded, over the 10 s per-file rule (ruling record 6181973363), and it declares no budget. Its "control: the unmasking reader filters and groups by every column…" test runs 3,377 ms of vitest's 5,000 ms default on an idle 4-core box. It timed out in this job's pass 1 on 10-02 and 10-14.The other timed-out tests are not carded. core and runtime each pass a second run in one tree. The 10-02 dogfood list tests run about 1 s unloaded and are load-only.
Verification
The branch merged
origin/mainat68ffa5dc1(a merge, no rebase). The gate union ran after the final commit, at240a2a565.Round 1 (verdict and nightly):
pnpm check:rerun-safety-verdict:rerun-safety-verdict --self-test: OK (23 batteries, 23 registered), exit 0. The fixtures are the 10-14 and 10-15 log shapes, colour codes included.scripts/ablation-replace.mjs:failing.add(id)on every ELIFECYCLE line. Landing proof: anchor 1 → 0, blob050a7fc6ae10→aec8629e130f.--self-test: 9 FAILED, including the end-to-end case printingNewly failing on pass 2: @objectstack/core#test, @objectstack/lint#test, @objectstack/spec#test.050a7fc6ae10),git diff HEADempty.ablation-replacerefused it as "a drop of 0", and that reading was discarded.Patch round (cadence):
node scripts/ci/scheduled-full-run.mjs --self-test:scheduled-full-run: self-test OK, exit 0. The identity battery now has 39 cases (28 before), and its floor rises from 23 to 34 for the eleven new ones:hourly full run: red on main (CI)andos-hourly-full-run-ci;Lint & Type Checkhourly;Rerun Safetynightly (title and marker);Spec Coverage) refused by name;renderBody;CIandLint & Type Check, title, marker, workflow name, labels and three body shapes match the module atb48f908ad. Result:byte-identical: 14 of 14.node scripts/ci/scheduled-full-run.mjs --check-concurrency: exit 0.240a2a565viascripts/ablation-replace.mjs, each restored to blob == HEADc33af68bc2eewithgit diff HEADempty. Every mutation landed (anchor 1 → 0) and each self-test went red:if (false …)in place of the map check) →Error: cadence: a workflow outside the map was not refused by name (accepted).CImapped to nightly →Error: identity: CI title is 'nightly full run: red on main (CI)'.everyword changed to'each hour'→Error: cadence: the CI marker line moved: …. The pre-existing cases do not pin this line; only the new marker-line pin catches it.Gates and tier:
240a2a565:node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderives 53 commands for this diff. Each was run with its exit code captured before any pipe, and all 53 exited 0.node scripts/ci/scheduled-full-run.mjs --self-test, andcheck:stall-guard-budget, which still sees both rerun-safety sites: window 15m, cap 30m, budget 170m, slack 140m.check:workflow-status-functions,check:agent-test-spelling,check:pm-dispatch-gates(fast tier, 1,884 cases, 320 s) andcheck:rerun-safety-verdict.--ranreconciliation:53 derived, 53 run, 0 NOT-MEASURED, 0 UNRUN, with a derived zero, because all 53 recorded an exit code.node scripts/pm/check-governed-merges.mjs --branch claude/issue-23880-rerun-safety-nightlyat240a2a565: GOVERNED, landing tier H,.github/workflows/**×2.58669da5e. Round 1 ended atfdc14b452with 908 (+713 / −195); the cadence leg adds the rest.Acceptance notes
Rerun Safetyrunstestand nottest:repo, while the rootpnpm testruns both. That is out of this card's scope, noted only.CARD_CADENCESandscheduled-full-run-card.yml'sworkflows:list are kept in step by the refusal, at the first red run of an unmapped workflow. No self-test reads the workflow file to compare them, deliberately: that would add a cross-file check to this gate family, and no new gate is in scope.--onlyand of the bound (this seat cannot dispatch the workflow);维护者速读(草稿)
改了什么:只动「Rerun Safety」夜间任务自己的四处。
为什么改:这个任务 30 晚里只绿过 1 晚,却没人收到通知。读日志发现:
cli失败其实是被 turbo 中途停掉的,真正失败的是 dogfood,而且都是 5 秒超时,属于负载问题。风险与代价(含回滚):
--only依赖「构建」一步已建好全部依赖;已用 dry-run 核对过,69 个构建任务全部包含在内。JOB_TIMEOUT_MINUTES与timeout-minutes两个数字保持一致;注释已写明。席位意见:
你要做的:
Generated by Claude Code