Skip to content

fix(release): the stale-prompts sweep retries a transient 5xx, and defers with a warning instead of reading as main red - #28272

Merged
objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-28165-sweep-5xx-retry
Oct 31, 2026
Merged

objectstack-fleet[bot] merged 4 commits into
mainfrom
claude/issue-28165-sweep-5xx-retry

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW

Fixes #28165
Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:release.yml 的 stale-prompts 清扫(scripts/release-pending-publish.mjs sweep)读 GitHub API 遇到 5xx 或连接被重置时,按共享策略重试(每次读 3 次、间隔 500/1000 ms、整次运行最多 4 次重试);重试用尽仍失败时写一条 warning 和 job summary 小节后以 0 退出,不再把 main 的 Release 染红。
  • 为什么改:2026-10-30 一天内 main 的 Release 因同一步骤遇到平台 502 红了 5 次,读起来像 main 红(p0),掩盖真正的发版失败;清扫每次 push 都会重跑,一次没读到只是推迟,不丢任何东西。
  • 风险与回滚:改动限于 scripts/ 下三个文件(未发布,无 changeset):清扫脚本本身;新的叶子模块 scripts/board-read.mjs(从 check-issue-citations.mjs 原样搬出的重试策略,零 import);check-issue-citations.mjs 改为 re-export,原有导入方不变。4xx、其它错误、取消请求失败仍然让 job 失败;in-flight 发布守卫的读取语义不变。回滚即 revert 本 PR。
  • 先看:scripts/release-pending-publish.mjs 的 retryingReads / expectOk(重试与分类)和 sweep 里 catch TransientReadFailure 的分支(唯一以 0 退出的地方);scripts/board-read.mjs(确认是原样搬移、零 import)。
  • 两分钟验证:node scripts/release-pending-publish.mjs --self-test,预期末行 146 cases across 35 batteries pass。
  • 需要你做的:无,席位落地。

What changes

scripts/release-pending-publish.mjs, plus the shared retry policy moved into a leaf module (scripts/board-read.mjs) and re-exported from scripts/check-issue-citations.mjs. No workflow file is touched, so this PR is not Tier H.

  • Retry, through the shared policy, not a copy. The sweep's GETs now go through retryingReads, which wraps the existing HTTP layer in readBoard under BOARD_READ_RETRY. That is the same policy and reader the release gate's full-battery read (scripts/ci/scheduled-full-run.mjs readRuns) uses since the publish-gate twin of this card: 3 attempts per read, 500 / 1000 ms backoff, at most 4 retries for the whole run (at most 3 s of sleep a sweep, against a 10-minute job timeout). Retried: an HTTP 5xx, or a named connection failure (ECONNRESET, UND_ERR_SOCKET; the policy's closed list). Final at once: any 4xx, and any other thrown error.
  • The outcome after the retries: a warning and exit 0. A read still answering a transient failure comes back flagged, expectOk throws it as a TransientReadFailure, and sweep catches only that class. It prints one ::warning:: line on stderr naming the read, the last answer and the attempts, appends a NOT READ this run section to $GITHUB_STEP_SUMMARY, and returns 0. Nothing was judged, so nothing was cancelled, and the next push to main re-runs the sweep from scratch. This is the only exit-0 failure path.
  • What still fails the job: a 4xx, a body that does not add up, any other thrown error, and a cancel POST that does not answer 202/409. Writes are not retried: they pass through retryingReads unchanged.
  • The publish guard keeps its semantics. The in-flight mode (publishInFlight) still reads through the bare HTTP layer. An unreadable answer is in-flight / unreadable after one request, which a new case pins.
  • sweep now takes injectable env, http, versionCommits, npmStateOf, now, sleep, log and warn, and returns the exit code (main exits with it). The live defaults are unchanged.
  • The policy now lives in a leaf module. BOARD_READ_RETRY, RETRYABLE_NETWORK_CODES, retryableNetworkCode, boardReadBudget, readBoard and readFailure moved unchanged from check-issue-citations.mjs into scripts/board-read.mjs, which imports nothing. The only addition is export on sleepMs, which the citation gate's defaults read. check-issue-citations.mjs re-exports every name, so its importers (scheduled-full-run.mjs) are unchanged. Its self-test still drives the policy. The import sits beside the re-export, so the census-pinned marker on line 204 does not move. release-pending-publish.mjs imports the leaf, so its ./x.mjs closure is now board-read.mjs and invoked-as.mjs only. Importing the citation gate would have pulled in pm/check-half-states.mjs and the rest of its closure, which release-verify-npm.mjs --self-test cannot copy into its fixture repo. That was this PR's first CI red.

H2: why the sweep keeps listing completed run 40078805767

By design, so this PR leaves it as it is. collectWaitingRuns finds each version commit's push run by head_sha, walking the commits newest first, and stops at the first run older than WAITING_READ_BACK_DAYS (30). It then checks that each run it found carries that version's publish job:

const verify = carried && id !== String(currentRunId) && (r.status === 'waiting' || r.status === 'completed');

(scripts/release-pending-publish.mjs L1228). This came in with eead9dcf40, the commit that added the targeted read. Its message says: "It checks that each run it finds carries that version's publish job". The self-test pins it: the quiet case reads completed runs 301 and 302, and the wrong case flags completed run 501, which has no publish job.

Read-only GETs show:

  • 40078805767 is the push run of the 18.1.0 version commit e649c2d9 ("chore: version packages (chore: version packages #25606)"), created 2026-10-24, completed / success.
  • 38841989023, the run named at 13:42Z, is the push run of the previous version commit e5136ff1 ("chore: version packages (next) (chore: version packages (next) #23481)"), created 2026-10-16.

Both are inside the 30-day window. So every push re-reads the jobs listing of every version-commit push run from the last 30 days. 40078805767 is simply the newest of them, and it is read on every sweep until about 2026-11-23. The 502 lands on whichever of those jobs listings GitHub fails to answer. The run itself is not special: a public read of its jobs listing answered 200 with 6 jobs in 1.4 s. With this PR, a 502 on any of those reads is retried, and if it persists the sweep is deferred instead of turning red.

Tests

The self-test has a new battery: the sweep -> a transient read is retried under one budget, still failing it defers with a warning and exit 0, and a 4xx or a failed cancel stays red. It has a floor of 10, and the battery floor went from 34 to 35. Its cases drive the whole sweep offline with a stubbed http and a real summary file in a temp dir:

  • control: every read answers, so the stale prompt is cancelled, with exit 0, no backoff and no warning;
  • 502 then 200 on a carried completed run's jobs listing (the incident's read): retried once after 500 ms, then swept (cancel sent, exit 0);
  • 502 on every attempt: 3 attempts and 500 + 1000 ms, nothing cancelled, exit 0;
  • the same case says so: one ::warning:: line naming the read, HTTP 502 after 3 attempts, and a NOT READ summary section;
  • ECONNRESET then 200: retried, then swept;
  • an unnamed network error (ENOTFOUND): rethrown on the first attempt, so the job goes red;
  • 404: fatal at once, with one request, no backoff, nothing cancelled, and the error is not a deferral;
  • a flapping API: one budget for the whole sweep, 4 retries, then the fifth transient answer defers;
  • a cancel answering 502: sent once, not retried, exit 1 as before;
  • the publish guard (in-flight) on a 502: still unreadable, so in-flight, after one request.

The existing pin a probe answering non-200 -> the read throws (a red job) still passes unchanged on the bare layer.

self-test at 88c63cd
✓ release-pending-publish self-test: 146 cases across 35 batteries pass.
os-verify-lock: VERDICT command-exit 0

(The baseline at 361ea24 was 136 cases across 34 batteries.)

Ablations, both via scripts/ablation-replace.mjs, restored byte-identical
  1. Retry removed (if (method !== 'GET') return http(method, path); replaced by an unconditional pass-through): the mutation landed (anchor 1 to 0, blob 186cbaab511d to fb6db796de53), and the self-test went red with 5 failures. Among them were "a 502 then a 200 ... swept", "a 502 on every attempt ...", "a connection reset ... swept" and "a flapping API ...". Restored: blob equals HEAD 186cbaab511d, and git diff HEAD is empty.
  2. Deferral removed (the TransientReadFailure catch rethrows every error): the mutation landed, and the self-test went red with 3 failures ("a 502 on every attempt ...", "and it says so ...", "a flapping API ..."). Restored byte-identical to HEAD.

Acceptance notes

  • .github/workflows/release.yml's stale-prompts comment says "a failed read here must be loud (a red job on main)". After this PR, a read that keeps failing transiently is a ::warning:: annotation plus a summary section, not a red job. Every other failure is still red. The comment is in a Tier H file, so this PR does not edit it. carrier: none.
  • The policy's network list stays closed (ECONNRESET, UND_ERR_SOCKET). Other connection failures, such as ENOTFOUND and ETIMEDOUT, stay final for the sweep exactly as they do for the release gate. Widening the list is a change to the shared policy, not to this script.
  • No live sweep was run. All proof comes from the stubbed harness.
  • scripts/ci/scheduled-full-run.mjs still imports the policy through check-issue-citations.mjs, so the citation gate's PM closure still loads in the full-battery job. Switching it to ./board-read.mjs is a one-line follow-up, left out of this PR on purpose. carrier: none.

Verification

I ran the local gates at 88c63cd785, the final commit. The derived list came from node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack: 34 commands, and all 34 exited 0. They include node scripts/release-pending-publish.mjs --self-test, node scripts/check-issue-citations.mjs, pnpm check:issue-citations and pnpm check:pm-dispatch-gates (fast tier, 1937 cases). The dispatch-gates --ran reconciliation answered 34 derived famil(ies) accounted for — 34 run, 0 NOT-MEASURED (a DERIVED zero — all 34 recorded an exit code and none of them is 3). Outside the derived set, these also exited 0: node scripts/release-verify-npm.mjs --self-test (169 cases, 16 batteries), node scripts/check-issue-citations.mjs --self-test, node scripts/ci/scheduled-full-run.mjs --self-test, and eslint --no-inline-config on the three changed files. No changeset: scripts/ publishes nothing, hence skip-changeset.


Generated by Claude Code

…efers instead of failing when it stays down

The sweep's reads now go through the shared transient-read policy
(BOARD_READ_RETRY / readBoard from check-issue-citations.mjs, the one the
release gate's full-battery read uses): a 5xx or a named connection failure
is retried, 3 attempts per read at 500 / 1000 ms, at most 4 retries a run.
A 4xx and any other error stay fatal. A read still failing transiently after
that is a ::warning:: plus a job-summary section and exit 0: nothing was
judged or cancelled, and the next push re-runs the sweep. The cancel write
and the in-flight guard's reads are unchanged.

Claude-Session: https://claude.ai/code/session_01QSeKYmTYfaPbLQD4BJfomW
Co-Authored-By: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 31, 2026
…he citation gate's closure

The transient-read policy (BOARD_READ_RETRY, retryableNetworkCode,
boardReadBudget, readBoard, readFailure) moves unchanged out of
check-issue-citations.mjs into scripts/board-read.mjs, which imports
nothing. check-issue-citations.mjs re-exports every name, so its importers
are unchanged; release-pending-publish.mjs imports the leaf, so its ./x.mjs
closure no longer drags pm/check-half-states.mjs and friends in -- the
closure release-verify-npm's self-test copies into its fixture repo.

Claude-Session: https://claude.ai/code/session_01QSeKYmTYfaPbLQD4BJfomW
Co-Authored-By: Claude <noreply@anthropic.com>
…port, keeping the census-pinned marker line

Claude-Session: https://claude.ai/code/session_01QSeKYmTYfaPbLQD4BJfomW
Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/l and removed size/m labels Oct 31, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW

席位核对 — ACCEPT · domain:devx#2 · head 88c63cd785 · 2026-10-31T01:08Z

维护者速读

  • 改了什么:Release 里「取消已上 npm 版本的发布提示」那一步(release-pending-publish.mjs 的 sweep),读 GitHub Actions 接口时遇到 5xx 或连接重置,会重试 3 次(间隔 500、1000 ms,每次运行最多重试 4 次)。重试完仍读不到,就写一条警告和一段 NOT READ 摘要,以 exit 0 结束。4xx、其他错误、以及取消发布提示失败,仍然让 job 变红。
  • 为什么改:昨天 main 的 Release 因 GitHub 偶发 502 红了 5 次,看起来就像 main 红了,还会把真正的发布失败淹没在噪音里。这一步的工作,下一次 push 会再做一遍,所以偶发读失败不需要变红。
  • 风险与回滚:改 scripts/ 下三个文件(+475/−156):sweep 脚本;新的叶子模块 board-read.mjs(重试策略原样搬出,零 import);check-issue-citations.mjs 改为 re-export、原有引用方不变。不碰 workflow,也不发布任何包。发布门禁的读取仍走原路径、不重试,有测试钉住。回滚 = 还原本 PR。
  • 先看:retryingReads 和 sweep() 里只吞 TransientReadFailure 的那个 catch。
  • 两分钟验证:node scripts/release-pending-publish.mjs --self-test 应输出 146 个用例全过。其中新增一组覆盖:502 后 200 能清掉;三次 502 时 exit 0 并告警;404 照旧报错;取消提示失败照旧红。
  • 需要你做的:无。席位落地。

席位意见:合入。终局按分诊卡定:重试耗尽后告警并 exit 0。重试策略直接 import 发布门禁那张孪生卡留下的共享 helper,没有复制第二份。分诊问的第二个问题,dev 实测结论是按设计如此,不是缺陷,理由见下。

Shape: draft → main, Fixes #28165. 3 files, +475/−156. Label skip-changeset. Non-governed (scripts/ only, no workflow); the seat lands it. Report 6536464560, claim 6536149684.

Checked against the diff:

  • Retry: retryingReads wraps only GETs. A write such as the cancel POST goes straight to http. It reads through readBoard / boardReadBudget from the new leaf scripts/board-read.mjs, which imports nothing. The seat diffed it: 124 of the 125 lines removed from check-issue-citations.mjs are in the leaf unchanged, and the 125th (sleepMs) only gained export. check-issue-citations.mjs re-exports all six names, so scheduled-full-run.mjs is unchanged.
  • Deferral: expectOk throws TransientReadFailure only for an unread transient result. sweep() catches only that class; anything else rethrows, so it stays red.
  • Gate unchanged: publishInFlight keeps reading through the bare layer, pinned by "in-flight 502 → unreadable after one request".
  • Teeth: ablation 1 removed the retry and 5 cases went red, including 502→200. Ablation 2 removed the deferral and 3 went red. The file was restored byte-identical both times.
  • Triage's second question (H2), by design: verify at ~L1228 came in with eead9dcf40 ("checks that each run it finds carries that version's publish job"). Every push re-lists the version-commit push runs inside WAITING_READ_BACK_DAYS = 30, and 40078805767 (the 18.1.0 version commit's push run) is the newest of them. The 502 lands on whichever listing GitHub fails, not on a special run: a read-only GET of its jobs answered 200 with 6 jobs.

First CI red, fixed in this PR: at dd93d69376 the PR imported the citation gate directly. Its closure (pm/check-half-states.mjs → ./seat-record.mjs) broke release-verify-npm.mjs --self-test, which the seat reproduced locally with an ENOENT and an exit 1 (exit 0 on main). The leaf module fixes it. At 88c63cd785 the seat re-ran release-verify-npm (exit 0), release-pending-publish (146 cases) and check-issue-citations (exit 0) self-tests locally.

Gates: on head 88c63cd785, 46 check-runs: 27 success, 19 skipped, 0 failure. Lint · CI, release & engine gates is green again (it was red at dd93d69376). mergeable_state: clean. check-expected-skips --pr 28272: OK, all 19 skips are in main's roster (exit 0). check-governed-merges --pr 28272: NOT governed, 631 changed lines. Dev's union at 88c63cd785: 34 derived, 34 run, all exit 0.

Out of scope / Acceptance notes: .github/workflows/release.yml:1455 says the stale-prompts job's "failed read here must be loud (a red job on main)". After this PR that holds for every failure except a transient read that keeps failing past the retries. The note is comment-only drift in a Tier H file, so it is recorded here and rides the next release.yml change. It is not worth a governed PR of its own. Also noted: scheduled-full-run.mjs still loads the policy through the citation gate. Switching it to the leaf is a one-line follow-up and is left out of this PR. Skills-docs: n/a.

Landing: non-governed, 631 changed lines. In this act the seat readies the PR and arms auto-merge. On merge, Fixes #28165 closes the card. The seat verifies the merged diff, and the next main push's Release run is the live reading.


Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 31, 2026 01:09
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 31, 2026 01:09
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit c5bf56b Oct 31, 2026
47 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-28165-sweep-5xx-retry branch October 31, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/l skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants