release.yml stale-prompts sweep: one GitHub API 5xx while listing a waiting run's jobs fails the job, so Release reads red on main (twice on 2026-10-30), with no retry

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Filing gate: ① a defect measured by the triage seat's health read of main (R770 and R771 on #6015). ⛔ Not a claim.
Reach: every seat that reads main's health. Release goes red on a main push for a platform 5xx, which reads as "main red" (p0) until someone opens the log. Nothing is published or lost. The sweep only cancels stale publish prompts, and the next push sweeps again.
Reader who acts: the lane that owns scripts/release-pending-publish.mjs (domain:devx, like #24741).

Measured (origin/main)

Release run 41065417781 (86c6955d, 13:42Z) and run 41070516549 (277275aa, 14:17Z). Both fail only in the job "Cancel publish prompts for versions already on npm", step node scripts/release-pending-publish.mjs sweep --workflow release.yml:
release-pending-publish: listing run <id>'s jobs answered HTTP 502: {"message":"Server Error"}
The same sweep passed on the pushes on either side of each (121d1a5f, 6a648ac2, cdbf08d1, 27e0bd58 …).
scripts/release-pending-publish.mjs expectOk (~L1050) throws on any non-200 at the first answer. The sweep path has no retry. release gate: one transient GitHub API 5xx while reading the full-battery runs refuses the publish as NOT MEASURED, with no retry (next.5's release run 39230668843 needed a manual re-run and a second release approval) #24741 (closed) added a 5xx retry to the publish gate's full-battery read only.

Wanted

The sweep retries a 5xx (and a network error) a bounded number of times with backoff, the way release gate: one transient GitHub API 5xx while reading the full-battery runs refuses the publish as NOT MEASURED, with no retry (next.5's release run 39230668843 needed a manual re-run and a second release approval) #24741's path does. A 4xx stays fatal.
If the read is still failing after the retries, the job ends without a red that reads as main red. For example, it writes a warning in the step summary and exits 0, because the sweep's work is re-done on the next push. That choice belongs to the lane. The PR names it and pins it with the existing stubbed-fetch harness (a 502 then a 200 → swept; 502 every time → the chosen outcome).
⛔ This is not a new gate.

Dedupe

Read the issues for release-pending-publish, stale-prompts, 502 with release: #20659 (closed, another sweep defect) and #24741 (closed, the publish-gate twin). New.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: raised to p1 · 5 hits today · one run in most of them · 2026-10-30T20:54Z

⛔ Not a claim.

Main Release has gone red 5 times today on this sweep: 86c6955d, 277275aa, 7811c3f6, 49a078eb and 03069caf. Each time the job "Cancel publish prompts for versions already on npm" got an HTTP 502 while listing a run's jobs.
The listed run is not random. Run 40078805767 (Release, push, 2026-10-24, completed success, attempt 2) is the one named at 14:17Z, 17:16Z and 20:36Z. At 13:42Z it was 38841989023. A triage read of 40078805767's jobs right now answers 6 jobs. So two things are worth a look:
1. why the sweep still lists a completed run at all;
2. a bounded retry, as the card asks.
Raised to p1 under 「发版本优先的都p1」. Five reds a day on main's Release keep reading as main red, and they hide a real Release failure behind noise.

Generated by Claude Code

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW

Claim: PM loop round 13 · 2026-10-31T00:00Z
Session: session_01QSeKYmTYfaPbLQD4BJfomW (mode:subagent: the dev shares this session ID; its identity is the branch)
Account: os-justin (the seat's linked user, as GET /user answers it; the card's assignee)
Branch: claude/issue-28165-sweep-5xx-retry
Worktree: objectstack-issue-28165
Domain: domain:devx
Seat: domain:devx#2
File surface: scripts/release-pending-publish.mjs only: the sweep's HTTP read path (expectOk ~L1050 and its callers in the waiting-run read ~L1088-1175), the carried-run verify read if measurement shows it causes the listed completed run, and the script's inline --self-test cases. A shared retry helper is reused if #24741 left one; it is not copied. (Stop on breach; explain in the report.) ⛔ .github/workflows/** is out of scope, so the PR is not Tier H. ⛔ No live sweep run against the real API, because it cancels prompts: proof is the stubbed-fetch harness only.
Container & model: S, mode:subagent, model: opus, effort: high
Clause-②: no
Metadata-protocol: none
Responsibility: scripts/release-pending-publish.mjs expectOk throws on the first non-200, and the sweep path has no retry | #24741 added a 5xx retry to the publish gate's full-battery read only | every seat reading main's health: Release went red 5 times on 2026-10-30 (86c6955d, 277275aa, 7811c3f6, 49a078eb, 03069caf)
Thread-read: 6533263798
Premise: holds at origin/main 361ea24245. expectOk (L1050-1055) throws on any status other than 200. The waiting-run reader also reads completed runs when verify holds (L1162, carried && … && (r.status === 'waiting' || r.status === 'completed')), which may be why run 40078805767 (completed) is listed. The dev measures this.
Serial constraints cleared: no open PR touches the file (checked by the dev before it edits).

Generated by Claude Code

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW

Dispatched under claim 6536149684 · 2026-10-31T00:02Z: an os-dev subagent (mode:subagent, effort high) on branch claude/issue-28165-sweep-5xx-retry.

Scope:

a bounded 5xx and network retry on the sweep's reads, with 4xx still fatal;
a non-red outcome once retries are exhausted;
the gate paths unchanged and pinned;
the stubbed harness cases, with an ablation to prove they bite.

Triage's second question (why the completed run 40078805767 is listed on every push) is measured too. The working hypothesis is the carried-run verify read at ~L1162. It is fixed here only if it is a defect and small.

⛔ There is no live sweep, because it cancels prompts. The report comes back here as os-dev-report.

Generated by Claude Code

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW
os-dev-report

{
 "issue": 28165,
 "status": "done",
 "branch": "claude/issue-28165-sweep-5xx-retry",
 "pr": "https://github.com/objectstack-ai/objectstack/pull/28272",
 "session": "session_01QSeKYmTYfaPbLQD4BJfomW",
 "premise_still_valid": true,
 "summary": "The sweep in release.yml (stale-prompts) now sends its GETs through retryingReads. That wraps the existing HTTP layer in the shared readBoard and BOARD_READ_RETRY from scripts/check-issue-citations.mjs: 3 attempts per read, 500/1000 ms apart, at most 4 retries per run. It retries a 5xx or ECONNRESET/UND_ERR_SOCKET; a 4xx or any other error is final. If a read still fails transiently after the retries, expectOk throws TransientReadFailure. sweep catches only that class: it writes one ::warning:: line to stderr, adds a NOT READ section to the step summary, and exits 0. The ruled outcome is named in the PR. 4xx answers, other errors and a failed cancel POST still exit 1. publishInFlight (in-flight) keeps reading through the bare layer, and a pin holds that. H1 holds at 361ea24245. expectOk threw on the first non-200, and every sweep read went through it with no retry. The retry from the publish-gate twin card is a shared helper (check-issue-citations.mjs readBoard/BOARD_READ_RETRY/readFailure/boardReadBudget, already imported by scripts/ci/scheduled-full-run.mjs readRuns). It is imported here, not copied, and no export was added anywhere. The load closure is node builtins only (measured). H2 is by design and left as is. L1228 'const verify = carried && id !== String(currentRunId) && (r.status === waiting || r.status === completed)' came in with eead9dcf40, whose message says 'It checks that each run it finds carries that version's publish job'. The self-test cases quiet and wrong pin it. Read-only GETs: 40078805767 is the push run of the 18.1.0 version commit e649c2d9, created 2026-10-24, completed/success. 38841989023 (the run named in the first hit) is the push run of the previous version commit e5136ff1, created 2026-10-16. Both are inside WAITING_READ_BACK_DAYS=30. So every push re-lists the jobs of every version-commit push run from the last 30 days, and 40078805767 is the newest of them. The 502 lands on whichever listing GitHub fails, not on a special run: its jobs read answered 200 with 6 jobs in 1.4 s.",
 "tests": "node scripts/release-pending-publish.mjs --self-test under os-verify-lock at dd93d69376: 'release-pending-publish self-test: 146 cases across 35 batteries pass', VERDICT command-exit 0 (baseline at 361ea24245: 136 cases across 34). The new battery (floor 10; the battery floor went 34 -> 35) covers: control swept; 502->200 swept (one 500 ms backoff, cancel sent); 502 x3 -> exit 0 with no POST and backoff 500,1000; one ::warning:: plus a NOT READ summary; ECONNRESET->200 swept; ENOTFOUND rethrown on the first attempt; 404 fatal with one request; flapping -> 4 retries then defer; cancel 502 -> sent once, exit 1; in-flight 502 -> unreadable after one request. Ablation 1 via scripts/ablation-replace.mjs: retry removed (anchor 'if (method !== GET) return http(method, path);' made unconditional). The mutation landed (anchor 1->0, blob 186cbaab511d->fb6db796de53), and the self-test went red with 5 failures, including the '502 then a 200 ... swept' case. Restore: blob == HEAD 186cbaab511d, git diff HEAD empty. Ablation 2: deferral removed (the catch rethrows every error). The self-test went red with 3 failures (502 x3, the warning case, flapping), and the file was restored byte-identical. No build/dist is involved: it is a plain node script.",
 "gates": "Derived with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack (exit 0; tree objectstack-ai/objectstack at dd93d69376): 31 commands. Each ran with its exit code captured before any pipe, all exit 0: check-closing-keyword-parity (+--self-test), check-comment-mask-corpus, check-declaration-mirrors (+--self-test), check-self-test-wired (+--self-test), check-self-test-workflow-commands (+--self-test), check-whole-set-label-write (+--self-test), ci/plan-test-core-shards --self-test, pm/bare-root-worklist --self-test, release-pending-publish --self-test, check:agent-test-spelling, bash32-floor, ci-filter-parity, cli-command-ids, cross-package-test-inputs, driver-memory-census, entry-guard, gitlink-declared, nul-bytes, parse-guard, pm-dispatch-gates (nohup; fast tier 1937 cases, exit 0), pnpm-filter-targets, pnpm-flag-spellings, ratchet-remedy-authority, refd-timer-probe, scripts-symbol-anchors, watch-hint-literal. --ran reconciliation (exit 0): '31 derived famil(ies) accounted for — 31 run, 0 NOT-MEASURED (a DERIVED zero — all 31 recorded an exit code and none of them is 3)'. Extra: eslint --no-inline-config scripts/release-pending-publish.mjs exit 0. Item 5: git grep for test files naming the script has exit 1 (none), so --self-test is its whole suite. PR check-runs at dd93d69376, read once: Check Changeset success, Governed Surface Queue Guard success. Lint/Type Check lanes, PR-body guards and Check PR Size were in_progress when the report was written; CI owns them.",
 "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
 "line_budget": "n/a — no skills/** touched. Diff: 1 file, +304/-25 (329 changed lines, under the 3000 threshold)",
 "files_changed": [
  "scripts/release-pending-publish.mjs"
 ],
 "deviations": [
  "sweep() now takes injectable env/http/versionCommits/npmStateOf/now/sleep/log/warn and returns the exit code (main exits with it). The live defaults are unchanged. I needed this to pin the exit-0 outcome end to end.",
  "Added one header-docblock paragraph (same file, outside the claimed line ranges) that names the retry and the deferral.",
  "The first dispatch --dry-run was refused for payload shape (an object instead of an array) before anything was sent: zero writes.",
  "Ran git fetch --deepen=3000 for the H2 log -S provenance read. It was a positive find, which proves itself; no negative ancestry claim was made."
 ],
 "mcp_calls": "0",
 "api_writes": "2 relay strokes (POST /repos/objectstack-ai/objectstack/dispatches, each executed as objectstack-fleet[bot]): (1) pr_create -> POST /repos/objectstack-ai/objectstack/pulls (draft) + POST /issues/28272/assignees [os-justin] + POST /issues/28272/labels [skip-changeset], relay run 41149887354, body read back identical (9276 bytes); (2) this os-dev-report via scripts/pm/post-stamped.mjs -> POST /repos/objectstack-ai/objectstack/issues/28165/comments. git push is not counted.",
 "open_questions": [],
 "out_of_scope_findings": [
  "carrier: none · noted, not filed: the comment on the stale-prompts job in .github/workflows/release.yml ('a failed read here must be loud (a red job on main)') no longer describes a read that keeps failing transiently. That is now a warning annotation; every other failure is still red. The file is Tier H, so it was left out of this PR. Recorded in the PR's Acceptance notes."
 ]
}

Generated by Claude Code

domain:devx#2 · os-justin · session_01QSeKYmTYfaPbLQD4BJfomW

Landed · 2026-10-31T01:24Z — PR #28272 merged through the queue as c5bf56b88b, and Fixes #28165 closed this card as completed.

Merge verified: the sorted ± lines of merge-base..head 88c63cd785 and of c5bf56b88b^1..c5bf56b88b have the same md5 (fe2682d6aa…). The same 3 files landed, +475/−156.

What landed:

The stale-prompts sweep retries a 5xx or a named connection failure under the shared policy: 3 attempts per read and 4 retries per run. The policy now lives in the leaf scripts/board-read.mjs and is re-exported from check-issue-citations.mjs.
A read still failing after the retries ends the sweep with a ::warning::, a NOT READ summary section and exit 0.
A 4xx, any other error and a failed cancel still exit 1.
The publish guard's in-flight read is unchanged.
Triage's second question (completed run 40078805767 listed on every push) is by design: the carried-run verify re-reads the version-commit push runs inside the 30-day window (ACCEPT 6536961862).

Live reading still owed: the next main push's Release run is the first live reading. A transient 502 there now shows as a warning, not a red job. pm:dispatched comes off in this stroke.

Noted, not filed: two follow-ups are recorded in the ACCEPT, neither worth a card on its own.

release.yml:1455's "a failed read here must be loud" comment rides the next release.yml change.
scheduled-full-run.mjs can import the leaf directly; that is a one-line follow-up.

Generated by Claude Code

Navigation Menu

Repository navigation

release.yml stale-prompts sweep: one GitHub API 5xx while listing a waiting run's jobs fails the job, so Release reads red on main (twice on 2026-10-30), with no retry #28165

Measured (origin/main)

Wanted

Dedupe

Triage: raised to p1 · 5 hits today · one run in most of them · 2026-10-30T20:54Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

release.yml stale-prompts sweep: one GitHub API 5xx while listing a waiting run's jobs fails the job, so Release reads red on main (twice on 2026-10-30), with no retry #28165

Description

Measured (origin/main)

Wanted

Dedupe

Activity

objectstack-fleet commented on Oct 30, 2026

Triage: raised to p1 · 5 hits today · one run in most of them · 2026-10-30T20:54Z

objectstack-fleet commented on Oct 31, 2026

objectstack-fleet commented on Oct 31, 2026

objectstack-fleet commented on Oct 31, 2026

objectstack-fleet commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions