Repository navigation
Investigate organization-wide GitHub Actions queue saturation blocking current-head PR merges #714
Description
Incident
Multiple current-head PRs in ContextualWisdomLab/naruon are mergeable and have clean/resolved review evidence, but every repository-local required workflow remains in queued state instead of starting. This prevents the central OpenCode/Noema/Strix review and guarded merge scheduler from publishing current-head approvals.
Observed exact-head examples on 2026-08-03 UTC:
naruon#1192head8792334e9b85e901d72f7115bf2a439db61efbbb- Application CI
30786852446 - Bandit
30786852465 - Security Scan
30786852447 - Semgrep
30786852459 - Dependency Review
30786852443 - Docker validation
30786852577
- Application CI
naruon#1219head807967e61fc6d0c263eb5693e1a6fd64dce465b1- all six corresponding repository-local workflows remain queued
naruon#1215head712a76feb477fbda8cd9b20de8ec6078450766bb- Application CI
30789099991, with both backend and frontend jobs queued
- Application CI
GitHub's public status reports Actions operational, so the exact cause is not established. Plausible administrative causes include organization/repository concurrency exhaustion, spending or minutes limits, hosted-runner policy, or a runner-group capacity restriction. Do not treat any one cause as confirmed until the organization Actions settings and billing/audit logs are inspected.
Required investigation
- Inspect organization and repository Actions usage, concurrent-job limits, spending/minutes limits, runner-group access, and policy denials.
- Confirm whether public-repository hosted runners are eligible for these workflows and whether any enterprise policy is holding jobs before assignment.
- Check whether the large PR fan-out (six repository-local workflows plus central required workflows per head) should be serialized or consolidated without weakening required security/review contexts.
- Restore execution for the listed immutable heads, then verify the central 15-minute scheduler can publish independent reviews and merge them without manual bypass.
- Record the root cause and an operational guardrail/alert so a release-ready PR cannot remain queued without a concrete reason.
Safety boundary
Do not remove required checks, bypass repository rulesets, synthesize successful statuses, or weaken OpenCode/Noema/Strix/security evidence. This issue concerns runner assignment and queue capacity only.
Activity
seonghobae commentedon Aug 3, 2026
Concrete evidence from driving
semantic-data-portal's PR queue on 2026-08-03/04 (UTC), adding a second failure dimension beyond queue saturation — the OpenCode review model pool itself is exhausting, so even dispatches that do run fail closed:Timeline observed
pull_requestruns from pushes; brief processing bursts at ~15:50 and ~20:22).30798884116(sdp#34): failed at "Publish OpenCode review outcome" after ~2h.30855431448(sdp#348130be13, ran 21:37–22:35):OpenCode model-pool outcome=exhausted model=none→MODEL_OUTPUT_UNAVAILABLEfail-closed. Contributing causes seen in sibling runs: nvidia-nim429 Too Many Requestsand github-models410 github_models_retirement_brownout.gh: Resource not accessible by integration (HTTP 403)onPOST /repos/{target}/statuses/{sha}and on code-scanning alert reads, so the review gap is not surfaced as a commit status either.4847823477, APPROVE at 19:25) but was dismissed after the head/evidence rotation, so intermittent capacity is not converging to merges.Impact: semantic-data-portal has its whole queue (16 open PRs) green-checked with auto-merge armed; nothing can merge because current-head OpenCode approvals cannot be produced reliably. Relates to #707 (model pool needs an operational fallback with real capacity) and #691 (403 on cross-repo status publish).
Generated by Claude Code
seonghobae commentedon Aug 4, 2026
Additional exact-head evidence on 2026-08-04 UTC:
.github#728head8d698cf5bbf68e31ecaeb76f1b9044f7361ba5dd: OSV, Semgrep, CodeQL, Security Scan, Scorecard, Secret Scan, Python Security, and SBOM runs are all queued. Direct merge reports 15 required checks not succeeded plus independent approval required..github#727head2133b5e1725124e3f9b233b59559f5a3be5d5564: the same eight hosted workflows are queued.newsdom-api#467head4ba5e81d8dba98fc97ba2f6964751cc958b58c9e: all nine repository-owned checks succeeded, but the central OpenCode coverage evidence failed while provisioning the protected base lock (pypdf==6.13.3unavailable on a reachable index), creating a self-healing deadlock. Focused repairs are fix(ci): keep CodeQL Action versions consistent #728 (existing CodeQL contract failure on central main) followed by fix(ci): defer stale base-lock pins when the index is reachable #727 (defer only reachable-index stale pins).This reinforces that the queue incident affects new central control-plane repairs themselves, so runner/billing/policy inspection remains an external administrative blocker. Required checks and approval rules have not been bypassed.
seonghobae commentedon Aug 4, 2026
Additional exact-head evidence from
ContextualWisdomLab/scopeweaveon 2026-08-04 UTC:scopeweave#414head7cf119fc4a4aef7d488e92cd51d7298a9caf0b25308830901603088309012630883090138308830901943088309020630883090560scopeweave#420head63a8f18b305b33b7a8c1b12584aa28aff448990b30883729755/ job91910311152308837199523088372026630883720021308837199353088371995430883719996scopeweave#419head99dc57b383b0ed599230c474c6fd7d8d495de677has all six repository-local workflows and CodeRabbit green, a valid non-author approval, and auto-merge enabled, but direct merge reports2 of 5 required status checks are expected; ready-for-review/auto-merge events and a Dependabot recreate request have not yet produced the missing central review contexts.GitHub's public status page currently reports all systems, including Actions, operational and no incident on August 4. This strengthens the hypothesis that the blocker is organization/repository capacity, spending, runner policy, or queue fan-out rather than a declared GitHub-wide incident, but it does not establish which cause. No required check or protection has been bypassed.
seonghobae commentedon Aug 7, 2026
Consolidating this into #712 as the single organization-wide queue-starvation investigation. #712 already covers the same root-cause space (hosted-runner concurrency, billing/usage, organization policy, reusable-workflow/concurrency behavior, stale/superseded runs), requires a machine-readable health snapshot and fail-closed merge behavior, and is broad enough to retain the Naruon examples recorded here. Keep this issue as historical Naruon evidence; continue implementation and current diagnosis in #712 to avoid two competing queue-governance tracks.