ci: stage all workflows on isolated self-hosted runners

[seonghobae]

Current exact-head security scanner isolation receipt — 2026-10-07

• Exact head: 62c051bd2668cee351a1648aba3d8e72af774f20; tree 43059a6fbdd174ed63e6f4612af512f364402ef9; parent 98beb17c43072f7f749ac8641d23db044436c055.
• Publication: RED cc42969ba8ed4abeb1819412c146af2a9dadae8f → GREEN 26f993c0bbe88d50bf81473e31d179ad7b317657 → evidence 98beb17c43072f7f749ac8641d23db044436c055 → full-baseline restore 62c051bd2668cee351a1648aba3d8e72af774f20, each an ordinary one-parent fast-forward with force=false; no rebase or Force Push.

RCA → RED → GREEN

Security run 37242578328 exposed two independent infrastructure false failures. Trivy job 112314054734 restored trivy-binary-v0.70.0-Linux-X64 but then exited 127 with trivy: command not found, so the heterogeneous isolated runner group may not reuse runner-local binary cache state. OSV job 112314054619 completed the base/head comparison with no PR-introduced finding and uploaded SARIF before the optional debug artifact failed because the GitHub artifact quota was exhausted.

RED adds contracts for both boundaries and failed 2/2 on the prior source. GREEN disables only the Trivy binary cache and makes only OSV debug retention non-blocking; scanner, reporter, finding, and SARIF gates remain fail-closed.

Exact-tree executable evidence

• focused contracts: 2 passed normally and 2 passed with GITHUB_ACTIONS=true
• warning-fatal repository suite at GREEN tree: 5,488 passed, 10 skipped, 40 subtests
• evidence contracts after CHANGELOG/Gap update: 9 passed, 4 subtests
• compileall, git diff --check, and clean worktree: GREEN
• docs/product-technical-gap-baseline.md (373,083 bytes; blob b50add6bef62fc4eb083a4a3e280e5c672995c51) and CHANGELOG carry the exact run/job/RCA/RED/GREEN receipt
• exact-tree transfer verification caught a truncated intermediate Gap blob and 62c051bd… restored the complete document; remote and verified local trees now both equal 43059a6f…

Lifecycle

Draft / Proposed / merge HOLD. Fresh exact-head hosted Security/SAST/quality evidence, the isolated-capacity cleanup canary, zero unresolved review threads, and a qualifying independent approval remain required. This COMMENT/evidence is not approval. No blind rerun, runner relabeling, access expansion, bypass, auto-merge, or paid/model fallback is authorized.

---

Current exact-head provenance repair receipt — 2026-10-03

• Exact head: f0f7ed988fba5d94eda6513f35d171f23c4bd9a5; tree 3a1407a8bd162566a291a2adef6c7b006a0884ad.
• Ordinary fast-forward ancestry: RED 665321c169500029ede42112a2aa72b994f1ce16 adds the executable provenance contract; GREEN f0f7ed988fba5d94eda6513f35d171f23c4bd9a5 corrects both canonical receipts. Parent remains reviewed combined head 650bc6e35409d52d24fec890c2ab74063794e9b8; Git Data publication used force=false. No rebase or Force Push occurred.
• RCA / repair: Git Data publication changed the locally predicted merge object, but CHANGELOG and the Gap baseline retained nonexistent ec7c1b58…. Both now name published two-parent merge 650bc6e35409d52d24fec890c2ab74063794e9b8 (tree 9522f0b9c96247b3412e147bc6f89fd2595c0776) and its 53/53 focused / 5,437 passed, 10 skipped, 40 subtests historical receipt. The new contract requires both records to agree on a resolvable current-ancestry commit with exact parents cd2c3f97… and 22902728….
• Exact-tree local verification: warning-fatal repository suite 5,439 passed, 10 skipped, 40 subtests passed; dependency-review, ADR-identity, and evidence contracts 24/24 normally and with GITHUB_ACTIONS=true; Python compile, git diff --check, clean worktree, and exact tree match pass. The unavailable local ruff executable was not treated as evidence.
• Independent review: no blocking finding; provenance, topology, counts, and Proposed/HOLD wording verified. One optional future hardening remains: derive and compare the documented tree SHA rather than treating that tree string as receipt text.
• Fresh hosted exact-head evidence: CodeQL 37120498641 is Draft-skipped. Security 37120498599, Metadata 37120498646, SAST 37120498634, Trusted uv 37120498647, and Cloudflare 37120498635 failed before executable steps; every materialized failing job has steps=null and logs_url=null. This confirms the absent isolated runner group/capacity admission blocker and does not authorize a blind rerun.
• Review state: both threads are resolved/outdated; zero unresolved threads and zero qualifying approvals remain.

Lifecycle: Draft / Proposed / merge HOLD. Missing CWL CI isolated group/capacity, disposable cleanup/canary proof, passing exact-head hosted Checks, and a qualifying independent approval still block ordinary merge. No runner relabeling, access expansion, bypass, auto-merge, source-neutral rerun, predecessor closure, or merge is authorized.

Scope

User instruction: move this SDP workstream entirely to self-hosted runners. All 73 central job declarations retain dedicated control/CodeQL/OpenCode boundaries and replace every hosted fallback.

Blocking review finding and RCA

The predecessor exact head d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f selected ordinary/non-main work with the mutable cwlab-ci-isolated label alone, even though its rollout document correctly stated that a generic label does not attest separation. GitHub runner labels are routing metadata and GitHub does not validate that OS/architecture labels match the machine. A persistent or privileged runner with matching labels and repository access could therefore receive untrusted PR work.

The repair scopes every affected direct, conditional, CodeQL and R-matrix selector through the dedicated CWL CI isolated runner group and the existing platform/isolation labels. The group is the repository-access boundary; labels express required capability inside it. Trusted-main control/CodeQL/OpenCode groups and repository predicates are unchanged.

Historical predecessor evidence — cd49a60f1299fb8f7984de8d8ba223db673f7b84

This section is retained for audit history and is superseded by the current exact-head receipt above.

• RED: test_isolated_label_is_scoped_to_dedicated_runner_group failed first at .github/workflows/actions-queue-health.yml on the label-only selector.
• Repair: 63 affected selectors updated; durable all-workflow and workflow-specific contracts updated rather than removed.
• Focused: 42 passed.
• Complete repository verification at that predecessor: 5,426 passed, 10 skipped, 40 subtests, exit 0.
• All 38 workflow YAML files parsed successfully; git diff --check passed.
• Historical predecessor head: cd49a60f1299fb8f7984de8d8ba223db673f7b84
• Historical tree: 5cbbaac086e19c94ac37abf2ca8a5914c88eac9b
• Parent: d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f (ordinary one-parent fast-forward; no force push or rebase).

Safety and rollout hold

DRAFT / HOLD. The retained organization inventory had no cwlab-ci-isolated capacity and no verified CWL CI isolated group/capacity record. Do not add a privileged runner to this group, relabel privileged runners, widen runner access, or merge until an authorized operator:

1. creates CWL CI isolated with least-privilege repository access;
2. provisions disposable isolated Linux/Windows/macOS capacity required by the matrices;
3. proves cleanup, tool availability, absence of privileged sockets/credentials/internal-network access, and separation from trusted control jobs;
4. returns a successful canary plus every exact-head required Check.

Source configuration is not runtime activation. No credentials, production deployment, existing runner ACLs, protections, inference budgets or active model runs were changed. Model-backed workflows remain fixed to orchestrator/free; no provider/model/group/paid fallback was added.

Historical hosted evidence — cd49a60f1299fb8f7984de8d8ba223db673f7b84

This predecessor evidence is superseded by the fresh cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba run receipt above.

At cd49a60f1299fb8f7984de8d8ba223db673f7b84, Cloudflare DNS, SAST Semgrep, Security Scan, Repository Metadata Reconcile and Trusted uv Materializer completed failure before any step existed; their job step lists are empty and job log blobs are absent. CodeQL PR was skipped. This was fail-closed pre-runner evidence consistent with the missing group/capacity prerequisite, not a passing runtime canary and not merge authorization. CodeRabbit status was success; no qualifying independent review or review threads existed.

The PR must remain Draft/HOLD until capacity and fresh exact-head hosted evidence are available.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Historical circular-admission receipt — 2026-10-03

Superseded: this receipt describes predecessor head d6ba56d04e22476b2ea41bfe7bb26e2cbc98f98f. The authoritative current-head receipt is in the PR body for cd2c3f9748e6bc50efd8bdfee9a385a74ec6e3ba.

• At that historical head, SAST 37094344652, Trusted uv 37094344666, Repository Metadata 37094344675, Security 37094344655, and Cloudflare DNS 37094344681 remained queued after every non-privileged route was pointed at the unprovisioned cwlab-ci-isolated label. Draft CodeQL 37094344661 was skipped.
• Canonical prerequisite owners remain linux-cluster-ops#326 for runner registration, repository access and operator evidence, and quarantine-sandbox-runtime#136 / #137 for host-side/LAN denial attestation.
• Keep Draft. Do not relabel a privileged central/control/OpenCode runner, substitute an arbitrary endpoint, blind-rerun, create a no-op event generation, or mark Ready before the owner prerequisites and canary are real.

[seonghobae]

Exact-head review for cd49a60f1299fb8f7984de8d8ba223db673f7b84 (tree 5cbbaac086e19c94ac37abf2ca8a5914c88eac9b):

Blocking predecessor defect repaired: label-only cwlab-ci-isolated routing did not establish the runner access boundary. All 63 isolated selectors now require the dedicated CWL CI isolated group plus their existing labels; trusted-main groups and predicates are preserved. RED was reproduced before repair, focused contracts pass 42/42, all 38 workflows parse, git diff --check passes, and the complete suite passes 5,426 tests with 10 explicit skips and 40 subtests.

This is a COMMENT, not approval. Hosted jobs at this exact head fail before step execution (empty step lists and absent job logs) because verified group/capacity is still missing. Keep Draft/HOLD. Required next evidence is operator-created least-privilege group, disposable capacity and cleanup proof, canary, fresh exact-head required Checks, then eligible independent approval and ordinary protected merge.

[seonghobae]

Exact-head COMMENT for 98beb17c43072f7f749ac8641d23db044436c055 / tree f20f58e28a0980b1401be6a54e36d9d5a011c228.

RCA: exact Security run 37242578328 mixed two infrastructure failures with scanner verdicts. Trivy job 112314054734 restored its pinned binary cache and then failed before scanning with trivy: command not found (127). OSV job 112314054619 completed the base/head comparison with no PR-introduced finding and uploaded SARIF, then failed only because optional debug-artifact storage quota was exhausted.

TDD publication is ordinary fast-forward only: RED cc42969ba8ed4abeb1819412c146af2a9dadae8f failed the two new contracts; GREEN 26f993c0bbe88d50bf81473e31d179ad7b317657 disables runner-local Trivy binary caching and makes only OSV debug retention non-blocking. Scanner, reporter, finding, and SARIF gates remain fail-closed. Evidence commit 98beb17c… updates CHANGELOG and the product/technical Gap baseline.

Verification: focused 2/2 normally and under GITHUB_ACTIONS=true; warning-fatal repository suite 5,488 passed, 10 skipped, 40 subtests; documentation/evidence contracts 9 passed plus 4 subtests; compileall and diff checks GREEN.

This is a COMMENT, not approval. Keep Draft / Proposed / HOLD pending fresh exact-head hosted Security/SAST/quality evidence, isolated-capacity cleanup canary, zero unresolved threads, and qualifying independent approval. No blind rerun, access expansion, runner relabeling, bypass, or merge.

[seonghobae]

Exact-head COMMENT for 62c051bd2668cee351a1648aba3d8e72af774f20 / tree 43059a6fbdd174ed63e6f4612af512f364402ef9.

The security RCA and RED→GREEN remain unchanged: RED cc42969ba8ed4abeb1819412c146af2a9dadae8f captures the runner-cache and optional-artifact failures; GREEN 26f993c0bbe88d50bf81473e31d179ad7b317657 disables Trivy binary caching and makes only OSV debug retention non-blocking while scanner/reporter/SARIF gates remain fail-closed.

Exact-tree verification then detected that intermediate evidence commit 98beb17c… carried a truncated product/technical Gap blob. The ordinary fast-forward corrective commit 62c051bd… restores the complete 373,083-byte baseline as blob b50add6bef62fc4eb083a4a3e280e5c672995c51. Fresh fetch proves the remote tree equals the fully verified local tree 43059a6f… byte-for-byte.

Final-tree verification: 5,488 passed, 10 skipped, 40 subtests; focused contracts 2/2 normally and under GITHUB_ACTIONS=true; compileall, diff check, clean worktree, remote/local tree equality, and full Gap blob size/hash all pass.

This is a COMMENT, not approval. Keep Draft / Proposed / HOLD pending fresh exact-head hosted Security/SAST/quality evidence, isolated-capacity cleanup canary, zero unresolved threads, and qualifying independent approval.

[seonghobae]

Exact-head owner repair review (COMMENT, not approval). RCA: mixed matrices silently filtered macOS/Windows/ARM rows and returned success whenever one Ubuntu row remained. RED proved a mixed unsupported matrix exited 0; GREEN rejects the entire matrix if any row is outside the supported Ubuntu set, with the workflow copy kept byte-equivalent to the owner script. Offline warnings-fatal focused verification: 10 passed; compile and git diff --check passed. Exact-head CodeQL PR run 37887239686 is skipped and Repository Metadata Reconcile run 37887239528 is queued. Draft / Proposed / merge HOLD remains pending terminal required Checks and qualifying independent approval.

[seonghobae]

Stack ownership receipt for exact #2565 head a206770680895adba000d7538fc3d5a5b499149e (COMMENT, not approval): #2607 no longer writes shared runner policy directly against main. Its ordinary merge head ef082eaa13b8840d2ceadbee7a129d0bcce0a91a includes this exact prerequisite as parent 2 and #2607 now targets branch ci/sdp-all-self-hosted-20261003. The child diff is reduced to 10 fast-mlsirm-owned files, and all 27 new runner selectors inherit this owner's CWL CI isolated + cwlab-ci-isolated contract. Full merged-tree verification is 5,494 passed, 10 skipped, 40 subtests with 46/46 workflow YAML parses. #2565 remains the prerequisite and remains Draft/HOLD; #2607 must not merge before this owner integrates normally.

[seonghobae]

Stack ownership receipt for exact #2565 head a206770680895adba000d7538fc3d5a5b499149e (COMMENT, not approval): #2606 no longer writes shared runner policy directly against main. Its exact head 9c209b3bed74c5af91d00905777400134be34e9c includes this owner through ordinary merge b8194dfb41b045b4830b54088994e39b4f351679 and now targets ci/sdp-all-self-hosted-20261003. The seven runner-routing conflicts were resolved identically to the published #2607 stack, preserving this owner's CWL CI isolated + cwlab-ci-isolated contract. #2606's comparison is reduced to three pg-erd-owned files plus one separately owned #2609 fixture file. Exact integrated-tree verification is 5,514 passed, 10 skipped, 40 subtests; #2565 remains Draft/HOLD and #2606 must not merge before this owner integrates normally.

[seonghobae]

Stack ownership receipt for exact #2565 head a206770680895adba000d7538fc3d5a5b499149e (COMMENT, not approval): #2602 has been refreshed from stale owner predecessor 6e924f32… by ordinary merge bb5b297be7b99f584ddf0d587e595e3bbdf0fac2, and now compares ahead 8 / behind 0 with merge-base exactly this head. Its GRC delta remains four additive files; #2565's runner-routing files and CWL CI isolated + cwlab-ci-isolated tests remain byte-equivalent to the owner. Exact combined-tree verification is 5,498 passed, 10 skipped, 40 subtests. #2565 and dependent #2602 remain Draft/HOLD pending real isolated capacity/canary and normal protected integration.

[seonghobae]

Downstream stack receipt — #2603

PR #2603 now non-force-stacks on this exact owner head a206770680895adba000d7538fc3d5a5b499149e.

• downstream exact head: 5d7fefa7e56300dfc8732929664ec199aa7fafa9
• merge base: this exact head
• child delta: three BandScope files plus test(codeql): keep current-time fixtures fresh #2609's fixture
• Draft retained because this prerequisite remains mutable/unready
• local exact-tree verification: 5,535 passed, 10 skipped, 40 subtests

No shared runner-policy rollback remains in the downstream diff.

[seonghobae]

Current authority addendum — exact head 8d4a2eb1061c060360741d4015e4350ffde6a3eb

Causal owner repair from downstream #2581:

• RED on the combined tree: CWL law CI was rejected by test_nonprivileged_runners_require_isolated_label (1 failed / 209 passed).
• GREEN: the owner contract now maps CWL law CI to exact label law-ai-agent-ci; unregistered groups still fail closed.
• Owner focused verification: 5 passed normally and 5 passed with GITHUB_ACTIONS=true.
• Owner full suite retained only the two known fixed-date failures now owned by test(codeql): keep current-time fixtures fresh #2609: 5,489 passed, 10 skipped, 40 subtests, 2 failed.
• Downstream ci: centralize law-agent quality on dedicated self-hosted runners #2581 consumed both this exact owner head and test(codeql): keep current-time fixtures fresh #2609, then passed the complete suite: 5,536 passed, 10 skipped, 40 subtests.

Branch advanced from a206770680895adba000d7538fc3d5a5b499149e with expected-head protection and force=false. Draft / Proposed / merge HOLD remains unchanged.

[seonghobae]

Stack ownership receipt for exact #2565 head 8d4a2eb1061c060360741d4015e4350ffde6a3eb (COMMENT, not approval): downstream #2607 detected this owner advance and repaired its stale comparison by ordinary two-parent merge 0975955d1bae39d634cc52e4e776c1c95b4f14d3, preserving predecessor ef082eaa… as parent 1 and this owner as parent 2. The effective child delta remains 10 paths; the owner's CWL law CI / law-ai-agent-ci and fail-closed unknown-group policy are inherited without copying or rollback. Exact integrated-tree verification is 5,495 passed, 10 skipped, 40 subtests; owner + child runner contracts are 9/9 and workflow parsing is 46/46. #2565 and #2607 remain Draft/HOLD pending real isolated capacity/canary, normal protected integration, and qualifying independent approval.

[seonghobae]

Downstream consumption receipts — exact head 8d4a2eb1061c060360741d4015e4350ffde6a3eb

The law-runner policy repair is now consumed by ordinary merge in every downstream stack changed during this run:

• ci: centralize law-agent quality on dedicated self-hosted runners #2581 → 431c575f9778e727ef2ec071e57eb334564bd737 (full exact-tree suite: 5,536 passed, 10 skipped, 40 subtests)
• ci(grc): stage central self-hosted Product producer #2602 → efb53f7b44d09614993702e4a20d75c06a190a37 (affected contracts: 43/43 normal and Actions mode)
• ci(bandscope): stage central isolated Linux quality gates #2603 → 49c1868d57236359485634042cee375d85d001c3 (affected contracts: 80/80 normal and Actions mode)
• ci(pg-erd): centralize quality on isolated self-hosted runners #2606 → 56c1435ab16fec9ef196e3bd4773845acf8f50a2 (affected contracts: 59/59 normal and Actions mode)

Each comparison has this exact owner head as merge base and zero behind commits. Draft / Proposed / merge HOLD remains unchanged.