Repository navigation
CI: one admission queue across zeron and zgx, a cache guard, nothing waits on a person or a cancelled run - #1356
Open
katulevskiy wants to merge 10 commits into
Open
katulevskiy wants to merge 10 commits into
katulevskiy wants to merge 10 commits into
Conversation
Google Play's upload leaves ci.yml for android-play.yml, run when CI passes
on main. Its approval wait (the google-play environment) kept ci.yml's run,
and so main's concurrency group, open: from Nov 5 to Nov 11 one wait held
every later main run and nightly, all cancelled unstarted, and main saved no
caches or passes. The new workflow holds nothing else, and a newer build
cancels an older one still waiting for approval, so only the newest waits.
The apk job marks a bundle signed with the release key (release-signed)
for it. android.yml keeps the upload by hand.
A pull request's CI run gets a concurrency group of its own. In one group
per pull request, a new push cancelled the older run, whose `CI result`
(`if: always()`) still queued for a runner while the new run waited behind
it ("waiting for CI #610 to complete"). ci-supersede.yml (pull_request_target,
so it can for forks too; it reads and cancels runs only, never the pull
request's code) force-cancels a pull request's older runs on each push and
all of them when it closes. A run still waiting for admission stops by
itself once its commit isn't the pull request's head (admit.py `superseded`).
The organization's 20 hosted runners (5 macOS) are one pool for zeronsh/zeron and zeronsh/zgx, but admission only saw this repository's CI. On Nov 10 zgx's CI alone held a median 18 of the 20 (max 20) for half a day, so runs admitted here as "all runners free" then queued behind it for hours, and release and beta jobs competed with admitted CI as if they weren't there. admit.py now reads, besides this workflow: - CI_PEERS (zeronsh/zgx:ci.yml by default): the peer repositories' admitted CI joins the same queue: their waiting runs wait in it, their admitted runs hold what they hold, their ci:priority / ci:lock labels count. A peer's pull request is keyed `owner/repo#n`, never confused with ours. - every other run in this repository and the peers (releases, betas, deploys, the peer's CI before it adopts admission): what its jobs run or queue for counts as held, so a release's queued jobs get the next runners and admitted CI doesn't take them. (A job waiting for an approval holds no runner and doesn't count.) CI_SEE_OUTSIDE=off turns this off. Another repository's runs are read at most every 2 minutes (the token's API quota). Tests: a peer's prioritised run ahead in the queue, a peer pull request's key, and outside runs' holdings.
…the queue admit.py is the same file on main, zeron/zgx and in zeronsh/zgx, which share the queue; its cap is the largest any of them reserves (linux 11, windows 4, macos 4: zeron/zgx's zgx suite). The cap test checks this branch's largest plan fits under it, within a job per runner type.
The repository's 10 GB of Actions cache is LRU: caches saved from tags and side branches (release.yml, zgx-beta.yml: 5 to 9 GB a run, never restored) and from pull requests (actions/cache, setup-node's npm cache, setup-zig's) evicted the ones main's CI restores on every run. scripts/ci/caches.py declares every cache as a family: a key pattern, the refs that may save it (main; zeron/zgx for zgx's suite), a priority by measured value (minutes saved per month per GB) and an expected size. The keep set is about 9.4 GB of a 9.5 GB budget: windows, linux-ui, linux-core, zgx-tests (macOS, Linux), ios-tests, macos-native, ios-core, the ui-browser .deb directory, zgx-ghostty and zgx-cli, plus pass records from any ref. test_ci_caches.py lints the workflows and actions (by indentation, no PyYAML) in plan: rust-cache needs save-if (false, main only, zgx's SAVE_CACHES, or a composite action's input and main) and may only save a declared family from its refs, worked out as rust-cache builds the key; actions/cache@ is refused; actions/cache/save needs an `if:` naming one allowed ref unless it's a pass record; setup-node/python/java/go may not cache; setup-zig needs use-cache: false. Stops saving: android, frame-source, TestFlight's ios-core, linux voice, voice's windows-runtime and .deb directory (linux-ci's apt cache is now saved only by the job given save-apt-cache), release.yml's and zgx-beta.yml's Rust caches, zgx-beta's Ghostty build (restore only) and setup-zig cache, and the npm caches. Nothing restores anything different. ci-caches.yml runs the janitor every 6 hours from main (or by hand, dry-run optional): across every ref it deletes undeclared entries, entries from refs their family doesn't allow, older generations (newest per family per ref), then the lowest-priority families while over budget, and summarises what it kept and deleted. docs/reference/ci.md: the caches policy, and the Play upload, pull request supersede, cross-repository admission and shared reservation cap from the previous commits.
…ogs mixed keys ci-supersede.yml runs on pull_request_target with actions: write. The head branch and repository names, which a fork's author chooses, were spliced into the jq filter text, so a crafted branch name could widen the filter and force-cancel any run, releases included. They're --arg values now (the branch also percent-encoded in the URL), the filter takes pull_request runs of ci.yml only, and only a run ID of digits is cancelled. admit.py's progress line sorted the priority and lock sets, which mix this repository's pull request numbers with a peer's `owner/repo#n` keys: sorting int with str raised TypeError. They're sorted as text.
…roup The workflow-level concurrency group put every run of android-play.yml in it, and every completed CI run on a branch named main starts one: a fork's pull request from its own main, the nightly, a run by hand. Each cancelled the upload waiting for its reviewer. The group is on the `play` job now, which only a trusted push to main with a bundle reaches, so the newest build still cancels the older one waiting for approval and nothing else does.
… read rank() ordered locks by lock time, then by their admit job's creation; decide()'s lock_is_older by lock time, then by the key as text. Two locks added in the same second could disagree in the two orders and each wait for the other. rank() breaks the tie by the key as text too. The reservation and pull request claim patterns take at most 4 digits: a name comes from another run's editable ci.yml, and int() of a number thousands of digits long raises ValueError, which failed every waiting run's poll until each gave up and started without a reservation.
passed_jobs listed `<key>-` entries for every ref in one page of 100 and kept the trusted ones. Pull requests, a fork's among them, can save entries under any key: enough of them fill the page and push main's and the pull request's own passes off it, so a suite that passed reran. It asks per trusted ref (`&ref=`), reading every page.
The janitor counted pass records in the budget total. Any pull request, a fork's too, can save any number of them, so enough of them made the janitor drop the lowest-priority caches. They don't count now. It also kept pass records from every ref. Only main's, zeron/zgx's and a pull request's merge ref's are read by any run (plan.py); the rest (a run by hand on a side branch, the merge queue, a tag) are deleted. A fork's pull request can't be told from a same-repository one by its ref, so all of refs/pull/<n>/merge stay.
The 4-digit cap kept a giant number in an edited admit name from raising in int(), but would also have stopped pull requests numbered 10000 and up from claiming ci:priority or ci:lock (zgx is past 2000, ~70 a day). Pull request numbers take up to 7 digits; reservations stay at 4.
katulevskiy
added a commit
to katulevskiy/zgx
that referenced
this pull request
Nov 12, 2026
…digits read zeron's fixes since 95500a27f (zeronsh/zeron#1356): - rank() and lock_is_older break a tie between locks added in the same second the same way, by the key as text, so two such locks can't each wait for the other; - the reservation and pull request claim patterns read at most 4 digits: int() of a number thousands of digits long, from another run's editable name, raised ValueError and failed every waiting run's poll. The vendored tests at the same commit, with zgx's differences kept.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of the CI speedup: the CI fixes
mainneeds, without new services or runners. It's one of three linked PRs (below); stage 2 adds a shared compile cache on R2, stage 3 boat.dev Linux runners.Why
mainwas blocked from Nov 5 to Nov 11. The Google Play upload waited for approval insideci.yml, holdingmain's concurrency group, so every latermainrun and nightly was cancelled before it started.mainsaved no caches or passes the whole time.CI resultjob (if: always()) still queued for a runner (zgx Windows: browser, terminal, annotation and intro fixes #1219's runs Unified Zeron and agent updates across devices #610/Opencode: title new threads from the session harness #611).ci.yml.zeronsh/zgx's CI alone held a median 18 of the org's 20 runners for half a day, and releases and betas competed with admitted CI unseen.mainrelies on. Tag and side-branch caches (release and zgx-beta, 5–9 GB per run, never restored) and PR-ref caches did this. A full set ofmain's caches was ~10.9 GB against a 10 GB limit.Changes
android-play.yml. The Google Play upload runs after CI passes onmain, in its own workflow. A newer build cancels an older one still waiting for approval, and nothing else waits on it. A run that built no bundle never asks a reviewer.ci-supersede.yml(pull_request_target; reads and cancels runs only, never checks out PR code) force-cancels a PR's older runs on each push, and all of them when the PR closes. It works for forks too.admit.pystops a waiting run whose commit is no longer the PR head.CI_PEERS(defaultzeronsh/zgx:ci.yml) puts the peer repo's admitted runs and itsci:priority/ci:locklabels in the same queue. Its PRs are keyedowner/repo#n.CI_SEE_OUTSIDE=offdisables this.admit.pyis byte-identical in every repo and branch, with one reservation cap (11/4/4).scripts/ci/caches.pyis the one allowlist: families ranked by measured minutes saved per GB, 9.43 GB declared against a 9.5 GB budget.test_ci_caches.pyruns inplanand fails CI on any undeclared cache, any combinedactions/cache, ungated saves, and setup-node/setup-zig self-caching.ci-caches.yml(every 6 h) deletes unlisted families, entries from refs a family doesn't allow, older generations, and the lowest-priority families if the total is over budget. A dry run against today's live list would delete 6.71 GB of dead entries.docs/reference/ci.mdis updated for all of the above.Verified
python -m unittest discover -s scripts/tests -p 'test_ci_*.py': 113 tests pass. That includes the admission simulations (3,000 random queues, no deadlock), the shared-pool scheduler, the superseded check and the cache lint.After merge
CI_PEERS,CI_SEE_OUTSIDE(defaults are fine).PLAY_UPLOAD_ENABLED,PLAY_TRACKandPLAY_RELEASE_STATUS.Linked PRs (CI stage 1)
Follow-up (release admission): #1358, #1359, zeronsh/zgx#2039