Repository navigation
CI for zeron/zgx: integration branch, zgx suite on plan and admission, faster betas - #1357
Open
katulevskiy wants to merge 17 commits into
Open
katulevskiy wants to merge 17 commits into
katulevskiy wants to merge 17 commits into
Conversation
Google Play's upload leaves ci.yml for android-play.yml, run when CI passes
on main. Its approval wait (the google-play environment) kept ci.yml's run,
and so main's concurrency group, open: from Nov 5 to Nov 11 one wait held
every later main run and nightly, all cancelled unstarted, and main saved no
caches or passes. The new workflow holds nothing else, and a newer build
cancels an older one still waiting for approval, so only the newest waits.
The apk job marks a bundle signed with the release key (release-signed)
for it. android.yml keeps the upload by hand.
A pull request's CI run gets a concurrency group of its own. In one group
per pull request, a new push cancelled the older run, whose `CI result`
(`if: always()`) still queued for a runner while the new run waited behind
it ("waiting for CI #610 to complete"). ci-supersede.yml (pull_request_target,
so it can for forks too; it reads and cancels runs only, never the pull
request's code) force-cancels a pull request's older runs on each push and
all of them when it closes. A run still waiting for admission stops by
itself once its commit isn't the pull request's head (admit.py `superseded`).
The organization's 20 hosted runners (5 macOS) are one pool for zeronsh/zeron and zeronsh/zgx, but admission only saw this repository's CI. On Nov 10 zgx's CI alone held a median 18 of the 20 (max 20) for half a day, so runs admitted here as "all runners free" then queued behind it for hours, and release and beta jobs competed with admitted CI as if they weren't there. admit.py now reads, besides this workflow: - CI_PEERS (zeronsh/zgx:ci.yml by default): the peer repositories' admitted CI joins the same queue: their waiting runs wait in it, their admitted runs hold what they hold, their ci:priority / ci:lock labels count. A peer's pull request is keyed `owner/repo#n`, never confused with ours. - every other run in this repository and the peers (releases, betas, deploys, the peer's CI before it adopts admission): what its jobs run or queue for counts as held, so a release's queued jobs get the next runners and admitted CI doesn't take them. (A job waiting for an approval holds no runner and doesn't count.) CI_SEE_OUTSIDE=off turns this off. Another repository's runs are read at most every 2 minutes (the token's API quota). Tests: a peer's prioritised run ahead in the queue, a peer pull request's key, and outside runs' holdings.
zeron/zgx is to the zgx interface's work what main is to the rest, but CI never ran there: nothing saved the zgx suite's caches (they saved on main only, where the suite never runs) or recorded its passes, so every zgx pull request compiled from nothing and re-ran everything. The zgx CLI was built again on every run, its cache saved under each pull request's own ref. - ci.yml runs on pushes to zeron/zgx too, on the pull request tier (CI_INTEGRATION_TIER), and a pull request into it reuses its passes as well as main's (plan.py INTEGRATION). - zgx-tests.yml saves its Rust, CLI and Ghostty caches from zeron/zgx only; everything else only reads them. - zgx-macos and the new zgx-windows run on the fast tier only when the change looks platform specific (plan.py ZGX_PLATFORM_JOBS: the app's own platform code, its lockfile, its vendored patches); zgx-linux runs for every zgx change. Each job is skipped on its own once it passed on the same inputs (SPLITTABLE). - The engine's plugin tests on macOS, a whole root-workspace build in that job (core-tests runs them on Linux on every change), run on pull requests only for the plugin host's own code (the engine-plugins flag). - zgx-windows: the unit tests on Windows with their own known failures (`rust-windows:` in zgx-known-failures.txt) and the Windows tests scripts/ci/zgx-windows-tests.txt lists, so Windows regressions run under plan and admission instead of a workflow of their own. - admit.py's cap is one for every branch and repository in the shared queue (linux 11, windows 4, macos 4): the file is the same everywhere.
…gates them Each platform built the engine (root workspace) and then, inside its packaging script, the zgx interface (apps/zgx, its own workspace): one after the other, on one runner. Windows took 29-38 min + 14.5-18.5, macOS 13-22 + 14-22, Linux 10-16.5 + 7-11. Now each platform has three jobs: <platform>-engine and <platform>-interface build in parallel from `version` and hand their binaries over as `beta-build-*` artifacts (kept a day), and <platform>-package takes both prebuilt and packages, with every check it had. package-zgx-beta.sh already took the interface prebuilt (ZGX_BETA_ZGX_BIN); package-linux.sh now takes ZGX_BETA_ZGX_BIN too, and package-windows.ps1 -BetaInterface. The interface jobs build it as the scripts did: from apps/zgx (its .cargo/config.toml), --locked --features beta, ZERON_BETA_VERSION, into apps/zgx/target. release and channel need the package jobs and download only `zgx-beta-*`. Only macos-package, which signs and notarizes, takes `apple-release`. A job asks for approval only once its needs are done, so it needs just `version`: it's approved while the builds run, then waits for their jobs (gh api, the run's jobs) and downloads their artifacts. No caches are saved: rust-cache has save-if: false, and the zgx-ghostty cache is restored only. A tag's caches (2.1-2.45 GB per platform) were read by no later tag and pushed main's out of the 10 GB.
… caches in budget The zgx unit tests had never run on Windows. Run on zeron/zgx (fbb89b2), 16 of 682 fail there: headless layout and paint measured at fixed pixel sizes, and paths read the Unix way. They're listed as rust-windows: known failures, so zgx-windows gates every other test from the start. The macOS engine-plugin variant and zgx-windows read the shared caches but save none: both run only for some changes, and their ~1 GB each would cost caches used on every run their place in the 10 GB.
The repository's 10 GB of Actions cache is LRU: caches saved from tags and side branches (release.yml, zgx-beta.yml: 5 to 9 GB a run, never restored) and from pull requests (actions/cache, setup-node's npm cache, setup-zig's) evicted the ones main's CI restores on every run. scripts/ci/caches.py declares every cache as a family: a key pattern, the refs that may save it (main; zeron/zgx for zgx's suite), a priority by measured value (minutes saved per month per GB) and an expected size. The keep set is about 9.4 GB of a 9.5 GB budget: windows, linux-ui, linux-core, zgx-tests (macOS, Linux), ios-tests, macos-native, ios-core, the ui-browser .deb directory, zgx-ghostty and zgx-cli, plus pass records from any ref. test_ci_caches.py lints the workflows and actions (by indentation, no PyYAML) in plan: rust-cache needs save-if (false, main only, zgx's SAVE_CACHES, or a composite action's input and main) and may only save a declared family from its refs, worked out as rust-cache builds the key; actions/cache@ is refused; actions/cache/save needs an `if:` naming one allowed ref unless it's a pass record; setup-node/python/java/go may not cache; setup-zig needs use-cache: false. Stops saving: android, frame-source, TestFlight's ios-core, linux voice, voice's windows-runtime and .deb directory (linux-ci's apt cache is now saved only by the job given save-apt-cache), release.yml's and zgx-beta.yml's Rust caches, zgx-beta's Ghostty build (restore only) and setup-zig cache, and the npm caches. Nothing restores anything different. ci-caches.yml runs the janitor every 6 hours from main (or by hand, dry-run optional): across every ref it deletes undeclared entries, entries from refs their family doesn't allow, older generations (newest per family per ref), then the lowest-priority families while over budget, and summarises what it kept and deleted. docs/reference/ci.md: the caches policy, and the Play upload, pull request supersede, cross-repository admission and shared reservation cap from the previous commits.
zeron/zgx moved the beta's and stable release's builds into zgx-build.yml: its zgx-beta.yml and packaging scripts are taken as they are there (the parallel engine/interface beta jobs are left to a follow-up on zgx-build.yml). zgx-build.yml follows the cache policy: no rust-cache saves from tags, Ghostty restored only, setup-zig's own cache off.
…ogs mixed keys ci-supersede.yml runs on pull_request_target with actions: write. The head branch and repository names, which a fork's author chooses, were spliced into the jq filter text, so a crafted branch name could widen the filter and force-cancel any run, releases included. They're --arg values now (the branch also percent-encoded in the URL), the filter takes pull_request runs of ci.yml only, and only a run ID of digits is cancelled. admit.py's progress line sorted the priority and lock sets, which mix this repository's pull request numbers with a peer's `owner/repo#n` keys: sorting int with str raised TypeError. They're sorted as text.
…roup The workflow-level concurrency group put every run of android-play.yml in it, and every completed CI run on a branch named main starts one: a fork's pull request from its own main, the nightly, a run by hand. Each cancelled the upload waiting for its reviewer. The group is on the `play` job now, which only a trusted push to main with a bundle reaches, so the newest build still cancels the older one waiting for approval and nothing else does.
… read rank() ordered locks by lock time, then by their admit job's creation; decide()'s lock_is_older by lock time, then by the key as text. Two locks added in the same second could disagree in the two orders and each wait for the other. rank() breaks the tie by the key as text too. The reservation and pull request claim patterns take at most 4 digits: a name comes from another run's editable ci.yml, and int() of a number thousands of digits long raises ValueError, which failed every waiting run's poll until each gave up and started without a reservation.
passed_jobs listed `<key>-` entries for every ref in one page of 100 and kept the trusted ones. Pull requests, a fork's among them, can save entries under any key: enough of them fill the page and push main's and the pull request's own passes off it, so a suite that passed reran. It asks per trusted ref (`&ref=`), reading every page.
The janitor counted pass records in the budget total. Any pull request, a fork's too, can save any number of them, so enough of them made the janitor drop the lowest-priority caches. They don't count now. It also kept pass records from every ref. Only main's, zeron/zgx's and a pull request's merge ref's are read by any run (plan.py); the rest (a run by hand on a side branch, the merge queue, a tag) are deleted. A fork's pull request can't be told from a same-repository one by its ref, so all of refs/pull/<n>/merge stay.
A pull request into zeron/zgx reused zeron/zgx's passes, and zgx-tests.yml
saved its caches from any push to it. zeron/zgx has no ruleset: anyone who
can push to it could record a pass, or save a zgx CLI or Ghostty build,
that every pull request into it then trusts and runs.
plan.py's integration_trusted reads the branch's rules on a push to it or a
pull request into it (GET /repos/{repo}/rules/branches/zeron%2Fzgx, which
the plan job's token reads on a public repository) and trusts it only with
both `pull_request` and `non_fast_forward`, or classic protection that
requires reviews and blocks force pushes when that is readable; any error
is no. Otherwise pull requests into it reuse main's passes only, a push to
it trusts main's, and plan's new `integration-trusted` output, passed to
zgx-tests.yml, keeps SAVE_CACHES false. The cache lint accepts that one
narrowed SAVE_CACHES form.
docs/reference/ci.md says what an admin has to do: give zeron/zgx main's
ruleset (pull request required, force pushes blocked).
…d itself A pull request's runs restore from its own cache scope first, and its earlier commits can write there: an edited zgx-tests.yml could save a fake zgx CLI under the real key, which a later, clean commit's tests restore and run. Ghostty's build, linked into the tests, the same, under any `zgx-ghostty-` key (its restore keys take any). On a pull request, a step lists the entries with that key in the pull request's own scope (actions/caches?key=...&ref=refs/pull/<n>/merge) and the restore runs only if there are none; otherwise, or if the API fails, the CLI and Ghostty are built. The zgx jobs get `actions: read` (and ci.yml grants it to the `zgx` call).
The 4-digit cap kept a giant number in an edited admit name from raising in int(), but would also have stopped pull requests numbered 10000 and up from claiming ci:priority or ci:lock (zgx is past 2000, ~70 a day). Pull request numbers take up to 7 digits; reservations stay at 4.
3 of 4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of the CI speedup for the zgx interface's work. It carries the same general commits as the
mainPR (linked below), plus whatzeron/zgxneeds.Why
main, where the zgx suite never runs, so every zgx PR compiled from scratch: zgx-linux 22–40 min, zgx-macos 29–89 min.apple-releaseapproval before building at all, ~190 min over the last 6 betas. Its tag-scoped caches (~9 GB per beta) evictedmain's.Changes
zeron/zgxis treated likemain(plan.py INTEGRATION).main's.zgx-linuxruns for every zgx change.zgx-macosand a newzgx-windowsrun only for platform-specific changes (the app's platform code, its lockfile, its vendored patches).zgx-windowsruns the unit tests on Windows. 16 of 682 fail on the untouched base when run locally on Windows (headless layout at fixed pixel sizes, Unix-style paths); they're listed asrust-windows:known failures. It also runs Windows regressions listed inscripts/ci/zgx-windows-tests.txt, so they run under plan and admission rather than as a workflow of their own.zeron/zgx. The rarely used variants (macOS with the engine, Windows) read the shared caches but save none, to stay in budget.apple-release, so it can be approved while the builds run.docs/reference/ci.md: zgx section.Verified
package-linux.sh ZGX_BETA_ZGX_BINandpackage-windows.ps1 -BetaInterface.Note for #1219
Its Windows regressions (
intro_handoff,windows_parity, the vendored terminal) can move from its ownzgx-windows.ymlintoscripts/ci/zgx-windows-tests.txtonce this lands.Linked PRs (CI stage 1)
Follow-up (release admission): #1358, #1359, zeronsh/zgx#2039