Skip to content

CI for zeron/zgx: integration branch, zgx suite on plan and admission, faster betas - #1357

Open
katulevskiy wants to merge 17 commits into
zeron/zgxfrom
zeron/ci-zgx-speed
Open

katulevskiy wants to merge 17 commits into
zeron/zgxfrom
zeron/ci-zgx-speed

Conversation

@katulevskiy

@katulevskiy katulevskiy commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Stage 1 of the CI speedup for the zgx interface's work. It carries the same general commits as the main PR (linked below), plus what zeron/zgx needs.

Why

  • zgx caches were never saved. Caches saved only on main, where the zgx suite never runs, so every zgx PR compiled from scratch: zgx-linux 22–40 min, zgx-macos 29–89 min.
  • The zgx CLI was rebuilt on every run, its cache saved under each PR's own ref.
  • zgx-macos rebuilt the root workspace on every PR just for the engine's plugin tests, which core-tests already runs on Linux. That's three compiles in one job, and runners were lost mid-job 4 times in 10.
  • Every beta built cold, one step after another. Windows took 44–56 min. The macOS job waited for apple-release approval before building at all, ~190 min over the last 6 betas. Its tag-scoped caches (~9 GB per beta) evicted main's.

Changes

  • zeron/zgx is treated like main (plan.py INTEGRATION).
    • CI runs on its pushes, on the PR tier.
    • PRs into it reuse its passes as well as main's.
    • Its zgx caches are saved from there only.
  • zgx suite:
    • zgx-linux runs for every zgx change.
    • zgx-macos and a new zgx-windows run only for platform-specific changes (the app's platform code, its lockfile, its vendored patches).
    • Each job is skipped on its own once it passed on the same inputs.
    • The macOS engine-plugin tests run on PRs only for the plugin host's own code.
  • zgx-windows runs the unit tests on Windows. 16 of 682 fail on the untouched base when run locally on Windows (headless layout at fixed pixel sizes, Unix-style paths); they're listed as rust-windows: known failures. It also runs Windows regressions listed in scripts/ci/zgx-windows-tests.txt, so they run under plan and admission rather than as a workflow of their own.
  • zgx caches: the CLI, Ghostty and Rust caches use restore/save split, gated on zeron/zgx. The rarely used variants (macOS with the engine, Windows) read the shared caches but save none, to stay in budget.
  • Betas:
    • Each platform's engine and interface build in parallel jobs, then package. Expected: Windows ~34–43 min (was 44–56), Linux ~14–20 min (was 17–28).
    • On macOS only the signing job waits for apple-release, so it can be approved while the builds run.
    • No beta saves a GitHub cache. Warm betas come with R2 in stage 2.
  • docs/reference/ci.md: zgx section.

Verified

  • 116 CI tests pass, including new zgx gating tests and the cache lint over these workflows.
  • Every workflow parses.
  • The beta job graph, artifacts and packaging inputs were checked by hand. The new optional inputs are package-linux.sh ZGX_BETA_ZGX_BIN and package-windows.ps1 -BetaInterface.
  • It hasn't run on GitHub yet.

Note for #1219

Its Windows regressions (intro_handoff, windows_parity, the vendored terminal) can move from its own zgx-windows.yml into scripts/ci/zgx-windows-tests.txt once this lands.

Linked PRs (CI stage 1)

Follow-up (release admission): #1358, #1359, zeronsh/zgx#2039

Google Play's upload leaves ci.yml for android-play.yml, run when CI passes
on main. Its approval wait (the google-play environment) kept ci.yml's run,
and so main's concurrency group, open: from Nov 5 to Nov 11 one wait held
every later main run and nightly, all cancelled unstarted, and main saved no
caches or passes. The new workflow holds nothing else, and a newer build
cancels an older one still waiting for approval, so only the newest waits.
The apk job marks a bundle signed with the release key (release-signed)
for it. android.yml keeps the upload by hand.

A pull request's CI run gets a concurrency group of its own. In one group
per pull request, a new push cancelled the older run, whose `CI result`
(`if: always()`) still queued for a runner while the new run waited behind
it ("waiting for CI #610 to complete"). ci-supersede.yml (pull_request_target,
so it can for forks too; it reads and cancels runs only, never the pull
request's code) force-cancels a pull request's older runs on each push and
all of them when it closes. A run still waiting for admission stops by
itself once its commit isn't the pull request's head (admit.py `superseded`).
The organization's 20 hosted runners (5 macOS) are one pool for zeronsh/zeron
and zeronsh/zgx, but admission only saw this repository's CI. On Nov 10
zgx's CI alone held a median 18 of the 20 (max 20) for half a day, so runs
admitted here as "all runners free" then queued behind it for hours, and
release and beta jobs competed with admitted CI as if they weren't there.

admit.py now reads, besides this workflow:
- CI_PEERS (zeronsh/zgx:ci.yml by default): the peer repositories' admitted
  CI joins the same queue: their waiting runs wait in it, their admitted
  runs hold what they hold, their ci:priority / ci:lock labels count. A
  peer's pull request is keyed `owner/repo#n`, never confused with ours.
- every other run in this repository and the peers (releases, betas,
  deploys, the peer's CI before it adopts admission): what its jobs run or
  queue for counts as held, so a release's queued jobs get the next runners
  and admitted CI doesn't take them. (A job waiting for an approval holds
  no runner and doesn't count.) CI_SEE_OUTSIDE=off turns this off.

Another repository's runs are read at most every 2 minutes (the token's API
quota). Tests: a peer's prioritised run ahead in the queue, a peer pull
request's key, and outside runs' holdings.
zeron/zgx is to the zgx interface's work what main is to the rest, but CI
never ran there: nothing saved the zgx suite's caches (they saved on main
only, where the suite never runs) or recorded its passes, so every zgx pull
request compiled from nothing and re-ran everything. The zgx CLI was built
again on every run, its cache saved under each pull request's own ref.

- ci.yml runs on pushes to zeron/zgx too, on the pull request tier
  (CI_INTEGRATION_TIER), and a pull request into it reuses its passes as
  well as main's (plan.py INTEGRATION).
- zgx-tests.yml saves its Rust, CLI and Ghostty caches from zeron/zgx only;
  everything else only reads them.
- zgx-macos and the new zgx-windows run on the fast tier only when the
  change looks platform specific (plan.py ZGX_PLATFORM_JOBS: the app's own
  platform code, its lockfile, its vendored patches); zgx-linux runs for
  every zgx change. Each job is skipped on its own once it passed on the
  same inputs (SPLITTABLE).
- The engine's plugin tests on macOS, a whole root-workspace build in that
  job (core-tests runs them on Linux on every change), run on pull requests
  only for the plugin host's own code (the engine-plugins flag).
- zgx-windows: the unit tests on Windows with their own known failures
  (`rust-windows:` in zgx-known-failures.txt) and the Windows tests
  scripts/ci/zgx-windows-tests.txt lists, so Windows regressions run under
  plan and admission instead of a workflow of their own.
- admit.py's cap is one for every branch and repository in the shared queue
  (linux 11, windows 4, macos 4): the file is the same everywhere.
…gates them

Each platform built the engine (root workspace) and then, inside its
packaging script, the zgx interface (apps/zgx, its own workspace): one
after the other, on one runner. Windows took 29-38 min + 14.5-18.5,
macOS 13-22 + 14-22, Linux 10-16.5 + 7-11.

Now each platform has three jobs: <platform>-engine and
<platform>-interface build in parallel from `version` and hand their
binaries over as `beta-build-*` artifacts (kept a day), and
<platform>-package takes both prebuilt and packages, with every check
it had. package-zgx-beta.sh already took the interface prebuilt
(ZGX_BETA_ZGX_BIN); package-linux.sh now takes ZGX_BETA_ZGX_BIN too,
and package-windows.ps1 -BetaInterface. The interface jobs build it as
the scripts did: from apps/zgx (its .cargo/config.toml), --locked
--features beta, ZERON_BETA_VERSION, into apps/zgx/target. release and
channel need the package jobs and download only `zgx-beta-*`.

Only macos-package, which signs and notarizes, takes `apple-release`.
A job asks for approval only once its needs are done, so it needs just
`version`: it's approved while the builds run, then waits for their jobs
(gh api, the run's jobs) and downloads their artifacts.

No caches are saved: rust-cache has save-if: false, and the zgx-ghostty
cache is restored only. A tag's caches (2.1-2.45 GB per platform) were
read by no later tag and pushed main's out of the 10 GB.
… caches in budget

The zgx unit tests had never run on Windows. Run on zeron/zgx (fbb89b2),
16 of 682 fail there: headless layout and paint measured at fixed pixel
sizes, and paths read the Unix way. They're listed as rust-windows: known
failures, so zgx-windows gates every other test from the start.

The macOS engine-plugin variant and zgx-windows read the shared caches but
save none: both run only for some changes, and their ~1 GB each would cost
caches used on every run their place in the 10 GB.
The repository's 10 GB of Actions cache is LRU: caches saved from tags and
side branches (release.yml, zgx-beta.yml: 5 to 9 GB a run, never restored)
and from pull requests (actions/cache, setup-node's npm cache, setup-zig's)
evicted the ones main's CI restores on every run.

scripts/ci/caches.py declares every cache as a family: a key pattern, the
refs that may save it (main; zeron/zgx for zgx's suite), a priority by
measured value (minutes saved per month per GB) and an expected size. The
keep set is about 9.4 GB of a 9.5 GB budget: windows, linux-ui, linux-core,
zgx-tests (macOS, Linux), ios-tests, macos-native, ios-core, the ui-browser
.deb directory, zgx-ghostty and zgx-cli, plus pass records from any ref.

test_ci_caches.py lints the workflows and actions (by indentation, no
PyYAML) in plan: rust-cache needs save-if (false, main only, zgx's
SAVE_CACHES, or a composite action's input and main) and may only save a
declared family from its refs, worked out as rust-cache builds the key;
actions/cache@ is refused; actions/cache/save needs an `if:` naming one
allowed ref unless it's a pass record; setup-node/python/java/go may not
cache; setup-zig needs use-cache: false.

Stops saving: android, frame-source, TestFlight's ios-core, linux voice,
voice's windows-runtime and .deb directory (linux-ci's apt cache is now saved
only by the job given save-apt-cache), release.yml's and zgx-beta.yml's Rust
caches, zgx-beta's Ghostty build (restore only) and setup-zig cache, and the
npm caches. Nothing restores anything different.

ci-caches.yml runs the janitor every 6 hours from main (or by hand, dry-run
optional): across every ref it deletes undeclared entries, entries from refs
their family doesn't allow, older generations (newest per family per ref),
then the lowest-priority families while over budget, and summarises what it
kept and deleted. docs/reference/ci.md: the caches policy, and the Play
upload, pull request supersede, cross-repository admission and shared
reservation cap from the previous commits.
zeron/zgx moved the beta's and stable release's builds into zgx-build.yml:
its zgx-beta.yml and packaging scripts are taken as they are there (the
parallel engine/interface beta jobs are left to a follow-up on
zgx-build.yml). zgx-build.yml follows the cache policy: no rust-cache saves
from tags, Ghostty restored only, setup-zig's own cache off.
@katulevskiy katulevskiy added the ci:priority CI: this PR's runs go to the front of the queue label Nov 12, 2026
…ogs mixed keys

ci-supersede.yml runs on pull_request_target with actions: write. The head
branch and repository names, which a fork's author chooses, were spliced
into the jq filter text, so a crafted branch name could widen the filter
and force-cancel any run, releases included. They're --arg values now (the
branch also percent-encoded in the URL), the filter takes pull_request runs
of ci.yml only, and only a run ID of digits is cancelled.

admit.py's progress line sorted the priority and lock sets, which mix this
repository's pull request numbers with a peer's `owner/repo#n` keys:
sorting int with str raised TypeError. They're sorted as text.
…roup

The workflow-level concurrency group put every run of android-play.yml in
it, and every completed CI run on a branch named main starts one: a fork's
pull request from its own main, the nightly, a run by hand. Each cancelled
the upload waiting for its reviewer. The group is on the `play` job now,
which only a trusted push to main with a bundle reaches, so the newest
build still cancels the older one waiting for approval and nothing else
does.
… read

rank() ordered locks by lock time, then by their admit job's creation;
decide()'s lock_is_older by lock time, then by the key as text. Two locks
added in the same second could disagree in the two orders and each wait
for the other. rank() breaks the tie by the key as text too.

The reservation and pull request claim patterns take at most 4 digits: a
name comes from another run's editable ci.yml, and int() of a number
thousands of digits long raises ValueError, which failed every waiting
run's poll until each gave up and started without a reservation.
passed_jobs listed `<key>-` entries for every ref in one page of 100 and
kept the trusted ones. Pull requests, a fork's among them, can save entries
under any key: enough of them fill the page and push main's and the pull
request's own passes off it, so a suite that passed reran. It asks per
trusted ref (`&ref=`), reading every page.
The janitor counted pass records in the budget total. Any pull request,
a fork's too, can save any number of them, so enough of them made the
janitor drop the lowest-priority caches. They don't count now.

It also kept pass records from every ref. Only main's, zeron/zgx's and a
pull request's merge ref's are read by any run (plan.py); the rest (a run
by hand on a side branch, the merge queue, a tag) are deleted. A fork's
pull request can't be told from a same-repository one by its ref, so all
of refs/pull/<n>/merge stay.
A pull request into zeron/zgx reused zeron/zgx's passes, and zgx-tests.yml
saved its caches from any push to it. zeron/zgx has no ruleset: anyone who
can push to it could record a pass, or save a zgx CLI or Ghostty build,
that every pull request into it then trusts and runs.

plan.py's integration_trusted reads the branch's rules on a push to it or a
pull request into it (GET /repos/{repo}/rules/branches/zeron%2Fzgx, which
the plan job's token reads on a public repository) and trusts it only with
both `pull_request` and `non_fast_forward`, or classic protection that
requires reviews and blocks force pushes when that is readable; any error
is no. Otherwise pull requests into it reuse main's passes only, a push to
it trusts main's, and plan's new `integration-trusted` output, passed to
zgx-tests.yml, keeps SAVE_CACHES false. The cache lint accepts that one
narrowed SAVE_CACHES form.

docs/reference/ci.md says what an admin has to do: give zeron/zgx main's
ruleset (pull request required, force pushes blocked).
…d itself

A pull request's runs restore from its own cache scope first, and its
earlier commits can write there: an edited zgx-tests.yml could save a fake
zgx CLI under the real key, which a later, clean commit's tests restore and
run. Ghostty's build, linked into the tests, the same, under any
`zgx-ghostty-` key (its restore keys take any).

On a pull request, a step lists the entries with that key in the pull
request's own scope (actions/caches?key=...&ref=refs/pull/<n>/merge) and
the restore runs only if there are none; otherwise, or if the API fails,
the CLI and Ghostty are built. The zgx jobs get `actions: read` (and ci.yml
grants it to the `zgx` call).
The 4-digit cap kept a giant number in an edited admit name from raising
in int(), but would also have stopped pull requests numbered 10000 and up
from claiming ci:priority or ci:lock (zgx is past 2000, ~70 a day). Pull
request numbers take up to 7 digits; reservations stay at 4.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:priority CI: this PR's runs go to the front of the queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant