Skip to content

ci: share small pruned uv caches, one saver per key, and cancel superseded PR runs (FDE-205) - #859

Open
vishal-bala wants to merge 4 commits into
mainfrom
ci/FDE-205-cache-and-concurrency
Open

vishal-bala wants to merge 4 commits into
mainfrom
ci/FDE-205-cache-and-concurrency

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

CI spends more time installing dependencies than running tests, because its caches are too big to stay in GitHub's cache. Each uv cache is about 3 GB, mostly CUDA torch, and together they come to about 44 GB against a 10 GB limit, so GitHub keeps evicting them. Two more problems make it worse. Any change to uv.lock misses every cache at once, and dozens of jobs race to save the same key, so the job that wins is often one that saved almost nothing. A cold Python 3.15 leg then rebuilds 16 packages from source and spends 5 to 6 minutes installing.

Changes

Keep only what is slow to rebuild, and let one job save it

Before saving, the job runs uv cache prune --ci. This keeps the wheels built from source, which are slow to rebuild, and drops the pre-built ones, which download faster than they restore. The cache shrinks from gigabytes to the handful of packages that need compiling.

Each cache now has one saver. service-tests saves the default Python version, and one matrix leg (redis-py 8.x on redis:latest) saves each of the others. Every other job only restores, and docs, coverage, the lock check and the release jobs don't cache at all.

When the key misses, the restore falls back to the newest cache for the same Python version. A uv.lock change, the release bot's version bump included, then rebuilds only what changed. setup-uv's built-in cache has no such fallback, so actions/cache/restore and actions/cache/save now do the caching around the install.

Fork PRs can no longer write caches that main reads

test-fork-pr.yml runs fork code on main, so anything it saved would be restored by every later run. Its job now sets cache-mode: read, which gives it a cache token that can restore but not save. As a backup, it runs the service-tests action from this repository rather than from the fork's copy, with saving off.

A new push to a PR cancels that PR's previous run

test.yml and lint.yml now cancel a PR's previous run when a new commit is pushed. Pushes to main, nightly runs and manual dispatches are never cancelled.

  • The HuggingFace model cache gets a new key, -v2. Only service-tests saves it, after its tests pass.
  • Jobs are pinned to ubuntu-24.04, because the cache key names the OS.
  • Test legs skip the docs dependency group, which no test imports.
  • actions/setup-python moves to v6 in test.yml, matching the other workflows.

Notes

The cache key is written out in five places, each marked "keep in sync". It can't live in a shared local action, because the fork workflow would load that action from the fork's checkout. A copy that drifts only costs its job a cache miss.

#840 is stacked on this PR and merges after it.

AI Assistance

  • No AI assistance was used for this PR.
  • This PR is AI-assisted. I have reviewed and understood everything in it, and I take responsibility for it.

Claude Code (claude-opus-5-5): implementation, pre-PR review and this description.

Reviewer Brief

Start here: .github/workflows/test.yml:155-201, the test job's uv restore, prune and save steps. Every other job copies their key and rules.

Where it could fail

  • .github/workflows/test-fork-pr.yml:71: GitHub documents cache-mode, but actionlint doesn't know it yet. If GitHub rejected it, the fork workflow would fail to start.
  • .github/workflows/test.yml:194-198: the saver leg is chosen by matrix values (8.x, redis:latest). If either value is removed, those caches stop being saved and nothing reports it.
  • .github/actions/run-service-tests/action.yml:112: the model cache is saved only when service-tests passes, so it stays empty while that job is red on main.

Not verified: no workflow was run. This leaves the first saves, the fallback after a lock change, the pruned sizes, cache-mode and a fork-PR dispatch unobserved.

Beyond CI: make check passes locally on Python 3.15.2 (3092 passed, 146 skipped). actionlint 1.7.12 passes every workflow except the cache-mode key.

Reviewed 8efea9ca, fixes in ca4e2a74 · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs


Note

Medium Risk
CI-only changes, but incorrect cache saver matrix pairing or ignored cache-mode could silently degrade installs or weaken fork isolation; release jobs intentionally skip cache to avoid competing with test caches.

Overview
Overhauls GitHub Actions caching so pruned uv and HuggingFace caches stay under the 10 GB limit and aren’t lost on every uv.lock change.

Caching model: Disables setup-uv’s built-in cache and uses actions/cache/restore + save with restore-keys fallbacks. Before save, jobs run uv cache prune --ci so only slow-to-build wheels are cached. One designated job per cache key writes: service-tests for the default Python version (and HF models after success); each other Python version is saved by a single matrix leg (redis-py 8.x + redis:latest). Other jobs restore only; release, lock-check, coverage, and docs skip caching entirely. HF key bumps to hf-cache-v2. Runners pin to ubuntu-24.04 to match hard-coded cache keys.

Workflow behavior: concurrency on test.yml and lint.yml cancels superseded PR runs. Test matrix installs use --no-group docs. setup-python@v6 in test.yml.

Fork PR safety: Fork testing uses cache-mode: read, checks out trusted .github/actions from upstream, save-cache: false, and persist-credentials: false on the fork checkout so untrusted code can’t poison shared caches.

Reviewed by Cursor Bugbot for commit ca4e2a7. Bugbot is set up for automated code reviews on this repo. Configure here.

… runs (FDE-205)

- Replace setup-uv's built-in cache (no restore-keys) with
  actions/cache/restore + save, pruned with `uv cache prune --ci` and
  saved after install, before the tests. A uv.lock change now gets a
  partial hit instead of a cold rebuild.
- One saver per Python version: service-tests saves 3.11, the
  redis-py 8.x x redis:8.4 leg saves the others; every other job is
  restore-only. docs and lint's lock job no longer cache.
- HuggingFace cache: versioned key (-v2), restore-only in the matrix
  and notebooks, saved only by the canary; off actions/cache@v4.
- test-fork-pr.yml runs the service-tests action from this repository
  with save-cache false, so a fork PR cannot save main-scoped caches.
- Cancel superseded PR runs in test.yml and lint.yml; never cancel
  pushes, schedule or dispatch.
- Pin runs-on: ubuntu-24.04 (the cache key names the OS), drop the docs
  group from test legs, setup-python v5 -> v6.
…-205)

- test-fork-pr.yml: cache-mode: read on the job that runs fork code,
  and persist-credentials: false on the fork checkout; the comment now
  says what this does and does not stop.
- Pick the non-3.11 uv saver by redis:latest, the image value that
  never rotates, and note the dependency beside the matrix.
- Correct comments on the docs group, the canary's HF save and the
  runs-on pins.
Resolve the run-service-tests action and test.yml conflicts with the
coverage change (#826): keep both the save-cache and pytest-args inputs,
and save the HF cache after the coverage-enabled test run. Bring the new
coverage and cluster-tests jobs onto the same caching rules: coverage
does not cache, cluster-tests restores the 3.11 uv and v2 HF caches
without saving, and both pin ubuntu-24.04.
…(FDE-205)

- auto-release.yml: enable-cache false in all three setup-uv steps. Its
  saves competed with the test caches for the 10 GB, and the publish job
  should not restore anything from the shared cache.
- test-fork-pr.yml: cache-mode read is enforced through a read-only
  cache token, so say so; the trusted-actions checkout is now described
  as the second guard it is.
- test.yml: name PYTHON_VERSION rather than 3.11 where the code keys off
  it, and drop two stale reasons from the coverage and docs comments.
@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@vishal-bala vishal-bala added the auto:ci Updates to CI/CD workflows and processes label Nov 20, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review November 20, 2026 10:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T10:13:16.947553Z ca4e2a7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:ci Updates to CI/CD workflows and processes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants