Skip to content

Cache ccache and the fetched dependency sources in CI - #59

Merged
dimitrivlachos merged 4 commits into
mainfrom
ci-cache
Oct 29, 2026
Merged

dimitrivlachos merged 4 commits into
mainfrom
ci-cache

Conversation

@dimitrivlachos

@dimitrivlachos dimitrivlachos commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

This PR gives the build/test workflow the same caching scheme as
fast-feedback-service's image build, including the fix from
FFS #149. Every run used to start from nothing: each
translation unit was recompiled, and mdspan and GTest were cloned
fresh at configure time, so one unreachable git host was enough to
fail the job before anything compiled.

Now the ccache store and the FetchContent sources are carried between
runs with actions/cache. ccache is keyed per OS on the commit with a
prefix restore key, so each run restores the newest store and saves
its own. Restoring the sources is not enough on its own: FetchContent
keeps its record of completed downloads in the build tree, which every
CI run starts without, so it deletes a restored source and clones it
again. FFS hit exactly this. Each restored source is therefore handed
to cmake as a FETCHCONTENT_SOURCE_DIR_<NAME> override, and a warm
run configures without reaching any host. The dependency key hashes
CMakeLists.txt and requirements.txt without the commit, so a warm
run is an exact hit and uploads nothing, and a changed pin or
toolchain starts from an empty directory.

FFS builds inside BuildKit and needs buildkit-cache-dance to move
the caches in and out of the container. dx2 builds on the runner, so
the cache directories are used in place, and fork pull requests can
restore and save their own caches without any gating.

Changes

  • Cached the ccache store and installed ccache from conda-forge in the
    workflow, leaving requirements.txt to describe what dx2 itself
    needs. The compiler check compares contents rather than mtime, since
    a fresh conda install may not preserve the compiler's mtime.
  • Cached the FetchContent base directory, passed each restored source
    to cmake as an override, and added a step that lists each restored
    dependency with its commit.
  • Configured and built through cmake -B / cmake --build with ccache
    as the compiler launcher, zeroing the ccache stats before the build
    and printing them after so the log shows the hit rate of that run.

Testing

The first attempt of the push run started with empty caches,
cloned both dependencies and missed ccache on every compile. The
re-run
at the same commit restored both caches as exact hits,
cloned nothing, and hit ccache on all 23 compiles:

Cold build step Warm build step Cold job Warm job
ubuntu 2m14s 5s 3m18s 1m18s
macOS 2m32s 14s 4m24s 1m28s

See: FFS #149

FetchContent keeps its record of completed downloads in the build tree,
which every run starts without, so it deleted each restored source and
cloned it again. The cache directories also move out of the job env,
where the runner context is not available.
@dimitrivlachos
dimitrivlachos marked this pull request as ready for review October 29, 2026 17:33
@dimitrivlachos
dimitrivlachos merged commit 264c821 into main Oct 29, 2026
9 checks passed
@dimitrivlachos
dimitrivlachos deleted the ci-cache branch October 29, 2026 17:35
dimitrivlachos added a commit to DiamondLightSource/fast-feedback-service that referenced this pull request Oct 29, 2026
…lace (#149)

The dependency cache from #146 was there to survive an unreachable git
host, and the first time it was needed it did not help: [this run](https://github.com/DiamondLightSource/fast-feedback-service/actions/runs/40894825263/job/123520943248)
restored every source, then re-cloned all of them and failed when
gitlab.com refused the Eigen submodule. FetchContent keeps
its record of completed downloads in the build tree, which every image
build starts without, so it deleted each restored source and cloned it
again. The cache never saved a fetch; it only stored what the run had
just cloned.

Now each cached source is handed to cmake as a
`FETCHCONTENT_SOURCE_DIR_<NAME>` override, so a warm run configures
without reaching any host, and anything missing is still fetched as
before. The Eigen clone that failed was for nothing: ffbidx already
compiles against the conda Eigen the top level finds, so the submodule
is now skipped and no configure-time fetch leaves GitHub. While tracing
which files decide the cache contents, the pins turned out to be spread
across five files with duplicates and drifting fallbacks, so they now
live in a single module.

### Changes

- Skipped ffbidx's only submodule, Eigen on gitlab.com. Every ffbidx
  source already compiles with the conda `include/eigen3` on its path,
  because it links the `Eigen3::Eigen` the top level found first; the
  submodule was about 140M of clone that no ffbidx source compiled
  against.
- Pinned lodepng and pocketfft, which followed a default branch and the
  `cpp` branch, to the commits CI was already building.
- Passed each restored source to cmake as an override in the
  `Dockerfile`, and told git to trust the sources, which belong to the
  runner's user rather than root. The cache key no longer carries the
  commit SHA, so a warm run is an exact hit and uploads nothing.
- Moved every `FetchContent_Declare` into
  `cmake/Modules/Dependencies.cmake`, each with a line on why it comes
  from git rather than conda. Since the first declaration of a name
  wins, this also fixes the versions dx2 would otherwise declare. The
  dead GitLab Eigen fallbacks in `baseline/` are gone, the GTest
  fallback matches conda's 1.17.0, and the deps cache key hashes only
  this module and the `Dockerfile`.
- Took nanobind 2.8.0 from conda, with the git tag kept as an exact
  `FIND_PACKAGE_ARGS` fallback.
- Replaced `Findlodepng.cmake` with a target defined at the top level.
  The module always fetched rather than found, and the integrator only
  linked against it because `spotfinder/` happened to be added first.
- Took mdspan 0.6.0 from conda. The conda package exports only
  `std::mdspan`, so this bumps dx2 to dials/dx2@32090bf, which links
  that target instead of including from the fetched source directory.

See: dials/dx2#59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant