Repository navigation
Build from the cached dependency sources and pin every fetch in one place - #149
Merged
Merged
Conversation
The cache never saved a fetch: every image build re-cloned each restored source and saved what it had just cloned.
dials/dx2@32090bf links dx2 against std::mdspan rather than including from the fetched source directory, which is what kept mdspan on git. The submodule bump also brings in dials/dx2#58.
dimitrivlachos
added a commit
to dials/dx2
that referenced
this pull request
Oct 29, 2026
This gives the build/test workflow the same caching scheme as [fast-feedback-service's image build][ffs-ci], including the fix from DiamondLightSource/fast-feedback-service#149. Every run used to start from nothing: each translation unit was recompiled, and mdspan and GTest were cloned fresh at configure time, so one unreachable git host was enough to fail the job before anything compiled. Now the ccache store and the FetchContent sources are carried between runs with `actions/cache`. ccache is keyed per OS on the commit with a prefix restore key, so each run restores the newest store and saves its own. Restoring the sources is not enough on its own: FetchContent keeps its record of completed downloads in the build tree, which every CI run starts without, so it deletes a restored source and clones it again. FFS hit exactly this. Each restored source is therefore handed to cmake as a `FETCHCONTENT_SOURCE_DIR_<NAME>` override, and a warm run configures without reaching any host. The dependency key hashes `CMakeLists.txt` and `requirements.txt` without the commit, so a warm run is an exact hit and uploads nothing, and a changed pin or toolchain starts from an empty directory. FFS builds inside BuildKit and needs `buildkit-cache-dance` to move the caches in and out of the container. dx2 builds on the runner, so the cache directories are used in place, and fork pull requests can restore and save their own caches without any gating. ### Changes - Cached the ccache store and installed ccache from conda-forge in the workflow, leaving `requirements.txt` to describe what dx2 itself needs. The compiler check compares contents rather than mtime, since a fresh conda install may not preserve the compiler's mtime. - Cached the FetchContent base directory, passed each restored source to cmake as an override, and added a step that lists each restored dependency with its commit. - Configured and built through `cmake -B` / `cmake --build` with ccache as the compiler launcher, zeroing the ccache stats before the build and printing them after so the log shows the hit rate of that run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dependency cache from #146 was there to survive an unreachable git
host, and the first time it was needed it did not help: this run
restored every source, then re-cloned all of them and failed when
gitlab.com refused the Eigen submodule. FetchContent keeps
its record of completed downloads in the build tree, which every image
build starts without, so it deleted each restored source and cloned it
again. The cache never saved a fetch; it only stored what the run had
just cloned.
Now each cached source is handed to cmake as a
FETCHCONTENT_SOURCE_DIR_<NAME>override, so a warm run configureswithout reaching any host, and anything missing is still fetched as
before. The Eigen clone that failed was for nothing: ffbidx already
compiles against the conda Eigen the top level finds, so the submodule
is now skipped and no configure-time fetch leaves GitHub. While tracing
which files decide the cache contents, the pins turned out to be spread
across five files with duplicates and drifting fallbacks, so they now
live in a single module.
Changes
source already compiles with the conda
include/eigen3on its path,because it links the
Eigen3::Eigenthe top level found first; thesubmodule was about 140M of clone that no ffbidx source compiled
against.
cppbranch, to the commits CI was already building.Dockerfile, and told git to trust the sources, which belong to therunner's user rather than root. The cache key no longer carries the
commit SHA, so a warm run is an exact hit and uploads nothing.
FetchContent_Declareintocmake/Modules/Dependencies.cmake, each with a line on why it comesfrom git rather than conda. Since the first declaration of a name
wins, this also fixes the versions dx2 would otherwise declare. The
dead GitLab Eigen fallbacks in
baseline/are gone, the GTestfallback matches conda's 1.17.0, and the deps cache key hashes only
this module and the
Dockerfile.FIND_PACKAGE_ARGSfallback.Findlodepng.cmakewith a target defined at the top level.The module always fetched rather than found, and the integrator only
linked against it because
spotfinder/happened to be added first.std::mdspan, so this bumps dx2 to dials/dx2@32090bf, which linksthat target instead of including from the fetched source directory.
See: dials/dx2#59