Skip to content

Build from the cached dependency sources and pin every fetch in one place - #149

Merged
dimitrivlachos merged 8 commits into
mainfrom
centralise-dependencies
Oct 29, 2026
Merged

dimitrivlachos merged 8 commits into
mainfrom
centralise-dependencies

Conversation

@dimitrivlachos

@dimitrivlachos dimitrivlachos commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

The dependency cache from #146 was there to survive an unreachable git
host, and the first time it was needed it did not help: this run
restored every source, then re-cloned all of them and failed when
gitlab.com refused the Eigen submodule. FetchContent keeps
its record of completed downloads in the build tree, which every image
build starts without, so it deleted each restored source and cloned it
again. The cache never saved a fetch; it only stored what the run had
just cloned.

Now each cached source is handed to cmake as a
FETCHCONTENT_SOURCE_DIR_<NAME> override, so a warm run configures
without reaching any host, and anything missing is still fetched as
before. The Eigen clone that failed was for nothing: ffbidx already
compiles against the conda Eigen the top level finds, so the submodule
is now skipped and no configure-time fetch leaves GitHub. While tracing
which files decide the cache contents, the pins turned out to be spread
across five files with duplicates and drifting fallbacks, so they now
live in a single module.

Changes

  • Skipped ffbidx's only submodule, Eigen on gitlab.com. Every ffbidx
    source already compiles with the conda include/eigen3 on its path,
    because it links the Eigen3::Eigen the top level found first; the
    submodule was about 140M of clone that no ffbidx source compiled
    against.
  • Pinned lodepng and pocketfft, which followed a default branch and the
    cpp branch, to the commits CI was already building.
  • Passed each restored source to cmake as an override in the
    Dockerfile, and told git to trust the sources, which belong to the
    runner's user rather than root. The cache key no longer carries the
    commit SHA, so a warm run is an exact hit and uploads nothing.
  • Moved every FetchContent_Declare into
    cmake/Modules/Dependencies.cmake, each with a line on why it comes
    from git rather than conda. Since the first declaration of a name
    wins, this also fixes the versions dx2 would otherwise declare. The
    dead GitLab Eigen fallbacks in baseline/ are gone, the GTest
    fallback matches conda's 1.17.0, and the deps cache key hashes only
    this module and the Dockerfile.
  • Took nanobind 2.8.0 from conda, with the git tag kept as an exact
    FIND_PACKAGE_ARGS fallback.
  • Replaced Findlodepng.cmake with a target defined at the top level.
    The module always fetched rather than found, and the integrator only
    linked against it because spotfinder/ happened to be added first.
  • Took mdspan 0.6.0 from conda. The conda package exports only
    std::mdspan, so this bumps dx2 to dials/dx2@32090bf, which links
    that target instead of including from the fetched source directory.

See: dials/dx2#59

dials/dx2@32090bf links dx2 against std::mdspan rather than including
from the fetched source directory, which is what kept mdspan on git.
The submodule bump also brings in dials/dx2#58.
dimitrivlachos added a commit to dials/dx2 that referenced this pull request Oct 29, 2026
This gives the build/test workflow the same caching scheme as
[fast-feedback-service's image build][ffs-ci], including the fix from
DiamondLightSource/fast-feedback-service#149.
Every run used to start from nothing: each
translation unit was recompiled, and mdspan and GTest were cloned
fresh at configure time, so one unreachable git host was enough to
fail the job before anything compiled.

Now the ccache store and the FetchContent sources are carried between
runs with `actions/cache`. ccache is keyed per OS on the commit with a
prefix restore key, so each run restores the newest store and saves
its own. Restoring the sources is not enough on its own: FetchContent
keeps its record of completed downloads in the build tree, which every
CI run starts without, so it deletes a restored source and clones it
again. FFS hit exactly this. Each restored source is therefore handed
to cmake as a `FETCHCONTENT_SOURCE_DIR_<NAME>` override, and a warm
run configures without reaching any host. The dependency key hashes
`CMakeLists.txt` and `requirements.txt` without the commit, so a warm
run is an exact hit and uploads nothing, and a changed pin or
toolchain starts from an empty directory.

FFS builds inside BuildKit and needs `buildkit-cache-dance` to move
the caches in and out of the container. dx2 builds on the runner, so
the cache directories are used in place, and fork pull requests can
restore and save their own caches without any gating.

### Changes

- Cached the ccache store and installed ccache from conda-forge in the
  workflow, leaving `requirements.txt` to describe what dx2 itself
  needs. The compiler check compares contents rather than mtime, since
  a fresh conda install may not preserve the compiler's mtime.
- Cached the FetchContent base directory, passed each restored source
  to cmake as an override, and added a step that lists each restored
  dependency with its commit.
- Configured and built through `cmake -B` / `cmake --build` with ccache
  as the compiler launcher, zeroing the ccache stats before the build
  and printing them after so the log shows the hit rate of that run.
@dimitrivlachos
dimitrivlachos marked this pull request as ready for review October 29, 2026 17:37
@dimitrivlachos
dimitrivlachos merged commit 822ad23 into main Oct 29, 2026
9 checks passed
@dimitrivlachos
dimitrivlachos deleted the centralise-dependencies branch October 29, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant