Repository navigation
ci(notebooks): give Notebook Validation its own HuggingFace model cache - #840
Open
vishal-bala wants to merge 7 commits into
Open
vishal-bala wants to merge 7 commits into
vishal-bala wants to merge 7 commits into
Conversation
The notebook job shared the fixed `Linux-hf-cache` key with the test jobs. Whichever test job misses first saves that entry, so it holds only the models the tests load, and actions/cache never re-saves after an exact hit. redis/langcache-embed-v2 and BAAI/bge-reranker-base are loaded only by notebooks, so they were downloaded inside nbval cells on every run; PR run 41813891110 timed out at 60s in huggingface_hub's xet_get right after an exact cache hit. The notebook job now restores a cache keyed on the hash of .github/scripts/prefetch_notebook_models.py, downloads the listed models before nbval, and saves straight after that step, so a failing notebook cannot stop the save. nbval runs with HF_HUB_OFFLINE=1, so a notebook that loads an unlisted model fails at once instead of downloading against the cell timeout. The `needs: service-tests` ordering existed only to wait for the shared cache, so it is dropped.
03_llmcache loaded v2 for its first cache but fell back to the SemanticCache default, v1, for the private and multi-field caches, and 10_embeddings_cache named v1 explicitly. The notebook job therefore downloaded and cached two 572 MB models where one does. 03 now builds one v2 vectorizer and passes it to all three caches, and 10 uses v2. The prefetch list drops v1. Outputs that depend on the model were refreshed by re-running those cells; the filtered check() results in 03 are unchanged under v2. The library default stays v1.
Dropping `needs: service-tests` was wrong: besides the shared HF cache, it orders the notebook job after the job that saves the Python 3.11 uv cache, which the notebook job restores. Restore it with the same `!cancelled()` guard the test matrix uses. Move the notebook HF restore and save steps to actions/cache@v5, the version the service-tests action already uses.
…otebook-hf-model-cache
A prefix restore brought back the previous entry whole, so a model removed from prefetch_notebook_models.py stayed in every later entry. A notebook still loading it would pass nbval offline until the chain of entries was evicted, which hid exactly the drift HF_HUB_OFFLINE=1 is there to catch. A model-list change now costs one cold download in the prefetch step, outside nbval's per-cell timeout.
Open
1 of 2 tasks
vishal-bala
changed the base branch from
main
to
ci/FDE-205-cache-and-concurrency
November 19, 2026 15:25
…d-concurrency' into fix/notebook-hf-model-cache
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## ci/FDE-205-cache-and-concurrency #840 +/- ##
====================================================================
- Coverage 79.59% 79.57% -0.02%
====================================================================
Files 121 121
Lines 14852 14852
Branches 2599 2599
====================================================================
- Hits 11821 11819 -2
- Misses 2329 2330 +1
- Partials 702 703 +1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
vishal-bala
marked this pull request as ready for review
November 20, 2026 10:12
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Notebook Validation intermittently fails with nbval's 60-second cell timeout on cells that construct
HFTextVectorizer(model="redis/langcache-embed-v2"). The job restored the HuggingFace cache that the test jobs share, and that cache holds only the models the tests load. Two models are loaded only by the user-guide notebooks,redis/langcache-embed-v2andBAAI/bge-reranker-base, so they were never in it, and actions/cache does not save again after an exact key hit, so nothing ever added them. Both models were therefore downloaded inside nbval cells on every run. On run 41813891110 the notebook job restored the shared cache with an exact hit, and then both cells that loadlangcache-embed-v2timed out insidehuggingface_hub'sxet_get, part-way through the weight download.Changes
A model cache owned by the notebook job
The notebook job restores its own cache,
<os>-hf-notebooks-<hash>, keyed on the hash of.github/scripts/prefetch_notebook_models.py. A new step runs that script before nbval. It loads each listed model throughHFTextVectorizerorHFCrossEncoderReranker, the classes the notebooks use, so the cache holds exactly the files they read. The cache is saved straight after that step, not in a post-job step, so a failing notebook cannot stop the save. The notebook job is the only job that saves this key. It runs only for same-repository, non-Dependabot events, and the fork-PR workflow does not run notebooks, so no untrusted code reaches the save.nbval then runs with
HF_HUB_OFFLINE=1. A notebook that loads a model missing from the script fails at once with anOSErrornaming offline mode, instead of downloading against the cell timeout. Adding a model to the notebooks therefore means adding it to the script, and the comment besideHF_HUB_OFFLINEsays so. The restore has norestore-keysfallback, because an older entry would still hold models since removed from the list and nbval would pass offline on them. A change to the list therefore starts from an empty cache and downloads the listed models once, in the prefetch step.One langcache model in the user guide
03_llmcache.ipynbusedlangcache-embed-v2for its first cache, but its private and multi-field caches passed no vectorizer and so loaded theSemanticCachedefault,langcache-embed-v1.10_embeddings_cache.ipynbnamed v1 explicitly. Both now use v2:03_llmcachebuilds one vectorizer and passes it to all three caches, which also stops the guide loading the model three times. The prefetch list therefore holds three models:langcache-embed-v2,sentence-transformers/all-mpnet-base-v2(theHFTextVectorizerdefault) andBAAI/bge-reranker-base. The saved outputs that depend on the model were refreshed by re-running those cells. The filteredcheck()results in03_llmcacheare the same under v2, and the stored key hash and printed model name in10_embeddings_cachechanged. The library default is unchanged.HF_TOKENmoves from the nbval step, where offline mode makes it unused, to the download step.service-testscomment now says the notebook job restores that job's uv cache but keeps its own model cache.Notes
The prefetch step runs on every run, including an exact cache hit. On a warm cache it only checks each file against the hub, and it is what fills in any file a newer sentence-transformers or transformers release needs from a model the cached snapshot lacks; skipping it on a hit would leave nbval to fail offline after such a dependency bump until the model list next changed.
The notebook job keeps
needs: service-testswith the!cancelled()guard. It no longer reads the shared HuggingFace cache, but it still restores the Python 3.11 uv cache thatservice-testssaves, and the ordering is what lets it.This PR is stacked on #859, which reworks caching and run concurrency across
.github/workflows/, and merges after it. Its base is #859's branch, so the diff shows only this PR's changes, including the notebook job's restore step that replaces #859's restore of the shared model cache.AI Assistance
Claude Code (claude-opus-5-5): diagnosis from the CI logs, the workflow and script changes, the notebook edits, this description and the pre-PR review.
Reviewer Brief
Start here:
.github/workflows/test.yml:477-522, the notebook job's restore, prefetch and save steps. They decide what is in the cache that nbval then reads offline.Where it could fail
.github/workflows/test.yml:574: withHF_HUB_OFFLINE=1, any notebook that loads a model not listed in.github/scripts/prefetch_notebook_models.py:15-22, including through a class default such asSemanticCache()orHFTextVectorizer(), now fails rather than downloading; check that the list covers every model a non-NBVAL_SKIPcell loads.docs/user_guide/03_llmcache.ipynband10_embeddings_cache.ipynb: nbval runs with--nbval-lax, so CI never compares the refreshed stored outputs with what the cells print.Not verified: the cold-cache path, which this PR's first CI run will exercise, and the fail-fast
OSErrorfrom an unlisted model inside nbval in CI.Beyond CI: the listed models load offline from a local cache in 0.5 to 5 s each, and an unlisted model fails at once with an
OSError;06_rerankers.ipynbpasses under nbval offline apart from its Cohere and VoyageAI cells, run without keys; the changed cells of03_llmcacheand10_embeddings_cacheran without errors against Redis 8.4 withHF_HUB_OFFLINE=1.Since the last brief: merged #859's latest head (
ca4e2a74) asdb65dca8; this PR's own lines are unchanged, apart from where they sit intest.yml.Reviewed
38de9eb8, fixes in9439e87a· Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docsNote
Low Risk
Changes are confined to CI notebook validation and user-guide notebooks; offline mode requires keeping the prefetch script aligned with every notebook model load.
Overview
Fixes Notebook Validation timeouts by moving HuggingFace downloads out of nbval cells and giving the job its own model cache.
The workflow adds
.github/scripts/prefetch_notebook_models.pyand notebook-job steps to restore, prefetch, and save models under a cache key hashed from that script (norestore-keysfallback). nbval runs withHF_HUB_OFFLINE=1so missing models fail fast instead of downloading during the 60s cell limit.User-guide notebooks
03_llmcacheand10_embeddings_cachenow standardize onredis/langcache-embed-v2(shared vectorizer in the LLM cache guide) with refreshed stored outputs.Reviewed by Cursor Bugbot for commit db65dca. Bugbot is set up for automated code reviews on this repo. Configure here.