Skip to content

ci(notebooks): give Notebook Validation its own HuggingFace model cache - #840

Open
vishal-bala wants to merge 7 commits into
ci/FDE-205-cache-and-concurrencyfrom
fix/notebook-hf-model-cache
Open

vishal-bala wants to merge 7 commits into
ci/FDE-205-cache-and-concurrencyfrom
fix/notebook-hf-model-cache

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Notebook Validation intermittently fails with nbval's 60-second cell timeout on cells that construct HFTextVectorizer(model="redis/langcache-embed-v2"). The job restored the HuggingFace cache that the test jobs share, and that cache holds only the models the tests load. Two models are loaded only by the user-guide notebooks, redis/langcache-embed-v2 and BAAI/bge-reranker-base, so they were never in it, and actions/cache does not save again after an exact key hit, so nothing ever added them. Both models were therefore downloaded inside nbval cells on every run. On run 41813891110 the notebook job restored the shared cache with an exact hit, and then both cells that load langcache-embed-v2 timed out inside huggingface_hub's xet_get, part-way through the weight download.

Changes

A model cache owned by the notebook job

The notebook job restores its own cache, <os>-hf-notebooks-<hash>, keyed on the hash of .github/scripts/prefetch_notebook_models.py. A new step runs that script before nbval. It loads each listed model through HFTextVectorizer or HFCrossEncoderReranker, the classes the notebooks use, so the cache holds exactly the files they read. The cache is saved straight after that step, not in a post-job step, so a failing notebook cannot stop the save. The notebook job is the only job that saves this key. It runs only for same-repository, non-Dependabot events, and the fork-PR workflow does not run notebooks, so no untrusted code reaches the save.

nbval then runs with HF_HUB_OFFLINE=1. A notebook that loads a model missing from the script fails at once with an OSError naming offline mode, instead of downloading against the cell timeout. Adding a model to the notebooks therefore means adding it to the script, and the comment beside HF_HUB_OFFLINE says so. The restore has no restore-keys fallback, because an older entry would still hold models since removed from the list and nbval would pass offline on them. A change to the list therefore starts from an empty cache and downloads the listed models once, in the prefetch step.

One langcache model in the user guide

03_llmcache.ipynb used langcache-embed-v2 for its first cache, but its private and multi-field caches passed no vectorizer and so loaded the SemanticCache default, langcache-embed-v1. 10_embeddings_cache.ipynb named v1 explicitly. Both now use v2: 03_llmcache builds one vectorizer and passes it to all three caches, which also stops the guide loading the model three times. The prefetch list therefore holds three models: langcache-embed-v2, sentence-transformers/all-mpnet-base-v2 (the HFTextVectorizer default) and BAAI/bge-reranker-base. The saved outputs that depend on the model were refreshed by re-running those cells. The filtered check() results in 03_llmcache are the same under v2, and the stored key hash and printed model name in 10_embeddings_cache changed. The library default is unchanged.

  • HF_TOKEN moves from the nbval step, where offline mode makes it unused, to the download step.
  • The service-tests comment now says the notebook job restores that job's uv cache but keeps its own model cache.

Notes

The prefetch step runs on every run, including an exact cache hit. On a warm cache it only checks each file against the hub, and it is what fills in any file a newer sentence-transformers or transformers release needs from a model the cached snapshot lacks; skipping it on a hit would leave nbval to fail offline after such a dependency bump until the model list next changed.

The notebook job keeps needs: service-tests with the !cancelled() guard. It no longer reads the shared HuggingFace cache, but it still restores the Python 3.11 uv cache that service-tests saves, and the ordering is what lets it.

This PR is stacked on #859, which reworks caching and run concurrency across .github/workflows/, and merges after it. Its base is #859's branch, so the diff shows only this PR's changes, including the notebook job's restore step that replaces #859's restore of the shared model cache.

AI Assistance

  • No AI assistance was used for this PR.
  • This PR is AI-assisted. I have reviewed and understood everything in it, and I take responsibility for it.

Claude Code (claude-opus-5-5): diagnosis from the CI logs, the workflow and script changes, the notebook edits, this description and the pre-PR review.

Reviewer Brief

Start here: .github/workflows/test.yml:477-522, the notebook job's restore, prefetch and save steps. They decide what is in the cache that nbval then reads offline.

Where it could fail

  • .github/workflows/test.yml:574: with HF_HUB_OFFLINE=1, any notebook that loads a model not listed in .github/scripts/prefetch_notebook_models.py:15-22, including through a class default such as SemanticCache() or HFTextVectorizer(), now fails rather than downloading; check that the list covers every model a non-NBVAL_SKIP cell loads.
  • docs/user_guide/03_llmcache.ipynb and 10_embeddings_cache.ipynb: nbval runs with --nbval-lax, so CI never compares the refreshed stored outputs with what the cells print.

Not verified: the cold-cache path, which this PR's first CI run will exercise, and the fail-fast OSError from an unlisted model inside nbval in CI.

Beyond CI: the listed models load offline from a local cache in 0.5 to 5 s each, and an unlisted model fails at once with an OSError; 06_rerankers.ipynb passes under nbval offline apart from its Cohere and VoyageAI cells, run without keys; the changed cells of 03_llmcache and 10_embeddings_cache ran without errors against Redis 8.4 with HF_HUB_OFFLINE=1.

Since the last brief: merged #859's latest head (ca4e2a74) as db65dca8; this PR's own lines are unchanged, apart from where they sit in test.yml.

Reviewed 38de9eb8, fixes in 9439e87a · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs


Note

Low Risk
Changes are confined to CI notebook validation and user-guide notebooks; offline mode requires keeping the prefetch script aligned with every notebook model load.

Overview
Fixes Notebook Validation timeouts by moving HuggingFace downloads out of nbval cells and giving the job its own model cache.

The workflow adds .github/scripts/prefetch_notebook_models.py and notebook-job steps to restore, prefetch, and save models under a cache key hashed from that script (no restore-keys fallback). nbval runs with HF_HUB_OFFLINE=1 so missing models fail fast instead of downloading during the 60s cell limit.

User-guide notebooks 03_llmcache and 10_embeddings_cache now standardize on redis/langcache-embed-v2 (shared vectorizer in the LLM cache guide) with refreshed stored outputs.

Reviewed by Cursor Bugbot for commit db65dca. Bugbot is set up for automated code reviews on this repo. Configure here.

The notebook job shared the fixed `Linux-hf-cache` key with the test jobs.
Whichever test job misses first saves that entry, so it holds only the
models the tests load, and actions/cache never re-saves after an exact
hit. redis/langcache-embed-v2 and BAAI/bge-reranker-base are loaded only
by notebooks, so they were downloaded inside nbval cells on every run;
PR run 41813891110 timed out at 60s in huggingface_hub's xet_get right
after an exact cache hit.

The notebook job now restores a cache keyed on the hash of
.github/scripts/prefetch_notebook_models.py, downloads the listed models
before nbval, and saves straight after that step, so a failing notebook
cannot stop the save. nbval runs with HF_HUB_OFFLINE=1, so a notebook
that loads an unlisted model fails at once instead of downloading
against the cell timeout. The `needs: service-tests` ordering existed
only to wait for the shared cache, so it is dropped.
03_llmcache loaded v2 for its first cache but fell back to the
SemanticCache default, v1, for the private and multi-field caches, and
10_embeddings_cache named v1 explicitly. The notebook job therefore
downloaded and cached two 572 MB models where one does.

03 now builds one v2 vectorizer and passes it to all three caches, and
10 uses v2. The prefetch list drops v1. Outputs that depend on the
model were refreshed by re-running those cells; the filtered check()
results in 03 are unchanged under v2. The library default stays v1.
Dropping `needs: service-tests` was wrong: besides the shared HF cache,
it orders the notebook job after the job that saves the Python 3.11 uv
cache, which the notebook job restores. Restore it with the same
`!cancelled()` guard the test matrix uses.

Move the notebook HF restore and save steps to actions/cache@v5, the
version the service-tests action already uses.
A prefix restore brought back the previous entry whole, so a model
removed from prefetch_notebook_models.py stayed in every later entry.
A notebook still loading it would pass nbval offline until the chain
of entries was evicted, which hid exactly the drift HF_HUB_OFFLINE=1
is there to catch. A model-list change now costs one cold download in
the prefetch step, outside nbval's per-cell timeout.
@vishal-bala vishal-bala added the auto:ci Updates to CI/CD workflows and processes label Nov 19, 2026
@vishal-bala
vishal-bala changed the base branch from main to ci/FDE-205-cache-and-concurrency November 19, 2026 15:25
@vishal-bala
vishal-bala added this pull request to stack #860 November 19, 2026 15:25
…d-concurrency' into fix/notebook-hf-model-cache
@vishal-bala vishal-bala changed the title ci: prefetch notebook HF models into their own cache ci(notebooks): give Notebook Validation its own HuggingFace model cache Nov 19, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.57%. Comparing base (ca4e2a7) to head (db65dca).

Additional details and impacted files
@@                         Coverage Diff                          @@
##           ci/FDE-205-cache-and-concurrency     #840      +/-   ##
====================================================================
- Coverage                             79.59%   79.57%   -0.02%     
====================================================================
  Files                                   121      121              
  Lines                                 14852    14852              
  Branches                               2599     2599              
====================================================================
- Hits                                  11821    11819       -2     
- Misses                                 2329     2330       +1     
- Partials                                702      703       +1     
Flag Coverage Δ
core 78.73% <ø> (-0.02%) ⬇️
service 79.41% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vishal-bala
vishal-bala marked this pull request as ready for review November 20, 2026 10:12
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T10:14:05.633719Z db65dca Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:ci Updates to CI/CD workflows and processes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants