fix: bypass response cache for Anthropic compaction - #1127
Merged
rapids-bot[bot] merged 2 commits intoSep 19, 2026
Merged
rapids-bot[bot] merged 2 commits into
rapids-bot[bot] merged 2 commits into
Conversation
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
github-actions
Bot
added
size:M
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (44)
🧰 Additional context used📓 Path-based instructions (4)Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.⚙️ CodeRabbit configuration file Files:
In MDX files, top-of-file comments must use JSX comment delimiters: `{/*` to open and `*/}` to close.📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md) Files:
Run `just docs` when the docs site changed; `./scripts/build-docs.sh html` remains the compatibility wrapper📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md) Files:
Verify MDX files use JSX delimiters for top-of-file SPDX comments.📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md) Files:
🔇 Additional comments (1)
WalkthroughAnthropic compaction requests now bypass response caching. The change covers compaction fields, blocks, beta headers, and malformed context-management data. It adds telemetry, integration coverage, unit coverage, and documentation. ChangesAnthropic compaction cache handling
Priority: ⬇️ Low Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant AnthropicMessagesRequest
participant ResponseCache
participant Provider
AnthropicMessagesRequest->>ResponseCache: submit compaction-enabled request
ResponseCache->>ResponseCache: detect compaction and mark bypass
ResponseCache->>Provider: execute request without cache lookup or storage
Provider-->>ResponseCache: return native compaction_delta events
Merge Risk: ⚪ Minimal · up to Compaction requests avoid cached responses while preserving native provider events, with ordinary requests retaining cacheability; no concrete current-head production risk is established. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
willkill07
approved these changes
Sep 19, 2026
Member
|
/merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Anthropic compaction changes both the response stream and how later requests continue a conversation. Relay's optional response cache cannot yet reproduce those semantics faithfully, so this PR keeps compaction-related requests on the live provider path while leaving ordinary Anthropic Messages requests cacheable.
It covers requests that enable compaction and follow-up requests that carry a previous compaction block.
Details
context_management, the top-levelcompactionfield, or ananthropic-betacompaction token.context_managementdeclarations as unsafe for caching.anthropic_compactionas the response-cache bypass reason.compaction_deltastream.This complements #1035, which preserves compaction deltas when Relay assembles a live Anthropic stream. That collector fix does not make cached replay of the stream safe.
Validation completed:
just test-rustjust test-pythonjust test-nodejust test-gojust docsjust docs-linkcheckuv run pre-commit run --all-filesWhere should the reviewer start?
Start with
anthropic_compaction_requestedincrates/adaptive/src/response_cache/key.rs. It contains the cache-eligibility decision. The integration testanthropic_compaction_streams_bypass_lookup_and_storagethen shows the resulting behavior across two identical requests.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
Bug Fixes
anthropic_compactionreason.Documentation