Forward-merge release/0.9 into main - #1129
Open
rapids-bot[bot] wants to merge 1 commit into
Open
rapids-bot[bot] wants to merge 1 commit into
rapids-bot[bot] wants to merge 1 commit into
Conversation
#### Overview Anthropic compaction changes both the response stream and how later requests continue a conversation. Relay's optional response cache cannot yet reproduce those semantics faithfully, so this PR keeps compaction-related requests on the live provider path while leaving ordinary Anthropic Messages requests cacheable. It covers requests that enable compaction and follow-up requests that carry a previous compaction block. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Recognize compaction configured through `context_management`, the top-level `compaction` field, or an `anthropic-beta` compaction token. - Recognize conversations that continue from an existing compaction content block. - Treat malformed or incomplete `context_management` declarations as unsafe for caching. - Keep ordinary Anthropic Messages requests and known non-compaction context-management edits cacheable. - Report `anthropic_compaction` as the response-cache bypass reason. - Add unit coverage for compaction, continuation, malformed, and non-compaction request shapes. - Add an integration test showing that repeated compaction requests both reach the provider and forward the native `compaction_delta` stream. - Document the cache limitation and bypass behavior. This complements #1035, which preserves compaction deltas when Relay assembles a live Anthropic stream. That collector fix does not make cached replay of the stream safe. Validation completed: - `just test-rust` - `just test-python` - `just test-node` - `just test-go` - `just docs` - `just docs-linkcheck` - `uv run pre-commit run --all-files` #### Where should the reviewer start? Start with `anthropic_compaction_requested` in `crates/adaptive/src/response_cache/key.rs`. It contains the cache-eligibility decision. The integration test `anthropic_compaction_streams_bypass_lookup_and_storage` then shows the resulting behavior across two identical requests. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to #1034 ## Summary by CodeRabbit * **Bug Fixes** * Anthropic Messages requests using server-side compaction or compaction-related beta features now bypass response caching, preserving the native compaction lifecycle. * Compaction events are forwarded unchanged during streaming responses. * Cache-bypass telemetry now reports the `anthropic_compaction` reason. * **Documentation** * Updated response-cache documentation to describe compaction-related bypass behavior and observability. * Documented the known limitation for requests continuing from compaction blocks. Authors: - Maryam Najafian (https://github.com/mnajafian-nv) Approvers: - Will Killian (https://github.com/willkill07) URL: #1127
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
github-actions
Bot
added
size:M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/0.9 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.