Explicit cache breakpoints in OpenAI Responses, fix copy beyond viewport - #179
Merged
Conversation
OpenAI's automatic prompt caching often misses in practice, showing up as ballooning uncached input tokens in agent sessions. GPT-5.6+ models support explicit cache control via prompt_cache_breakpoint markers on input content blocks, analogous to Anthropic's cache_control. - Extract the Anthropic cache-marker placement (stable prefix via volatile flag, markers every 5 messages) into a shared prompt_caching module and reuse it for both OpenAI Responses clients (HTTP and WebSocket). - Place breakpoints on input_text/input_image blocks; markers anchored on non-eligible items (tool calls/outputs, assistant text) shift backwards to the nearest eligible block. The HTTP client additionally marks the developer/system message. - Gate the field on GPT-5.6+ (older models reject it). - WS client: markers serialize as part of the input items, so compute_delta forces a full resend when marker positions move; the cached token prefix itself stays valid. - Parse cache_write_tokens from usage and normalize OpenAI usage to Anthropic semantics: input_tokens now excludes cached and cache-write tokens (consumers like context-usage tracking already assumed this).
…ages The quantized scheme left the tail behind the last marker (up to ~5 messages, often including large tool results) billed at full input price on several consecutive requests. A cache write (1.25x) pays for itself as soon as the prefix is reused once, so caching as early as possible is strictly cheaper for any conversation that continues. New stateless placement: leading marker on the last stable message, trailing marker on the message directly before the last stable assistant message. Since every request ends on a user message and the response is appended right behind it, that position is exactly where the previous request placed its leading marker — the lookup is guaranteed to hit and only the newly appended messages are written, regardless of how many messages arrived in between (tool results, pending user messages). The OpenAI Responses clients keep the quantized scheme: implicit mode already caches up to the latest message on every request, and rarely-moving markers keep the WS client's incremental input intact (per-request movement would force a full resend every turn).
Bumps gpui-component to ab37c62a (markdown-source-copy-consume-v3): markdown-source-copy + the copy-beyond-viewport fix (select the full geometric band, not just visible glyphs) + the zed 1a246efd pin. Also adapts BlockView::markdown_view to the SelectionFormat API (.selectable_source() -> .selection_format()).
stippi
changed the title
Consume the merged 'Copy Markdown source' work (gpui-component PR #2628) from stippi/gpui-component. Pin gpui-component to the thin pin-zed-cc053a4a branch (upstream main + one commit pinning the zed git deps) and pin the zed crates to cc053a4a so the graph resolves to a single copy of gpui.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.