Skip to content

fix(chat): reserve at the long-context step for huge prompts - #165

Merged
VickyXAI merged 1 commit into
mainfrom
fix/leftovers-2026-10
Oct 2, 2026
Merged

VickyXAI merged 1 commit into
mainfrom
fix/leftovers-2026-10

Conversation

@VickyXAI

@VickyXAI VickyXAI commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

estimateChatCost reserved at base rates for every prompt size. Past a model's pricing.long_context threshold, the gateway reprices the whole request, so a huge prompt under-reserved. Examples: gpt-6-astra over 272K reserved $10/$50 against a $20/$75 quote, and Grok from 200K reserved half the quote.

  • LONG_CONTEXT_PRICE_PER_MTOKEN copies the live steps from GET /v1/models (both chains agree; the Qwen steps are Base-only). That covers GPT-6, GPT-5.4/5.5/5.6 (>272K, 2x in / 1.5x out), Gemini 3.1/2.5 Pro (>200K, 2x/1.5x), Grok 4.3–4.7 and build-0.1 (>=200K, 2x/2x), and Qwen 3.7 plus/flash (inclusive; flash steps twice).
  • chatRateAt / tierWorstPriceAt pick the highest step reached, using the gateway's own inclusive/exclusive rule. They are pure in prompt tokens and never lower the base rate. They are used by estimateChatCost for both an explicit model and a mode, and by accountLedgerUsd.
  • Tests cover the exact boundaries: 272,000 stays base and 272,001 steps for the exclusive models; 199,999 stays base and 200,000 steps for inclusive Grok. They also cover the multi-step ladder, tier modes, bare ids, prototype keys and the ledger.
  • verify:prices now fails when a live step would settle above the reserve. A control run with the astra step removed and grok made exclusive flagged both models on both chains.

Refs BlockRunAI/blockrun#781

@VickyXAI
VickyXAI merged commit 1ce2f2e into main Oct 2, 2026
3 checks passed
@VickyXAI
VickyXAI deleted the fix/leftovers-2026-10 branch October 2, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant