Skip to content

fix(gateway): keep hard budgets conservative when stream usage is incomplete - #255

Open
islahuddinn wants to merge 1 commit into
theam:mainfrom
islahuddinn:fix/gateway-incomplete-stream-budget-248
Open

fix(gateway): keep hard budgets conservative when stream usage is incomplete#255
islahuddinn wants to merge 1 commit into
theam:mainfrom
islahuddinn:fix/gateway-incomplete-stream-budget-248

Conversation

@islahuddinn

Copy link
Copy Markdown

Summary

When an Anthropic (or other streaming) provider connection drops after partial usage telemetry arrives, the gateway was reconciling hard-budget reservations down to that partial measured cost. That releases budget headroom the provider may still charge against.

This change tracks usage completeness separately from observed usage:

  • UsageTee.usageComplete is set only on terminal provider usage frames
  • billableCostCents() keeps hard-budget reconciliation conservative when a provider-charged error ends with incomplete usage
  • Observed token counts are still stored on the llm_requests row; only budget billing uses the preflight estimate when final usage is unknown

Fixes #248

Test plan

  • pnpm --filter @facility/gateway test (33 passed)
  • Unit tests for billableCostCents and usageComplete
  • Integration test: incomplete client abort keeps full reservation; complete abort bills measured cost

Made with Cursor

@islahuddinn
islahuddinn force-pushed the fix/gateway-incomplete-stream-budget-248 branch 2 times, most recently from 065a897 to f90f7ac Compare September 2, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway can undercount interrupted streams when partial usage is nonzero

2 participants