Quota check on write path times out for heavy accounts (node↔billing mutual-call storm)

[huntertinycloud]

Symptom

Users with large/multi-space accounts get SQL batch failed: upstream request timed out on ordinary writes (observed on prod via Listen's schema-ensure for a ~78.5MB space, 2026-07-09). Small accounts are unaffected. Because writes fail, data loss is silent for webhook-driven ingestion (e.g. Listen transcripts).

Trace

1. Write arrives → node consults QuotaCache.get_limit() (tinycloud-node-server/src/quota.rs): cache miss → synchronous HTTP GET {quota_url}/api/quota/{space_id} with a 10s timeout, inside the user's write request.
2. Billing sidecar (billing/src/index.ts, GET /api/quota/:space_id) computes effective limit = subscription budget − live usage of all other spaces the DID owns, fetching each space's usage by calling back to the node (GET /admin/quota/:space_id → store_size, a real storage scan).
3. N spaces × storage scans > timeout → the user's write dies with an upstream timeout before even reaching the honest 402.
4. admin.rs get_quota already carries a comment warning about this recursion ("mutual-call storm") — the hazard is half-known.
5. Aggravator: QuotaCache.overrides is an in-memory HashMap — every node restart is a cold start that re-triggers the storm, and admin overrides silently evaporate.

Root cause

The quota contract requires live cross-space usage aggregation at read time. That makes slowness structural.

Proposed fixes

• billing: precompute per-space effective limits into D1; recompute asynchronously on Stripe webhooks and periodic/pushed usage reports. /api/quota becomes a plain row read.
• node: cache store_size/quota answers with a TTL; move quota refresh off the write path (background refresh, stale-while-revalidate).
• policy: on quota-service timeout, fail open with last-known limit (or env default) instead of failing the write — a billing hiccup should never deny storage writes.
• persist admin quota overrides (or document that they are ephemeral).

Severity

High — hits the heaviest/most engaged accounts first, and the failure mode upstream of the 402 is silent write loss.

Found during the Listen UX/freshness audit (branch ux/library-navigation); quota UX on the Listen side is handled separately (402 'Storage full' state shipped; P8 usage-meter planned).

🤖 Generated with Claude Code

[huntertinycloud]

PRs up for both halves:

• node: Quota: never block writes on the quota service (stale-while-revalidate) (#104) #105 — quota lookups never block writes for known spaces (stale-while-revalidate, fail-open last-known → env default, 30s negative-cache backoff; first-sight fetch bounded at 3s). Admin override ephemerality documented.
• billing: TinyCloudLabs/billing#27 — cross-space aggregation precomputed into a space_quotas D1 table (migration 0005); /api/quota/:space_id becomes a row read with waitUntil revalidation + eager recompute on Stripe webhooks.

One correction to the issue trace found while implementing: store_size is an in-memory counter on all current backends (no storage scan), and get_limit already fell back to the env default on fetch failure — the prod timeout was most plausibly the synchronous 10s quota fetch (cold cache after restart × billing fan-out) pushing the write past the ingress timeout. Both PRs remove that wait from the hot path. Overrides persistence was taken as the 'document' option; durable limits belong to the billing sidecar.

[huntertinycloud]

Tracking in Linear: TC-154 (parent), with the review follow-ups as sub-issues TC-155–TC-158 (node: inflight dedup, remove_limit race, cache bounds, injectable clock) and TC-159–TC-164 (billing: revalidation dedup, webhook waitUntil, fail-open rows, F10 revenue bug, ul-cutover migration renumbering, CI scoping).

[huntertinycloud]

Fresh measurements from a live repro today changed part of this diagnosis — the quota path is currently fast (~1.2s from billing), but every SQL batch on the heavy space (reads included) still dies at the TEE gateway. Split out as #106 / TC-165. #105/billing#27 remain valid for the write-path quota class but won't alone unblock heavy accounts.