Follow-on from #104, with fresh measurements (2026-07-10, account 0x6Ac0…FD6, space …:applications, 78.5MB) that change the diagnosis: the quota path is currently healthy, and the live blocker is the node's SQL batch latency vs the TEE gateway timeout.
Measurements (local Listen backend → prod tee.node, scripted harness)
| Probe |
Result |
billing GET /api/quota/<space> (×2) |
200 in 1.35s / 1.14s — storage_limit_bytes: 1028398468, usage 78.5MB, total 123.9MB |
node /healthz (×3) |
200 in 0.6–0.9s |
Listen POST /api/schema/ensure (idempotent DDL batch, ×3) |
fails in 127.9s / 127.6s / 129.5s — SQL batch failed: upstream request timed out |
Listen GET /api/conversations (read path, gated on ensureSchema) |
fails in 42.6s — same error |
The string upstream request timed out appears in neither tinycloud-node nor listen source — it's the dstack ingress/gateway in front of the TEE killing the response. The node is still grinding when the gateway gives up.
Why this is not (just) #104
The #104 trace blamed the node↔billing quota storm, but billing now answers in ~1.2s (post billing#26 / #91 timeouts, warm caches). Even with quota resolved instantly, every SQL batch against this space — reads included — dies at the gateway. #105 + billing#27 remain correct and worth merging (they remove the write-path quota wait class of failure), but they will not unblock heavy accounts on their own.
Hypothesis
The SQL service's per-batch artifact round-trip (load ~78MB SQLite/DuckDB artifact → execute → save + metering, post-#89) exceeds the gateway's upstream budget on heavy spaces. The 42s vs ~128s variance suggests more than one timeout tier in the path.
Directions
- node: cache/pin hot SQL artifacts in the SQL service instead of per-batch load/save; or stream saves; or size-tiered handling
- infra: measure the actual dstack-ingress timeout tiers; consider a longer budget for
/invoke SQL
- listen: stop hard-gating list/read routes on
ensureSchema (an idempotent DDL batch pays the full artifact round-trip on every request; relates to the TC-146 error-model work) — schema existence could be checked once and cached
- data shape: transcripts stored inline in SQL (
conversation.transcript_json) is what makes these DBs heavy — worth revisiting the storage contract
Repro harness + timings from the Listen debugging session on 2026-07-10; happy to share the scripts.
🤖 Generated with Claude Code
Follow-on from #104, with fresh measurements (2026-07-10, account
0x6Ac0…FD6, space…:applications, 78.5MB) that change the diagnosis: the quota path is currently healthy, and the live blocker is the node's SQL batch latency vs the TEE gateway timeout.Measurements (local Listen backend → prod tee.node, scripted harness)
GET /api/quota/<space>(×2)storage_limit_bytes: 1028398468, usage 78.5MB, total 123.9MB/healthz(×3)POST /api/schema/ensure(idempotent DDL batch, ×3)SQL batch failed: upstream request timed outGET /api/conversations(read path, gated on ensureSchema)The string
upstream request timed outappears in neither tinycloud-node nor listen source — it's the dstack ingress/gateway in front of the TEE killing the response. The node is still grinding when the gateway gives up.Why this is not (just) #104
The #104 trace blamed the node↔billing quota storm, but billing now answers in ~1.2s (post billing#26 / #91 timeouts, warm caches). Even with quota resolved instantly, every SQL batch against this space — reads included — dies at the gateway. #105 + billing#27 remain correct and worth merging (they remove the write-path quota wait class of failure), but they will not unblock heavy accounts on their own.
Hypothesis
The SQL service's per-batch artifact round-trip (load ~78MB SQLite/DuckDB artifact → execute → save + metering, post-#89) exceeds the gateway's upstream budget on heavy spaces. The 42s vs ~128s variance suggests more than one timeout tier in the path.
Directions
/invokeSQLensureSchema(an idempotent DDL batch pays the full artifact round-trip on every request; relates to the TC-146 error-model work) — schema existence could be checked once and cachedconversation.transcript_json) is what makes these DBs heavy — worth revisiting the storage contractRepro harness + timings from the Listen debugging session on 2026-07-10; happy to share the scripts.
🤖 Generated with Claude Code