Busy-space epoch history makes every authenticated op take 60s+ — Listen backend (and all source syncs) down post-v1.4.8

[huntertinycloud]

Summary

After v1.4.8 (which fixed the SQL export throttle, #112/#113 — confirmed on prod), a second, distinct slow class remains: every operation the Listen backend makes against the node hangs ~60s+, so delegation activation (12 /delegate calls inside a 30s timeout) can never complete and every Listen source sync (Fireflies, Soundcore, Google Meet, Granola) is down. Feed's fresh-space bootstrap latency (~2.5 min vs prod, ~45s locally) looks like the same underlying cost.

Measurements (2026-07-15 02:30–02:45 UTC, prod)

• GET api.listen.tinycloud.xyz/api/delegations/status — a single kv.get of a <1KB row in the backend's own KV space — took 60.5s ±0.05 across 4 consecutive probes (one 200, three cut at the 60s ingress timeout). The consistency rules out network jitter.
• Backend space (did:pkh:…:0x8c48…9056f6aEF5b2) storage: ~1.1MB (quota API) — not data volume.
• Same op classes from outside against the same tee.node endpoint (tc CLI, quiet space): 2–3s.
• Fresh backend process (CVM redeployed 02:22) — not stale in-process state. Fresh node CVM (v1.4.8 deploy 02:0x) — not stale node state.
• User-space SQL and KV (browser session): sub-second. Only the busiest identity is pathological.

Hypothesis

Per-op epoch bookkeeping cost scales with a space's epoch history: every authenticated op reads the epoch tip via the LEFT JOIN epoch_order … WHERE child IS NULL anti-join and appends a child, under chain-scoped guards. The Listen backend's space is by far the busiest on the node (every gated request from every user + sync polling + ~28h of retry storm during the #112 outage ≈ 10^5–10^6 epoch rows). If the tip anti-join lacks a covering index (or the planner seq-scans at that cardinality), each op costs tens of seconds on that space alone — matching every observation, while quiet spaces stay fast.

Suggested 10-minute check (prod Postgres)

SELECT space, count(*) FROM epoch GROUP BY space ORDER BY 2 DESC LIMIT 5;
EXPLAIN ANALYZE SELECT epoch.id FROM epoch LEFT JOIN epoch_order ON epoch.id = epoch_order.parent
  WHERE epoch.space = '<busiest space>' AND epoch_order.child IS NULL;
SELECT indexname, indexdef FROM pg_indexes WHERE tablename IN ('epoch','epoch_order');

Candidate fixes

• Covering/composite indexes for the tip lookup (epoch_order(parent), epoch(space, id)), if missing.
• Maintain a per-space tip pointer (single-row update) instead of the anti-join per op.
• Epoch pruning/compaction for high-traffic spaces.

Impact

Listen transcripts read fine (browser-direct), but all backend-mediated functionality — source syncs, secret reads, schema seeding via backend — is down for all users. Related: #112 (fixed SQL half), #106 (July 10 report of the SQL half), TC-212 (the SERIALIZABLE conflicts were plausibly this same epoch-tip contention under load).

[huntertinycloud]

Mystery solved — the 60s wall was a dead pinned host, not epoch cost

Server-side instrumentation on the Listen backend (temporary node-probe endpoint running the real store.load → deserialize → activate sequence) caught the actual failure:

Failed to activate delegated resource kv:vault/secrets/GRANOLA_API_KEY:
Failed to activate delegated session: … 525: SSL handshake failed
host: 350d124c81d53a57e8899e9f85a904918e663b61-8000.dstack-pha-prod5.phala.network

Browser-minted delegations can carry the node URL current at grant time in their host field — including the ephemeral per-CVM dstack app URL. useDelegation prefers delegation.host over the configured host, so activation targets whatever instance existed at grant time. Every node CVM replacement (three tonight: v1.4.7, v1.4.8) turns those pinned hosts into dead origins: while the old CVM drained, connections hung (the 60.5s walls measured earlier); once it was gone, Cloudflare 525s fast. Current probe timings for everything else: store.load 2.6s, node round-trips 1.3–2.4s — healthy.

Fixed on the Listen side by stripping the pinned host before activation (backend activates against its configured TINYCLOUD_HOST) — deploying now. SDK-level follow-up recommended: delegations should pin the stable domain (or nothing), never the per-instance dstack URL; anything else breaks all stored delegations on every redeploy, for every app.

What remains real in this issue

The epoch-tip scaling problem, now measured locally: the tip lookup does two full parallel seq scans + hash anti-join (no index serves the space filter — epoch PK leads with id), 540ms at 300k rows, O(total table size) per authenticated op, table never pruned. That's the ~1–2s/op prod tax everyone pays and it grows monotonically. Suggest retitling this issue to the epoch-scan fix (index or per-space tip pointer) — happy to PR the migration.