perf(retrieval): optimize map-nav snapshot and scoring

[suguanYang]

Summary

• Replace process-local decoded namespace snapshot cache with Redis compressed-blob caching keyed by snapshot format and namespace generation.
• Add routing-only namespace snapshot v2 while preserving the v1 reader.
• Replace descendant-leaf rescans with linear post-order map score aggregation while preserving score semantics.
• Add snapshot stage and process CPU/RSS instrumentation.

Validation

• Retrieval contract subset: 29 passed.
• Full API contract suite: 229 passed before the final generation-validation hardening; the final hardening was re-run against the retrieval subset.
• Random-tree map-score parity: 100/100 exact matches against the previous implementation.
• Synthetic 60k-chunk snapshot: compressed size 656,050 -> 419,178 bytes (~36% smaller); decode 0.729s -> 0.163s (~4.5x faster).
• Synthetic 50k-section map pooling: 0.172s -> 0.147s (~17% faster).
• Local Redis compressed-blob read: mean 7.6ms, p95 16.6ms.

Expected production impact

The 150s+ retrieval time is dominated by LLM/planner/harvest/control and database/network waits, so this change should not be expected to turn 150s into a short request. The expected gain is in the non-LLM portion:

• First request / Redis cold miss: primarily from the smaller routing snapshot and linear map pooling; likely seconds-level or sub-second savings depending on the current snapshot transfer/decode share.
• Warm Redis hit: avoids transferring the large PostgreSQL snapshot blob between API and database; expected to save the snapshot payload-fetch portion, typically tens to hundreds of milliseconds in-VPC, plus decode savings.
• Map scoring: expected to remove descendant-rescan growth; largest benefit is on deep/large section trees.
• Overall 150s request: conservatively expect low single-digit percentage improvement unless snapshot/map scoring currently accounts for a substantial fraction of non-LLM latency. LLM time is unchanged.

A staging-VPC benchmark with 10+ cold/warm/fallback runs is still required before making a production p95 claim or changing CPU allocation.

[suguanYang]

Additional optimization validation

Added and pushed:

• 6ed286d3 — scope map-unit frequency lookup to the current episode before token frequency filtering.
• a3ad93a5 — select only JobResult.job_id during connected hydration, preventing implicit loading of legacy job_chunks.

Validation:

• Retrieval/map-unit contract tests: 9 passed.
• Ruff and git diff --check: passed.
• Local cold connected hydration: ~266 ms → ~19 ms.
• Local warm connected hydration median: ~123 ms → ~6.4 ms.
• Production read-only EXPLAIN for the frequency query: ~80.5 s → ~26.4 s with identical result rows.

These changes do not modify prompts, LLM behavior, ranking semantics, public API, schema, migration, or backfill requirements. Staging cold-run validation remains the deployment gate.