fix: harden retrieval serving consistency and incomplete-index fallback

[suguanYang]

Summary

This PR hardens retrieval serving consistency across publication, scoring, and fallback paths.

Changes

• Advance both old and new namespace generations when a document moves namespaces, preventing stale process-local snapshot reuse.
• Make the public retrieval contract explicit: the persisted map-unit route supports its fixed path/content scorer; legacy channels and channel_weights controls are deprecated and rejected when supplied.
• Compute namespace-level average IDF from published token document-frequency statistics instead of combining per-revision averages. The synchronous map scorer uses the same exact namespace calculation when available.
• Add bounded PostgreSQL lexical fallback for classic retrieval when serving indexes are incomplete.
• Add legacy scorer fallback for map-nav when persisted indexes are unavailable, with explicit degraded-path logging.
• Publish short-lived Redis index-readiness status (ready, expected revisions, indexed revisions). Redis is status-only; PostgreSQL remains the source of truth and retrieval does not depend on Redis availability.
• Add rebuild telemetry for namespace statistics (generation, documents, units, token-stat count, duration).

Validation

• Retrieval contract suite: 26 passed.
• Ruff: passed.
• Pyright: 0 errors.

No production database writes or deployment operations were performed by this PR.

Operational notes

After deployment, monitor the retrieval map index incomplete and namespace-statistics rebuild logs. The readiness marker is advisory and should not replace the database/backfill readiness check.