fix(db): use global fact-type partial vector indexes

[Sanderhoff-alt]

Summary

This change replaces PostgreSQL per-bank vector indexes with three global fact-type partial vector indexes. The indexes cover all banks while keeping a separate ANN index for each fact type: world, experience, and observation.

The goal is to remove bank-count-dependent planning overhead without restoring the old unfiltered-global-index post-filter path that caused filtered ANN recall regressions.

Closes #3485

Background

The previous layout created three vector indexes for every bank. As the number of banks grew, PostgreSQL had to consider an increasingly large index catalog for parameterized recall queries. Planning time therefore grew with bank count, and end-to-end recall latency could degrade approximately linearly even when each bank contained relatively few memories.

The target layout is:

Layout	Index coverage	Predicates
Current	All banks	One global partial index per fact_type
Previous	One bank	One index per bank and fact_type
Removed fallback	All banks	One unfiltered global ANN index, then bank_id post-filter

Design

The migration creates these canonical indexes:

• idx_mu_emb_world
• idx_mu_emb_experience
• idx_mu_emb_observation

For pgvector they are HNSW indexes using the existing cosine operator class. The backend abstraction remains intact for pgvectorscale and vchord. ScaNN keeps its existing unfiltered global-index path because it does not use the PostgreSQL partial-index layout.

The migration removes legacy per-bank and stale unfiltered indexes, then uses concurrent DDL inside an Alembic autocommit block. A short lock timeout makes a busy deployment fail quickly and retry idempotently instead of waiting indefinitely. Bank creation, archive import, and bank deletion no longer issue vector-index DDL.

Backend Switches

A populated deployment that changes vector backend must use:

hindsight-admin vector-indexes --rebuild

The command checks the migration-owned layout and rebuilds only missing, invalid, or unexpected indexes. PostgreSQL ANN indexes use DROP INDEX CONCURRENTLY and CREATE INDEX CONCURRENTLY on a raw autocommit connection. Startup reconciliation rebuilds an empty table automatically and fails closed with the command above in the error message when data is present.

Downgrading removes the global indexes and recreates the prior per-bank layout for existing banks. Upgrading is idempotent and removes both named legacy indexes and catalog-discovered per-bank indexes.

Recall Correctness

Every recall arm retains its fact_type predicate. Semantic ANN arms over-fetch by 5x, with a minimum of 100 candidates, before trimming and reranking. The existing ef_search=200 setting remains in effect.

The global layout can still encounter the filtered-ANN under-return shape documented in #2645: a small bank may be underrepresented when bank_id is applied after scanning a large shared index. On pgvector 0.8+, connection setup detects pg_extension.extversion and enables:

hnsw.iterative_scan = relaxed_order
hnsw.max_scan_tuples = 20000

20000 is explicitly pinned to the pgvector upstream default. Older pgvector versions continue to use the existing over-fetch, fact-type predicate, and reranking safeguards. The scan budget is intentionally not increased until a multi-bank benchmark demonstrates that a higher value is necessary.

Backend Compatibility

Oracle is unchanged. Oracle 23ai uses its existing partitioned global vector index and does not support PostgreSQL-style partial vector indexes; the new migration is dispatched only for PostgreSQL.

Verification

The following areas are covered by the updated tests:

• migration upgrade, downgrade, and idempotency;
• empty-table reconciliation and populated-table fail-closed behavior;
• concurrent vector-index rebuilds and unexpected-index cleanup;
• link ANN SQL and lifecycle operations;
• pgvector iterative-scan version gating and process-level version caching;
• vector-index health and backend validation.

ruff, ruff format, and ty check hindsight_api pass. Focused migration, vector-index, health, and link-utils tests pass. The repository lint hook's ESLint phase requires the separately installed @eslint/js package and could not run in this worktree.

Validation on the repository pg0 instance confirmed pgvector 0.8.5. A real asyncpg pool connection initialized by the new setup path reported hnsw.iterative_scan = relaxed_order and hnsw.max_scan_tuples = 20000. This validates parameter wiring and the #2645 regression guard; it is not a substitute for a recall benchmark across bank sizes.

Baseline Validation

Before merge, the benchmark suites should be run on both origin/main and this branch under the same PostgreSQL/pgvector version, dataset, warm-up, and concurrency settings:

• ./scripts/benchmarks/run-longmemeval.sh
• ./scripts/benchmarks/run-locomo.sh
• ./scripts/benchmarks/run-perf-test.sh

Results should be compared between baseline and branch rather than evaluating this branch in isolation. The performance run should include multiple banks with strongly different sizes and report small-bank recall, planning time, execution time, and tail latency. LongMemEval and LoCoMo should be checked for end-to-end quality or latency regressions.

The default run-perf-test.sh --suite recall setup creates one bank, so a multi-bank fixture or equivalent data setup is required for the #2645 filtered-ANN scenario. The multi-bank run should also be used to decide whether the explicit 20000 scan budget should be increased.

[Sanderhoff-alt]

Closing this — it was superseded by #3561 (merged) and #3541.

Why

The two approaches this PR explored were the global fact-type index layout (#3535) and the per-bank-by-size layout that #3561 shipped. The evidence in #3485 settled it in favor of per-bank-by-size:

• The planner never picks the global index for small partitions — in both benchmarks it chooses idx_memory_units_bank_fact_type + a top-N sort. The measured 1,500ms → 23ms win came entirely from deleting 6,000 per-bank indexes, not from adding 3 global ones.
• A global index is then maintained on every write to serve a minority of queries, and served worst exactly when used: filtered ANN over a whole-corpus graph is weakest when the bank is a small fraction of the table (Per-(bank, fact_type) vector indexes never self-heal: banks populated outside the create-time path fall back to global HNSW + post-filter → slow and under-returning recall #2645 shape).

#3561 keeps the per-bank layout but earns indexes by size (HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS, default 0 = unchanged behavior), which makes index count proportional to the number of large banks instead of the bank count — same goal, no global-index downside.

#3541 (still open) complements it by enabling hnsw.iterative_scan so a recall budget actually reaches the index on the large partitions that do have one. The iterative-scan capability detection this PR added (pgvector extversion → relaxed_order + max_scan_tuples) is worth folding into #3541 if it merges, but as a separate layout change this PR has no remaining path to merge.

Closing to avoid two competing layouts in the tree. Happy to reopen if a multi-bank benchmark shows the global layout is needed.