Skip to content

perf(read): page pinned hydration catalog lookups - #581

Closed
forhappy wants to merge 2 commits into
codex/indexed-staging-integrationfrom
codex/selected-catalog-consumers
Closed

forhappy wants to merge 2 commits into
codex/indexed-staging-integrationfrom
codex/selected-catalog-consumers

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Remote snapshot hydration and mount setup materialized every file binding in the captured Xet catalog before requesting a file. Pass captured authenticated lookup roots through SharedFileIndexLookup instead. Point/batch queries fetch the selected file, shard and xorb closure with the existing page cache and query budget, preserving file precedence and descriptor conflicts.

Mount setup also switches to the existing paged Git-view capture; retaining its previous complete-checkpoint opener would leave an eager read before lookup. Shared-reader initialization, active-read guards and exclusive close ownership remain unchanged. Missing pages are errors and never reopen mutable metadata.

Evidence map

  • Entry points: crab/src/read/mod.rs::remote_snapshot_git_dir and crab/src/cmd/mount.rs::resolve_mount_read_context_from_remote_url.
  • Owner: metadata SharedFileIndexLookup retains the captured PointerCatalogLookup; lookup_batch calls its authenticated selected-closure reader. Hydration continues through the canonical ShardHydrator and whole-file hash checks.
  • Siblings: current-state FileIndexLookupSession::open_with_mode remains eager; shard-index sessions retain their existing behavior. Shared receive conversion is tracked separately in perf(remote): select publication catalog dependencies #579. Complete maintenance inspection remains intentional.
  • Tests: 100/20,000-row catalogs, ordered duplicates and misses, cold/warm page bounds, unchanged captures, missing pages despite a later catalog, rejection after close; real shard/xorb hydration with the pointer hint removed. Existing shared/snapshot lookup tests retain close and concurrent-read coverage.
  • Retention contract: checkpoint publication records authenticated history before the root transition; GC marks retained checkpoint/run sources. Full compaction/GC capture qualification and explicit history-prune semantics remain open gates.
  • Main comparison: stacked on feat: integrate indexed catalogs with unified staging #578; not a main-level completion verdict. No claim that repository opening, current-state hydration, complete blame or the eight-stage initiative is finished.

Is this the best fix? Reusing metadata's captured page reader and shared session owner removes eager pinned catalog maps without a second selection or reconstruction policy. The remaining current-state opener needs a shared capture-owner extraction rather than another independent ref-ordering implementation.

Validation

Passed: all 29 metadata file-lookup regression tests, including the new page-bound and corruption cases. Passed: pinned reconstruction through the paged catalog with real shard/xorb descriptors and no pointer shard hint. The first fixture compile failed with E0283; the explicit MerkleHash conversion fixes type inference without changing assertions. Passed on 43203ff48f6: CLI library/binary check with simd-accel,gix-pathmatch,nfs and no default features (7m13s; 12 minimal-profile warnings, not a strict-lint pass). Pending: source-bound runtime and retained-capture lifecycle proof. Draft only until evidence is complete. No benchmark audit reports or architecture/expected-failure baselines are modified.

Keep captured catalog roots in the shared file lookup instead of copying all file bindings. Mount capture uses the paged Git view so checkpoint bodies stay cold. Preserve close ownership and authenticate the selected dependency closure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant