Skip to content

perf(metadata): page current catalog lookups - #584

Closed
forhappy wants to merge 4 commits into
codex/current-catalog-openfrom
codex/paged-current-catalog
Closed

forhappy wants to merge 4 commits into
codex/current-catalog-openfrom
codex/paged-current-catalog

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Current-state file lookup reconstructs the complete historical pointer catalog before answering a selected file query. This change captures authenticated checkpoint/run controls once and uses the existing paged catalog lookup for requested file bindings and their dependency closure.

Stacked on #582. The combined indexed-read/shared-staging work is in #578; this PR is the next bounded catalog-reader slice. No storage format, dependency, or configuration changes.

Changes

  • Replace the current-state eager file map with a captured PointerCatalogLookup; preserve batch order, duplicate requests, missing-file results, session closure, and fail-closed v2 authority.
  • Move captured visibility and ref/control ordering into crab-metadata, so Git and catalog readers share authentication and publication ordering rules. Remove the superseded read-side implementation and full catalog scan.
  • Assemble full inspection catalogs across the captured visible set before requiring dependency completeness. Independent refs can publish descriptors and bindings in an order different from canonical transaction sorting. Immutable conflicts still fail immediately; publication-time PointerCatalog::apply remains strict.
  • Add current-state 100/20,000-row request/byte bounds, captured-root isolation, and a real cross-ref publication regression. Preserve the existing lazy Git visibility opening test.

Validation

  • Metadata library: 458 passed, 0 failed; includes the new current-state and cross-ref regressions.
  • The cross-ref regression failed before the assembly fix: a selected lookup succeeded while full catalog reconstruction rejected a shard supplied by another visible ref.
  • cargo fmt --all --check and git diff --check: passed.
  • Read capsule-protocol r5: 47 passed, 0 failed (27.12 s). The earlier r4 run exposed an error-classification regression (46 passed, 1 failed). Restoring the read boundary makes the original conflicting-capsule assertion pass unchanged; transport errors retain their metadata sources. The negative receipt is retained.
  • Metadata payload-only compilation and strict all-target metadata Clippy (file-index-reader, -D warnings) passed. CLI/NFS library/binary compilation and the debug CLI build passed (minimal-profile warnings remain; not a strict CLI lint claim).
  • All three GC cleanup tests pass, including the new 20,000-file captured checkpoint regression: replace the checkpoint, sweep past grace, delete an actual orphan, then fetch cold old pages and preserve ordered duplicate answers. This is an in-memory metadata proof, not payload hydration or concurrent real-store proof.
  • The six-case force rewrite policy/visibility regression passes after awaiting its now-async test call; assertions are unchanged. Eleven Python fixture tests pass. The known staging-version fixture correction is carried forward from feat: integrate indexed catalogs with unified staging #578.
  • Full current-source CLI smoke remains failed: 152 commands / 61 checks stop at the shallow-since expectation. The helper parses the ISO date option as an integer timestamp, while the harness expects the older unsupported-option diagnostic. This needs a source adapter fix and native Git dated-boundary oracles; the check is not waived or rewritten to accept a generic failure. The earlier failed GC receipts (borrow/API compile errors and an invalid empty-source fixture) are also retained.
  • The new qualify_remote ... <catalog-file-hash> catalog-current operation builds and passes against the four-version, two-file 10 GiB RustFS dataset. Opening: 28 operations / 149,860 bytes. Cold and warm duplicate-file queries each use 497 ranges / 7,434,846 bytes; results match full inspection, with zero writes and drained operations. The repeated warm recipe reads are a measured remaining performance gap.
  • The first qualifier attempt remains a failed receipt: it supplied raw pointer-wire hex instead of Xet catalog MerkleHash::hex(). Correcting that external input made the unchanged binary pass. This is one local debug-profile functional sample, not cloud latency or independent history-scaling proof.
  • Broader runtime qualification, retained-root GC/compaction races, representative S3 latency, and the final combined 10 GiB matrix remain pending. Earlier component/combined staging receipts do not qualify later reader changes.

Evidence map and review question

Surface Evidence
Entry/caller FileIndexLookupSession::open_with_mode in crates/crab-metadata/src/file_index_lookup.rs
Ownership/callee load_pointer_catalog_lookup_from_root in capsule_protocol/catalog_capture.rs; existing authenticated checkpoint/run page readers
Shared invariant Git control views and current catalog capture use ref_view.rs and captured_visibility.rs; both full materializers use PointerCatalogAssembly
Tests file_index_lookup/catalog_tests.rs, capsule_protocol/store/tests.rs, pointer assembly tests, retained read visibility opening test
Before change The stacked base eagerly reconstructs the v2 catalog and copies all file bindings; main does not contain this complete capsule/indexed-read stack
Remaining proof Read/CLI checks and real runtime qualification listed above; no claim of complete initiative readiness

Is this the best fix? Sharing metadata capture avoids separate Git/catalog publication policies, and one opaque assembly boundary fixes cross-ref inspection without weakening strict publication validation. Review the new public storage API surface and the remaining lifecycle qualification before marking ready.

The net source growth supplies current-state capture plus cross-ref assembly; moved ordering/visibility replaces duplicate implementations. The two requested large-file-push-audit benchmark reports are excluded.

The lifecycle qualification and pending CLI-smoke failure are recorded in 8c7d20ea393. The separate bounded verified-binding optimization addresses the measured repeated warm recipe reads; its parent API and cold dependency proof remain the ones established here.

Share captured ref ordering and visibility with Git readers.

Validate full catalog dependencies after assembling the visible cross-ref set.
Preserve read-side corruption classification when shared metadata code detects conflicting captured controls. Keep transport errors as metadata sources.
Measure opening before cold and warm selected queries. Compare duplicate results with full inspection only after closing the session.
Prove cold catalog pages survive checkpoint replacement and a real orphan sweep. Repair the async visibility test caller and carry forward the qualified staging-version fixture.

Record the remaining shallow-date adapter failure without weakening its CLI smoke assertion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant