Skip to content

perf: integrate paged current-state catalog lookup - #587

Closed
forhappy wants to merge 6 commits into
codex/catalog-staging-integrationfrom
codex/catalog-current-integration
Closed

forhappy wants to merge 6 commits into
codex/catalog-staging-integrationfrom
codex/catalog-current-integration

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Current-state file lookup previously materialized the historical Xet catalog. This integrates authenticated paged catalog capture with the unified-staging/publication stack and retains verified file-to-shard results when a large recipe exceeds the decoded-page cache.

Draft, stacked on #585; shared current-catalog commits originate in #582/#584. Review this incremental diff against codex/catalog-staging-integration. No merge requested.

Behavior

  • Share ref-head capture, activation validation and captured visibility between metadata and read orchestration; preserve corruption errors and authority-epoch filtering.
  • Open current-state catalog sessions from authenticated checkpoint/run descriptors and fetch selected file dependency closures lazily. Full materialization remains an explicit maintenance operation.
  • Reuse the session's existing cache for at most 4,096 verified file bindings or misses. Release its mutex before catalog I/O; cache results only after complete query validation succeeds. Preserve aggregate input admission, duplicate ordering, snapshot identity and exclusive close behavior.
  • Add qualify_remote catalog-current, separating opening from cold/warm duplicate lookups and comparing against the full maintenance oracle after session close.

Validation

  • Current head: focused metadata file-index and capsule protocol tests 169 passed, including the duplicate-boundary and distinct-miss memory cap regressions.
  • Strict metadata all-target Clippy with file-index-reader: passed.
  • Format and diff checks: passed.
  • Parent 6a8a8e75b8a: payload-only metadata tests 274 passed (payload-only sources unchanged by the follow-up).
  • Parent 6a8a8e75b8a: read capsule protocol tests 47 passed. The follow-up changes only binding-cache insertion and its tests; the current-head live qualifier recompiles and exercises the consumer path.
  • Publication-enabled remote qualifier compile check: passed.
  • Duplicate-batch boundary: 4,097 occurrences of one large-recipe file originally bypassed binding retention (487 extra warm reads). The corrected cache counts distinct identities; the same regression and an 8,192-distinct-miss memory-bound test pass. Every duplicate still consumes input admission.
  • Before the cache fix, the new 10 GiB-recipe metadata regression performed 487 cold reads and another 487 warm reads and failed its assertion. With the fix, the same regression passes with no additional warm reads. This fixture contains recipe metadata, not 10 GiB payload bytes.
  • Current head 6fab757a397 real-store check passed: release qualifier rebuilt from unchanged tracked sources, then queried an existing four-version/two-file 10 GiB RustFS history. Opening: 28 operations / 149,860 bytes. Cold selected recipe: 506 ranges / 7,537,592 bytes. Warm duplicate batch: 0 operations / 0 bytes. Both batches match the full maintenance oracle after close; all measured phases perform no writes and drain backend operations. This is current-head recipe lookup proof, not the final push matrix, history-growth or representative S3 latency proof.

Review evidence and boundaries

Surface Evidence
Entry point / owner FileIndexLookupSession::open_for_storage -> metadata-owned current catalog capture
Caller / callee Shared file lookup / hydrator and term resolver -> PointerCatalogLookup authenticated dependency reads
Sibling surfaces Read capture uses the shared ref/activation authority; explicit maintenance materialization remains available; v1 shard lookup remains separate
Tests Ordering, misses, root movement, missing dependency repair, large recipes, input budgets, concurrent reads and close/abandonment
Current main File lookup uses the manifest/shard path; this stack adds the capsule current-state paged path

Is this the best fix? Reusing the existing session cache keeps one immutable-capture authority and avoids increasing the decoded-page cache merely to fit a 10 GiB recipe. Cold queries still verify their full selected dependency closure; the cache does not resolve GC retention or cold lookup amplification.

Open release gates: cold captured browse-index unit regressions are fixed in sibling #591, with real provider-aged GC qualification still pending. The final combined six-case 10 GiB multi-edit/push matrix and representative S3 latency/scaling qualification remain open.

The first frozen earlier-source entropy/native run passed runtime checks but the independent acceptance wrapper rejected generated Python bytecode as a dirty source tree. That result remains rejected. The fresh R2 matrix uses bytecode-disabled Python, unchanged assertions and fresh buckets. On clean frozen component a6eeca4319d, entropy/native, entropy/Git and repeated/native are accepted (75 commands and 57 checks each, including eight cold-file SHA checks and remote fsck). Repeated/Git is running; zero/native and zero/Git remain pending. This proves those older component cases only; it is not current #591 or final combined-stack qualification, and it does not solve incremental local staging write amplification. Do not treat this draft as completion of the broader eight-stage initiative.

The latest commit also has a completed successful official make install build from a clean pinned checkout, using a separate Cargo target and hashes for all 2,630 tracked input files; no inputs changed during the build. It does not replace the frozen earlier-source matrix proof.

The two excluded large-file audit reports and the user-owned advisor plan are not included.

Share captured ref ordering and visibility with Git readers.

Validate full catalog dependencies after assembling the visible cross-ref set.
Preserve read-side corruption classification when shared metadata code detects conflicting captured controls. Keep transport errors as metadata sources.
Measure opening before cold and warm selected queries. Compare duplicate results with full inspection only after closing the session.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant