Skip to content

perf(metadata): retain verified catalog bindings - #586

Closed
forhappy wants to merge 1 commit into
codex/paged-current-catalogfrom
codex/verified-catalog-bindings
Closed

forhappy wants to merge 1 commit into
codex/paged-current-catalogfrom
codex/verified-catalog-bindings

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Repeated file-to-shard lookup for a 10 GiB file rebuilt its authenticated recipe after the generic page cache evicted it: both cold and warm batches issued 497 range reads and fetched 7,434,846 bytes. Retain compact verified bindings within the captured file-index session so a fully cached batch needs no origin reads.

Stacked on #584. No format, dependency, configuration or public API changes.

Behavior

  • A private catalog session retains at most 4,096 verified file-to-shard answers or authenticated misses. It does not retain full recipes.
  • The first lookup still validates the complete shard/xorb dependency closure. A batch containing any uncached binding follows the original complete batch proof and its aggregate byte accounting.
  • Requested-key admission runs before allocating cached answers. Fully cached duplicate batches cannot bypass the existing input-byte limit.
  • Preserve order, duplicate requests and absent-file results, including batches larger than cache capacity. Failed proofs populate no results; evicted bindings are revalidated against the same immutable capture.
  • Cache locks cover local bookkeeping only. Existing shared close/cancellation guards remain in their owner. The released manifest/SlateDB path and direct full-catalog consumers are unchanged.

Validation

  • 463 metadata library tests passed, including snapshot isolation, shared close/cancellation, error retry, mixed hit/miss cache bounds and 5,000-file batch coverage.
  • The wide-recipe regression failed before the cache with 308 repeated backend reads and passes after it.
  • A temporary removal of the admission guard makes the corrected regression fail by returning an oversized successful batch. Restoring the guard passes; rejection is compared exactly with the canonical uncached query rather than a guessed diagnostic. The faulty negative-control source is not committed.
  • Strict all-target metadata Clippy with file-index-reader and -D warnings passed. Payload-only compilation, storage-only compilation and the real qualifier build passed. Format and whitespace checks passed.
  • Real RustFS qualification against the four-version, two-file 10 GiB history passes with results matching complete inspection. Cold proof work remains 497 ranges / 7,434,846 bytes; warm proof work becomes zero backend operations / zero bytes. No writes, and all operations drain.
  • These are local debug-profile functional/work measurements, not cloud latency or independently increasing-history qualification. The final source artifact also passes a post-rebase run against that fixture; all shared-crate source trees match the tested commit. Binary SHA-256 91d437990f626aa39b3473588c0f3995ef246d445fc0726eaa79a849ff003837; result JSONL SHA-256 4a5999c73ab4c28e4cc7cfeef0e984b0569d276de0eaf6195144a1f8b2a25eb4.

Evidence map

Boundary Evidence
Entry/caller FileIndexLookupSession::lookup_batch in crates/crab-metadata/src/file_index_lookup.rs; current and captured catalog constructors share the same private session
Owner/callee catalog_session.rs owns compact verified results; PointerCatalogLookup::files retains complete authenticated closure and canonical ordering
Budget Shared key-count admission in capsule_protocol/pointer/paged/reader.rs; existing page/query limits remain unchanged
Siblings Hydration and mount use shared lookup sessions; direct publication/full-inspection APIs keep their full result behavior; manifest acceleration remains unchanged
Regression tests file_index_lookup/catalog_tests.rs and the bounded binding-cache property; existing file-index lifecycle tests pass
Before change The stacked base reconstructs the closure on every file-index query; the 256-page/8 MiB cache cannot retain the wide recipe. Current main lacks this complete capsule protocol stack
Remaining gates Full combined-source pattern/order/failure matrix, retained-root races and hosted-S3 latency. The parent's CLI smoke also exposed a shallow-date adapter failure requiring a separate source fix

Is this the best fix? Cache the small verified answer at its consumer boundary instead of increasing a generic page cache to fit one file size or weakening the initial dependency proof. About 110 net production lines own bounded results, preserve admission and replace the previous inline catalog conversion. Review the deliberately complete proof on partially uncached batches and the remaining combined qualification before readiness.

The two requested large-file audit reports remain excluded. No assertion baseline, architecture inventory or expected-failure list is relaxed.

Cache bounded file-to-shard answers after complete dependency proofs. Preserve requested-key admission and full-batch proof accounting on misses; never hold the cache lock across origin I/O.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant