Skip to content

perf(metadata): coalesce compacted visibility reads - #603

Closed
forhappy wants to merge 1 commit into
codex/indexed-staging-final-integrationfrom
codex/bounded-catalog-opening
Closed

forhappy wants to merge 1 commit into
codex/indexed-staging-final-integrationfrom
codex/bounded-catalog-opening

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opening a current catalog issued one header read and one change-page read per logical visibility edit, even after compaction put the history in one physical run. In the isolated canonical-session fixture, 2 / 8 / 32 edits took 8 / 20 / 68 provider read attempts. This change takes 5 / 5 / 5 while preserving reconciliation and the distinct-object proof limit.

Compacted runs copy their visibility sections into one contiguous proof-only region. The footer commits its span and digest; complete inspection also checks each copy against the original capsule. Logical controls share one cache for authenticated decoded pages and 64 KiB opaque proof blocks, bounded to 8 MiB / 256 entries per run. Page hashes remain mandatory, and cold/warm admission charges full blocks before I/O. Four asynchronous fetch lanes coalesce neighboring page misses while warm hits remain independent. Leaf runs retain their original coordinates. Existing expected-old, inherited closure, removal, ordering and catalog precedence rules are unchanged.

Stacked on #598. Run footer version 5 rejects retired unshipped versions 1–4. A fresh audit of six published tag objects and 19 known local tag objects found no current capsule format; the differing published/local v1.2.4 objects were both checked without replacing the local tag. Released v1 storage is unaffected. The corruption harness now targets version 5 and explicitly rejects version 4.

Validation on commit 9e5c081b351 (all guarded source bytes unchanged):

  • 601 metadata tests pass; the existing thousand-publication qualification stress test remains ignored in the default slice.
  • 47 shared-read and 22 shared-writer capsule tests pass, including visibility, compaction, restoration and publication races. These focused slices are not a workspace-wide proof.
  • 14 Python fault-range/source-fixture tests pass.
  • Strict metadata all-target Clippy, payload-only and storage-only builds, formatting and whitespace checks pass.
  • Official make install and both qualification binaries completed. The first CLI variant reports the base SHA because its build began before the commit; source fingerprints bind the compiled inputs to this change. Stable-head identity checks for all final release artifacts remain required.
  • Fresh RustFS qualification passed 408 commands / 34 checks: 128 publications, independent cold reconstruction and native Git / remote fsck. An independent audit matched the guarded source inputs to this commit. This is a 4 MiB component workload, not the final 10 GiB matrix.
  • At 32 / 64 / 128 publications, the selected cold catalog query stayed at five range reads / 3,764 bytes; warm queries used zero origin reads. Opening used 8 / 9 / 11 operations and 139,326 / 274,473 / 610,126 bytes. Historical opening bytes still grow; the bounded-opening acceptance gate is not satisfied.
  • Explicitly running the default-ignored 1,000-publication locator qualification failed: 47 active layers exceeded the unchanged bound of 40. This maintenance gate remains unresolved; diagnostic instrumentation is outside this PR.

Evidence map: current file-index opening enters shared catalog capture; Git and catalog consumers both load authenticated run controls through load_capsule_run_git_control. CapsuleRun owns format/compaction, CapsuleVisibilityReader owns rooted proof lookup, and the existing metadata page cache owns bounded read coalescing. Full maintenance readers still verify original capsule sections and the copied pool. Captured-visibility tests exercise closure membership separately from physical presence. The source-bound before-change regression establishes the transport problem on #598’s head; it is not a measured current-main latency claim.

Is this the best fix? Co-location addresses the measured request amplification at the existing compaction and page-cache boundaries without skipping proof checks or reading across Git payloads. It adds one copy of compacted visibility metadata; Git payloads are not duplicated by this change. The additional implementation owns the wire region, shared eviction/admission and concurrent block reads; tests cover re-signed routing, substituted evidence, prefetched-page tampering and resource bounds.

This is a partial performance step, not the completed opening redesign: logical control replay, differing addition-set enumeration and increasing historical bytes/runs remain required work. Hosted S3 latency, broader lifecycle gates, fuller Git blame and the final six-case 10 GiB matrix also remain open. The #598 matrix’s first fresh push failed a readback digest check; a later diagnostic retry succeeded with 161 exact wire-body comparisons. That retry does not turn the failed matrix into a pass.

The user-owned advisor plan and both excluded large-file audit reports are absent from this diff. No inventory, baseline, assertion or lint policy is relaxed.

Compacted capsules separate proof pages with Git payloads. Copy proof-only sections into one footer-bound region and share a bounded page/block cache to remove per-edit network round trips.

Keep visibility reconciliation and dictionary limits intact. Advance the unshipped run footer to version 5; audited published release objects do not contain this capsule format. Logical replay and historical byte growth remain separate required work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant