Skip to content

Integrate large-file push and indexed-read fixes - #609

Closed
forhappy wants to merge 6 commits into
codex/indexed-staging-final-integrationfrom
codex/indexed-staging-qualification-combined
Closed

forhappy wants to merge 6 commits into
codex/indexed-staging-final-integrationfrom
codex/indexed-staging-qualification-combined

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Combine the fixes for repeated large-file edits and catalog opening into one source revision for qualification. Residual push preparation decodes bounded ranges, reuses complete prepared xorbs, repacks only unresolved chunks, and moves compression/finalization off the async executor. Immutable publication retries corrupt readbacks while preserving verified provider identity. Catalog opening coalesces visibility-proof ranges; locator maintenance limits layer pressure.

Stacked on #598. Component review units: #602, #603, #604, #605 and #608. This PR combines those changes; nothing has been merged.

Validation on exact head 4093308d542c9f7a606daa65f237346bd38cfb9a:

  • Metadata: 601 passed, 1 ignored; storage: 223 passed, 1 ignored.
  • Focused CLI publication: 19 passed; staging: 4 passed; shared read/write capsule slices: 47 and 22 passed; Python source/fault fixtures: 14 passed. These are scoped slices, not a full workspace result.
  • Explicit 1,000-publication locator stress passed the unchanged resource bound, with original and latest checkpoint lookups preserved.
  • Strict metadata/storage/staging Clippy, minimal metadata and storage-only feature checks, formatting, and whitespace checks passed.
  • Official make install completed for all five artifacts; release qualify_remote and qualify_browse examples built. Source and binary fingerprints are recorded outside the repository.
  • Real RustFS smoke passed: two 64 MiB entropy files, four versions alternating native/Git ingestion, 100 code files, 75 commands and 57 checks. Source/cache removal, eight cold historical SHA-256 reconstructions, native Git fsck and full origin fsck passed.
  • Current origin/main (cc2e4e25f46) is an ancestor; the qualification checkout is clean.

Remaining gates before merge:

  • The six-case 10 GiB matrix is running on this exact head: entropy/repeated/zero data, native/Git-first ingestion, two files and four versions per case. The first entropy/native case also qualifies captured readers through compaction, actual aged GC, and cold reconstruction. No full 10 GiB acceptance claim yet.
  • API/protected HTTP fault proof must be repeated on this combined head; previous component/integration results are not counted as this head's evidence.
  • Opening still reconciles historical controls. Coalesced requests do not establish bounded historical bytes or eliminate replay. Incremental catalog checkpoints, distinct-history/ref scaling and representative hosted-S3 latency remain open.
  • Crash/restart, concurrent publication/pruning, stale-cache and migration qualification remain required. Fuller merge/rename/source-line blame remains a separate stage.
  • Existing source-policy inventory and broader CLI lint issues remain open; no inventories, baselines, assertions or lint policies were relaxed.

Evidence map: native add/Git clean feed staging; push preparation consumes the bounded prepared-source API and verified storage publication; selected catalog capture consumes shared authenticated pages; locator writers own layer compaction. Hydration, fsck, GC and shared/protected publication retain the existing ownership and durability boundaries from #598. Component PRs document their callers, callees, sibling paths and before-change behavior.

Is this the best fix? The component changes address measured decoding, executor blocking, readback verification and request amplification at their existing owners. This integration PR exists to qualify their interaction. It deliberately leaves the unresolved full-history replay gate visible.

Excluded: crab/docs/benchmarks/large-file-push-audit-summary.json, crab/docs/benchmarks/large-file-push-audit.md, and the user's advisor-plan edits. The uncommitted checkpoint-format/catalog-object work in the other checkout is not included.

Share bounded hash verification and its existing corruption retry with acknowledged immutable writes. Retry only the GET and retain both identity fields from the response whose bytes passed verification.

Persistent corruption still fails after two reads; exact and logical writes issue one PUT. Keep checksum-qualified and occupied-key contracts unchanged.
Compacted capsules separate proof pages with Git payloads. Copy proof-only sections into one footer-bound region and share a bounded page/block cache to remove per-edit network round trips.

Keep visibility reconciliation and dictionary limits intact. Advance the unshipped run footer to version 5; audited published release objects do not contain this capsule format. Logical replay and historical byte growth remain separate required work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant