Skip to content

perf(push): repack only unresolved prepared chunks - #605

Closed
forhappy wants to merge 2 commits into
codex/bounded-prepared-residual-readsfrom
codex/incremental-xorb-publication
Closed

forhappy wants to merge 2 commits into
codex/bounded-prepared-residual-readsfrom
codex/incremental-xorb-publication

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

A prepared xorb containing mostly committed chunks could be adopted and uploaded whole for a tiny edit. Adopt only wholly selected, unresolved prepared xorbs; repack mixed sources using missing chunks while retaining committed placements. Gather residual requests across files before reading so each shared source is verified once. Run residual compression and finalization on blocking workers so lease-renewal and upload tasks remain schedulable.

Depends on #604. Ordered recipe occurrences still build complete file terms. Source verification, remote authority revalidation, snapshot lifetime and flush-before-pointer publication remain at their existing owners.

Before-change evidence reproduces full prepared-body upload and repeated source reads. The regression uses a real 32 MiB staged prepared body with 516 chunks, tests both 515 and one committed chunks, verifies the reconstructed file recipe, and requires the uploaded chunk set to equal exactly the unresolved set. Both cases open the prepared source once; observed upload bytes are 21,434 and 33,454,308 respectively.

Validation: 19 focused CLI tests pass, including catalog corruption/size conflicts, existing encoding adoption, protected publication, cross-repository cold add, prepared concurrency, bounded packing/backpressure and upload error draining. Four staging cancellation/corruption/batching tests, strict staging all-target Clippy and formatting pass. The original r6 runner failed after its 19 product tests passed because diagnostics were captured; r7 enables --nocapture and preserves every assertion. Exact-source fresh 10 GiB and broader lifecycle qualification remain open; these focused checks do not establish the full performance goal.

Fresh native-CLI RustFS qualification on dependent combined commit 4a9b8059d4e passed 75 commands / 57 checks: two 64 MiB entropy files, four versions through alternating native/Git staging, no second full local payload, retired published staging owners, eight independent cold byte oracles and full remote integrity. New xorb bytes per version were 67,156,162 / 2,216,454 / 2,284,332 / 2,354,458. Source and CLI hashes stayed unchanged. This is component E2E proof, not the final 10 GiB matrix.

Evidence map: capsule push calls prepare_delta; verified catalog/global placements determine missing chunks; shared CoalescedChunkReadPlan supplies bounded source reads; XorbBuilder packs residuals off the async worker; immutable publication provides xorb receipts before shard/catalog visibility. Native packing uses the same bounded staging reader. Existing complete prepared-body adoption remains for wholly new outgoing payloads. Current main has no capsule Xet publication module; #598 permits mixed prepared adoption, and its source-bound failing probes are retained.

Is this the best fix? Selection belongs at publication, where current committed placements are known. Repacking only unresolved chunks preserves deduplication without changing add-time ownership or creating per-file payload copies. Compression moves to the existing Tokio blocking pool, one bounded batch at a time. Publication code growth is limited to selection and offloading; the regression protects full recipes and single-source reads.

Draft only. The advisor plan and both excluded large-file audit reports are absent. No inventory, baseline, ignore, or product assertion is relaxed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant