Skip to content

fix(push): upload only unresolved prepared chunks - #607

Closed
forhappy wants to merge 2 commits into
codex/bounded-residual-stagingfrom
codex/incremental-xorb-publication
Closed

forhappy wants to merge 2 commits into
codex/bounded-residual-stagingfrom
codex/incremental-xorb-publication

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

A prepared xorb containing mostly committed chunks could be adopted and uploaded whole for a tiny edit. Adopt only wholly selected, unresolved prepared xorbs; repack mixed sources using missing chunks while retaining committed placements. Gather residual requests across files before reading so each shared source is verified once. Run residual compression and finalization on blocking workers so lease-renewal and upload tasks remain schedulable.

Depends on the bounded residual staging PR. Ordered recipe occurrences still build complete file terms. Source verification, remote authority revalidation, snapshot lifetime and flush-before-pointer publication remain at their existing owners.

Before-change evidence reproduces full prepared-body upload and repeated source reads. The regression uses a real 32 MiB staged prepared body with 516 chunks, tests both 515 and one committed chunks, verifies the reconstructed file recipe, and requires the uploaded chunk set to equal exactly the unresolved set. Both cases open the prepared source once; observed upload bytes are 21,434 and 33,454,308 respectively.

Validation: 19 focused CLI tests pass, including catalog corruption/size conflicts, existing encoding adoption, protected publication, cross-repository cold add, prepared concurrency, bounded packing/backpressure and upload error draining. Four staging cancellation/corruption/batching tests, strict staging all-target Clippy and formatting pass. The original r6 runner failed after its 19 product tests passed because diagnostics were captured; r7 enables --nocapture and preserves every assertion. Exact-source fresh 10 GiB and broader lifecycle qualification remain open; these focused checks do not establish the full performance goal.

Evidence map: capsule push calls prepare_delta; verified catalog/global placements determine missing chunks; shared CoalescedChunkReadPlan supplies bounded source reads; XorbBuilder packs residuals off the async worker; immutable publication provides xorb receipts before shard/catalog visibility. Native packing uses the same bounded staging reader. Existing complete prepared-body adoption remains for wholly new outgoing payloads. Current main has no capsule Xet publication module; #598 permits mixed prepared adoption, and its source-bound failing probes are retained.

Is this the best fix? Selection belongs at publication, where current committed placements are known. Repacking only unresolved chunks preserves deduplication without changing add-time ownership or creating per-file payload copies. Compression moves to the existing Tokio blocking pool, one bounded batch at a time. Publication code growth is limited to selection and offloading; the regression protects full recipes and single-source reads.

Draft only. The advisor plan and both excluded large-file audit reports are absent. No inventory, baseline, ignore, or product assertion is relaxed.

@forhappy forhappy closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant