Skip to content

fix(xet): split xorbs at the protocol chunk limit - #580

Closed
forhappy wants to merge 1 commit into
codex/selected-publication-catalogfrom
codex/xorb-chunk-count-limit
Closed

forhappy wants to merge 1 commit into
codex/selected-publication-catalogfrom
codex/xorb-chunk-count-limit

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Packing enough small chunks could create a xorb that Crab immediately rejects on read. The 10,000-file protected-receive qualification reproduced xorb contains 10000 chunks; maximum is 8192 with real recipes and payloads.

Seal the current xorb before accepting a new unique chunk beyond the locked Xet chunk-count limit. The check lives beside the existing decoded-size rollover in the common state-mutating builder path. Deduplication still runs first; sealing still requires the existing materialization admission callback before mutation. Run continuity cannot override a format limit.

Stacked on #579 (codex/selected-publication-catalog). This PR contains only the Xet builder, its regression tests and the crate README; the protected-consumer changes are a separate follow-up. No dependency, format, parser-limit or configuration changes.

Validation:

  • Both new focused regressions fail before the fix: parser rejection at 8,194 unique chunks, and missing admission at the count boundary.
  • All 88 xorb builder/parser/compression tests pass with no default features.
  • Strict all-target cargo clippy --no-deps -p crab-xet --no-default-features -- -D warnings passes.
  • Regression covers serial, precompressed, batch and admitted ingestion, with byte-identical serialized outputs, payload/hash/chunk verification, dense placements, boundary deduplication, and denied-admission retry.
  • All 88 focused tests also pass with chunker,upload-concurrency enabled. The dependent protected-receive integration in fix(auth): isolate capsule verification and page catalogs #583 passes all 81 receive tests, including four real 100/10,000-file run/checkpoint cases that exercise splitting at the chunk limit. Deployed/S3 and 10 GiB qualification remain separate gates.

Evidence map:

  • Shared owner / changed surface: XorbBuilder::push_precompressed_with_reserve_and_admission in crates/crab-xet/src/xorb/builder.rs.
  • Callers: CLI pack_decoded_chunk_group in crab/src/git/push.rs; direct prepared staging in crates/crab-staging/src/stream.rs.
  • Callee: finalize_current_after_admission retains writer admission and serialized-payload pool ordering.
  • Siblings: all four ingestion APIs converge on the changed check; parser and shard coverage checks stay unchanged.
  • Dependency contract: locked xet-core-structures 1.6.0 exports MAX_XORB_CHUNKS = 8 * 1024; its raw-xorb constructor asserts the same maximum. Crab's parser rejects larger chunk counts before allocating metadata.
  • Current main: fetched origin/main at cc2e4e25f46eeaee3cfd30025d257397c17a6049 has the same builder function without the count rollover; the original builder file blob matches this PR's base.

Is this the best fix? Enforcing the existing format limit at the common ingestion boundary covers ordinary push and prepared staging together, preserves dedup/admission ownership, and avoids increasing reader limits or adding path-specific packing logic. Production change is four net lines; the larger diff is regression coverage.

Already-written oversized xorbs remain invalid and are still rejected; this change prevents new invalid output and does not silently repair stored data. Full installed-CLI/10 GiB and hosted-S3 performance qualification remain separate gates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant