Skip to content

feat(index): publish authenticated capsule chunk hints - #593

Closed
forhappy wants to merge 4 commits into
codex/immutable-origin-identityfrom
codex/capsule-chunk-authority
Closed

forhappy wants to merge 4 commits into
codex/immutable-origin-identityfrom
codex/capsule-chunk-authority

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Capsule pushes currently publish Xet payloads without populating the shared chunk index queried by a cold native add in another repository. The consumer repacks identical content instead of reusing the committed payload.

This draft publishes bounded capsule chunk candidates after ref visibility and validates them through the source repository's authenticated shard closure plus the canonical object's verified identity. Direct CLI publication and protected receive share the metadata encoder/writer. Immutable placement-digest records preserve the original add-time proof when shared heads change. Push revalidates those exact placement/origin IDs and retains authenticated source descriptors and original receipts, avoiding a redundant full xorb GET. Partial add-time hints expand only the requested closure; mixed physical assertions fail explicitly. New key families keep capsule provenance separate from released manifest receipts; whole-body and payload digests remain distinct. The shared writer initializes the canonical format marker for new databases and rejects noncanonical existing databases.

Stacked on #590 (codex/immutable-origin-identity, 3637d2d0aae). Review the incremental diff against that base. No merge requested.

Validation

  • Final-source capsule candidate codec, immutable placement history, writer and key-codec tests: 19 passed.
  • Paged catalog reader and dependency-closure tests: 12 passed.
  • Final-source metadata strict all-target Clippy with remote-index: passed. Strict minimal-feature Clippy and 28 payload-only run tests also passed.
  • Current-source CLI and protected-receive library compile checks: passed with the selected minimal CLI feature set (nfs,simd-accel,gix-pathmatch).
  • Final-source CLI/Xet tests: 10 passed, including source-root removal, changed head, forged IDs, current-origin checks, warm caches, partial hints and mixed old/new origins. The unchanged I/O bound failed before the fix (one full xorb GET) and passes after (zero).
  • Existing origin-receipt and remote-range verification siblings: 12 passed. Protected receive: 80 passed, using native Git and an in-memory store.
  • Capsule smoke proof oracle: 5 passed in normal and optimized Python; rejects counters spread across events. Requires exact source/payload counts and zero mismatches/stale/global reclassification.
  • Production CLI binary build: passed.
  • Current-source isolated RustFS cross-repository add/push/cold reconstruction: 34 commands, 32 checks passed, using a 4 MiB deterministic file. Consumer add keeps no segment/prepared payload; push revalidates all 58 exact proofs with zero mismatches/stale/global reclassification and uploads no new xorb. Add and push each fetch only 2,658 xorb metadata bytes in two ranges, with zero full xorb GETs. Empty-cache hydration performs a real remote payload GET and matches the original SHA-256. Recorded source hashes remain unchanged throughout.
  • Binary SHA-256: 5002c1c59f8e2d6d3fa82abed1c5686c67d46feb93f975c0c1d76c1930f3dc70. Report SHA-256: 36d6ec1b744a2f5545a9a998710271bc2fa092ceadf756dd56c95acf75eeff2f.
  • Rust format and diff checks: passed.
  • The cache-free real publisher-to-cold-consumer regression passed on 3c93bbe140b (0.55 s runtime; 6m45s test build). It uses real native Git and staging with an in-memory object store, a different consumer repository, and no advisory candidate caches. This proves publisher/classifier integration, not real S3 qualification.

Required before readiness

  • Qualify failure, cancellation and cleanup paths, concurrent writers, stale candidates, source retirement, protected namespace isolation, and the shared crab-remote publication sibling.
  • Repeat the full 10 GiB alternating native/Git multi-edit matrix on the eventual combined source. Earlier 10 GiB successes do not qualify this commit.

This is an unfinished draft, not completion of the indexing/staging initiative or a merge-ready correctness verdict.

Review evidence map

Surface Evidence and remaining scope
Changed entry points crab/src/git/capsule_push.rs and crates/crab-auth-server/src/receive/capsule.rs publish hints only after a successful ref transition
Caller / owner / callee Native add classifier -> ChunkIndexStore::capsule_candidates -> metadata codecs and paged PointerCatalogLookup::shards -> existing canonical-origin verifier
Physical identity CLI Xet publication and protected promotion carry the exact verified PUT/GET identity from #590; no later HEAD certifies earlier bytes
Siblings Existing manifest-receipt lookups retain separate keys; files() reuses the selected-shard closure reader; protected receive tests pass; shared remote publication still needs integration proof
Dependency contract Locked SlateDB 0.15.0 source: buffered writes require awaited close for durability; writer paths explicitly attempt close on success and failure
Current main / parent Fetched origin/main (cc2e4e25f46) is an ancestor and has no capsule candidate publisher; parent #590 retains the publisher/classifier gap
Tests New codec/source-root/format-marker tests plus existing remote-index and paged-reader suites; publisher/classifier integration, exact staged-proof regressions and isolated real RustFS duplicate reuse/cold reconstruction passed

Is this the best fix? Shared metadata ownership and operation-bound origin identity address the observed publication gap. The full best-fix verdict remains open until all publication siblings, fault/lifecycle paths and final large-file qualification are proven. This draft deliberately records those gaps.

The added production code introduces one capsule provenance contract shared by direct publication, protected receive and add lookup; it does not replace the existing released manifest contract. The follow-up production LOC growth preserves immutable proof history, returns bounded caller-owned descriptors and adds exact revalidation through the existing origin verifier; the long end-to-end regression lives in its own test module. No dependency or lockfile changes. The two large-file audit reports and user-owned advisor plan are excluded.

Bind shared chunk hints to authenticated source shards and exact immutable object identities. Keep capsule provenance separate from released manifest receipt keys.

Draft: exact staged-proof revalidation and current-source real-store qualification remain incomplete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant