Skip to content

Integrate selected publication with unified staging - #585

Closed
forhappy wants to merge 6 commits into
codex/selected-catalog-consumersfrom
codex/catalog-staging-integration
Closed

forhappy wants to merge 6 commits into
codex/selected-catalog-consumersfrom
codex/catalog-staging-integration

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Native/Git staging, selected Xet publication, captured hydration and protected receive need to run together on one source tree. This integration brings the selected-publication, xorb format-limit and protected-receive changes onto the existing unified staging and captured hydration branch. It adds checks that publication does not widen older hydration captures and that shared catalog reads preserve close and abandonment behavior.

Stacked on #581 (codex/selected-catalog-consumers, 43203ff48f6), which includes #578's canonical ingestion and staging-retirement implementation. Imports #579, #580 and #583; production code applied without conflicts. Documentation retains both consumer descriptions. The current-state capture extraction in #582 remains a separate series.

Additional qualification:

  • At 100/10,000 historical catalog entries, run-backed and checkpoint-backed publication each exercise both existing file bindings and newly supplied bindings. The captured lookup remains on its original state after the new ref becomes visible; a fresh capture resolves the published binding.
  • The same cases require zero catalog I/O during lookup construction and at most 24 range requests / 256 KiB during the fixed hydration lookup. No mutable-root/index reads or checkpoint writes are permitted in that measured query.
  • A paused cold catalog page read must not serialize a warm lookup. Close waits for the active query, rejects new work and completes after either successful read completion or abandonment.
  • The existing real shard/xorb hydration test removes the pointer shard hint and checks byte-identical reconstruction. Metadata-only historical descriptors are not evidence that every unrelated payload was materialized or hydrated.

Validation on the frozen a6eeca4319d tree, guarded by 1,224 Rust source/manifest/lock hashes:

  • 30 metadata file-lookup tests pass.
  • 88 xorb builder/parser tests pass.
  • 42 hydrator tests pass, including real shard/xorb reconstruction without a pointer hint.
  • The eight selected publication/hydration cases pass (one table-driven integration test).
  • All 81 receive tests pass, including private-source and retained-but-invisible object rejection.
  • All seven push-snapshot, publication-history recovery and retained-recipe visibility tests pass.
  • CLI library/binary check passes with simd-accel,gix-pathmatch,nfs and no default features. Its 12 minimal-profile warnings are not a strict CLI-lint pass.
  • Strict all-target, no-deps Clippy passes for metadata/read/remote/auth/xet; formatting and diff whitespace checks pass.

A subsequent CLI lib-test build found an imported test still calling the now-async git_visibility_index synchronously. This PR also adds its missing await, retaining the policy/visibility assertions. The focused force-rewrite positive control now passes, retaining its six force/lease/deny policy cases and visibility assertions. Two new captured-reader GC regressions fail on the unchanged production implementation: after either ref movement or readiness replacement, repository GC deletes an aged physical-index body even though the reader was just captured and its pages remained cold. Both fail with the expected typed storage NotFound; the runtime result is retained separately from the earlier compile failure. Retained-reader lifecycle correctness is therefore a confirmed open gate, not an inferred risk.

The official native installation and the six-case 10 GiB matrix use a separate frozen a6eeca4319d checkout. Those source-bound results will be component evidence for that exact tree, not a claim about later source changes. Installation and qualification are still pending.

Evidence map:

  • Entry points: native crab add, Git clean/push, Prepared::upload_capsule, protected verify_receive, revision downloads and remote mount setup.
  • Owners: staging owns byte ingestion and payload lifetime; callers own Git-index and remote publication transactions. Metadata owns captured catalog identity, selected closure validation and shared lookup lifecycle. Read owns reconstruction and native pack intake; auth owns separate source/filtered authority and candidate graph admission.
  • Callers/callees: shared publication selects outgoing pointer and local artifact keys through PointerCatalogLookup::dependencies. Snapshot/mount hydration passes SharedFileIndexLookup::for_pointer_catalog to the canonical hydrator and StoreClient. All xorb admission variants use the same hard format-limit rollover. Protected receive validates filtered/incoming Git visibility before installing private source packs.
  • Siblings: current-state file lookup is still eager; refactor(metadata): share ref capture authority #582 is extracting its shared authority owner. Full catalog maintenance remains explicit. v1 protected Git-object isolation, end-to-end workflow cancellation and retained-capture GC races remain required work.
  • Current main: fetched origin/main at cc2e4e25f46eeaee3cfd30025d257397c17a6049 is an ancestor; the unmerged series supplies the capsule/staging changes. This PR is an integration gate, not a main-level completion verdict. All 19 audited local release tags omit the captured catalog constructor; no compatibility API is added.

Is this the best fix? It exercises the existing canonical staging and catalog owners on one tree rather than adding a competing hydration implementation. The additional tests cover their shared snapshot and cleanup contracts. Imported production changes retain their separate review history.

Remaining release gates: source-bound integrated 10 GiB alternating native/Git edits and pushes, cold historical reconstruction, full protocol/provider and mounted runtime qualification, lifecycle/GC/publication races, current-state capture, representative hosted-S3 cold/warm latency and fuller Git blame semantics. Earlier binaries and sibling-branch receipts are component evidence only. No merge or production rollout is included.

The two requested benchmark audit reports and the user-owned advisor plan are excluded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant