Skip to content

perf(remote): select publication catalog dependencies - #579

Closed
forhappy wants to merge 2 commits into
codex/sqlite-wal-safetyfrom
codex/selected-publication-catalog
Closed

forhappy wants to merge 2 commits into
codex/sqlite-wal-safetyfrom
codex/selected-publication-catalog

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Shared capsule publication decoded the complete historical Xet catalog before checking an outgoing pointer. A fixed pointer publication now queries its file binding, candidate shard/xorb identities, and their authenticated dependency closure from the captured view in one aggregate budget. Candidate identities remain necessary when a new file reuses historical artifacts.

Stacked on #577 (codex/sqlite-wal-safety); review this incremental diff against that branch. No format, dependency, or configuration changes in this PR.

Validation:

  • Paged-reader tests: 16 passed, including mixed identity queries, aggregate admission, descriptor conflicts, missing dependencies, corruption and warm-cache behavior.
  • Public prepare → upload → commit regression: all eight cases passed, varying 100/10,000 unrelated catalog entries, run/checkpoint storage and existing-pointer/new-binding reuse. Each measured upload stays within 24 reads / 512 KiB and performs no payload PUTs; a fresh view observes the committed ref and file binding.
  • The same regression with the old complete-catalog caller fails the unchanged read bound at 10,000 entries.
  • cargo fmt --all --check and git diff --check pass.
  • Strict combined Clippy stops on six unfulfilled double_must_use expectations in crab-coordination, unchanged relative to this PR's base. These annotations differ from current origin/main; this is an inherited stack issue, not a demonstrated failure on main. The broader metadata capsule suite passed all 129 tests. Strict selected-package cargo clippy --no-deps passed for both changed crates, all targets, and the publication/storage/remote-index feature combination after replacing a test-only cloned slice with a borrowed slice.

The regression uses real 8 KiB Xet shard/xorb content for the selected file and synthetic unrelated metadata without payload bodies. It proves selected publication behavior, not complete-repository, S3 latency, or 10 GiB qualification. Those gates remain open.

Evidence map:

  • Entry point / caller: HTTP receive in crates/crab-http-server/src/receive/publish.rs calls Prepared::upload_capsule.
  • Ownership: crab-remote selects outgoing/candidate keys; crab-metadata::PointerCatalogLookup::dependencies authenticates the loaded closure; storage retains transport ownership.
  • Callees: captured-view pointer_catalog_lookup, dependency proof, then upload_capsule_content and existing per-ref commit. Leases/GC fences and payload-before-visibility ordering remain required.
  • Siblings: CLI preparation already uses selected catalog reads. Protected receive's separately authorized source/filtered views and hydration still need their own integration; explicit full maintenance inspection remains intact.
  • Existing tests: file/chunk/run precedence and corruption coverage remains; new mixed-query tests and public publication regression exercise this consumer.
  • Baseline: the stacked parent materializes the catalog at this call; current main does not contain this capsule publication method. This PR depends on the preceding series.

Is this the best fix? The selected dependency API extends the existing authenticated reader and makes file-only queries delegate to it. It replaces the foreground full scan without a fallback or a second codec. Approximately 53 net production lines add combined candidate selection and closure validation; tests cover the cross-path reuse invariant.

Excluded: the two large-file audit reports and the unrelated local advisor-plan edits. Unified staging, protected receive/hydration, lifecycle qualification and fuller Git blame remain follow-up work; this PR does not claim the overall initiative is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant