Skip to content

Integrate indexed catalogs, staging and lifecycle - #598

Closed
forhappy wants to merge 17 commits into
codex/catalog-source-retentionfrom
codex/indexed-staging-final-integration
Closed

forhappy wants to merge 17 commits into
codex/catalog-source-retentionfrom
codex/indexed-staging-final-integration

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Integrate authenticated selected catalog reads with unified staged-payload publication so native add, Git clean, direct push and protected/shared preparation can be qualified on one revision. Stacked on #597; component PRs remain review units. Nothing is merged to main.

The integrated path preserves one aggregate dependency-query budget, complete selected file/shard/xorb closures, verified chunk placements, exact provider identities and flush-before-publication ordering. Catalog and browse source lifetime is publisher-owned and followed by GC. Full fsck retains source/browse authority through later phases and proves every reachable Crab/LFS pointer’s origin bytes using the shared integrity verifier. Mirror recipe probes retain their narrower workload. The HTTP fault fixture waits for its repair worker and reloads snapshot-bound readiness before each injected corruption; production readiness semantics are unchanged.

Exact combined head 3daffd89531:

  • 561 metadata tests, 95 CLI fsck/GC/history/chunk/publication tests and 21 storage-only lifetime tests pass. These are focused slices with overlapping coverage, not a summed workspace total.
  • Strict all-target Clippy for metadata/shared/protected crates, minimal metadata/remote feature builds, formatting and whitespace checks pass.
  • Official make install and both publication-feature qualification binaries build successfully. All 2,645 guarded source inputs remain unchanged; installed binary hashes are recorded. The checkout is clean and current upstream main cc2e4e25f46 is already an ancestor.
  • Exact-source live API preparation, native HTTP push, all six receive-fault scenarios and native mixed Crab/LFS checks pass. The installed native proof completes 28 commands / 46 checks, including missing/corrupt origin rejection and independent cold reconstruction. The six-case 10 GiB matrix stopped at the initial entropy/native-first push: immutable xorb readback returned a different BLAKE3 digest. Add and commit completed; the five remaining cases have not started. Three later direct origin reads exactly match the preserved local body, including its expected BLAKE3 identity and every Xet chunk. The initial response mismatch remains unexplained. A separately recorded retry of the owned initial push succeeded: all 161 audited origin xorb responses matched the local payloads. A fresh independent clone then read 11,090,949,360 backend response bytes, verified both full 10 GiB SHA-256 oracles, passed native Git strict/full fsck, and passed full origin fsck with zero errors (12 commands / 23 checks). This is one recovered version with the original source preserved, not source-removal, four-version, lifecycle or six-case acceptance. The original matrix remains failed.

Earlier source-bound evidence:

  • The prior combined source passed metadata/shared/protected/staging/shallow/publication and HTTP checks, including real RustFS native API and HTTP fault/restart qualification. The HTTP fixture’s original 202/503 failure, delayed-worker reproduction and passing fix remain retained externally.
  • Frozen source 909a40a2559 completed a two-file, four-version, 10 GiB entropy/native-first run with 5,000 code files, 532 additional publications, actual aged GC and eight independent cold historical reconstructions: 1,678 commands / 117 checks, all passing. It predates this final combination and the reachable-pointer fsck fix.
  • Component fix(fsck): verify reachable pointer payloads #601 has a real native mixed Crab/LFS proof: 28 commands / 45 checks. A healthy local LFS copy does not mask missing or same-size corrupt origin data; the fixed binary fails, exact restoration succeeds, and a fresh clone reproduces both payloads. This is component proof, not final combined-source qualification.

Remaining gates:

  • Resolve and reproduce the initial 10 GiB readback mismatch, then complete all six pattern/order cases and the captured-reader / aged-GC gate; retain exact-source receipts and independent byte oracles.
  • Opening still reconciles historical visibility controls. The earlier fixed-query baseline held the selected catalog query at seven requests while opening grew from 9 to 1,079 requests across 532 publications. A separate, uncommitted regression now isolates two reads per historical visibility edit inside one physical run; it is outside this PR’s frozen source. Opening, distinct-chunk/ref scaling and representative S3 latency remain required.
  • Complete crash/restart, concurrent publication/pruning, stale-cache and migration qualification. Fuller merge/rename/source-line Git blame semantics remain a separate required stage.
  • The strict CLI lint backlog and stale rusqlite source-policy inventory remain open. No inventory, baseline, assertion or repository lint policy is relaxed. This draft is not ready to merge.

Evidence map: native/Git add/push, shared Prepared upload/commit and protected receive enter the existing staging, metadata, storage and write ownership boundaries. Selected dependency/shard lookup and operation-bound immutable receipts are shared callees. Hydration, fsck, GC and pruning share lifetime ownership. The changed fsck entry point is main -> run_fsck -> StoreChecker -> shared reachable-dependency verification. Historical recovery already uses that verifier; mirror probes do not initialize full browse/Git maintenance. Current main lacks the complete combined path; before-change false-clean and HTTP failures are retained as source-bound evidence.

Is this the best fix? The implementation keeps verification and lifetime policy at their existing owners, with one shared origin verifier and bounded control rechecks. The HTTP change synchronizes fault injection with its repair owner. The unresolved opening replay is explicitly retained as required implementation work rather than hidden behind selected-query measurements.

The advisor plan and both excluded large-file audit reports are absent from this PR.

Qualification-runner failures were preserved: one runner prefix was rejected by the unchanged HTTP namespace guard before traffic; a long generated log filename then stopped recording the native cold-clone result. Correct inputs and a short operation label resolved those runner issues. Passing API/HTTP stages and the corrected native proof are independently audited on the same exact source and binary; no product test, guard or assertion changed.

An unopened global index cannot supply per-chunk absence facts. Keep that state in the existing OnceCell instead of persisting a negative row for every chunk in a new file.

Preserve initialized-index cache policy, cancellation, close ownership and push-side proof validation.
Prove cold catalog pages survive checkpoint replacement and a real orphan sweep. Repair the async visibility test caller and carry forward the qualified staging-version fixture.

Record the remaining shallow-date adapter failure without weakening its CLI smoke assertion.
Remote helpers receive Git date expressions before wire normalization. Apply shallow selectors before requested-tip selection so an empty result cannot become a depth-one checkout.
Advertise an archive side tip instead of the ancestor under raw-fetch admission test. Its exclusion closure still contains that ancestor, preserving both selector and admission assertions.
V2 stores Git packs and indexes inside capsule runs. Target those ranges and compare authority bytes directly: damaged discovery controls cannot prove unchanged refs through ls-remote.
Keep the provider token beside the bytes it identifies. Preserve both ETag and version from the PUT or verified GET so origin proofs need no later HEAD.

Update all consumers of this unreleased Rust outcome shape; stored receipt formats remain unchanged.
Bind shared chunk hints to authenticated source shards and exact immutable object identities. Keep capsule provenance separate from released manifest receipt keys.

Draft: exact staged-proof revalidation and current-source real-store qualification remain incomplete.
Drain the preceding repair worker and bind each injected fault to current readiness. A published ready record can precede worker completion, so overlapping mutations can legitimately observe graph-only readiness.
Retain decoded head CAS tokens and the pruning floor through source, data and Git verification. Reject concurrent authority movement or corrupt control bytes without rescanning immutable history.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant