Skip to content

Retain captured browse indexes and original sources across GC - #591

Closed
forhappy wants to merge 5 commits into
codex/catalog-current-integrationfrom
codex/captured-browse-retention
Closed

forhappy wants to merge 5 commits into
codex/catalog-current-integrationfrom
codex/captured-browse-retention

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

A reader opened before ref movement or a browse-index rebuild can lose its cold index pages and original pack/visibility sources during GC. Replacement readiness currently drops the old dependency roots, even though the captured reader still addresses those immutable bytes.

This change separates query readiness from storage lifetime. Browse maintenance authenticates and CAS-publishes an immutable capture before advertising readiness. Repository GC follows retained captures, including original checkpoints and ref-only capsule runs, alongside graph, path-state and physical-index pages. Explicit history pruning trims superseded captures only after the authoritative history transition succeeds; an unchanged capture does not append another record.

Stacked on #587 (codex/catalog-current-integration, 6fab757a397). Current head: 1fd3251d794. No merge requested.

Changes

  • Metadata owns the bounded, authenticated capture chain; storage owns its paths. The new lifetime format uses v2/browse-retention and v2/browse-roots/. Existing version-3 browse readiness is unchanged; no release tag contains this browse-readiness protocol.
  • Publication retains original sources before readiness CAS under existing generation ownership and GC writer fences. A failed later publication may retain extra bytes, never advertise unretained dependencies.
  • GC rejects published readiness without a matching retained capture. Storage retention does not grant current object membership or ref visibility; normal reads retain captured inventory, generation and Git-validation checks.
  • GC uses seven actual candidate LIST streams, with one metadata scan filtered to exact owned index directories. Tests record those prefixes and protect unrelated/lookalike metadata from collection.
  • History pruning preserves newest/current captures, releases its fences on cleanup failure, and retries cleanup after a partially completed prior prune.
  • Qualification captures the immutable view and readiness, waits for external publication/GC, then constructs the reader and queries the original view. Repository construction performs normal visible-tip graph validation after GC; all graph/physical/path pages must remain cold before the handshake.

Validation

At 703f02a4a8e, source hashes unchanged throughout:

  • Metadata browse/capture tests: 12 passed.
  • Remote publication tests: 2 passed.
  • CLI GC, cold captures, cleanup and history-recovery tests: 17 passed. Both original ref-movement and readiness-replacement regressions now pass; their assertions were preserved.
  • Release qualification executable: built.
  • Strict release all-target Clippy for metadata, storage and remote publication: passed.
  • Workspace format check and diff check: passed.
  • Earlier physical-index authentication/retention slice: 17 passed; those sources are unchanged by the three latest follow-up commits.

The example-only follow-up at 1fd3251d794 also passes a release build and strict example Clippy. Production sources are byte-identical to the tested 703f02a4a8e; the source comparison is retained with the qualification evidence.

The earlier draft's readiness-replacement failure is fixed by retained capture ownership. Earlier failed runs remain in local evidence; assertions were not waived.

Commands (use a checkout-specific target on the mounted workspace volume):

cargo test -p crab-metadata --release --locked --no-default-features --features remote-index --lib capsule_protocol::browse
cargo test -p crab-remote --release --locked --no-default-features --features publication --lib browse_indexes
cargo test -p crab --release --locked --no-default-features --features simd-accel,gix-pathmatch --lib -- cmd::gc::captured_index_tests cmd::gc::tests::capsule_protocol_gc cmd::gc::capsule_cleanup_tests cmd::history_recovery::v2
cargo clippy -p crab-metadata -p crab-storage -p crab-remote --release --locked --all-targets --features crab-metadata/remote-index,crab-remote/publication -- -D warnings

Remaining release gates

This remains draft. Official 703f02a4a8e and 1fd3251d794 installations completed with unchanged source hashes. The first raw RustFS reader-coherence smoke passed, but independent lifecycle acceptance rejects it: the source objects were aged past one hour while the command used the default 24-hour grace period. It does not prove protection from eligible collection. The raw report is preserved; no lifecycle success is claimed from it.

The separate ref-only lifecycle case now passes 61 checks at 1fd3251d794. A fresh captured view stayed cold while 532 real pushes rolled its aged visibility-only source out of active coverage, replacement readiness was published, and repository GC persisted its root transition. No checkpoint history masked retention. Cold old blob identity, directory attribution and path history remained correct; measured reads wrote nothing and drained backend operations.

The task-owned local overlay was atomically configured with the existing gc_grace_period_secs = 3600 setting before the final 66 pushes. Its hash and value remained unchanged through GC, and all target provider timestamps preceded the effective one-hour cutoff. The matching source's local config merge/GC call chain was audited; the command had no override or force flag. Independent acceptance records this configuration proof alongside the original raw report. This stronger case covers ref movement, readiness replacement and an obsolete ref-only source; it does not validate explicit pruning, publication faults, catalog-only sessions or the final combined stack.

Earlier probe setup failures are also preserved: path admission compared a resolved volume path with an unresolved symlink; full repository construction warmed graph pages during normal tip validation. The revised capture protocol delays construction until after GC and preserves the zero-decoded-pages assertion.

Retained-chain fsck is implemented separately in #594 with focused tests; its provider-backed qualification remains open. Fault/cancellation/concurrent-publication qualification, catalog-only captured-session lifetime, history/resource scaling and broad CI remain open. This PR does not complete the whole indexing/staging initiative or claim final combined 10 GiB qualification. The ongoing older-component large-file matrix is separate evidence.

Review evidence

Boundary Evidence
Entry point Browse maintenance ensure; repository GC sweep_capsule_objects; explicit prune_history
Lifetime owner Metadata capture codec/authentication and CAS; CLI owns pruning policy
Publication caller/callee browse_indexes::ensure -> retain_browse_indexes -> verified immutable write -> conditional lifetime head -> readiness CAS
Reader sibling CapsuleRepositoryView::git_repository_from_store and normal physical-index loading retain captured inventory/visibility checks
Other dependency siblings Graph/path-state pages, physical descriptor/body, original checkpoint and capsule/pack-layer sources are retained together; ref-only runs are included
Cleanup sibling History CAS precedes capture pruning; normal lease/root-fence release handles cleanup errors
Main behavior This is a stacked feature change. The recorded pre-stack main lacks the browse marker and paged physical index; current-main integration proof remains pending
Parent behavior Ref movement drops current-state retention; readiness replacement loses former roots; both cold-reader regressions demonstrated missing bodies
Tests Cold captures across ref movement/replacement, metadata integrity/resource checks, publication, exact LIST streams, unrelated metadata, history/restore and cleanup

Is this the best fix? A separate lifetime authority addresses replacement and source ownership without making a stale readiness record authorize new reads or adding foreground history scans. It adds a persistent codec, CAS lifecycle and pruning boundary, which account for the production LOC growth; final release acceptance still depends on the open gates above.

No configuration, dependency or lockfile changes. The two requested large-file audit reports and user-owned advisor plan are excluded. Add-classifier cache I/O is isolated in #592.

Keep published readiness dependencies after refs move. Resolve physical index retention from its own authenticated descriptor without treating old inventory as current authority.

Retain regressions for ref movement and readiness replacement. Replacement lifetime remains an explicit draft release gate.
Publish a separate authenticated lifetime chain before readiness CAS. Retain original Git source ranges with cold index pages until explicit history pruning retires their generation.

Prune lifetime roots only after history CAS commits, retry cleanup after partial success, and preserve current readiness. Keep foreground readiness reads unchanged.

Refs #591
Captured visibility and catalog pages can live in runs without Git members. Retain every captured capsule pointer instead of only the physical pack-source directory.

The metadata regression uses a real ref-only run and retains its source-path assertion. No dependency contract or assertion is weakened.
Keep immutable capture roots in the v2 control namespace and scan derived index bodies through one bounded ownership filter. Report seven actual LIST streams without changing the existing assertions.

Cover lookalike metadata directories and recorded LIST prefixes. Move the captured view through readiness attachment instead of cloning its metadata.
@forhappy forhappy changed the title Retain published browse-index pages after ref movement Retain captured browse indexes and original sources across GC Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant