Skip to content

fix(storage): retain verified immutable object identity - #590

Closed
forhappy wants to merge 1 commit into
codex/capsule-fault-smokefrom
codex/immutable-origin-identity
Closed

forhappy wants to merge 1 commit into
codex/capsule-fault-smokefrom
codex/immutable-origin-identity

Conversation

@forhappy

Copy link
Copy Markdown
Contributor

Immutable create/adopt outcomes discarded the ETag/version returned with the bytes or successful PUT. A later origin receipt would have to query the object again and could attach verification to a replacement version. Retain the exact operation identity in the existing outcome.

Draft, stacked on #589. This is the storage prerequisite for repairing capsule cross-repository reuse; it does not publish the missing capsule chunk index or fix the failing end-to-end assertion yet. No merge requested.

Behavior

  • Checksum-acknowledged creation returns the successful PUT’s ETag and version.
  • Required readback returns the identity of the GET whose bytes passed the existing full-body hash check. Corrupt readback returns an error before exposing verified identity.
  • Occupied-key results return bounded bytes and their same-GET identity. Bytes remain untrusted until each caller’s existing logical-Xet checks succeed.
  • CLI Xet publication, shared prepared-content upload, and xorb optimization retain their existing created/adopted behavior and validation. The return contract carries origin identity for the following proof-publication change. No extra HEAD, GET, upload, format, dependency, or configuration change.

Validation

  • Focused storage tests: 6 passed, including deliberately different PUT/GET ETags and versions, corrupt readback, occupied-key encoding, attributes, and existing-body size bounds.
  • Strict storage all-target Clippy: passed.
  • Consumer compile (crab --lib, no default features, nfs,simd-accel,gix-pathmatch): passed; warnings are in unchanged surfaces.
  • CLI build (crab, same feature set): passed.
  • Current-head RustFS committed-restage replay: 34 commands / 22 checks passed, including byte-identical latest and committed-prior version reconstruction. This is a 4 MiB correctness smoke, not final 10 GiB or representative S3 latency proof.
  • Format and diff checks: passed.

Review evidence

Boundary Evidence
Entry / owner Store::create_or_read_immutable[_with_attributes] and ImmutableCreateOutcome, owned by crab-storage
Caller / callee CLI publish_xorb_candidate, shared upload_capsule_content, optimizer upload_destination -> object_store::ObjectStore::put_opts and bounded GET
Dependency contract Locked object_store 0.14.1 UpdateVersion requires retaining both ETag and version; From<PutResult> preserves them. Bounded GET already captures both from the response carrying the bytes.
Siblings All three production consumers updated; existing-byte logical, digest, and chunk validation retained. Staged protected writes keep their separate service-promotion boundary.
Tests Real InMemory conditional writes and byte reads, with a provider adapter giving PUT/GET different metadata; corrupt bytes remain rejected.
Current main / release Latest origin/main is cc2e4e25f46; this outcome API is absent there and at v1.2.4. This changes an unreleased Rust contract in the capsule stack, not stored receipt encodings. Released receipt fields remain unchanged.

Is this the best fix? Preserve identity where verified bytes and provider response are still together. The following capsule proof work can carry that identity without introducing a second read or trusting a later HEAD. Production LOC grows only to return metadata already available at this boundary; most added code is focused test coverage.

Remaining initiative gates

The cache-free publisher/classifier regression still returns [None] after a successful source push (0.80 seconds). Fresh-bucket RustFS also fails the unchanged remote-authority assertion after 27 commands / 17 checks. Next: carry both whole-body and payload digest identities with verified origin identity; publish shared capsule-bound candidates for direct/protected publication; revalidate source catalog and origin; invalidate matching local misses after visibility. Captured-reader GC retention, final integrated 10 GiB multi-edit/push qualification, fuller blame, source-policy mismatch, and representative S3 scaling/latency remain open. Parent #588 CI also has a Blob response-loss shutdown failure with local_disk_reserved_bytes: 160688 and an image qualification blocked after repeated AWS CLI registry pulls; their diagnoses remain separate from this change. The excluded audit reports and user-owned advisor plan are omitted.

Keep the provider token beside the bytes it identifies. Preserve both ETag and version from the PUT or verified GET so origin proofs need no later HEAD.

Update all consumers of this unreleased Rust outcome shape; stored receipt formats remain unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant