Skip to content

fix: preserve inline provenance in redb change exports - #257

Closed
vinceblock99 wants to merge 1 commit into
devfrom
fix/redb-inline-provenance
Closed

vinceblock99 wants to merge 1 commit into
devfrom
fix/redb-inline-provenance

Conversation

@vinceblock99

Copy link
Copy Markdown
Contributor

RedbChangeStore discarded inline provenance on import, then exported files that claimed to contain it and could not be read back. Preserve the original section bytes in a registered redb table and write them back during export.

Tested signed export/reimport, reopen, legacy database consolidation, the repository/libatomic suites (1,703 passed), and 8 migration checks.

Previously discarded provenance requires reimporting the original change file. Extended HEADER preservation is covered separately by #256.

@graywolf336 graywolf336 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At PR head b9c12dd, export preserves the decoded provenance payload but not necessarily its original hashed representation.

The call chain is:

  1. Export decompresses the stored payload and passes it to the writer — atomic-repository/src/redb_change_store/queries.rs:436–439:

    let payload = zstd::decode_all(value.value()) /* ... */?;
    writer.write_provenance_payload(&payload)?;
  2. The provenance writer delegates to the raw-section writer — atomic-core/src/change/format_v3/writer/state_machine.rs:428:

    self.write_raw_section(SectionType::Provenance, payload)?;
  3. The raw-section writer recompresses the payload — atomic-core/src/change/format_v3/writer/helpers.rs:139–140:

    let compressed = zstd::encode_all(uncompressed, self.options.compression_level())
        .map_err(|e| FormatError::Compress(e.to_string()))?;

    Export creates the writer with WriterOptions::default() at queries.rs:413, so this uses level 3 regardless of the original compression settings.

  4. The newly compressed bytes contribute to the content hash — writer/helpers.rs:150–153:

    if section_type.is_hashed() {
        self.hasher.update(&header_bytes);
        self.hasher.update(&compressed);
    }

The new import code at atomic-repository/src/redb_change_store/mod.rs:463–467 stores a level-3 compression of the already-decompressed payload rather than retaining the original compressed section.

Runtime-confirmed: A valid change with only its PROVENANCE section compressed at level 19 imports under its original hash, but export produces a different verified content hash despite identical decoded data. This can break dependency references and signatures bound to the original hash.

Requested fix: Retain the original compressed PROVENANCE section bytes during import and replay them without recompression. Add a nondefault-compression regression asserting that import → export → reimport preserves the original content hash.

@leefaus leefaus closed this Oct 31, 2026
@leefaus
leefaus deleted the fix/redb-inline-provenance branch October 31, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants