perf(dng): benchmark encode and decode against the Adobe DNG SDK - #558
Open
justin13888 wants to merge 24 commits into
Open
justin13888 wants to merge 24 commits into
justin13888 wants to merge 24 commits into
Conversation
`read_raw_dng` writes the bytes to a temporary file and exports the decoded samples across the FFI boundary. Both are fine for a conformance check and wrong for a timed one: they charge the reference implementation for file I/O and for a full-image `malloc` + `memcpy` that gamut's in-memory `DngDecoder` never pays. `decode_dng_in_memory` runs the same parse -> build-negative -> `ReadStage1Image` flow over a `dng_stream` on the caller's bytes and reports only the extent of the image it produced. Pinned against the file-stream path on an Adobe sample DNG, so a benchmark cannot be timing a cheaper, different decode. Refs #163
`--bench compression` answers the #196 question on packed payloads only. This adds `--bench codec`: encode and decode throughput across the whole shipped matrix -- uncompressed, Deflate and lossless JPEG, for CFA and LinearRaw -- with gamut's decode next to the reference implementation's. Fixtures are synthesised in-process, so the harness needs no sample corpus and runs by default. The codec call, the buffer it produces and that buffer's teardown are inside the timed region; fixture synthesis and the encode a decode benchmark reads are outside it. The teardown is placed explicitly because divan otherwise defers a returned value's drop past the timed region, which would charge gamut nothing for freeing a decoded image while the SDK's negative destructor runs inside its call. Two gamut-versus-SDK comparisons are published because their biases point in opposite directions and neither can be normalised away: the whole-file decode favours the SDK (gamut also unpacks IFD 0's preview and rebuilds the metadata, a margin the fixture table prints), the lossless-JPEG codestream decode favours gamut (the oracle's export path costs the SDK two extra passes). No encode comparison: the shim wraps the SDK's reader, not its writer, so no reference number exists to compare against. Refs #163
State what the harness measures, what sits inside and outside each timed region, and -- the part a reader cannot re-derive -- which of the two gamut-versus-SDK comparisons favours which side and by how much. Also state that no absolute throughput figure is pinned: unlike the #196 ratios, MB/s is a property of the machine that produced it. Refs #163
The linear-raw branch derived a colour from `(plane % 2, plane / 2)`, which sends plane 2 to the fall-through arm -- so blue photosites were generated at green's gain. Harmless to a timing number, but the fixture no longer matched what its own comment claimed it was, and a fixture nobody can read is one nobody can check. Index a named `GAINS` table by colour instead, derived from the RGGB tile for a mosaic and from the plane for a linear image. Refs #163
The fixture table's first column did not line up. `{case:<26}` sets a
width on the formatter, but both `Display` impls wrote through
`write_str`/`write!`, which go straight to the underlying buffer and
ignore width and alignment -- so every row was ragged and the table was
harder to read than the plain text beside it.
`Formatter::pad` is the method that honours those flags. `Case` has to
build its composite string first, which costs an allocation in a
function that runs once per case outside every timed region.
Refs #163
A harness that is never read is worth nothing, so its first run is written down where the crate's other measurements live. Lossless JPEG is the outlier at both ends. Decode is ~60x slower than the reference implementation while every other scheme is within 1.25x, which localises the cost to `decode_symbol`'s linear scan of the whole code table rather than to decode overhead (#583). And a CFA file gets *larger* when the scheme is turned on, because the mosaic goes to the encoder as one full-width component and predictor 1 then differences a red photosite against its green neighbour; the spec's reshape takes the payload from 119.7% of raw to 91.5% (#584). Both quantities are bytes or ratios rather than absolute times, so both reproduce off the box that measured them. Refs #163
The codestream comparison in the codec benchmark carried one residual bias that was described rather than measured: crossing the FFI boundary costs the Adobe DNG SDK a `malloc`, a `memcpy` out of its spool buffer and a copy into a `Vec`, none of which gamut's single `Vec` pays. Add `decode_lossless_jpeg_extent`, which runs the identical `DecodeLosslessJPEG<Scalar>` into the identical spool buffer and stops before those copies, so the gap between the two entry points is the export path and nothing else. A differential test pins that the two reach the same decode. Also rename the memory-stream oracle test to what it asserts: it compares the stage-1 *extent*, not the pixels, because the entry point it covers deliberately exports none.
The two implementations sat in separate divan benchmarks, which run in name order, so every reference case was measured minutes away from its counterpart. On a shared machine that drifts, a ratio measured minutes apart is not a ratio: across two runs of the split harness a 1.24x Deflate figure moved to below 1.0x. Take the implementation as a benchmark argument instead, naming the arguments so divan's own name sort keeps the members of a pair adjacent. The codestream benchmark gains a third arm, `adobe-sdk-no-export`, whose distance from `adobe-sdk` prices the FFI export path. Put the IFD-0 preview volume into gamut's divan counter for the whole-file decode and for encode, both of which handle the preview while the SDK's stage-1 read does not. The median-time column is then the uncorrected ratio and the throughput column the preview-corrected one, so no reader has a subtraction to do. The fixture table prints both volumes and the factor.
The isolation claim for #583 was stated as "every decode path is within 1.25x except lossless JPEG", and the two rows that contradict it -- both uncompressed cases, at 1.8x and 2.4x -- were absent from the table. Publish the whole matrix, both runs, with the preview-corrected column beside it, and rest the isolation on the codestream pair instead: that pair carries no container asymmetry and its one residual bias is now measured at under 2%. Quote #584 against a single denominator throughout. Read in sequence the previous bullet switched from the whole file to the codestream mid-sentence, which inflates the remedy about fivefold; on the file denominator the reshape lands 6.2% below the uncompressed baseline. Record that the fixture table must not become a codec gate: the margin is a property of frame-uniform synthetic gains, and pinning an encoder requirement to one synthetic fixture is the failure a benchmark harness exists to avoid.
Two header sentences still described the previous shape: that every counter is the raw sample volume, and that the preview asymmetry belongs to two benchmarks that no longer exist by those names.
The in-memory decode entry point repeated the SDK's parse -> post-parse -> validate -> make-negative -> parse -> post-parse -> read-stage-1 sequence that `read_negative` already ran for the file-stream entry point, so the two could drift apart silently. Take a `dng_stream &` in `read_negative` and keep the path form as a two-line overload that opens the file and delegates. Opening the stream is then the whole of the difference between the two flows, which is what the benchmark's fairness claim rests on.
`gdng_decode_lossless_jpeg_extent` narrowed its `size_t` length to the `uint32` `dng_stream` takes without checking it fits, so a buffer above 4 GiB would have been decoded from a silently truncated view. Its sibling `gdng_decode_dng_in_memory` guards the same narrowing; this one did not.
The pin for `decode_dng_in_memory` sat in `tooling/gamut-dng-oracle`, which is excluded from the workspace, so it never ran in automation and could not detect the drift it was written to detect. Its sibling `decode_lossless_jpeg_extent` was already pinned inside `gamut-dng`. Move it there too: encode a DNG with this crate, decode it both ways, and assert the memory-stream entry point reports the extent the exporting one produces and that this is the encoded image's own extent. Note on both oracle entry points where their pin now lives, and why it is not beside them.
Three of the harness's published quantities were not what they claimed. The preview correction modelled the IFD-0 preview at its stored width, one byte per sample. The decoder surfaces every sub-image as `SubImageData::Decoded( Vec<u16>)` whatever the stored depth, so the buffer it allocates, fills and tears down is twice that. Model the width the decoder materialises. The correction charges preview bytes at the raw path's per-byte rate, which holds only where both paths do comparable work per byte. Under Deflate and lossless JPEG a raw byte carries entropy-coding work a preview byte does not, so there the arithmetic yields a lower bound on gamut's ratio rather than a measurement of it. Suppress it on those rows -- gamut's counter is the raw volume and both divan columns are uncorrected -- and print which rows those are, in the fixture table and in the epilogue an operator actually reads. The export-path arm bounds the FFI copy; it does not price it. Its magnitude sits at the measurement floor, where its sign is not resolved. Say bound. Also record, at the epilogue, that a pair's arms cannot be interleaved per sample: one always runs first and the bias points one way for a whole run, so a published ratio is the mean of one run each way (`--sortr name` reverses divan's sort and with it the arm order).
The oracle links the system libz dynamically, because the SDK includes <zlib.h> unconditionally. So on the two `*/deflate` rows -- and only there -- the reference arm runs code that is not built from source committed to this repository, and which libz it runs is a property of the machine and even of the launcher: cargo puts every build script's native search path on `LD_LIBRARY_PATH`, so `cargo bench` resolves the stock zlib that another dev oracle happens to have built under `target/`, while running the same binary directly resolves the platform's, which on this box is a zlib-ng fork. Inflate implementations differ by more than the margin that decides which side of 1.0 a Deflate ratio falls on, so two correct runs of this harness can disagree on those rows with no defect in either. Print the resolved library -- version plus the path `dladdr` reports, since zlib-ng's compatibility build answers "1.3.1" exactly as stock zlib does -- above the divan output, and say in the epilogue that a Deflate figure travels with it or not at all.
Re-measured across eight runs -- two repetitions of {stock zlib, zlib-ng} x
{reference arm first, gamut arm first}, 100 samples each, at loads from 15 to
38, each bracketed by `uptime`.
The two Deflate ratios an independent re-measurement could not reproduce are
reproduced here, both of them: 0.94 under stock zlib 1.3.1 and 1.17-1.26 under
zlib-ng 2.3.3, flat in load and in arm order. Neither figure was wrong. The
gamut arm does not move between the two libraries; only the reference arm does,
by 1.2-1.3x, which is enough to reverse which side of 1.0 the row falls on. Say
so, and say which library each figure belongs to.
The lossless-JPEG finding survives -- no fastest-sample ratio below 34x in any
run -- but not at the precision "56-59x" claimed from two runs; quote the bound.
The export-path figure does not survive at all: over sixteen case-runs it spans
-4% to +64%, so publish it as a bound below the run-to-run spread rather than as
a cost with a sign. Correct the preview volume to the width the decoder
materialises, record that the correction is now suppressed where it would be a
lower bound, and withdraw the claim that interleaving accounted for the Deflate
shift, which is not this section's to explain.
Two doc sites still called the gap between the two SDK lossless-JPEG arms the export path's cost. Measured over sixteen case-runs it spans -4% to +64%, so what the pair supports is a bound at the measurement floor, not a signed price. The module header already said so; these did not.
`gdng_decode_lossless_jpeg_extent` rejects a length that does not fit the `uint32` `dng_stream` takes; `gdng_decode_lossless_jpeg` narrowed the same `size_t` unguarded. The guard's own rationale named `gdng_decode_dng_in_memory` as the sibling to match, which is the wrong one: the entry point the extent arm is *timed against* is the exporting decode, and an asymmetric guard is both a difference in what the two accept and a difference inside the measured region. Give the exporting arm the identical check and state the reason at both sites. Unreachable from this crate's fixtures either way -- the streams are kilobytes.
The harness printed which zlib the reference arm called, which makes a Deflate ratio interpretable but not reproducible: a resolution that came from the build graph rather than from the platform is one nobody else gets. `cargo` exports every build script's native search path on the runner's library path, and `gamut-dng` dev-depends on `libtiff-oracle`, which builds a `libz.so` of its own -- so `cargo bench -p gamut-dng` alone is enough to measure stock zlib where the same binary run directly measures the platform's zlib-ng, and the two move that row by 1.2-1.3x. Split the resolved path out of the identity string so a caller can test it instead of reading it, and warn when it has a `target` component.
The header, the printed epilogue and the STATUS section all said that on the `*/deflate` rows "one measured path is not built from this repository" and that "every other row runs only code built here". A reader takes that as a claim about gamut's own codec, and it is not one: `gamut-deflate` is deliberately encoder-only, so this crate inflates with `miniz_oxide`. Neither arm on those rows is gamut-authored. Say so, and give the distinction that carries the section's real content -- `miniz_oxide` is pinned by `Cargo.lock` to one version and one checksum, and the system libz is pinned by nothing, not even by the machine.
The section's two findings are filed as #583 and #584, which were written from its first revision and still quote figures three later passes withdrew -- the Deflate ratios, the localisation argument resting on them, a fixture-table column the harness no longer prints, and a verification command naming benchmarks that no longer exist. Neither issue can be corrected in place from here, so #617 carries the correction; name it, and name #618 for pinning the libz the reference arm links.
Four doc sites and the benchmark's printed epilogue justified the resolved path print by claiming a zlib-ng compatibility build answers zlibVersion() with stock zlib's string. Executed on this box, it does not: the platform build answers "1.3.1.zlib-ng" and the stock copy under target/ answers "1.3.1", so the version separates that particular pair. The worked example in the oracle -- version "1.3.1" resolving to a path ending .zlib-ng -- is a composite that cannot occur. The mechanism and the print survive on the narrower true claim: the pair the loader actually collides is two *stock* builds of one version, the copy a dev oracle left under target/ and an installed libz.so.1.3.1, and those are indistinguishable by version string. The path is what identifies the resolution; a fork renaming itself is not something the identification may rest on. Refs #163
… they are The previous commit replaced one unexecuted claim with a second one: that the pair the loader collides is two stock builds of a version. On this box it is not -- the platform ships a fork that renames itself, which the same commit's own message says. What is true of every box is the shape: the loader chooses between a copy some dev oracle built under target/ and whatever the platform installed, and zlibVersion() separates those only when the platform's build renamed itself. A box shipping stock zlib 1.3.1 gives two resolutions that answer identically, so the identification rests on the path either way. Refs #163
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #163 asks for a DNG benchmark harness compared against the Adobe DNG SDK. This adds
cargo bench -p gamut-dng --bench codec: encode and decode throughput across the whole shippedcodec matrix — uncompressed, Deflate and lossless JPEG, each for CFA and
LinearRawphotometry —with gamut's decode placed next to the reference implementation's.
This section is current as of the sixth pass, and it has been corrected in place. Four review
passes changed what the harness measures and withdrew figures earlier revisions of this section
stated, and a fifth corrected a justification without changing any measurement; each is recorded
below under its own heading —
## Second pass,## Third pass,## Fourth pass,## Fifth pass,## Sixth pass— and those headings, not this one, are the append-only record.The convention protects the decision record, not the summary: a reader meets the summary first, so
it says what is true now rather than what was true when it was first written. Every correction made
here is named where it was made.
Stacked on PR #508 (
feat/442-dng-c2pa-manifest-storeat677d38ea), not onmaster: #508 isopen, delivered and green, and basing on it avoids a
crates/gamut-dng/STATUS.mdcollision. Thisrun does not merge anything. Review #508 first; this PR's own diff is everything above
677d38ea—git log --oneline 677d38ea..HEAD. The count is given as a command rather than as anumber because it has been wrong in this body twice.
What landed:
crates/gamut-dng/benches/codec.rs— the harness. It followsbenches/compression.rs's shape(divan, a printed table then the timed benchmarks) rather than inventing a second one.
tooling/gamut-dng-oracle— three dev-only entry points and their C++ shim:decode_dng_in_memory(the same parse → build-negative →
ReadStage1Imageflow the oracle already runs, but over adng_streamon the caller's bytes and stopping before the export copy),decode_lossless_jpeg_extent(the no-export arm that bounds the FFI export path), and
zlib_identity/zlib_path, which reportthe libz the SDK's Deflate reader actually called.
crates/gamut-dng/STATUS.md— a "Codec benchmark harness (Setup benchmark harnesses for DNG #163)" section.crates/gamut-dng/tests/roundtrip.rsandsrc/lossless_jpeg.rs's test module — the pins for thoseoracle entry points, inside the gated crate so a gate runs them.
What the harness runs
Three benchmarks, not six:
encode_gamut,decode_dng,decode_lossless_jpeg. Each decodebenchmark takes the implementation as a divan argument rather than living in one benchmark per
arm. divan runs benchmarks in name order, so separate benchmarks would measure every reference case
minutes away from its counterpart, and on a shared machine that drifts a ratio measured minutes
apart is not a ratio. The argument names sort adjacent, so a pair runs back to back:
So the ratio for a row is read inside one benchmark, between its
gamutandadobe-sdkarguments —
cargo bench -p gamut-dng --bench codec -- decode_lossless_jpegis a completeinvocation.
--sortr namereverses the sort and therefore which arm runs first; a published ratiois the mean of one run each way.
There is no
encodearm for the SDK: the oracle shim wraps the SDK's reader, not its writer, sono reference encode number exists and none is fabricated. Encode is reported for gamut alone.
What is inside each timed region, and what is not
Inside, every benchmark: the codec call, the allocation and growth of the buffer it produces,
and that buffer's teardown. The teardown is placed explicitly (each closure returns
()and dropsits result) because divan otherwise defers a returned value's drop past the timed region — which
would charge gamut nothing for freeing a decoded image while the SDK, whose
dng_negativedestructor runs inside its own call, pays in full.
Outside, every benchmark: synthesising the sensor samples, building the
RawImageandCameraProfile, and the encode that produces the bytes a decode benchmark reads. No benchmarktouches the filesystem.
Is the gamut-versus-SDK comparison fair?
Every asymmetry is either removed or measured; none is left as an adjective. Which
direction each one favours is deliberately not claimed. An earlier revision of this section said
decode_dngfavours the SDK anddecode_lossless_jpegfavours gamut "by less"; the second iswithdrawn — the export-path gap that would justify it sits at this harness's measurement floor
and its sign is not resolved (see
## Fourth pass, N5).decode_dng_in_memoryhands the SDK adng_streamover the caller's own bytes, so the reference implementation parses the very buffergamut parses: no temporary file, no import copy on either side.
reports the decoded image's extent and exports no samples, so the SDK is not charged for a
malloc+memcpythat exists only because the caller is in Rust.decode_dngis the IFD-0 preview, plus the metadatareconstruction:
DngDecoder::decodeis a whole-file decode andReadStage1Imageis not. Itsvolume is exact, and it is the volume the decoder materialises, not the one the file stores —
the preview is written at 8 bits, but every sub-image surfaces as
SubImageData::Decoded(Vec<u16>),so the buffer gamut allocates, fills and frees is
⌊w/2⌋ × ⌊h/2⌋ × 3 × 2bytes against the raw'sw × h × planes × 2: 75 % of a 16-bit CFA frame and 25 % of aLinearRawone. (An earlierrevision of this section modelled it at the stored width and so halved both figures; corrected in
## Fourth pass, N4.) The fixture table prints that volume per case together with whether thecorrection was applied — it is applied on the uncompressed rows, where both paths do comparable
work per byte, and suppressed on the compressed rows, where the same arithmetic would print a
lower bound where a reader takes a measurement. It is not normalised away: gamut exposes no
raw-image-only decode entry point, and adding one so a benchmark reads better would be the wrong
direction of causation.
decode_lossless_jpegis the FFI export path, and a third arm boundsit.
adobe-sdk-no-exportruns the identicalDecodeLosslessJPEG<Scalar>into the identicalspool buffer and stops before the
malloc/memcpy/Veccopies. Read that gap as a magnitudeonly: it sits at the measurement floor, so what it supports is "the codestream comparison is fair
to within the bound", not a price for the export path.
*/deflaterows neither arm's inflate is gamut-authored.gamut-deflateisdeliberately encoder-only, so this crate inflates with
miniz_oxide, while the reference armcalls the system libz the oracle links dynamically (
-lz, because the SDK includes<zlib.h>unconditionally). What separates the two is pinning, not authorship:
miniz_oxideis pinnedby
Cargo.lockto one version and one checksum, and the system libz is pinned by nothing — not bya version, since the loader picks between a copy under
target/and whatever the platforminstalled and
zlibVersion()separates those only when the platform's build renamed itself, and not even by themachine, since
cargoexports every build script's native search path and this crate dev-dependson
libtiff-oracle, which builds alibz.soof its own undertarget/. The harness prints theresolved library and warns when its path lies inside a build directory. Pinning it is filed as
tooling/gamut-dng-oracle: pin the zlib the benchmark's reference arm links, keeping -lz for conformance #618.
Numbers
None are pinned in the repository. The measured run is
## Fourth pass— eight runs across twolibz builds and both arm orders, with raw divan output and per-run load averages. This box is
shared with other concurrent sessions; a throughput figure taken under load is not a throughput
figure, so absolute MB/s is reported as indicative only and
STATUS.mdtells a reader to run theharness on the box they care about. The #196 ratios already in
STATUS.mdare unaffected — theycompare two encoders inside one process, which survives a loaded machine; MB/s does not.
Validation
Commands verbatim, with outcome. Correcting two claims this paragraph made in earlier passes.
It said nothing in the gate set executes this file: that is false.
mise run bench-testruns everybench in the workspace once, without timings, precisely so a bench cannot rot silently (issue
#437), and it is in this lane's gate set — its run is recorded below. What no gate does is time
the harness, which is deliberate: a timing number from a shared runner would be noise. It also said
the one piece of the diff a gate executes is the oracle's new entry point: that was false too, for
the reason under N2 below — the oracle crate is outside the workspace, so its tests never ran. Both
of that entry point's pins now live inside
gamut-dng, wheremise run testruns them. Nothingunder
.github/is touched.Commands verbatim, with outcome, are recorded per pass:
## Second pass,## Third pass,## Fourth pass,## Fifth passand## Sixth passeach carry aValidation, this passtable.The gate set run on the current head is this one (
## Sixth pass; the diff is two commits ofcomments, doc comments, a printed string and a
STATUS.mdparagraph — no source behaviour changes):__CARGO_TEST_ROOT=$(git rev-parse --show-toplevel) mise run fmt, then the same prefix formise run fmt-checkfmtchanged nothingmise run check-testsconvco check origin/feat/442-dng-c2pa-manifest-store..HEADmise run bench-testmise run lintcargo test -p gamut-dng --all-featuresGAMUT_MUTANTS_BASE=origin/feat/442-dng-c2pa-manifest-store mise run mutants-diff__CARGO_TEST_ROOTis set because cargo otherwise walks past this nested worktree's root whenloading the
tooling/*manifests andfmt-tooling-checkexits 101 on an untouched tree; it changesnothing about what is formatted. The mutation base is stated beside the count because this branch is
stacked and the default base folds in the mutants of the pull request underneath.
mise run check-release-depsandmise run check-ffi-featureswere not re-run: no manifest changedthis pass, and both passed on the manifest change recorded in
## Third pass. No timing ispublished in this pass — the measured run remains
## Fourth pass.Risks and rollout
dev-only oracle entry point (
tooling/**, never published), and documentation. No newdependency.
--bench codeclinks the Adobe SDK oracle, whichcargo test -p gamut-dng --all-featuresalready builds, somise run test/lintgain a link target and no new nativebuild.
mitigations are structural rather than promissory: the asymmetry that cannot be removed is
printed by the harness itself next to the numbers it distorts, and the second comparison is
biased the other way so a reader is never left with one flattering ratio.
benches/codec.rs, its[[bench]]stanza, the STATUS section and theoracle entry point; nothing depends on them.
Issue
Refs #163. NotCloses: the issue asks to "still keep it comprehensive", and two axes of theformat are measurable but not measured here — JPEG XL (encoding it needs the
jxl-encodefeatureand a C++ toolchain, so a default
cargo benchcannot build its fixture) and the real-camera corpusbehind
mise run fetch-dng-samples(a ~178 MiB submodule, which would make the harness unrunnableby default — rejected in the decision record below). Remainder filed as #556.
Two further issues are filed by the fifth pass and are not closed by this PR: #617 corrects the record in #583 and #584 (withdrawn figures, and a verification command that matches no benchmark), which this run may not edit in place; #618 proposes pinning the zlib the oracle's reference arm links, keeping
-lzfor conformance. The sixth pass files a third, #620, correcting one sentence of #618's justification:zlibVersion()does separate this box's two libz builds, and the claim that holds is the narrower one about two stock builds of a version. #618's proposal is unaffected.Decisions taken
No human approved this plan. This is an unattended run; the record below is what a human reads
afterwards. The frozen record this lane was given, verbatim:
Appended by this lane, in the same shape:
Second pass: the table fix, and what the harness found
The harness's first full run left one thing unfinished and turned up two defects. Neither defect is
fixed here — the record's "will not" forbids changing the codec, and a benchmark is the wrong place
to change what it measures — so both are filed with their evidence.
The fix that landed (
a589d61). The fixture table's first column was ragged:{case:<26}setsa width on the formatter, but both
Displayimpls wrote throughwrite_str/write!, which gostraight to the underlying buffer and discard width and alignment.
Formatter::padis the methodthat honours them. Cosmetic; it changes no measurement, and the fixture byte counts are identical
before and after.
#583 — lossless-JPEG decode is ~60x slower than the reference implementation. Medians of 100
samples, 512x384 at 16 bits:
decode_lossless_jpegcfadecode_lossless_jpeglinear-rawdecode_dngcfa/deflatedecode_dnglinear-raw/deflateThat every other scheme is within 1.25x and only this one is 60x out is what localises the cost:
lossless_jpeg::decode_symbolscans the whole 256-entry code table once per candidate bit length,so a symbol costs ~1000 comparisons against the reference implementation's single table probe.
#584 — CFA lossless-JPEG encode expands the payload past uncompressed. From the fixture table
the harness prints:
cfa/lossless-jpegis 618 800 bytes againstcfa/uncompressed's 541 440 —turning compression on makes the file 14.3 % larger. The encoder hands the mosaic to
lossless_jpeg::encodeas one full-width component, so predictor 1 differences a red photositeagainst its green neighbour. Declaring the same samples as
(width / 2, height, 2)— the reshapeDNG 1.7.1.0 p. 20 describes, which needs no sample reordering and which this crate's decoder
already reads — takes the payload from 119.7 % of raw to 91.5 %, measured through the crate's own
public
lossless_jpeg::encode.STATUS.mdrecords both (3ec14d6), as ratios and byte counts rather than absolute times.On the numbers, and the box that produced them
This machine is shared and was under heavy concurrent load for most of this session. The
measurement above was taken in a quiet window —
uptimeat the start of the run reported a 1-minuteload average of 7.65 on 16 cores, and 9.60 at the end — and that
uptimebracket isrecorded in the run log rather than inferred. Even so: no absolute MB/s figure from this run is
published anywhere in the repository, because a throughput figure taken on a shared box is not a
throughput figure. What is published is (a) ratios between two implementations measured in the same
process, which reproduced at 57-71x across two runs an order of magnitude apart in load, and
(b) fixture byte counts, which are deterministic.
Fairness of the comparison, re-established this pass
decode_dng_*both parse the same in-memoryVec<u8>.DecodeLosslessJPEG<Scalar>, the referenceimplementation's non-vectorised path, so the 60x is scalar against scalar.
the codestream pair favours gamut) and they still point in opposite directions.
DngDecoder::decodedoes not verifyNewRawImageDigest— that is a separate opt-in method —so gamut is not being charged for hidden work in the timed region.
Validation, this pass
All from the worktree, all completed in this run:
mise run fmt-check(with__CARGO_TEST_ROOT)mise run check-testsconvco check <base>..HEADmise run lintmise run testmise run check-release-depsmise run check-ffi-featuresmise run mutants-diffcargo bench -p gamut-dng --bench codecis not a gate: benchmarks are not tests, and no CI checkruns this file. What does cover it is
mise run lintandmise run test, both of which build--all-targetsand so compile the bench under-D warnings; nothing asserts on its output.Decisions appended to the record, in the record's shape
This is an unattended run: no human approved this plan or these decisions. The record above is
what a human reads afterwards.
Third pass: the table published in full, and each pair measured against itself
A review of the second pass raised eight findings. All eight are addressed below. Two of them
changed what the harness measures, and one of those changed a published conclusion. Corrections are
appended rather than written over the entry they correct, and each names its entry.
F4 (Medium) — each pair is now one benchmark, and that moved every small ratio
benches/codec.rs:353and:373put gamut and the reference implementation in separate divanbenchmarks. Divan runs benchmarks in name order, so every reference case was measured minutes away
from its counterpart on a shared box that drifts, and every conclusion at the 1.25x scale sat
inside its own noise.
decode_dnganddecode_lossless_jpegnow take the implementation as a benchmark argument,with the argument names ordered so divan's own name sort keeps the members of a pair adjacent. The
pair runs back to back, under the same instantaneous load. This repository's position is that
ratios are the durable quantity; a ratio measured minutes apart on a shared box is not one.
Correction, naming its entry. The second pass's table (this body, "#583 — lossless-JPEG decode
is ~60x slower than the reference implementation") reported
decode_dng cfa/deflateat 1.24xand
linear-raw/deflateat 1.25x. Interleaved, both sit below 1.0x — 0.94-0.97x across tworuns, i.e. gamut decodes Deflate DNGs slightly faster than the reference implementation. Those
two figures were the split-group measurement moving, not the codec. The ~60x lossless-JPEG result
is unaffected and reproduces.
F1 (High) — the isolating sentence was false, and the rows that falsify it were missing
crates/gamut-dng/STATUS.md:397claimed every decode path was within 1.25x of the SDK exceptlossless JPEG. It is not true — both uncompressed cases are well outside it — and those two rows
appeared in no table published anywhere. Omitting the contradicting rows was the defect, not the
ratio. The whole matrix is now published, in this body and in
STATUS.md.Whole-file decode, gamut / Adobe DNG SDK, median of 100 samples, 512x384 at 16 bits. "Corrected"
divides out the IFD-0 preview volume gamut additionally unpacks:
decode_dngcasecfa/uncompressedcfa/deflatecfa/lossless-jpeglinear-raw/uncompressedlinear-raw/deflatelinear-raw/lossless-jpegBare codestream decode — same SOF3 stream in, same samples out, no container work on either side,
one counter for all three arms:
decode_lossless_jpegcasecfalinear-rawCorrection, naming its entry. The
STATUS.mdsentence "Every decode path is within 1.25x ofthe SDK except lossless JPEG, which is ~60x slower" (second pass, commit
3ec14d6) is withdrawn.#583's isolation now rests on the codestream pair, which carries no container asymmetry at all
and whose one residual bias is measured at under 2 %: gamut is 56-59x the reference implementation
there, in both runs, on both photometries. The whole-file rows are published in full beside it. The
preview correction explains part of the uncompressed gap (2.4x -> 1.7-1.8x, 1.8x -> 1.6x); the rest
is the fixed IFD and metadata reconstruction, which does not scale with the frame and so dominates
exactly where the raw path is little more than a
memcpy. The harness measures that gap and doesnot attribute it further — and #583 does not depend on it.
Issues #583 and #584 carry the same incomplete table in their bodies. This run does not edit an
existing issue, so the corrected table lives here and in
crates/gamut-dng/STATUS.md, which is thedurable record that gets cited.
The preview-corrected ratio is now printed, so no reader does the subtraction. The harness
applies one counter rule: every benchmark's divan counter is the pixel volume that implementation
actually moves — the raw sample volume for every SDK arm and for gamut's codestream decode, and the
raw volume plus the IFD-0 preview for gamut's whole-file decode and for gamut's encode, both of
which handle the preview too. In
decode_dngthe median-time column is therefore the uncorrectedratio and the throughput column the preview-corrected one. The correction charges preview bytes
at the raw path's per-byte rate: close to exact on the uncompressed rows (both paths just move
bytes) and generous to gamut on the compressed ones, where the corrected figure is a lower bound.
The fixture table prints the raw volume, the preview volume and the resulting factor per case.
Design question 6 — the fairness framing, replaced by what was measured
The second pass called one comparison "the SDK is favoured" and the other "gamut is favoured", both
unquantified. Neither adjective survives:
decode_dng_in_memoryhands the SDK adng_streamover the caller's own bytes — the same buffer gamut parses. No temporary file, noimport copy, on either side.
reports the decoded image's extent and exports no samples.
adobe-sdk-no-export, runs the identicalDecodeLosslessJPEG<Scalar>into the identical spoolbuffer and stops before the
malloc/memcpy/Veccopies. Measured, that path costs thereference implementation 1.9 % / 0.3 % (cfa) and 1.1 % / 0.7 % (linear-raw) across the two
runs — so the codestream comparison is fair to within 2 %, a number rather than a claim. It
required one new dev-only oracle entry point (
decode_lossless_jpeg_extent) plus its C++ shim,pinned by a differential test that the two entry points reach the same decode.
F2 (Medium) — "no CI check runs this file" was false
Correction, naming its entry. The second pass's Validation section said "
cargo bench -p gamut-dng --bench codecis not a gate: benchmarks are not tests, and no CI check runs this file."The second clause is wrong.
.github/workflows/extended.ymldefines abenchesjob ("Benchesstill run") whose final step is
mise run bench-test(mise.toml:243-245,cargo bench --workspace --benches -- --test), which exists precisely to catch a bench that builds and then panics insetup. The
[[bench]]stanza this PR adds moves this file into that job's scope. Correctedstatement: this bench is not one of the four required per-PR checks, but it is run once per
extended-lane CI run, and that gate was not listed or executed in the second pass. It is run and
listed below:
mise run bench-test, exit 0, all 24 cases execute.F3 (Medium) — one denominator, and it is named
Correction, naming its entry.
STATUS.md:403-407(second pass) quotedcfa/lossless-jpegas"157.4 % of the raw samples against 137.7 %" — whole encoded files — and then, in the same bullet,
the remedy as "119.7 % to 91.5 %" — bare codestreams. Read in sequence that inflates the remedy
about fivefold. Restated against one denominator, the raw sample volume (393 216 bytes for this
fixture):
cfa/uncompressedwrites 541 440 bytes, 137.7 % of raw.cfa/lossless-jpegwrites 618 800 bytes, 157.4 % — turning compression on makes the file14.3 % larger.
148 224 bytes of preview and directory, so the reshaped file is 508 112 bytes, 129.2 % of raw:
6.2 % below the uncompressed baseline, not ~33 %.
Re-measured this pass through the crate's own public
lossless_jpeg::encode; the byte countsreproduce exactly.
F5 (Low-Medium) — the oracle test now names what it asserts
in_memory_decode_reaches_the_same_image_as_the_file_decodecompared extents, not pixels. Theentry point deliberately carries no pixels, so the name was the defect, not the assertion. Renamed
to
in_memory_decode_reports_the_same_extent_as_the_file_decode, with the reason in its doccomment.
F6 (Low) — the commit range
Correction, naming its entry. The second pass's summary sent a reviewer to "the three commits
above
677d38ea". This PR's own diff is the ten commits above677d38ea(
git log --oneline 677d38ea..HEAD). The two the second pass skipped are83461e7(fix(dng): give the benchmark's blue plane blue's gain) and
a589d61(style(dng): pad thebenchmark's case column through the formatter) — the ones that changed what is measured and that
carry the fixture byte counts quoted above. This pass adds three more.
F7 (Low) — the load factor
Correction, naming its entry. The second pass wrote that the ~60x result "reproduced at 57-71x
across two runs an order of magnitude apart in load". "An order of magnitude" overstates the
difference by roughly 2.5x; the two runs differed by about a factor of four in one-minute load
average. The claim that survives is narrower and is the one made here: the result reproduced across
two runs taken in the same quiet window, at 56-60x.
F8 (Low) — the fixture table is not promoted to a codec gate
#584's verification section proposes pinning the encoder to the sizes this harness prints. It
should not be, and
STATUS.mdnow records why in-repo: the margin is a property of frame-uniformsynthetic gains — one gain per CFA colour across the entire frame, which is what makes the
interleaved-component reshape win so cleanly — and pinning an encoder requirement to a single
synthetic fixture is exactly the failure this harness exists to avoid. The recommendation recorded
in-repo is to re-measure on the real-camera corpus (
mise run fetch-dng-samples, thenmise run test-dng-real) before the encoder changes, and to gate on that if anything is gated. This run doesnot edit #584.
Measurement discipline
Both runs above were taken in a quiet window on a shared machine, each bracketed by
uptime(theone-minute load average, 16 cores):
--testordering check--sample-count 100)--sample-count 100)No absolute throughput figure from either run is published, here or in the repository: on a
shared box a MB/s figure is a property of the box. What is published is same-process ratios and
deterministic byte counts.
Validation, this pass
All from the worktree, all completed in this run. Workspace-wide gates ran inside a 16 GiB
memory-capped systemd scope with
CARGO_BUILD_JOBS=2.cargo test -p gamut-dng --all-features --lib sdk_differentialcargo test -p gamut-dng-oraclecargo clippy -p gamut-dng -p gamut-dng-oracle --all-targets --all-features -- -D warningsmise run fmtthenmise run fmt-check(with__CARGO_TEST_ROOT)mise run check-testsconvco check 677d38ea..HEADmise run lintmise run testmise run bench-testmise run check-release-depsmise run check-ffi-featuresmise run mutants-diffWhat these gates cover and what they do not:
mise run testandmise run lintcover the neworacle entry point (
decode_lossless_jpeg_extent, via the differential test that it reaches thesame decode as the exporting entry) and compile the bench under
-D warnings.mise run bench-testexecutes every case of the bench once, which is what catches a fixture or dimensionmistake. Nothing asserts on the bench's timings, by design — a timing assertion on a shared box
is a flaky test — so the ratios above are evidence from a recorded run, not a gate.
mise run coveragewas not run: no new module was added with low test reach.Decisions appended to the record, in the record's shape
This is an unattended run: no human approved this plan, these decisions, or these corrections. The
record above is what a human reads afterwards.
Correction to F6, appended
F6 above says this PR's own diff is "the ten commits above
677d38ea". One further commit(docs(dng): describe the benchmark's counter rule where it is stated) landed after that sentence
was written, so the count is now eleven. The durable form is the command, not the number:
git log --oneline 677d38ea..HEAD. The two commits the second pass skipped are unchanged:83461e7anda589d61.Fourth pass: the disputed rows reproduce — both of them, and here is why
An independent re-measurement could not reproduce two third-pass ratios:
cfa/deflateandlinear-raw/deflate, published at 0.94–0.97 (gamut faster), re-measured eighteen times at1.20–1.26 (the SDK faster), flat in load from 2.8 to 49, ±2 %, on identical fixture bytes. Its
argument was that if interleaved measurement is stable to ±2 %, 0.95 and 1.24 cannot both be
measurements of the same thing on the same box.
That argument is correct, and it is what finds the defect. They are not measurements of the
same thing. Both figures reproduce here on demand, and what selects between them is which
libzthe reference arm called.
tooling/gamut-dng-oracle/build.rslinks the system zlib dynamically (-lz), because the SDKincludes
<zlib.h>unconditionally. So on the two*/deflaterows — and on no other row — thereference arm runs code not built from source committed here. Worse, which libz is not even a
property of the machine alone: cargo puts every build script's native search path on
LD_LIBRARY_PATH, so the loader picks whichever stock zlib some other dev oracle built undertarget/, while running the same binary directly picks the platform's. Here those are stock zlib1.3.1 and zlib-ng 2.3.3 — and
mise run bench-testresolves a third copy again (libpng-oracle's),because the set of oracles in the graph differs.
The measurement
Eight runs of
decode_dng+decode_lossless_jpeg, 100 samples each, 512×384 at 16 bits: tworepetitions of
{stock zlib 1.3.1, zlib-ng 2.3.3} × {reference arm first, gamut arm first}, eachbracketed by
uptime. One-minute load 15.5 → 38.1; the box did not go quiet in the window and isreported as it was. Every ratio below is derived from the raw output reproduced further down —
nothing here is published alone.
decode_dng, gamut ÷ Adobe DNG SDK, median time; the four cells per column are{rep 1, rep 2} × {reference first, gamut first}:cfa/uncompressedcfa/deflatecfa/lossless-jpeglinear-raw/uncompressedlinear-raw/deflatelinear-raw/lossless-jpegThe parenthesised
cfacells are the noisiest run (load 36.8); its fastest-sample ratios are 0.97and 27, in line with the rest.
Only one arm moves — that is the isolating evidence.
cfa/deflatemedians from the dedicatedA/B (three repetitions back to back at a fixed load of 13.2 → 12.3, raw output below):
gamutadobe-sdkgamut's arm is the control and does not move; the reference arm moves by 1.2–1.3×, exactly enough
to carry the ratio across 1.0. Neither implementation changed; the harness never said which inflate
it had measured. It now prints that, above its own output.
Answering the ±2 % argument directly. The stability claim holds, and this round corroborates it
from the other side: within a library the Deflate ratio is stable to ±2 % across loads 15–38 and
both arm orders (0.94, 0.94, 0.94 stock; 1.23, 1.25, 1.26 zlib-ng). That stability is what makes the
inference sound, and why the explanation had to be a difference in what was measured rather than
in how well — and why waiting for a quieter box was never going to settle it.
Withdrawn, and standing
1.20–1.26 is the ratio against a zlib-ng-class libz, and this round reproduces both. Both are
published, each with its library.
magnitude and sign as the libz effect, and the non-interleaved arrangement was not re-run under a
pinned library. Interleaving is kept on its own argument — a ratio measured minutes apart on a
drifting machine is not a ratio — and is no longer offered as an explanation of that shift.
JPEG.
ratio is below 34×; medians centre near 50–60×. The precision is withdrawn, the finding is not
near parity under any measurement here, and its measured path contains no code from outside this
repository — the SDK's lossless-JPEG decoder is built here from the committed SDK source — so
unlike the Deflate rows it does not depend on what the machine has installed.
lossless-JPEG.
though they do not fit a tidy story.
N2 — the pin now sits where a gate runs it
tooling/gamut-dng-oracleis under[workspace].exclude, so a#[cfg(test)]test in it is nevercompiled by
cargo test --workspace, and the one pinningdecode_dng_in_memorynever ran. Itssibling
decode_lossless_jpeg_extentwas already pinned insidegamut-dng, which makes theomission an inconsistency rather than a judgement call. Moved to
crates/gamut-dng/tests/roundtrip.rsasadobe_in_memory_decode_matches_the_file_decode: encode aDNG with this crate, decode it both ways, assert the memory-stream entry reports the extent the
exporting one produces and that this is the encoded image's own extent. Both oracle entry points now
carry a doc line naming where their pin lives and why it is not beside them. The false claim under
## Validationis corrected in place, above.N3 — one flow, not two
gdng_decode_dng_in_memoryrepeated the seven-call SDK sequence (Parse→PostParse→IsValidDNG→Make_dng_negative→Parse→PostParse→ReadStage1Image) thatread_negativealready ran for the file path.read_negativenow takes adng_stream &and thepath form is a two-line overload that opens the file and delegates — so opening the stream is the
whole of the difference between the two flows, which is exactly what the fairness claim asserts.
N4 — the correction is suppressed where the model is a bound
The preview is now modelled at the width the decoder materialises: it is stored at 8 bits, but
every sub-image surfaces as
SubImageData::Decoded(Vec<u16>)whatever the stored depth, so thebuffer gamut allocates, fills and frees is twice the stored size. The model under-subtracted by
half; it reads
⌊w/2⌋ × ⌊h/2⌋ × 3 × 2.And the correction is applied only on the uncompressed rows, where both paths unpack and store
at comparable cost per byte, and suppressed on the compressed ones, where a raw byte carries
entropy-coding work a preview byte does not and the arithmetic yields a lower bound dressed as a
measurement. There gamut's counter is the raw volume and both divan columns are uncorrected. The
fixture table gained a
correctioncolumn readingappliedorSUPPRESSEDper case, and theterminal epilogue printed on every run states why, and what a reader should take a compressed row
to mean instead.
N5 — the bound, not the sign
Across sixteen case-runs the gap between
adobe-sdkandadobe-sdk-no-exportspans −4 % to+64 %: three above 30 %, two negative, the rest 0.2–3.1 %. An effect below this harness's
run-to-run spread, whose sign is not resolved — exactly as the review said. What the fairness claim
needs is a bound (the export path cannot account for a 30×-plus ratio), and that is what is now
published. 0.3–1.9 % is withdrawn: two samples of a quantity at the noise floor.
N7 — the guard its sibling has
gdng_decode_lossless_jpeg_extentnarrowed itssize_tlength to theuint32dng_streamtakeswithout checking it fits;
gdng_decode_dng_in_memoryguards the same narrowing. Added.Decision 8 — alternating the arm order
divan cannot interleave a pair per sample, so one arm always runs first and the bias points one way
for a whole run. divan's sort is already reversible, so no machinery was invented:
--sortr nameruns the gamut arm first (verified from the listing — pair members stay adjacent, only their order
flips), the module header and the printed epilogue both instruct an operator to take one run each
way and publish the mean, and the eight runs above are four of each. Across them the order is worth
about a percent, well under the run-to-run spread; it is corrected for because it is
one-directional, not because it is large.
Validation, this pass
__CARGO_TEST_ROOT=$(git rev-parse --show-toplevel) mise run fmt-checkmise run check-testsconvco check origin/feat/442-dng-c2pa-manifest-store..HEADmise run lintcargo clippy --workspace --all-targets --all-features -- -D warningsmise run testtest result: ok, 0 failedmise run bench-testGAMUT_MUTANTS_BASE=origin/feat/442-dng-c2pa-manifest-store mise run mutants-diff#[cfg(test)]modulemise run check-release-depsmise run check-ffi-featurescargo test -p gamut-dng --all-features --test roundtrip adobe_in_memorycargo test -p gamut-dng --all-features --lib sdk_extentWorkspace-wide gates ran inside a
systemd-run --user --scope -p MemoryMax=16Gscope withCARGO_BUILD_JOBS=2andulimit -v.fmt-checkneeds__CARGO_TEST_ROOTin a nested worktree —an environment artefact, not a manifest problem.
Raw output
Every ratio in this pass is derived from the output below. Three edits, all stated: the throughput
(MB/s) columns are dropped (this run publishes no absolute throughput figure), the constant
samples/iterscolumns are dropped (100 for every row), and the column padding is collapsed —all three because the body would otherwise exceed GitHub's 65 536-character limit. Every timing,
load average and library identity is verbatim.
Eight runs: {stock zlib, zlib-ng} × {reference first, gamut first} × 2 reps
The isolating A/B: {zlib-ng, stock zlib} back to back, three reps, Deflate rows
Decisions appended to the record, in the record's shape
Taken — the harness reports the libz its reference arm called, and does not pin it.
The SDK includes
<zlib.h>unconditionally andbuild.rslinks-lz; vendoring a zlib into theoracle would change what the oracle is — a build of the reference implementation as the platform
builds it — to make one benchmark row tidier.
zlib_identity()returnszlibVersion()plus thepath
dladdrreports, because zlib-ng's compatibility build answers"1.3.1"exactly as stockzlib does and the version alone does not discriminate.
Rejected with evidence: vendoring a zlib (changes the reference implementation to suit a
measurement); publishing one library's Deflate figure as the figure (that is the defect being
repaired).
Reverses: nothing.
Taken — the preview correction is applied only where it is a measurement.
Charging preview bytes at the raw path's per-byte rate is sound where both paths unpack and store,
unsound where the raw path also entropy-decodes. On the latter the harness prints no corrected
number and says why, at the fixture table and at the terminal epilogue.
Rejected with evidence: keeping the correction everywhere and labelling it a lower bound in prose
— a printed number is read as measured, and the two prose sites were not where an operator reads.
Reverses: the third pass's counter rule, which corrected every row.
Taken — the arm order is alternated with divan's own reversible sort, not new machinery.
--sortr nameflips the arm order within every pair while keeping them adjacent; verified from thelisting. A published ratio is the mean of one run each way, stated in the module header and the
printed epilogue.
Rejected with evidence: an env var or a rank prefix in the argument names — both invent a control
divan already has, and the prefix would change the row labels between the two runs a reader is
meant to compare. Reverses: nothing.
Taken — record the withdrawn attributions rather than quietly replacing them.
STATUS.mdnames which claims are withdrawn and what replaced them; a benchmark section whosehistory is invisible is how a superseded number gets cited.
Rejected with evidence: silently rewriting the tables. Reverses: nothing.
This is an unattended run: no human approved this plan, these decisions, or these corrections. The
record above is what a human reads afterwards.
Fifth pass: where the correction did not reach
The fourth pass's mechanism was re-established independently, and tighter than this body stated:
the interfering build is
libtiff-oracle, a direct dev-dependency ofgamut-dngitself, socargo bench -p gamut-dngalone is enough to flip which zlib the reference arm calls — no unrelatedcrate need be involved. Reproduced again this pass:
cargo bench -p gamut-dng --bench codecprints1.3.1 from …/target/release/build/libtiff-oracle-*/out/zlib-prefix/lib/libz.so.1.3.1, while theplatform's
/usr/lib64/libz.so.1islibz.so.1.3.1.zlib-ng(zlib-ng-compat-2.3.3-3.fc44), andmise run bench-testresolves a third copy again (libpng-oracle's). No shipped code neededchanging for that. What follows is the four places the correction had not reached.
F3 — name what each arm actually runs
On the disputed rows neither arm's inflate is gamut-authored:
gamut-deflateis deliberatelyencoder-only, so this crate inflates with
miniz_oxide. The module header, the printed epilogue andthe
STATUS.mdsection all said "one measured path is not built from this repository" and"every other row runs only code built here", which a reader takes as a claim about gamut's own
codec. Corrected in all three. The distinction that carries the section's real content is now the
one stated:
miniz_oxideis pinned byCargo.lockto one version and one checksum, so every run ofthis harness anywhere inflates with the same code; the system libz is pinned by nothing — not by a
version, since zlib-ng answers
zlibVersion()with stock zlib's own string, and not by the machine.Decision 5 (design question 2) — the harness flags the accident, not just the identity
Printing the resolved library makes a Deflate ratio interpretable; it does not make it
reproducible.
print_zlib_identitynow warns when the resolved path has atargetcomponent,because a resolution that came out of the build graph belongs to that build graph and to nobody
else. The path is split out of the identity string into
gamut_dng_oracle::zlib_pathso a callercan test it rather than parse a printed line. Observed firing on both
cargo bench -p gamut-dng --bench codecandmise run bench-test— on two different oracles' copies of zlib, which is itselfthe point.
F4 — the guard is symmetric now, and its rationale had named the wrong sibling
gdng_decode_lossless_jpeg_extentrejects asize_tlength that does not fit theuint32dng_streamtakes; the exporting arm it is timed against,gdng_decode_lossless_jpeg, narrowedthe same value unguarded. The guard's rationale cited
gdng_decode_dng_in_memoryas the sibling tomatch — the wrong one. Both arms carry the identical check now, and both sites state why: these two
entry points are subtracted from each other, so a check one runs and the other does not is a
difference inside the measured region as well as a difference in what each accepts. Unreachable from
this crate's fixtures either way; the streams are kilobytes.
F1 / decision 2 — the two filed issues, corrected by a third
#583 and #584 are what someone will actually act on, and both still carry withdrawn material. This
run may not edit, comment on, label or close an existing issue under any authority, so the
correction is filed as #617, which states per issue which figure is withdrawn and what replaced
it:
them — "that every other scheme lands within 1.25x and only this one is 60x out" — while the
rows that falsify that sentence appear nowhere in it:
decode_dng's uncompressed pair is2.1–2.5× and 1.7–1.8× uncorrected. The replacement argument is the codestream pair, which needs no
other row. Its Verification section instructs
decode_lossless_jpeg_gamutagainstdecode_lossless_jpeg_adobe_sdk, which matches zero benchmarks since this PR merged the armsinto one benchmark taking the implementation as an argument; gamut-dng: correct the record in #583 and #584 — withdrawn benchmark figures and a verification command that matches no benchmark #617 gives the working command.
doubled, and mixes whole-file percentages with codestream percentages in a way that yields a ~33 %
file saving that does not exist — restated against one denominator it is 6.2 %.
Both findings themselves stand, and neither issue is edited by this run.
STATUS.mdnow points at#617, and at #618 for the linkage question.
Decision 6 (design question 1) — the row stays, the linkage question is filed
Pinning a zlib for benchmarking while keeping
-lzfor conformance is the right idea, and a realbuild-system change to a crate every
gamut-dngtest links; taking it here would widen this round'sblast radius across the run. The Deflate row keeps its place with the accurate label F3 gives it,
and the proposal is filed as #618 with its evidence.
The mutation counts published in earlier passes were taken against the wrong base
## Third passand## Fourth passboth reportmise run mutants-diffas "78 mutants, 69 caught,9 unviable, 0 missed". That selection used the runner's default base,
origin/master, and thisbranch is stacked — so it folded in every mutant belonging to #508 underneath. Re-run this pass
with the correct base,
GAMUT_MUTANTS_BASE=origin/feat/442-dng-c2pa-manifest-store, the selectionis 0 mutants, which is the honest answer for this diff: its only
src/change lives inside a#[cfg(test)] mod testsblock, bench targets produce no mutants, andtooling/**is excluded by.cargo/mutants.toml. The tool is working —cargo mutants -p gamut-dng --listlists 1964 mutantsin the crate. The 78/69/9/0 figures are withdrawn as evidence about this pull request. Nothing
this PR claims rested on them.
Validation, this pass
__CARGO_TEST_ROOT=$(git rev-parse --show-toplevel) mise run fmt, then the same prefix formise run fmt-checkmise run check-testsconvco check origin/feat/442-dng-c2pa-manifest-store..HEADcargo test -p gamut-dng --all-featuresmise run lintmise run bench-testGAMUT_MUTANTS_BASE=origin/feat/442-dng-c2pa-manifest-store mise run mutants-diffcargo bench -p gamut-dng --bench codec -- --testOne-minute load averages bracketing the runs: 27.3 → 33.9, on a shared box. No timing claim is
made in this pass — no ratio is published here, so the load bounds what a re-measurement would
have cost rather than what this pass concluded; the measured run remains
## Fourth pass.mise run check-release-depsandmise run check-ffi-featureswere not re-run: no manifest changedthis pass, and both passed on the manifest change recorded in
## Third pass.Decisions appended to the record, in the record's shape
Taken — correct
## Summaryin place rather than appending a fifth correction to it. Theappend-only convention protects the decision record, not the summary; the summary is what a reader
meets first, and it still carried the pre-N4 preview model (halved), two withdrawn adjectives, and a
description of a benchmark layout that no longer exists. It now describes what the harness runs, and
carries a forward pointer to all five passes.
Rejected with evidence: appending a fifth "what the summary should now say" section — four such
sections already exist, and the finding is precisely that a reader does not reach them.
Reverses: restore the superseded text from this PR's edit history.
Taken — file the record correction as a new issue rather than editing #583/#584. This run may
not edit, comment on, label or close an existing issue under any authority. #617 names both issues,
states which figures are withdrawn and what replaced them, gives the working verification command,
and says why the originals could not be corrected in place. It is linked from
## Issueand left asan unresolved note, because closing that loop needs a human with write access.
Rejected with evidence: leaving the issues as they stand — #583's localisation sentence is false
as written and its verification command runs nothing, so an implementer follows it into an empty
result.
Reverses: close #617 once #583 and #584 carry the corrections.
Taken — name
miniz_oxideand make pinning, not authorship, the section's claim. Saying "onemeasured path is not built from this repository" is true and misleading: it implies the other path
is gamut's codec. Neither arm's inflate is gamut-authored on those rows; what makes one of them
reproducible is
Cargo.lock.Rejected with evidence: dropping the Deflate rows instead — they are the rows a DNG user's files
actually take, and an unlabelled absence is worse than a labelled dependency.
Reverses: restore the one-path wording.
Taken — warn when the resolved libz path lies inside a build directory. The print made the row
interpretable; the warning makes its irreproducibility visible without a reader having to recognise
a
target/path. Implemented as a path test on a value the oracle now returns unformatted, not asstring-matching on a printed line.
Rejected with evidence: comparing against this build's own
target/only — the loader may resolveany build script's copy in the graph, and every one of them is equally unreproducible elsewhere.
Reverses: drop the warning.
Taken — give the exporting lossless-JPEG arm the guard its timed twin has. An asymmetric guard
is the shape a later reader mistakes for a deliberate distinction, and here the two entry points
exist to be subtracted from each other.
Rejected with evidence: removing the guard from the extent arm instead, which would restore
symmetry by deleting a correct check.
Reverses: revert the guard commit.
Taken — keep the Deflate row and file the oracle-linkage question rather than changing
-lz.Pinning the benchmark's zlib while keeping
-lzfor conformance is right, and it is a build-systemchange to a crate every
gamut-dngtest links; taking it inside a benchmark PR widens the blastradius across concurrent work.
Rejected with evidence: pinning it here — the conformance oracle's behaviour is not this PR's to
change, and the row is usable once labelled.
Reverses: implement the pinned link under #618.
Taken — publish the mutation selection against the stacked base, and withdraw the earlier
counts. A count taken against
origin/masteron a stacked branch includes the base PR's mutantsand is not evidence about this diff. The correct base selects zero, and zero is stated with its
reason rather than presented as coverage.
Rejected with evidence: re-publishing the 78/69/9/0 figures with a caveat — they measure #508 and
this PR together, so no caveat makes them evidence about this one.
Reverses: nothing.
This remains an unattended run: no human approved this plan, these decisions, or these corrections.
The record above is what a human reads afterwards.
Sixth pass: the redaction that was clobbered, and a justification that was never executed
Two findings, both about text rather than about what the harness measures. No source behaviour
changes in this pass — the diff is one commit of comments, doc comments, a printed string and a
STATUS.mdparagraph.F1 (High) — the seven raw-output lines named the tooling again, and this pass's own publish did it
The
### Raw outputblock under## Fourth passprints the harness's own zlib-identity line seventimes, once per recorded run, and each carried an absolute path whose leading components name the
tooling that produced this pull request rather than anything about the measurement. That path was
redacted on the published body out of band between rounds — and the next publish restored it,
because this lane's publish step rebuilds the body from a local draft file and overwrites
whatever is live. Any edit made to the published body is silently clobbered by the next publish; the
draft never saw the redaction. Confirmed by comparing the live body against the draft that produced
it: byte-identical apart from a trailing newline, seven occurrences of the full path in each, zero
of the redacted form.
That also refutes something the fifth round reported when it handed this pull request on — not in
this body, but in its own account of what it had changed: that everything below the Summary was
byte-identical to the previous publish. The prose was. The raw-output block was not: the seven
lines it re-emitted differed from what was live at the time, because they came from the draft.
The seven lines are now redacted to
<worktree>/target/release/build/libtiff-oracle-…, whichkeeps the whole of the fact those lines carry — that the loader resolved libz from inside a Cargo
build directory, out of
libtiff-oracle's build script, rather than from the platform — and dropsonly the prefix that identifies the machine and the tooling. The measurement is unchanged; nothing
below the redacted prefix is edited. This is declared rather than done silently, because it edits
text inside an append-only record: the replaced substring, the seven lines it appears on, and the
reason are all stated here.
The pipeline, not the habit, is what changed. The publish step now runs a check on the assembled
body before it is sent, which greps for a tooling path and exits non-zero without publishing if it
finds one, printing every offending line number. Executed both ways: it refuses on the pre-redaction
body, naming all seven lines, and passes on the body this pass publishes. A check that runs after
publishing, or a resolution to remember, would both have failed the same way the last six rounds
did.
F2 (Medium) — the stated reason for the path print is false on this box
Every artefact that explains why the harness prints the resolved libz path justified it the same
way: that a zlib-ng compatibility build answers
zlibVersion()with stock zlib's string, so theversion cannot separate the two builds. Executed here:
(
dlopen,dlsym("zlibVersion"), print;stringson both objects agrees.) The platform buildanswers
1.3.1.zlib-ngand the build-tree copy answers1.3.1, so for that pair the versionstring is sufficient — and
## Fourth pass's own raw output is labelled with exactly that split,three lines from where the claim was made. The worked example in the oracle source was worse than
unsupported: it showed a version of
1.3.1resolving to a path ending.zlib-ng, a composite thatcannot occur.
The mechanism and the print survive; only the justification was wrong. What is true of every
box is the shape, not the pair: the loader chooses between a copy some dev oracle built under
target/and whatever the platform installed, andzlibVersion()separates those two only when theplatform's build renamed itself. This box's did. A box shipping stock zlib 1.3.1 gives two
resolutions that answer identically, so the identification cannot rest on a fork choosing to rename
itself — the path tells them apart either way. That is what every site now says.
The first correction commit did not get this right either: it replaced the false claim with a
second unexecuted one — that the pair the loader collides is two stock builds — which contradicts
its own commit message three lines down.
8b34b973fixes that to the shape above. Both commits arein the diff; the second is named here rather than folded into the first, because the first is
pushed.
Corrected at every site:
crates/gamut-dng/STATUS.md(Deflate-row paragraph)zlibVersion()with"1.3.1", exactly as stock zlib does"target/copy and the platform's; the version separates them only if the platform renamed itself, which this box's did and a stock box's does notcrates/gamut-dng/STATUS.md(why the path is printed)crates/gamut-dng/benches/codec.rsmodule headerzlibVersion()with stock zlib's own string"crates/gamut-dng/benches/codec.rsprinted epilogue (FIXTURE_TABLE_EPILOGUE)tooling/gamut-dng-oracle/src/lib.rsdoc comment"1.3.1 from /usr/lib64/libz.so.1.3.1.zlib-ng", justified by the zlib-ng claim"1.3.1 from /usr/lib64/libz.so.1.3.1"— a resolution that can occur — and the corrected reasontooling/gamut-dng-oracle/src/oracle_shim.cppcomment## Summary(Deflate bullet)zlibVersion()with stock zlib's string"zlibVersion()cannot tell them apart… Only the path discriminates"Two dated entries in this body carry the superseded wording and are deliberately left standing.
## Fourth pass→Decisions appended to the recordsays "because zlib-ng's compatibility buildanswers
"1.3.1"exactly as stock zlib does and the version alone does not discriminate", and## Fifth passsays "the system libz is pinned by nothing — not by a version, since zlib-ng answerszlibVersion()with stock zlib's own string". A correction may not rewrite the text it corrects: anedit in place would leave this pass pointing at sentences that no longer say what it says they said.
Both decisions stand on the corrected reason — the path is what identifies the resolution — and
neither figure they were taken beside is affected.
The claim's origin is recorded, not hidden: it was not invented here. It was carried verbatim in
the shared contract this run's lanes read before writing anything, and it propagated into four
in-repo sites, a printed epilogue, this body and a filed issue before anyone executed
zlibVersion()against both libraries. The contract has since been corrected at its source. Thenext reader deserves to know a shared contract carried it into three artefacts before it was
checked — not as an excuse, but because "four artefacts agree" is exactly the evidence that made it
look settled.
F2's filed-issue site — a new issue, not a fold into #617
#618 carries the false sentence and this run may not edit an existing issue under any authority.
Filed as a new correction issue rather than folded into #617: #617 is itself already published,
so folding would mean editing it, which is the same prohibition. #617 corrects #583 and #584; the
new issue corrects #618. Both are linked from
## Issueand both need a human to fold them in andclose them.
F3 (Low) — a withdrawn direction claim survives in a commit message
8956ad00("perf(dng): benchmark encode and decode against the Adobe DNG SDK") says "thelossless-JPEG codestream decode favours gamut (the oracle's export path costs the SDK two extra
passes)", and
c6620b29promises to state "which of the two gamut-versus-SDK comparisons favourswhich side and by how much". Both were withdrawn by
## Fourth pass(N5): the export-path gap spans−4 % to +64 % over sixteen case-runs, so it is a bound at the measurement floor, not a signed cost.
Later commits do retract in-message —
31f01840and347c8adeeach withdraw a different claim —so the pattern exists and simply was not applied here. Both commits are pushed and this run may not
rewrite pushed history, so it is recorded in the unresolved notes with the commits named rather
than fixed.
F4 (Low, pre-existing) —
## Validationpromised commands and carried a placeholder## Validationopened with "Commands verbatim, with outcome" and then contained an unsubstitutedtemplate marker instead of any command. Substituted, with this pass's gate set; earlier passes'
commands stay under their own
Validation, this passheadings, which the section now names.Design questions 1, 2 and 3 — declined, and recorded for a human
A version-mismatch warning, a build-time recorded path to compare the run-time one against, and
printing every dynamic library the reference arm links are each reasonable, and each adds machinery
to a loop that is closing. None is taken here. All three are in
## Unresolved review notesfora human to pick up, with what each would buy.
Design question 4 — restated, not taken
This body is ~84 KB across six dated passes. Whether to collapse it into a current description plus
a short changelog is a maintainer's call, not this run's: the append-only convention is what
makes every withdrawn figure traceable, and trading it for readability is a decision about the
repository's conventions rather than about this change. Restated here so it is not lost.
Decisions appended to the record, in the record's shape
Taken — re-redact the seven raw-output paths to
<worktree>/target/…and declare the redactionrather than making it silently. The lines sit inside an append-only dated pass, so editing them
without saying so leaves a reader who diffs two revisions of this body with an unexplained change
inside a record that promises not to change. The replaced substring, the seven lines, and the reason
are stated in
## Sixth pass.Rejected with evidence: leaving the paths and noting them as a known artefact — the rule is about
what is published, and eleven bodies in this run have already carried the same prefix inside
otherwise-correct provenance notes.
Rejected with evidence: deleting the seven lines outright — they carry the fact that the loader
resolved libz out of
libtiff-oracle's build directory, which is the finding those runs exist toshow. Only the prefix is dropped.
Reverses: restore the paths.
Taken — put the check in the publish step, before the bytes are sent. The redaction was lost
because the publish step rebuilds the body from a local draft and overwrites the live one; a
redaction applied to the live body cannot survive that, and neither can a resolution to be careful.
The step now refuses to publish a body containing a tooling path and prints every offending line.
Verified in both directions on real inputs.
Rejected with evidence: checking the published body afterwards — that is what has been happening,
and it detects the sixth instance rather than preventing the seventh.
Reverses: drop the check.
Taken — correct the zlib justification at every site and state the narrower claim that holds.
Two stock builds of one version are indistinguishable by version string; a fork that renames itself
is separable and cannot be what the identification rests on. Six in-repo sites including the printed
epilogue, plus the worked example, which was a composite resolution that cannot occur.
Rejected with evidence: dropping the path print, or softening the sentence to "the version may not
be enough" — the mechanism is real and reproduces, and a hedge is not a claim an implementer can
act on.
Reverses: restore the wording.
Taken — record that the false claim was inherited from the run's shared contract. It reached
four in-repo sites, a printed epilogue, this body and a filed issue before anyone executed
zlibVersion(). Naming the source is what tells the next reader why four agreeing artefacts werenot evidence.
Rejected with evidence: correcting silently — the propagation is the more useful finding, and
without it the correction reads as one lane's slip.
Reverses: remove the origin note.
Taken — file the #618 correction as a new issue rather than folding it into #617. #617 is
already published, and this run may not edit an existing issue under any authority, including one it
filed itself; folding would be the same prohibition by another name. #617 corrects #583 and #584,
the new issue corrects #618, and both are linked from
## Issuefor a human to fold in and close.Rejected with evidence: editing #618 in place — forbidden, and the proposal there is otherwise
sound, so a replacement issue would discard a valid filing.
Reverses: close the new issue once #618 carries the correction.
Taken — substitute
## Validation's placeholder with this pass's gate set rather than removingthe section's promise. A section that says "commands verbatim, with outcome" and contains none is
a defect in the artefact a reader checks first. Earlier passes' commands stay under their own
Validation, this passheadings, which the section now names, so nothing is duplicated or moved.Rejected with evidence: deleting the promise instead — the commands exist and are recorded; the
placeholder was the only thing missing.
Reverses: restore the placeholder.
Taken — decline design questions 1, 2 and 3 and record them for a human. A version-mismatch
warning, a build-time recorded path, and printing every linked dynamic library each add machinery to
a change that is closing, and none of them repairs a defect this round found.
Rejected with evidence: implementing them here — each widens a benchmark PR's blast radius for a
diagnostic nobody has asked a question of yet.
Reverses: implement any of them under a new issue.
Not taken — design question 4, collapsing this body. It is a maintainer's call about the
repository's append-only convention, not a decision this run may take on its own authority.
Restated in
## Sixth passso it is not lost.This remains an unattended run: no human approved this plan, these decisions, or these corrections.
The record above is what a human reads afterwards.
Unresolved review notes
of 15.5–38.1; the box did not go quiet during the window and the load is attached to each run in
the raw output. This matters least for the finding it is used for — the Deflate result is stable
to ±2 % within a library across that whole load range, and eighteen independent runs found the
same flatness from 2.8 to 49 — and most for the two
lossless-jpegmedian columns, whosespread (18 to 101) is dominated by scheduling, which is why the fastest-sample ratio is quoted
beside them.
exist on this machine. The claim is that the choice of libz moves the Deflate rows across 1.0,
which two points establish; it is not a claim about any third build.
attribution of a 25–30 % Deflate shift to interleaving is withdrawn rather than replaced, and
interleaving is retained on its own argument. Settling what that shift was would mean reverting
the interleaving commit and re-measuring under both libraries; nothing this PR claims needs it.
now taken at the width the decoder materialises and applied only to the uncompressed rows, where
both paths do comparable work per byte. That argues the model is sound there; it does not measure
the preview's cost, which would need a decode entry point skipping the preview that gamut
deliberately does not expose.
decode_lossless_jpeg's medians are not usable on a loaded machine. gamut's arm is ~100 ms,long enough to absorb a scheduling event whole. The bound quoted for gamut-dng: lossless-JPEG decode is ~60x slower than the reference implementation #583 — no fastest-sample
ratio below 34× — is the defensible form; a tighter figure needs a quiet box.
to gamut-dng: lossless-JPEG decode is ~60x slower than the reference implementation #583 and gamut-dng: lossless-JPEG CFA encode expands the payload past uncompressed #584 that this run may not apply in place; until a human folds it in, an implementer
who opens gamut-dng: lossless-JPEG decode is ~60x slower than the reference implementation #583 reads a false localisation sentence and a verification command that matches no
benchmark. tooling/gamut-dng-oracle: pin the zlib the benchmark's reference arm links, keeping -lz for conformance #618 carries the oracle-linkage proposal. Both are linked from
## Issue.libstdc++is also unpinned, and linked asymmetrically. Only the reference arm links it —the SDK is C++ and gamut is not — so it is an asymmetry of the same kind as the libz one but
without the symmetric-comparison excuse. The work it does inside the measured region (the
std::vectorspool, thedng_negativedestructor) is orders of magnitude below inflate, so norow published here is plausibly moved by it; that is an argument from magnitude, not a
measurement. Nothing in this pass measured a second
libstdc++.expected_samples * sizeof(uint16_t)touint32unguarded. This pass made the length guard symmetric, which is what the finding named. The
sample-count narrowing is identical in both arms, so it does not distort the subtraction, and it
is unreachable from fixtures three orders of magnitude below the boundary — but it is the same
class of defect and is recorded rather than fixed silently.
targetcomponent. A platform library installed under a directory named
targetwould be flaggedspuriously, and a build-graph library resolved from a directory named something else would not be
flagged at all. It is a flag on the accident this repository actually produces, not a general
guarantee.
8956ad00states "thelossless-JPEG codestream decode favours gamut (the oracle's export path costs the SDK two extra
passes)" and
c6620b29promises to say "which of the two gamut-versus-SDK comparisons favourswhich side and by how much".
## Fourth pass(N5) withdrew both: over sixteen case-runs theexport-path gap spans −4 % to +64 %, so the pair supports a bound at the measurement floor, not a
signed cost. Later commits in this branch do retract in-message (
31f01840,347c8ade), so thepattern exists; these two predate it and are pushed. This run may not rewrite pushed history, so
the retraction lives here and in
## Fourth passrather than in the commits themselves. A readerof
git logalone meets the withdrawn wording with nothing beside it.taken. It would catch the case where a reader compares two Deflate figures taken against
different implementations, which is the failure this whole thread is about, and it needs a second
recorded identity to compare against, which the harness does not have. Declined as machinery added
to a closing change; it belongs with tooling/gamut-dng-oracle: pin the zlib the benchmark's reference arm links, keeping -lz for conformance #618, which would give it the pinned identity to compare to.
resolution — is not taken. It would turn the build-directory warning from a heuristic on a path
component into an exact statement about whether the loader picked the library the build script
chose. It needs a build-script change to
gamut-dng-oracle, which is tooling/gamut-dng-oracle: pin the zlib the benchmark's reference arm links, keeping -lz for conformance #618's territory.just libz — is not taken.
lddon the built bench binary lists five shared objects:libstdc++.so.6,libz.so.1,libgcc_s.so.1,libm.so.6andlibc.so.6. Onlylibzisproduced by a build script in this graph and so only
libzcan be resolved out oftarget/; theother four come from the platform, and none of them does work inside the measured region within
orders of magnitude of inflate (see the
libstdc++note above). Printing all five would make theone that actually moves a row harder to see, not easier.
~84 KB the record is longer than the change, and every withdrawn figure in it is traceable to the
pass that withdrew it. Trading that for a current description plus a short changelog is a decision
about the repository's append-only convention and belongs to a maintainer.
worktree path, a long agent identifier, or a checkout path with a hidden tooling directory in it.
A tooling path in a shape none of those match would pass. It is a guard against the accident this
run actually produces, and it is asserted in both directions before every publish, not a general
guarantee.