diff --git a/crates/gamut-exif/README.md b/crates/gamut-exif/README.md index 7680d163..d2c5eb4f 100644 --- a/crates/gamut-exif/README.md +++ b/crates/gamut-exif/README.md @@ -44,7 +44,34 @@ let out = edited.to_bytes(); // Exif\0\0 + TIFF, ready to re-embe ``` For a bare TIFF stream (PNG `eXIf` / WebP `EXIF`) or a byte-order override, use [`ExifWriter`]; -[`ExifReader`] carries the read-side options (`require_marker`, `strict`). +[`ExifReader`] carries the read-side options (`require_marker`, `strict`) and two further entry +points: + +- **`parse_from`** reads through [`gamut_ifd::ReadAt`] instead of a slice, so the EXIF of a + 300 MB raw file costs a few hundred bytes of I/O rather than the whole file. `parse` is the + `&[u8]` case of it — one parse engine, two entry points. It is deliberately synchronous: an + async caller drives the source itself, which keeps a runtime dependency out of the crate. +- **`parse_with_report`** (and its `parse_from_with_report` twin) returns a `ReadReport` alongside + the `Exif`, naming each region the lenient reader discarded — a malformed Exif/GPS/Interop + sub-IFD, an unusable thumbnail range, or a top-level directory past the 1st IFD — with the tag + that addressed it, the offset it carried, and a typed reason. `parse` stays silent, as before. + The report is complete over those regions but is **not** a byte-completeness verdict: an empty + report does not mean the parse lost nothing (see the deferred items below). A `strict` report is + not always empty either: strictness rejects *malformed* regions, and a trailing directory is + well-formed and merely unrepresentable, so it is reported in both modes. + +```rust +# use gamut_exif::ExifReader; +# fn demo(bytes: &[u8]) -> Result<(), gamut_exif::ExifError> { +let (exif, report) = ExifReader::new().parse_with_report(bytes)?; +for dropped in report.dropped() { + // e.g. "dropped GPS (tag 0x8825) at offset 65535: addresses bytes outside the EXIF blob" + eprintln!("{dropped}"); +} +# let _ = exif; +# Ok(()) +# } +``` Enable the optional `geocoordinates` feature (also included by `full`) to convert a complete [`GpsInfo`] with `TryFrom` into `geocoordinates::Wgs84` or `geocoordinates::Coordinate`. The latter @@ -52,6 +79,30 @@ preserves EXIF sea-level altitude as an orthometric height; the 2D `Wgs84` newty drops altitude. Malformed references, rationals, DMS components, and out-of-range positions return the typed [`GpsConversionError`]. +## Compatibility + +**One read verdict changed after 1.0.0.** A 1st IFD carrying `JPEGInterchangeFormat` with no +`JPEGInterchangeFormatLength` used to parse as a thumbnail that simply had no bytes; it is a +**loss** — named in the report as `DropReason::ThumbnailLengthMissing`, and **rejected by +`strict`** with `ExifError::BadThumbnail`. A caller running `strict` over blobs 1.0.0 accepted +should know the verdict moved. + +The rule is about **readability**: an offset with nothing to size it addresses bytes that cannot be +read, which is what `strict` is for. It is not about a support level, and the reader does not +consult `Compression` at all. + +**Conformance** is the separate question of whether the move is a *fix* or a *redefinition*, and it +is a fix, so no major version is forced. Exif 3.0 §4.6.9.2 Table 21 states each 1st IFD tag's +support level per thumbnail-format column: three uncompressed ones distinguished by photometric +interpretation and planar configuration (Chunky, Planar, YCC), plus **Compressed**. That axis is not +the two-valued `Compression` tag. The table gives `JPEGInterchangeFormat` and +`JPEGInterchangeFormatLength` the *same* level in each column: "not allowed to record" under the +three uncompressed columns, mandatory under Compressed. An offset with no length is therefore +non-conformant under every column, and no conformant 1st IFD changes verdict. That grounding is what +this repository ships — the table is vendored under `references/exif/`; the before/after comparison +measured behind it is recorded in the pull request that introduced the report, not committed here as +a harness. + ## Scope v1 covers the **standard CIPA DC-008 tag dictionary** ([`ExifTag`]), full read/write round-trips @@ -67,6 +118,17 @@ designed to be added without breaking the 1.0 API — the catalogue and vendor e - **exiftool-parity tag breadth** beyond the standard dictionary (unknown tags still round-trip losslessly via the raw `Ifd`). - **Uncompressed strip-based thumbnails** are read but not re-embedded (JPEG thumbnails are). +- **Per-tag error recovery inside one directory.** A single unparseable entry fails its whole + directory in `gamut-ifd`, so the report's granularity is the sub-IFD, not the individual tag + (issue #521). +- **A signal for a shadowed duplicate tag.** Two entries for one tag decode to the last, and the + earlier one is discarded a layer below this crate. `gamut-exif` could *detect* the loss (compare + `RawIfd::entries` against the decoded `Ifd::fields()`), but not describe it without re-decoding + the shadowed entry, so the signal belongs where the discarding happens — a layer three crates + share (issue #528). +- **A byte-completeness verdict** over the whole blob (which source bytes no parsed structure + claims). `gamut-ifd`'s audit engine has the machinery; `ReadReport` today reports only what was + dropped, not what was never reached (issue #521). ## Status diff --git a/crates/gamut-exif/STATUS.md b/crates/gamut-exif/STATUS.md index 3a355f5f..511b6480 100644 --- a/crates/gamut-exif/STATUS.md +++ b/crates/gamut-exif/STATUS.md @@ -24,6 +24,7 @@ fixtures** (`tests/fixtures/`, regenerate with `GAMUT_REGEN_GOLDEN=1`). | P6 | §4.6 | **Keystone** — writer round-trip (endianness/pointers/thumbnail preserved) | ✅ | | P7 | §4.6 | MakerNote: opaque passthrough + vendor detection (no per-vendor decode) | ✅ | | P8 | — | exiv2 differential gate + golden fixtures | ✅ | +| P9 | §4.6 | `ReadAt` streaming entry point + lenient-drop report (`ReadReport`, scoped to the regions `DroppedRegion` names) | ✅ | ## Intentionally deferred (additive under the `#[non_exhaustive]` surface) @@ -35,5 +36,27 @@ fixtures** (`tests/fixtures/`, regenerate with `GAMUT_REGEN_GOLDEN=1`). round-trip losslessly because the raw `gamut_ifd::Ifd` is retained. - **Uncompressed strip-based thumbnails** are read (as their directory) but not re-embedded; JPEG thumbnails round-trip fully. -- **A reader coverage/validation report** — `gamut-ifd` has the machinery; exposing a toggle on - `ExifReader` is a non-breaking future addition. +- **Per-tag error recovery inside a directory.** `ExifReader::parse_with_report` names the + sub-IFDs, thumbnail ranges and trailing top-level directories the lenient reader discards (issue + #419), but the granularity is the directory: a single unparseable entry fails its whole IFD in + `gamut-ifd`, which is the layer that would have to recover per entry. nom-exif's + `entry.into_result()` is finer-grained here. +- **A signal for a shadowed duplicate tag.** `gamut_ifd::IfdReader::decode_ifd` builds a directory + with `Ifd::set`, which is last-wins, so two entries for one tag decode to the second and the + first is discarded. `ReadReport::is_empty()` is therefore a verdict over the regions it covers, + **not** "this parse lost nothing"; both the report's module docs and the README say so. + + The deferral is a layering decision, **not** an inability to observe: `RawIfd::entries` is public + and in on-disk order and `follow` already holds the `RawIfd`, so `raw.entries.len() != + ifd.fields().len()` would detect a shadowed tag here in three lines. What `gamut-exif` cannot do + is say *what* was lost without re-decoding the shadowed entry — and `gamut-tiff` and `gamut-dng` + need the same signal, so it belongs in the shared layer (issue #528). Issue #528's own body + states the weaker, incorrect reason; read it with this correction. +- **A byte-completeness verdict.** `ReadReport` says what was *dropped*, not which source bytes no + parsed structure claims. `gamut-ifd`'s audit engine (`Tracked`, `SegmentMap`, `read_audited`) is + the machinery for it and is already used by `gamut-dng` and `gamut-tiff`; wiring it behind a + toggle on `ExifReader` stays a non-breaking future addition. +- **An async entry point.** Declined rather than deferred: `parse_from` is synchronous over + `gamut_ifd::ReadAt`, and an async caller drives that source itself. A `tokio` feature would put a + runtime dependency in a crate that has none and constrain the public shape against the + C-portability convention, for a capability the caller can supply. diff --git a/crates/gamut-exif/src/lib.rs b/crates/gamut-exif/src/lib.rs index cd9d9c5f..04e21c52 100644 --- a/crates/gamut-exif/src/lib.rs +++ b/crates/gamut-exif/src/lib.rs @@ -15,6 +15,30 @@ //! [`Exif::parse`] reads a blob and [`Exif::to_bytes`] re-serialises it (preserving the byte order); //! read tags with the typed accessors or the [`ExifTag`] catalogue. //! +//! Two further read entry points sit on [`ExifReader`]: +//! [`parse_from`](ExifReader::parse_from) reads through [`gamut_ifd::ReadAt`] rather than a slice, +//! so EXIF can be pulled out of a large raw file without loading it, and +//! [`parse_with_report`](ExifReader::parse_with_report) returns a [`ReadReport`] naming the +//! sub-IFDs, thumbnail ranges and trailing directories the lenient reader discarded — see +//! [`report`] for what that covers and what it deliberately does not. `parse` is the `&[u8]` case +//! of `parse_from` and stays silent. +//! +//! `parse` keeps its signature, its accept/reject verdict and its re-serialised bytes, with two +//! narrow exceptions, both measured over a 3 144-case truncation-and-corruption sweep against the +//! previous release. An error message's offset is now a position in the buffer the caller handed +//! in, so for a marked blob it is six bytes larger than before — the `Exif\0\0` marker — while a +//! [`Dropped::offset`] stays relative to the TIFF stream; and a 1st IFD carrying +//! `JPEGInterchangeFormat` with no `JPEGInterchangeFormatLength` is now named in the report +//! instead of vanishing, and rejected in [`strict`](ExifReader::strict) mode as the unreadable +//! range it is — an offset with nothing to size it addresses bytes that cannot be read. That rule +//! is structural, not a support level: Exif 3.0 §4.6.9.2 Table 21 states the pair's level only *per +//! thumbnail-format column*, an axis that is not the two-valued `Compression` tag — which this +//! reader does not consult, though [`Thumbnail::compression`](thumbnail::Thumbnail::compression) +//! exposes it to callers (issue #574). It refuses no conformant input: the table gives both tags +//! the same level in all four columns — `M` under **Compressed**, `N` (not allowed to record) under +//! the three uncompressed ones — so an offset with no length is non-conformant under every one of +//! them. +//! //! ``` //! use gamut_exif::{ByteOrder, Exif, ExifTag, Value}; //! @@ -35,6 +59,8 @@ pub mod exif; pub mod gps; pub mod maker_note; pub mod reader; +pub mod report; +mod stream; pub mod tag; pub mod thumbnail; pub mod value; @@ -50,6 +76,7 @@ pub use gps::GpsConversionError; pub use gps::{GpsAltitude, GpsCoordinate, GpsInfo, GpsReference}; pub use maker_note::{MakerNote, MakerNoteVendor}; pub use reader::ExifReader; +pub use report::{DropReason, Dropped, DroppedRegion, ReadReport}; pub use tag::{ExifTag, IfdKind}; pub use thumbnail::Thumbnail; pub use value::{Rational, SRational, as_text}; diff --git a/crates/gamut-exif/src/reader.rs b/crates/gamut-exif/src/reader.rs index ce84211f..1186b1ca 100644 --- a/crates/gamut-exif/src/reader.rs +++ b/crates/gamut-exif/src/reader.rs @@ -1,31 +1,19 @@ //! Reading an EXIF blob into the typed [`Exif`] model. //! //! An EXIF blob is an optional `Exif\0\0` marker followed by a TIFF stream. The 0th IFD and (when -//! present) the 1st IFD are the top-level chain [`gamut_ifd::read`] returns; the Exif, GPS, and -//! Interoperability directories hang off pointer *tags* that the generic reader cannot follow (it -//! cannot know which `LONG`s are offsets), so this reader chases those pointers explicitly and -//! removes them, representing each sub-IFD structurally on [`Exif`] instead. - -use gamut_ifd::{ByteOrder, Ifd, IfdReader, Variant, tags as ifd_tags}; - -use crate::error::{ExifError, Result}; -use crate::exif::{EXIF_IFD_POINTER, Exif, GPS_IFD_POINTER, INTEROP_IFD_POINTER, MARKER}; -use crate::tag::ExifTag; -use crate::thumbnail::Thumbnail; +//! present) the 1st IFD are the top-level chain; the Exif, GPS, and Interoperability directories +//! hang off pointer *tags* that the generic TIFF reader cannot follow (it cannot know which +//! `LONG`s are offsets), so this reader chases those pointers explicitly and removes them, +//! representing each sub-IFD structurally on [`Exif`] instead. +//! +//! This module holds the reader's options and its `&[u8]` entry points. The parse itself is +//! generic over [`gamut_ifd::ReadAt`] and lives in the crate's private `stream` module; a slice is +//! simply one such source, so there is exactly **one** parse engine and the two entry points cannot +//! drift. -/// The absolute offset of the Exif sub-IFD's out-of-line `MakerNote` value in `tiff`, or `None` -/// if the note is absent or inline. -fn maker_note_offset( - tiff: &[u8], - exif_ifd_at: u64, - order: ByteOrder, - variant: Variant, -) -> Option { - let mut reader = IfdReader::with_layout(tiff, order, variant); - let raw = reader.read_ifd(exif_ifd_at).ok()?; - let entry = raw.entry(ifd_tags::MAKER_NOTE)?; - reader.value_offset(entry) -} +use crate::error::Result; +use crate::exif::Exif; +use crate::report::ReadReport; /// Reads an EXIF blob into an [`Exif`], with options for how the parse is bounded. /// @@ -33,8 +21,8 @@ fn maker_note_offset( /// lenient: a malformed Exif/GPS/Interop sub-IFD is dropped rather than failing the whole parse. #[derive(Debug, Clone, Default)] pub struct ExifReader { - require_marker: bool, - strict: bool, + pub(crate) require_marker: bool, + pub(crate) strict: bool, } impl ExifReader { @@ -45,7 +33,7 @@ impl ExifReader { } /// Requires the `Exif\0\0` marker; a bare TIFF stream is then rejected with - /// [`ExifError::MissingMarker`]. + /// [`ExifError::MissingMarker`](crate::ExifError::MissingMarker). /// /// Off by default: the JPEG `APP1` segment carries the marker, but the WebP `EXIF` and PNG /// `eXIf` chunks carry a bare TIFF stream. @@ -65,121 +53,79 @@ impl ExifReader { /// Parses an EXIF blob into an [`Exif`]. /// + /// The `&[u8]` case of [`parse_from`](Self::parse_from) — a slice is a + /// [`ReadAt`](gamut_ifd::ReadAt) source — so the two share one parse engine. + /// /// # Errors /// - /// Returns [`ExifError::MissingMarker`] when the marker is required but absent, an - /// [`ExifError::Ifd`] when the TIFF stream is malformed, or (in [`strict`](Self::strict) mode) - /// [`ExifError::InvalidIfd`] when a sub-IFD pointer addresses a malformed directory. + /// Returns [`ExifError::MissingMarker`](crate::ExifError::MissingMarker) when the marker is + /// required but absent, an [`ExifError::Ifd`](crate::ExifError::Ifd) when the TIFF stream is + /// malformed, or (in [`strict`](Self::strict) mode) + /// [`ExifError::InvalidIfd`](crate::ExifError::InvalidIfd) when a sub-IFD pointer addresses a + /// malformed directory or [`ExifError::BadThumbnail`](crate::ExifError::BadThumbnail) when the + /// 1st IFD's JPEG range is unusable — outside the blob, or an offset with no + /// `JPEGInterchangeFormatLength` to size it. + /// + /// An offset inside an error message is a position in `bytes` — the buffer the caller handed + /// in — so for a marked blob it counts the six-byte `Exif\0\0` marker. That is deliberately a + /// different frame from [`Dropped::offset`](crate::Dropped::offset), which is relative to the + /// start of the TIFF stream and therefore six smaller for the same position: a diagnostic + /// points into the caller's own bytes, while a report offset addresses the TIFF structure. pub fn parse(&self, bytes: &[u8]) -> Result { - let tiff = match bytes.strip_prefix(MARKER) { - Some(rest) => rest, - None if self.require_marker => return Err(ExifError::MissingMarker), - None => bytes, - }; - - let file = gamut_ifd::read(tiff)?; - let order = file.order; - let variant = file.variant; - let mut ifds = file.ifds.into_iter(); - let mut image = ifds.next().ok_or(ExifError::Truncated)?; - // The next-IFD chain's second entry is the thumbnail directory (1st IFD), if any. - let thumbnail = match ifds.next() { - Some(ifd) => Some(self.read_thumbnail(ifd, tiff)?), - None => None, - }; - - // The Exif sub-IFD's own offset, captured before `follow` strips the pointer: the - // maker-note pin needs the note value's absolute source position. - let exif_ifd_at = image.get_u32(EXIF_IFD_POINTER).map(u64::from); - let exif = self.follow(&mut image, tiff, order, variant, EXIF_IFD_POINTER, "Exif")?; - let gps = self.follow(&mut image, tiff, order, variant, GPS_IFD_POINTER, "GPS")?; - let maker_note_at = match (&exif, exif_ifd_at) { - (Some(_), Some(at)) => maker_note_offset(tiff, at, order, variant), - _ => None, - }; - - // The Interoperability directory is reached from *inside* the Exif sub-IFD, not the 0th IFD. - let (exif, interop) = match exif { - Some(mut e) => { - let interop = - self.follow(&mut e, tiff, order, variant, INTEROP_IFD_POINTER, "Interop")?; - (Some(e), interop) - } - None => (None, None), - }; - - Ok(Exif::from_parts( - order, - image, - exif, - gps, - interop, - thumbnail, - maker_note_at, - )) + self.parse_from(bytes) } - /// Reads pointer tag `ptr` from `parent`, removes it (the pointer is represented structurally, - /// not as a data field), and parses the sub-IFD it addresses. + /// Parses an EXIF blob and reports what a lenient parse discarded. /// - /// Returns `Ok(None)` when the pointer is absent, or — in lenient mode — when the pointed-at - /// directory is malformed. - fn follow( - &self, - parent: &mut Ifd, - tiff: &[u8], - order: ByteOrder, - variant: Variant, - ptr: u16, - name: &'static str, - ) -> Result> { - let Some(offset) = parent.get_u32(ptr) else { - return Ok(None); - }; - parent.remove(ptr); - match gamut_ifd::read_ifd_at(tiff, u64::from(offset), order, variant) { - Ok(ifd) => Ok(Some(ifd)), - Err(_) if !self.strict => Ok(None), - Err(_) => Err(ExifError::InvalidIfd(name)), - } - } - - /// Builds a [`Thumbnail`] from the 1st IFD, slicing out its JPEG bytes (from the - /// `JPEGInterchangeFormat` offset / length) when present. In lenient mode an out-of-bounds - /// JPEG range yields a thumbnail without bytes; in strict mode it errors. - fn read_thumbnail(&self, ifd: Ifd, tiff: &[u8]) -> Result { - let offset = ifd.get_u32(ExifTag::JpegInterchangeFormat.tag_id()); - let length = ifd.get_u32(ExifTag::JpegInterchangeFormatLength.tag_id()); - let jpeg = match (offset, length) { - (Some(offset), Some(length)) => { - let range = (offset as usize).checked_add(length as usize); - match range.and_then(|end| tiff.get(offset as usize..end)) { - Some(bytes) => Some(bytes.to_vec()), - None if self.strict => { - return Err(ExifError::BadThumbnail("JPEG offset out of bounds")); - } - None => None, - } - } - _ => None, - }; - // The JPEGInterchangeFormat offset is structural — the bytes are captured above and the - // writer re-synthesises the offset — so drop it from the stored directory (mirroring how the - // sub-IFD pointer tags are stripped), leaving a value the model can't carry stale. - let mut ifd = ifd; - if jpeg.is_some() { - ifd.remove(ExifTag::JpegInterchangeFormat.tag_id()); - } - Ok(Thumbnail::from_parts(ifd, jpeg)) + /// [`parse`](Self::parse) is silent about what leniency drops; this returns the same [`Exif`] + /// alongside a [`ReadReport`] naming each discarded region, so a caller can tell a blob that + /// never carried GPS from one whose GPS pointer was dangling. + /// + /// The report covers the regions [`DroppedRegion`](crate::DroppedRegion) enumerates and is + /// complete over them. It is **not** a byte-completeness verdict, and an empty report does not + /// mean the parse lost nothing — see the [`report`](crate::report) module for the two known + /// losses below this crate. + /// + /// ``` + /// # use gamut_exif::{ByteOrder, Exif, ExifReader}; + /// # let bytes = Exif::new(ByteOrder::LittleEndian).to_bytes()?; + /// let (exif, report) = ExifReader::new().parse_with_report(&bytes)?; + /// assert!(report.is_empty()); // no covered region was discarded + /// for dropped in report.dropped() { + /// eprintln!("{dropped}"); + /// } + /// # let _ = exif; + /// # Ok::<(), gamut_exif::ExifError>(()) + /// ``` + /// + /// # Errors + /// + /// As [`parse`](Self::parse). In [`strict`](Self::strict) mode the first *malformed* region + /// fails the parse instead of being reported — but a strict report is **not** therefore always + /// empty: [`DroppedRegion::TrailingIfd`](crate::DroppedRegion::TrailingIfd) is well-formed and + /// merely unrepresentable, so strictness has no grounds to reject it and it is reported in both + /// modes. + /// + /// The two offset frames of [`parse`](Self::parse) meet here: an offset in a returned + /// [`ExifError`](crate::ExifError) is a position in `bytes` and counts any `Exif\0\0` marker, + /// while every [`Dropped::offset`](crate::Dropped::offset) in the report is relative to the + /// start of the TIFF stream — six smaller for the same position in a marked blob. A caller + /// that renders both beside each other must normalise one of them. They *meet* rather than + /// always arrive together: the return is a sum type, so a blob whose strict-fatal defect is + /// reached after a reportable one yields the error alone and no report. + pub fn parse_with_report(&self, bytes: &[u8]) -> Result<(Exif, ReadReport)> { + self.parse_from_with_report(bytes) } } #[cfg(test)] mod tests { - use gamut_ifd::{TiffFile, Value, write}; + use gamut_ifd::{ByteOrder, Ifd, TiffFile, Value, Variant, write}; use super::*; - use crate::IfdKind; + use crate::error::ExifError; + use crate::exif::{EXIF_IFD_POINTER, GPS_IFD_POINTER, INTEROP_IFD_POINTER, MARKER}; + use crate::tag::{ExifTag, IfdKind}; /// Builds a small but structurally complete EXIF TIFF stream (0th IFD with Make/Orientation, /// an Exif sub-IFD with FNumber + a nested Interop sub-IFD, a GPS sub-IFD, and a thumbnail @@ -331,6 +277,45 @@ mod tests { ); } + /// A thumbnail offset with no length is an unreadable range, and strict mode says so. + /// + /// A `JPEGInterchangeFormat` with nothing to size the read by addresses bytes that cannot be + /// fetched, so it fails strictness for the same reason an out-of-bounds range does — the + /// sibling case above — rather than passing as a thumbnail that simply has no bytes. The + /// message is pinned because it must state that structural fact and *not* claim a missing + /// mandatory tag: Exif 3.0 §4.6.9.2 Table 21 makes the pair mandatory only under + /// `Compression = Compressed`, and forbids recording either tag under the uncompressed + /// columns (issue #574). The lenient half of the contract is the report, pinned in + /// `tests/report.rs`. + #[test] + fn a_thumbnail_offset_without_a_length_is_rejected_strictly() { + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); + let mut thumb = Ifd::new(); + thumb.set(ExifTag::Compression.tag_id(), Value::Short(vec![6])); + thumb.set( + ExifTag::JpegInterchangeFormat.tag_id(), + Value::Long(vec![4]), + ); + // ...and deliberately no JpegInterchangeFormatLength. + let bytes = write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds: vec![image, thumb], + }) + .expect("write"); + + let err = ExifReader::new() + .strict(true) + .parse(&bytes) + .expect_err("strict must reject half a thumbnail pair"); + assert_eq!( + err.to_string(), + "invalid thumbnail: JPEGInterchangeFormat offset with no length to size it", + "the message must name the unreadable range, not a missing mandatory tag" + ); + } + #[test] fn lenient_drops_a_dangling_sub_ifd_pointer_that_strict_rejects() { // An ExifIFD pointer that addresses far past the end of the stream. diff --git a/crates/gamut-exif/src/report.rs b/crates/gamut-exif/src/report.rs new file mode 100644 index 00000000..594d6fae --- /dev/null +++ b/crates/gamut-exif/src/report.rs @@ -0,0 +1,391 @@ +//! What a lenient parse discarded. +//! +//! [`ExifReader`](crate::ExifReader) is lenient by default: a malformed Exif/GPS/Interop sub-IFD or +//! an out-of-bounds thumbnail range is dropped so the rest of the blob still parses. That is the +//! right default for real-world files, but on its own it is *silent* — a caller cannot tell a blob +//! that never carried GPS from one whose GPS pointer was dangling. +//! +//! A [`ReadReport`], returned by +//! [`ExifReader::parse_with_report`](crate::ExifReader::parse_with_report), names each discarded +//! region: **where** it was (a [`DroppedRegion`] and the tag that addressed it), **what offset** +//! addressed it, and **why** it went ([`DropReason`]). +//! +//! # What this report does and does not claim +//! +//! It covers exactly the regions [`DroppedRegion`] enumerates — the three pointer-addressed +//! sub-IFDs, the thumbnail's JPEG bytes, and any top-level directory past the 1st IFD. Within that +//! set it is complete: nothing in it is discarded without an entry. +//! +//! It is **not** a byte-completeness verdict over the blob, and +//! [`is_empty`](ReadReport::is_empty) does not mean "this parse lost nothing". Two known losses sit +//! outside it, both below this crate in [`gamut_ifd`]: +//! +//! * a **duplicate tag** within one directory keeps the last occurrence and discards the earlier +//! one. This crate *could* detect that a directory lost an entry — `RawIfd::entries` is public +//! and in on-disk order, so comparing its length against the decoded `Ifd::fields()` finds it in +//! three lines — but it could not say what was lost without re-decoding the shadowed entry +//! itself. The signal belongs at the layer that does the discarding, which three crates share +//! (issue #528); +//! * a single unparseable **entry** fails its whole directory rather than being skipped, so the +//! report's granularity is the directory, never the individual tag (issue #521). +//! +//! Bytes that no parsed structure ever claimed are likewise not reported; that verdict would need +//! `gamut-ifd`'s audit engine and is tracked in #521. + +use core::fmt; + +use crate::exif::{EXIF_IFD_POINTER, GPS_IFD_POINTER, INTEROP_IFD_POINTER}; +use crate::tag::ExifTag; + +/// A region of an EXIF blob that a lenient parse can discard. +/// +/// Fieldless with an explicit `repr` and append-only discriminants, so the value crosses an FFI +/// boundary as a plain integer. `#[non_exhaustive]`: regions can be added post-1.0 without a +/// breaking change. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[repr(u8)] +#[non_exhaustive] +pub enum DroppedRegion { + /// The Exif sub-IFD, addressed by the `ExifIFD` pointer (`0x8769`) in the 0th IFD. + ExifIfd = 0, + /// The GPS sub-IFD, addressed by the `GPSInfo` pointer (`0x8825`) in the 0th IFD. + GpsIfd = 1, + /// The Interoperability sub-IFD, addressed by the `Interoperability` pointer (`0xA005`) + /// *inside* the Exif sub-IFD. + InteropIfd = 2, + /// The 1st IFD's embedded JPEG thumbnail bytes, addressed by `JPEGInterchangeFormat` + /// (`0x0201`) and sized by `JPEGInterchangeFormatLength` (`0x0202`). The thumbnail's own + /// directory survives; only its bytes are lost. + /// + /// Reported when the range lies outside the blob ([`DropReason::OutOfBounds`]) and when the + /// offset has no length beside it ([`DropReason::ThumbnailLengthMissing`]): an offset with + /// nothing to size it addresses bytes that cannot be read, which is a loss rather than an + /// absent thumbnail. That rule is structural and unconditional here — see + /// [`ThumbnailLengthMissing`](DropReason::ThumbnailLengthMissing) — and is *not* derived from + /// the pair's support level, which Exif 3.0 §4.6.9.2 Table 21 states only per + /// thumbnail-format column. That axis is not the two-valued `Compression` tag, which this + /// *reader* does not consult (a parsed [`Thumbnail`](crate::Thumbnail) does expose it, through + /// [`compression`](crate::Thumbnail::compression)). + ThumbnailJpeg = 3, + /// A top-level directory past the 1st IFD. + /// + /// EXIF defines exactly two: the 0th IFD (primary image) and the 1st (thumbnail). A stream + /// whose next-IFD chain runs on has more, and the [`Exif`](crate::Exif) model has nowhere to + /// put them — so they parse cleanly and are then discarded. No tag addresses one (the chain is + /// followed through the structural next-IFD pointer), so [`Dropped::tag`] is `None`. + /// + /// Reported in [`strict`](crate::ExifReader::strict) mode too: strictness rejects *malformed* + /// regions, and nothing about a trailing directory is malformed — it is well-formed and + /// unrepresentable. A strict report is therefore empty of everything *but* this. + TrailingIfd = 4, +} + +impl DroppedRegion { + /// The region's short name, as it appears in [`Dropped`]'s `Display`. + /// + /// For the three sub-IFDs this is also the spelling + /// [`ExifError::InvalidIfd`](crate::ExifError::InvalidIfd) uses in strict mode. The other two + /// have no such error: a strict out-of-bounds thumbnail is + /// [`ExifError::BadThumbnail`](crate::ExifError::BadThumbnail), and a trailing directory is + /// discarded in strict mode exactly as in lenient mode, since nothing about it is malformed. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::ExifIfd => "Exif", + Self::GpsIfd => "GPS", + Self::InteropIfd => "Interop", + Self::ThumbnailJpeg => "Thumbnail", + Self::TrailingIfd => "TrailingIFD", + } + } + + /// The tag whose value addressed this region, or `None` when no tag does. + pub(crate) const fn tag(self) -> Option { + match self { + Self::ExifIfd => Some(EXIF_IFD_POINTER), + Self::GpsIfd => Some(GPS_IFD_POINTER), + Self::InteropIfd => Some(INTEROP_IFD_POINTER), + Self::ThumbnailJpeg => Some(ExifTag::JpegInterchangeFormat.tag_id()), + Self::TrailingIfd => None, + } + } +} + +/// Why a region was discarded. +/// +/// Fieldless with an explicit `repr` and append-only discriminants; `#[non_exhaustive]` so reasons +/// can be distinguished more finely post-1.0 without a breaking change. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[repr(u8)] +#[non_exhaustive] +pub enum DropReason { + /// The address itself lay outside the blob — a pointer at or past the end of the TIFF stream, + /// or a byte range that is not wholly inside it. Nothing could have been read there. + OutOfBounds = 0, + /// The address was inside the blob, but the structure at it did not parse: a bad entry count, + /// an unreadable entry, or a value offset the directory could not resolve. + Malformed = 1, + /// Nothing was wrong with the region — it parsed cleanly — but the EXIF model has no place to + /// put it, so it could not be carried across. + Unrepresentable = 2, + /// The 1st IFD carried a `JPEGInterchangeFormat` offset with no `JPEGInterchangeFormatLength` + /// beside it, so there was no range to read and the JPEG behind the offset is lost. + /// + /// Deliberately named for that one site rather than for the shape of the defect: it is the + /// only thing this reason ever means, and a generic name on a single-site variant invites + /// unrelated reuse that a `#[non_exhaustive]` enum can add a *new* variant for instead. + /// + /// Distinct from [`OutOfBounds`](Self::OutOfBounds) — the address may be perfectly valid — and + /// from [`Malformed`](Self::Malformed), which is about bytes that *were* read and did not + /// parse. The repair is different in each case, which is why they are different reasons. + /// + /// Recorded whatever the thumbnail's `Compression` says, because the reason it is a loss is + /// **readability**: the read has no length, not a tag is missing where the spec requires one. + /// Exif 3.0 §4.6.9.2 Table 21 gives the pair's support level *per column*: mandatory under + /// **Compressed**, and `N` (not allowed to record) under all three uncompressed columns. + /// + /// **Conformance** is a separate axis, and it is only what shows the unconditional rule names + /// nothing it should not: the two tags carry the *same* level in every column, so an offset + /// with no length is non-conformant under all four and there is no conformant 1st IFD this + /// rule wrongly names. Whether to condition it on `Compression` anyway is filed as issue #574, + /// and it is a cheap option rather than a costly one — not because + /// [`Thumbnail::compression`](crate::Thumbnail::compression) exists (that accessor reads a + /// *finished* thumbnail, and the arm #574 would condition returns before one is built) but + /// because the 1st IFD is already in scope there and `Compression` is the same one-line + /// lookup that reads the offset and the length two lines above. Nothing needs plumbing. + /// + /// The mirror case in that issue — a length with no offset — is genuinely + /// asymmetric and not merely unreached: an offset with no length *addresses bytes*, so + /// something is lost, while a length with no offset addresses nothing, so nothing is. + ThumbnailLengthMissing = 3, +} + +impl DropReason { + /// The clause [`Dropped`]'s `Display` uses for this reason. + const fn clause(self) -> &'static str { + match self { + Self::OutOfBounds => "addresses bytes outside the EXIF blob", + Self::Malformed => "is not a well-formed directory", + Self::Unrepresentable => "parsed cleanly but has no place in the EXIF model", + Self::ThumbnailLengthMissing => "has no JPEGInterchangeFormatLength to size the read", + } + } +} + +/// One region a lenient parse discarded, named by where it was and why it went. +/// +/// `Copy` plain data reachable through accessors, so it is representable across an FFI boundary. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct Dropped { + region: DroppedRegion, + tag: Option, + offset: u64, + reason: DropReason, +} + +impl Dropped { + /// Records a drop of `region`, addressed by its tag at `offset`, for `reason`. + pub(crate) const fn new(region: DroppedRegion, offset: u64, reason: DropReason) -> Self { + Self { + region, + tag: region.tag(), + offset, + reason, + } + } + + /// Which region was discarded. + #[must_use] + pub const fn region(self) -> DroppedRegion { + self.region + } + + /// The tag whose value addressed the discarded region — the pointer tag for a sub-IFD, + /// `JPEGInterchangeFormat` for the thumbnail bytes. + /// + /// `None` when no tag addresses the region, which today means only + /// [`DroppedRegion::TrailingIfd`]: a top-level directory is reached through the structural + /// next-IFD pointer, not through a tag. This is an `Option` rather than a `0` sentinel because + /// `0` is a real tag number (`GPSVersionID`), and the region set is `#[non_exhaustive]` — the + /// next region without an addressing tag might well be one inside a GPS directory. + #[must_use] + pub const fn tag(self) -> Option { + self.tag + } + + /// The offset of the discarded region, relative to the start of the TIFF stream (i.e. *after* + /// any `Exif\0\0` marker) — the same frame of reference EXIF's own offsets use. + /// + /// For a sub-IFD or the thumbnail bytes this is the value the addressing tag carried; for a + /// [`TrailingIfd`](DroppedRegion::TrailingIfd) it is the directory's own position in the + /// stream. + /// + /// This is **not** the frame the crate's *error* messages use. An [`ExifError`](crate::ExifError) + /// carries the offset of the byte the reader could not read in the source the caller handed in, + /// so for a marked blob it is 6 bytes (`MARKER.len()`) larger than the same position expressed + /// here. The two frames are deliberately different: a diagnostic points into the caller's own + /// buffer, while a report offset addresses the TIFF structure the report describes and matches + /// every offset stored inside the file. + #[must_use] + pub const fn offset(self) -> u64 { + self.offset + } + + /// Why the region was discarded. + #[must_use] + pub const fn reason(self) -> DropReason { + self.reason + } +} + +impl fmt::Display for Dropped { + /// One grammar for every drop — `dropped (tag ) at offset : ` — with + /// `none` as the explicit absent marker rather than a second shape. A caller that scrapes this + /// line should not have to recognise two forms, and an absent tag is a fact worth stating. + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let name = self.region.name(); + let clause = self.reason.clause(); + match self.tag { + Some(tag) => write!( + f, + "dropped {name} (tag {tag:#06x}) at offset {}: {clause}", + self.offset + ), + None => write!( + f, + "dropped {name} (tag none) at offset {}: {clause}", + self.offset + ), + } + } +} + +/// What a lenient parse discarded, over the regions [`DroppedRegion`] enumerates. +/// +/// Obtained from [`ExifReader::parse_with_report`](crate::ExifReader::parse_with_report) or +/// [`ExifReader::parse_from_with_report`](crate::ExifReader::parse_from_with_report). In +/// [`strict`](crate::ExifReader::strict) mode the first *malformed* region fails the parse instead +/// of being reported — but a strict report is **not** therefore always empty: +/// [`DroppedRegion::TrailingIfd`] is well-formed and merely unrepresentable, so strictness has no +/// grounds to reject it and it is reported in both modes. +/// +/// Read the module documentation for what this report deliberately does **not** cover: it is not a +/// byte-completeness verdict, and losses inside a single directory belong to [`gamut_ifd`]. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ReadReport { + dropped: Vec, +} + +impl ReadReport { + /// An empty report — nothing discarded. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Every discarded region, in the order the reader met it. + #[must_use] + pub fn dropped(&self) -> &[Dropped] { + &self.dropped + } + + /// Whether any of the regions this report covers was discarded. + /// + /// **Not** a "this parse lost nothing" verdict — see the module documentation. An empty report + /// means no sub-IFD, thumbnail range or trailing directory was dropped; it says nothing about + /// a shadowed duplicate tag (#528) or about source bytes no structure claimed (#521). + #[must_use] + pub fn is_empty(&self) -> bool { + self.dropped.is_empty() + } + + /// Appends a discarded region. + pub(crate) fn record(&mut self, dropped: Dropped) { + self.dropped.push(dropped); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Each region reports the tag that actually addresses it — the value a caller uses to find + /// the pointer back in the source directory — and a region no tag addresses reports `None`. + #[test] + fn each_region_carries_the_tag_that_addresses_it() { + for (region, tag) in [ + (DroppedRegion::ExifIfd, Some(0x8769)), + (DroppedRegion::GpsIfd, Some(0x8825)), + (DroppedRegion::InteropIfd, Some(0xA005)), + (DroppedRegion::ThumbnailJpeg, Some(0x0201)), + (DroppedRegion::TrailingIfd, None), + ] { + assert_eq!( + Dropped::new(region, 0, DropReason::OutOfBounds).tag(), + tag, + "wrong addressing tag for {region:?}" + ); + } + } + + /// The rendered form names the region, the tag, the offset and the reason — the four facts a + /// caller reporting a lossy parse needs, and the strings are part of the contract. + #[test] + fn the_rendered_drop_names_region_tag_offset_and_reason() { + assert_eq!( + Dropped::new(DroppedRegion::GpsIfd, 65_535, DropReason::OutOfBounds).to_string(), + "dropped GPS (tag 0x8825) at offset 65535: addresses bytes outside the EXIF blob" + ); + assert_eq!( + Dropped::new(DroppedRegion::ExifIfd, 26, DropReason::Malformed).to_string(), + "dropped Exif (tag 0x8769) at offset 26: is not a well-formed directory" + ); + assert_eq!( + Dropped::new(DroppedRegion::InteropIfd, 8, DropReason::Malformed).to_string(), + "dropped Interop (tag 0xa005) at offset 8: is not a well-formed directory" + ); + assert_eq!( + Dropped::new(DroppedRegion::ThumbnailJpeg, 1, DropReason::OutOfBounds).to_string(), + "dropped Thumbnail (tag 0x0201) at offset 1: addresses bytes outside the EXIF blob" + ); + assert_eq!( + Dropped::new( + DroppedRegion::ThumbnailJpeg, + 42, + DropReason::ThumbnailLengthMissing + ) + .to_string(), + "dropped Thumbnail (tag 0x0201) at offset 42: has no JPEGInterchangeFormatLength to \ + size the read" + ); + } + + /// A region no tag addresses says so explicitly, in the same grammar as every other drop — + /// rather than claiming tag `0x0000`, which is a real tag number (`GPSVersionID`) and would + /// read as a fact about the source. + #[test] + fn a_drop_with_no_addressing_tag_says_so_in_the_same_grammar() { + assert_eq!( + Dropped::new(DroppedRegion::TrailingIfd, 120, DropReason::Unrepresentable).to_string(), + "dropped TrailingIFD (tag none) at offset 120: parsed cleanly but has no place in the \ + EXIF model" + ); + } + + /// A fresh report is empty and stays consistent with what has been recorded — `is_empty` is + /// the verdict over the covered regions, so it must not be independent of the contents. + #[test] + fn a_report_is_empty_until_something_is_recorded() { + let mut report = ReadReport::new(); + assert!(report.is_empty()); + assert_eq!(report.dropped(), &[]); + + let drop = Dropped::new(DroppedRegion::ExifIfd, 7, DropReason::Malformed); + report.record(drop); + assert!(!report.is_empty()); + assert_eq!(report.dropped(), &[drop]); + assert_eq!(report.dropped()[0].region(), DroppedRegion::ExifIfd); + assert_eq!(report.dropped()[0].offset(), 7); + assert_eq!(report.dropped()[0].reason(), DropReason::Malformed); + } +} diff --git a/crates/gamut-exif/src/stream.rs b/crates/gamut-exif/src/stream.rs new file mode 100644 index 00000000..559849b9 --- /dev/null +++ b/crates/gamut-exif/src/stream.rs @@ -0,0 +1,754 @@ +//! Reading EXIF from a positioned byte source rather than a slice. +//! +//! EXIF is usually small, but the file it is embedded in need not be: a raw `.NEF`/`.CR3` can be +//! hundreds of megabytes whose EXIF is a few kilobytes near the front. [`gamut_ifd`]'s +//! [`IfdReader`] already reads a TIFF stream through the positioned [`ReadAt`] trait, fetching only +//! the directory bodies and the values they reference, so this module lifts +//! [`ExifReader`](crate::ExifReader) onto the same source type: open a file, hand the reader a +//! [`gamut_ifd::StreamSource`], and pull the EXIF out without loading the image. +//! +//! This is the crate's **one** parse engine — [`ExifReader::parse`](crate::ExifReader::parse) is +//! the `&[u8]` case of it (`&[u8]` implements [`ReadAt`]), so the slice and streaming paths cannot +//! drift. It is deliberately synchronous: an async caller drives a [`ReadAt`] source itself, which +//! keeps a runtime dependency out of a crate that has none. + +use gamut_core::ErrorKind; +use gamut_ifd::{Ifd, IfdReader, RawIfd, ReadAt, tags as ifd_tags}; + +use crate::error::{ExifError, Result}; +use crate::exif::{EXIF_IFD_POINTER, Exif, MARKER}; +use crate::reader::ExifReader; +use crate::report::{DropReason, Dropped, DroppedRegion, ReadReport}; +use crate::tag::ExifTag; +use crate::thumbnail::Thumbnail; + +impl ExifReader { + /// Parses EXIF from a positioned byte source, reading only the parts it needs. + /// + /// The streaming twin of [`parse`](Self::parse), which is this method over a `&[u8]`. `source` + /// is taken by value; `&mut S` and `&mut dyn ReadAt` both implement [`ReadAt`], so a caller + /// that must keep its source can pass a borrow, and a caller that needs dynamic dispatch can + /// erase the type. + /// + /// ```no_run + /// use std::fs::File; + /// + /// use gamut_exif::ExifReader; + /// use gamut_ifd::StreamSource; + /// + /// // The EXIF of a large raw file, without reading the image. + /// let mut file = File::open("capture.dng")?; + /// let exif = ExifReader::new().parse_from(StreamSource::new(&mut file))?; + /// println!("{:?}", exif.make()); + /// # Ok::<(), Box>(()) + /// ``` + /// + /// # Errors + /// + /// Returns [`ExifError::MissingMarker`] when the marker is required but absent, an + /// [`ExifError::Ifd`] when the TIFF stream is malformed or the source fails, or (in + /// [`strict`](Self::strict) mode) [`ExifError::InvalidIfd`] / + /// [`ExifError::BadThumbnail`] when a sub-IFD pointer or thumbnail range is unusable. + /// + /// An offset inside an error message is a position in `source` — the byte source the caller + /// handed in — so for a marked source it counts the six-byte `Exif\0\0` marker. That is + /// deliberately a different frame from [`Dropped::offset`](crate::Dropped::offset), which is + /// relative to the start of the TIFF stream and therefore six smaller for the same position. + /// It matters most here: a caller streaming from a file is the one likeliest to correlate an + /// error offset against bytes on disk, and it can do so directly only for the error frame — + /// a report offset must have the source's own start (and the marker) added back first. + pub fn parse_from(&self, source: S) -> Result { + self.parse_source(source, &mut ReadReport::new()) + } + + /// Parses EXIF from a positioned byte source and reports what a lenient parse discarded. + /// + /// The streaming twin of [`parse_with_report`](Self::parse_with_report). See [`ReadReport`]. + /// + /// The two frames of [`parse_from`](Self::parse_from) meet here: every + /// [`Dropped::offset`](crate::Dropped::offset) in the returned report is relative to the start + /// of the TIFF stream, while an offset in a returned [`ExifError`] is a position in `source` + /// and includes any `Exif\0\0` marker. A caller that renders both beside each other must + /// normalise one of them. They *meet* rather than always arrive together: the return is a sum + /// type, so a source whose strict-fatal defect is reached after a reportable one yields the + /// error alone and no report. + /// + /// # Errors + /// + /// As [`parse_from`](Self::parse_from). + pub fn parse_from_with_report(&self, source: S) -> Result<(Exif, ReadReport)> { + let mut report = ReadReport::new(); + let exif = self.parse_source(source, &mut report)?; + Ok((exif, report)) + } + + /// The crate's one parse engine: marker handling, the top-level chain, the thumbnail, and the + /// three pointer-addressed sub-IFDs, recording into `report` whatever leniency discards. + pub(crate) fn parse_source( + &self, + mut source: S, + report: &mut ReadReport, + ) -> Result { + let base = self.tiff_base(&mut source)?; + // Everything below addresses the TIFF stream, so offsets read out of it — and the offsets + // this crate hands back — stay in EXIF's own frame of reference. + let mut reader = IfdReader::open(source.rebased(base))?; + let order = reader.order(); + + let file = reader.read_file()?; + let has_trailing = file.ifds.len() > 2; + let mut ifds = file.ifds.into_iter(); + let mut image = ifds.next().ok_or(ExifError::Truncated)?; + // The next-IFD chain's second entry is the thumbnail directory (1st IFD), if any. + let thumbnail = match ifds.next() { + Some(ifd) => Some(self.read_thumbnail(ifd, &mut reader, report)?), + None => None, + }; + // EXIF defines exactly two top-level directories, so anything further down the chain has + // nowhere to go in the model. Name it rather than letting the iterator drop it silently. + record_trailing_ifds(&mut reader, has_trailing, report)?; + + // The Exif sub-IFD's own offset, captured before `follow` strips the pointer: the + // maker-note pin needs the note value's absolute source position. + let exif_ifd_at = image.get_u32(EXIF_IFD_POINTER).map(u64::from); + let exif = self.follow(&mut image, &mut reader, DroppedRegion::ExifIfd, report)?; + let gps = self.follow(&mut image, &mut reader, DroppedRegion::GpsIfd, report)?; + let maker_note_at = match (&exif, exif_ifd_at) { + (Some(_), Some(at)) => maker_note_offset(&mut reader, at)?, + _ => None, + }; + + // The Interoperability directory is reached from *inside* the Exif sub-IFD, not the 0th IFD. + let (exif, interop) = match exif { + Some(mut e) => { + let interop = + self.follow(&mut e, &mut reader, DroppedRegion::InteropIfd, report)?; + (Some(e), interop) + } + None => (None, None), + }; + + Ok(Exif::from_parts( + order, + image, + exif, + gps, + interop, + thumbnail, + maker_note_at, + )) + } + + /// The offset at which the TIFF stream starts in `source`: past the `Exif\0\0` marker when it + /// is there, 0 when it is not. + /// + /// A source too short to hold the marker is treated as unmarked — the TIFF header that follows + /// is longer than the marker, so such a source cannot parse either way. + fn tiff_base(&self, source: &mut S) -> Result { + let mut head = [0u8; MARKER.len()]; + let marked = match source.read_exact_at(0, &mut head) { + Ok(()) => head.as_slice() == MARKER, + // Too few bytes to hold a marker: unmarked. Keyed on the error *kind*, never on the + // source's length, so a slice keeps its exact behaviour while a transport failure + // (a disk error, a dropped network mount) is not misread as "no marker". + Err(e) if e.kind() == ErrorKind::InvalidInput => false, + Err(e) => return Err(e.into()), + }; + if marked { + Ok(MARKER.len() as u64) + } else if self.require_marker { + Err(ExifError::MissingMarker) + } else { + Ok(0) + } + } + + /// Reads `region`'s pointer tag from `parent` and parses the sub-IFD it addresses, then removes + /// the pointer (it is represented structurally, not as a data field). + /// + /// Returns `Ok(None)` when the pointer is absent, or — in lenient mode — when the pointed-at + /// directory is malformed, in which case the drop is recorded in `report`. Either way the + /// pointer is removed: preserving it would change what `to_bytes` emits for a malformed blob, + /// which is beyond this crate's remit here — #419's "the pointer was lost too" is answered by + /// *naming* the tag and offset in the report, not by keeping the entry. + /// + /// A failure that is not the *bytes* being wrong — a source whose transport failed — is + /// propagated unchanged in both modes. Leniency exists to tolerate corrupt files, and + /// reporting a structurally perfect directory as malformed because a disk read failed would be + /// a lie about the file. + fn follow( + &self, + parent: &mut Ifd, + reader: &mut IfdReader, + region: DroppedRegion, + report: &mut ReadReport, + ) -> Result> { + // A region no tag addresses has, by definition, no pointer to follow — so `Ok(None)` is + // the answer, not a special case. Today `follow` is only ever called for the three + // pointer-addressed sub-IFDs, all of which have one. + let Some(ptr) = region.tag() else { + return Ok(None); + }; + let Some(offset) = parent.get_u32(ptr) else { + return Ok(None); + }; + let offset = u64::from(offset); + let followed = match reader.read_ifd(offset) { + Ok(raw) => reader.decode_ifd(&raw), + Err(e) => Err(e), + }; + parent.remove(ptr); + match followed { + Ok(ifd) => Ok(Some(ifd)), + // Not the file's fault: hand the transport failure back untouched. + Err(e) if e.kind() != ErrorKind::InvalidInput => Err(e.into()), + Err(_) if self.strict => Err(ExifError::InvalidIfd(region.name())), + Err(_) => { + let reason = address_reason(reader, offset)?; + report.record(Dropped::new(region, offset, reason)); + Ok(None) + } + } + } + + /// Builds a [`Thumbnail`] from the 1st IFD, fetching its JPEG bytes (from the + /// `JPEGInterchangeFormat` offset / length) when the range is wholly inside the stream. In + /// lenient mode an unusable range yields a thumbnail without bytes and a recorded drop; in + /// strict mode it errors. + /// + /// An offset with no `JPEGInterchangeFormatLength` beside it addresses bytes nothing can size, + /// so it is a loss rather than an absent thumbnail — the JPEG behind the offset is unreadable. + /// A length with no offset addresses nothing at all, so nothing was dropped and nothing is + /// reported. Both halves of that rule are structural and apply whatever the thumbnail's + /// `Compression` says: Exif 3.0 §4.6.9.2 Table 21 gives the pair's support level *per + /// thumbnail-format column* (mandatory under **Compressed**, `N` — not allowed to record — + /// under all three uncompressed ones); that axis is not the two-valued `Compression` tag, which + /// this reader does not consult. Whether it should, and whether the length-only case should be + /// rejected for symmetry, is issue #574. + fn read_thumbnail( + &self, + ifd: Ifd, + reader: &mut IfdReader, + report: &mut ReadReport, + ) -> Result { + let ptr = ExifTag::JpegInterchangeFormat.tag_id(); + let offset = ifd.get_u32(ptr); + let length = ifd.get_u32(ExifTag::JpegInterchangeFormatLength.tag_id()); + let jpeg = match (offset, length) { + (Some(offset), Some(length)) => match read_range(reader, offset, length)? { + Some(bytes) => Some(bytes), + None if self.strict => { + return Err(ExifError::BadThumbnail("JPEG offset out of bounds")); + } + None => { + report.record(Dropped::new( + DroppedRegion::ThumbnailJpeg, + u64::from(offset), + DropReason::OutOfBounds, + )); + None + } + }, + (Some(_), None) if self.strict => { + // States the structural fact — a range with no size — rather than naming a missing + // mandatory tag: Table 21 makes the sibling mandatory only under `Compression = + // Compressed`, and forbids recording it at all under the uncompressed columns. + return Err(ExifError::BadThumbnail( + "JPEGInterchangeFormat offset with no length to size it", + )); + } + (Some(offset), None) => { + report.record(Dropped::new( + DroppedRegion::ThumbnailJpeg, + u64::from(offset), + DropReason::ThumbnailLengthMissing, + )); + None + } + (None, _) => None, + }; + // The JPEGInterchangeFormat offset is structural — the bytes are captured above and the + // writer re-synthesises the offset — so drop it from the stored directory (mirroring how the + // sub-IFD pointer tags are stripped), leaving a value the model can't carry stale. + // + // The removal is conditioned on bytes having been read, which is #548: when `jpeg` is + // `None` the pointer survives into the model and `to_bytes` re-emits it, so the emitted + // blob claims a thumbnail the report says was dropped. #548 names only the OutOfBounds + // case; the ThumbnailLengthMissing arm above is a SECOND instance of it. What separates + // them is not that a strict parse of the re-emitted blob fails — it fails for BOTH, since + // an out-of-bounds offset survives the round trip just as an unsized one does — but that + // the OutOfBounds instance is pre-existing (`master` already rejects it strictly) while + // this one is created by adding the strict arm. Fixing the condition is a + // writer behaviour change and belongs to #548, not here. + let mut ifd = ifd; + if jpeg.is_some() { + ifd.remove(ptr); + } + Ok(Thumbnail::from_parts(ifd, jpeg)) + } +} + +/// Records the top-level directories past the 1st IFD, which parse cleanly but have nowhere to go +/// in the [`Exif`] model. +/// +/// The offsets come from a second walk of the next-IFD chain, which is the single source of truth +/// for *which* directories are trailing — `has_trailing` only says whether the walk is worth +/// starting. That costs a re-read of the directory bodies, so it runs **only** when there is +/// something to report: a well-formed EXIF blob has one or two directories and never reaches it, +/// leaving the lazy read bound untouched. +/// +/// A source that dies between the two walks turns a would-be success into an error. That is the +/// same rule the rest of this module follows — a transport failure is propagated, never swallowed — +/// and swallowing it only here would be the inconsistency. +fn record_trailing_ifds( + reader: &mut IfdReader, + has_trailing: bool, + report: &mut ReadReport, +) -> Result<()> { + if !has_trailing { + return Ok(()); + } + let mut offsets = Vec::new(); + for raw in reader.ifds().skip(2) { + offsets.push(raw?.offset); + } + for offset in offsets { + report.record(Dropped::new( + DroppedRegion::TrailingIfd, + offset, + DropReason::Unrepresentable, + )); + } + Ok(()) +} + +/// Why an address that failed to parse failed: past the end of the stream, or inside it but +/// structurally bad. Separating the two is what makes a report actionable — a dangling pointer is +/// a different defect from a corrupt directory. +fn address_reason(reader: &mut IfdReader, offset: u64) -> Result { + let len = reader.source_mut().len()?; + if offset < len { + Ok(DropReason::Malformed) + } else { + Ok(DropReason::OutOfBounds) + } +} + +/// Fetches `length` bytes at `offset`, or `None` when that range is not wholly inside the stream. +/// +/// The bound is checked against the source's length *before* anything is allocated, so a hostile +/// `JPEGInterchangeFormatLength` cannot make the reader reserve more than the stream can hold. +fn read_range( + reader: &mut IfdReader, + offset: u32, + length: u32, +) -> Result>> { + // Widened to 64 bits first: the sum of two `u32`s cannot overflow a `u64`. + let end = u64::from(offset) + u64::from(length); + if end > reader.source_mut().len()? { + return Ok(None); + } + let mut buf = vec![0u8; length as usize]; + reader + .source_mut() + .read_exact_at(u64::from(offset), &mut buf)?; + Ok(Some(buf)) +} + +/// The absolute offset of the Exif sub-IFD's out-of-line `MakerNote` value in the TIFF stream, or +/// `None` if the note is absent or inline. +/// +/// A failure here is propagated rather than folded into `None`. The offset is what +/// [`ExifWriter`](crate::ExifWriter) uses to *pin* the note in place on a rewrite, so losing it +/// silently re-emits a vendor MakerNote unpinned — wrong bytes, no error, and nothing in the +/// report. No later read is guaranteed to resurface the failure either: `follow` returns before +/// reading at all when the GPS and Interop pointers are absent, which is the common case. +fn maker_note_offset( + reader: &mut IfdReader, + exif_ifd_at: u64, +) -> Result> { + // Every error propagates, with no lenient arm — deliberately. This is reached only when + // `follow` has already read and decoded this exact directory at this exact offset, so a + // *deterministic* source cannot fail here for a reason the bytes explain: over such a source a + // malformed-input arm is unreachable, and an unreachable arm is a branch no test can falsify. + // + // It is not unreachable in general. A `ReadAt` may answer differently on a second read — a file + // rewritten underneath the reader is precisely the case `parse_from` exists to enable — and then + // the directory really can fail here. A hard error is still the right answer for it: the pin is + // what keeps a vendor MakerNote's TIFF-absolute internal offsets valid on a rewrite, so + // continuing would re-emit the note unpinned, with wrong bytes and nothing in the report. + let raw: RawIfd = reader.read_ifd(exif_ifd_at)?; + let Some(entry) = raw.entry(ifd_tags::MAKER_NOTE) else { + return Ok(None); + }; + Ok(reader.value_offset(entry)) +} + +#[cfg(test)] +mod tests { + use std::io::Cursor; + + use gamut_ifd::{ByteOrder, StreamSource, TiffFile, Value, Variant, write}; + + use super::*; + use crate::exif::{GPS_IFD_POINTER, INTEROP_IFD_POINTER}; + + /// A minimal marked EXIF blob whose 0th IFD carries `Make`. + fn blob() -> Vec { + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); + let tiff = write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds: vec![image], + }) + .expect("write"); + let mut out = MARKER.to_vec(); + out.extend(tiff); + out + } + + /// The streaming entry point reads a `Read + Seek` source that is not a slice at all — the + /// capability the slice-only API could not offer. + #[test] + fn parse_from_reads_a_seekable_stream() { + let source = StreamSource::new(Cursor::new(blob())); + let exif = ExifReader::new().parse_from(source).expect("parse_from"); + assert_eq!(exif.make(), Some("Canon")); + } + + /// The marker is detected through the source, not by slicing: the same `require_marker` + /// contract `parse` has must hold for a stream, or the two entry points disagree. + #[test] + fn the_marker_is_detected_through_the_source() { + let marked = blob(); + let bare = marked[MARKER.len()..].to_vec(); + + assert!( + ExifReader::new() + .require_marker(true) + .parse_from(StreamSource::new(Cursor::new(marked))) + .is_ok() + ); + let err = ExifReader::new() + .require_marker(true) + .parse_from(StreamSource::new(Cursor::new(bare.clone()))) + .expect_err("a bare TIFF stream must be rejected"); + assert!(matches!(err, ExifError::MissingMarker), "{err:?}"); + // ...and without the requirement the same bare stream parses. + assert!( + ExifReader::new() + .parse_from(StreamSource::new(Cursor::new(bare))) + .is_ok() + ); + } + + /// A source shorter than the 6-byte marker cannot be read for one; it is unmarked, and + /// `require_marker` says so rather than reporting a torn read. + #[test] + fn a_source_too_short_for_the_marker_is_unmarked() { + let err = ExifReader::new() + .require_marker(true) + .parse_from(&b"Exi"[..]) + .expect_err("three bytes cannot carry the marker"); + assert!(matches!(err, ExifError::MissingMarker), "{err:?}"); + } + + /// A `ReadAt` whose *transport* fails after `budget` successful reads — a disk error, a + /// dropped network mount. Distinct from a source whose bytes are merely wrong: `gamut-core` + /// maps the former to `Error::Io` and the latter to `Error::InvalidInput`. + struct FailingAfter { + inner: S, + budget: usize, + } + + impl ReadAt for FailingAfter { + fn read_exact_at(&mut self, offset: u64, buf: &mut [u8]) -> gamut_core::Result<()> { + let Some(left) = self.budget.checked_sub(1) else { + return Err(gamut_core::Error::Io(std::io::Error::other( + "transport lost", + ))); + }; + self.budget = left; + self.inner.read_exact_at(offset, buf) + } + + fn len(&mut self) -> gamut_core::Result { + self.inner.len() + } + } + + /// A structurally perfect blob with both 0th-IFD sub-directories and a nested Interop, so + /// every sub-IFD drop path is reachable and none of them *should* fire. + fn healthy_blob() -> Vec { + let mut interop = Ifd::new(); + interop.set(0x0001, Value::Ascii("R98".into())); + let mut exif = Ifd::new(); + exif.set(0x829D, Value::Rational(vec![(28, 10)])); + exif.set_sub_ifd(INTEROP_IFD_POINTER, vec![interop]); + let mut gps = Ifd::new(); + gps.set(0x0000, Value::Byte(vec![2, 3, 0, 0])); + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); + image.set_sub_ifd(EXIF_IFD_POINTER, vec![exif]); + image.set_sub_ifd(GPS_IFD_POINTER, vec![gps]); + let tiff = write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds: vec![image], + }) + .expect("write"); + let mut out = MARKER.to_vec(); + out.extend(tiff); + out + } + + /// The bytes `deep_blob`'s thumbnail range addresses, appended past the directories. + const THUMB_BYTES: &[u8] = b"\xFF\xD8__jpg\xFF\xD9"; + /// A `JPEGInterchangeFormat` value patched to the real offset once the layout is known — + /// `write` lays the directories out, so the payload's position is not knowable before it runs. + const THUMB_SENTINEL: u32 = 0xDEAD_BEEF; + + /// A structurally perfect blob reaching every read site `healthy_blob` cannot. + /// + /// `healthy_blob` has no `MakerNote`, no thumbnail bytes and no trailing directory, and its + /// Interop pointer means a failure inside `maker_note_offset` is always resurfaced by the later + /// Interop read. Here the GPS and Interop pointers are both absent — so `follow` returns without + /// reading at all — while three additions each open one otherwise-unswept read: an out-of-line + /// `MakerNote` gives the pin an offset a caller can lose, an in-bounds thumbnail range makes + /// `read_range` fetch, and a third top-level directory makes `record_trailing_ifds` re-walk the + /// chain. + fn deep_blob() -> Vec { + let mut exif = Ifd::new(); + exif.set(0x829A, Value::Rational(vec![(1, 250)])); // ExposureTime + // Nine bytes: too wide to sit inline in the entry, so it has a real source offset. + exif.set( + ifd_tags::MAKER_NOTE, + Value::Undefined(b"Canon\0\0\0\0".to_vec()), + ); + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); + image.set_sub_ifd(EXIF_IFD_POINTER, vec![exif]); + + let mut thumb = Ifd::new(); + thumb.set(ExifTag::Compression.tag_id(), Value::Short(vec![6])); + thumb.set( + ExifTag::JpegInterchangeFormat.tag_id(), + Value::Long(vec![THUMB_SENTINEL]), + ); + thumb.set( + ExifTag::JpegInterchangeFormatLength.tag_id(), + Value::Long(vec![THUMB_BYTES.len() as u32]), + ); + + let mut trailing = Ifd::new(); + trailing.set(0x0131, Value::Ascii("trailing".into())); // Software + + let mut tiff = write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds: vec![image, thumb, trailing], + }) + .expect("write"); + + // The payload goes after the directories, so the sentinel is patched to where it lands. + let at = u32::try_from(tiff.len()).expect("the fixture fits in 32 bits"); + let sentinel = THUMB_SENTINEL.to_le_bytes(); + let hits: Vec = tiff + .windows(4) + .enumerate() + .filter(|(_, w)| *w == sentinel) + .map(|(i, _)| i) + .collect(); + assert_eq!( + hits.len(), + 1, + "the sentinel must name exactly one value field" + ); + tiff[hits[0]..hits[0] + 4].copy_from_slice(&at.to_le_bytes()); + tiff.extend_from_slice(THUMB_BYTES); + + let mut out = MARKER.to_vec(); + out.extend(tiff); + out + } + + /// The deep fixture really carries every loss the sweep below claims to watch for. + /// + /// Without this the sweep could pass against a blob that never had a pin, thumbnail bytes or a + /// trailing directory to lose — and against a clean report that already blamed the file, which + /// would make the sweep's report-equality law vacuous. + #[test] + fn the_deep_fixture_has_a_pin_a_thumbnail_and_a_trailing_directory_to_lose() { + let data = deep_blob(); + let (exif, report) = ExifReader::new() + .parse_from_with_report(&data[..]) + .expect("parse"); + assert!( + exif.maker_note_offset().is_some(), + "the fixture must pin an out-of-line MakerNote" + ); + assert_eq!( + exif.thumbnail().and_then(Thumbnail::jpeg), + Some(THUMB_BYTES), + "the fixture must have thumbnail bytes that were really fetched" + ); + assert!( + exif.gps_ifd().is_none(), + "no GPS pointer to rescue a failure" + ); + assert!( + exif.interop_ifd().is_none(), + "no Interop pointer to rescue a failure" + ); + assert_eq!(report.dropped().len(), 1, "{:?}", report.dropped()); + assert_eq!(report.dropped()[0].region(), DroppedRegion::TrailingIfd); + assert_eq!( + report.dropped()[0].reason(), + DropReason::Unrepresentable, + "the clean report must blame nothing on the file" + ); + } + + /// A failing source is propagated, never reported as a malformed file. + /// + /// Leniency exists to tolerate corrupt *bytes*. If the transport fails instead, the data may be + /// perfect, so silently returning `Ok` with the sub-IFDs missing — and a report blaming the + /// file — would be a lie, and the worst case is the network-backed source this entry point + /// exists to enable. The whole parse is swept one read at a time over two fixtures, between + /// them reaching every site that reads bytes: the marker probe, the header, each directory + /// body, each out-of-line value, the three sub-IFD reads (`healthy_blob`), and the thumbnail + /// fetch, the trailing-chain re-walk and the maker-note pin (`deep_blob`). The one `ReadAt` + /// method left unswept is `len`, which answers a length rather than reading bytes. + /// + /// The law is that an `Ok` from a failing source is the **whole** answer — equal to the clean + /// parse in report, pin and thumbnail bytes — which catches both shapes a silent loss takes: a + /// spurious report entry blaming the file, or, as the maker-note pin did, an `Exif` quietly + /// missing something with nothing in the report at all. Equality against the clean report + /// rather than emptiness is what lets `deep_blob` be swept at all: its trailing directory is a + /// legitimate drop that a clean parse reports too. + #[test] + fn a_failing_source_is_propagated_not_reported_as_a_malformed_file() { + for (name, data) in [("healthy", healthy_blob()), ("deep", deep_blob())] { + let (clean, clean_report) = ExifReader::new() + .parse_from_with_report(&data[..]) + .expect("clean parse"); + let (mut failures, mut successes) = (0, 0); + for budget in 0..40 { + let source = FailingAfter { + inner: &data[..], + budget, + }; + match ExifReader::new().parse_from_with_report(source) { + Ok((exif, report)) => { + successes += 1; + assert_eq!( + report, clean_report, + "{name} budget {budget}: a transport failure changed the report" + ); + assert_eq!( + exif.maker_note_offset(), + clean.maker_note_offset(), + "{name} budget {budget}: the maker-note pin was silently lost" + ); + assert_eq!( + exif.thumbnail().and_then(Thumbnail::jpeg), + clean.thumbnail().and_then(Thumbnail::jpeg), + "{name} budget {budget}: the thumbnail bytes were silently lost" + ); + } + Err(ExifError::Ifd(e)) => { + failures += 1; + assert_eq!( + e.kind(), + ErrorKind::Io, + "{name} budget {budget}: a transport failure must keep its kind" + ); + } + // `MissingMarker` here would mean the marker probe swallowed the error and + // decided the blob was unmarked; anything else is equally a misdiagnosis. + Err(other) => { + panic!("{name} budget {budget}: transport failure became {other:?}") + } + } + } + assert!( + failures > 0 && successes > 0, + "{name}: the sweep proved nothing" + ); + } + } + + /// A transport failure while probing for the marker is not a *missing* marker. + /// + /// The marker probe is the one read that happens before any parsing, and its result is a + /// three-way question — marked, unmarked, or unknown — collapsed onto a boolean. Deciding + /// "unmarked" from a failed read makes `require_marker(true)` answer `MissingMarker` for a blob + /// that may well carry one, which sends a caller to the wrong conclusion entirely. The split is + /// keyed on the error kind, so a short slice still reads as genuinely unmarked — which + /// `a_source_too_short_for_the_marker_is_unmarked` pins. + #[test] + fn a_transport_failure_probing_the_marker_is_not_a_missing_marker() { + let data = healthy_blob(); + let source = FailingAfter { + inner: &data[..], + budget: 0, + }; + let err = ExifReader::new() + .require_marker(true) + .parse_from(source) + .expect_err("a source that cannot be read must not parse"); + match err { + ExifError::Ifd(e) => assert_eq!( + e.kind(), + ErrorKind::Io, + "the transport failure must keep its kind" + ), + other => panic!("transport failure became {other:?}"), + } + } + + /// `address_reason` splits the two defects the report distinguishes, and the boundary is the + /// stream's length itself: the last byte is inside, the length is not. + #[test] + fn an_address_is_out_of_bounds_from_the_streams_length_onwards() { + let data = [0u8; 8]; + let mut reader = + IfdReader::with_layout(&data[..], ByteOrder::LittleEndian, Variant::Classic); + assert_eq!( + address_reason(&mut reader, 7).expect("reason"), + DropReason::Malformed, + "the last byte of the stream is inside it" + ); + assert_eq!( + address_reason(&mut reader, 8).expect("reason"), + DropReason::OutOfBounds, + "one past the last byte is outside it" + ); + } + + /// A range is fetched only when it ends inside the stream — the bound that stops a hostile + /// length from being allocated. The boundary case (a range ending exactly at the end) is the + /// one an off-by-one would get wrong. + #[test] + fn a_range_is_fetched_only_when_it_ends_inside_the_stream() { + let data = [1u8, 2, 3, 4, 5, 6, 7, 8]; + let mut reader = + IfdReader::with_layout(&data[..], ByteOrder::LittleEndian, Variant::Classic); + assert_eq!( + read_range(&mut reader, 4, 4).expect("read"), + Some(vec![5, 6, 7, 8]), + "a range ending exactly at the end is inside" + ); + assert_eq!( + read_range(&mut reader, 4, 5).expect("read"), + None, + "one byte past the end is not" + ); + assert_eq!( + read_range(&mut reader, 0, u32::MAX).expect("read"), + None, + "a hostile length is refused before it is allocated" + ); + } +} diff --git a/crates/gamut-exif/tests/report.rs b/crates/gamut-exif/tests/report.rs new file mode 100644 index 00000000..2ab9f375 --- /dev/null +++ b/crates/gamut-exif/tests/report.rs @@ -0,0 +1,450 @@ +//! The honesty contract for the lenient reader: nothing is discarded without being named. +//! +//! [`ExifReader`] drops a malformed sub-IFD or an out-of-bounds thumbnail range so the rest of a +//! real-world blob still parses. That leniency is only defensible if a caller can *ask* what it +//! cost — otherwise a blob that never carried GPS and one whose GPS pointer was dangling are +//! indistinguishable. Each test below feeds one deliberately broken blob to +//! [`ExifReader::parse_with_report`] and pins that the discarded region is named with the tag that +//! addressed it, the offset it carried, and a reason that separates "nothing could be there" from +//! "something was there and it was corrupt". Three further tests pin the contract's edges: that a +//! *strict* report is not empty of the one loss strictness has no grounds to reject, that the +//! report's offsets and the crate's error offsets are deliberately in different frames, and — over +//! a truncation sweep — that no sub-IFD is ever dropped without being named. + +use gamut_exif::{DropReason, DroppedRegion, ExifReader}; +use gamut_ifd::{ByteOrder, Ifd, IfdReader, TiffFile, Value, Variant, write}; + +/// `ExifIFD` pointer (Exif 3.0 §4.6.3). +const EXIF_IFD: u16 = 0x8769; +/// `GPSInfo` pointer. +const GPS_INFO: u16 = 0x8825; +/// `Interoperability` pointer, which lives inside the Exif sub-IFD. +const INTEROP_IFD: u16 = 0xA005; +/// `JPEGInterchangeFormat` — the 1st IFD's thumbnail offset. +const THUMB_OFFSET: u16 = 0x0201; +/// `JPEGInterchangeFormatLength`. +const THUMB_LENGTH: u16 = 0x0202; +/// An offset far past the end of any fixture here. +const DANGLING: u32 = 0xFFFF; +/// The `Exif\0\0` marker a JPEG `APP1` payload carries before the TIFF stream. +const MARKER: &[u8] = b"Exif\x00\x00"; + +/// Serialises `ifds` as a bare little-endian TIFF stream. +fn tiff(ifds: Vec) -> Vec { + write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds, + }) + .expect("write") +} + +/// A 0th IFD carrying `Make`, so every fixture has one tag that must survive the drop. +fn image_ifd() -> Ifd { + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); + image +} + +/// A dangling Exif, GPS or Interop pointer is reported with the tag that addressed it, the offset +/// it carried, and `OutOfBounds` — while the rest of the blob still parses. +#[test] +fn a_dangling_sub_ifd_pointer_is_named_with_its_tag_offset_and_reason() { + for (tag, region) in [ + (EXIF_IFD, DroppedRegion::ExifIfd), + (GPS_INFO, DroppedRegion::GpsIfd), + (INTEROP_IFD, DroppedRegion::InteropIfd), + ] { + // The Interop pointer is reached from inside the Exif sub-IFD, not the 0th IFD. + let mut image = image_ifd(); + if tag == INTEROP_IFD { + let mut exif = Ifd::new(); + exif.set(INTEROP_IFD, Value::Long(vec![DANGLING])); + image.set_sub_ifd(EXIF_IFD, vec![exif]); + } else { + image.set(tag, Value::Long(vec![DANGLING])); + } + + let (exif, report) = ExifReader::new() + .parse_with_report(&tiff(vec![image])) + .expect("a dangling pointer must not fail a lenient parse"); + + assert_eq!(exif.make(), Some("Canon"), "the rest of the blob survives"); + assert_eq!( + report.dropped().len(), + 1, + "exactly one region was dropped for {region:?}: {:?}", + report.dropped() + ); + let dropped = report.dropped()[0]; + assert_eq!(dropped.region(), region); + assert_eq!( + dropped.tag(), + Some(tag), + "named by the tag that addressed it" + ); + assert_eq!(dropped.offset(), u64::from(DANGLING)); + assert_eq!(dropped.reason(), DropReason::OutOfBounds); + } +} + +/// A pointer that lands *inside* the blob but on bytes that are not a directory is `Malformed`, +/// not `OutOfBounds` — the two are different defects, and a report that conflated them would send +/// a caller looking in the wrong place. +#[test] +fn an_in_bounds_pointer_to_corrupt_bytes_is_reported_as_malformed() { + let mut image = image_ifd(); + // Offset 1 is inside the stream but straddles the byte-order mark and the magic, so the entry + // count read there is nonsense (0x2A49 entries) and the directory cannot be read. + image.set(EXIF_IFD, Value::Long(vec![1])); + let bytes = tiff(vec![image]); + assert!(bytes.len() > 1, "offset 1 must really be inside the stream"); + + let (exif, report) = ExifReader::new() + .parse_with_report(&bytes) + .expect("lenient parse"); + + assert!(exif.exif_ifd().is_none(), "the sub-IFD was dropped"); + assert_eq!(report.dropped().len(), 1, "{:?}", report.dropped()); + assert_eq!(report.dropped()[0].reason(), DropReason::Malformed); + assert_eq!(report.dropped()[0].offset(), 1); +} + +/// A thumbnail whose JPEG range runs past the end of the blob loses its bytes, and the loss is +/// named — the thumbnail's own directory survives, so without the report the missing bytes look +/// like a thumbnail that never had any. +#[test] +fn an_out_of_bounds_thumbnail_range_is_named() { + let mut thumb = Ifd::new(); + thumb.set(THUMB_OFFSET, Value::Long(vec![DANGLING])); + thumb.set(THUMB_LENGTH, Value::Long(vec![16])); + + let (exif, report) = ExifReader::new() + .parse_with_report(&tiff(vec![image_ifd(), thumb])) + .expect("lenient parse"); + + let thumbnail = exif.thumbnail().expect("the 1st IFD is still a thumbnail"); + assert_eq!( + thumbnail.jpeg(), + None, + "no bytes, rather than bytes from nowhere" + ); + assert_eq!(report.dropped().len(), 1, "{:?}", report.dropped()); + let dropped = report.dropped()[0]; + assert_eq!(dropped.region(), DroppedRegion::ThumbnailJpeg); + assert_eq!(dropped.tag(), Some(THUMB_OFFSET)); + assert_eq!(dropped.offset(), u64::from(DANGLING)); + assert_eq!(dropped.reason(), DropReason::OutOfBounds); +} + +/// A blob that parses in full reports nothing. +/// +/// `is_empty` is the verdict over the regions the report *covers* — deliberately not "this parse +/// lost nothing", which is a stronger claim this crate cannot make (see the `report` module). What +/// it must still guarantee is the direction tested here: a healthy file names nothing, or the +/// signal would be noise. +#[test] +fn a_well_formed_blob_reports_no_drops() { + let mut interop = Ifd::new(); + interop.set(0x0001, Value::Ascii("R98".into())); + let mut exif = Ifd::new(); + exif.set(0x829D, Value::Rational(vec![(28, 10)])); + exif.set_sub_ifd(INTEROP_IFD, vec![interop]); + let mut gps = Ifd::new(); + gps.set(0x0000, Value::Byte(vec![2, 3, 0, 0])); + + let mut image = image_ifd(); + image.set_sub_ifd(EXIF_IFD, vec![exif]); + image.set_sub_ifd(GPS_INFO, vec![gps]); + + let (parsed, report) = ExifReader::new() + .parse_with_report(&tiff(vec![image])) + .expect("parse"); + + assert!(parsed.exif_ifd().is_some()); + assert!(parsed.gps_ifd().is_some()); + assert!(parsed.interop_ifd().is_some()); + assert!( + report.is_empty(), + "healthy blob reported {:?}", + report.dropped() + ); +} + +/// A top-level directory past the 1st IFD is named rather than silently discarded. +/// +/// EXIF defines exactly two — the 0th (primary image) and the 1st (thumbnail) — so a longer +/// next-IFD chain parses cleanly and then has nowhere to go in the model. That is a real loss (the +/// bytes do not survive `to_bytes`), and it is the one drop with no addressing tag: the chain is +/// followed through the structural next-IFD pointer, so the reported tag is `None` and the reported +/// offset is the directory's own position. +#[test] +fn a_top_level_directory_past_the_thumbnail_is_named() { + for extra in 1..=2 { + let mut thumb = Ifd::new(); + thumb.set(0x0103, Value::Short(vec![6])); // Compression = JPEG + + let mut ifds = vec![image_ifd(), thumb]; + for n in 0..extra { + let mut trailing = Ifd::new(); + trailing.set(0x0131, Value::Ascii(format!("trailing {n}"))); // Software + ifds.push(trailing); + } + let bytes = tiff(ifds); + + let (exif, report) = ExifReader::new() + .parse_with_report(&bytes) + .expect("a long chain must still parse"); + assert_eq!(exif.make(), Some("Canon"), "the 0th IFD survives"); + assert!(exif.thumbnail().is_some(), "the 1st IFD survives"); + + // Where those directories actually sit, read back independently of the model. + let mut raw = IfdReader::open(&bytes[..]).expect("open"); + let offsets: Vec = raw + .ifds() + .map(|ifd| ifd.expect("chain link").offset) + .collect(); + assert_eq!( + offsets.len(), + 2 + extra, + "the fixture really has a long chain" + ); + + assert_eq!( + report.dropped().len(), + extra, + "every trailing directory is named: {:?}", + report.dropped() + ); + for (dropped, expected) in report.dropped().iter().zip(&offsets[2..]) { + assert_eq!(dropped.region(), DroppedRegion::TrailingIfd); + assert_eq!( + dropped.tag(), + None, + "no tag addresses a top-level directory" + ); + assert_eq!(dropped.offset(), *expected, "named at its own position"); + assert_eq!(dropped.reason(), DropReason::Unrepresentable); + } + } +} + +/// The law, over a truncation sweep: whenever a lenient parse succeeds, a sub-IFD pointer that was +/// present in the source has either been followed into the model or been named in the report — +/// never silently missing. +/// +/// Truncation is the cheapest generator of *varied* corruption: each prefix breaks a different +/// structure (a value, a directory body, a pointer target), so the sweep reaches drop paths no +/// hand-written fixture enumerates. +#[test] +fn a_truncated_blob_never_drops_a_sub_ifd_without_naming_it() { + let mut interop = Ifd::new(); + interop.set(0x0001, Value::Ascii("R98".into())); + let mut exif = Ifd::new(); + exif.set(0x829D, Value::Rational(vec![(28, 10)])); + exif.set_sub_ifd(INTEROP_IFD, vec![interop]); + let mut gps = Ifd::new(); + gps.set(0x0000, Value::Byte(vec![2, 3, 0, 0])); + let mut image = image_ifd(); + image.set_sub_ifd(EXIF_IFD, vec![exif]); + image.set_sub_ifd(GPS_INFO, vec![gps]); + let full = tiff(vec![image]); + + let mut parses = 0_usize; + let mut drops = 0_usize; + for end in 0..full.len() { + let data = &full[..end]; + let Ok((parsed, report)) = ExifReader::new().parse_with_report(data) else { + continue; + }; + parses += 1; + + // What the source actually carried, read back independently of the model. + let Ok(mut raw_reader) = IfdReader::open(data) else { + continue; + }; + let first = raw_reader.first_ifd_offset(); + let Ok(raw) = raw_reader.read_ifd(first) else { + continue; + }; + + for (tag, followed) in [ + (EXIF_IFD, parsed.exif_ifd().is_some()), + (GPS_INFO, parsed.gps_ifd().is_some()), + ] { + if raw.entry(tag).is_none() { + continue; + } + let named = report.dropped().iter().any(|d| d.tag() == Some(tag)); + assert_ne!( + followed, named, + "at truncation {end}, tag {tag:#06x} was followed={followed} and named={named}" + ); + drops += usize::from(named); + } + } + + assert!( + parses > 0, + "no truncation parsed — the sweep proved nothing" + ); + assert!( + drops > 0, + "no truncation dropped a sub-IFD — the sweep proved nothing" + ); +} + +/// A thumbnail offset with no length beside it is named rather than silently ignored. +/// +/// An offset with no `JPEGInterchangeFormatLength` is not "no thumbnail" — it is an address with +/// nothing to size the read by, and the JPEG behind it is lost. Before this the pair fell into the +/// reader's catch-all `None` arm: no bytes, no error, no report entry, inside the very region this +/// report claims completeness over. The rule is structural, not a support level: Exif 3.0 §4.6.9.2 +/// Table 21 states the pair's level per thumbnail-format column, an axis that is not the two-valued +/// `Compression` tag — which this reader does not consult at all: `Thumbnail::compression` exposes +/// it, but nothing in the parse branches on it (issue #574) — so the fixture's `Compression` value +/// is scene-setting, not the trigger. +#[test] +fn a_thumbnail_offset_without_a_length_is_named() { + let mut thumb = Ifd::new(); + thumb.set(0x0103, Value::Short(vec![6])); // Compression = JPEG + thumb.set(THUMB_OFFSET, Value::Long(vec![4])); // ...in bounds, so not OutOfBounds + // ...and deliberately no THUMB_LENGTH. + + let (exif, report) = ExifReader::new() + .parse_with_report(&tiff(vec![image_ifd(), thumb])) + .expect("lenient parse"); + + assert_eq!( + exif.thumbnail().and_then(|t| t.jpeg()), + None, + "there is no length, so there are no bytes" + ); + assert_eq!(report.dropped().len(), 1, "{:?}", report.dropped()); + let dropped = report.dropped()[0]; + assert_eq!(dropped.region(), DroppedRegion::ThumbnailJpeg); + assert_eq!(dropped.tag(), Some(THUMB_OFFSET)); + assert_eq!(dropped.offset(), 4, "named at the offset the tag carried"); + assert_eq!( + dropped.reason(), + DropReason::ThumbnailLengthMissing, + "not OutOfBounds — the address is inside the blob; the length is what is missing" + ); +} + +/// A thumbnail with neither JPEG tag is an uncompressed thumbnail, not a loss, and reports nothing. +/// +/// The other direction of the pair: a `JPEGInterchangeFormatLength` on its own addresses no bytes +/// at all, so there is nothing to name. Without this, reporting the incomplete pair could be +/// "fixed" by reporting every thumbnail that has no JPEG, which would make the signal noise. +/// +/// This pins the *reporting* contract only. Whether a length-only 1st IFD should nonetheless be +/// *rejected* in strict mode is open, and filed as issue #574: it is equally non-conformant (Exif +/// 3.0 §4.6.9.2 Table 21 gives both tags one level per thumbnail-format column — `M` under +/// **Compressed**, `N` under the three uncompressed ones — and that axis is not the two-valued +/// `Compression` tag), but it is not equally a *loss*, which is what this report names. An offset +/// with no length addresses bytes; a length with no offset addresses nothing. Both fixtures here +/// are uncompressed, where the table forbids either tag. +#[test] +fn a_thumbnail_with_no_jpeg_range_reports_nothing() { + for extra in [None, Some((THUMB_LENGTH, 16))] { + let mut thumb = Ifd::new(); + thumb.set(0x0103, Value::Short(vec![1])); // Compression = uncompressed + if let Some((tag, value)) = extra { + thumb.set(tag, Value::Long(vec![value])); + } + let (exif, report) = ExifReader::new() + .parse_with_report(&tiff(vec![image_ifd(), thumb])) + .expect("lenient parse"); + assert!( + exif.thumbnail().is_some(), + "the 1st IFD is still a thumbnail" + ); + assert!( + report.is_empty(), + "nothing was addressed, so nothing was dropped: {:?}", + report.dropped() + ); + } +} + +/// A strict report is not always empty: it still carries the loss strictness cannot reject. +/// +/// Strictness rejects *malformed* regions. A top-level directory past the 1st IFD is not malformed +/// — it parses cleanly and the [`Exif`](gamut_exif::Exif) model simply has nowhere to put it — so +/// strict has no grounds to fail on it, and dropping it silently would re-hide exactly the loss +/// `DroppedRegion::TrailingIfd` exists to surface. +#[test] +fn a_strict_parse_still_reports_a_trailing_directory() { + let mut thumb = Ifd::new(); + thumb.set(0x0103, Value::Short(vec![6])); // Compression = JPEG + let mut trailing = Ifd::new(); + trailing.set(0x0131, Value::Ascii("trailing".into())); // Software + + let (exif, report) = ExifReader::new() + .strict(true) + .parse_with_report(&tiff(vec![image_ifd(), thumb, trailing])) + .expect("a well-formed long chain must not fail even in strict mode"); + + assert_eq!(exif.make(), Some("Canon"), "the 0th IFD survives"); + assert_eq!(report.dropped().len(), 1, "{:?}", report.dropped()); + assert_eq!(report.dropped()[0].region(), DroppedRegion::TrailingIfd); + assert_eq!(report.dropped()[0].reason(), DropReason::Unrepresentable); +} + +/// The report's offsets and the crate's error offsets are in different frames, by the marker. +/// +/// A `Dropped::offset` addresses the TIFF stream, so it matches every offset stored inside the file +/// and is unchanged by whether the caller's buffer carries the six-byte `Exif\0\0` marker. An error +/// message instead names a byte of the buffer that was handed in, so the marker shifts it. Pinning +/// the pair together is what stops either frame drifting onto the other: unifying them would aim a +/// diagnostic outside the caller's buffer or renumber every reported offset. +#[test] +fn report_offsets_ignore_the_marker_but_error_offsets_include_it() { + let mut image = image_ifd(); + image.set(GPS_INFO, Value::Long(vec![DANGLING])); + let bare = tiff(vec![image]); + let mut marked = MARKER.to_vec(); + marked.extend(&bare); + + // The report frame is marker-invariant. + let offsets = |blob: &[u8]| -> Vec { + let (_, report) = ExifReader::new().parse_with_report(blob).expect("parse"); + report.dropped().iter().map(|d| d.offset()).collect() + }; + assert_eq!(offsets(&bare), vec![u64::from(DANGLING)]); + assert_eq!( + offsets(&marked), + offsets(&bare), + "a report offset addresses the TIFF stream, not the caller's buffer" + ); + + // The diagnostic frame is marker-shifted. The same corruption is applied to the TIFF stream in + // both blobs, so any offset difference is the marker and nothing else. + let mut broken_bare = bare.clone(); + broken_bare[4] ^= 0xFF; // the first-IFD offset in the TIFF header + let mut broken_marked = MARKER.to_vec(); + broken_marked.extend(&broken_bare); + + let message = |blob: &[u8]| -> String { + ExifReader::new() + .parse(blob) + .expect_err("a corrupt first-IFD offset must fail") + .to_string() + }; + let (bare_msg, marked_msg) = (message(&broken_bare), message(&broken_marked)); + let at = |m: &str| -> u64 { + let tail = m + .rsplit_once("byte offset: ") + .expect("the diagnostic names a byte offset") + .1; + tail.trim_end_matches(']') + .parse() + .expect("the byte offset parses") + }; + assert_eq!( + at(&marked_msg) - at(&bare_msg), + MARKER.len() as u64, + "an error offset counts the marker: {marked_msg} vs {bare_msg}" + ); +} diff --git a/crates/gamut-exif/tests/streaming.rs b/crates/gamut-exif/tests/streaming.rs new file mode 100644 index 00000000..d42e3851 --- /dev/null +++ b/crates/gamut-exif/tests/streaming.rs @@ -0,0 +1,131 @@ +//! The laziness contract for [`ExifReader::parse_from`]: pulling EXIF out of a large file reads +//! only the metadata, never the image payload. +//! +//! This is the whole point of the streaming entry point — a raw `.NEF`/`.CR3` is hundreds of +//! megabytes whose EXIF is a few kilobytes near the front — and it is not visible from a test that +//! only checks the parsed values, because a reader that slurped the file whole would return exactly +//! the same [`Exif`](gamut_exif::Exif). It is pinned the way `gamut-ifd`'s own `tests/streaming.rs` +//! pins its "≤64 read bytes" contract: a counting [`ReadAt`] wrapper and a bound that a +//! payload-touching read would blow past by three orders of magnitude. + +use gamut_core::Result; +use gamut_exif::ExifReader; +use gamut_ifd::{ByteOrder, Ifd, ReadAt, TiffFile, Value, Variant, write}; + +/// The `Exif\0\0` marker that precedes the TIFF stream in a JPEG `APP1` payload. +const MARKER: &[u8] = b"Exif\x00\x00"; +/// `ExifIFD` pointer (Exif 3.0 §4.6.3). +const EXIF_IFD: u16 = 0x8769; +/// `GPSInfo` pointer. +const GPS_INFO: u16 = 0x8825; +/// `Interoperability` pointer. +const INTEROP_IFD: u16 = 0xA005; + +/// Four megabytes: the file the metadata is embedded in. +const FILE_LEN: usize = 4 * 1024 * 1024; +/// Where the (nonexistent) strip data claims to start — well inside the padding. +const STRIP_AT: u32 = 1024 * 1024; + +/// A [`ReadAt`] wrapper that counts the bytes fetched, pinning the laziness contract. +struct Counting { + inner: S, + bytes_read: u64, +} + +impl ReadAt for Counting { + fn read_exact_at(&mut self, offset: u64, buf: &mut [u8]) -> Result<()> { + self.bytes_read += buf.len() as u64; + self.inner.read_exact_at(offset, buf) + } + + fn len(&mut self) -> Result { + self.inner.len() + } +} + +/// A marked EXIF blob — 0th IFD, Exif and GPS sub-IFDs, a nested Interop sub-IFD and a thumbnail +/// directory — followed by four megabytes of image payload the strip tags point into. +fn large_file_with_exif() -> Vec { + let mut image = Ifd::new(); + image.set(0x010F, Value::Ascii("Canon".into())); // Make + image.set(0x0110, Value::Ascii("EOS R5".into())); // Model + image.set(0x0111, Value::Long(vec![STRIP_AT])); // StripOffsets, into the payload + image.set(0x0117, Value::Long(vec![STRIP_AT])); // StripByteCounts + + let mut interop = Ifd::new(); + interop.set(0x0001, Value::Ascii("R98".into())); // InteroperabilityIndex + + let mut exif = Ifd::new(); + exif.set(0x829D, Value::Rational(vec![(28, 10)])); // FNumber + exif.set(0x8827, Value::Short(vec![400])); // PhotographicSensitivity + exif.set_sub_ifd(INTEROP_IFD, vec![interop]); + + let mut gps = Ifd::new(); + gps.set(0x0000, Value::Byte(vec![2, 3, 0, 0])); // GPSVersionID + + image.set_sub_ifd(EXIF_IFD, vec![exif]); + image.set_sub_ifd(GPS_INFO, vec![gps]); + + let mut thumb = Ifd::new(); + thumb.set(0x0103, Value::Short(vec![6])); // Compression = JPEG + + let tiff = write(&TiffFile { + order: ByteOrder::LittleEndian, + variant: Variant::Classic, + ifds: vec![image, thumb], + }) + .expect("write"); + + let mut data = MARKER.to_vec(); + data.extend(tiff); + assert!( + data.len() < STRIP_AT as usize, + "the metadata must precede the payload" + ); + data.resize(FILE_LEN, 0xAB); + data +} + +/// Extracting the EXIF of a four-megabyte file reads only the marker, the header, the directory +/// bodies and the values they reference — a bounded number of bytes that does not grow with the +/// file. +/// +/// The source is passed as `&mut Counting<_>` rather than by value, which is also the contract that +/// `&mut S` is itself a [`ReadAt`] source: without it the counter would be moved into the reader +/// and unreadable afterwards. +#[test] +fn extracting_exif_from_a_large_file_never_reads_the_payload() { + let data = large_file_with_exif(); + let mut counting = Counting { + inner: &data[..], + bytes_read: 0, + }; + + let exif = ExifReader::new() + .parse_from(&mut counting) + .expect("parse_from"); + + // The whole metadata tree really was reached — a reader that gave up early would also be + // "lazy", and this is what separates the two. + assert_eq!(exif.make(), Some("Canon")); + assert_eq!(exif.model(), Some("EOS R5")); + assert_eq!(exif.iso(), Some(400)); + assert!(exif.gps_ifd().is_some(), "the GPS sub-IFD was followed"); + assert!( + exif.interop_ifd().is_some(), + "the Interop sub-IFD was followed" + ); + assert!(exif.thumbnail().is_some(), "the 1st IFD was read"); + + // 251 bytes today: the marker, the header, five directory bodies and their out-of-line + // values. Four megabytes is the failure mode a slurping reader would show — but the bound is + // tighter than "not the payload" on purpose. A reader that re-walks the next-IFD chain when it + // has no trailing directory to report costs 347 bytes here, so 300 is the value that makes + // that a *failure* rather than an invisible inefficiency; it still leaves ~20% headroom for a + // tag or two. + assert!( + counting.bytes_read <= 300, + "streaming parse read {} bytes of a {FILE_LEN}-byte file", + counting.bytes_read + ); +}