diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8dde08e9..617cc72d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -60,6 +60,11 @@ jobs: run: mise run check-ffi-features - name: Release dependency graph run: mise run check-release-deps + - name: README crates table + # Static: `cargo metadata --no-deps` (the same call the step above makes) plus awk over + # README.md. Nothing is built and nothing is fetched. A drift guard that runs nowhere + # cannot guard against drift, so it runs here beside the other three static checks. + run: mise run check-readme-crates - name: Test conventions (docs/testing.md) # Static: greps the tree, builds nothing, sub-second. Enforces that every integration-test # file names what it pins, that every proptest configuration pins its seed and disables diff --git a/AGENTS.md b/AGENTS.md index c30149d5..00dc6630 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,15 +20,16 @@ Dependency edges (a crate depends on those to its right): `cmm`, `codec-abi`, `all`); `default = []`. `primitives` re-exports shared `color`/`dsp`/`bitstream`; `isobmff`/`metadata`/`tonemap`/`codec-abi` re-export their respective primitive crates; `all` includes all of these. -- **gamut-core** — `Encoder`/`Decoder` traits, image buffers, `Dimensions`, `Error`, plus the +- **gamut-core** — `EncodeImage`/`DecodeImage` traits, image buffers, `Dimensions`, `Error`, plus the format-agnostic `convert` module: the one place any `Pixel` layout converts to another (grey↔RGB, alpha add/drop/composite, 8↔16-bit), lossless by default with loss opted into per decoder via a `ConvertPolicy`. Format crates decode to what the file carries and delegate the - layout change there rather than hand-rolling it. No internal deps; everything else depends on it. + layout change there rather than hand-rolling it. No internal deps of its own; each entry below + states its own edges, and most — not all — of them include `gamut-core`. - **gamut-color** / **gamut-dsp** / **gamut-bitstream** — shared primitives. ← core. -- **gamut-tonemap** — scalar tone-mapping curves (`ToneCurve` + Reinhard/ACES/Hable/Drago) - for HDR→SDR pipelines, between `gamut-color`'s transfer functions and the SDR re-encode. - ← core. +- **gamut-tonemap** — scalar tone-mapping curves for HDR→SDR pipelines: the `ToneCurve` trait + over eight operators (Linear/Clamp/Exposure/Reinhard/ReinhardExtended/ACES/Hable/Drago), + between `gamut-color`'s transfer functions and the SDR re-encode. ← core. - **gamut-codec-abi** — shared codestream-backend seam: `repr(C)` vtables (`DecoderVTable`/`EncoderVTable` + `StreamConfig`/`EncodeConfig`/`ImageDesc`) and their object-safe Rust twin traits, plus the registry fallback contract by which a foreign @@ -64,7 +65,8 @@ Dependency edges (a crate depends on those to its right): decode itself is out of scope here). Differential oracle: libheif+libde265 (+kvazaar fixture generation), dev-only. ← isobmff, core, color. - **gamut-deflate** — pure-Rust DEFLATE/zlib **encoder** (zopfli-class) under gamut-png; - deliberately encoder-only — workspace decoders inflate via `miniz_oxide`. ← core. + deliberately encoder-only — workspace decoders inflate via `miniz_oxide`. ← nothing: no dependency + at all, not even `gamut-core` (`src/lib.rs:6`). - **gamut-png** — PNG codec (3rd edition, W3C): space-efficient encoder and spec-compliant decoder — all colour types/bit depths, Adam7 *decoding*, all filters, decode limits for hostile input, ancillary metadata surfaced as raw `MetadataBlock`-ready payloads diff --git a/README.md b/README.md index 57a77973..210d2b78 100644 --- a/README.md +++ b/README.md @@ -69,18 +69,30 @@ In 2026, `gamut` started when there were no robust, well-tested Rust implementat ### Scope The initial focus is **AVIF, WebP, and JPEG** — the formats with the best -size-versus-compatibility tradeoff today. **JPEG XL** (`gamut-jxl`) is now implemented as an +size-versus-compatibility tradeoff today. **JPEG XL** (`gamut-jxl`) is implemented as an encoder + decoder (issue #243) — uniquely, by wrapping the format's reference implementations (libjxl for encode, the pure-Rust jxl-rs for decode) rather than clean-slate, a deliberate -maintainer decision documented in that crate. The other format crates in the tree (HEIC, VVC, -AV2) are scaffolding, and may move or be dropped as the focus sharpens. **TIFF 6.0** -(`gamut-tiff`) is newly scaffolded and under active implementation (issue #107) as a -royalty-free, natively still-image format — a good long-term fit for the image-first focus. - -Alongside the codecs, gamut is growing **shared image-metadata primitives** (issue #34) — EXIF, -XMP, ICC, and IPTC, plus the TIFF/IFD container core (`gamut-ifd`) that EXIF builds on — so the -format crates can read, preserve, and embed metadata. These are newly scaffolded; the long-term -goal is de-facto, fully-featured implementations (for EXIF, exiftool-class tag coverage). +maintainer decision documented in that crate. **TIFF 6.0** (`gamut-tiff`) is implemented at v1 +(issue #107) as a royalty-free, natively still-image format — a good long-term fit for the +image-first focus — with YCbCr/Lab and JPEG-in-TIFF still deferred, and **DNG** (`gamut-dng`) +is a raw encoder + decoder built as a TIFF/EP profile over the same `gamut-ifd` container core. +`gamut-heic` is a decode-only HEIF container, with the HEVC bitstream itself left to a pluggable +backend rather than decoded here. Of the format crates, only `gamut-vvc` and `gamut-av2` are +still scaffolding, and they may move or be dropped as the focus sharpens. + +Alongside the codecs, gamut ships **shared image-metadata primitives** (issue #34) — EXIF, +XMP, ICC, and IPTC, plus the TIFF/IFD container core (`gamut-ifd`) that EXIF builds on. These are +all at v1 or later; EXIF, IPTC and XMP are gated differentially against exiv2 (which bundles +Adobe's XMPCore), and ICC against Little-CMS. `gamut-cmm` adds the ICC transform engine over the +profiles `gamut-icc` parses — conformance-gated against Little-CMS, though still well behind it +on throughput — and is the one crate here not yet on crates.io. + +The long-term goal is de-facto, fully-featured implementations — for EXIF, exiftool-class tag +coverage — and today's crates do not reach it yet. The largest gap is reach rather than depth: +the `gamut-metadata` facade is consumed by `gamut-dng` and the umbrella only, so every other +format crate still carries EXIF/XMP/ICC as raw byte blocks instead of calling the typed path. +Behind that sit per-vendor MakerNote payloads, which round-trip verbatim but are not decoded, and +tag breadth beyond the standard dictionary. Issue #416 tracks the whole gap, measured. **gamut is image-first.** Even where a format's codec (AV1, AV2, VVC, HEVC) is fundamentally a video codec, gamut implements only the intra-frame, still-image subset those formats use — no @@ -106,40 +118,52 @@ format. ## Crates +Every row describes the crate **in this tree**: any version it cites is the crate's own +`Cargo.toml` — what `mise run versions` prints and what a workspace `path` dependency builds — not +what crates.io currently serves. Where the two differ, [Releases](#releases) says which crate and +why; deciding that comparison needs the network, so it is stated there rather than counted here. + | Crate | Purpose | Status | | ----------------- | ---------------------------------------------------------------------- | -------------------------------------- | -| `gamut` | Umbrella crate; re-exports the format crates behind Cargo features | implemented | -| `gamut-core` | Core traits (`Encoder`/`Decoder`), image buffers, dimensions, errors | WIP | -| `gamut-color` | Color spaces, pixel formats, bit depths, chroma subsampling, transfers | stabilizing api | -| `gamut-dsp` | Shared DSP: DCT, wavelet transforms, quantization, filtering | stabilizing api | -| `gamut-bitstream` | Bit readers/writers and entropy coders (ANS, arithmetic, Huffman) | stabilizing api | -| `gamut-isobmff` | ISOBMFF container utilities (AVIF, HEIC) | finalizing api | +| `gamut` | Umbrella crate; most sibling crates it re-exports sit behind a Cargo feature of their own — the format/codec crates, the ISOBMFF container, tone mapping, colour management, the codec ABI — and the rest behind one of the two shared Cargo features `primitives` and `metadata`, covering the colour/DSP/bitstream layer and the EXIF/XMP/ICC/IPTC/IFD crates plus the metadata facade; the exception is the core re-export, unconditional because it is the umbrella's one always-on dependency: `gamut-core` | implemented | +| `gamut-core` | Core traits (`EncodeImage`/`DecodeImage`), image buffers, dimensions, errors, `convert` | stable (v2; v1 under #177), pixel conversion added by #268 | +| `gamut-color` | Coded-plane bit depths and chroma subsampling, CICP code points and planar buffers, the `ycbcr` matrixing layer (H.273, plus the libwebp-exact BT.601 one VP8 needs), and the `f64` colour science (transfer, Lab/OKLab, XYB, `matrix`/`linalg`, gamut map, CCT, profile) | stable (v2; v1 under #179); the colour science is Tier-1 `f64`, not bit-reproducible | +| `gamut-dsp` | Shared DSP kernels: AV1 DCT/ADST/identity/WHT, the JPEG 8×8 forward/inverse DCT, quantization rounding | stable (v2; v1 under #192) | +| `gamut-bitstream` | Bit readers/writers, LEB128, the AV1 §8.2 symbol (arithmetic) coder, MSB-first sample packing | v0.2, no v1 release issue yet; the ANS and Huffman coders are not implemented | +| `gamut-tonemap` | Tone-mapping curves for HDR→SDR: the `ToneCurve` trait over the Linear, Clamp, Exposure, Reinhard, ReinhardExtended, ACES, Hable and Drago operators | stable (v1, #188); surface frozen | +| `gamut-codec-abi` | Shared codestream-backend seam: `repr(C)` vtables, their object-safe Rust twins, and the registry fallback *contract*; this crate declares no registry — each host crate that offers a seam keeps its own | consumed by `gamut`, `gamut-avif`, `gamut-ffi`, `gamut-heic`, `gamut-jpeg`, `gamut-jxl`, `gamut-png` and `gamut-webp`; the umbrella's edge is optional, behind Cargo feature `codec-abi` | +| `gamut-isobmff` | ISOBMFF container utilities (AVIF, HEIC) | stable (v2); structure only, codestream carried opaquely | | `gamut-riff` | RIFF container utilities (WebP) | stable (v1, #186) | -| `gamut-av1` | AV1 still-image (intra-frame) encoder — the codec layer beneath AVIF | implemented lossless and lossy (alpha) | +| `gamut-av1` | AV1 still-image (intra-frame) encoder + decoder — the codec layer beneath AVIF | encoder: lossless and lossy intra keyframes; decoder (default Cargo feature `decode`): 8-bit 4:4:4 intra frames, key and intra-only (#259) | | `gamut-av2` | AV2 still-image (intra-frame) encoder/decoder — AV1's successor | placeholder | -| `gamut-avif` | AVIF encoder — AV1 still frames in an ISOBMFF container | stabilizing with gamut-av1 | -| `gamut-jxl` | JPEG XL encoder (libjxl wrap) + decoder (pure-Rust jxl-rs) | encoder + decoder (v1, #243) | -| `gamut-jxl-sys` | Static libjxl 0.12.0 FFI declarations — native core of gamut-jxl encode | encoder backend (v1, #243) | -| `gamut-webp` | WebP (intra-frame VP8/VP8L) encoder/decoder | implemented VP8 + VP8L (+alpha, metadata, effort/near-lossless) | -| `gamut-heic` | HEIC/HEIF still-image (HEVC intra) encoder/decoder | placeholder | +| `gamut-avif` | AVIF encoder + container decoder — AV1 still frames in ISOBMFF | encoder (v1, 8/10/12-bit) + container decode (#250); AV1 codestream decode via the `Av1StillDecoder` seam | +| `gamut-jxl` | JPEG XL encoder (libjxl wrap) + decoder (pure-Rust jxl-rs) | encoder + decoder (#243) | +| `gamut-jxl-sys` | Static libjxl 0.12.0 FFI declarations — native core of `gamut-jxl` encode | encoder backend (#243) | +| `gamut-jpeg` | JPEG-1 (ISO/IEC 10918-1) encoder + decoder — baseline & progressive; the jpegli-style XYB colour mode is encode-only | encoder + decoder (#28, P1–P13) | +| `gamut-webp` | WebP (intra-frame VP8/VP8L) encoder/decoder, with a public `backend` seam for an alternate VP8/VP8L codestream implementation | implemented VP8 + VP8L (+alpha, metadata, effort/near-lossless) | +| `gamut-heic` | HEIC/HEIF still-image container **decoder** — HEVC via a pluggable backend | decode-only container (S1–S7); no encoder, by charter | | `gamut-vvc` | VVC (H.266) still-image (intra) encoder/decoder | placeholder | -| `gamut-ifd` | TIFF/IFD container core (byte order, field types, IFD I/O) — EXIF+TIFF | scaffolding (impl in progress, #34) | -| `gamut-exif` | EXIF (Exif 3.0) metadata parser/serializer — built on gamut-ifd | scaffolding (impl in progress, #34) | +| `gamut-ifd` | TIFF/IFD container core (byte order, field types, IFD I/O) — EXIF+TIFF | stable (v2, byte completeness #263); BigTIFF behind Cargo feature `bigtiff` | +| `gamut-exif` | EXIF (Exif 3.0) metadata parser/serializer — built on `gamut-ifd` | stable (v1, #194); MakerNote preserved verbatim, not decoded | | `gamut-icc` | ICC color profile (ICC.1:2022) parser/serializer | stable (v1, #180) | -| `gamut-cmm` | ICC colour management module (transform engine) over gamut-icc profiles | epic #323 complete (P1–P7); unpublished | -| `gamut-xmp` | XMP (RDF/XML) metadata parser/serializer | scaffolding (impl in progress, #34) | -| `gamut-iptc` | IPTC photo metadata (IIM + Core/Extension over XMP) | scaffolding (impl in progress, #34) | -| `gamut-metadata` | Unified metadata facade over EXIF/XMP/ICC/IPTC (extract + embed) | scaffolding (impl in progress, #34) | -| `gamut-tiff` | TIFF 6.0 encoder/decoder — self-contained (own IFD/tag container) | baseline + extensions (YCbCr/Lab/JPEG WIP) | -| `gamut-deflate` | DEFLATE/zlib encoder (zopfli-class) — the compression under gamut-png | encoder (decoding stays on miniz_oxide) | -| `gamut-png` | PNG (W3C 3rd edition) encoder + spec-compliant decoder | encoder (#24) + decoder (#249) | -| `gamut-cli` | `gamut` CLI sandbox: encode AVIF + inspect the shared primitives | ready for use | +| `gamut-cmm` | ICC colour management module (transform engine) over `gamut-icc` profiles | P1–P8 complete per its STATUS.md — P1–P7 are epic #323, while P8 (pipeline optimization) is #372, which #323 lists out of scope | +| `gamut-xmp` | XMP (RDF/XML) metadata parser/serializer | stable (v1, #189); canonical serializer, UTF-8 packets only | +| `gamut-iptc` | IPTC photo metadata (IIM + Core/Extension over XMP) | stable (v1, #182); Extension structures pass through as raw XMP | +| `gamut-metadata` | Unified metadata facade over EXIF/XMP/ICC/IPTC (extract + embed) | stable (v1); orchestration only. A C2PA manifest store is extracted verbatim but never re-embedded: embedding drops it, or refuses, because its hard binding cannot survive the rewrite | +| `gamut-tiff` | TIFF 6.0 encoder/decoder — on the shared `gamut-ifd` container core | stable (v1, #107); YCbCr/Lab and JPEG-in-TIFF deferred | +| `gamut-dng` | DNG 1.7.1 raw encoder + decoder — a TIFF/EP profile over `gamut-ifd` | encoder + decoder (v1, #109), Adobe DNG SDK-gated | +| `gamut-deflate` | DEFLATE/zlib encoder (zopfli-class) — the compression under gamut's own writers | encoder (#195); consumed by `gamut-dng`, `gamut-png` and `gamut-tiff`, which inflate with the external dependency `miniz_oxide` rather than here — this crate has no always-on dependencies of its own | +| `gamut-png` | PNG (W3C 3rd edition) encoder + decoder, over the still-image subset | encoder (#24) + decoder (#249); APNG out of scope, so an animated PNG decodes as its default image | +| `gamut-cli` | `gamut` CLI sandbox: `convert` (decode PNG/JPEG/PPM/WebP/JXL, re-encode AVIF/WebP/TIFF/PNG/JXL/JPEG), `inspect` (TIFF/DNG byte accounting), `icc`, `isobmff`, `av1`, and the `color`, `dsp` and `bitstream` primitive inspectors | ready for use | | `gamut-wasm` | WebAssembly bindings | placeholder | -| `gamut-ffi` | C-compatible FFI bindings | placeholder | +| `gamut-ffi` | C-compatible FFI bindings | provider boundary shipped (#280); consumer entry points pending (#242) | -All cargo metadata except per-crate `version` is centralized in the root -`[workspace.package]` / `[workspace.dependencies]`; each crate inherits the shared fields via -`.workspace = true` and sets its own `version` (see [Versioning](#versioning)). +Each crate manifest sets its own `name`, `version`, `description` and `readme` — the root +manifest says why `description` is per-crate: crates.io discoverability — and `links` where it +drives a native build. Everything else (`edition`, `rust-version`, `authors`, `license`, +`repository`, `homepage`, `keywords`, `categories`, and the shared dependency versions) is +centralized in the root `[workspace.package]` / `[workspace.dependencies]` and inherited via +`.workspace = true` (see [Versioning](#versioning)). ## Prerequisites @@ -213,6 +237,7 @@ cargo test --workspace | `mise run lint` | Lint with Clippy (warnings as errors) | | `mise run lint-fix` | Lint and auto-fix | | `mise run check-commits` | Check commits are Conventional Commits | +| `mise run check-readme-crates` | Check this README's crates table names every workspace crate, no phantom or duplicate ones, stands under its delimiter row, and gives each crate a three-cell row | | `mise run coverage` | Run tests with coverage (min 80%) | | `mise run check-cross ` | Cross-compile-check the libs for a target (extended CI; master/manual) | | `mise run check-msrv` | Check the libs compile on the documented MSRV (extended CI; master/manual) | @@ -322,9 +347,20 @@ pipeline stalls silently when either is missing (issue #377): [`release.yml`](.github/workflows/release.yml) prefers it over `GITHUB_TOKEN` when present. 2. **A brand-new crate needs one manual first publish.** Trusted Publishing mints a token *for a crate that already exists*, so the very first version of a new crate must be published once - with an owner-scoped token, after which OIDC takes over. Crates still awaiting that bootstrap - have no `-v` tag: currently `gamut-cmm`, `gamut-deflate`, `gamut-dng`, - `gamut-jpeg`, `gamut-jxl-sys` and `gamut-png`. + with an owner-scoped token, after which OIDC takes over. **`gamut-cmm` is the only crate still + awaiting that bootstrap** — it is the one workspace crate absent from crates.io. + +A missing `-v` tag is **not** evidence of a missing publish, and this section +previously read it that way. `gamut-deflate`, `gamut-dng`, `gamut-jpeg`, `gamut-jxl-sys` and +`gamut-png` carry no release tag and are nonetheless on crates.io, each at the version its +`Cargo.toml` declares. That is the stall above showing itself in the tag-and-release half of the +pipeline rather than in the publish half. + +The opposite skew exists too, and it is the one place a version in the [crates table](#crates) +differs from what crates.io serves: `gamut-riff`'s manifest declares `1.0.0` while the newest +published release is `0.1.3`. Whether that 1.0 is real is a release decision rather than a +documentation one, so the table states the manifest version and this note states the gap +(issue #546). ## License diff --git a/mise.toml b/mise.toml index 50ccb405..5fdc5986 100644 --- a/mise.toml +++ b/mise.toml @@ -296,6 +296,39 @@ run = "./tooling/mutants/run.sh --crate" description = "Enforce the mechanical test conventions in docs/testing.md" run = "./tooling/check-tests/run.sh" +# Nothing read the README crates table, so it drifted: fixing issue #425 meant correcting +# twenty-three of its twenty-eight rows and adding four crates that had no row at all. `versions` +# only lists `name version` and asserts nothing, and `check-msrv` greps the README for one number, +# so neither would have caught it. This gates the table's *membership* -- every workspace crate +# has a row, every row names a real crate, no crate twice -- each crate row's *shape* (three cells +# whose Purpose and Status each render as something, and not hidden inside an HTML comment or a +# code fence), and the table's *context*: EVERY crate row in the section must render inside ONE +# table, headed by a delimiter row directly under a header row of the same width, since anything +# else renders some or all of the rows as literal text. It also checks the claim forms `cargo +# metadata` can decide, because a hand-maintained list of crates is the defect this task exists +# to catch, one level down. Those forms are NOT enumerated here: this comment named three of them +# while the script checked five, which is the same defect a third time. The script header says +# where the list lives and how to print it -- `grep -n 'CLAIM FORM:'` over the script. +# +# It gates nothing else about the prose, so a green run means "every crate has a well-formed row +# inside a real table, and the claims cargo can settle are settled", not "the table is true". A +# row's prose answers to the crate's own source -- the code that ships. lib.rs, then Cargo.toml, +# then STATUS.md is where to look, but each is a summary and any of them can be stale (issues +# #545, #560), so a row must be true of the crate rather than faithful to a file. What the guard +# deliberately does not check is written down once, in the script header. +[tasks.check-readme-crates] +description = "Verify the README crates table covers every workspace crate, lists no phantom or duplicate rows, renders as one table under a header and delimiter row of matching width, gives each crate a three-cell row whose Purpose and Status render as something, and agrees with cargo metadata wherever the section makes a claim in one of the forms the guard checks" +run = "./tooling/check-readme-crates/run.sh" + +# The guard's own regression battery: single-edit fixtures over the repository README, each +# asserting the exit code and the message the guard owes it. Not part of `check-readme-crates`, +# and not in CI: it is for whoever changes the guard, and it costs a `cargo metadata` per fixture. +# `CHECK_README_AWK` selects the interpreter, so the portability claim is re-runnable rather than +# asserted -- e.g. `CHECK_README_AWK='gawk --posix' mise run check-readme-crates-fixtures`. +[tasks.check-readme-crates-fixtures] +description = "Run the README crates-table guard against its own fixture battery" +run = "./tooling/check-readme-crates/fixtures.sh" + [tasks.check-release-deps] description = "Reject publishable dev-only workspace edges that release-plz cannot order" run = '''cargo metadata --no-deps --format-version 1 | jq -r -e ' diff --git a/tooling/check-readme-crates/fixtures.sh b/tooling/check-readme-crates/fixtures.sh new file mode 100755 index 00000000..93a7d58f --- /dev/null +++ b/tooling/check-readme-crates/fixtures.sh @@ -0,0 +1,185 @@ +#!/usr/bin/env bash +# The regression battery for `run.sh`, the README crates-table guard beside this file. +# +# The guard grew five checks over four rounds of review, and each round proved itself with a +# battery of single-edit fixtures that lived only in the pull request description. A claim in a +# description is not a regression test: nobody can re-run it, and the next round starts from an +# assertion rather than from an execution. This file is that battery, committed. +# +# Each fixture is ONE edit to the repository README, applied to a copy. The fixture asserts the +# exit code the guard must return for it and, where the reason matters, a fragment the guard must +# print -- so a fixture that starts failing for a different reason is caught rather than counted +# as a pass. A fixture is named for the single property it holds the guard to. +# +# The three groups are the guard's three jobs: that the table lists the crates (membership and row +# shape), that it renders as one table (structure), and that the claims its prose makes agree with +# `cargo metadata` (the claim forms). Legal renderings are fixtures too, with expected code 0: a +# guard that rejects legal prose is a guard someone turns off, and both over-rejections this +# battery pins were found by a reviewer after they shipped. +# +# Usage: +# tooling/check-readme-crates/fixtures.sh # system awk +# CHECK_README_AWK='gawk --posix' tooling/check-readme-crates/fixtures.sh +# +# With no argument the fixtures are cut from the repository's own README, so they follow it as it +# changes; pass a path to cut them from another file instead. +set -uo pipefail + +root="$(git rev-parse --show-toplevel)" || { + echo "fixtures: not inside a git repository" + exit 1 +} +cd "$root" || exit 1 + +guard="./tooling/check-readme-crates/run.sh" +base="${1:-README.md}" + +test -f "$base" || { + echo "fixtures: no such file: $base" + exit 1 +} + +# The guard must agree with the tree before any fixture means anything: every fixture is a single +# edit AWAY from this file, so a base that already fails would make every expectation unreadable. +work="$(mktemp -d)" || exit 1 +trap 'rm -rf "$work"' EXIT + +passed=0 +failed=0 + +# check +# The transform reads the base README on stdin and writes the fixture on stdout. +check() { + name="$1" + want="$2" + fragment="$3" + shift 3 + + "$@" <"$base" >"$work/README.md" || { + echo "FAIL $name -- the fixture transform itself failed" + failed=$((failed + 1)) + return + } + + out="$("$guard" "$work/README.md" 2>&1)" + got=$? + + if [ "$got" != "$want" ]; then + echo "FAIL $name -- expected exit $want, got $got" + printf '%s\n' "$out" | sed 's/^/ /' + failed=$((failed + 1)) + return + fi + if [ -n "$fragment" ] && ! printf '%s\n' "$out" | grep -qF -- "$fragment"; then + echo "FAIL $name -- exit $got as expected, but the output does not say '$fragment'" + printf '%s\n' "$out" | sed 's/^/ /' + failed=$((failed + 1)) + return + fi + passed=$((passed + 1)) +} + +# Transforms. Each reads stdin and writes stdout; `sedx`/`awkx` keep the fixture table to one +# readable line apiece. +unchanged() { cat; } +sedx() { sed "$1"; } +awkx() { awk "$1"; } + +# --------------------------------------------------------------------------------------------- +# Membership and row shape: the table lists every crate, once, in a row a reader can read. +# --------------------------------------------------------------------------------------------- +check "baseline: the repository README passes" 0 "lists every workspace crate" unchanged +check "a crate row deleted" 1 "no row in" sedx '/^| `gamut-png` /d' +check "a crate row misspelled into a phantom" 1 "not workspace members" sedx 's/^| `gamut-png` /| `gamut-pngg` /' +check "a crate listed twice" 1 "listed more than once" sedx '/^| `gamut-png` /p' +check "a crate row hidden in an HTML comment" 1 "no row in" awkx '/^\| `gamut-png` /{print ""; next} {print}' +check "a crate row hidden in a fenced code block" 1 "no row in" awkx '/^\| `gamut-png` /{print "```"; print; print "```"; next} {print}' +check "a phantom row hiding behind a capital" 1 "not workspace members" sedx 's/^| `gamut-png` /| `Gamut-png` /' +check "a phantom row hiding behind an underscore" 1 "not workspace members" sedx 's/^| `gamut-png` /| `gamut_png` /' +check "a two-cell crate row" 1 "malformed crate rows" sedx 's/^| `gamut-png` .*/| `gamut-png` | only one cell |/' +check "a four-cell crate row" 1 "malformed crate rows" sedx '/^| `gamut-png` /s/$/ extra |/' +check "an empty Status cell" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | |/' +check "a Status cell of one  " 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \  |/' +check "a Status cell of one ‎" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \‎ |/' +check "a Status cell of one ‎" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \‎ |/' +check "a Status cell of one ‎ (leading zero)" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \‎ |/' +check "a Status cell of one ‎" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \‎ |/' +check "a Status cell of one  " 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \  |/' +check "a Status cell of one  " 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | \  |/' +check "a Status cell of one non-breaking space" 1 "empty Status cell" awkx '{ if ($0 ~ /^\| `gamut-png` /) { print "| `gamut-png` | PNG codec | \302\240 |"; next } print }' +check "a Status cell of one tab" 1 "empty Status cell" awkx '{ if ($0 ~ /^\| `gamut-png` /) { print "| `gamut-png` | PNG codec |\t|"; next } print }' +check "a Status cell of one empty HTML element" 1 "empty Status cell" sedx 's/^| `gamut-png` .*/| `gamut-png` | PNG codec | <\/span> |/' +check "a crate row with no trailing pipe" 1 "no row in" sedx '/^| `gamut-png` /s/|$//' +check "a crate row indented four spaces (a code block)" 1 "no row in" sedx '/^| `gamut-png` /s/^/ /' +check "a crate row indented two spaces (legal)" 0 "lists every workspace crate" sedx '/^| `gamut-png` /s/^/ /' + +# --------------------------------------------------------------------------------------------- +# Structure: every crate row renders inside ONE table, whatever the bytes of the row look like. +# --------------------------------------------------------------------------------------------- +check "the delimiter row deleted" 1 "do not render inside a table" sedx '/^| -----/d' +check "a blank line directly after the delimiter" 1 "do not render inside a table" sedx '/^| -----/G' +check "a blank line between two crate rows" 1 "do not render inside a table" sedx '/^| `gamut-png` /G' +check "the header row blanked" 1 "do not render inside a table" sedx 's/^| Crate .*/| | | |/' +check "a two-column header over a three-column delimiter" 1 "do not render inside a table" sedx 's/^| Crate .*/| Crate | Purpose |/' +check "the delimiter row indented four spaces" 1 "do not render inside a table" sedx '/^| -----/s/^/ /' +check "an earlier table donates its delimiter" 1 "do not render inside a table" awkx '/^\| Crate /{print "| Decoy | Decoy | Decoy |"; print "| --- | --- | --- |"; print ""} /^\| -----/{next} {print}' +check "a decoy table whose own first row is a crate row" 1 "do not render inside a table" awkx '/^\| Crate /{print "| `gamut-png` | PNG codec | shipped |"; print "| --- | --- | --- |"; print ""} /^\| -----/{next} /^\| `gamut-png` /{next} {print}' +check "the crate rows split over two valid tables" 1 "spread over" awkx '/^\| `gamut-png` /{print ""; print "| Crate | Purpose | Status |"; print "| --- | --- | --- |"} {print}' +check "alignment colons in the delimiter (legal)" 0 "lists every workspace crate" awkx '/^\| -----/{print "|:---|:---:|---:|"; next} {print}' +check "an escaped pipe inside a cell (legal)" 0 "lists every workspace crate" awkx '{ if ($0 ~ /^\| `gamut-riff` /) sub(/RIFF container utilities \(WebP\)/, "RIFF a \\| b"); print }' +check "a non-crate separator row inside the table (legal)" 0 "lists every workspace crate" awkx '/^\| `gamut-png` /{print "| **Codecs** | | |"} {print}' +check "an unrelated table earlier in the section (legal)" 0 "lists every workspace crate" awkx '/^## Crates$/{print; print ""; print "| A | B |"; print "| --- | --- |"; print "| 1 | 2 |"; print ""; next} {print}' + +# --------------------------------------------------------------------------------------------- +# The section heading, in every form CommonMark gives it. +# --------------------------------------------------------------------------------------------- +check "the heading with a closing sequence (legal)" 0 "lists every workspace crate" sedx 's/^## Crates$/## Crates ##/' +check "the heading indented three spaces (legal)" 0 "lists every workspace crate" sedx 's/^## Crates$/ ## Crates/' +check "the heading in setext form (legal)" 0 "lists every workspace crate" awkx '{ if ($0 == "## Crates") { print "Crates"; print "------"; next } print }' +check "a trailing space on the heading (legal)" 0 "lists every workspace crate" sedx 's/^## Crates$/## Crates /' +check "a CRLF file (legal)" 0 "lists every workspace crate" sedx 's/$/\r/' +check "a heading whose text is Crates#, not Crates" 1 "found no crate rows" sedx 's/^## Crates$/## Crates#/' + +# --------------------------------------------------------------------------------------------- +# The claim forms. Each is opt-in and marker-driven; each fixture writes one claim cargo refutes, +# and each is paired with the legal claim it must not reject. +# --------------------------------------------------------------------------------------------- +check "a version token contradicting the manifest" 1 "version claims" sedx '/^| `gamut-core` /s/stable (v2;/stable (v3;/' +check "a minor version contradicting the manifest" 1 "version claims" sedx '/^| `gamut-bitstream` /s/v0.2,/v0.3,/' +check "a consumed-by list one crate short" 1 "consumed by" sedx 's/`gamut-png` and `gamut-webp`;/`gamut-png`;/' +check "a consumed-by list naming a non-consumer" 1 "consumed by" sedx 's/`gamut-png` and `gamut-webp`;/`gamut-png`, `gamut-riff` and `gamut-webp`;/' +check "an always-on list naming an optional edge" 1 "always-on" sedx 's/always-on dependency: `gamut-core`/always-on dependency: `gamut-avif`/' +check "an always-on list naming an edge that is absent" 1 "always-on" sedx 's/always-on dependency: `gamut-core`/always-on dependency: `gamut-jxl-sys`/' +check "a feature no crate the row names declares" 1 "feature claims" sedx 's/behind Cargo feature `codec-abi`/behind Cargo feature `codec-abii`/' +check "an off-by-default feature called a default one" 1 "feature claims" sedx 's/BigTIFF behind Cargo feature `bigtiff`/BigTIFF behind default Cargo feature `bigtiff`/' +check "a default feature owned by a crate not named here" 1 "feature claims" sedx '/^| `gamut-riff` /s/stable (v1, #186)/stable (v1, #186); default Cargo feature `decode`/' +check "another real feature of the row crate (legal)" 0 "lists every" sedx 's/BigTIFF behind Cargo feature `bigtiff`/BigTIFF behind Cargo feature `bigtiff`, with Cargo feature `test-support` for fixtures/' +check "a feature of another crate the row names (legal)" 0 "lists every" sedx 's/behind Cargo feature `codec-abi`/behind Cargo feature `codec-abi` and Cargo feature `avif`/' +check "a cell naming a crate that does not exist" 1 "do not exist" sedx 's/built on `gamut-ifd`/built on `gamut-ifdd`/' +check "a cell naming a real crate (legal)" 0 "lists every" sedx '/^| `gamut-riff` /s/(WebP)/(WebP), read by `gamut-webp`/' +check "an external dependency cargo does not record" 1 "external-dependency" sedx 's/external dependency `miniz_oxide`/external dependency `miniz_oxidee`/' +check "an external dependency no crate in the row has" 1 "external-dependency" sedx 's/external dependency `miniz_oxide`/external dependency `serde`/' + +# The two over-rejections a reviewer found after they shipped. Both are legal prose and both must +# stay legal: bare `features` is an English verb, which is why the marker carries `Cargo`. +check "the word features as an English verb (legal)" 0 "lists every" sedx '/^| `gamut-riff` /s/(WebP)/(WebP); the crate features `chunk` walking/' +check "the underscore spelling of a real crate" 1 "do not exist" sedx 's/built on `gamut-ifd`/built on `gamut_ifd`/' + +# --------------------------------------------------------------------------------------------- +# The precondition the name checks rest on, and the section prose they now reach. +# --------------------------------------------------------------------------------------------- +check "a crate name in a cell without a code span" 1 "without a code span" sedx 's/built on `gamut-ifd`/built on gamut-ifd/' +check "a crate name in section prose without a code span" 1 "without a code span" sedx 's/^Each crate manifest sets/Each gamut-png crate manifest sets/' +check "the bare word gamut in prose (legal)" 0 "lists every" sedx 's/^Each crate manifest sets/Each gamut crate manifest sets/' +check "a crate name inside a fence in the section (legal)" 0 "lists every" awkx '/^## Prerequisites/{print "```rust"; print "use gamut_png::Encoder;"; print "```"; print ""} {print}' +check "a phantom crate named in section prose" 1 "do not exist" sedx 's/^Each crate manifest sets/Each `gamut-ifdd` crate manifest sets/' +check "a feature claim in section prose cargo refutes" 1 "feature claims" sedx 's/^Each crate manifest sets/The `gamut-png` Cargo feature `nope` is gone; each crate manifest sets/' +check "a consumed-by list outside a crate row" 1 "no crate to be about" sedx 's/^Each crate manifest sets/Each is consumed by `gamut-png`; each crate manifest sets/' +check "an always-on list outside a crate row" 1 "no crate to be about" sedx 's/^Each crate manifest sets/Each has an always-on dependency: `gamut-core`; each crate manifest sets/' +check "a version token outside a crate row" 1 "no crate to be about" sedx 's/^Each crate manifest sets/Each v9 crate manifest sets/' +check "a phantom crate named in a sub-heading" 1 "do not exist" awkx '/^Each crate manifest sets/{print "### The `gamut-ifdd` note"; print ""} {print}' +check "a claim in a sub-heading cargo refutes" 1 "feature claims" awkx '/^Each crate manifest sets/{print "### The `gamut-png` Cargo feature `nope`"; print ""} {print}' + +echo +echo "check-readme-crates fixtures: $passed passed, $failed failed (awk: ${CHECK_README_AWK:-awk})" +test "$failed" -eq 0 diff --git a/tooling/check-readme-crates/run.sh b/tooling/check-readme-crates/run.sh new file mode 100755 index 00000000..1ab9729a --- /dev/null +++ b/tooling/check-readme-crates/run.sh @@ -0,0 +1,948 @@ +#!/usr/bin/env bash +# The README's "## Crates" table is hand-maintained prose, and nothing read it. By issue #425 it +# had drifted far enough that twenty-three of its twenty-eight rows needed correcting -- five +# shipped crates were still described as unstarted scaffolding citing a closed issue -- and four +# crates (gamut-codec-abi, gamut-dng, gamut-jpeg, gamut-tonemap) had no row at all. +# +# This guard checks what is mechanical about the table -- its *membership*, the *shape* of a row, +# that the table is a table, and the handful of claim forms `cargo metadata` can decide: +# +# * every workspace crate has a row, so a new crate cannot be added without documenting it; +# * every row names a crate that still exists, so a renamed or deleted crate cannot be left +# behind as a phantom row. A row is recognised by a backticked cargo package name, so the +# character set is cargo's own -- letters, digits, `-` and `_`, in either case -- and a +# phantom row cannot hide behind a capital or an underscore; +# * no crate is listed twice, so the table stays a bijection rather than a set; +# * every `gamut`-prefixed name a Purpose or Status cell backticks is a workspace crate too, so +# a rename leaves no phantom behind in the prose either; +# * every crate row is a three-cell row whose Purpose and Status each *render as something*: +# a cell holding one non-breaking space, one tab, or one empty HTML element is empty, because +# that is what a reader sees; +# * a row inside an HTML comment or a fenced code block is not a row. Both render as something +# other than a table cell, so a crate documented only there is documented nowhere, and the +# membership check must see it as missing rather than as present; +# * EVERY crate row in the section lives inside ONE rendered table. A table is a header row +# immediately followed by a GFM delimiter row of the same width, neither indented four spaces +# or more, and it ends at the first line that is not a table row. That is a claim about the +# whole section rather than about the first row's two neighbours: a blank line between two +# crate rows, a decoy table earlier in the section donating its head, a deleted delimiter, a +# blanked header and a header of the wrong width all leave some or all of the rows rendering +# as a paragraph of literal pipes while each row's bytes stay intact, and all of them fail +# here. +# +# A set of claim FORMS is checked against `cargo metadata`, and they are the only prose this +# guard reads. Each is opt-in and marker-driven: text that writes no marker claims nothing and is +# not checked. Write a claim a reader could check against cargo in one of these forms, or do not +# write it -- a hand-maintained list of crates is exactly the defect this file exists to catch, +# one level down. +# +# The forms are deliberately NOT enumerated here. An enumeration in the documentation of an +# enumeration checker is the one place a hand-written list must not be, and this header was stale +# the moment the fifth check landed: the same commit left it saying "four". `claims()` below is the +# list. Every form is one `match()` in it, tagged and documented at that match site, and every +# failure message names the form it rejects and the shape it wants: +# +# grep -n 'CLAIM FORM:' tooling/check-readme-crates/run.sh +# +# In every name-list form the list ends at the first character that is not a backticked name, a +# comma, a colon, a space or the word "and", so ordinary prose may follow it on the same line. +# +# WHAT THE CONTRACT COVERS. The whole `## Crates` section, not only its rows. The structural +# checks always judged the section; the claim checks do too, so a machine-decidable claim written +# as ordinary prose above or below the table -- or in a `###` sub-heading inside it -- is read +# exactly as it would be inside a cell. Two forms -- the version token and the `consumed +# by`/`always-on` lists -- name no subject of their own and take the row's crate as their subject, +# so outside a crate row they have nobody to be about: writing one there is a failure rather than +# a silent pass, and the message says to move it into a row. The one exception is a fenced or +# indented code block, which is outside every check here: its content is a code sample, not a +# claim this workspace has to answer for. +# +# WHAT THE TABLE IS FORBIDDEN TO WRITE, and why forbidding beats widening. A check that reads +# backticked names can only be as good as the assumption that names are backticked, and that +# assumption was not enforced: an unbackticked `gamut-ifdd` in a cell sailed past the cite check. +# The fix is not a wider pattern -- widening is how a phantom gets in -- but the requirement made +# real. Inside the section: +# +# * a `gamut-`/`gamut_` compound OUTSIDE a code span is a failure, whether or not it names a +# real crate, because that is the precondition every name check rests on. The bare word +# `gamut` is exempt: it is the project's name in English as well as the umbrella's package +# name, and requiring a code span around every mention of it would reject the prose this +# README is made of. A fenced or indented code block is exempt too -- there a compound is a +# sample of Rust or of a shell line, where `gamut_png` is the correct spelling; +# * the underscore spelling of a workspace crate is REJECTED, deliberately, and this is the one +# place that decision is written down. `gamut_ifd` is a Rust identifier, not a cargo package +# name; cargo publishes `gamut-ifd` and `cargo add gamut_ifd` does not resolve. Accepting it +# would also split the guard against itself, because the crate-cell pattern reads `_` on +# purpose so a phantom row cannot hide behind one -- one half would fold the spelling while +# the other treats it as a distinct name. Cells name packages; the umbrella's module aliases +# (`gamut::png`) are how the Rust side is spelled in prose. +# +# CRLF input is accepted: a trailing carriage return is removed from every line before anything is +# matched, so a CRLF README does not silently lose its heading and report itself as rowless. +# +# The `## Crates` heading is matched in every form CommonMark gives it: any ATX level-2 heading +# with the text `Crates`, indented up to three spaces, with or without a closing `##` sequence, or +# the setext form underlined with dashes. Any level-1 or level-2 heading ends the section; a +# `###` or deeper heading does not, so a sub-table inside the section is still part of it. +# +# What is deliberately NOT checked, and why. These are limitations, not oversights, and this +# comment is the one place they are written down: +# * The *wording* of the Purpose and Status cells, beyond the claim forms. They are +# prose a human maintains, and their authority is **the crate's own source** -- the code that +# ships. No single file outranks it: `STATUS.md` can list a shipped module as deferred (issue +# #545) and a module doc can defer a capability the crate's own `EncodeImage` impls already +# provide (issue #560) -- which is why the `gamut-avif` row states 8/10/12-bit encode against +# a doc comment that still calls 10/12-bit deferred. `lib.rs`, `Cargo.toml` and `STATUS.md` +# are where to look first, in that order, but each is a summary and a row must be true of the +# crate, not faithful to a file. A text gate over the cells would fossilise a particular +# phrasing, and a generated table would move prose a human writes into a generator. +# * The header row's own text. Its *presence* directly above the delimiter and its *column +# count* are asserted; what the three column headings are called is prose like any other cell. +# * Anything that needs the network. Whether crates.io serves the version a manifest declares +# cannot be decided here, so the table states the manifest version and [Releases](#releases) +# states the gaps. +# * A row whose crate cell is written in legal Markdown this pattern does not recognise -- an +# omitted leading OR trailing pipe, a linked or annotated crate cell, prose after the code +# span. Each is reported as a *missing crate*, and the failure message says so, because +# widening the pattern is how a phantom row gets in. Narrow and loud beats wide and quiet. +# * Whether an HTML element in a cell renders. Every tag is treated as rendering nothing, so a +# cell whose only content is `` is reported empty. Telling a replaced element from a +# wrapper needs an HTML model, and a crates-table cell that says nothing in plain text +# documents nothing to a reader of the file. +set -euo pipefail + +# With no argument, check the repository's own README from wherever the task was invoked. With +# one, check that file as given -- so a caller can point the guard at a candidate README without +# the working directory changing what "README.md" means. +if [ "$#" -eq 0 ]; then + root="$(git rev-parse --show-toplevel)" || { + echo "check-readme-crates: not inside a git repository, and no README path was given" + exit 1 + } + cd "$root" || exit 1 + readme="README.md" +else + readme="$1" +fi + +test -f "$readme" || { + echo "check-readme-crates: no such file: $readme" + exit 1 +} + +# One pass over the file emits every stream -- `NAME` for membership, `BAD` for shape, `ORPHAN` +# and `SPLIT` for the table context, `VER`/`CONS`/`ALWAYS`/`FEAT`/`CITE` for the claims -- so the +# checks can never disagree about which lines are crate rows. Bounded to the "## Crates" section +# so the README's other tables (the `mise run ...` table) can never be mistaken for a crate row. +# +# LC_ALL=C makes every regexp below byte-wise, which is what the octal escapes assume; the awk +# is POSIX (no gensub, no interval expressions, dynamic regexps built as strings) so it behaves +# the same under mawk, which is what `awk` is on the CI runner. +# +# `CHECK_README_AWK` picks the interpreter, unquoted so `CHECK_README_AWK='gawk --posix'` works. +# The portability claim is only worth what someone can re-run: `fixtures.sh` beside this file +# drives the whole battery through here once per interpreter available on the machine. +scan="$( + LC_ALL=C ${CHECK_README_AWK:-awk} ' + BEGIN { + # An escaped pipe is content, not a column separator, so it is swapped for a control + # byte before the split and swapped back before the cell is judged. + SENTINEL = "\001" + CR = sprintf("%c", 13) + # ASCII space and every C0/DEL control byte. + ASCII_BLANK = "[ \001-\037\177]" + # The UTF-8 encodings of the Unicode blanks a Markdown renderer shows as nothing: + # U+00A0, U+1680, U+2000-U+200F, U+2028, U+2029, U+202F, U+205F, U+2060, U+3000 and + # U+FEFF. Without this fold a single non-breaking space reconstructs exactly the + # degenerate row the shape check exists to reject. + UNI_BLANK = "\302\240|\341\232\200|\342\200[\200-\217\250\251\257]" + UNI_BLANK = UNI_BLANK "|\342\201[\237\240]|\343\200\200|\357\273\277" + # The same blanks written as HTML character references, which a Markdown renderer + # resolves before it shows the cell. The named list is the common set, NOT an + # exhaustive one -- HTML5 names some 2000 references -- but the numeric lists below + # cover every code point the two lists above name, in decimal and in hex, with + # leading zeros allowed. A named reference whose numeric twin was missing was the + # whole hole: `‎` folded while `‎` did not. + ENT_BLANK = "&(nbsp|NonBreakingSpace|ensp|emsp|emsp13|emsp14|numsp|puncsp|thinsp" + ENT_BLANK = ENT_BLANK "|hairsp|VeryThinSpace|MediumSpace|ThickSpace|ThinSpace" + ENT_BLANK = ENT_BLANK "|zwnj|zwj|ZeroWidthSpace|NegativeThinSpace|NoBreak|lrm|rlm);" + ENT_BLANK = ENT_BLANK "|�*(160|5760|819[2-9]|820[0-7]|8232|8233|8239|8287|8288" + ENT_BLANK = ENT_BLANK "|12288|65279);" + ENT_BLANK = ENT_BLANK "|&#[xX]0*([Aa]0|1680|200[0-9A-Fa-f]|202[89Ff]|205[Ff]" + ENT_BLANK = ENT_BLANK "|2060|3000|[Ff][Ee][Ff][Ff]);" + } + + # Returns the part of `line` that is outside an HTML comment, carrying `in_comment` + # across lines so a multi-line comment hides every line it spans. + function strip_comments(line, out, p) { + out = "" + while (length(line) > 0) { + if (in_comment) { + p = index(line, "-->") + if (p == 0) { return out } + in_comment = 0 + line = substr(line, p + 3) + } else { + p = index(line, "