source of truth catalog file

.claude/skills/porting-5-quants/SKILL.md

@@ -1,14 +1,14 @@
 ---
 name: porting-5-quants
-description: Produces the shipped quant matrix from the reference-dtype GGUF, smoke-tests each, publishes the matrix to a private HF repo, and takes a tentative WER read (Modal or local) for human review. Use after porting-4-cpp has finalized tolerances and passed validate.py + the full ref-dtype WER gate. Input: models/<variant>/<variant>-<REFDTYPE>.gguf. Output: F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M alongside the reference-dtype GGUF; a CLI smoke pass per file; quants pushed to a private HF repo; tentative per-quant WER. Authoritative quant WER is Stage 7. No tensor-level numerical comparison is required for quant acceptance — that is intentional.
+description: Produces the shipped quant matrix from the reference-dtype GGUF, smoke-tests each, publishes the matrix to a private HF repo, and takes a preliminary 512-utterance WER read (Modal or local) for human review. Use after porting-4-cpp has finalized tolerances and passed validate.py + the full ref-dtype WER gate. Input: models/<variant>/<variant>-<REFDTYPE>.gguf. Output: F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M alongside the reference-dtype GGUF; a CLI smoke pass per file; quants pushed to a private HF repo; preliminary per-quant WER. Authoritative full-split quant WER is Stage 7. No tensor-level numerical comparison is required for quant acceptance — that is intentional.
 ---
 
 # porting-5-quants
 
 Stage 5 of the porting pipeline. Builds the quantizer, runs
 `scripts/quantize-all.py`, smoke-tests each GGUF, publishes the matrix to a
-private HF repo, and takes a tentative WER read for human review.
-Authoritative quant WER is Stage 7.
+private HF repo, and takes a preliminary 512-utterance WER read for human
+review. Authoritative full-split quant WER is Stage 7.
 
 ## Preconditions
 
@@ -21,7 +21,7 @@ Authoritative quant WER is Stage 7.
 - `build/bin/transcribe-cli` and `build/bin/transcribe-quantize` are
   buildable.
 - `hf` authenticated for the target org (private upload). Modal optional
-  for the tentative WER sweep.
+  for the preliminary 512-utterance WER sweep.
 
 ## Workflow
 
@@ -31,7 +31,7 @@ Quants progress:
 - [ ] Step 2: Run quantize-all
 - [ ] Step 3: CLI output-validity smoke per produced GGUF
 - [ ] Step 4: Publish quants to a private HF repo
-- [ ] Step 5: Tentative WER sweep (Modal or local)
+- [ ] Step 5: Preliminary 512-utterance WER sweep (Modal or local)
 - [ ] Step 6: Sign-off review
 ```
 
@@ -81,35 +81,40 @@ hf repo create <org>/<variant>-gguf --repo-type model --private  # if absent
 hf upload <org>/<variant>-gguf models/<variant> . --repo-type model
 ```
 
-### Step 5: Tentative WER sweep (execute)
+### Step 5: Preliminary 512-utterance WER sweep (execute)
 
-Per-quant WER for human review on the **full acceptance manifest**, not
-a subset. "Tentative" here means "not the published number" (Stage 7
-re-runs and confirms), NOT "small N". Use Modal if credentials are
-available; otherwise run locally. Do not pass `--n-utts` unless you have
-a specific debugging reason and call it out in the sign-off.
+Run each quant on the first 512 utterances of the acceptance manifest. This is
+a bring-up signal only: it catches a clearly bad quant before benchmarking,
+while Stage 7 remains the authoritative full-split publication run. Subset
+artifacts carry `.n512` in their names and are never ingested into the catalog.
+Use Modal if credentials are available; otherwise run locally.
 
 ```bash
-# Modal: sweeps the private repo from Step 4 on GPU (full dataset)
+# Modal: the output paths include .n512.
 modal run scripts/wer/remote/modal_sweep.py::sweep \
-  --models <org>/<variant>-gguf --quants ""
-# local
+  --models <org>/<variant>-gguf --quants "" --n-utts 512
+
+# Local: materialize a separately named subset manifest and output.
+mkdir -p build/wer
+head -n 512 "$MANIFEST" > build/wer/<variant>.<dataset>.n512.manifest.jsonl
 for q in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do
   uv run scripts/wer/run.py --model models/<variant>/<variant>-$q.gguf \
-    --manifest "$MANIFEST" --out reports/wer/<variant>-$q.<dataset>.jsonl
-  uv run scripts/wer/score.py reports/wer/<variant>-$q.<dataset>.jsonl
+    --manifest build/wer/<variant>.<dataset>.n512.manifest.jsonl \
+    --out reports/wer/<variant>-$q.<dataset>.n512.jsonl
+  uv run scripts/wer/score.py reports/wer/<variant>-$q.<dataset>.n512.jsonl
 done
 ```
 
-Report the per-quant WER table for user review before Stage 6.
+Report the preliminary per-quant table for user review before Stage 6. Do not
+copy these rows into `catalog/<variant>.json`; Stage 7 owns published accuracy.
 
 ### Step 6: Sign-off
 
 Report:
 - Every produced GGUF with file size.
 - Any GGUF that failed the CLI smoke (with the failing output).
 - The private HF repo the matrix was pushed to.
-- Tentative per-quant WER table (preliminary; Stage 7 authoritative).
+- Preliminary 512-utterance per-quant WER table (Stage 7 authoritative).
 
 **Do not commit.**
 
@@ -123,8 +128,11 @@ Report:
 - No tensor-level numerical comparison is required (or expected) for
   quant acceptance.
 - Quant matrix pushed to a private HF repo (`<org>/<variant>-gguf`).
-- Tentative per-quant WER produced and reviewed; authoritative WER is
-  Stage 7.
+- Preliminary 512-utterance per-quant WER produced and reviewed; authoritative
+  full-split WER is Stage 7, and only Stage 7 results enter the catalog.
+- The GGUFs are now the input Stage 6 seeds `catalog/<variant>.json` from
+  (`scripts/catalog/new_record.py`), so a wrong capability KV or licence
+  read propagates into the catalog. Fix it here, not in the record.
 
 ## Pointers (read, not execute)
 

.claude/skills/porting-6-bench/SKILL.md

@@ -17,6 +17,8 @@ requirement.
 
 - `models/<variant>/<variant>-<PRESET>.gguf` exists for every shipped
   preset (F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M) — i.e. Stage 5 complete.
+- `catalog/<variant>.json` exists, or Step 0 creates it. Nothing this stage
+  measures can be published without a record to hold it.
 - `build/bin/transcribe-bench` and `build/bin/transcribe-cli` are built
   under `build/`.
 - `scripts/bench/run.py` is runnable.
@@ -47,9 +49,9 @@ Rules:
   rig's own `reports/perf/<slug>/`. Vulkan cells only exist on the Ryzen
   box; Metal cells only exist on the Mac.
 - `reports/` is gitignored (`.gitignore:66`), so the per-rig JSON never
-  travels with the repo. The durable artifact is the **rendered table in
-  `docs/models/<variant>.md`** plus the `perf:` block in
-  `scripts/hf_cards/<variant>.yaml`. Transcribe the numbers into both.
+  travels with the repo. The durable artifact is the speed rows in
+  **`catalog/<variant>.json`**, ingested from those reports; the doc table
+  and the HF card are rendered from them. Never transcribe a number by hand.
 - A dev box that is neither rig (for example a base `apple-m4`) is
   iteration data. It may be added as an extra card section, but it does
   **not** substitute for either required rig.
@@ -60,25 +62,50 @@ Rules:
 
 ## Standardized bench schema
 
-Every per-cell report under
-`reports/perf/<machine>/<name>_<variant>_<backend>.json` is checked by the
-`required_*` sets in Step 5. Missing required fields block Stage 6;
-optional gaps are surfaced but do not gate.
+`scripts/bench/run.py` refuses to write a report under
+`reports/perf/<machine>/<name>_<variant>_<backend>.json` that lacks any field
+the catalog ingester or `compare.py` reads (`report_gaps()` in the driver). A
+report on disk is complete by construction; a refusal is a bench-harness
+regression and halts Stage 6.
 
 ## Workflow
 
 ```
 Bench progress:
+- [ ] Step 0: Catalog record exists (create it on a first port)
 - [ ] Step 1: Confirm full quant matrix present
 - [ ] Step 2: Rebuild transcribe-bench
 - [ ] Step 3: Confirm bench scope (publication default, optional widening)
 - [ ] Step 4: Capture publication baseline
 - [ ] Step 4b: Batch throughput sweep (good-to-know, non-gating)
-- [ ] Step 5: Validate schema completeness
-- [ ] Step 6: Iteration loop (human-driven, with validate gate per accept)
-- [ ] Step 7: Sign-off review
+- [ ] Step 5: Iteration loop (human-driven, with validate gate per accept)
+- [ ] Step 6: Sign-off review
 ```
 
+### Step 0: Catalog record (execute, first port only)
+
+Stage 6 is the first stage that writes to the catalog, so the record has to
+exist before a report can be ingested. A model that was ported before the
+catalog existed already has one; a new port does not.
+
+```bash
+ls catalog/<variant>.json || uv run scripts/catalog/new_record.py <variant> \
+    --long-form <chunked-unbounded|hard-cap|soft-window> --docs-page <family>.md
+```
+
+Everything mechanical is read from artifacts Stage 5 already produced: the
+intake supplies family, upstream repo and revision, and languages; the GGUFs
+under `models/<variant>/` supply the download table, byte sizes, parameter
+count, capability KVs, licence and display name. The two required flags are
+the facts no artifact carries: which `docs/input-limits.md` bucket the family
+falls into, and which page under `docs/models/` documents it. Benchmark rows
+are left empty for this stage and Stage 7 to fill.
+
+Check the seeded record before benching. `--published-repo`,
+`--display-name`, `--license`, `--license-display` and `--language-tag-form`
+override a wrong guess; a capability KV that the GGUF states wrongly is a
+Stage 5 export bug, so fix it there and re-run rather than editing the record.
+
 ### Step 1: Matrix presence (execute)
 
 ```bash
@@ -102,19 +129,18 @@ capturing baseline.
 **Publication scope (default, required for sign-off).** This is the
 matrix that ends up rendered in `docs/models/<variant>.md`:
 
-- Quants: `q8_0,q4_k_m` (the two columns the per-model perf table ships)
-- Samples: `jfk,dots` (short + medium-length sample)
-- Backends: `metal,cpu,vulkan` — `run.py` filters to whatever this machine
-  actually supports (Metal on macOS, Vulkan on Linux with the Vulkan
-  build, CPU everywhere)
-- Iters: `3`, Warmup: `1`
-- `--name <variant>-publication`
-- Rigs: **both** of Apple M4 Max and AMD Ryzen 7 PRO 4750U (see
-  Reference machine matrix above). Run the identical command on each.
+The checked-in profile `asr-publication-v2` is the source of truth. It measures
+the publication quants Q8_0 and Q4_K_M when downloaded, on both `jfk` and
+`dots`, with three iterations after one warmup, using the backends assigned to
+the detected publication machine (M4 Max CPU/Metal or Ryzen 4750U CPU/Vulkan).
+Every selected quant/sample/target cell is required. The Russian-only GigaAM
+variants currently override the defaults with their published short `ru`
+fixture; add a long Russian fixture to those overrides when one is available.
+Add every future mandatory sample to the profile rather than restating the
+matrix as command-line flags.
 
-Confirm publication scope with the user, including which rigs are
-reachable this session. Narrowed or widened sweeps are
-allowed for iteration, but sign-off is decided on publication scope.
+Narrowed or widened sweeps are allowed for iteration, but sign-off is decided
+by the profile.
 
 ### Step 4: Baseline capture (execute)
 
@@ -125,13 +151,7 @@ pollute timings.
 Publication-scope baseline (default):
 
 ```bash
-uv run scripts/bench/run.py \
-  --models <variant> \
-  --quants q8_0,q4_k_m \
-  --samples jfk,dots \
-  --backends metal,cpu,vulkan \
-  --iters 3 --warmup 1 \
-  --name <variant>-publication-baseline-$(date -u +%Y%m%dT%H%M%SZ)
+uv run scripts/bench/run.py --profile --models <variant>
 ```
 
 Writes one report per (variant, backend) pair to `reports/perf/<machine>/`.
@@ -156,36 +176,7 @@ Emits per-batch `{batch_size, per_utt_ms, wall_ms}` to
 `reports/perf/<machine>/<name>_<variant>_batch_<backend>.json`. This is
 exploratory and does not gate sign-off. Batch runs stay strictly serial.
 
-### Step 5: Schema validation (execute)
-
-For each report file, confirm every required field is present and
-surface any optional gaps:
-
-```python
-# uv run python -c '...'
-import json, pathlib, sys
-required_top = {"git_sha", "machine"}
-required_run = {"schema", "model_path", "backend", "per_iter",
-                "sample_duration_s", "rtf_wall_mean", "transcript_sha256"}
-required_iter = {"mel_ms", "encode_ms", "decode_ms", "total_ms", "wall_ms"}
-optional_top = {"git_dirty"}
-optional_run = {"rtf_compute_mean", "token_ids_sha256"}
-for p in sys.argv[1:]:
-    d = json.loads(pathlib.Path(p).read_text())
-    missing_top = required_top - d.keys()
-    missing_run = any(required_run - r.keys() for r in d["runs"])
-    missing_iter = any(required_iter - it.keys() for r in d["runs"] for it in r["per_iter"])
-    absent_opt_top = optional_top - d.keys()
-    absent_opt_run = any(optional_run - r.keys() for r in d["runs"])
-    print(f"{p}: required_top_missing={missing_top} required_run_missing={missing_run} "
-          f"required_iter_missing={missing_iter} optional_absent_top={absent_opt_top} "
-          f"optional_absent_run={absent_opt_run}")
-```
-
-Any missing **required** field is a bench-harness regression — halt Stage
-6 sign-off. Absent **optional** fields are surfaced but do not block.
-
-### Step 6: Iteration loop (human-driven)
+### Step 5: Iteration loop (human-driven)
 
 For each optimization hypothesis:
 
@@ -225,32 +216,57 @@ For each optimization hypothesis:
 
 Repeat until the user is satisfied.
 
-### Step 7: Sign-off
+### Step 6: Sign-off
 
 Report:
 - Baseline reports and machine matrix covered. Name **each of the two
   required rigs** and its state: covered, or `INCOMPLETE — pending`. Do
   not report Stage 6 as complete while either rig is missing.
-- Any schema gaps observed.
 - Total iterations run, net timing improvement, and that every accepted
   iteration passed `validate.py all`.
 
 **Do not commit.** Bench reports under `reports/perf/` may or may not be
 committed at the user's discretion.
 
+## Catalog (mandatory exit step)
+
+`reports/` is gitignored, so a bench report exists only on the machine that
+produced it. The stage is not finished until the numbers are in the catalog.
+
+1. Run the profile-selected publishable measurement, not a hand-written matrix:
+
+   ```bash
+   uv run scripts/bench/run.py --profile --models <variant>
+   ```
+
+   `--profile` stamps both `publication: true` and the profile id in the
+   report. Hypothesis-loop runs (the optimization iterations above) MUST NOT
+   carry it: on CPU they differ from the shipped figure by tens of percent,
+   and the whole point is that they never reach a doc.
+
+2. Fold them in and regenerate the tables:
+
+   ```bash
+   uv run scripts/catalog/ingest_perf.py
+   uv run scripts/catalog/render.py
+   uv run scripts/catalog/check.py --publication-profile --models <variant>
+   ```
+
+   `ingest_perf.py` ingests publication runs only. It refuses a row whose xRT
+   would move more than 5% against what the doc already published, because
+   that is a different build rather than a better reading of the same one;
+   re-bench at the current sha instead of passing `--force`.
+
+3. `check.py` must report every speed row for this variant as measured and
+   sourced. A row carrying a bare xRT with no `engine_sha` is not publishable.
+
 ## Postconditions
 
-- **Both required rigs benched** (Apple M4 Max and AMD Ryzen 7 PRO
-  4750U), each with a bench report covering every **publication-scope**
-  cell (`q8_0`/`q4_k_m` × `jfk`/`dots` × that rig's backends, iters 3,
-  warmup 1) under that rig's `reports/perf/<machine>/`, and each rig's
-  numbers transcribed into `docs/models/<variant>.md` and the `perf:`
-  block of `scripts/hf_cards/<variant>.yaml`. A missing rig is an
-  explicit, user-signed exception, never a silent default. The final on-doc run
-  uses `--name <variant>-publication` so the reproduction command in
-  `docs/models/<variant>.md` matches a real artifact.
-- Schema completeness reported to the user; any gap is a known bench-
-  harness task, not a porting task.
+- `catalog/<variant>.json` exists and is schema-valid.
+- A sourced speed measurement for Q8_0 and Q4_K_M when downloaded, on both
+  `jfk` and `dots`, for every profile machine/backend target. Legacy xRT-only
+  rows may satisfy a cell but are explicitly marked and should be replaced
+  during the long-form/memory sweep.
 - Optimization iteration loop scripted end-to-end (user drives
   hypotheses; skill runs the loop).
 - Every accepted performance iteration was followed by a passing
@@ -266,6 +282,8 @@ committed at the user's discretion.
 ## Pointers (read, not execute)
 
 - `docs/porting/5-benchmarks.md` — bench procedure context
+- `scripts/catalog/new_record.py` — seeds the record from intake + GGUFs
+- `catalog/_schema.json` — what a record is allowed to hold
 - `scripts/bench/run.py` — driver, already discovers `build/bin/` first
 - `scripts/bench/compare.py` — baseline-vs-candidate delta table
 - `tools/transcribe-bench/main.cpp` — bench binary source if the schema